This PR was originally focused on improving support for Kibana docs, in particular the missing operator docs, but it has expanded to cover a bunch of related things:
* Primarily the main work was to improve operators support. ESQL generated docs cover all functions and most operators for which their is a clear operator class and test class. However, some are built-in behaviour and need additional support. This PR adds more generated content for those operators.
* Various specific operators requested by Kibana: Cast & null-predicates, and in particular the addition of examples
* Two functions without examples: mv_append and to_date_nanos
* Many small visual document cleanups (spelling, grammar, capitalization, etc.)
* Initial support for `applies_to` for multi-version differentiation.
This last point requires more work, as it is not yet agreed on just how we want this to look. We'll probably need to do refinements in followup PR. Consider the version in this PR as a first step into how this could look.
Adds a new cache and setting
TransportGetAllocationStatsAction.CACHE_TTL_SETTING
"cluster.routing.allocation.stats.cache.ttl" to configure the max age
for cached NodeAllocationStats on the master. The default
value is currently 1 minute per the suggestion in issue 110716.
Closes#110716
Did a few things:
* Rewrite Kibana docs asciidoc links to be MD links
* Make kibana docs links absolute to planned publication path
* Clarify which operators are generated and which are static
* Removed the trailing .md from kibana docs links
This commit adds a conversion function from numerics (and aggregate
metric doubles) to aggregate metric doubles.
It is most useful when you have multiple indices, where one index uses
aggregate metric double (e.g. a downsampled index) and another uses a
normal numeric type like long or double (e.g. an index prior to
downsampling).
Earlier work on the ES|QL port of docs to V3 introduced an issue in the build.gradle file making it fail with --configuration-cache. This fixes that, as well as one other broken link and removes some unused files.
In addition we bring back partial support for deleting unused files. It is tricky to have full support for this due to the mix of static and generated content, particularly in the operators snippets.
In a few previous PR's we restructured the ES|QL docs to make it possible to generate them dynamically.
This PR just moves a few files around to make the query languages docs easier to work with, and a little more organized like the ES|QL docs.
A bit part of this was setting up redirects to the new locations, so other repo's could correctly link to the elasticsearch docs.
Clarify that it is expected sometimes to see inter-node connections
sending zero-window advertisements as part of the usual TCP backpressure
mechanism.
This adds a new parameter to the quantized index mapping that allows
default oversampling and rescoring to occur.
This doesn't adjust any of the defaults. It allows it to be configured.
When the user provides `rescore_vector: {oversample: <number>}` in the
query it will overwrite it.
For example, here is how to use it with bbq:
```
PUT rescored_bbq
{
"mappings": {
"properties": {
"vector": {
"type": "dense_vector",
"index_options": {
"type": "bbq_hnsw",
"rescore_vector": {"oversample": 3.0}
}
}
}
}
}
```
Then, when querying, it will auto oversample the `k` by `3x` and rerank
with the raw vectors.
```
POST _search
{
"knn": {
"query_vector": [...],
"field": "vector"
}
}
```
Building on the work started in https://github.com/elastic/elasticsearch/pull/123904, we now want to auto-generate most of the small subfiles from the ES|QL functions unit tests.
This work also investigates any remaining discrepancies between the original asciidoc version and the new markdown, and tries to minimize differences so the docs do not look too different.
The kibana json and markdown files are moved to a new location, and the operator docs are a little more generated than before (although still largely manual).
* Inline cast to date
* Update docs/changelog/123460.yaml
* New capability for `::date` casting
* More tests
* Update tests
---------
Co-authored-by: Fang Xing <155562079+fang-xing-esql@users.noreply.github.com>
Resolves#123053
This adds the thread name to the driver sleep profile output.
---------
Co-authored-by: elasticsearchmachine <infra-root+elasticsearchmachine@elastic.co>
Adds options to QSTR function.
#118619 added named function parameters. This PR uses this mechanism for allowing query string function parameters, so query string parameters can be used in ES|QL.
Closes#120933
The original work at https://github.com/elastic/elasticsearch/pull/106065 did not support geospatial types with this comment:
> I made this work for everything but geo_point and cartesian_point because I'm not 100% sure how to integrate with those. We can grab those in a follow up.
The geospatial types should be possible to collect using the VALUES aggregation with similar behavior to the `ST_COLLECT` OGC function, based on the Elasticsearch convention that treats multi-value geospatial fields as behaving similarly to any geometry collection. So this implementation is a trivial addition to the existing values types support.
* Allow data stream reindex tasks to be re-run after completion
* Docs update
* Update docs/reference/migration/apis/data-stream-reindex.asciidoc
Co-authored-by: Keith Massey <keith.massey@elastic.co>
---------
Co-authored-by: Keith Massey <keith.massey@elastic.co>
* Allow setting the `type` in the reroute processor
This allows configuring the `type` from within the ingest `reroute` processor. Similar to `dataset`
and `namespace`, the type defaults to the value extracted from the index name. This means that
documents sent to `logs-mysql.access.default` will have a default value of `logs` for the type.
Resolves#121553
* Update docs/changelog/122409.yaml
This updates the kibana signature json files in two ways:
* Renames `eval` to `scalar` - that's the name we use inside of ESQL and
we may as well make the name the same.
* Calls the `CATEGORIZE` and `BUCKET` function `grouping` because they
can only be used in the "grouping" positions of the `STATS` command.
Closes#113411
The semantic text format was updated in #119183. This commit removes the last remaining reference to the old format from the documentation to ensure consistency.
This commit introduces the `MappedFieldType#getDefaultHighlighter`, allowing a specific highlighter to be enforced for a field.
The semantic field mapper utilizes this new functionality to set the `semantic` highlighter as the default.
All other fields will continue to use the `unified` highlighter by default.
Today, Elasticsearch supports two models to establish secure connections
and trust between two Elasticsearch clusters:
- API key based security model
- Certificate based security model
This PR deprecates the _Certificate based security model_ in favour of *API key based security model*.
The _API key based security model_ is preferred way to configure remote clusters,
as it allows to follow security best practices when setting up remote cluster connections
and defining fine-grained access control.
Users are encouraged to migrate remote clusters from certificate to API key authentication.
* (Doc+) Expand watermark resolution
Relaunch https://github.com/elastic/elasticsearch/pull/116892 since the original one seems to be outdated and hard to update branch.
* Apply suggestions from code review
Co-authored-by: shainaraskas <58563081+shainaraskas@users.noreply.github.com>
---------
Co-authored-by: shainaraskas <58563081+shainaraskas@users.noreply.github.com>
API keys are high-entropy secure random strings. This means that the
additional work factor of functions like PBKDF or bcrypt are not
necessary, and a faster hash function like salted SHA-256 provides
adequate security against offline attacks (hash collision, brute force,
etc.).
This PR adds `SSHA-256` to the list of supported stored hash algorithms
for API key secrets, and makes it the default algorithm. Additionally,
this PR changes the format of API key secrets, moving from an encoded
UUID to a random string which increase the entropy of API keys from 122
bits to 128 bits, without changing overall secret length.
Relates: ES-9504
With the introduction of our new backing algorithm and making rescoring
easier with the `rescore_vector` API, let's mark bbq as GA.
Additionally, this commit adds rolling upgrade tests to ensure
stability.
Building off of `stats` and multi-value aggregations, including the
limitation:
- all values of extended_stats will be mapped to `double` if mapping
deduction is used
Relates #51925
Enable logsdb by default if logsdb.prior_logs_usage has not been set to true.
Meaning that if no data streams were created matching with the logs-- pattern in 8.x, then logsdb will be enabled by default for data streams matching with logs-*-* pattern.
Also removes LogsPatternUsageService as with version 9.0 and beyond, this component is no longer necessary.
Followup from #120708Closes#106489
* (Doc+) Clarify dimension field requirements for time_series aggregation
👋 howdy, team!
This PR adds a note explaining that time series indices require:
- index.mode set to "time_series"
- at least one dimension field with time_series_dimension: true
- a routing_path array listing those dimension fields
Without these settings, the time_series aggregation may return empty buckets or behave unexpectedly. By emphasizing the dimension field requirement, we help users configure their time series indices correctly and see meaningful aggregation results.
* Apply suggestions from code review
Co-authored-by: shainaraskas <58563081+shainaraskas@users.noreply.github.com>
---------
Co-authored-by: Liam Thompson <32779855+leemthompo@users.noreply.github.com>
Co-authored-by: shainaraskas <58563081+shainaraskas@users.noreply.github.com>
Add documentation for new REST endpoints related to data stream upgrade.
Endpoints:
- /_migration/reindex
- /_migration/reindex/{index}/_status
- /_migration/reindex/{index}/_cancel
- /_create_from/{source}/{dest}
There are many features of the Elasticsearch ecosystem that may malfunction, or fail to work entirely, if these templates are not installed. This commit adds documentation cautioning against disabling the installation of templates.
* [DOCS] Update documentation for index sorting and routing for logsdb
* update
* Apply suggestions from code review
Co-authored-by: Marci W <333176+marciw@users.noreply.github.com>
* Update logs.asciidoc
* Update docs/reference/data-streams/logs.asciidoc
Co-authored-by: Marci W <333176+marciw@users.noreply.github.com>
* Update logs.asciidoc
---------
Co-authored-by: Marci W <333176+marciw@users.noreply.github.com>
Karen Metts
Tommaso Teofili
Colleen McGinnis
Alexander Spies
Craig Taverner
Jeremy Dahlgren
Liam Thompson
Carlos Delgado
Mike Pellegrini
Lisa Cawley
Kofi B
Larisa Motova
Charlotte Hoblik
David Turner
George Wallace
Benjamin Trent
kanoshiou
Mark Tozzi
Jan Calanog
David Kilfoyle
Svilen Mihaylov
Kostas Krikellas
Marci W
Lee Hinman
shainaraskas
John Wagster
Colleen McGinnis
Kathleen DeRusso
Luke Whiting
Nik Everett
Fang Xing
elasticsearchmachine
Jim Ferenczi
Chris Hegarty
Slobodan Adamović
Kuni Sen
Luiz Santos
Nikolaj Volgushev
Michael Peterson
Stanislav Malyshev
Peter Straßer
Pat Whelan
Martijn van Groningen
Valeriy Khakhutskyy
Jihyun(Brian) Jeong
Stef Nestor
Parker Timmins
Panagiotis Bailis
István Zoltán Szabó
Amine GANI
Roberto Seldner
Pius Fung
Sean Story
Sylvain Morin
Joe Gallo
Luigi Dell'Aquila