This PR adds unit tests and docs for first_over_time, last_over_time,
and rate. For the rate function, the tests currently only verify that
the output is a double, not the actual value.
With the rise of larger CPU count nodes our current write queue size
might be too conservative. Indexing pressure will still provide protect
against out of memories.
* Use `throwInvalidIndexNameException()` to throw invalid ex after
dropping asterisk in `IdentifierBuilder#resolveAndValidateIndex()`
* Assert the message in test
* Refactor
* drop invalid chars from assertion string due to randomisation issue
* Re-assert invalid chars
* Update docs/changelog/130027.yaml
* Update semantic text docs to suggest using index options for customization
* Correct type of index_options
* Move example
* PR feedback
* Copy warning fix
There was a bug in the code for deleting unused and orphan ML data. When deletion using DBQ occurred, the bug caused the request to time out. This PR resolves the issue.
Keep better track of shard contexts using RefCounted, so they can be released more aggressively during operator processing. For example, during TopN, we can potentially release some contexts if they don't pass the limit filter.
This is done in preparation of TopN fetch optimization, which will delay the fetching of additional columns to the data node coordinator, instead of doing it in each individual worker, thereby reducing IO. Since the node coordinator would need to maintain the shard contexts for a potentially longer duration, it is important we try to release what we can eariler.
An even more advanced optimization is to delay fetching to the main cluster coordinator, but that would be more involved, since we need to first figure out how to transport the shard contexts between nodes.
Summary of main changes:
DocVector now maintains a RefCounted instance per shard.
Things which can build or release DocVectors (e.g., LuceneSourceOperator, TopNOperator), can also hold RefCounted instances, so they can pass them to DocVector and also ensure contexts aren't released if they can still be potentially used later.
Driver's main loop iteration (runSingleLoopIteration), now closes its operators even between different operator processing. This is extra aggressive, and was mostly done to improve testability.
Added a couple of tests to TopNOperator and a new integration test EsqlTopNShardManagementIT, which uses the pausable plugin framework to check that TopNOperator releases things as early as possible..
The local plan optimizer should not change the layout, as it has already
been agreed upon. However, CombineProjections can violate this when some
grouping elements refer to the same attribute. This occurs when
ReplaceFieldWithConstantOrNull replaces missing fields with the same
reference for a given data type.
Closes#128054Closes#129811
ES|QL index patterns validation: Ensure that the patterns in the query are syntactically and semantically valid
---------
Co-authored-by: elasticsearchmachine <infra-root+elasticsearchmachine@elastic.co>
Co-authored-by: Andrei Stefan <astefan@users.noreply.github.com>
Co-authored-by: Alexander Spies <alexander.spies@elastic.co>
* [DOCS] Update ESQL metadata fields page
**esql-metadata-fields.md:**
- restructured from bullet list to table format for metadata fields
- added `_index_mode` and `_source` fields to available metadata
- improved field descriptions (more detailed)
- added "usage and limitations" section
- reorganized examples into subsections with headers
- added `_score` sorting example
- added tip box linking to search documentation
* 🚙Drive by updates to search functions ref page
moved tutorial link into tip box at top
added cross-reference to search overview documentation
minor text flow improvements and punctuation fixes
* Fix id typo
* Apply suggestions from review
Co-authored-by: Bogdan Pintea <sig11@mailbox.org>
With the introduction of entitlements (#120243) and exclusive file
access (#123087) it is no longer safe to watch a whole directory.
In a lot of deployments, the parent directory for SSL config files
will be the main config directory, which also contains exclusive files
such as SAML realm metadata or File realm users. Watching that
directory will cause entitlement warnings because it is not
permissible for core/ssl-config to read files that are exclusively
owned by the security module (or other modules)
* Declare LU JOIN GA in 9.1
* Align applies_to tags for sample, change_point
Co-authored-by: Liam Thompson <32779855+leemthompo@users.noreply.github.com>
Introduces a new `RemoveBlock` API that complements the existing `AddBlock` API by allowing users to remove index blocks using `DELETE /{index}/_block/{block}`.
Resolves#128966
---------
Co-authored-by: Niels Bauman <nielsbauman@gmail.com>
The comment in `TransportHandshaker` indicates (correctly) that we emit
a warning when talking to a chronologically-newer-yet-numerically-older
version, but the wording of the warning message is inverted and says
that the remote is chronologically-older-yet-numerically-newer. This
commit straightens out the message to match the situation it is
describing.
Relates #123397
* Update docs for v9.0.3 release
* Add merge scheduler issue to known issues pages
* Add patch release information
---------
Co-authored-by: Charlotte Hoblik <116336412+charlotte-hoblik@users.noreply.github.com>
Co-authored-by: Charlotte Hoblik <sarolta@saroltah.hu>
* Add vector tile examples
* Add new page to TOC
* Add internal translation example
* Update docs/reference/elasticsearch/rest-apis/vector-tile-search.md
Co-authored-by: István Zoltán Szabó <istvan.szabo@elastic.co>
* Update docs/reference/elasticsearch/rest-apis/vector-tile-search.md
Co-authored-by: István Zoltán Szabó <istvan.szabo@elastic.co>
---------
Co-authored-by: István Zoltán Szabó <istvan.szabo@elastic.co>
This adds `counter_long` and `counter_double` to the signatures of
supported fields for `IS NULL` and `IS NOT NULL`. We hadn't been
generating those signatures since the docs v3 migration, so this had to
plug those in. In addition, it changes the wording on a few things and
adds a note that if a field is only in some documents then the ones
missing the field will have `NULL` - which is important information for
`IS NULL` and `IS NOT NULL`.
* Initial checkin of refactored index_options code
* [CI] Auto commit changes from spotless
* initial unit testing
* complete unit tests; add yaml tests
* [CI] Auto commit changes from spotless
* register test feature for sparse vector
* Update docs/changelog/129089.yaml
* update changelog
* add docs
* explicit set default index_options if null
* [CI] Auto commit changes from spotless
* update yaml tests; update docs
* fix yaml tests
* readd auth for teardown
* only serialize index options if not default
* [CI] Auto commit changes from spotless
* serialization refactor; pass index version around
* [CI] Auto commit changes from spotless
* fix transport versions merge
* fix up docs
* [CI] Auto commit changes from spotless
* fix docs; add include_defaults unit and yaml test
* [CI] Auto commit changes from spotless
* override getIndexReaderManager for SemanticQueryBuilderTests
* [CI] Auto commit changes from spotless
* cleanup mapper/builder/tests; index vers. in type
still need to refactor / clean YAML tests
* [CI] Auto commit changes from spotless
* cleanups to mapper tests for clarity
* [CI] Auto commit changes from spotless
* move feature into mappers; fix yaml tests
* cleanups; add comments; remove redundant test
* [CI] Auto commit changes from spotless
* escape more periods in the YAML tests
* cleanup mapper and type tests
* [CI] Auto commit changes from spotless
* rename mapping for previous index test
* set explicit number of shards for yaml test
---------
Co-authored-by: elasticsearchmachine <infra-root+elasticsearchmachine@elastic.co>
Co-authored-by: Kathleen DeRusso <kathleen.derusso@elastic.co>
Send the Elastic API Payload to a SageMaker endpoint, and parse the
response as if it were an Elastic API response.
- SageMaker now supports all task types in the Elastic API format.
- Streaming is supported using the SageMaker client/server rpc,
rather than SSE. Payloads must be in a complete and valid JSON
structure.
- Task Settings can be used for additional passthrough settings, but
they will not be saved alongside the model. Elastic cannot make
guarantees on the structure or contents of this payload, so Elastic
will treat it like the other input payloads and only allow them during
inference.
* [DOCS][ESQL] Flip preview booleans, to GA search functions
* render docs, tweak some applies_to metadata in docs gen code
- **rendered docs (md):**
- kql: removed serverless preview, added ga 9.1.0
- match: removed serverless preview, added ga 9.1.0
- match_phrase: changed from preview 9.1.0 to unavailable 9.0 + ga 9.1.0
- qstr: removed serverless preview, added ga 9.1.0
- search functions list: removed bullet point before term function
- **docs generation code (java):**
- match_phrase: updated function info annotations to unavailable 9.0 + ga 9.1.0
- query_string: uncommented ga 9.1.0 annotation
Ioana Tagirta
Gal Lalouche
Nhat Nguyen
Graeme Mjehovich
Martijn van Groningen
Lisa Cawley
David Kilfoyle
Mike Pellegrini
Tim Brooks
Artem Shelkovnikov
Kathleen DeRusso
Simon Cooper
Nick Tindall
James Baiera
Pawan Kartik
Valeriy Khakhutskyy
Stanislav Malyshev
Stef Nestor
elasticsearchmachine
Lorenzo Dematté
Liam Thompson
Tim Vernum
Alexander Spies
David Kyle
HYUNSANG HAN (한현상, Travis)
elasticsearchmachine
David Turner
Mark J. Hoy
Panagiotis Bailis
Charlotte Hoblik
Nik Everett
Pat Whelan