Enhance ES|QL responses to include information about `took` time (search latency), shards, and
clusters against which the query was executed.
The goal of this PR is to begin to provide parity between the metadata displayed for
cross-cluster searches in _search and ES|QL.
This PR adds the following features:
- add overall `took` time to all ES|QL query responses. And to emphasize: "all" here
means: async search, sync search, local-only and cross-cluster searches, so it goes
beyond just CCS.
- add `_clusters` metadata to the final response for cross-cluster searches, for both
async and sync search (see example below)
- tracking/reporting counts of skipped shards from the can_match (SearchShards API)
phase of ES|QL processing
- marking clusters as skipped if they cannot be connected to (during the field-caps
phase of processing)
Out of scope for this PR:
- honoring the `skip_unavailable` cluster setting
- showing `_clusters` metadata in the async response **while** the search is still running
- showing any shard failure messages (since any shard search failures in ES|QL are
automatically fatal and _cluster/details is not shown in 4xx/5xx error responses). Note that
this also means that the `failed` shard count is always 0 in ES|QL `_clusters` section.
Things changed with respect to behavior in `_search`:
- the `timed_out` field in `_clusters/details/mycluster` was removed in the ESQL
response, since ESQL does not support timeouts. It could be added back later
if/when ESQL supports timeouts.
- the `failures` array in `_clusters/details/mycluster/_shards` was removed in the ESQL
response, since any shard failure causes the whole query to fail.
Example output from ES|QL CCS:
```es
POST /_query
{
"query": "from blogs,remote2:bl*,remote1:blogs|\nkeep authors.first_name,publish_date|\n limit 5"
}
```
```json
{
"took": 49,
"columns": [
{
"name": "authors.first_name",
"type": "text"
},
{
"name": "publish_date",
"type": "date"
}
],
"values": [
[
"Tammy",
"2009-11-04T04:08:07.000Z"
],
[
"Theresa",
"2019-05-10T21:22:32.000Z"
],
[
"Jason",
"2021-11-23T00:57:30.000Z"
],
[
"Craig",
"2019-12-14T21:24:29.000Z"
],
[
"Alexandra",
"2013-02-15T18:13:24.000Z"
]
],
"_clusters": {
"total": 3,
"successful": 2,
"running": 0,
"skipped": 1,
"partial": 0,
"failed": 0,
"details": {
"(local)": {
"status": "successful",
"indices": "blogs",
"took": 43,
"_shards": {
"total": 13,
"successful": 13,
"skipped": 0,
"failed": 0
}
},
"remote2": {
"status": "skipped", // remote2 was offline when this query was run
"indices": "remote2:bl*",
"took": 0,
"_shards": {
"total": 0,
"successful": 0,
"skipped": 0,
"failed": 0
}
},
"remote1": {
"status": "successful",
"indices": "remote1:blogs",
"took": 47,
"_shards": {
"total": 13,
"successful": 13,
"skipped": 0,
"failed": 0
}
}
}
}
}
```
Fixes https://github.com/elastic/elasticsearch/issues/112402 and https://github.com/elastic/elasticsearch/issues/110935
Lucene 10 has upgraded its Snowball stemming support, as part of those
upgrades, two no longer supported stemmers were removed, `KpStemmer` and
`LovinsStemmer`. These are `dutch_kp` and `lovins`, respectively.
We will deprecate in 8.16 and will remove support for these in a future
version.
* Add new Previous Step Info field to LifecycleExecutionState
* Add new field to IndexLifecycleExplainResponse
* Add new field to TransportExplainLifecycleAction
* Add logic to IndexLifecycleTransition to keep previous setp info
* Switch tests to use Java standard Clock class
for any time based testing, this is the recommended method
* Fix tests for new field
Also refactor tests to newer style
* Add test to ensure step info is preserved
Across auto retries
* Add docs for new field
* Changelog Entry
* Update docs/changelog/113187.yaml
* Revert "Switch tests to use Java standard Clock class"
This reverts commit 241074c735.
* PR Changes
* PR Changes - Improve docs wording
Co-authored-by: Mary Gouseti <mgouseti@gmail.com>
* Integration test for new ILM explain field
* Use ROOT locale instead of default toLowerCase
* PR Changes - Switch to block strings
* Remove forbidden API usage
---------
Co-authored-by: Mary Gouseti <mgouseti@gmail.com>
Creating an endpoint for the built in multilingual e5 model failed for
linux optimised version due to an error in the logic that checks model
compatibility.
We are current using Float.MIN_VALUE which is the smallest positive non-zero value, not the smallest possible value.
This commit change it to -Float.MAX_VALUE to be symmetric to the double max aggregation.
Adds a new option trace_redact in redact processor to indicate a document has been redacted in the ingest pipeline. If a document is processed by a redact processor AND any field is redacted, ingest metadata _ingest._redact._is_redacted = true will be set.
Closes#94633
* Note in docs about incorrect IO stats when running in docker
* Update docs/reference/cluster/nodes-stats.asciidoc
Co-authored-by: David Turner <david.turner@elastic.co>
* Requested PR changes to wording
* Update docs/reference/cluster/nodes-stats.asciidoc
Co-authored-by: David Turner <david.turner@elastic.co>
---------
Co-authored-by: David Turner <david.turner@elastic.co>
Resolves#111842
This adds a conversion function that yields DATE_NANOS. Mostly this is straight forward.
It is worth noting that when converting a millisecond date into a nanosecond date, the conversion function truncates it to 0 nanoseconds (i.e. first nanosecond of that millisecond). This is, of course, a bit of an assumption, but I don't have a better assumption we can make. I'd thought about adding a second, optional, parameter to control this behavior, but it's important that TO_DATE_NANOS extend AbstractConvertFunction, which itself extends UnaryScalarFunction, so that it will work correctly with union types. Also, it's unlikely the user will have any better guess than we do for filling in the nanoseconds.
Making that assumption does, however, create some weirdness. Consider two comparisons:
TO_DATETIME("2023-03-23T12:15:03.360103847") == TO_DATETIME("2023-03-23T12:15:03.360") will return true while TO_DATE_NANOS("2023-03-23T12:15:03.360103847") == TO_DATE_NANOS("2023-03-23T12:15:03.360") will return false. This is akin to casting between longs and doubles, where things may compare equal in one type that are not equal in the other. This seems fine, and I can't think of a better way to do it, but it's worth being aware of.
---------
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
Collapse dynamically will add values to the DocumentField values array.
There are a few scenarios where this is immutable and most of these are
OK. However, we get in trouble when we create an immutable set for
StoredValues which collapse later tries to update.
The other option for this fix was to make an array copy for `values` in
every `DocumentField` ctor, this seemed very expensive and could get out
of hand. So, I decided to fix this one bug instead.
closes https://github.com/elastic/elasticsearch/issues/112646
Deprecate to, from, include_lower, include_upper range query params.
These params have been removed from our documentation in v. 0.90.4 (d6ecdecc19),
but did not got through deprecation cycle.
These params to be removed in v9.0.
Related to #81276Closes#48538
Restores the changes from #111684 which uses multiple streams to improve the
time to download and install the built in ml models. The first iteration has a problem
where the number of in-flight requests was not properly limited which is fixed here.
Additionally there are now circuit breaker checks on allocating the buffer used to
store the model definition.
Adds a search_inference_id parameter to the semantic_text mapping. This parameter defines the inference endpoint that is used to generate embeddings at query time.
This speeds up the `CASE` function when it has two or three arguments
and both of the arguments are constants or fields. This works because
`CASE` is lazy so it can avoid warnings in cases like
```
CASE(foo != 0, 2 / foo, 1)
```
And, in the case where the function is *very* slow, it can avoid the
computations.
But if the lhs and rhs of the `CASE` are constant then there isn't any
work to avoid.
The performance improvment is pretty substantial:
```
(operation) Before Error After Error Units
case_1_lazy 97.422 ± 1.048 101.571 ± 0.737 ns/op
case_1_eager 79.312 ± 1.190 4.601 ± 0.049 ns/op
```
The top line is a `CASE` that has to be lazy - it shouldn't change. The
4 nanos change here is noise. The eager version improves by about 94%.
There may be many older indices in need of merging, but today we do not
throttle this work across shards so an upgrade could lead to an
overwhelming spike in merges. With this commit we make it so that the
automatic merge-on-recovery behaviour only applies to newly-created
indices.
Current ensureGreen test helper method uses client() directly.
Sometimes is useful to call ensureGreen with adminClient() or
another rest client. This PR allows passing a RestClient into
ensureGreen.
This implements the `parseBytesRef` method for the `_ts_routing_hash` field so we
can parse the values generated by the companion `format` method.
We parse the values when fetching them from the source when the field is used
as a `sort` paired with `search_after`.
Before this change a sort by and search_after `_ts_routing_hash` would yield
an `UnsupportedOperationException`
Create `POST _inference/<task>/<id>/_stream` and
`POST _inference/<id>/_stream` API.
REST Streaming API will reuse InferenceAction.
For now, all services and task types will return an
HTTP 405 status code and error message.
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
Allows setting index total_shards_per_node in the SearchableSnapshot action of ILM to remediate hot spot in shard allocation for searchable snapshot index.
Closes#112261
Here we introduce a new index-level setting, `ignore_above`, similar to what we have
for `ignore_malformed`. The setting will apply to all `keyword`, `wildcard` and `flattened`
fields. Each field mapping will still be allowed to override the index-level setting using a
mapping-level `ignore_above` value.
Ignacio Vera
Luca Cavanna
Michael Peterson
Benjamin Trent
Stanislav Malyshev
Patrick Doyle
Liam Thompson
Luke Whiting
Simon Cooper
David Kyle
István Zoltán Szabó
Kostas Krikellas
Sam Xiao
Nikolay Benlioglu
Mike Pellegrini
Carson Ip
Oleksandr Kolomiiets
elasticsearchmachine
Fang Xing
Mark Tozzi
Stef Nestor
kosabogi
Keith Massey
Mayya Sharipova
Smriti
Luigi Dell'Aquila
Nik Everett
David Turner
Parker Timmins
Andrei Dan
Valeriy Khakhutskyy
Pat Whelan
Salvatore Campagna
YeonghyeonKo