We ran into a situation where dynamic mapping updates where retried in a
fairly hot loop. The problem that triggered this was waiting for any cluster state
update in this logic. This is mostly fine but adds a lot of overhead for
retries when there's other actions running at a higher priority than the
mapping update. Lets make it specific so that we at least wait for there
to be any mapping and for its version to be different from the version
that made us request a mapping update in the first place.
Also added a breakout in case the index got concurrently deleted so we
don't run out the clock in that case.
Fixes https://github.com/elastic/elasticsearch/issues/110184
When a DISSECT command overwrites the input, eg.
```
FROM idx | DISSECT foo "%{foo} %{bar}" | KEEP foo, bar
```
The input field (`foo` in this case) could be excluded from the index
resolution (incorrectly masked by the `foo` that is the result of the
DISSECT). This PR makes sure that the input field does not get lost and
is correctly passed to the indexResolver.
This commit adds `bit` vector support by adding `element_type: bit` for
vectors. This new element type works for indexed and non-indexed
vectors. Additionally, it works with `hnsw` and `flat` index types. No
quantization based codec works with this element type, this is
consistent with `byte` vectors.
`bit` vectors accept up to `32768` dimensions in size and expect vectors
that are being indexed to be encoded either as a hexidecimal string or a
`byte[]` array where each element of the `byte` array represents `8`
bits of the vector.
`bit` vectors support script usage and regular query usage. When
indexed, all comparisons done are `xor` and `popcount` summations (aka,
hamming distance), and the scores are transformed and normalized given
the vector dimensions. Note, indexed bit vectors require `l2_norm` to be
the similarity.
For scripts, `l1norm` is the same as `hamming` distance and `l2norm` is
`sqrt(l1norm)`. `dotProduct` and `cosineSimilarity` are not supported.
Note, the dimensions expected by this element_type are always to be
divisible by `8`, and the `byte[]` vectors provided for index must be
have size `dim/8` size, where each byte element represents `8` bits of
the vectors.
closes: https://github.com/elastic/elasticsearch/issues/48322
Rate aggregation is special because it must be computed per time series,
regardless of the grouping keys. The keys must be `_tsid` or a pair of
`_tsid` and `time_bucket`. To support user-defined grouping keys, we
first execute the rate aggregation using the time-series keys, then
perform another aggregation with the resulting rate using the
user-specific keys.
This PR translates the aggregates in the METRICS commands to standard
aggregates. This approach helps avoid introducing new plans and
operators for metrics aggregations only.
Examples:
**METRICS k8s max(rate(request))** becomes:
```
METRICS k8s
| STATS rate(request) BY _tsid
| STATS max(`rate(request)`)
```
**METRICS k8s max(rate(request)) BY host** becomes:
```
METRICS k8s
| STATS rate(request), VALUES(host) BY _tsid
| STATS max(`rate(request)`) BY host=`VALUES(host)`
```
**METRICS k8s avg(rate(request)) BY host** becomes:
```
METRICS k8s
| STATS rate(request), VALUES(host) BY _tsid
| STATS sum=sum(`rate(request)`), count(`rate(request)`) BY host=`VALUES(host)`
| EVAL `avg(rate(request))` = `sum(rate(request))` / `count(rate(request))`
| KEEP `avg(rate(request))`, host
```
**METRICS k8s avg(rate(request)) BY host, time_bucket=bucket(\@timestamp, 1minute)** becomes:
```
METRICS k8s
| EVAL `bucket(@timestamp, 1minute)`=datetrunc(@timestamp, 1minute)
| STATS rate(request), VALUES(host) BY _tsid,`bucket(@timestamp, 1minute)`
| STATS sum=sum(`rate(request)`), count(`rate(request)`) BY host=`VALUES(host)`, `bucket(@timestamp, 1minute)`
| EVAL `avg(rate(request))` = `sum(rate(request))` / `count(rate(request))`
| KEEP `avg(rate(request))`, host, `bucket(@timestamp, 1minute)`
```
Add event.ingested min/max range to cluster state for searchable snapshots in order
to support shard skipping on the search coordinator node for event.ingested
like we do for the @timestamp field.
The key motivation is that the Elastic Security solution uses two timestamp fields:
@timestamp which is user populated and event.ingested which is populated by
an ingest processor which tells the time that the event was ingested.
In some cases, queries are filtered by @timestamp, while in some others they are filtered
by event.ingested. When data is stored in the cold or frozen tier, we have a shard skipping
mechanism on the coordinating node, which allows the coordinating node to skip shards
based on min and max value stored in the cluster state for the @timestamp field. This is
done to prevent returning irrelevant errors in case shards in frozen/cold are unavailable
(as they have no replicas) and queries that don’t target them (as the datetime filter won’t
match them) are executed.
This works well for the @timestamp field and the Security solution could benefit from having
the same skipping mechanism applied to the event.ingested field.
Note that it is important that the value of the event.ingested range in IndexMetadata
in cluster state be set to UNKNOWN when the min cluster TransportVersion is less
than EVENT_INGESTED_RANGE_IN_CLUSTER_STATE.
No point in wrapping in unmodifiable here on every call, just adds
allocations and we escape the mutable iterator anyway. Also we dont't
need to call `asList` to iterate to begin with.
We are only called the method every 10K buckets, so for deeply nested aggregations might never call the method
although the final number of buckets can be enormous.
This drops a bunch of unused code in the esql-core package that was used
to power SQL and EQL. We copied it when we forked the shared ql code
into esql-core. But we don't need it.
This causes the ESQL tests to be ok with a different cancellation
message in the cancellation tests - this one is just as descriptive as
the previous allowed ones though I'm not entirely sure where it comes
from. But that's ok - it tells us the request is cancelled.
Closes#109890
This adds a metric to count the total number of search responses generated by the search action and
the scroll search action. This uses the key es.search_response.response_count.total. Each response
also has an attribute to specify whether the response was a success, a partial failure, or a failure.
Follow up to #109227. As discussed there, this makes it clear what is being excluded from the type list. It also makes adding new types easier, as implementers do not need to remember to add the type to the list.
I do not know why these two data types (DOC_DATA_TYPE and TSID_DATA_TYPE) are excluded from this list. It was like that when I found it.
The test that is set up assumes a single shard. Since the test uses so
few vectors and few dimensions, the statistics are pretty sensitive.
CCS tests seem to allow more than one write shard (via more than one
cluster). Consequently, the similarity detected can vary pretty wildly.
However, through empirical testing, I found that the desired vector
seems to always have a score > 0.0034 and all the other vectors have a
score < 0.001. This commit adjusts this similarity threshold
accordingly. This should make the test flakiness go away in CCS testing.
closes: https://github.com/elastic/elasticsearch/issues/109881
Sort fields are usually rewritten into their merge form during transport serialization. For searches where all shards are local to the coordinating node, this serialization is not applied, and sort fields remain in their original form. This is typically not an issue except when checking sort compatibility, as we rely on the primary type of the sort field.
This change ensures we extract the correct type even if the sort field is in its primary form (with a CUSTOM type) using the original comparator source. The only field type that benefits from this change afaik is the constant_keyword field type. When a sort field is a mix of numeric and constant_keyword, this change ensures the discrepancy is correctly reported to the user.
There were some optimizations that broke collapse fields automatically
being added to `docvalue_fields` during the fetch phase.
Consequently, users will get really weird errors like
`unsupported_operation_exception`. This commit corrects the intended
behavior of automatically including the collapse field in the
docvalue_fields context during fetch if it isn't already included.
closes: https://github.com/elastic/elasticsearch/issues/96510
Similar to https://github.com/elastic/elasticsearch/pull/110036, this
creates the opportunity for the compiler to flag this as a place which
needs to be changed when we add a data type. It's also fewer lines of
code, and makes it explicit which types are not supported.
Ideally, this would also become a property on DataType, but that can't
happen until we pull it out of core.
Armin Braun
Tim Rühsen
Luigi Dell'Aquila
Huaixinww
Nick Tindall
Nik Everett
elasticsearchmachine
Benjamin Trent
Nhat Nguyen
Michael Peterson
Oleksandr Kolomiiets
Simon Cooper
Ignacio Vera
István Zoltán Szabó
Tim Grein
Kostas Krikellas
Rene Groeschke
David Kyle
Pius
Quentin Pradet
Jack Conradson
Mark Tozzi
Kathleen DeRusso
Jim Ferenczi