This replaces the `script` docs for bucket aggregations with runtime
fields. We expect runtime fields to be nicer to work with because you
can also fetch them or filter on them. We expect them to be faster
because their don't need this sort of `instanceof` tree:
a92a647b9f/server/src/main/java/org/elasticsearch/search/aggregations/support/values/ScriptDoubleValues.java (L42)
Relates to #69291
Co-authored-by: James Rodewig <40268737+jrodewig@users.noreply.github.com>
Co-authored-by: Adam Locke <adam.locke@elastic.co>
Currently the canonical form takes into account only the node itself and
not its children.
For commutative cases this creates subtle issues in that the children
are swapped based on their canonical form but not canonicalize and thus
semantic comparison fail.
This PR fixes that by taking into account the canonical children and,
for commutative expressions, applies semantic ordering.
In the process, improve handling of nested negated expressions.
Currently HAVING is modeled as a Filter - while this works well
conceptually, it leads to false positives when validation correct usage
of HAVING (only on aggregate functions or grouped expressions).
This occurs because there's no way to detect whether a filter was
declared as a WHERE or HAVING call - the existing heuristic works in
basic queries but fails when dealing with subqueries.
This PR addresses the issue by introducing a dedicated HAVING node that,
through its definition, clearly disambiguate between regular Filter
(WHERE) and aggregated one (HAVING).
Fix#69758
IsNull/IsNotNull inside a conjunction influences other operations done
on the same expression:
f > 10 AND f IS NULL -> f IS NULL
IFNULL(f + 1, f - 1) AND f IS NOT NULL -> f + 1 AND f IS NOT NULL
Fix#70683
Previously, the search states are stored in ReaderContext on data nodes.
Since 7.10, we send them to the coordinating node in a QuerySearchResult
of a ShardSearchRequest and the coordinating node then sends them back
in ShardFetchSearchRequest. We must keep the search states in data nodes
unless they are sent back in the fetch phase. We used the channel version
to determine this guarantee. However, it's not correct in CCS requests in
mixed clusters.
1. The coordinating node of the local cluster on the old version sends a
ShardSearchRequest to a proxy node of the remote cluster on the new
version. That proxy node delivers the request to the data node. In this
case, the channel version between the data node and the proxy node is
>= 7.10, but we won't receive the search states in the fetch phase as they
are stripped out in the channel between the old coordinating node and
the new proxy.
```
[coordinating node v7.9] --> [proxy node v7.10] --> [data node on v7.10]
```
2. The coordinating node of the local on the new version sends a
ShardSearchRequest to a proxy node of the remote cluster on the new
version. However, the coordinating node sends a ShardFetchSearchRequest
to another proxy node of the remote cluster that is still on an old
version. The search states then are stripped out and never reach the
data node.
```
-> query phase: [coordinating node v7.10] --> [proxy node v7.10] --> [data node on v7.10]
-> fetch phase: [coordinating node v7.10] --> [proxy node v7.9] --> [data node on v7.10]
```
This commit fixes the first issue by explicitly serializing the channel version in
a ShardSearchRequest and the second by continue storing the search states
in ReaderContext unless all nodes are upgraded.
Relates #52741
Clean-up the range optimization rule by avoid the use of indices during
iteration and removal (since it skips an element each time)
Remove redundant ternary calls
* Add warning admonition for removing runtime fields.
* Add cross-link to runtime fields.
* Expanding examples for runtime fields in a search request.
* Clarifying language and simplifying response tests.
We previously allowed but deprecated the ability for the shared cache to
be positively sized on nodes without the frozen role. This is because we
only allocate shared_cache searchable snapshots to nodes with the frozen
role. This commit completes our intention to deprecate/remove this
ability.
If a `max` value in a given composite aggregation bucket is the same as the current after page floor, the datafeed could cancel processing composite aggregation pages too early.
It will see that `max` timestamp aligned with the interval and stop processing. This is a bug. There may be still other terms
to process within that `date_histogram` bucket in subsequent pages as the order of the buckets are done by term NOT by max timestamp.
This commit corrects this to verify that if the process is canceled, the datafeed continues to finish out the current date_histogram bucket, regardless if the first timestamp seen after cancelling aligns with the current page or not.
closes https://github.com/elastic/elasticsearch/issues/71212
In the high level rest client tests, we should clean up trained models, same as we do for datafeeds, jobs, etc.
But, to delete a trained model, it needs to not be used in a pipeline, so we grab the model stats.
This commit does:
- waiting for no more initializing shards when ML integration tests are cleaning up before attempting to delete things
- The specific inference stats test now waits for the `.ml-stats-00001` index to exist before continuing
- And we make sure that config index is green before attempting to run the `pipeline_inference.yml` tests
closes https://github.com/elastic/elasticsearch/issues/71139
When computing node’s ARS rank, we use the number of outstanding search
requests to the node. If there are no connections to the node, we consider
there to be 1 outstanding request. This isn’t accurate, the default should be 0
to indicate no outstanding requests. The ARS rank we return in node stats
actually uses 0 instead of 1.
This small fix lets us remove a test workaround. It also ensures the ARS ranks
we return in node stats match the ranks we use to select shards during search.
Follow-up to #70283.
Today when creating an internal test cluster, we allow the test to
supply the node settings that are applied. The extension point to
provide these settings has a single integer parameter, indicating the
index (zero-based) of the node being constructed. This allows the test
to make some decisions about the settings to return, but it is too
simplistic. For example, imagine a test that wants to provide a setting,
but some values for that setting are not valid on non-data nodes. Since
the only information the test has about the node being constructed is
its index, it does not have sufficient information to determine if the
node being constructed is a non-data node or not, since this is done by
the test framework externally by overriding the final settings with
specific settings that dicate the roles of the node. This commit changes
the test framework so that the test has information about what settings
are going to be overriden by the test framework after the test provide
its test-specific settings. This allows the test to make informed
decisions about what values it can return to the test framework.
There's a bogus assertion in CacheFile that expects a channel to be available whenever we continue writing to it. In case
where the range listener does not care about the write anymore (i.e. all reads are ok), it does not continues to hold onto
the file, however, which is ok.
Closes#71083
The test was originally waiting on a lookback only datafeed to complete and autoclose the job
But, there were times where the lookback completed right when the node was shutting down.
Consequently, the job never closed, but the datafeed did. So, the job never autoclosed.
This commit makes the test's datafeed a realtime datafeed so it always gets reassigned.
closes: https://github.com/elastic/elasticsearch/issues/67756
`readByte` was used for all primitive reads which via the expensive `maybeNextSlice`
which did not inline well caused slowness when reading large aggregate messages.
This commit makes these reads do bulk operations on the underlying `byte[]` where possible
instead. Also, it extracts cold path from `maybeNextSlice` so that we get a fast bounds check
branch instead of the loop with a large body for the hot case.
closes#70800
We have been developing deprecation checks infrastructure in 7.x and not
porting it to master. This commit addresses this by pulling forward some
of the infrastructure that has accreted there.
The ML annotations index is created automatically as soon as
any other ML index is created. It's done this way so that the
ML UI can index into it as soon as ML is being used, but the
index is owned by the backend.
However, if it's detected that the annotations index needs to
be created when upgrade mode is enabled then we shouldn't
instantly create it, as we may be fighting against a user
trying to repair the index in some way.
Additionally, the annotations index should not be created
during a feature reset. Doing this could fight against the
reset process if the annotations index happened to get removed
before some other ML index but then immediately got added back.
(We get away with this at the moment as there is custom test
cleanup code to ensure .ml-annotations is deleted after other
ML indices, but we want ML feature reset to completely and
robustly replace custom ML test cleanup.)
Increases the write buffer size to 2MB, which has shown in benchmarks to provide better throughput on the frozen tier.
Also makes the corresponding cache fetch thread pool size adaptive, as we're otherwise consuming too much memory
on smaller instance nodes. For the instance types (e.g. i3en.2xl) considered on Cloud (8 CPU), the number of
cache_fetch threads stay roughly the same.
Recently merged was a change that modified the default value of: `action.destructive_requires_name` to `true`.
The original merge was fairly careful to adjust all the test clusters to account for the default value change.
However, the machine learning test runner was missed. We create custom node settings and thus require
the handling of the default value change.
relates to https://github.com/elastic/elasticsearch/issues/71150
Nik Everett
James Rodewig
Adam Locke
Henning Andersen
Stuart Tettemer
Joe Gallo
Costin Leau
Nhat Nguyen
Ryan Ernst
Jason Tedor
Benjamin Trent
Jake Landis
Julie Tibshirani
Yannick Welsch
Dan Hermann
Dimitris Athanasiou
Armin Braun
Mark Vieira
David Roberts