Sometimes our fancy "run this agg as a Query" optimizations end up
slower than running the aggregation in the old way. We know that and use
heuristics to dissable the optimization in that case. But it turns out
that the process of running the heuristics itself can be slow, depending
on the query. Worse, changing the heuristics requires an upgrade, which
means waiting. If the heurisics make a terrible choice folks need a
quick way out. This adds such a way: a cluster level setting that
contains a list of queries that are considered "too expensive" to try
and optimize. If the top level query contains any of those queries we'll
disable the "run as Query" optimization.
The default for this settings is wildcard and term-in-set queries, which
is fairly conservative. There are certainly wildcard and term-in-set
queries that the optimization works well with, but there are other queries
of that type that it works very badly with. So we're being careful.
Better, you can modify this setting in a running cluster to disable the
optimization if we find a new type of query that doesn't work well.
Closes#73426
We never do (nor should) call this method to overwrite a snapshots existing entry,
checks and assertions upstream make sure of that.
=> simplified the code accordingly
Today we use `SimpleFSDirectory` in `PersistedClusterStateService` since
we don't need anything fancy. `SimpleFSDirectory` is today deprecated
since it's strictly worse than `NIOFSDirectory` so this commit removes
usages of the deprecated class.
Today we do not set the `LucenePersistedState#writeNextStateFully` flag
on all failures, notably on an `OutOfMemoryError`. Since we don't exit
immediately on an OOME we may have failed part-way through writing a
full state but still proceed with another apparently-incremental write.
With this commit we ensure `LucenePersistedState#writeNextStateFully` is
only set if the previous write was successful.
Serializing and compressing `RepositoryData` seems to have been the wrong trade-off in hindsight.
While saving some heap on a quiet master it makes every repository operation cost heap for
the a newly instantiated `RepositoryData`. Concurrent repository operations and snapshot API
requests can thus easily lead to many duplicate instances on heap causing memory pressure.
Limiting caching to smaller instances also appears to have been the wrong choice in hindsight.
While duplication of a few 100kb instances of `RepositoryData` is mostly not a big deal, duplicating
a `5MB` instance a couple of times (e.g. seen during heavily concurrent get snapshots requests)
eventually becomes a problem.
We've been discussing possibly removing `FieldTypeLookup#getMatchingFieldTypes`, or at least its `SearchExecutionContext` variant that applies runtime mappings.
This is another step in that direction: the exists query can rely on getMatchingFieldNames instead, and look up field types by name.
We've had a series of bug fixes for cases where an OsProbe gives negative
values, most often just -1, to the OsStats class. We added assertions to catch
cases where we were initializing OsStats with bad values. Unfortunately, these
fixes turned to not be backwards-compatible. In this commit, we simply coerce
bad values to 0 when data is coming from nodes that don't have the relevant bug
fixes.
Relevant PRs:
* #42725
* #56435
* #57317Fixes#73459
* Add product response header to all responses
* share header value privately across package
* Make the product header lowercase.
* Do not expose the product header if request is unauthenticated.
* Fix checkstyle
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
Today when upgrading to the next major version we have a so-called
_major version barrier_: once the cluster comprises nodes of the new
major version then nodes of the previous major version are prevented
from joining the cluster. This means we can be certain that
`clusterState.nodes().getMinNodeVersion().major` will never decrease, so
we can implement upgrade logic that relies on the cluster remaining in
its wholly-upgraded state.
This commit generalises this behaviour to apply to all upgrades, so that
we can be certain that `clusterState.nodes().getMinNodeVersion()` will
never decrease in a running cluster.
Closes#72911
This commit fixes a small inconsistency in the FrozenEngine
and Engine classes when they refer to the source used to
open a Searcher for loading doc stats.
This allows indexing documents into a data stream alias.
The ingestion is that forwarded to the write index of the data stream
that is marked as write data stream.
The `is_write_index` parameter can be used to indicate what the write data stream is,
when updating / adding a data steam alias.
Relates to #66163
We use DynamicFieldType to dynamically resolve fields names at their lookup. This is currently used by the flattened field mapper and we intend to use it to expose emitting multiple fields from a runtime field scripts. Those multiple sub-fields will be resolved dynamically, but we need to make a distinction between sub-fields that are known in advance, which we would like to be discoverable e.g. through field_caps at all times, and additional sub-fields that may be resolved at lookup time but are not necessarily known in advance.
For this we are introducing a new method to DynamicFieldType that FieldTypeLookup consults in its getMatchingFieldNames and getMatchingFieldTypes method when the argument is a wilcard pattern.
We recently streamlined support for dynamic field lookups in FieldTypeLookup. That is now used by the flattened field mapper. We would also like to use it for runtime fields, hence this commit adds support for dynamic runtime fields.
This will be useful to support emitting multiple fields from a single runtime field script, as the sub-fields will be dynamically emitted.
Fixes the incorrect assumption in the snapshot state machine that a finished
snapshot delete could only start shard snapshots: in fact it can also move
snapshots to a completed state.
Currently when attempting to an alias to points to both data streams and regular indices
the alias does get created, but points only to data streams. With this change attempting
to add such aliases results in a client error.
Currently when adding data stream aliases with unsupported parameters (e.g. filter or routing)
the alias does get created, but without the unsupported parameters. With this change
attempting to create aliases to point to data streams with unsupported parameters will result
in a client error.
Relates to #66163
This object should be completely immutable. Also added a useful
assertion that makes sure we don't accidentally overwrite a valid
generation with `null` when dealing dealing failed status updates.
Previously, we would always return 0 total hits when there were no groups. Now
that collapsing supports search_after, it's possible for total hits to be
greater than 0 but no groups to return.
This PR also fixes a test bug where we set the wrong missing value for sorting
on doubles.
Fixes#73270.
When a search or search_quote analyzer on a text mapper is not defined,
we fallback to a configured default search/search_quote analyzer if it
exists. However, if an index analyzer has been configured on the mapper
then we should first fall back to that.
Fixes#73333
This method is taking about 4% of CPU time with internal cluster tests
for me. 80% of that were coming from the slow immutability assertion,
the rest was due to the slow way we were building up the new map.
The CPU time slowness likely translates into outright test slowness,
because this was mainly hit through adding transport handlers when starting
nodes (which happens on the main test thread).
Fixed both to save a few % of test runtime.
When data stream aliases are resolved then the includeDataStreams flag of an action request should be taken into account,
so that data stream aliases aren't resolved to backing indices for apis that don't support data streams.
Closes#73195
* Validate that system indices aren't also hidden inidices
* Remove hidden from ingest geo system index
* Add test coverage
* Remove hidden setting from system index even if not upgrading
These settings were deprecated in 7.13+ in #72835 and are now removed by this commit.
This commit also ensures that the settings are removed from index metadata when the metadata is
loaded. The reason for this is that if we allow the settings to remain (because they are not
technically "invalid"), then the index will not be able to be allocated, because the
FilterAllocationDecider will be looking for nodes with the _tier attribute.
Fix a test that was failing to correctly identify that we can't optimize
the `exist` filter on keyword fields. We can't optimize it most of the
time, but it thought we could *sometimes* because we can optimize it
*sometimes*. Specifically, when there are no values for that field in
the segment at all. The test bumped into segments like that and,
correctly, optimized the filter. This changes the test to make sure we
never bump into segments like that when we're asserting that we *can't*
optimize the agg.
Closes#73185
Similar to #73268 we should be stricter here, especially when we are super-strict
about additional fields anyway. Also, use our parser exception utils to get better
exceptions if parsing fails.
Make the restore path a little easier to follow by splitting it up into
the cluster state update and the steps that happen before the CS update.
Also, document more pieces of it and remove some confusing redundant code.
The parsing here was causing trouble with the new streaming deserialization because it
did not fully consume the parser so if the internal buffer of the parser was just enough
to finish reading the `"snapshot"` field but missed the closing bracket, then the stream behind
the parser would not have been consumed fully. Also it was strangely lenient and would just read
a broken in-progress `SnapshotInfo` if it ran into SMILE that contained any object field
under any key that isn't "snapshot". I made it a little stricter now to enforce that
we have a "snapshot" field and not just an object field by any name.
This change adds support for using `search_after` with field collapsing. When
using these in conjunction, the same field must be used for both sorting and
field collapsing. This helps keep the behavior simple and predictable.
Otherwise it would be possible for a group to appear on multiple pages of
results.
Currently search after is handled directly in `CollapsingTopDocsCollector`. As
a follow-up, we could generalize the logic and move support to the Lucene
grouping framework.
Closes#53115.
This commit adjusts the behavior of the Get Aliases API to more
thoroughly prevent errors and warnings from being emitted unnecessarily
from the Get Aliases API by retrieving all indices including system ones
and only warning in the post processing of the action.
Additionally, the IndexAbstractionResolver has been updated to properly
handle system data streams when evaluating visibility.
Closes#73218
Co-authored-by: jaymode <jay@elastic.co>
The get alias api should take into account the aliases parameter when
returning aliases that refer to data streams and don't return entries
for data streams that don't have any aliases pointing to it.
Relates to #66163
This method today corrupts bytes until the checksum changes, but (a)
it's comparing the checksum vs one computed before even reading the
file, and (b) changing a single byte will always invalidate a CRC-32
checksum so the loop is unnecessary as is the checksum calculation. It
also doesn't ever try truncating the file which is a realistic kind of
corruption that we must be able to detect.
This commit addresses all that.
Adds new snapshot meta pool that is used to speed up the get snapshots API
by making `SnapshotInfo` load in parallel. Also use this pool to load
`RepositoryData`.
A follow-up to this would expand the use of this pool to the snapshot status
API and make it run in parallel as well.
Nik Everett
Armin Braun
Ignacio Vera
David Turner
Martijn van Groningen
Luca Cavanna
William Brafford
James Baiera
Tanguy Leroux
Nhat Nguyen
Julie Tibshirani
Julie Tibshirani
Alan Woodward
Przemysław Witek
Mark Vieira
Lee Hinman
Mark Tozzi
Gordon Brown
Joe Gallo
Przemyslaw Gomulka
Ryan Ernst