This committ changes the searchable snapshot's CacheService so that
it now periodically fsync cache files using the method introduced in #64201.
The synchronization is executed every 10 seconds by default (this interval
can be changed using a new xpack.searchable.snapshot.cache.sync.interval
setting).
The recovery stats assertions in this test ran without any waiting for
the recoveries to actually finish. The fact that they ran after the concurrent
searches checks generally meant that they would pass (because of searches warming caches
+ general relative slowness of searches) but there is no hard guarantees this will work
reliably as the pre-fetch threads which will update the recovery state might still be slow
to do so randomly, causing the assertions to trip.
closes#65302
If we fail to create the `FileChannelReference` (e.g. because the directory it should be created in
was deleted in a test) we have to remove the listener from the `listeners` set to not trip internal
consistency assertions.
Relates #65302 (does not fix it though, but reduces noise from failures by removing secondary
tripped assertions after the test fails)
Previously it looked like `_source` was exposed as a top-level parameter in
runtime field scripts. The method is only used in tests and can be removed
altogether.
This PR simplifies how the document source is passed to each fetch subphase. A summary of the strategy:
* For each document, we try to eagerly load the source and store it on `HitContext`. Most subphases that access source, like source filtering and highlighting, use `HitContext`. For nested hits, we filter the parent source and also store this source on `HitContext`.
* Only for non-nested documents, we also store the loaded source on `QueryShardContext#lookup`. This allows subphases that access source through `SearchLookup` to use the pre-loaded source when possible. This is now a common occurrence, since runtime fields are supported in the 'fields' option and may soon be supported in highlighting.
There is no longer a special `SearchLookup` just for the fetch phase. This was not necessary and was mostly caused by a misunderstanding of how `QueryShardContext` should be used.
Addresses #62511.
* Clarify that field data cache includes global ordinals
* Describe that the cache should be cleared once the limit is reached
* Clarify that the `_id` field does not supported aggregations anymore
* Fold the `fielddata` mapping parameter page into the `text field docs
* Improve cross-linking
This change fixes a bug where when doing compound assignment involving String concatenation, the
right-hand side will fail to cast to String appropriately and throw a ClassCastException.
In PR #64274, we make it possible to run our Docker images more easily
under a different group, but in so doing increased the Docker layer
sizes dramatically, effectively doubling the size of the images.
Fix this by reworking what commands get run and where, in order to bring
down the final size of the images.
When handling nested documents in the Fetch phase, we currently find
a document's nested identity by loading ObjectMappers from a MapperService,
and iterating over their filters. This logic is difficult to follow, end requires
exposing internals of the mappers that we would rather hide away.
This commit refactors all NestedIdentity loading into a single NestedDocuments
object, with per-segment iterators to handle the individual documents. This way
of iterating over documents allows us to share parent filters, meaning that we
can avoid rebuilding the filter for every nested level of every visited document.
Most of the Elasticsearch Docker build process implements retries
in order to make the process more robust in the face of transient
errors e.g. network failures. However, we missed out the `apk`
command in `master`'s Dockerfile. This PR adds the same retry loop
around `apk` as features elsewhere in the `Dockerfile`.
As part of this, I implemented a helper closure that generates the
same loop construct throughout the `Dockerfile`.
If a _remote/info request is executed on a node without the
remote_cluster_client role, then we will always respond with an empty
result. With this commit, we will fail in such a request instead to
avoid confusing CCS and CCR users.
Security manager policies within plugins currently can ask to grant any
permission (though we block some within the security manager itself at
runtime). Yet most of these permissions should never be necessary, and
some we would actively not want any plugins to be allowed to use. This
commit adds validation of plugins' policy files to restrict the
permissions allowed to be granted to a subset that is reasonable for
plugins to need. The allowed permissions are not ideal (still containing
things like suppressAccessChecks), but it is a step forward in defining
a stricter model for plugins that reduces the surface area of potential
abuse.
Runtime fields are defined in a separate runtime section in the mappings. Since the runtime section was introduced, runtime fields are not taken into account when parsing documents. That means that if a document gets indexed that holds a field that's already defined as a runtime field, the field gets dynamically mapped as a concrete field although it will always be shadowed by the runtime field defined with the same name.
A more sensible default would be to instead consider runtime fields like ordinary mapped fields, so a dynamic update is not necessary whenever a field is defined as part of the runtime section. As a consequence, the field does not get indexed. If users prefer to keep indexing the field although it is shadowed, we consider this an exception, and they can do so by mapping the field under properties explicitly.
Relates to #62906
This commit updates the IndexAbstractionResolver so that hidden indices
are properly resolved when date math is in use and when we are checking
if the index is visible.
Closes#65157
This commit changes code that previously pinned to TLSv1.2 when running
on JDK 12+ to allow the use of TLSv1.3 if on JDK 16 or newer. There was
a bug in the HttpsServer code that has finally been fixed, which
prevented the use of TLSv1.3 as the HttpsServer would endlessly loop.
The JDK bug is JDK-8254967.
Closes#38646
The codebases files are hints that allow the test framework to map
codebase names required by plugin security policy files to urls that may
exist as classes directories on disk. These files end up in the same
test resources directory in gradle, but in eclipse they exist in
different directories. This commit reorganizes the files so they exist
within the same plugin-metadata source, thereby existing in the same
output directory used by tests. Finally, the codebases files are
filtered out of the final jar within the plugin build.
In aa1ea96b86 I made all
`testReduceRandom` tests for aggs mimick production more precisely.
More precisely, they pick the correct "lead" result when performing
partial reduction. This is great, but, sadly, some tests assumed that we
always reduced against the "first" aggregator. This fixes those tests.
Closes#65163
All spots using this method would do a fsync with metadata afterwards
so we might as well do a full fsync right away and save a redundant
fsync + re-opening the file.
When collecting information about analyzers or field types, we do not care much
about usage in system indices which are managed by Elasticsearch, however we
care about indices that are created by users.
A class with 2 fields does not need a builder, especially
when in many cases the builder result is just equivalent to the
`EMPTY` singleton to begin with.
Removed the builder and simplified related code accordingly.
Add syntactic sugar for performing case-insensitive comparison,
including support for the wildcard pattern, against multiple values.
type where field : ("a", "b", "c*")
Fix#64388
When an array is encountered while following a path through the map,
we should only add array values if we're at the final path. So following the
path `foo.bar.baz` shouldn't add concrete values that sit in the array
at `foo.bar`.
Intellij has a special junit runner jar that needs additional security
manager permissions. However, this jar is not present in eclipse. This
commit adds a new (empty) policy for eclipse, and then chooses which to
load based on the presence of the special intellij junit jar. This is a
bit of a hack, but there are no special system properties for intellij
or eclipse that I could find (there are substrings that could be
searched for in existing properties, but looking for a specific key is
much easier to do that searching a substring).
relates #65231
There are still CI failures in watcher related tests even after #65186.
I wasn't able to reproduce it locally at all so this change adds additional logging to pinpoint the problem.
Tanguy Leroux
Andrei Stefan
Armin Braun
Julie Tibshirani
Wylie Conlon
Jack Conradson
Ryan Ernst
István Zoltán Szabó
Rory Hunter
Alan Woodward
Dan Hermann
Howard
Hendrik Muhs
bellengao
Benjamin Trent
Nhat Nguyen
James Rodewig
Luca Cavanna
Jay Modi
Nik Everett
Adrien Grand
Costin Leau
Martijn van Groningen
Rene Groeschke
Henning Andersen
Mark Vieira
DeDe Morton
Przemko Robakowski