Some indices requests support expanding wildcards. During authorization,
wildcard expansion is performed against the list of authorized indices
that the user has access to. The expansion (resolvedIndices) are then
used to replace the original indices. These indices requests shares a
common interface of IndicesRequest.Replaceable.
The observation is that indices requests that are not Replaceable do not
need expanding wildcards which in turn remove the need for the list of
authorized indices.
This PR skips loading of authorized indices when the indices requests
are not replaceable, e.g. ShardSearchRequest,
FieldCapabilitiesIndexRequest. This should help the node on the receving
end of the request handling, i.e. handler of fan-out requests.
* Index prefixes for searchable snapshots
added a note about how ILM managed indices are prefixed with "restored-" or "partial-" when they are either fully or partially mounted for searchable snapshots
* Apply suggestions from code review
Co-authored-by: debadair <debadair@elastic.co>
This reverts commit fd138f578b.
The revert was effectively conflict free, there was only a very minor
static import that needed fixing.
relates #78525
relates #71205
The test base class ESTestCase provides methods to get a random
timezone. It has methods for Joda, java.time and java.util timezones.
This commit reworks the test helper methods to no longer use Joda. Due
to backcompat tests for sql, the random timezones need to be limited to
those that are still useable by Joda. To filter the available timezones
not available in Joda, the unsupported timezone ids are hardcoded in
testcase initialization.
Add a deprecation warning and a system property es.cluster_health.request_timeout_200 to opt in for returning 200 which will be the default in 8.0.0
Fixes#70849
This reverts commit 6a7298e555.
The revert was not clean. There were two adjustements necessary.
First, this effectively must revert #73136. Second, some tests
were written after MDP removal so needed to be made multi path aware.
relates #78525
relates #71205
This fixes a bug where the range aggregation always treats the range end points as doubles, even if the field value doesn't have enough resolution to fill a double. This was creating issues where the range would have a "more precise" approximation of an unrepresentable number than the field, causing the value to fall in the wrong bucket.
Note 1: This does not resolve the case where we have a long value that is not precisely representable as a double. Since the wire format sends the range bounds as doubles, by the time we get to where this fix is operating, we've already lost the precision to act on a big long. Fixing that problem will require a wire format change, and I'm not convinced it's worth it right now.
Note 2: This is probably still broken for ScaledFloats, since they don't implement NumberFieldType.
Resolves#77033
Currently we exclude metadata fields from being looked up using the `fields` option in search.
However, as issue like #75836 show, they can still be retrieved e.g. via aliases and then fetching
their values causes errors.
With this change, we enable retrieval of metadata fields (like `_id`) using the fields option when
the field is explicitely asked for. We still continue to exclude any metadata field from matching
wildcard patterns, but they should be retrievable via an exact name or if there is an alias definition
with a path to a metadata field. This change adds support for the `_id` field in particular.
Fix for wildcard field query optimisation that rewrites to a match all for regexes like .*
A bug was found in this complex rewrite logic so we have simplified the detection of .* type regexes by examining the Automaton for the regex rather than our parsed form of it which is expressed as a Lucene BooleanQuery. The old logic relied on a recursive "simplify" function on the BooleanQuery which has now been removed. We now rely on Lucene's query rewrite logic to simplify expressions at query time and consequently some of the tests had to change to do some of this rewriting before running test comparisons.
Closes#78391
This is mostly motivated by the ILM steps changes, that showed up as hot in
profiling since instantiating a `Settings` instance turns out to be somewhat expensive.
Also cleans up a couple of other spots and some duplication as well.
Allow passing FileCollection instead of single Jar files. This makes using the API way easier
as gradle configurations for resolving jars do not need to be resolved eagerly
Allow individual queries within a sequence to be repeated through a
dedicated keyword without having physical duplication.
Change from using [runs=2] to "with runs=2"
Before:
sequence
queryA [runs=2]
queryB
queryC [runs=3]
queryD
Now:
sequence
queryA with runs=2
queryB
queryC with runs=3
queryD
Which essentially is the same as:
sequence
queryA
queryA
queryB
queryC
queryC
queryC
queryD
but more concise.
Supersedes #75082
Fix the split package org.elasticsearch.common.xcontent, between server and the x-content lib. Move the x-content lib exported package from org.elasticsearch.common.xcontent to org.elasticsearch.xcontent ( following the naming convention of similar libraries ). Removing split packages is a prerequisite to modularization.
The data tier preference is very expensive to parse out of the setting string
repeatedly for large number of indices when using it in the data tier allocation decider.
=> as done with other index settings relevant to allocation, this commit moves the data tier
preference to a field in `IndexMetadata`. The required moving the `DataTier` class itself to
`server`. In a follow-up we can look into making the setting a list setting to remove the
duplication around turning the string value into a list in various places.
* Increase node shutdown logging level when stalled
* Add allocation explanation to STALLED response
* Adjust logging levels per review
* Include all SingleNodeShutdownMetadata fields in logs
* Depluralize `node_shutdown_decisions` per review
* Repluralize node_allocation_decisions when we're actually reading from the Allocation Explain API
This reverts commit 2dc796a747.
The revert was mostly clean, but it required adjusting
NodeEnvironment.upgradeLegacyNodeFolders to retain #74921.
relates #78525
relates #71205
* Implement and test get feature upgrade status API
* Add integration test for feature upgrade endpoint
* Use constant enum for statuses
* Add unit tests for transport class methods
- Add test about partial failure on multi-paths
- Add test for deleteMetaState
Co-authored-by: Henning Andersen <33268011+henningandersen@users.noreply.github.com>
This adds support for the headers necessary for REST version compatibility to the High Level Rest
Client (HLRC).
Compatibility mode can be turned on either with the .setAPICompatibilityMode(true) when creating
the client, or by setting the ELASTIC_CLIENT_APIVERSIONING to true similar to our other
Elasticsearch clients.
Resolves#77859
* WIP, basic implementation
* Pull `if` branch into a variable
* Remove outdated javadoc
* Remove map iteration, use target name instead of id (whoops)
* Remove streaming from isReplacementSource
* Simplify getReplacementName
* Only calculate node shutdowns if canRemain==false and forceMove==false
* Move canRebalance comment in BalancedShardsAllocator
* Rename canForceDuringVacate -> canForceAllocateDuringReplace
* Add comment to AwarenessAllocationDecider.canForceAllocateDuringReplace
* Revert changes to ClusterRebalanceAllocationDecider
* Change "no replacement" decision message in NodeReplacementAllocationDecider
* Only construct shutdown map once in isReplacementSource
* Make node shutdowns and target shutdowns available within RoutingAllocation
* Add randomization for adding the filter that is overridden in test
* Add integration test with replicas: 1
* Go nuts with the verbosity of allocation decisions
* Also check NODE_C in unit test
* Test with randomly assigned shard
* Fix test for extra verbose decision messages
* Remove canAllocate(IndexMetadat, RoutingNode, RoutingAllocation) overriding
* Spotless :|
* Implement 100% disk usage check during force-replace-allocate
* Add rudimentary documentation for "replace" shutdown type
* Use RoutingAllocation shutdown map in BalancedShardsAllocator
* Add canForceAllocateDuringReplace to AllocationDeciders & add test
* Switch from percentage to bytes in DiskThresholdDecider force check
* Enhance docs with note about rollover, creation, & shrink
* Clarify decision messages, add test for target-only allocation
* Simplify NodeReplacementAllocationDecider.replacementOngoing
* Start nodeC before nodeB in integration test
* Spotleeeessssssss! You get me every time!
* Remove outdated comment
By using a BuildService we can decouple the configuration of projects but keep making
expensive operations (e.g. parsing properties file or reading minimumJavaCompiler from file) only once
per build.
This bring us closer to decouple projects and get ultimatively configuration cache compliant. In
general we will bring in more BuildServices for the main build to follow along that Pattern and
ultimatevely remove usages of allprojects and subprojects in our build.
Today `GET /_cat/templates` retrieves the whole cluster metadata from
the master, which includes all sorts of unnecessary junk and consumes
significant resources. This commit reimplements these endpoints using
`GetIndexTemplatesAction` and `GetComposableIndexTemplateAction` which
are much more efficient.
The docs for this API indicate that it accepts a comma-separated list of
template names/patterns of the form `GET /_cat/templates/name1,name2`
but in fact today it only accepts a single name or pattern. This commit
also adds support for multiple names/patterns as the docs claim.
Martijn van Groningen
István Zoltán Szabó
Armin Braun
Yang Wang
Roberto Seldner
Ryan Ernst
Adam Locke
Mark Tozzi
Artem Prigoda
Lukas Wegmann
Christoph Büscher
James Rodewig
markharwood
xiaoping
Przemysław Witek
Hendrik Muhs
Rene Groeschke
weizijun
Costin Leau
Joe Gallo
Chris Hegarty
Lisa Cawley
Gordon Brown
Asce Ma
William Brafford
Nikola Grcevski
Lee Hinman
Keith Massey
David Turner
Henning Andersen