- Creates a new StackTemplateRegistry that uses the new names
- The new registry only respects stack.templates.enabled for index templates
- Renames the old registry to LegacyStackTemplateRegistry
- Component templates are not duplicated but registered under two different names
- Documents the new naming convention
- Index templates are not renamed, at least for now, as there are some challenges with it
See 7fd0423 for more details.
Adds a test framework that validates instruments are registered before they are called and are not double registered.
Also records all invocations of Instruments and allows test authors to add validation to instruments.
While dot expansion is disabled when parsing percolator queries at index
time, as that would interfere with query parsing, we still use a wrapper parser
that is conservative about what methods it supports, assuming that
document parsing needs nextToken and not much more. Turns out that when
parsing queries instead, we need to support all the XContentParser
methods including map, list etc.
This commit adds a test for script score query parsing through document
parsing via percolator field mapper, and removes the limitations in the
wrapper parser when dots expansion is disabled.
The anti-contention delay in the S3 repository's compare-and-exchange
operation is hard-coded at 1 second today, but sometimes we encounter a
repository that needs much longer to perform a compare-and-exchange
operation when under contention. With this commit we make the
anti-contention delay configurable.
This splits out the registry and the service, which makes testing easier and removes much of the delegation from the old `APMMeter` to `Instruments` (now renamed `APMMeterRegistry`).
APMMeterService takes care of the lifecycle and APMMeterRegistry holds the instruments.
Just found that we have a lot of inconsistency and needless verbosity
here in tests. We can just use `assertAcked` in a couple spots
to save `.get`, `.actionGet` etc., especially with the signature
change I added here.
Today using painless execute api with tsdb index can fail with a `_id must be unset or set to [cn4exTOUtxytuLkQAAABeRnR_mY] but was [_id] because [test_index] is in time_series mode` error.
This change addresses this.
The painless execute api shouldn't set use a static _id, but
let the TsidExtractingIdFieldMapper generate it.
Otherwise validation TsidExtractingIdFieldMapper fails.
Closes#101072
Follow-up to #100966.
Add more assertion overloads that consume a requestBuilder as in the
other PRs and start using `assertHitCount` in more places that were
duplicating what it does. Also add a shortcut for
`client().prepareSearch()` to integ tests and bulk-replace some usages
of this pattern to avoid these changes from blowing up test code line
count further.
Similar to the TransportVersions holder class, IndexVersions is the new
place to contain all constants for IndexVersion. This commit moves all
existing constants to the new class. It is purely mechanical.
Remove `assertSearchResponse` which was just an alias for
`assertNoFailures` and then cleanup many spots in the result
by combining the hit count and no failure assertion into a single
method.
follow-up to #100966
Follow up to #100966 introducing new combined assertion `assertSearchHitsWithoutFailures`
to combine no-failure, count, and id assertions into one block.
Painless sandboxes some errors from Java for which it can recover. These
errors are wrapped within a ScriptException. However, retaining the
error as a cause can be confusing when walking the error chain. This
commit wraps the error so that the real error type does not appear,
but maintains the same error message in xcontent serialized form.
We'd like to make `SearchResponse` reference counted and pooled but there are around 6k
instances of tests that create a `SearchResponse` local variable that would need to be
released manually to avoid leaks in the tests.
This does away with about 10% of these spots by adding an override for `assertHitCount`
that handles the actual execution of the search request and its release automatically
and making use of it in all spots where the `.get()` on the request build could be inlined
semi-automatically and in a straight-forward fashion without other code changes.
This PR is migrating some of the ITs that use either the
`elasticsearch.legacy-java-rest-test` or the
`elasticsearch.legacy-yaml-rest-test` gradle test plugins to the new
`elasticsearch.internal-java-rest-test` and
`elasticsearch.internal-yaml-rest-test` equivalents. This is the list of
the affected ITs: * SamlAuthenticationIT * OperatorPrivilegesIT *
ProfileIT * SetSecurityUserProcessorWithWithSecurityDisabledIT *
AsyncSearchSecurityIT * SecurityRealmSmokeTestCase *
KibanaSystemIndexIT * KerberosAuthenticationIT * ReindexWithSecurityIT
and ReindexWithSecurityClientYamlTestSuiteIT *
ReloadSecureSettingsWithPasswordProtectedKeystoreRestIT * PermissionsIT
from slm:qa:with-security * Permissions IT from
runtime-fields:with-security * Permissions IT from ilm:qa:with-securiy
* GraphWithSecurityIT and GraphWithSecurityInsufficientRoleIT
Related: ES-6751
There are no remote invocations of any actions derived from
`TransportNodesAction` so there is no need to register the top-level
action with the `TransportService`, and that means that all the code
related to de/serialization of the top-level request and response is
unused and can be removed.
Relates #100111 Relates #100878
Emmited metrics could not be index as elasticsearch.metrics.s3.exceptions field
is both long counter and a parent object for a histogram. This change renames
histogram to avoid the conflict.
This commit adds the possibility to create runtime fields of type geo-shape. In order to create them, users can
define an emit function that takes either a geojson object or a WKT string that internally creates a geometry object.
SourceConfirmedTestQuery uses a QueryVisitor to collect terms from
its inner query to build its internal SimScorer. It is important to hold these
terms in a consistent order so that when scores for each term are summed,
the order of summation is the same as it would be for the inner query. This
commit changes the call to visit to use a LinkedHashSet to ensure that
terms are iterated in the order in which they are collected.
Fixes#98712
This PR builds on top of #100464 to publish s3 request count via the metrics API.
The metric takes the name of `repositories.requests.count` with
attributes/dimensions of
`{"repo_type": "s3", "repo_name": "xxx", "operation": "xxx", "purpose": "xxx"}`.
Closes: ES-6801
* Don't print synthetic source in mapping for bwc tests
* Move comment.
* Don't print synthetic source in mapping for bwc tests #2
* Don't print synthetic source in mapping for bwc tests #2
* Revert "Don't print synthetic source in mapping for bwc tests #2"
This reverts commit 034262c5d2.
* Revert "Don't print synthetic source in mapping for bwc tests #2"
This reverts commit 44e815635e.
* Revert "Don't print synthetic source in mapping for bwc tests (#100572)"
This reverts commit 9322ab9b91.
* Exclude synthetic source test from mixedClusterTests
* Update comment.
* Mute all tsdb tests in mixedClusterTests
This is an interim step to stop sporadic test failures, while we try to
fix version skip for mixed cluster tests.
* Remove old exclusion
* Add aggregations too
* Mute tests for versions between 8.7-8.10
* Remove mute
Terms aggregation supports a `global_ordinals` execution hint that
switches the underlying implementation to use
`GlobalOrdinalsStringTermsAggregator`. If applied on a text field,
fielddata get populated. However, in mixedClusterTests there are many
nodes, and some contain no data so fielddata won't get populated either.
Fixes#99774
* Support complex datemath expressions in index and index alias names
* Update docs/changelog/100646.yaml
* Follow PR comments
* fix
---------
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
Today we rely on the caller computing the appropriate repository format
version based on the nodes in the cluster and the snapshots in (some
recent copy of) the `RepositoryData`. This commit moves that computation
into `createSnapshotsDeletion` so that (a) we can be sure to use the
same `RepositoryData` used for the rest of the process, and (b) we avoid
dispatching work to the SNAPSHOT pool twice.
Relates a comment on #100657
The only reason this method is throwing an exception is because the
method ByteArrayOutputStream#close() is declaring it although it is a
noop. Therefore it can be safely ignored.
Thanks @romseygeek for bringing into attention.
TSDB indices are expected to receive a large amounts of writes whilst
their time bounds are "active" (i.e they include `now`).
This ensures TSDB doesn't execute any ingest disruptive operations (like
delete, forcemerge, downsample) until the `end_time` for the TSDS
backing indices has lapsed.
Introduces the `SnapshotsDeletion` class which encapsulates the process
of deleting some collection of snapshots. In particular this class gives
us somewhere to store various deletion-wide data which significantly
reduces the length of some argument lists.
Relates #100568
* Allow prefix index naming while reindexing from remote
* Update docs/changelog/96968.yaml
* Add ignore_unavailable index option to query parameters
* Code style
* Fix asciidoc
* Exclude remote index names from reindex alias validation
* spotless fix
* Fix test
* Fix test
* code style fix
* Do not eval remote expr locally + IT test
* Fix test
* in progress
* Reverting back a bit (sync with main)
* Ignore remote names in ReindexValidator
* Add IT test, fix double re-indexing
* codestyle
* reduce scope of PR (do not handle ignore_unavailable request option)
* minus api specs file
* add datemath index name tests for within-cluster reindexing
* Move out (to separate PR) logic which handles complex datemath expressions
OTEL gauges should follow the callback model otherwise they will not be sent by
apm java agent. (or use BatchCallback)
This commit changes the gagues creation model to return Observable*Gauge
and uses AtomicLong/Double to store current value which will be polled when
metrics are exported (and callback is called)
Some tests rely on the natural index order for test assertions. This
works when the index has a single primary shard but fails otherwise.
This PR adjusts the relevant tests so that they explicitly configure the
number of shards to 1.
Relates: ES-6540
Reorders the methods involved in snapshot deletion to be closer together
and better match the flow of execution, and harmonises the names of many
parameters and local variables to make it easier to follow them through
the process.
This PR wires the new Meter interface into S3BlobStore. The new meter
field remains unused in this PR. Actual metric collection will be
addressed in follow-ups.
Relates: ES-6801
A new no-op OperationPurpose parameter is added in #99615 to all blob
store/container operation method. This PR updates the s3 stats
collection code to actually use this parameter for finer grained stats
collection and reports. This differentiation between purposes are kept
internally for now. The stats are currently aggregated over operations for
existing stats reporting. This means responses from both
GetRepositoriesMetering API and GetBlobStoreStats API will not be changed.
We will have follow-ups to expose the finer stats separately.
Relates: #99615
Relates: ES-6800
Felix Barnsteiner
Ignacio Vera
Stuart Tettemer
Luca Cavanna
Yang Wang
David Turner
Lorenzo Dematté
Armin Braun
Martijn van Groningen
Ryan Ernst
Mark Vieira
Keith Massey
Albert Zaharovits
Ievgen Degtiarenko
Alan Woodward
Kostas Krikellas
Volodymyr Krasnikov
Mary Gouseti
Andrei Dan
Przemyslaw Gomulka
Simon Cooper