EnsureNoWarnings method should assert that there is no other warnings
than the allowed "predefined" warnings in filteredWarnings() method
bug introduced in #71207
We already ship templates that include the _meta map for things like component and composable
index templates, and we use the "managed": true indicator for many of our built-in items within
Elasticsearch.
This commit adds a description and managed flag to each of the ILM policies we ship by default.
Sometimes our fancy "run this agg as a Query" optimizations end up
slower than running the aggregation in the old way. We know that and use
heuristics to dissable the optimization in that case. But it turns out
that the process of running the heuristics itself can be slow, depending
on the query. Worse, changing the heuristics requires an upgrade, which
means waiting. If the heurisics make a terrible choice folks need a
quick way out. This adds such a way: a cluster level setting that
contains a list of queries that are considered "too expensive" to try
and optimize. If the top level query contains any of those queries we'll
disable the "run as Query" optimization.
The default for this settings is wildcard and term-in-set queries, which
is fairly conservative. There are certainly wildcard and term-in-set
queries that the optimization works well with, but there are other queries
of that type that it works very badly with. So we're being careful.
Better, you can modify this setting in a running cluster to disable the
optimization if we find a new type of query that doesn't work well.
Closes#73426
This commit adds some clean up logic to ESRestTestCase so
that searchable snapshots indices are deleted after test case
executions, before the snapshot and repositories are wipe out.
Backport of #73555
There are some changes in GCP. The service accounts are now their own button and not a child under the IAM/permissions anymore.
Co-authored-by: Philipp Kahr <philipp.kahr@elastic.co>
Changes:
* Reuses the same `aliases` object properties in the following API docs:
* Clone index API
* Create index API
* Put component template API
* Put legacy index template API
* Put index template API
* Rollover index API
* Shrink index API
* Simulate template API
* Split index API
* Updates the `aliases` object properties for the simulate index API docs.
Closes#73044
We never do (nor should) call this method to overwrite a snapshots existing entry,
checks and assertions upstream make sure of that.
=> simplified the code accordingly
Today we use `SimpleFSDirectory` in `PersistedClusterStateService` since
we don't need anything fancy. `SimpleFSDirectory` is today deprecated
since it's strictly worse than `NIOFSDirectory` so this commit removes
usages of the deprecated class.
Today we do not set the `LucenePersistedState#writeNextStateFully` flag
on all failures, notably on an `OutOfMemoryError`. Since we don't exit
immediately on an OOME we may have failed part-way through writing a
full state but still proceed with another apparently-incremental write.
With this commit we ensure `LucenePersistedState#writeNextStateFully` is
only set if the previous write was successful.
Serializing and compressing `RepositoryData` seems to have been the wrong trade-off in hindsight.
While saving some heap on a quiet master it makes every repository operation cost heap for
the a newly instantiated `RepositoryData`. Concurrent repository operations and snapshot API
requests can thus easily lead to many duplicate instances on heap causing memory pressure.
Limiting caching to smaller instances also appears to have been the wrong choice in hindsight.
While duplication of a few 100kb instances of `RepositoryData` is mostly not a big deal, duplicating
a `5MB` instance a couple of times (e.g. seen during heavily concurrent get snapshots requests)
eventually becomes a problem.
Recent JDK releases have disabled TLS v1.0 and TLS v1.1 by default
See
- https://java.com/en/jre-jdk-cryptoroadmap.html
- https://bugs.openjdk.java.net/browse/JDK-8202343
This change adds documentation clarifying which TLS versions are
supported on which JDKs (in general terms, rather than specific builds)
and how to change the configuration if necessary.
Co-authored-by: Adam Locke <adam.locke@elastic.co>
This commit moves the logic for deleting job data
from the `TransportDeleteJobAction` into the `JobDataDeleter`
class in order to allow reusing it. In particular, it will
be necessary for the implementation of the reset API.
We've been discussing possibly removing `FieldTypeLookup#getMatchingFieldTypes`, or at least its `SearchExecutionContext` variant that applies runtime mappings.
This is another step in that direction: the exists query can rely on getMatchingFieldNames instead, and look up field types by name.
We've had a series of bug fixes for cases where an OsProbe gives negative
values, most often just -1, to the OsStats class. We added assertions to catch
cases where we were initializing OsStats with bad values. Unfortunately, these
fixes turned to not be backwards-compatible. In this commit, we simply coerce
bad values to 0 when data is coming from nodes that don't have the relevant bug
fixes.
Relevant PRs:
* #42725
* #56435
* #57317Fixes#73459
This template was added in #64978, however, there can be some test failures if we try to remove
built-in templates. It was missing from the list and now needs to be added back.
* Add product response header to all responses
* share header value privately across package
* Make the product header lowercase.
* Do not expose the product header if request is unauthenticated.
* Fix checkstyle
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
The cluster will not automatically react to the node shutdown being
registered unless we notify it that something has changed that may
require a change in shard allocation. This commit modifies the Put
and Delete Shutdown actions to invoke a reroute after the cluster
state has been updated, as well as an integration test to verify that
shards quickly move away from nodes which are shutting down for
removal.
Relates to #70755.
The main changes of this PR are:
Add an optional _meta field to ILM policy.
Add some test code about the change.
Update the doc of Create or update lifecycle policy API.
Changes:
* Updates the write index snippet to use data streams
* Notes data streams aliases don't set an implicit write stream, even if the alias points to one data stream.
Categorization jobs created once the entire cluster is upgraded to
version 7.14 or higher will default to using the new ml_standard
tokenizer rather than the previous default of the ml_classic
tokenizer, and will incorporate the new first_non_blank_line char
filter so that categorization is based purely on the first non-blank
line of each message.
The difference between the ml_classic and ml_standard tokenizers
is that ml_classic splits on slashes and colons, so creates multiple
tokens from URLs and filesystem paths, whereas ml_standard attempts
to keep URLs, email addresses and filesystem paths as single tokens.
It is still possible to config the ml_classic tokenizer if you
prefer: just provide a categorization_analyzer within your
analysis_config and whichever tokenizer you choose (which could be
ml_classic or any other Elasticsearch tokenizer) will be used.
To opt out of using first_non_blank_line as a default char filter,
you must explicitly specify a categorization_analyzer that does not
include it.
If no categorization_analyzer is specified but categorization_filters
are specified then the categorization filters are converted to char
filters applied that are applied after first_non_blank_line.
Closeselastic/ml-cpp#1724
This moves the public build api and plugins into a separete included build called 'build-tools'
and we removed the duplication of included buildSrc twice (2nd import as build-tools).
The elasticsearch internal build logic is kept in build-tools-internal as included build which allows us better handling of this project that its just being an buildSrc project (e.g. we can reference tasks directly from the root build etc.)
Convention logic applied to both projects will live in a new build-conventions project.
Today when upgrading to the next major version we have a so-called
_major version barrier_: once the cluster comprises nodes of the new
major version then nodes of the previous major version are prevented
from joining the cluster. This means we can be certain that
`clusterState.nodes().getMinNodeVersion().major` will never decrease, so
we can implement upgrade logic that relies on the cluster remaining in
its wholly-upgraded state.
This commit generalises this behaviour to apply to all upgrades, so that
we can be certain that `clusterState.nodes().getMinNodeVersion()` will
never decrease in a running cluster.
Closes#72911
This commit fixes a small inconsistency in the FrozenEngine
and Engine classes when they refer to the source used to
open a Searcher for loading doc stats.
Przemyslaw Gomulka
István Zoltán Szabó
Luca Belluccini
Lee Hinman
Nik Everett
Tanguy Leroux
Alexander Reelsen
James Rodewig
Armin Braun
Ignacio Vera
David Turner
Martijn van Groningen
Dimitris Athanasiou
Tim Vernum
David Roberts
Luca Cavanna
Nhat Nguyen
Mark Vieira
William Brafford
James Baiera
Gordon Brown
bellengao
Rene Groeschke
Adam Locke