This commit introduces into the node stats API various statistics to
track the time that the elected master spends in various phases of the
cluster state publication process.
Relates #76625
We spawn a background task to mark a task as cancelled when its inbound
connection closes. In these tests this might not happen until we've shut
the threadpool down, in which case it never happens. This change ensures
that we're not waiting for the cancellation to happen when the test
exits.
Closes#76558
We tend to include the phrase `warn threshold` in log messages
indicating that something took much longer than we expected. The
`MasterService` has some log messages that date from before the adoption
of this convention. This commit adds the phrase `warn threshold` to
those logs.
Relates #76625
This change #76636 actually resolved the #75097 by only filtering out
the CAs certs/keys from xpack.security.http.ssl keystore
This change re-enables the tests previously affected by #75097Resolves: #75097
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
Composite runtime fields do not have a mapped type - add null check, test and Nullable annotation to SearchExecutionContext.getObjectMapper(name)
Closes#76716
This commit adjusts `NodeSeenService` to check if the `NodesShutdownMetadata`
is `null` before trying to work with it. If it is `null`, we can simply
return from the method early as we know there are no shutdowns registered.
Closes#74795.
Introduce two Docker image variants for Cloud. The first bundles
(actually installs) the S3, Azure and GCS repository plugins. The
second bundles all official plugins, but only installs the repository
plugins.
Both images also bundle Filebeat and Metricbeat.
The testing utils have been refactored to introduce a `docker`
sub-package. This allows the static `Docker.containerId` to be
shared without needing all the code in one big class. The code for
checking file ownership / permissions has also been refactored to
a more Hamcrest style, using a custom Docker file matcher.
In #72820 we improved the error message when attempting to `copy_to` with an
object value. The change accidentally started throwing errors for null values
too, although they're allowed. This PR fixes the regression and adds an
explicit test for `copy_to` with null values.
Whilst testing the p_value scoring heuristic for significant terms introduced
in #75313 it became clear we can assign arbitrarily low p-values if the overall
counts are high enough for terms which constitute a very small fraction of the
foreground set. Even if the difference in their frequency on the foreground and
background set is statistically significant they don't explain the majority of the
foreground cases and so are not of significant interest (certainly not in the use
cases we have for this aggregation).
We already have some mitigation for the cases that 1. the term frequency is
small on both the foreground and background set, 2. the term frequencies are
very similar. These offset the actual term counts by a fixed small fraction of
the background counts and make the foreground and background frequencies
more similar by a small relative amount, respectively. This change simply applies
offsets to the term counts before making frequencies more similar. For frequencies
much less than the offset we therefore get equal frequencies on the foreground
and background sets and p-value tends to 1. This retains the advantage of being
a smooth correction to the p-value so we get no strange discontinuities in the
vicinity of the small absolute and difference thresholds for the frequency.
We added configuration to AllocateAction to set the total shards per node property on the index. This makes it possible that a user could set this to a value lower than the total number of shards in the index that is about to be shrunk, meaning that all of the shards could not be moved to a single node in the ShrinkAction. This commit unsets the total shards per node property so that we fall back to the default value (-1, unlimited) in the ShrinkAction to avoid this.
Relates to #44070
For package installations (DEB,RPM), we are generating a
random strong password for the elastic user on installation time
so that we can show it to the user.
We subsequently hash and store this password in the
elasticsearch.keystore so that the node can pick it up on the first
run and use it to populate the relevant document for the elastic
user in the security index.
This change implements a class that can be called from the package
installation scripts (postinst) to
- Generate a strong password
- Hash it with the configured(default) password hashing algo
- Store it in the elasticsearch.keystore
- Print it in stdout so that it the bash script can capture it.
System indices should be treated as a special set of indices and not be
accessible by all users. The existing security codebase has the notion
of restricted indices, which are currently a subset of system indices.
This change unifies the two concepts by making system indices the set
of restricted indices. This means that going forward, consumers of
system indices will need access to restricted indices.
Our intention is that this will be handled internally as much as
possible. For example, restricted index access can be used implicitly
by setting a valid origin on a request or using a system index plugin.
In these cases, the security module will apply internally defined
privileges when necessary.
The main impact of this change for developers is that system index
deletions will require superuser privileges, so we have to make sure
we are using an admin role for test cleanup.
Closes#69298
Co-authored-by: William Brafford <william.brafford@elastic.co>
Co-authored-by: Albert Zaharovits <albert.zaharovits@elastic.co>
To return the JVM `uptime` metric, the `human` query parameter must be `true`.
Co-authored-by: Adam Locke <adam.locke@elastic.co>
Co-authored-by: James Rodewig <40268737+jrodewig@users.noreply.github.com>
Updates the put SLM policy API's `config` parameter to reuse the create snapshot API's request body parameters.
Previously, the `config` parameter was missing the `feature_states` parameter. This change should keep the two docs in sync.
When building empty responses for shards that don't have the term field in question,
significant terms ignored the background filter.
This commit fixes this bug by respecting the background filter count, even
when building empty results.
closes#76729
Adds a more detailed description to get-snapshots, get-shard-snapshot,
snapshots-status and field-capabilities tasks since these may somtimes
be long-running things and it may occasionally be necessary to track
down why they are running and what they're doing.
This PR add more resilience to testExpiredTokensDeletedAfterExpiration
by wrapping the empty error assertion into an assertBusy block. In rare
cases, it is possible for search to return a deleted but not yet
refreshed document. Trying to invalidate this document then result into
MissDocumentException. This state is transient and should go away once
all shards refresh.
Resolves: #67347
This is the final step in the removal of the X-Pack specific SSL
configuration code (replaced by libs/ssl-config)
For some time we have had two implementations of SSL Configuration
code. One was in org.elasticsearch.xpack.core.ssl, the other in
org.elasticsearch.common.ssl
These two implementations had essentially the same functionality:
- Handle settings such as '*.ssl.certificate`, `*.ssl.key` etc
- Load certificates and keys from PEM files and Keystores
- Build and configure Java class such as SSLContext, KeyManager, etc
based on the configuration and certificates.
As of this common the X-Pack version is no more, and all SSL
configuration in Elasticsearch is handled by the libs/ssl-config
version instead.
Resolves: #68719
* Remove unused recent trial license version from license state
The license state previously used the most recent trial license version
to determine the default security enabled state. Since that no longer
exists, this can be removed from the license state.
* compile
* fix tests
* iter
* more compile
Licensed feature tracking of long running features happens now by
starting and stopping the tracking of a feature. This commit adds an
intermediate base class to be used by features that utilize persistent
tasks. The base class only requires specifying the feature and context,
and then any tasks created automatically are tracked. An example using
marchine learning's JobTask is also added here, using a new machine
learning job feature object.
Today we use `ClusterChangedEvent` to represent a committed change to
the cluster state while it's being applied, and also to represent the
proposed change while it's being published. These are quite different
usages in practice, so this commit separates them by introducing a
`ClusterStatePublicationEvent` to represent the change to be published.
Relates #76625 in that we will be able to use the new
`ClusterStatePublicationEvent` to track various stats about the
publication as it progresses, but which don't make sense on a
`ClusterChangedEvent`.
* Put Shutdown API docs
* Get Shutdown API docs
* Properly escape regex
* Doc tests build now! (but don't pass)
* Doc tests pass now! (with a code fix that will go in a different PR)
* DELETE shutdown docs
* Edit for language & consistency
* Fix doctest
* Add cloud-only banner
* Add allocation_delay docs
* Restore file that got deleted somehow?
* Use `restart` type in example to demonstrate `allocation_delay` param
* Fix typo per review
Co-authored-by: debadair <debadair@elastic.co>
* Vastly improve wording per review
Co-authored-by: debadair <debadair@elastic.co>
* Adjust test request & response so it passes
Co-authored-by: Deb Adair <debadair@elastic.co>
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
This fixes a problem with using NLP processors concurrently.
In particular, the `BertRequestBuilder` holds a local reference
to the tokenization result which is needed for processing the
result. However, this is not thread-safe. Multiple concurrent
calls will override the cached tokenization result and, consuquently,
the final results will be wrong.
This commit fixes this by making `BertRequestBuilder` stateless
and caching the tokenization result in the stack in the method
that's waiting for the result to come through from the process.
The feature usage tracking data currently contains an opaque "name"
attribute which identifies the feature that was used. This name needs to
be unique enough that certain features can be identified independently
of others. For example, distinguishing machine learning jobs from
trained models. Yet both those examples are all "machine learning".
This commit adds a "family" attribute so that similar tracked features
can be grouped together. The output format of the feature usage api is
essentially the same; it is still a flat list of features and their last
used times. The family attribute can be used on the receiving end to
group many features.
Mark Vieira
David Turner
Rory Hunter
Lyudmila Fokina
Henning Andersen
markharwood
Keith Massey
James Rodewig
Martijn van Groningen
István Zoltán Szabó
David Kyle
Gordon Brown
Rory Hunter
Armin Braun
Julie Tibshirani
Tom Veasey
Ioannis Kakavas
Jay Modi
Benjamin Trent
Peter Dyson
Yang Wang
Tim Vernum
Ryan Ernst
Dimitris Athanasiou