Today if a node shutdown is stalled due to unmoveable shards then we say
to use the allocation explain API to find details. In fact, since #78727
we include the allocation explanation in the response already so we
should tell users just to look at that instead. This commit adjusts the
message to address this.
Today if there are multiple nodes with the same name then
`POST /_cluster/voting_config_exclusions?node_names=ambiguous-name` will
return a `500 Internal Server Error` and a mysterious message. This
commit changes the behaviour to throw an `IllegalArgumentException`
(i.e. `400 Bad Request`) along with a more useful message describing the
problem.
Adds `WriteScript` as the common base class for the write scripts: `IngestScript`, `UpdateScript`, `UpdateByQueryScript` and `ReindexScript`.
This pulls the common `getCtx()` and `metadata()` methods into the base class and prepares for the implementation of the ingest fields api (https://github.com/elastic/elasticsearch/issues/79155).
As part of the refactor, `IngestScript` now takes a `CtxMap` directly rather than taking "sourceAndMetadata" (`CtxMap`) and `Metadata` (from `CtxMap`). There is a new `getCtxMap()` getter to get the typed `CtxMap`. `getSourceAndMetadata` could have been refactored to do this, but most of the callers of that don't need to know about `CtxMap` and are happy with a `Map<String, Object>`.
Fixing the names of the internal actions used by CoordinationDiagnosticsService to begin with "internal:" so
that they can be used in the system context with security enabled.
The equality checks on these in `DiskThresholdDecider` become very expensive
during reroute in a large cluster. Deduplicating these when building the `ClusterInfo`
saves more than 2% CPU time during many-shards benchmark bootstrapping because
the lookup of the shard data path by shard-routing mostly hit instance equality.
Also, this saves a little memory.
This PR also moves the callback for building `ClusterInfo` from the stats response to
the management pool as it is now more expensive (though the overall CPU use from it is trivial
relative to the cost savings during reroute) and was questionable to run on
a transport thread in a large cluster to begin with.
Co-authored-by: David Turner <david.turner@elastic.co>
No need to check for the null token manually and parse `textOrNull`.
Either we can just use `text()` since we know we don't have to deal
with a null token or use `textOrNull` and check the return value for
`null`.
I chose the latterbecause it benchmarked slightly faster in `BeatsMapperBenchmark`
but both save the expensive call to `currentToken` on the heavily nested x-content
parser that we use here.
This PR adds BloomFilter to Elasticsearch and enables it for the _id
field of non-data stream indices. BloomFilter should speed up the
performance of mget and update requests at a small expense of refresh,
merge, and storage.
This commit causes non-master-eligible nodes to poll a random master-eligible node every 10 seconds
whenever the elected master goes null for diagnostic information in support of the health API's master
stability check.
If a master-eligible node comes up and has never seen an elected master
node (and assuming that a quorum requires more than one node), then it
ought to report that the master stability health is red because it
cannot form a quorum.
This method was introduced to fix datastream snapshots during
concurrent index/datastream changes but was never actually
used because we went with a different approach in the end.
=> remove it and its tests
If a docvalues field matches multiple field patterns, then ES will
return the value of that doc-values field multiple times. Like fetching
fields from source, we should deduplicate the matching doc-values
fields.
We previously removed support for `fields` in the request body, to ensure there
was only one way to specify the parameter. We've now decided to undo the
change, since it was disruptive and the request body is actually the best place to
pass variable-length data like `fields`.
This PR restores support for `fields` in the request body. It throws an error
if the parameter is specified both in the URL and the body.
Closes#86875
To assist the user in configuring the visualizations correctly while leveraging TSDB
functionality, information about TSDB configuration should be exposed via the field
caps API per field.
Especially for metrics fields, it must be clear which fields are metrics and if they belong
to only time-series indexes or mixed time-series and non-time-series indexes.
To further distinguish metric fields when they belong to any of the following indices:
- Standard (non-time-series) indexes
- Time series indexes
- Downsampled time series indexes
This PR modifies the field caps API so that the mapping parameters time_series_dimension
and time_series_dimension are presented only when they are set on fields of time-series indexes.
Those parameters are completely ignored when they are set on standard (non-time-series) indexes.
This PR revisits some of the conventions adopted by #78790
Removing the custom dependency checksum functionality in favor of Gradle build-in dependency verification support.
- Use sha256 in favor of sha1 as sha1 is not considered safe these days.
Closes https://github.com/elastic/elasticsearch/issues/69736
This change deprecates the kNN search API in favor of the new 'knn' option
inside the search API. The 'knn' option is now the preferred way of performing
kNN search.
Relates to #87625
This makes sure that the test cluster is stable in CoordinationDiagnosticsServiceIT::testBlockClusterStateProcessingOnOneNode before proceeding with the rest of test.
Part of #84369. Implement the `Tracer` interface by providing a
module that uses OpenTelemetry, along with Elastic's APM
agent for Java.
See the file `TRACING.md` for background on the changes and the
reasoning for some of the implementation decisions.
The configuration mechanism is the most fiddly part of this PR. The
Security Manager permissions required by the APM Java agent make
it prohibitive to start an agent from within Elasticsearch
programmatically, so it must be configured when the ES JVM starts.
That means that the startup CLI needs to assemble the required JVM
options.
To complicate matters further, the APM agent needs a secret token
in order to ship traces to the APM server. We can't use Java system
properties to configure this, since otherwise the secret will be
readable to all code in Elasticsearch. It therefore has to be
configured in a dedicated config file. This in itself is awkward,
since we don't want to leave secrets in config files. Therefore,
we pull the APM secret token from the keystore, write it to a config
file, then delete the config file after ES starts.
There's a further issue with the config file. Any options we set
in the APM agent config file cannot later be reconfigured via system
properties, so we need to make sure that only "static" configuration
goes into the config file.
I generated most of the files under `qa/apm` using an APM test
utility (I can't remember which one now, unfortunately). The goal
is to setup up a complete system so that traces can be captured in
APM server, and the results in Elasticsearch inspected.
As discussed in #73569 the current implementation is too slow in certain scenarios.
The inefficient part of the code can be stated as the following problem:
Given a text (getText()) and a position in this text (offset), find the sentence
boundary before and after the offset, in such a way that the after boundary is
maximal but respects end boundary - start boundary < fragment size.
In case it's impossible to produce an after boundary that respects the said
condition, use the nearest boundary following offset.
The current approach begins by finding the nearest preceding and following boundaries,
and expands the following boundary greedily while it respects the problem restriction. This
is fine asymptotically, but BreakIterator which is used to find each boundary is sometimes
expensive.
This new approach maximizes the after boundary by scanning for the last boundary
preceding the position that would cause the condition to be violated (i.e. knowing start
boundary and offset, how many characters are left before resulting length is fragment size).
If this scan finds the start boundary, it means it's impossible to satisfy the problem
restriction, and we get the first boundary following offset instead (or better, since we
already scanned [offset, targetEndOffset], start from targetEndOffset + 1).
We used this in three spots, where it copies potentially huge arrays.
One of those spots doesn't need the `copyShards` call at all
and can use the normal iterator as there's no concurrent modfication.
The other two spots can at least just use an array, which will iterate
a little faster than a mutable list and also potentially saves another
round copying the array in the `ArrayList` constructor that the compiler
seems to not be able to eliminate in all cases.
Part of #84369. Split out from #88443. This PR wraps parts of the code
in a new tracing context. This is necessary so that a tracing
implementation can use the thread context to propagate tracing headers,
but without the code attempting to set the same key twice in the thread
context, which is illegal. In order to avoid future diff noise, the wrapped
code has mostly been refactored into methods.
Note that in some places we actually clear the tracing context
completely. This is done where the operation to be performed should have
no association with the current trace context. For example, when
creating a new index via a REST request, the resulting background tasks
for the index should not be associated with the REST request in
perpetuity.
This PR introduces the local health monitoring functionality needed for
#84811 . The monitor uses the `NodeService` to get the disk usage stats
and determines the node's disk health.
When a change in the disk's is detected or when the health node changes,
this class would be responsible to send the node's health to the health
node. Currently this is simulated with a method that just logs the
current health.
The monitor keeps the last reported health, this way, if something fails
on the next check it will try to resend the new health state.
This is a pure refactoring of the WildcardExpressionResolver.
The objective is to restrict access to the indices lookup through the context parameter only.
Eventually, Security is going to plug into the context and only show a restricted view of the
indices lookup, particular to the user context.
This change adds a SourceValueFetcherSortedDoubleIndexFieldData to support double doc values types for source fallback. This also adds support for double, float and half_float field types.
Part of #84369.
ML uses the task framework to register a tasks for each loaded model.
These tasks are not executed in the usual sense, and it does not make
sense to trace them using APM. Therefore, make it possible to register
a task without also starting tracing.
Computing routing nodes and the indices lookup takes considerable time
for large states. Both are needed during cluster state application and
Prior to this change would be computed on the applier thread in all cases.
By running the creation of both objects concurrently to publication, the
many shards benchmark sees a 10%+ reduction in the bootstrap time to
50k indices.
We can save another two loops here by checking for shards to fail in the same loop that updates or creates shards.
Also, we only need to loop over all indices services locally once for deleting indices as a whole or just shards
out of existing indices.
This change adds source fallback support for byte, short, and long fields. These use the already
existing class SourceValueFetcherSortedNumericIndexFieldData.
This fixes some possible race conditions in the cluster formation polling of the stable master code.
It also prevents the list of tasks from growing indefinitely.
This removes many calls to the last remaining `createParser` method that
I deprecated in #79814, migrating callers to one of the new methods that
it created.
David Turner
Mary Gouseti
Ievgen Degtiarenko
Nikola Grcevski
Stuart Tettemer
Keith Massey
Armin Braun
Jack Conradson
Nhat Nguyen
likzn
Christos Soulios
zhouhui
Rene Groeschke
Julie Tibshirani
Marcin Słowiak
Rory Hunter
Denilson das Mercês Amorim
Albert Zaharovits
Ievgen Degtiarenko
Nik Everett