We have found that ContentPath.pathAsText() ends up being a very hot path
in indexing, and that it is specifically called a lot when checking that the current
path does not lead to a metadata mapper. The metadata mappers that accept
input only exist at the root, however, so instead of building the full path we can
instead check that we are on the root mapper and then just pass the current
field to the lookup.
The cluster allocation explain API includes a top-level status
indicating to the user whether the shard can be assigned/rebalanced/etc
or not. Today this status is fairly terse and experience shows that
users sometimes struggle to understand how to interpret it and to decide
on follow-up actions.
This commit makes the top-level explanation more detailed and
actionable. For instance, in the cases like `THROTTLED` where the status
is transient we instruct the user to wait; if a shard is lost we say to
restore it from a snapshot; if a shard cannot be assigned we say to
choose a specific node where its assignment is expected and to address
the obstacles.
Co-authored-by: James Rodewig <james.rodewig@elastic.co>
Lucene issues that resulted in elasticsearch changes:
LUCENE-9820 Separate logic for reading the BKD index from logic to intersecting it.
LUCENE-10377: Replace 'sortPos' with 'enableSkipping' in SortField.getComparator()
LUCENE-10301: make the test-framework a proper module by moving all test
classes to org.apache.lucene.tests
LUCENE-10300: rewrite how resources are read in ukrainian morfologik analyzer:
LUCENE-10054 Make HnswGraph hierarchical
This test would fail if we introduce the network partition while the
master is still publishing a cluster state update and hasn't received
the ack from the victim node. In this case the default publish timeout
means that the master will wait for 30s before completing the stalled
publication and moving on to the `node-left` one, but
`ensureStableCluster` also times out after 30s which leaves not much
time for the master to remove the victim node.
This commit reduces the publish timeout to 10s so that the master
recovers well before `ensureStableCluster` times out.
Closes#84172
* Add a note that the http_ca.crt certificate that is generated and
stored in config/certs can be used to configure any client to trust
the certificate that elasticsearch uses for TLS on the HTTP layer
* Add a note that the elasticsearch-create-enrollment-token CLI
tool can only be used with auto-configured TLS settings.
Replaces some funny code formatting created when we applied spotless to
the code globally. Spotless does a fine job, but this was a little funky
looking.
This PR addressed the behavior described in #83917, in which Feature migrations
which have encountered errors are not re-run in some cases. As of this PR, Features
which have encountered errors during migration are treated the same as Features
requiring migration.
This PR also adds a test which artificially replicates #83917.
Clarifies that the `orientation` mapping parameter only applies to WKT polygons. GeoJSON polygons use a default orientation of `RIGHT`, regardless of the mapping parameter.
Also notes that the document-level `orientation` parameter overrides the default orientation for both WKT and GeoJSON polygons.
Closes https://github.com/elastic/elasticsearch/issues/84009.
This commit utilizes the index mapping hash to share the fields-caps for
indices with the same index mapping to reduce the memory usage and the
size of transport messages.
Closes#78665Closes#82879
This removes the defaults from the slow log for the remaining queries in
the `parent-join` module. So it should be easier to read the slow log
when it contains these queries.
Relates to #76515
At the moment we really don't know what configuring a
`time_series_dimension` should *do* when there are nested documents.
So, for now, we're going to disable it. One day when someone has a good
idea of how it should work we can build that. But for now we don't want
to guess wrong and then lock us into some annoying behavior that no one
needs but we have to support for backwards compatibility reasons.
Closes: #83915
A regular data stream can be migrated to a tsdb data stream if in template that created the data stream, the `index_mode` field is set to `time_series` and the data stream's `index_mode` property is either not specified or set to `standard`. Then on the next rollover the data stream is migrated to be a tsdb data stream.
When that happens the data stream's `index_mode` property is set to `time_series` and the new backing index's `index.mode` index setting is also set to `time_series`.
Closes#83520
The current `ignore_unavailable` definition is a bit misleading. The parameter primarily determines if a request that targets a missing or closed index returns an error.
In #84038 we added a dependency on having a valid `threadContext` in a
repository, but some tests use mocking and may end up with a `null`
here. This seems not to be a problem in recent branches but causes
failures in 8.0. With this commit we ensure that we always have a valid
`threadContext` to avoid any problems.
In a large cluster we expect most nodes not to have a copy of most
shards, but today during replica shard allocation we create a new (and
nontrivial) object for each node that has no copy of a shard. With this
commit we check at deserialization time whether the response is empty
and, if so, avoid the unnecessary instantiation.
Relates #77466
We have two implementations of XContentParser that both delegate all of its methods to a delegate, either an inner parser provided at construction (FilterXContentParser) or a more dynamic variant that is returned by overriding the delegate method (DelegatingXContentParser).
Effectively the two classes do exactly the same, the only difference being how the delegate parser is provided. While these two are two separate implementations, they could inherit from each other.
With this change we make FilterXContentParser be the previous DelegatingXContentParser, that allows to override the delegate method, and we introduce a new FilterXContentParserWrapper that takes the fixed delegate as a constructor argument.
Additionally, XContentSubParser is rewritten to extend FilterXContentParserWrapper.
Adds an API that can be used to find out how much memory ML
is permitted to use and is currently using on each node, both
within the JVM heap, and natively, outside of the JVM.
rewrites checkpointing as internal actions, reducing several sub-calls to
only 1 per data node that has at least 1 primary shard of the indexes of
interest.
Robustness: The current checkpointing sends a request to every shard
- primary and replica - and collects the results. If 1 request fails, even
for a replica, checkpointing fails. See #75780 for details.
Performance: The current checkpointing is wasteful, it uses get index
and get index stats which results in a lot more calls and executes a
lot more code which produces results we are not interested in.
Number of requests before and after:
before: 1 + #shards * #indices * (#replicas + 1)
after: #data_nodes_holding_gt1_shard
Fixes#75780
Today the `ResultDeduplicator` may complete a collection of listeners in
contexts different from the ones in which they were submitted. This
commit makes sure that the context is preserved in the listener.
This test was failing in rare cases when there are only few operations
and maxSeqNoOfUpdatesOrDeletes is greater then maxSeqNoOfUpdatesOrDeletesBeforeRollback
This commit removes the exception throwing and assumes that when the area is zero, the polygon has
the right orientation. The exception will be thrown at indexing time when the polygon is invalid .
We have recently seen a number of failures in ActiveDirectorySessionFactoryTests
where we fail to get a response from the Samba Server we use in
the default time frame of 5 sec. The fixture seems to be up and
running successfully so there is the case that it's just too slow
to respond within the 5 sec. This commit bumps the timeout to 15s
Authentication headers are persisted as part of a task definition including ML
jobs, CCR following etc. The persistence process store them into either an
index or the cluster state. In both cases, the headers are retrieved from
ThreadContext as a string which is the serialised form of the Authentication
object. This string is always serialised with the node's version.
The problem is: In a mixed cluster, the task can be created in a newer node and
persisted into an index but then needs to be loaded by a older node. The older
node does not understand the newer format of the serialised Authentication
object and hence error out on reading it.
This PR adds additional logic in places where the headers are persisted. It
compares the Authentication version with minNodeVersion and rewrites it if the
minNodeVersion is older. Since we already filter security headers in places
where headers are persisted, the new logic is hooked into the same places and
essentially another enhancement on how to handle security headers for persisted
tasks.
Resolves: #83567
This commit corrects the snapshot creation and restoration docs to
describe the usage of `"none"` to restore no feature states. Previously,
they incorrectly stated that using an empty array would accomplish this,
but specifying an empty array results in the default behavior (rather
than preventing feature state snapshot/restoration).
Removes the following methods from the SecurityClient component
of the High Level Rest Client
- putUser
- deleteUser
- changePassword
- authenticate
As part of this change, I renamed the SecurityClientTestHelper class
to TestSecurityClient and made it a real object rather than a set of
utility methods.
This was needed because different tests need different RequestOptions
objects, but passing it into every method made it cumbersome.
The code is clearer if we use a field in the test client itself.
Relates: #83423
This PR adds a new global privilege which can be used to restrict writes
for user profile data. The privilege is configurable for the names of
the top level keys in the profile data maps (`data` and `access`), which
by convetion are "application" names. Lastly it adds such a privilege,
for the `kibana-*` application namespace, to the `kibana_system`
built-in role.
Eg:
```
{
"global": {
"application": {
"manage": {
"applications": [...]
}
},
"profile": {
"write": {
"applications": [...]
}
}
}
}
```
Notes: * for every role there can be only one list of application names
for the write profile privilege, and the list does not support excludes
(and it supports wildcards) * there is no validation that the privilege
refers to valid application names (eg empty application name)
This adds a warning-level deprecation if a user has set the node.attr.data setting, since it is a sign that they are
trying to create a hot/warm setup in the way that is no longer supported.
Closes#83800
Christoph Büscher
Alan Woodward
David Turner
Mayya Sharipova
Joe Gallo
Ignacio Vera
Artem Prigoda
Justin Cranford
Ioannis Kakavas
Julie Tibshirani
Nik Everett
Gordon Brown
James Rodewig
Nhat Nguyen
weizijun
Martijn van Groningen
Seth Michael Larson
Yang Wang
Elasticsearch addict
Luca Cavanna
David Roberts
Hendrik Muhs
Ievgen Degtiarenko
Ioannis Kakavas
Tim Vernum
Albert Zaharovits
Keith Massey
James Baiera
Lisa Cawley
Mark Vieira