Now that #74115 is backported to 7.x the code to check whether
local abort is supported within a cluster is redundant, as 8.x
only supports running in a mixed cluster with 7.last, and 7.last
contains the local abort functionality.
This change removes the redundant code.
Followup to #74115
synced flush is going to be replaced by flush. This commit allows to synced_flush api only in v7 compatibility mode.
Worth noting - sync_id is gone and won't be available in v7 responses from indices.stats
relates removal pr #50882
relates #51816
This assertion was simply wrong, but took time to catch as it was very rarely
triggered. The test already checks the correct assertion at an earlier time.
Fixes#75736.
In this commit we extend the Fields API formats for geo data in order to produce vector tiles features directly
on the data nodes. That helps the vector tile API to reduce the size of the data that needs to pull in order to
create the answer.
* Adding shard count to _nodes/stats api
Added a shards section to each node returned by the _nodes/stats api. Currently this new section only contains a total count of all shards on the node.
The node executing a shard level operation would in many cases communicate `null` for the shard state update,
leading to follow-up operations incorrectly assuming an empty shard snapshot directory and starting from scratch.
closes#75598
Follow up to #75332, there is a somewhat unlikely but possible edge case
left in this test. If the random deletes execute before the clones
we can see the clones conflicting.
Also, even less likely the snapshot delete could have finished before the
clone completes so we have to catch that case also.
Today when a task is cancelled we record the reason for the cancellation
but this information is very rarely exposed to users. This commit
centralises the construction of the `TaskCancellationException` and
includes the reason in the exception message.
Closes#74825
Even though it would be nice to wait for the clones and restores to queue up before
triggering the deletes this does not work. At times the clones or restores are so quick
that they complete before we even get to look at the cluster state, thus making the test
wait for a state that is never to come and fail.
We only create a `ReceiveTimeoutTransportException` in one place, the
timeout handler for the corresponding transport request, so the stack
trace contains no useful information and just adds noise if ever it is
logged. With this commit we drop the stack trace from these exceptions.
Today when a node is removed from the cluster because of repeated
follower check failures it is not clear whether those failures are due
to active rejections or timeouts. It's important for further
investigation to distinguish these two cases. This commit adds tracking
for the number of timeouts separately from the number of active
failures, and includes the counts of each in the removal reason.
When the node shutdown API is used to indicate that
an ML node is going to be shut down, ML will do the
following:
1. Isolate all datafeeds on the node and unassign
their persistent tasks.
2. Tell all anomaly detection jobs on the node to go
through the motions of closing, but not actually
close and instead unassign their persistent tasks.
3. Report that the node is safe to shut down once all
persistent tasks associated with anomaly detection
jobs and model snapshot upgrades have either
completed or been unassigned.
When performing incremental reductions, 0 value of docCountError may mean that
the error was not previously calculated, or that the error was indeed previously
calculated and its value was 0. We end up rejecting true values set to 0 this
way. This may lead to wrong upper bound of error in result. To fix it, this PR
makes docCountError nullable. null values mean that error was not calculated
yet.
Fixes#40005
Co-authored-by: Igor Motov <igor@motovs.org>
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
In #75454 we changed our dynamic shadowing logic to check that an unmapped
field was truly shadowed by a runtime field before returning no-op mappers. However,
this does not handle the case where the runtime field can have multiple subfields, as
will be true for the upcoming composite field type. We instead need to check that
the field in question would not be shadowed by any field type returned by any
runtime field.
This commit abstracts this logic into a new isShadowed() method on
DocumentParserContext, which uses a set of runtime field type names built from
the mapping lookup at construction time. It also simplifies the no-op mapper
slightly by making it a singleton object, as we don't need to preserve field names
here.
This commit adds a new master transport action TransportGetShardSnapshotAction
that allows getting the last successful snapshot for a particular
shard in a set of repositories. It deals with the different
implementation details around BwC for repositories.
Relates #73496
Closes#61767.
When Elasticsearch fails to start up, it tries to print a message to
suggest to the user that they consult the log for more information.
However, if a fatal error is encountered, Elasticsearch exits directly and
no such message is printed.
Improve this situation by printing the same message when Elasticsearch
is about to exit due to a fatal unexpected error.
This commit adds a new p_value score heuristic to significant terms.
The p_value is calculating assuming that the foreground set and the background set are independent Bernoulli trials with the null hypothesis that the probabilities are the same.
We have to put the entry into `endingSnapshots`, otherwise we trip
an assertion about listener consistency with the cluster state if
there's other concurrent operations.
The original PR #75264 made some test mistakes
NXY Significant term heuristics have additional values that need to be set when testing
basicScore properties.
Additionally, previous refactor kept the abstract test class in a package that other plugins
don't have access to.
closes#75442, #75561
This is likely one source of bugs in at least snapshotting as it can lead
to looking up the wrong index from an old shard id (if an index has been
deleted and a new index is created in its place concurrently)
In the upcoming Lucene 9 release, `indices.query.bool.max_clause_count` is
going to apply to the entire query tree rather than per `bool` query. In order
to avoid breaks, the limit has been bumped from 1024 to 4096.
The semantics will effectively change when we upgrade to Lucene 9, this PR
is only about agreeing on a migration strategy and documenting this change.
To avoid further breaks, I am leaning towards keeping the current setting name
even though it contains `bool`. I believe that it still makes sense given that
`bool` queries are typically the main contributors to high numbers of clauses.
Co-authored-by: James Rodewig <40268737+jrodewig@users.noreply.github.com>
Prior to this change, supplying a wildcard expression in the `indices` field
of an alias action would always result in a 404, despite data streams existing
that could match with the provided wildcard expression.
Closes#75456
This introduces a new cmd line tool that generates the security configuration
for a new node in a new cluster (as opposed to joining an existing cluster).
The security configuration consists of TLS key and certificates, which
are stored in a directory inside the config path, as well as settings appended
to the elasticsearch.yml referencing the aforementioned certs.
Date histogram interval parameter was deprecated in 7.2, in favor of the more specific fixed_interval and calendar_interval parameters. The old logic used some poorly understood guessing to decide if it should operate in fixed or calendar mode. The new logic requires a specific choice by the user, which is more explicit. In 7.x REST compatibility mode, we will parse the interval as calendar if possible, and otherwise interpret it as fixed.
In #74678 we released an early fix for a Lucene bug around `combined_fields`
queries with missing fields. This PR brings our fix up-to-date with what was
actually committed to Lucene.
We should use getStringRep instead of toString to avoid serializing the
keep_alive of a point in time in fractional format; otherwise, we won't
be able to parse it.
Fixes#75446
Previously the equals and hashCode methods only compared the query terms. This
meant that queries on different fields, or with different field weights, were
considered equal.
During boolean query rewrites, duplicate clauses are removed. So because equals/
hashCode was incorrect, rewrites could accidentally drop combined_fields query
clauses.
We used to throw a parsing exception when trying to overwrite a multi field on
indexing a document. These errors got lost on some recent changes in
DocumentParser around runtime fields and not detected in tests. This PR adds a
test for this case and restores parts of the logic in
DocumentParser#getLeafMapper to prevent this kind of overwrite from happening.
Fixes#75343
Previously the compatibility layer was always returning an _doc in mappings for get
template.
This commit does not return _doc in empty mappings.
Returning just {} empty object (v7 and v8 behaviour)
also moving term lookups tests which are already fixed (relates #74544)
relates #70966
relates main meta issue #51816
relates types removal meta #54160
Nested objects are implemented via a Nested class directly on object mappers,
even though nested and non-nested objects have quite different semantics. In
addition, most call-sites that need to get an object mapper in fact need a nested
object mapper. To make it clearer that nested and object mappers are different
beasts with different implementations and different requirements, we should
split them into different classes.
* Fix up shard generations in `SnapshotsInProgress` during snapshot finalization (don't do it earlier because it's a really heavy computation and we have a ton of places where it would have to run).
* Adjust finalization queue to be able to work with changing snapshot entries after they've been enqueued for finalisation
* Still one remaining bug left after this (see TODO about leaking generations) that I don't feel confident in fixing for `7.13.4` due to the complexity of a fix and how minor the blob leak is (+ it's cleaned up just fine during snapshot deletes)
Closes#75336
previously disallowed in #70209. However since now the recommendation is
to not set the field at all - and rely on a default - this would be a
shape change for users prefering to set -1 so far.
With request compatible with v7 they will still be allowed to do this
and will get a warning.
relates #69548
relates #51816
Some system indices, such as .tasks, had integer _meta.version
properties before we switched to using an Elasticsearch version there.
This commit ensures the code that reads the _meta.version from the
mapping can handle integers as well as "real" versions.
Today there is no relationship between the lifecycle of a
snapshot mounted as an index and the lifecycle of index
itself. This lack of relationship makes it possible to delete
a snapshot in a repository while the mounted index still
exists, causing the shards to fail in the future.
On the other hand creating a snapshot that contains a
single index to later mount it as a searchable snapshot
index becomes more and more natural for users and
ILM; but it comes with the risk to forget to delete the
snapshot when the searchable snapshot index is
deleted from the cluster, "leaking" snapshots in
repositories.
We'd like to improve the situation and provide a way
to automatically delete the snapshot when the mounted
index is deleted. To opt in for this behaviour, a user has
to enable a specific index setting when mounting the
snapshot (the proposed name for the setting is
index.store.snapshot.delete_searchable_snapshot).
Elasticsearch then verifies that the snapshot to mount
contains only 1 snapshotted index and that the snapshot
is not used by another mounted index with a different
value for the opt in setting, and mounts the snapshot
with the new private index setting.
This is the part implemented in this commit.
In follow-up pull requests this index setting will be used
when the last mounted index is deleted and removed
from the cluster state in order to add the searchable
snapshot id to a list of snapshots to delete in the
repository metadata. Snapshots that are marked as
"to delete" will not be able to be restored or cloned, and
Elasticsearch will take care of deleting the snapshot as
soon as possible. Then ILM will be changed to use this
setting when mounting a snapshot as a cold or frozen
index and delete_searchable_snapshot option in ILM
will be removed. Finally, deleting a snapshot that is
still used by mounted indices will be prevented.
Elasticsearch's server sometimes has to do things
differently in order to handle searchable snapshots
shards. In such cases it relies on index settings
and hard coded values to know if a shard is a
searchable snapshot one.
This commit adds a new SearchableSnapshotsSettings
class in server that provides methods to check if
an index is a searchable snapshot index. This class
also contains the names of some index settings
related to searchable snapshots that are required
by the server.
This PR implements a system index access level that includes all system indices, except net-new system indices.
This new access level is leveraged to fix the error in the Get Alias API described in #74687.
David Roberts
Keith Massey
Przemyslaw Gomulka
Julie Tibshirani
Ignacio Vera
Rory Hunter
Armin Braun
Yannick Welsch
David Turner
Nikita Glashenko
Dan Hermann
Alan Woodward
Francisco Fernández Castaño
Benjamin Trent
Martijn van Groningen
Adrien Grand
Albert Zaharovits
Mark Tozzi
Julie Tibshirani
Joe Gallo
Nhat Nguyen
Christoph Büscher
Christos Soulios
Gordon Brown
Tanguy Leroux