This adds the _ilm/migrate_to_data_tiers API to expose the service for
migrating the elasticsearch abstractions (indices, ILM policies and an
optional legacy template to delete) to data tiers routing allocation
(away from custom node attributes)
Added the dimension parameter to the following field types:
keyword
ip
Numeric field types (integer, long, byte, short)
The dimension parameter is of type boolean (default: false) and is used
to mark that a field is a time series dimension field.
Relates to #74014
* [DOCS] Remove beta label for most service accounts docs
* Remove beta label from additional service account files
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
This PR returns the get snapshots API to the 7.x format (and transport client behavior) and enhances it for requests that ask for multiple repositories.
The changes for requests that target multiple repositories are:
* Add `repository` field to `SnapshotInfo` and REST response
* Add `failures` map alongside `snapshots` list instead of returning just an exception response as done for single repo requests
* Pagination now works across repositories instead of being per repository for multi-repository requests
closes#69108closes#43462
During implementation we discovered that the clusters should not
necessarily have a unique name and thus we don't need to convey
this information in the response of the Enroll Node API.
This commit adds the "in_use_by" object to the response for ILM policies. This map shows the
indices, data streams, and composable templates that use the ILM policy.
An example output may look like:
```json
{
"logs" : {
"version" : 1,
"modified_date" : "2021-06-23T18:42:08.381Z",
"policy" : {
...
},
"in_use_by" : {
"indices" : [".ds-logs-foo-barbaz-2021.06.23-000001", ".ds-logs-foo-other-2021.06.23-000001"],
"data_streams" : ["logs-foo-barbaz", "logs-foo-other"],
"composable_templates" : ["logs"]
}
}
}
```
Resolves#73869
This is a quality of life improvement for typical users. Almost all anomaly jobs will receive their data through a datafeed.
The datafeed config can now be supplied and is available in the datafeed field in the job config for creation and getting jobs.
This converts the system property feature flag 'es.shutdown_feature_flag_enabled' to a regular
non-dynamic node setting. This setting can only be set to 'true' on a snapshot build of
Elasticsearch (not a release build).
Relates to #70338
A reindex from a remote cluster doesn't support automatic or manual slicing.
This reuses a related note from the reindex docs in the upgrade docs.
Closes#54243.
Previously it was a requirement of the close job API that if the
job had an associated datafeed that that datafeed was stopped
before the job could be closed. Experience has shown that this
is just a pedantic nuisance. If a user closes the job without
first stopping the datafeed then it's just a mistake, and they
then have to make two further calls, to stop the datafeed and
then attempt to close the job again.
This PR changes the behaviour so that if you ask to close a job
whose datafeed is running then the datafeed gets stopped first
as part of the same call. Datafeeds are stopped with the same
level of force as the job close request specified.
This commit adds two related changes:
* ILM WaitForDataTierStep
* Autoscaling frozen_existence decider
The first part ensures that we wait mounting an index until a node that
can hold the index is available, avoiding a failed restore and red
cluster state. This is in particular important for the frozen phase, but
is done generically in the searchable snapshot action.
The second part triggers on indices in the ILM frozen phase to scale the
tier into existence by requiring a minimal amount of memory and storage.
Closes#72771
I was helping some folks debug an issue with the terms agg and noticed
that we didn't always have the `total_buckets` debug information. I also
noticed that we can't tell how many buckets we build, so I added that
too as `built_buckets`.
Finally, I noticed that when we're using segment ords we count segments
without any values as "multi-valued". We can do better there and count
them as no-valued. That will, mostly, just improve the profiling. When
we collect from global ords we have no way to tell how many values are
on the segment so segments without any values will, sadly, in this case
still be miscounted as multi-valued.
When we introduced dynamic:runtime (#65489) we decided to have it create objects dynamically under properties, as the runtime section did not (and still does not) support object fields. That proved to be a poor choice, because the runtime section is flat, supports dots in field names, and does not really need objects. Also, these end up causing unnecessary mapping conflicts.
With this commit we adapt dynamic:runtime to not dynamically create objects.
Closes#70268
Today we don't really describe why using `index.shard.check_on_startup`
is such a bad idea, or what to do instead. This commit expands the docs
to clarify what it does, why it's not really necessary and what to do
instead. It also now logs a warning every time the startup checks run to
encourage users to stop using this setting.
https://github.com/elastic/elasticsearch/pull/74201 documents `null` handling to the arg descriptions of several string functions.
This PR moves pre-existing docs for `null` handling and similar edge case handling for string functions to arg descriptions for consistency.
Relates to #74193
Pagination and snapshots for get snapshots API, build on top of the current implementation to enable work that needs this API for testing. A follow-up will leverage the changes to make things more efficient via pagination.
Relates https://github.com/elastic/elasticsearch/pull/73570 which does part of the under-the-hood changes required to efficiently implement this API on the repository layer.
Both of these APIs don't parse request bodies, the parameters are all taken
from the query string. Also, included the master timeout param include
as it was missing here also.
In #74138 we noted that index settings aren't copied in a clone. In fact
that's not true, we copy everything except explicitly-excluded ones,
`number_of_replicas` and `auto_expand_replicas`. This fixes the mistake.
Co-authored-by: James Rodewig <40268737+jrodewig@users.noreply.github.com>
Today if sending file chunks is CPU-bound (e.g. when using compression)
then we tend to concentrate all that work onto relatively few threads,
even if `indices.recovery.max_concurrent_file_chunks` is increased. With
this commit we fork the transmission of each chunk onto its own thread
so that the CPU-bound work can happen in parallel.
Andrei Dan
Przemko Robakowski
Ignacio Vera
Armin Braun
James Rodewig
Christos Soulios
Adam Locke
Lisa Cawley
Ioannis Kakavas
Lee Hinman
Robin Clarke
Benjamin Trent
bellengao
David Roberts
Henning Andersen
Dan Hermann
Nik Everett
Yang Cheng
Luca Cavanna
Hendrik Muhs
István Zoltán Szabó
David Turner
Yang Wang
François-Clément Brossard
Nicole Albee