We added configuration to AllocateAction to set the total shards per node property on the index. This makes it possible that a user could set this to a value lower than the total number of shards in the index that is about to be shrunk, meaning that all of the shards could not be moved to a single node in the ShrinkAction. This commit unsets the total shards per node property so that we fall back to the default value (-1, unlimited) in the ShrinkAction to avoid this.
Relates to #44070
This adds a new optional field to the allocate ILM action called "total_shards_per_node". If present, the
value of this field is set as the value of "index.routing.allocation.total_shards_per_node" before the allocation
takes place.
Relates to #44070
* Allow ILM move-to-step without `action` or `name`
This commit enhances ILM's move-to-step API to allow dropping the `name`, or dropping both the
`action` and `name`. For example:
```json
POST /_ilm/move/foo-1
{
"current_step": {
"phase": "hot",
"action": "rollover",
"name": "check-rollover-ready"
},
"next_step": {
"phase": "warm",
"action": "forcemerge"
}
}
```
Will move to the first step in the `forcemerge` action in the `warm` phase (without having to know
the specific step name).
Another example:
```json
POST /_ilm/move/foo-1
{
"current_step": {
"phase": "hot",
"action": "rollover",
"name": "check-rollover-ready"
},
"next_step": {
"phase": "warm"
}
}
```
Will move to the first step in the `warm` phase (without having to know the specific action name).
Bear in mind that the execution order is still entirely an implementation detail, so "first" in the
above sentences means the first step that ILM would execute.
Resolves#58128
* Apply Andrei's wording change (thanks!)
Co-authored-by: Andrei Dan <andrei.dan@elastic.co>
* Log index and policy name when the concrete step key can't be resolved
Co-authored-by: Andrei Dan <andrei.dan@elastic.co>
This adds support for a `dry_run` parameter for the
`_ilm/migrate_to_data_tiers` API. This defaults to `false`, but when
configured to `true` it will simulate the migration of elasticsearch
entities to data tiers based routing, returning the entites that need to
be updated (indices, ILM policies and the legacy index template that'd
be deleted, if any was configured in the request).
To switch an index's lifecycle policy, you must first remove the existing
policy. Otherwise, phase execution for the index may silently fail.
Closes#70151
This adds the _ilm/migrate_to_data_tiers API to expose the service for
migrating the elasticsearch abstractions (indices, ILM policies and an
optional legacy template to delete) to data tiers routing allocation
(away from custom node attributes)
This commit adds the "in_use_by" object to the response for ILM policies. This map shows the
indices, data streams, and composable templates that use the ILM policy.
An example output may look like:
```json
{
"logs" : {
"version" : 1,
"modified_date" : "2021-06-23T18:42:08.381Z",
"policy" : {
...
},
"in_use_by" : {
"indices" : [".ds-logs-foo-barbaz-2021.06.23-000001", ".ds-logs-foo-other-2021.06.23-000001"],
"data_streams" : ["logs-foo-barbaz", "logs-foo-other"],
"composable_templates" : ["logs"]
}
}
}
```
Resolves#73869
Relates to #70755.
The main changes of this PR are:
Add an optional _meta field to ILM policy.
Add some test code about the change.
Update the doc of Create or update lifecycle policy API.
* [DOCS] Document how to migrate to node roles from node attrs. Closes#65855
* [DOCS] Incorporated review comments
* Update docs/reference/data-management/migrate-index-allocation-filters.asciidoc
Co-authored-by: Andrei Dan <andrei.dan@elastic.co>
If enabled, the `delete_searchable_snapshot` option will attempt to delete the
index snapshot generated in any previous phase, for the purpose of mounting the
index as a searchable snapshot.
This aims at making the shrink action retryable. Every step is
retryable, but in order to provide an experience where ILM tries
to achieve a successful shrink even when the target node goes
missing permanently or the shrunk index cannot recover, this also
introduces a retryable shrink cycle within the shrink action.
The shrink action will generate a unique index name that'll be the
shrunk index name. The generated index name is stored in the lifecycle
state.
If the shrink action ends up waiting for the source shards to
colocate or for the shrunk index to recover for more than the configured
`LIFECYCLE_STEP_WAIT_TIME_THRESHOLD` setting, it will move back
to clean up the attempted (and failed) shrunk index and will retry
generating a new index name and attempting to shrink the source
to the newly generated index name.
This commit changes the frozen phase within ILM in the following ways:
- The `searchable_snapshot` action now no longer takes a `storage` parameter. The storage type is
determined by the phase within which it is invoked (shared cache for frozen and full copy for
everything else).
- The frozen phase in ILM now no longer allows *any* actions other than `searchable_snapshot`
- If a frozen phase is provided, it *must* include a `searchable_snapshot` action.
These changes may seem breaking, but since they are intended to go back to 7.12 which has not been
released yet, they are not truly breaking changes.
It can be confusing to configure policies with phase timings that get smaller, because phase timings
are absolute. To make things a little clearer, this commit now rejects policies where a configured
min_age is less than a previous phase's min_age.
This validation is added only to the PutLifecycleAction.Request instead of the
TimeseriesLifecycleType class because we cannot do this validation every time a lifecycle is
created or else we will block cluster state from being recoverable for existing clusters that may
have invalid policies.
Resolves#70032
This commit moves away from the static `rollup-{indexName}` rollup index
naming strategy and moves towards a randomized rollup index name scheme.
This will reduce the complications that exist if the RollupStep fails and retries
in any way. A separate cleanup will still be required for failed temporary indices,
but at least there will not be a conflict.
This commit generates the new rollup index name in the LifecycleExecutionState so
that it can be used in RollupStep and UpdateRollupIndexPolicyStep on a per-index
basis.
This moves the execution of the `searchable_snapshot` action before the
`migrate` action in the `cold` and `frozen` phases for more efficient
data migration (ie. mounting it as a searchable snapshot directly on the
target tier)
Now that searchable_snapshot can precede other actions in the same phase
(eg. in frozen it is followed by `migrate`) we need to allow the mounted
index to resume executing the ILM policy starting with a step that's part
of a new action (ie. migrate).
This adds support to resume the execution of the mounted index from another
action.
With older versions, the execution would resume from the PhaseCompleteStep
as it was the last action in a phase, which was handled as a special case
in the `CopyExecutionStateStep`. This generalises the `CopyExecutionStateStep`
to be able to resume from any `StepKey`.
This commit adds support for the recently introduced partial searchable snapshot (#68509) to ILM.
Searchable snapshot ILM actions may now be specified with a `storage` option, specifying either
`full_copy` or `shared_cache` (similar to the "mount" API) to mount either a full or partial
searchable snapshot:
```json
PUT _ilm/policy/my_policy
{
"policy": {
"phases": {
"cold": {
"actions": {
"searchable_snapshot" : {
"snapshot_repository" : "backing_repo",
"storage": "shared_cache"
}
}
}
}
}
}
```
Internally, If more than one searchable snapshot action is specified (for example, a full searchable
snapshot in the "cold" phase and a partial searchable snapshot in the "frozen" phase) ILM will
re-use the existing snapshot when doing the second mount since a second snapshot is not required.
Currently this is allowed for actions that use the same repository, however, multiple
`searchable_snapshot` actions for the same index that use different repositories is not allowed (the
ERROR state is entered). We plan to allow this in the future in subsequent work.
If the `storage` option is not specified in the `searchable_snapshot` action, the mount type
defaults to "shared_cache" in the frozen phase and "full_copy" in all other phases.
Relates to #68605
This commit adds the `data_frozen` node role as part of the formalization of data tiers. It also
adds the `"frozen"` phase to ILM, currently allowing the same actions as the existing cold phase.
The frozen phase is intended to be used for data even less frequently searched than the cold phase,
and will eventually be loosely tied to data using partial searchable snapshots (as oppposed to full
searchable snapshots in the cold phase).
Relates to #60848
Changes:
- Reworks the ILM tutorial to focus on the Elastic Agent and a built-in ILM policy
- Updates several screenshots in the docs for the new ILM UI
Co-authored-by: debadair <debadair@elastic.co>
debadair
Keith Massey
Lee Hinman
Stef Nestor
Andrei Dan
James Rodewig
bellengao
Frederic Dartayre
Henning Andersen
Joe Gallo
Tal Levy
Adam Locke