2020-01-10 08:12:02 +08:00
[[snapshots-restore-snapshot]]
2020-07-13 20:34:42 +08:00
== Restore a snapshot
2020-01-10 08:12:02 +08:00
////
[source,console]
-----------------------------------
PUT /_snapshot/my_backup
{
"type": "fs",
"settings": {
"location": "my_backup_location"
}
}
PUT /_snapshot/my_backup/snapshot_1?wait_for_completion=true
-----------------------------------
// TESTSETUP
////
2020-07-22 23:42:17 +08:00
Use the <<restore-snapshot-api,restore snapshot API>> to restore
a snapshot of a cluster or specified data streams and indices:
2020-01-10 08:12:02 +08:00
[source,console]
-----------------------------------
POST /_snapshot/my_backup/snapshot_1/_restore
-----------------------------------
2020-07-22 23:42:17 +08:00
// TEST[s/_restore/_restore?wait_for_completion=true/]
2020-01-10 08:12:02 +08:00
2020-07-13 20:34:42 +08:00
By default, all data streams and indices in the snapshot are restored, but the cluster state is
2020-07-22 23:42:17 +08:00
*not* restored. Use the `indices` parameter to restore only specific data streams or indices. This parameter
2021-07-01 00:24:42 +08:00
supports <<api-multi-index,multi-target syntax>>. To include the global cluster state, set
2020-07-22 23:42:17 +08:00
`include_global_state` to `true` in the restore request body.
2020-07-13 20:34:42 +08:00
Introduce "Feature States" for managing snapshots of system indices (#63513)
This PR expands the meaning of `include_global_state` for snapshots to include system indices. If `include_global_state` is `true` on creation, system indices will be included in the snapshot regardless of the contents of the `indices` field. If `include_global_state` is `true` on restoration, system indices will be restored (if included in the snapshot), regardless of the contents of the `indices` field. Index renaming is not applied to system indices, as system indices rely on their names matching certain patterns. If restored system indices are already present, they are automatically deleted prior to restoration from the snapshot to avoid conflicts.
This behavior can be overridden to an extent by including a new field in the snapshot creation or restoration call, `feature_states`, which contains an array of strings indicating the "feature" for which system indices should be snapshotted or restored. For example, this call will only restore the `watcher` and `security` system indices (in addition to `index_1`):
```
POST /_snapshot/my_repository/snapshot_2/_restore
{
"indices": "index_1",
"include_global_state": true,
"feature_states": ["watcher", "security"]
}
```
If `feature_states` is present, the system indices associated with those features will be snapshotted or restored regardless of the value of `include_global_state`. All system indices can be omitted by providing a special value of `none` (`"feature_states": ["none"]`), or included by omitting the field or explicitly providing an empty array (`"feature_states": []`), similar to the `indices` field.
The list of currently available features can be retrieved via a new "Get Snapshottable Features" API:
```
GET /_snapshottable_features
```
which returns a response of the form:
```
{
"features": [
{
"name": "tasks",
"description": "Manages task results"
},
{
"name": "kibana",
"description": "Manages Kibana configuration and reports"
}
]
}
```
Features currently map one-to-one with `SystemIndexPlugin`s, but this should be considered an implementation detail. The Get Snapshottable Features API and snapshot creation rely upon all relevant plugins being installed on the master node.
Further, the list of feature states included in a given snapshot is exposed by the Get Snapshot API, which now includes a new field, `feature_states`, which contains a list of the feature states and their associated system indices which are included in the snapshot. All system indices in feature states are also included in the `indices` array for backwards compatibility, although explicitly requesting system indices included in a feature state is deprecated. For example, an excerpt from the Get Snapshot API showing `feature_states`:
```
"feature_states": [
{
"feature_name": "tasks",
"indices": [
".tasks"
]
}
],
"indices": [
".tasks",
"test1",
"test2"
]
```
Co-authored-by: William Brafford <william.brafford@elastic.co>
2021-02-12 02:55:14 +08:00
Because all indices in the snapshot are restored by default, all system indices will be restored
by default as well.
2020-07-13 20:34:42 +08:00
[WARNING]
====
Each data stream requires a matching
2021-04-01 05:28:55 +08:00
<<create-index-template,index template>>. The stream uses this
2020-07-13 20:34:42 +08:00
template to create new backing indices.
When restoring a data stream, ensure a matching template exists for the stream.
You can do this using one of the following methods:
* Check for existing templates that match the stream. If no matching template
2021-04-01 05:28:55 +08:00
exists, <<create-index-template,create one>>.
2020-07-13 20:34:42 +08:00
* Restore a global cluster state that includes a matching template for the
stream.
If no index template matches a data stream, the stream cannot
<<manually-roll-over-a-data-stream,roll over>> or create new backing indices.
====
The `rename_pattern`
and `rename_replacement` options can be also used to rename data streams and indices on restore
2020-07-22 23:42:17 +08:00
using regular expression that supports referencing the original text, according to the https://docs.oracle.com/javase/8/docs/api/java/util/regex/Matcher.html#appendReplacement-java.lang.StringBuffer-java.lang.String-[`appendReplacement`] logic.
2020-07-13 20:34:42 +08:00
2020-07-22 23:42:17 +08:00
[[rename-restored-data-stream]]
// tag::rename-restored-data-stream-tag[]
2020-07-13 20:34:42 +08:00
If you rename a restored data stream, its backing indices are also
renamed. For example, if you rename the `logs` data stream to `restored-logs`,
2020-12-15 06:46:54 +08:00
the backing index `.ds-logs-2099.03.09-000005` is renamed to
`.ds-restored-logs-2099.03.09-000005`.
2020-07-13 20:34:42 +08:00
[WARNING]
====
If you rename a restored stream, ensure an index template matches the new stream
name. If no index template matches the stream, it cannot
<<manually-roll-over-a-data-stream,roll over>> or create new backing indices.
====
2020-07-22 23:42:17 +08:00
// end::rename-restored-data-stream-tag[]
2020-07-13 20:34:42 +08:00
2021-06-03 20:43:14 +08:00
To prevent aliases from being restored with their associated data streams and
indices, set `include_aliases` to `false`.
2020-01-10 08:12:02 +08:00
[source,console]
-----------------------------------
POST /_snapshot/my_backup/snapshot_1/_restore
{
2020-07-13 20:34:42 +08:00
"indices": "data_stream_1,index_1,index_2",
2020-01-10 08:12:02 +08:00
"ignore_unavailable": true,
2020-05-26 23:59:50 +08:00
"include_global_state": false, <1>
2020-01-10 08:12:02 +08:00
"rename_pattern": "index_(.+)",
2020-05-26 23:59:50 +08:00
"rename_replacement": "restored_index_$1",
"include_aliases": false
2020-01-10 08:12:02 +08:00
}
-----------------------------------
// TEST[continued]
2020-05-26 23:59:50 +08:00
<1> By default, `include_global_state` is `false`, meaning the snapshot's
Introduce "Feature States" for managing snapshots of system indices (#63513)
This PR expands the meaning of `include_global_state` for snapshots to include system indices. If `include_global_state` is `true` on creation, system indices will be included in the snapshot regardless of the contents of the `indices` field. If `include_global_state` is `true` on restoration, system indices will be restored (if included in the snapshot), regardless of the contents of the `indices` field. Index renaming is not applied to system indices, as system indices rely on their names matching certain patterns. If restored system indices are already present, they are automatically deleted prior to restoration from the snapshot to avoid conflicts.
This behavior can be overridden to an extent by including a new field in the snapshot creation or restoration call, `feature_states`, which contains an array of strings indicating the "feature" for which system indices should be snapshotted or restored. For example, this call will only restore the `watcher` and `security` system indices (in addition to `index_1`):
```
POST /_snapshot/my_repository/snapshot_2/_restore
{
"indices": "index_1",
"include_global_state": true,
"feature_states": ["watcher", "security"]
}
```
If `feature_states` is present, the system indices associated with those features will be snapshotted or restored regardless of the value of `include_global_state`. All system indices can be omitted by providing a special value of `none` (`"feature_states": ["none"]`), or included by omitting the field or explicitly providing an empty array (`"feature_states": []`), similar to the `indices` field.
The list of currently available features can be retrieved via a new "Get Snapshottable Features" API:
```
GET /_snapshottable_features
```
which returns a response of the form:
```
{
"features": [
{
"name": "tasks",
"description": "Manages task results"
},
{
"name": "kibana",
"description": "Manages Kibana configuration and reports"
}
]
}
```
Features currently map one-to-one with `SystemIndexPlugin`s, but this should be considered an implementation detail. The Get Snapshottable Features API and snapshot creation rely upon all relevant plugins being installed on the master node.
Further, the list of feature states included in a given snapshot is exposed by the Get Snapshot API, which now includes a new field, `feature_states`, which contains a list of the feature states and their associated system indices which are included in the snapshot. All system indices in feature states are also included in the `indices` array for backwards compatibility, although explicitly requesting system indices included in a feature state is deprecated. For example, an excerpt from the Get Snapshot API showing `feature_states`:
```
"feature_states": [
{
"feature_name": "tasks",
"indices": [
".tasks"
]
}
],
"indices": [
".tasks",
"test1",
"test2"
]
```
Co-authored-by: William Brafford <william.brafford@elastic.co>
2021-02-12 02:55:14 +08:00
cluster state and feature states are not restored.
2020-05-26 23:59:50 +08:00
+
2021-05-19 23:16:41 +08:00
If `true` then the restore operation merges the legacy index templates in your
cluster with the templates contained in the snapshot, replacing any existing
ones whose name matches one in the snapshot. It completely removes all
persistent settings, non-legacy index templates, ingest pipelines and
{ilm-init} lifecycle policies that exist in your cluster and replaces them with
the corresponding items from the snapshot.
2020-05-26 23:59:50 +08:00
2020-07-22 23:42:17 +08:00
The restore operation must be performed on a functioning cluster. However, an
2021-08-10 22:02:00 +08:00
existing index can only be restored if it's <<indices-close,closed>> and
2020-01-10 08:12:02 +08:00
has the same number of shards as the index in the snapshot. The restore
operation automatically opens restored indices if they were closed and creates
2020-05-26 23:59:50 +08:00
new indices if they didn't exist in the cluster.
2020-01-10 08:12:02 +08:00
2020-07-13 20:34:42 +08:00
If a data stream is restored, its backing indices are also restored. The restore
operation automatically opens restored backing indices if they were closed.
In addition to entire data streams, you can restore only specific backing
indices from a snapshot. However, restored backing indices are not automatically
2020-12-15 06:46:54 +08:00
added to any existing data streams. For example, if only the
`.ds-logs-2099.03.08-000003` backing index is restored from a snapshot, it is
not automatically added to the existing `logs` data stream.
2020-07-13 20:34:42 +08:00
2020-07-22 23:42:17 +08:00
[discrete]
2020-01-10 08:12:02 +08:00
=== Partial restore
2020-07-13 20:34:42 +08:00
By default, the entire restore operation will fail if one or more indices or backing indices participating in the operation don't have
2020-01-10 08:12:02 +08:00
snapshots of all shards available. It can occur if some shards failed to snapshot for example. It is still possible to
restore such indices by setting `partial` to `true`. Please note, that only successfully snapshotted shards will be
restored in this case and all missing shards will be recreated empty.
2020-07-22 23:42:17 +08:00
[discrete]
2020-11-25 08:59:52 +08:00
[[change-index-settings-during-restore]]
2020-01-10 08:12:02 +08:00
=== Changing index settings during restore
2020-07-22 23:42:17 +08:00
Use the <<restore-snapshot-api-index-settings,`index_settings`>> parameter
to override index settings during the restore process. For example, the
following request will restore the index `index_1` without creating any
replicas while switching back to the default refresh interval:
2020-01-10 08:12:02 +08:00
[source,console]
-----------------------------------
POST /_snapshot/my_backup/snapshot_1/_restore
{
"indices": "index_1",
"ignore_unavailable": true,
"index_settings": {
"index.number_of_replicas": 0
},
"ignore_index_settings": [
"index.refresh_interval"
]
}
-----------------------------------
// TEST[continued]
2020-07-22 23:42:17 +08:00
NOTE: Some settings such as `index.number_of_shards` cannot be changed during the restore operation.
2020-01-10 08:12:02 +08:00
2020-07-13 20:34:42 +08:00
For data streams, these index settings are applied to the restored backing
indices.
2020-07-22 23:42:17 +08:00
// tag::index-settings-data-stream-warning[]
2020-07-13 20:34:42 +08:00
[IMPORTANT]
====
The `index_settings` and `ignore_index_settings` parameters affect
restored backing indices only. New backing indices created for a stream use the index
settings specified in the stream's matching
2021-04-01 05:28:55 +08:00
<<create-index-template,index template>>.
2020-07-13 20:34:42 +08:00
If you change index settings during a restore, we recommend you make similar
changes in the stream's matching index template. This ensures new backing
indices created for the stream use the same index settings.
====
2020-07-22 23:42:17 +08:00
// end::index-settings-data-stream-warning[]
2020-07-13 20:34:42 +08:00
2020-07-22 23:42:17 +08:00
[discrete]
2020-01-10 08:12:02 +08:00
=== Restoring to a different cluster
2020-07-22 23:42:17 +08:00
The information stored in a snapshot is not tied to a particular cluster or a cluster name. Therefore, it's possible to
restore a snapshot created from one cluster into another cluster by registering the repository that contains the snapshot in the new cluster and starting the restore process.
2020-01-10 08:12:02 +08:00
2020-07-22 23:42:17 +08:00
The topology of the new cluster does not have to match the cluster where the snapshot was created. However, the version of the new cluster must be the same or only one major version newer than the cluster that was used to create the snapshot. For example, you can restore a 5.x snapshot to a 6.x cluster, but not a 5.x snapshot to a 7.x cluster.
If the new cluster has a smaller capacity, consider the following changes:
* Ensure that the new cluster has enough capacity to store all data streams and indices in the snapshot.
* Change the index settings during the restore operation to reduce the
<<dynamic-index-number-of-replicas,number of replicas>>.
* Use the `indices` parameter to choose only specific data streams or indices to restore.
2020-01-10 08:12:02 +08:00
2020-07-13 20:34:42 +08:00
If indices or backing indices in the original cluster were assigned to particular nodes using
2020-07-22 23:42:17 +08:00
<<shard-allocation-filtering,shard allocation filtering>>, the same rules will be enforced in the new cluster. If the new cluster does not contain nodes with appropriate attributes that a restored index can be allocated on, the
index will not be successfully restored unless these index allocation settings are changed during the restore operation.
2020-01-10 08:12:02 +08:00
The restore operation also checks that restored persistent settings are compatible with the current cluster to avoid accidentally
restoring incompatible settings. If you need to restore a snapshot with incompatible persistent settings, try restoring it without
2020-07-22 23:42:17 +08:00
the <<restore-snapshot-api-include-global-state,global cluster state>>.
