* Documents `GET _snapshot/_status` and `GET _snapshot/<repository>/_status`.
* Notes the `<repository>` and `<snasphot>` parameters are optional.
* Removes erroneous mention of the `<snapshot>` parameter supporting the `_current` value.
Closes#81600
Relates to #80931
With https://github.com/elastic/elasticsearch/pull/81870, the Azure, GCS, and S3 repository types have separate, dedicated pages in the Elasticsearch guide. For consistency, this PR creates separate pages for the shared file system, read-only URL, and source-only repository types.
Related changes:
- Adds redirects to the plugins docs
- Fixes a few breaking changes that refer to the Azure, GCS, and S3 repositories as plugins.
Co-authored-by: Adam Locke <adam.locke@elastic.co>
Closes#81652.
Convert the `repository-azure`, `repository-gcs` and `repository-s3`
plugins into modules, so that they are always included in the
Elasticsearch distribution. Also change plugin installation, removal
and syncing so that attempting to add or remove these plugins still
succeeds but is now a no-op.
Makes several changes to consolidate snapshot and backup-related docs.
Highlights:
* Adds info about supported ESS snapshot repository types
* Adds docs for Kibana's Snapshot and Restore feature
* Combines tutorial pages related to taking and managing snapshots
* Consolidates explanations of the snapshot process
* Incorporates SLM into the snapshot tutorial
* Removes duplicate "back up a cluster" pages
The put repository API doesn't accept these parameters in the request body.
Co-authored-by: Ivonne Botello <87008515+ibotello@users.noreply.github.com>
When restoring a snapshot to a new cluster, users may expect the cluster
to not contain any conflicting indices or data streams. However, some
features, such as the GeoIP processor, automatically create indices at
startup.
This adds and updates related procedures in the restore a snapshot tutorial.
I plan to improve other documentation related to feature states in snapshots
in a separate PR(s).
This PR also updates the restore snapshot API's example to include
the `indices` and `feature_states` parameters.
Relates to #79675
The create SLM policy API's `max_count` parameter limits the number of
snapshots for a policy. Only successful snapshot attempts count toward
this limit. Failed snapshot attempts do not.
Changes:
- Notes snapshot names support date math
- Sorts request body parameters alphabetically
- Adds the `expand_wildcards` request body parameter
- Reuses cluster state contents list from the restore snapshot API
- Notes the `indices` and `feature_states` parameters support a special `none` value
Relates to #79081
Related to issue #77823
This does the following:
- Updates several asciidoc files that contained code snippets with
invalid JSON, most involving unnecessary trailing commas.
- Makes the switch from the Groovy JSON parser to the Jackson parser,
pursuant to the general goal of eliminating Groovy dependence.
- Makes testing of JSON validity at build time more strict.
Note that this update still allows backslash escaping for any
character. Currently that matters because of the file
"docs/reference/ml/anomaly-detection/apis/get-datafeed-stats.asciidoc",
specifically this part:
"attributes" : {
"ml.machine_memory" :
"$body.datafeeds.0.node.attributes.ml\.machine_memory",
"ml.max_open_jobs" : "512"
}
It's not clear to me what change, if any, is appropriate there. So,
I've left in the escaped period and configured the parser to ignore
it for the time being.
Updates the put SLM policy API's `config` parameter to reuse the create snapshot API's request body parameters.
Previously, the `config` parameter was missing the `feature_states` parameter. This change should keep the two docs in sync.
This PR returns the get snapshots API to the 7.x format (and transport client behavior) and enhances it for requests that ask for multiple repositories.
The changes for requests that target multiple repositories are:
* Add `repository` field to `SnapshotInfo` and REST response
* Add `failures` map alongside `snapshots` list instead of returning just an exception response as done for single repo requests
* Pagination now works across repositories instead of being per repository for multi-repository requests
closes#69108closes#43462
Pagination and snapshots for get snapshots API, build on top of the current implementation to enable work that needs this API for testing. A follow-up will leverage the changes to make things more efficient via pagination.
Relates https://github.com/elastic/elasticsearch/pull/73570 which does part of the under-the-hood changes required to efficiently implement this API on the repository layer.
Both of these APIs don't parse request bodies, the parameters are all taken
from the query string. Also, included the master timeout param include
as it was missing here also.
Today the docs indicate that restoring a snapshot with
`include_global_state` set will merge the ingest pipelines, ILM
policies, settings etc in the snapshot with those already in the
cluster. This isn't the case, we simply replace all the things. This
commit corrects the docs.
We rely on the repository implementation correctly handling the case where a
write is aborted before it completes. This is not guaranteed for third-party
repositories.
This commit adds a rare action during analysis which aborts the write
just before it completes and verifies that the target blob is not found
by any node.
This commit adds some per-index statistics to the `SnapshotInfo` blob:
- number of shards
- total size in bytes
- maximum number of segments per shard
It also exposes these statistics in the get snapshot API.
The endpoint `_snapshottable_features` is long and implies incorrect
things about this API - it is used not just for snapshots, but also for
the upcoming reset API. Following discussions on the team, this commit
changes the endpoint to `_features` and removes the connection between
this API and snapshots, as snapshots are not the only use for the output
of this API.
Today we rely on blob stores behaving in a certain way so that they can be used
as a snapshot repository. There are an increasing number of third-party blob
stores that claim to be S3-compatible, but which may not offer a suitably
correct or performant implementation of the S3 API. We rely on somesubtle
semantics with concurrent readers and writers, but some blob stores may not
implement it correctly. Hitting a corner case in the implementation may be rare
in normal use, and may be hard to reproduce or to distinguish from an
Elasticsearch bug.
This commit introduces a new `POST /_snapshot/.../_analyse` API which exercises
the more problematic corners of the repository implementation looking for
correctness bugs and measures the details of the performance of the repository
under concurrent load.
This PR expands the meaning of `include_global_state` for snapshots to include system indices. If `include_global_state` is `true` on creation, system indices will be included in the snapshot regardless of the contents of the `indices` field. If `include_global_state` is `true` on restoration, system indices will be restored (if included in the snapshot), regardless of the contents of the `indices` field. Index renaming is not applied to system indices, as system indices rely on their names matching certain patterns. If restored system indices are already present, they are automatically deleted prior to restoration from the snapshot to avoid conflicts.
This behavior can be overridden to an extent by including a new field in the snapshot creation or restoration call, `feature_states`, which contains an array of strings indicating the "feature" for which system indices should be snapshotted or restored. For example, this call will only restore the `watcher` and `security` system indices (in addition to `index_1`):
```
POST /_snapshot/my_repository/snapshot_2/_restore
{
"indices": "index_1",
"include_global_state": true,
"feature_states": ["watcher", "security"]
}
```
If `feature_states` is present, the system indices associated with those features will be snapshotted or restored regardless of the value of `include_global_state`. All system indices can be omitted by providing a special value of `none` (`"feature_states": ["none"]`), or included by omitting the field or explicitly providing an empty array (`"feature_states": []`), similar to the `indices` field.
The list of currently available features can be retrieved via a new "Get Snapshottable Features" API:
```
GET /_snapshottable_features
```
which returns a response of the form:
```
{
"features": [
{
"name": "tasks",
"description": "Manages task results"
},
{
"name": "kibana",
"description": "Manages Kibana configuration and reports"
}
]
}
```
Features currently map one-to-one with `SystemIndexPlugin`s, but this should be considered an implementation detail. The Get Snapshottable Features API and snapshot creation rely upon all relevant plugins being installed on the master node.
Further, the list of feature states included in a given snapshot is exposed by the Get Snapshot API, which now includes a new field, `feature_states`, which contains a list of the feature states and their associated system indices which are included in the snapshot. All system indices in feature states are also included in the `indices` array for backwards compatibility, although explicitly requesting system indices included in a feature state is deprecated. For example, an excerpt from the Get Snapshot API showing `feature_states`:
```
"feature_states": [
{
"feature_name": "tasks",
"indices": [
".tasks"
]
}
],
"indices": [
".tasks",
"test1",
"test2"
]
```
Co-authored-by: William Brafford <william.brafford@elastic.co>
James Rodewig
Joe Gallo
Ievgen Degtiarenko
Rory Hunter
James Rodewig
edh-oss
Armin Braun
Mincong Huang
David Turner
Martijn van Groningen
Gordon Brown