Commit Graph

103 Commits

Author SHA1 Message Date
Armin Braun 0920e21445
Implement Sort By Repository Name in Get Snapshots API (#77049)
This one is the last sort column not yet implemented but used by Kibana.
2021-09-01 13:01:58 +02:00
Armin Braun 48f3784a6d
Add Sort By Shard Count and Failed Shard Count to Get Snapshots API (#77011)
It's in the title. As requested by the Kibana team, adding these two additional sort columns.

relates #74350
2021-08-30 13:39:51 +02:00
James Rodewig 38fe33a870 [DOCS] Fix whitespace to hide attribute 2021-08-22 21:18:51 -04:00
James Rodewig 1c355bfda9
[DOCS] Reuse snapshot config in put SLM policy API docs (#76712)
Updates the put SLM policy API's `config` parameter to reuse the create snapshot API's request body parameters.
Previously, the `config` parameter was missing the `feature_states` parameter. This change should keep the two docs in sync.
2021-08-20 08:29:16 -04:00
James Rodewig 5c954df521
[DOCS] Fix data type for create snapshot API's `metadata` param (#76465) 2021-08-12 16:38:56 -04:00
James Rodewig 20f000e95a
[DOCS] Fix query params for create snapshot API (#76436)
Moves the `master_timeout` and `wait_for_completion` parameters to a query parameters section.
2021-08-12 13:11:12 -04:00
Glen Smith 263d9f2dac
[DOCS] Fix typo (#76213)
"can be only restored" -> "can only be restored"
2021-08-10 10:02:00 -04:00
Armin Braun ffaa0f2742
Implement Numeric Offset Parameter in Get Snapshots API (#76233)
Add numeric offset parameter to this API.

Relates #74350
2021-08-09 16:23:43 +02:00
Armin Braun a4983f5ab9
Return Total Result Count and Remaining Count in Get Snapshots Response (#76150)
Add total result count and remaining count to get snapshots response.
2021-08-09 11:34:24 +02:00
debadair 29bb8fd67e
[DOCS] Add info about GET/POST to the API conventions. (#74529)
* [DOCS] Add info about GET/POST to the API conventions.

* Fixed anchor

* Added redirect

* Incorporated review suggestions
2021-06-30 09:24:42 -07:00
Mincong Huang dc70bd6092
[DOCS] Fix missing word for snapshot API (#74657) 2021-06-29 08:07:17 +01:00
Armin Braun 5f89f8be3f
Introduce Next Field in Paginated GetSnapshots Response (#74236)
Follow up to #73952 adding documentation for the `after` query parameter
and the related `next` response field.
2021-06-28 22:28:30 +02:00
Armin Braun 939c4c6c3f
Remove Repository Size Limit Default (#74592)
Now that we actively improve the scalability there is no point
of having a `500` limit in `master`.
2021-06-27 21:14:56 +02:00
Armin Braun cbf48e0633
Flatten Get Snapshots Response (#74451)
This PR returns the get snapshots API to the 7.x format (and transport client behavior) and enhances it for requests that ask for multiple repositories.
The changes for requests that target multiple repositories are:
* Add `repository` field to `SnapshotInfo` and REST response
* Add `failures` map alongside `snapshots` list instead of returning just an exception response as done for single repo requests
* Pagination now works across repositories instead of being per repository for multi-repository requests

closes #69108
closes #43462
2021-06-24 16:58:33 +02:00
Armin Braun c1e9590a69
Pagination and Sorting for Get Snapshots API (#73952)
Pagination and snapshots for get snapshots API, build on top of the current implementation to enable work that needs this API for testing. A follow-up will leverage the changes to make things more efficient via pagination.

Relates https://github.com/elastic/elasticsearch/pull/73570 which does part of the under-the-hood changes required to efficiently implement this API on the repository layer.
2021-06-17 09:00:11 +02:00
Armin Braun e8662e3cc0
Fix Snapshot Docs Listing Query Params in Body Incorrectly (#74196)
Both of these APIs don't parse request bodies, the parameters are all taken
from the query string. Also, included the master timeout param include
as it was missing here also.
2021-06-16 20:07:25 +02:00
James Rodewig 51ec860b48
[DOCS] Clarify criteria for restore completion (#74094)
A restore operation is complete when all attempts to recover primary shards have finished, even if unsuccessful.

Closes #70854
2021-06-15 08:30:51 -04:00
David Turner 9146606e49
Note repo must not be registered during restore (#73740)
This commit adds a short note to the docs on repository backups
indicating that the repository must not be modified while registered, so
that a restore from a repository backup must complete before
registration.

Relates #73730
2021-06-07 10:14:13 +01:00
James Rodewig 68bb763343
[DOCS] Note `include_aliases` supports data stream aliases (#73687)
With #73595, data stream aliases now support the restore snapshot API's
`include_aliases` option.
2021-06-03 08:43:14 -04:00
James Rodewig 2fca666b47
[DOCS] Update snapshot/restore for data stream aliases (#73438)
In 7.14+, you can now create data stream aliases. This updates the related
snapshot/restore documentation.
2021-05-27 10:57:06 -04:00
James Rodewig 81de369635
[DOCS] Fix query parameters for restore API (#73015) 2021-05-20 08:38:37 -04:00
David Turner 8170376bf0
Restore global state does not merge (#73226)
Today the docs indicate that restoring a snapshot with
`include_global_state` set will merge the ingest pipelines, ILM
policies, settings etc in the snapshot with those already in the
cluster. This isn't the case, we simply replace all the things. This
commit corrects the docs.
2021-05-19 16:16:41 +01:00
Dan Hermann 3c08d821e1
[DOCS] Data stream snapshot behavior with excluded backing indices (#72761) 2021-05-06 07:35:08 -05:00
James Rodewig 15e42fd748 [DOCS] Fix `<repository>` param 2021-05-05 15:56:28 -04:00
David Turner 1c4791e398
Abort writes in repo analyzer (#72077)
We rely on the repository implementation correctly handling the case where a
write is aborted before it completes. This is not guaranteed for third-party
repositories.

This commit adds a rare action during analysis which aborts the write
just before it completes and verifies that the target blob is not found
by any node.
2021-04-27 14:13:22 +01:00
Pierre Grimaud 3c44dfec60
[DOCS] Fix typos (#72227) 2021-04-26 12:40:38 -04:00
Yannick Welsch 8c1938a524
Remove experimental status from frozen tier docs (#71923)
Removes the experimental status for the frozen tier / shared_cache searchable snapshots for the 7.13 release. 
Also adapts docs that URL repositories are now supported in 7.13 for searchable snapshots.
2021-04-20 15:37:56 +02:00
David Turner c8fb9aad40
Track index details in SnapshotInfo (#71754)
This commit adds some per-index statistics to the `SnapshotInfo` blob:

- number of shards
- total size in bytes
- maximum number of segments per shard

It also exposes these statistics in the get snapshot API.
2021-04-19 14:57:32 +01:00
James Rodewig 55f3db74a6
[DOCS] Fix 'Register a snapshot repo' title (#71224) 2021-04-02 09:26:20 -04:00
James Rodewig f41320616c
[DOCS] Refactor data stream setup tutorial (#71074) 2021-03-31 17:28:55 -04:00
James Rodewig 693807a6d3
[DOCS] Fix double spaces (#71082) 2021-03-31 09:57:47 -04:00
Martijn van Groningen 9089e45173
Update docs that closing a data stream's write index is allowed (#71039)
This was forgotten as part of #70908
2021-03-30 11:49:09 +02:00
James Rodewig 7bddd7db5c
[DOCS] Fix collapsible properties role (#70756) 2021-03-23 11:14:46 -04:00
James Rodewig 5c75d004fa
[DOCS] Replace `put` with `create or update` in API names (#70330)
Co-authored-by: debadair <debadair@elastic.co>
Co-authored-by: Lisa Cawley <lcawley@elastic.co>
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
2021-03-15 14:49:44 -04:00
James Rodewig d51a04cd8c
[DOCS] Add operator privileges to APIs and settings (#69903) 2021-03-15 09:20:09 -04:00
David Turner 3aa8ded328
Snapshot process details docs improvements (#70012)
- adds a bit more overview on the process, including noting that it
works in terms of files

- notes that the snapshot is a point-in-time view of each shard, and not
necessarily exactly at the start of the snapshot process

- documents the `snapshot.max_concurrent_operations` setting

Co-authored-by: James Rodewig <40268737+jrodewig@users.noreply.github.com>
2021-03-08 14:42:06 +00:00
Francisco Fernández Castaño ae5308c638
Add support for range reads and retries to URL repositories (#69521) 2021-03-08 13:14:12 +01:00
Gordon Brown ce8a0c0cea
Change Get Snapshottable Features endpoint to `_features` (#69755)
The endpoint `_snapshottable_features` is long and implies incorrect
things about this API - it is used not just for snapshots, but also for
the upcoming reset API. Following discussions on the team, this commit
changes the endpoint to `_features` and removes the connection between
this API and snapshots, as snapshots are not the only use for the output
of this API.
2021-03-02 11:30:02 -07:00
David Turner d3e0a571eb
URL repos and searchable snapshots don't mix (#69197)
Provides docs and a better error message regarding using URL
repositories with searchable snapshots.

Relates #68918
2021-02-18 17:50:50 +00:00
James Rodewig 9b88ae92e6
[DOCS] Fix typos for duplicate words (#69125) 2021-02-17 10:34:20 -05:00
David Turner 0ca9f6a4f0
Adjust repository analyzer docs (#69107)
Fixes a handful of typos/wording issues, and adds the new human-readable
fields to the response format docs.

Relates #67247
2021-02-17 13:50:42 +00:00
David Turner 92d13a3f7d
Introduce repository test kit/analyser (#67247)
Today we rely on blob stores behaving in a certain way so that they can be used
as a snapshot repository. There are an increasing number of third-party blob
stores that claim to be S3-compatible, but which may not offer a suitably
correct or performant implementation of the S3 API. We rely on somesubtle
semantics with concurrent readers and writers, but some blob stores may not
implement it correctly. Hitting a corner case in the implementation may be rare
in normal use, and may be hard to reproduce or to distinguish from an
Elasticsearch bug.

This commit introduces a new `POST /_snapshot/.../_analyse` API which exercises
the more problematic corners of the repository implementation looking for
correctness bugs and measures the details of the performance of the repository
under concurrent load.
2021-02-16 14:24:40 +00:00
Gordon Brown 3f6472de74
Introduce "Feature States" for managing snapshots of system indices (#63513)
This PR expands the meaning of `include_global_state` for snapshots to include system indices. If `include_global_state` is `true` on creation, system indices will be included in the snapshot regardless of the contents of the `indices` field. If `include_global_state` is `true` on restoration, system indices will be restored (if included in the snapshot), regardless of the contents of the `indices` field. Index renaming is not applied to system indices, as system indices rely on their names matching certain patterns. If restored system indices are already present, they are automatically deleted prior to restoration from the snapshot to avoid conflicts.

This behavior can be overridden to an extent by including a new field in the snapshot creation or restoration call, `feature_states`, which contains an array of strings indicating the "feature" for which system indices should be snapshotted or restored. For example, this call will only restore the `watcher` and `security` system indices (in addition to `index_1`):

```
POST /_snapshot/my_repository/snapshot_2/_restore
{
  "indices": "index_1",
  "include_global_state": true,
  "feature_states": ["watcher", "security"]
}
```

If `feature_states` is present, the system indices associated with those features will be snapshotted or restored regardless of the value of `include_global_state`. All system indices can be omitted by providing a special value of `none` (`"feature_states": ["none"]`), or included by omitting the field or explicitly providing an empty array (`"feature_states": []`), similar to the `indices` field.

The list of currently available features can be retrieved via a new "Get Snapshottable Features" API:
```
GET /_snapshottable_features
```

which returns a response of the form:
```
{
    "features": [
        {
            "name": "tasks",
            "description": "Manages task results"
        },
        {
            "name": "kibana",
            "description": "Manages Kibana configuration and reports"
        }
    ]
}
```

Features currently map one-to-one with `SystemIndexPlugin`s, but this should be considered an implementation detail. The Get Snapshottable Features API and snapshot creation rely upon all relevant plugins being installed on the master node.

Further, the list of feature states included in a given snapshot is exposed by the Get Snapshot API, which now includes a new field, `feature_states`, which contains a list of the feature states and their associated system indices which are included in the snapshot. All system indices in feature states are also included in the `indices` array for backwards compatibility, although explicitly requesting system indices included in a feature state is deprecated. For example, an excerpt from the Get Snapshot API showing `feature_states`:
```
"feature_states": [
    {
        "feature_name": "tasks",
        "indices": [
            ".tasks"
        ]
    }
],
"indices": [
    ".tasks",
    "test1",
    "test2"
]
```

Co-authored-by: William Brafford <william.brafford@elastic.co>
2021-02-11 11:55:14 -07:00
David Turner 71763ca37f
Add docs on repository reliability and backups (#68740)
This commit spells out how important repository reliability is to
searchable snapshots, and also documents a procedure for taking a backup
of a snapshot repository.

Relates #54944
2021-02-09 14:57:06 +00:00
James Rodewig b49576fabf
[DOCS] Add security privileges to snapshot/restore API docs (#67955) 2021-02-02 09:34:35 -05:00
David Turner e5a15d4fcb
Introduce repository UUIDs (#67829)
Today a snapshot repository does not have a well-defined identity. It
can be reregistered with a different cluster under a different name, and
can even be registered with multiple clusters in readonly mode.

This presents problems for cases where we need to refer to a specific
snapshot in a globally-unique fashion. Today we rely on the repository
being registered under the same name on every cluster, but this is not a
safe assumption.

This commit adds a UUID that can be used to uniquely identify a
repository. The UUID is stored in the top-level index blob, represented
by `RepositoryData`, and is also usually copied into the
`RepositoryMetadata` that represents the repository in the cluster
state. The repository UUID is exposed in the get-repositories API; other
more meaningful consumers will be added in due course.
2021-01-25 12:17:52 +00:00
David Turner e8cba29d01
Further emphasise filesystem backups don't work (#67634)
In #33102 we added a warning against using filesystem backups.
Experience has shown that the wording we added was insufficiently
general and open to misinterpretation. This commit reworks it to be
clearer.

This commit also clarifies that snapshots are not incremental across
repositories.
2021-01-19 17:55:11 +00:00
Dan Hermann 83a5256dc2
Include date in data stream backing index names (#65205) 2020-12-14 16:46:54 -06:00
bellengao 80df5fe2f3
[DOCS] Correct restore snapshot API request example (#65525) 2020-11-30 13:55:37 -05:00
David Turner 60442b8ec4
Clarify snapshot incrementality (#65587)
Today we describe snapshots as "incremental" but their incrementality is
rather different beast from e.g. incremental filesystem backups. With
traditional backups you take a large and relatively infrequent "full"
backup and then a sequence of smaller "incremental" ones, and this whole
sequence of backups is required for a restore so it must be kept around
until at least the next full backup. In contrast, Elasticsearch
snapshots are logically independent and each can be deleted without
affecting the integrity of the others.

This distinction frequently causes confusion amongst newer users, so
this commit clarifies what we mean by "incremental" in the docs.
2020-11-30 14:49:41 +00:00