Commit Graph

59 Commits

Author SHA1 Message Date
James Rodewig b49576fabf
[DOCS] Add security privileges to snapshot/restore API docs (#67955) 2021-02-02 09:34:35 -05:00
David Turner e5a15d4fcb
Introduce repository UUIDs (#67829)
Today a snapshot repository does not have a well-defined identity. It
can be reregistered with a different cluster under a different name, and
can even be registered with multiple clusters in readonly mode.

This presents problems for cases where we need to refer to a specific
snapshot in a globally-unique fashion. Today we rely on the repository
being registered under the same name on every cluster, but this is not a
safe assumption.

This commit adds a UUID that can be used to uniquely identify a
repository. The UUID is stored in the top-level index blob, represented
by `RepositoryData`, and is also usually copied into the
`RepositoryMetadata` that represents the repository in the cluster
state. The repository UUID is exposed in the get-repositories API; other
more meaningful consumers will be added in due course.
2021-01-25 12:17:52 +00:00
David Turner e8cba29d01
Further emphasise filesystem backups don't work (#67634)
In #33102 we added a warning against using filesystem backups.
Experience has shown that the wording we added was insufficiently
general and open to misinterpretation. This commit reworks it to be
clearer.

This commit also clarifies that snapshots are not incremental across
repositories.
2021-01-19 17:55:11 +00:00
Dan Hermann 83a5256dc2
Include date in data stream backing index names (#65205) 2020-12-14 16:46:54 -06:00
bellengao 80df5fe2f3
[DOCS] Correct restore snapshot API request example (#65525) 2020-11-30 13:55:37 -05:00
David Turner 60442b8ec4
Clarify snapshot incrementality (#65587)
Today we describe snapshots as "incremental" but their incrementality is
rather different beast from e.g. incremental filesystem backups. With
traditional backups you take a large and relatively infrequent "full"
backup and then a sequence of smaller "incremental" ones, and this whole
sequence of backups is required for a restore so it must be kept around
until at least the next full backup. In contrast, Elasticsearch
snapshots are logically independent and each can be deleted without
affecting the integrity of the others.

This distinction frequently causes confusion amongst newer users, so
this commit clarifies what we mean by "incremental" in the docs.
2020-11-30 14:49:41 +00:00
James Rodewig 0805ff17a5
[DOCS] Fix redirects and anchors (#65472) (#65474) 2020-11-24 19:59:52 -05:00
Rafi Estrada 5e4db2291f
[DOCS] Add note about NFS repositories (#65149)
Co-authored-by: James Rodewig <40268737+jrodewig@users.noreply.github.com>
2020-11-17 14:21:18 -05:00
James Rodewig b31a8ff244
[DOCS] Fix put repository API docs (#64811) 2020-11-09 14:20:08 -05:00
James Rodewig 357ab0073e
[DOCS] Improve docs for Windows DOS/UNC paths in `path.*` settings (#64668) 2020-11-09 10:36:48 -05:00
Armin Braun 6924236118
Limit the Number of Snapshots in a BlobStoreRepository (#64461)
Adds a limit to the maximum number of snapshots that are allowed
to be added to a snapshot repository as a safety measure of last resort
against repositories that grow to an unmanagable size due to e.g. incorrect SLM
settings.

Co-authored-by: David Turner <david.turner@elastic.co>
2020-11-03 18:02:11 +01:00
James Rodewig 1ea83359bb
[DOCS] Fix case for 'Boolean' (#64299) 2020-10-29 09:04:43 -04:00
debadair b95d9c4aa4
[DOCS] Add searchable snapshots topic. (#63040)
* [DOCS] Add searchable snapshots topic.

* [DOCS] Add definitions & remove fully-remote storage.

* [DOCS] Fixed duplicate anchor.

* Expand conceptual docs for searchable snapshots

* Rewordings

* Glossary tidy-up

* Beta

* Reword

* More performance idea to a TIP

* use -> manage

* red -> not green

* Missing space?

* Update docs/reference/glossary.asciidoc

* Fix beta label

* Use more attributes, fix link titles

* Apply suggestions from code review

Co-authored-by: debadair <debadair@elastic.co>

* Reformat

* Minor rewordings

* More minor rewordings

* Address Henning's comments

Co-authored-by: David Turner <david.turner@elastic.co>
Co-authored-by: James Rodewig <40268737+jrodewig@users.noreply.github.com>
2020-10-22 14:34:32 -07:00
Adam Locke 954d6eeeba
Move clone snapshot API page. (#63902) 2020-10-20 08:45:38 -04:00
Adam Locke 789ee2d73e
[DOCS] Combining important config settings into a single page (#63849)
* Combining important config settings into a single page.

* Updating ids for two pages causing link errors and implementing redirects.
2020-10-19 10:02:22 -04:00
Armin Braun f7f239d39a
Clone Snapshot API (#61839)
Adds clone snapshot API to clone part of a snapshot into a new snapshot.
2020-10-02 21:28:44 +02:00
Armin Braun 1bf5550498
Remove Outdated Snapshot Docs (#61684)
Removing some now outdated statements that refer to a time
when snapshot operations could not run concurrently.

Closes #61680

Co-authored-by: Adam Locke <adam.locke@elastic.co>
2020-08-31 15:52:49 +02:00
Adam Locke 186feaa437
Adding ignore_unavailable param. (#61368) 2020-08-20 09:39:25 -04:00
James Rodewig 4131f9ad01
[DOCS] Fix typo in take snapshot docs (#60204) (#60382)
Co-authored-by: VLADIMIR MIRONOV <mironov.v@torrowtech.com>
2020-07-29 09:15:51 -04:00
Adam Locke 714c8861dc
[DOCS] Adds table with icons for version compatibility (#60159)
* Adds table with icons for simplicity.

* Updating table for clarity.

* Changing table formatting and incorporating more feedback.

* Changing table alignment.
2020-07-28 10:27:55 -04:00
James Rodewig 2774cd6938
[DOCS] Swap `[float]` for `[discrete]` (#60124)
Changes instances of `[float]` in our docs for `[discrete]`.

Asciidoctor prefers the `[discrete]` tag for floating headings:
https://asciidoctor.org/docs/asciidoc-asciidoctor-diffs/#blocks
2020-07-23 11:48:22 -04:00
Adam Locke 396dcffe39
[DOCS] Adding new page for restore snapshot API (#59937)
* Adding new page for restore snapshot API.

* Improving test cases, lots of edits, and streamlining content.

* Incorporating review suggestions and feedback.

* Specify `index alias` vs `alias`

* Change parameter order

* Provide clarity around regular expression

* Add link to SLM parameters

* Split sentences in example

* Adding link to master node page.
2020-07-22 11:42:17 -04:00
James Rodewig 80b674fb25
[DOCS] Reformat snippets to use two-space indents (#59973) 2020-07-21 12:24:26 -04:00
Armin Braun 9b155d9201
Fix Snapshot Status API Docs Test (#59902)
The clock resolution for this API is our default 200ms. It is unlikely but
possible that a shard snapshot starts and ends on separate clock ticks and that breaks the test.
Just allowing any value here seems fine to me (seems we can't match for integer specifically).
2020-07-20 18:26:05 +02:00
James Rodewig 6a02528e91
[DOCS] Fix erroneous data stream ref (#59805)
Removes an erroneous data stream reference added in #58513.

While technically possible, we don't encourage using date math to name
data streams.
2020-07-17 13:43:43 -04:00
Adam Locke c143bb56cb
[DOCS] Updating snapshot/restore pages to align with API changes (#59730)
* Updating snapshot/restore pages to align with API changes.

* Fixing texts in delete snapshot page.

* Removing duplicate code sample and making editorial changes.

* Change "deleted" to "delete"

* Incorporating review feedback and making minor editorial changes.

* Remove titleabbrev

* Add paragraph break

* Remove titleabbrev from restore page

* Remove titleabbrev from create page

* Change "Create" to lowercase

* Change API names to lowercase

* Remove extraneous delimiters

* Change "Delete" to lowercase

* Single-sourcing warning and clarifying warning text.
2020-07-17 13:08:13 -04:00
Armin Braun 3cef12368c
Fix Snapshot Status API Docs Test (#59775)
We can't just assume a fixed number for the overall file count.
Depending on how the merging/flushing works out we won't always have
4 files for the index across all versions, systems etc.
Also, we could have x-pack concurrently create some system indices
which could mess up the total numbers here.
Fixed by only snapshotting a single index+shard in the snapshot that
we get the status for and verifying consistency instead of equality
for total file counts.

Closes #59767
2020-07-17 16:34:29 +02:00
Adam Locke 556f19d55d
[DOCS] Adding get snapshot status API docs (#59355)
* Adding get snapshot status API docs.

* Adding more fields and a link to the new page.

* Adding missing spaces in TESTRESPONSES

* Adding more parameters and making some edits.

* Marking snapshot as optional

* Marking repository as optional

* Add data type for stats

* Add data type for shard_stats

* Incorporating review feedback.

* Lots of review feedback incorporated.

* Fixing tests to unbreak CI builds.

* Changing indices to index.
2020-07-15 16:28:43 -04:00
James Rodewig 786104eef7
[DOCS] Update snapshot/restore and SLM docs for data streams (#58513)
Updates the existing snapshot/restore and SLM docs to make them
aware of data streams.
2020-07-13 08:34:42 -04:00
Armin Braun f571ea7e8b
Remove Outdated Documentation On Snapshots (#59358)
* We now have concurrent repository operations so the one at a time limit does not apply any longer
* Initialization was never slow solely due to loading information about all existing snaphots (though this contributed)
but also because two cluster state updates and a few writes to the repository had to happen before initialization could return
   * Repo data necessary for a snapshot create operation is now cached on heap so loading it is effectively instant
   * Snapshot initialization is just a single CS update now
   * Initialization does no writes to the repository whatsoever
* Fixed missing `repository`
2020-07-10 19:19:02 +02:00
Adam Locke b2beb9c892
[DOCS] Adding get snapshot API docs (#59098)
* Adding page for get snapshot API.

* Adding values for state and cleaning up some other formatting.

* Adding missing forward slash to GET request.

* Updating values for start_time and end_time in TESTRESPONSE.

* Swap "return" for "retrieve"

* Swap "return" for "retrieve" 2

* Change .snapshot to .response

* Adding response parameters and incorporating edits from review.

* Update response example to include repository info

* Change dash to underscore

* Add data type for snapshot in response

* Incorporating review comments and adding missing response definitions.

* Minor rewording in description.
2020-07-08 11:28:24 -04:00
David Turner 283cd8369d
Restores do not cause red health (#59015)
Since 2.0.0 (56a264cf6d) we have documented that restoring a snapshot
typically results in `red` cluster health. However since 5.0.0 (#19516)
this hasn't been true, we report `yellow` health for unassigned
primaries that will be recovered from a snapshot in the future. This
commit adjusts these docs to match today's behaviour.
2020-07-04 11:16:02 +01:00
Adam Locke 3a1258fe97
[DOCS] Add supported ESS settings to ES docs (#57953)
* Adding ESS icons to supported ES settings.

* Adding new file for supported ESS settings.

* Adding supported ESS settings for HTTP and disk-based shard allocation.

* Adding more supported settings for ESS.

* Adding descriptions for each Cloud section, plus additional settings.

* Adding new warehouse file for Cloud, plus additional settings.

* Adding node settings for Cloud.

* Adding audit settings for Cloud.

* Resolving merge conflict.

* Adding SAML settings (part 1).

* Adding SAML realm encryption and signing settings.

* Adding SAML SSL settings.

* Adding Kerberos realm settings.

* Adding OpenID Connect Realm settings.

* Adding OpenID Connect SSL settings.

* Resolving leftover Git merge markers.

* Removing Cloud settings page and link to it.

* Add link to mapping source

* Update docs/reference/docs/reindex.asciidoc

* Incorporate edit of HTTP settings

* Remove "cloud" from tag and ID

* Remove "cloud" from tag and update description

* Remove "cloud" from tag and ID

* Change "whitelists" to "specifies"

* Remove "cloud" from end tag

* Removing cloud from IDs and tags.

* Changing link reference to fix build issue.

* Adding index management page for missing settings.

* Removing warehouse file for Cloud and moving settings elsewhere.

* Clarifying true/false usage of http.detailed_errors.enabled.

* Changing underscore to dash in link to fix ci build.
2020-07-02 14:13:06 -04:00
Adam Locke 39c1244b7b
[DOCS] Adding delete snapshot API docs. (#58865)
* [DOCS] Adding delete snapshot API docs.

* Adding TESTSETUP snippets and fixing original TEST.

* Removing extraneous TESTSETUP.

* Revising <snapshot> description.

* Removing TEST.

* Streamline delete API description.

* Improve TESTSETUP for snippets.
2020-07-01 14:34:02 -04:00
James Rodewig 483bab2281
[DOCS] Add data streams to API conventions (#58695)
Updates the existing API conventions docs to make them aware of data
streams.

Co-authored-by: debadair <debadair@elastic.co>
2020-06-30 17:06:17 -04:00
Yannick Welsch 118521d022
Account for recovery throttling when restoring snapshot (#58658)
Restoring from a snapshot (which is a particular form of recovery) does not currently take recovery throttling into account
(i.e. the `indices.recovery.max_bytes_per_sec` setting). While restores are subject to their own throttling (repository
setting `max_restore_bytes_per_sec`), this repository setting does not allow for values to be configured differently on a
per-node basis. As restores are very similar in nature to peer recoveries (streaming bytes to the node), it makes sense to
configure throttling in a single place.

The `max_restore_bytes_per_sec` setting is also changed to default to unlimited now, whereas previously it was set to
`40mb`, which is the current default of `indices.recovery.max_bytes_per_sec`). This means that no behavioral change
will be observed by clusters where the recovery and restore settings were not adapted.

Relates https://github.com/elastic/elasticsearch/issues/57023

Co-authored-by: James Rodewig <james.rodewig@elastic.co>
2020-06-30 13:08:21 +02:00
Adam Locke 23abe8ec44
[DOCS] Adding create index snapshot API docs (#58519)
* Adding create index snapshot API page.
* Condense API description.
* Remove parameter from query.
* Add POST method and remove `-name` from the snapshot variable.
* Expand description of `<snapshot>`.
* Add data streams to introduction and expand the overall description.
* Add support for data streams.
* Add support for data streams.
* Add data stream and reference for "point-in-time view".
* Add data streams.
* Change `my_backup` to `my_repository`.
* Add description of boolean options for `wait_for_completion` parameter.
* Change command --> response
* Clarify `indices` parameter description
* Update `ignore-unavailable` parameter description
* Reword example description
* Remove "index" from API name
* Incorporating review comments from James R.
* Adding a much better request + response
* Clarify `include_global_state` description
* Incorporating additional edits.
* Changing my_backup to my_repository in example.
* Update snippet test to avoid failures
* Update TESTRESPONSE snippets
* Remove errant space
* Removing the  parameter per reviewer comments
2020-06-29 14:53:30 -04:00
debadair 99c75f1e6c
[DOCS] Fixes problematic terminology (#58178)
* [DOCS] Fixes problematic terminology.

* Update docs/reference/snapshot-restore/register-repository.asciidoc

Co-authored-by: James Rodewig <james.rodewig@elastic.co>
2020-06-16 09:23:11 -07:00
Gordon Brown 34018f2b12
Clarify `expand_wildcard` option in snapshot docs (#58016)
This commit clarifies that the `expand_wildcards` option (as well as other
`IndicesOptions` parameters) can be used with the Create Snapshot API, but that
they must be in the body of the request.

Also clarifies the connection between `expand_wildcards` and hidden indices as
it relates to snapshots.
2020-06-12 09:00:43 -06:00
James Rodewig 7826bbee87
[DOCS] Move search API's `docvalue_fields` examples (#57760)
Changes:

* Condenses and relocates the `docvalue_fields` example to the 'Run a search' 
   page.
* Adds docs for the `docvalue_fields` request body parameter.
* Updates several related xrefs.

Co-authored-by: debadair <debadair@elastic.co>
2020-06-11 10:57:15 -04:00
James Rodewig 51e3d5ab63
[DOCS] Fix source filtering xrefs (#57720) 2020-06-05 08:46:26 -04:00
James Rodewig 3bb11cf269
[DOCS] Refactor admons for multi-parameter options (#57491)
Several APIs support options that can be specified as a query parameter or a
request body parameter.

Currently, this is documented using notes, which can get rather lengthy. This
replaces those multiple notes with a single note and a footnote.
2020-06-02 11:58:46 -04:00
Lisa Cawley 8b9293b3bf
[DOCS] Replace docdir attribute with es-repo-dir (#57489) 2020-06-01 15:55:05 -07:00
James Rodewig 9b337ec021
[DOCS] Add verify snapshot repository API docs (#57253) 2020-05-28 09:35:19 -04:00
Toby Sutor 80778f5a0d
[DOCS] Add note about restoring snapshot versions (#55225)
As discussed at https://elastic.slack.com/archives/C0D1XEXEZ/p1586939752242300 is it not possible to restore snapshots taken on newer versions into clusters running lower versions. For example, a snapshot created in a 7.6.0 cluster cannot be restored on a 7.5.0 cluster. This needs to be documented.
2020-05-27 11:38:48 -04:00
James Rodewig 1fde128c40 [DOCS] Fix snapshot repo API jump list 2020-05-27 10:54:22 -04:00
James Rodewig 96a8ba40ad
[DOCS] Add get snapshot repo API docs (#57150) 2020-05-27 10:52:27 -04:00
James Rodewig 626f67e651
[DOCS] Fix clarity of URL snapshot repo docs (#56920)
Makes following changes to better clarify docs for read-only URL
snapshot repositories:

* Adds an example snippet for registering a URL repository
* Rewrites the protocols paragraph
* Adds a note to explicitly point out that only URLs using the `ftp`,
  `http`, `http`, and `jar` protocols do not need the `path.repo`
  setting.

Fixes #16280
2020-05-27 10:34:26 -04:00
James Rodewig 7fca5e12bb
[DOCS] Document impact of restore API's `include_global_state` param (#57067) (#57157)
Previously, the restore API snippet included a `include_global_state` value of `true`.

Some users copy and paste the code example verbatim, updating only the index and
snapshot value names. Running the snippet could inadvertently wipe out a
cluster's current ILM policies, index templates, and ingest pipelines.

This change updates the snippet to use a `include_global_state` value of
`false`. It also adds a callout that better describes impacts of
using a `include_global_state` argument of `true`.

Co-authored-by: Mike Wong <mike.wong@elastic.co>
Co-authored-by: James Rodewig <james.rodewig@elastic.co>
Co-authored-by: David Turner <david.turner@elastic.co>
2020-05-26 11:59:50 -04:00
James Rodewig eae4a1c953
[DOCS] Add delete snapshot repo API docs (#57043)
Changes:

* Adds API reference docs for the delete snapshot repo API.

* Corrects an error in the delete snapshot repo API spec. Comma-separated
repository names are not supported.

* Relocates the existing delete snapshot repo API example docs.
2020-05-21 13:59:48 -04:00