It's often useful to quote these docs to users encountering problems
with their not-quite-S3-compatible storage system. In practice we don't
need to quote the bits in the middle but we do need the last sentence
about working with the supplier to address incompatibilities. This
commit reorders things so that the most commonly quoted sentences form a
standalone paragraph.
👋 howdy, team! Expanding reference to [internal](https://github.com/elastic/cloud/pull/118105) update, we've just confirmed ILM requires the repository name to be the same among migrating clusters. This is a hard block for Searchable Snapshots which requires un-Searchable-Snapshotting or redoing migration to resolve.
* [DOC+] snapshot-restore single index example
👋🏼 howdy, team! I'd like to append an example to snapshot-restore a single index. Support usually points users to [this page](https://www.elastic.co/guide/en/elasticsearch/reference/master/restore-snapshot-api.html) but then users attempt the `rename_pattern` example (which makes sense!). I'd like to point them to a more literal "close index > restore on that index" example in the future.
* Fix test failure and reword
---------
Co-authored-by: Abdon Pijpelink <abdon.pijpelink@elastic.co>
👋🏼 Regardless of if we decide to validation enforce #78276, may we please drop a doc note that users should avoid duplicating repositories (particularly bucket / base paths).
To make it clear that repository snapshots should be available and reliable for any mounted searchable snapshots.
Co-authored-by: David Turner <david.turner@elastic.co>
If the recovery node bandwidth settings exist,
then the default value for max snapshot speed will
be infinite, and the speed will be rate limited
by the recovery rate limit as well.
Fixes#57023
The get snapshot status API will currently return a value of `STARTED` for the state of a snapshot that is currently running. The documentation says that the `state` value for a running snapshot is `IN_PROGRESS`. This documentation change will align the docs with the actual result of the get snapshot status API.
Co-authored-by: Austin Smith <76973609+asmith-elastic@users.noreply.github.com>
1. Adds a note that you can restore older snapshots (to recover from a
failed upgrade) even after newer snapshots were taken.
2. Copies the note about incompatible S3 repo implementations to the top
level to avoid misunderstandings.
Adds a parameter `index_names` to the get snapshots API so that users may exclude the potentially very long index name lists when listing out snapshots.
closes#82937
Today the docs say that new major versions need new snapshot
repositories or else corruption may occur. This isn't true, we support
using the same repository across upgrades.
This commit adds back some notes which were lost when we consolidated
the snapshot/restore documentation into a single location.
The notes in question are that:
1) If a Snapshot repository contains Security's feature state, then
that repository contains security-sensitive information. This may
be obvious to some, but is good to state explicitly.
2) Some files, such as the keystore and TLS keys, are not included in
snapshots and are important to back up via other means.
Today the note in the docs about S3-compatible repositories notes that
the repo must behave correctly, but it's also important that it has the
same performance profile. This commit extends the docs to include this
info.
This commit corrects the snapshot creation and restoration docs to
describe the usage of `"none"` to restore no feature states. Previously,
they incorrectly stated that using an empty array would accomplish this,
but specifying an empty array results in the default behavior (rather
than preventing feature state snapshot/restoration).
Elastic Cloud Enterprise (ECE) shares snapshot repositories across multiple deployments. As a result, the `base_path` is generated by ECE, and the `base_path` setting is not allowed. This PR adds a note to the S3, Azure, and GCS snapshot repository docs.
* Documents `GET _snapshot/_status` and `GET _snapshot/<repository>/_status`.
* Notes the `<repository>` and `<snasphot>` parameters are optional.
* Removes erroneous mention of the `<snapshot>` parameter supporting the `_current` value.
Closes#81600
Relates to #80931
With https://github.com/elastic/elasticsearch/pull/81870, the Azure, GCS, and S3 repository types have separate, dedicated pages in the Elasticsearch guide. For consistency, this PR creates separate pages for the shared file system, read-only URL, and source-only repository types.
Related changes:
- Adds redirects to the plugins docs
- Fixes a few breaking changes that refer to the Azure, GCS, and S3 repositories as plugins.
Co-authored-by: Adam Locke <adam.locke@elastic.co>
Today we note that the `repository-s3` plugin uses the JVM-wide
truststore in the docs for the `protocol` client setting, but it turns
out that this is easy to overlook since most installations will not need
to change the `protocol`. This commit adds the same detail to the
section on S3-compatible repositories where it is more likely to be
found.
* Add support for HTTP Proxies for the GCS repository
The change adds 3 new client properties for the GCS repository:
* gcs.client.default.proxy.type
* gcs.client.default.proxy.host
* gcs.client.default.proxy.port
They allow to configure a [java.net.Proxy](https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/net/Proxy.html)
for the GCS SDK to use when communicating with the GCS API.
Resolves#82444
There have been many requests to support repository-s3 authentication via IAM roles in Kubernetes service accounts.
The AWS SDK is supposed to support them out of the box with the aws-java-sdk-sts library. Unfortunately, we can't use WebIdentityTokenCredentialsProvider from the SDK. It reads the token from AWS_WEB_IDENTITY_TOKEN_FILE environment variable which is usually mounted to /var/run/secrets/eks.amazonaws.com/serviceaccount/token and the S3 repository doesn't have the read permission to read it. We don't want to hard-code a file permission for the repository, because the location of AWS_WEB_IDENTITY_TOKEN_FILE can change at any time in the future and we would also generally prefer to restrict the ability of plugins to access things outside of their config directory.
To overcome this limitation, this change adds a custom WebIdentityCredentials provider that reads the service account from a symlink to AWS_WEB_IDENTITY_TOKEN_FILE created in the repository's config directory. We expect the end user to create the symlink to indicate that they want to use service accounts for authentification.
Service accounts are checked and exchanged for session tokens by the AWS STS. To test the authentification flow, this change adds a test fixture which mocks the assume-role-with-web-identity call to the service and returns a response with test credentials.
Fixes#52625
David Turner
Tanguy Leroux
Stef Nestor
Peter Dyson
David Kilfoyle
debadair
Daniel Mitterdorfer
Francisco Fernández Castaño
Stef Nestor
Iraklis Psaroudakis
amyjtechwriter
Andrei Dan
Anthony McGlone
Joe Gallo
Pooya Salehi
Seth Michael Larson
Yannick Welsch
Armin Braun
Gordon Brown
James Rodewig
Tobias Stadler
Dan Roscigno
Artem Prigoda
Ievgen Degtiarenko