It's often useful to quote these docs to users encountering problems
with their not-quite-S3-compatible storage system. In practice we don't
need to quote the bits in the middle but we do need the last sentence
about working with the supplier to address incompatibilities. This
commit reorders things so that the most commonly quoted sentences form a
standalone paragraph.
Today the note in the docs about S3-compatible repositories notes that
the repo must behave correctly, but it's also important that it has the
same performance profile. This commit extends the docs to include this
info.
Elastic Cloud Enterprise (ECE) shares snapshot repositories across multiple deployments. As a result, the `base_path` is generated by ECE, and the `base_path` setting is not allowed. This PR adds a note to the S3, Azure, and GCS snapshot repository docs.
Today we note that the `repository-s3` plugin uses the JVM-wide
truststore in the docs for the `protocol` client setting, but it turns
out that this is easy to overlook since most installations will not need
to change the `protocol`. This commit adds the same detail to the
section on S3-compatible repositories where it is more likely to be
found.
There have been many requests to support repository-s3 authentication via IAM roles in Kubernetes service accounts.
The AWS SDK is supposed to support them out of the box with the aws-java-sdk-sts library. Unfortunately, we can't use WebIdentityTokenCredentialsProvider from the SDK. It reads the token from AWS_WEB_IDENTITY_TOKEN_FILE environment variable which is usually mounted to /var/run/secrets/eks.amazonaws.com/serviceaccount/token and the S3 repository doesn't have the read permission to read it. We don't want to hard-code a file permission for the repository, because the location of AWS_WEB_IDENTITY_TOKEN_FILE can change at any time in the future and we would also generally prefer to restrict the ability of plugins to access things outside of their config directory.
To overcome this limitation, this change adds a custom WebIdentityCredentials provider that reads the service account from a symlink to AWS_WEB_IDENTITY_TOKEN_FILE created in the repository's config directory. We expect the end user to create the symlink to indicate that they want to use service accounts for authentification.
Service accounts are checked and exchanged for session tokens by the AWS STS. To test the authentification flow, this change adds a test fixture which mocks the assume-role-with-web-identity call to the service and returns a response with test credentials.
Fixes#52625
Closes#81652.
Convert the `repository-azure`, `repository-gcs` and `repository-s3`
plugins into modules, so that they are always included in the
Elasticsearch distribution. Also change plugin installation, removal
and syncing so that attempting to add or remove these plugins still
succeeds but is now a no-op.
David Turner
Tanguy Leroux
debadair
Francisco Fernández Castaño
Dan Roscigno
James Rodewig
Artem Prigoda
Rory Hunter