This commit adds a new test framework for configuring and orchestrating
test clusters for both Java and YAML REST testing. This will eventually
replace the existing "test-clusters" Gradle plugin and the build-time
cluster orchestration.
* Add support for building against defined build-ids of dependent artifacts
* Introduce DraResolvePlugin and functional tests
* Declare initial dra snapshot ci job for elasticsearch
We run the DRA builds against ubuntu as other release related ci jobs have in the past. Also we only have libs2xmlutills available on our ubuntu based ci workers
As part of #50277, we removed the `TAKE_FILE_OWNERSHIP` option from the
Docker entrypoint script and the associated chroot calls, and instead
just defaulted to running the image as `elasticsearch` instead of
`root`.
However, we didn't check that it was still possible to pass CLI options
to Elasticsearch via CLI arguments, and broke this by mistake. This is
probably an uncommon pattern, versus environment variables or a config
file. Nevertheless, it is supposed to be possible and is mentioned in
the documentation.
Fix the problem by suppling the missing positional params when calling
Elasticsearch, and add a test case so that we don't break it again.
As part of #50277, we removed the `TAKE_FILE_OWNERSHIP` option from the
Docker entrypoint script and the associated chroot calls, and instead
just defaulted to running the image as `elasticsearch` instead of
`root`.
However, we didn't check that it was still possible to pass CLI options
to Elasticsearch via CLI arguments, and broke this by mistake. This is
probably an uncommon pattern, versus environment variables or a config
file. Nevertheless, it is supposed to be possible and is mentioned in
the documentation.
Fix the problem by suppling the missing positional params when calling
Elasticsearch, and add a test case so that we don't break it again.
This changes the LoggedExec task to be configuration cache compatible. We changed the implementation
to use `ExecOperations` instead of extending `Exec` task. As double checked with the Gradle team this task
is not planned to be made configuration cache compatible out of the box anytime soon.
This is part of the effort on https://github.com/elastic/elasticsearch/issues/57918
Deprecation log indexing can make test very fragile
and randomly failing during cluster cleanup.
This commit disables indexing of deprecation logs for :distribution:docker tests
closes#79803
Fixes#81999.
Change `EnrollmentProcessTests` and the testing infrastructure so
that `ServerUtils` can send requests to the correct local port.
This is also a step towards addressing #79688.
Also change the Iron Bank build to that it works on aarch64. This
isn't currently a supported configuration, but it's useful for
testing.
Closes#81652.
Convert the `repository-azure`, `repository-gcs` and `repository-s3`
plugins into modules, so that they are always included in the
Elasticsearch distribution. Also change plugin installation, removal
and syncing so that attempting to add or remove these plugins still
succeeds but is now a no-op.
For some reason, the way we generate `/app/elasticsearch.sh` for Cloud
images has broken. It may be the result of changing the default shell to
`bash`. Fix by adding `-e` to the `echo` command.
As a result of changing the base Docker to Ubuntu in #80820, the default shell
i.e. `/bin/sh` changed to `dash`, rather than `bash`, which could impact anyone
invoking `/bin/sh` and expecting it to still propagate environment variables with
periods in their names.
Reconfigure the default shell back to `bash` so that this type of situation works
again.
Apply suggestions from Docker Inc about how to update the `cacerts` in
our Ubuntu-based Docker image. Instead of copying around files and
symlinking, instead install `ca-certificates` and `p11-kit`, and use the
latter to regenerate Java's `cacerts`, as well as ensuring it is
regenerated if the system ca certs are updated.
Allow users to pass an ENROLLMENT_TOKEN environment variable that
would be passed as the `--enrollment-token` parameter to the
elasticsearch executable running in the container. This allows an
elasticsearch node running in docker to enroll itself to an
existing secured cluster.
Resolves: #81068
Closes#81208. Elasticsearch uses zlib for two purposes: *
Compression of stored fields with `index.codec: best_compression`,
which we use for observability and security data. * Request /
response compression. Historically, zlib was packaged within the JDK, so
that users wouldn't have to have zlib installed for basic usage of Java.
However, the original zlib optimizes for portability and misses a number
of important optimizations such as leveraging vectorization support for
x86 and ARM architectures. Several forks have been created in order to
address this. Since version 9, the JDK uses the system's zlib when
available and falls back to the zlib that is packaged within the JDK if
a system zlib cannot be found. This commit changes the Docker image to
install the Cloudflare fork of zlib, and run Java using the fork instead
of the original zlib, so that users of the Docker image can get better
performance. Other ES distribution types are out-of-scope, since
configuring the JVM to use an alternative zlib requires an environment
config as well as installed another zlib, and Docker is the only
distribution type where we can control both.
Part of #81150. Sometimes, building the Cloud Docker images fails due to
some issue with the `.tar.gz` files. Add extra checks and error messages
so that we have a better idea what the precise problem is.
Closes#70219.
Introduce a declarative way for the Elasticsearch server to manage plugins,
which reads the `elasticsearch-plugins.yml` file and works which out
plugins need to be added and / or removed to match the configuration. Also
make it possible to configure a proxy in the config file, instead of
through the environment.
Most of the work of adding and removing is still done in the
`InstallPluginAction` and `RemovePluginAction` classes, so the
behaviour should be the same as with the `install` and `remove`
commands. However, these commands will now abort if the above config
file exists. The intent is to make it harder for the configuration
to drift.
This new method only applies to `docker` distribution types at the
moment.
Since this syncing mechanism declarative, rather than imperative,
the Cloud-specific plugin wrapper script is no longer required.
Instead, an environment variable informs `InstallPluginAction` to
install plugins from an archive directory instead of downloading
them, where possible.
Closes#76681. Our approach to using `scratch` for building Docker
images has caused problems at Docker Hub. Fix this situation by
removing the whole process of using scratch and instead bases the
default distribution on `almalinux:8.4-minimal`. Alma Linux is
binary-compatible with RHEL, and therefore very similar to UBI.
Firstly: we tag our Docker images with various pieces of information,
including a timestamp for when the image was built. However, this makes
it impossible completely cache the image. When developing the Docker
images, it's very tedious to completely rebuild an image for every
single change. Therefore, provided we're not building a proper release
build, we fix the build time to midnight so that the Docker build cache
is usable.
Secondly: the `DockerBuildTask` outputs a marker file to indicate that
an image has been built, but that isn't enough for a meaningful
up-to-date check by Gradle. Improve this by fetching the newly-built
image's hash, and writing that to the output file.
Thirdly: improve the Docker tests to make them more ergonomic, and also
disable `ingest.geoip.downloader.enabled` by default.
Fourthly: add missing test coverage for sourcing settings from env vars.
Closes#77414.
The generated shell logic in ShellRetry doesn't work in more basic shells, so instead
of using the {1..10} construct, expand the sequence.
* Only depend on beats when necessary
The unified build happens in stages, and for release builds the main
part of Elasticsearch will be built before the Beats distributions are
available. To work around this, infer in the Docker part of the build
whether to include beats as a dependency. In the future, we should find
a more robust mechanism.
* Move decision about including beats
Our packaging tests set up dependencies so that the tests depend on the
artifact that is going to be tested. For Docker, the tests depend on an
export of the Docker image, partly to follow the pattern, but partly to
support loading the image when the tests are executed inside Vagrant.
There is a sub-project for each type of image that can be exported. The
main Docker build file adds an export task and an artifact to the
sub-project. However, it also makes the export task a dependency of the
sub-project's `assemble` task. This last step isn't necessary for the
tests to run, and causes extra and needless work to be done when running
`assemble` on the entire build. Therefore, removing the last bit of task
wiring.
Rene Groeschke
Michael Montgomery
Mark Vieira
Dimitrios Liappis
Rory Hunter
Rory Hunter
Przemyslaw Gomulka
Ioannis Kakavas