Elasticsearch's keystore initial md5sum was added in #28928 with
the intention to allow us to remove the elasticsearch.keystore
file upon package removal, if this hadn't been altered after
installation. At that time this decision made perfect sense as
the elasticsearch keystore only contains transient data by
default ( keystore.seed ) that is meant to be useful for bootstrap
related actions, and doesn't need to survive re-installations.
With Security ON by default, we will be storing additional
settings in the keystore upon installation(namely, the passwords
for the PKCS#12 keystores used for TLS) and these have a more
persistent nature. Since `remove` doesn't delete the configuration
directories and files where said PKCS#12 keystores are stored, it
makes sense to also not delete the elasticsearch.keystore which
stores the passwords.
Our packaging tests set up dependencies so that the tests depend on the
artifact that is going to be tested. For Docker, the tests depend on an
export of the Docker image, partly to follow the pattern, but partly to
support loading the image when the tests are executed inside Vagrant.
There is a sub-project for each type of image that can be exported. The
main Docker build file adds an export task and an artifact to the
sub-project. However, it also makes the export task a dependency of the
sub-project's `assemble` task. This last step isn't necessary for the
tests to run, and causes extra and needless work to be done when running
`assemble` on the entire build. Therefore, removing the last bit of task
wiring.
Closes#74795.
Introduce two Docker image variants for Cloud. The first bundles
(actually installs) the S3, Azure and GCS repository plugins. The
second bundles all official plugins, but only installs the repository
plugins.
Both images also bundle Filebeat and Metricbeat.
The testing utils have been refactored to introduce a `docker`
sub-package. This allows the static `Docker.containerId` to be
shared without needing all the code in one big class. The code for
checking file ownership / permissions has also been refactored to
a more Hamcrest style, using a custom Docker file matcher.
* Update redline library to 1.2.10
The redline team just released version 1.2.10 of the redline
library which contains our fix of the rpm signatures / headers.
Also a PR to update that dependency in the ospackage plugin has been
raised at https://github.com/nebula-plugins/gradle-ospackage-plugin/pull/402
* Update common about enforcing redline 1.2.10
Second attempt after reverting #76087.
This PR fixes a number of issues with the Iron Bank Docker context. Most
notably it changes the generated `Dockerfile` in the build context to
`COPY` the Elasticsearch archive, instead of downloading it, since Iron
Bank images have a different process and will never be built on Docker Hub.
This requires that the context task be wired to `assemble` instead of the
build image task. The latter is only used for testing purposes.
Closes#76148. Previously, in #74327, we added support for setting
Elasticsearch settings in the Docker image via environment variables.
However there are scenarios e.g in Kubernetes where it is easy to
accidentally define environment variables that the ES startup process
rejects.
Work around this by using the more explicit env var prefix
`ES_SETTING_`.
A MultiCommand requires that a (sub)command is provided for each
execution. Previously, the error handling for MultiCommand would
simply state "ERROR: Missing command" but give no assistance about
what commands were accepted. The user was required to pass "-help" in
order to get the command list.
This commit changes the behaviour so that the list of commands is
printed after the error message.
The test cluster took slightly more than 10s to apply the cluster state for the mapping update
randomly which failed the run in #75941. This only ever failed once but since there's no real harm in doing so
increasing the timeout to 30s here like we did in other similar spots.
closes#75941
Explicitly set permissions for all files in the Elasticsearch home
directory to the minimum required set, and change ownership to
`root:root` where possible.
Part of #70219.
Split the install and remove plugin CLI commands into `Command` and
`Action` classes, so that the main implementations can be reused
in a later PR to create a `Sync` command.
This adds support for Sha256 header signature in our RPMs by
updating the dependency to the readline library to a version
we have patched until the provided PR (https://github.com/craigwblake/redline/pull/157)
got merged and released by the redline folks.
This work is related to #58257
The process for building `log4j2.properties` in the default distribution
is complicated, expensive and requires all modules and plugins to be
built in order to generate the file. Improve this by directly grabbing
the files and concatenating them.
Also remove code for building the OSS distro config.
Closes#74036. Since some orchestration platforms forbid periods in
environment variable names, allow Docker users to pass settings to ES
using an alternative name scheme. For example:
bootstrap.memory_lock
...becomes:
ES_BOOTSTRAP_MEMORY__LOCK
The setting name is uppercased, prefixed, all underscores are converted
to double underscores, and all periods are converted to underscores.
We are using the `-n` bash operator to determine whether a second
argument is passed to the `postinst` script during installation of
our DEB packages. The reasoning is that we want to differentiate
between installations and upgrades and a second argument to
postinst is only passed when this is an upgrade ( the identifier
of the version the package is upgraded to ).
The problem is that we use `-n` without quoting the $2 and this is
unsafe practice as it might yield unexpected results for `-n` and
`-z` operators used within test brackets. Testing with bash
5.0-6ubuntu1.1 `[ -n $2 ]` returns false both for upgrades and
installations, while `[ -n "$2" ]` behaves as expected.
References:
https://tldp.org/LDP/abs/html/comparison-ops.htmlhttps://wiki.debian.org/MaintainerScripts
SimpleFS is deprecated and removed in Lucene 9. This commit replaces its
internal usages with NIOFS. Two other usages (store type and SMB) need
to be deprecated before switching to niofs.
- Fix new introduced deprecated usages
- Update to newer ospackage snapshot to include provided PR for fixing deprecated usage
This gradle release comes with improvements on incremental compilation which we should benefit from
Alpine Linux 3.14.0 is incompatible with older versions of Docker, so pin the
version that we use to 3.13. At some point in the future, it will
be possible to upgrade Alpine.
Also when compiling curl, if the configure step fails and a config.log
file exists, then dump it out before exiting to assist diagnosis.
Change the formatter config to sort / order imports, and reformat the
codebase. We already had a config file for Eclipse users, so Spotless now
uses that.
The "Eclipse Code Formatter" plugin ought to be able to use this file as
well for import ordering, but in my experiments the results were poor.
Instead, use IntelliJ's `.editorconfig` support to configure import
ordering.
I've also added a config file for the formatter plugin.
Other changes:
* I've quietly enabled the `toggleOnOff` option for Spotless. It was
already possible to disable formatting for sections using the markers
for docs snippets, so enabling this option just accepts this reality
and makes it possible via `formatter:off` and `formatter:on` without
the restrictions around line length. It should still only be used as
a very last resort and with good reason.
* I've removed mention of the `paddedCell` option from the contributing
guide, since I haven't had to use that option for a very long time. I
moved the docs to the spotless config.
Recursively remove write access from the bin, jdk, lib and
modules directories, since this access is not required, and removing
it makes it harder to exploit other issues in an ES distribution.
Modularization of the JDK has been ongoing for several years. Recently
in Java 16 the JDK began enforcing module boundaries by default. While
Elasticsearch does not yet use the module system directly, there are
some side effects even for those projects not modularized (eg #73517).
Before we can even begin to think about how to modularize, we must
Prepare The Way by enforcing packages only exist in a single jar file,
since the module system does not allow packages to coexist in multiple
modules.
This commit adds a precommit check to the build which detects split
packages. The expectation is that we will add the existing split
packages to the ignore list so that any new classes will not exacerbate
the problem, and the work to cleanup these split packages can be
parallelized.
relates #73525
When libs/core was created, several classes were moved from server's
o.e.common package, but they were not moved to a new package. Split
packages need to go away long term, so that Elasticsearch can even think
about modularization. This commit moves all the classes under o.e.common
in core to o.e.core.
relates #73784
The org.elasticsearch.bootstrap package exists in server with classes
for starting up Elasticsearch. The elasticsearch-core jar has a handful
of classes that were split out from there, namely java version parsing
and jarhell. This commit moves those classes to a new
org.elasticsearch.jdk package so as to not split the server owned
bootstrap package.
relates #73784
Extract usage of internal API from TestClustersPlugin and PluginBuildPlugin and related plugins and build logic
This includes a refactoring of ElasticsearchDistribution to handle types
better in a way we can differentiate between supported Elasticsearch
Distribution types supported in TestCkustersPlugin and types only supported
in internal plugins.
It also introduces a set of internal versions of public plugins.
As part of this we also generate the plugin descriptors now.
As a follow up on this we can actually move these public used classes into
an extra project (declared as included build)
We keep LoggedExec and VersionProperties effectively public And workaround for RestTestBase
This commit adds a new signature file for snakeyaml, to avoid certain
constructors. It is not added as part of server signatures so that it
can be used in launchers where there is also a use of snakeyaml.
Related to #71593 we move all build logic that is for elasticsearch build only into
the org.elasticsearch.gradle.internal* packages
This makes it clearer if build logic is considered to be used by external projects
Ultimately we want to only expose TestCluster and PluginBuildPlugin logic
to third party plugin authors.
This is a very first step towards that direction.
This PR adds documentation for GeoIPv2 auto-update feature.
It also changes related settings names from geoip.downloader.* to ingest.geoip.downloader to have the same convention as current setting.
Relates to #68920
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
Co-authored-by: James Rodewig <40268737+jrodewig@users.noreply.github.com>
This change enables GeoIP downloader by default.
It removes feature flag but adds flag that is used by tests to disable it again (as we don't want to hammer GeoIP database service with every test cluster we spin up).
Relates to #68920
* Warn users if security is implicitly disabled
Elasticsearch has security features implicitly disabled by default for
Basic and Trial licenses, unless explicitly set in the configuration
file.
This may be good for onboarding, but it also lead to unintended insecure
clusters.
This change introduces clear warnings when security features are
implicitly disabled.
- a warning header in each REST response if security is implicitly
disabled;
- a log message during cluster boot.
This change fixes number of problems in GeoIPv2 code:
- closes streams from Files.list in GeoIpCli, which should fix tests on Windows
- makes sure that total download time in GeoIP stats is non-negative (we serialize it as vInt which can cause problems with negative numbers and it can happen when clock was changed during operation)
- fixes handling of failed/simultaneous downloads, #69951 was meant as a way to prevent 2 persistent tasks to index chunks but it would prevent any update if single download failed mid indexing, this change uses timestamp (lastUpdate) as sort of UUID. This should still prevent 2 tasks to step on each other toes (overwriting chunks) but in the end still only single task should be able to update task state (this is handled by persistent tasks framework)
Closes#71145
- Update gradle wrapper to gradle 7.0
- Remove deprecated usages to make build 7.0 compatible
- Fix excludes in docs snippet tasks (See https://github.com/gradle/gradle/issues/16160 for details)
- Fix deprecation warnings in 7.0
- Add explicit dependencies that have been missed
- Make extract native licenses tasks output dir more explicit
- Use a snapshot of the ospackage plugin that includes a fix for 7.0 already
- fix test runtime classpath setup in repository-hdfs
- Make task dependency explicit to fix further deprecation warnings
- Remove manual check for http repo usages that has been deprecated in gradle 7.0
- Update spock to latest 2.0 milestone required for groovy 3
For the Docker distribution, we transform the archive distribution's
log4j2 config so that all messages are logged to the console by default.
However this transformation step happens when the Docker image is built,
which means that the source for the transformation must be included in
the Docker context.
Improve this by making the archive distribution's log4j config available
as an artifact in the build, then simply copying it into the Docker context.
The transform logic has been reimplemented as a simple copy filter.
Consequently, the `transform-log4j-config` project has been removed.
Mark Vieira
Ioannis Kakavas
Rory Hunter
Rory Hunter
Rene Groeschke
Tim Vernum
Armin Braun
nibbl3
Jake Landis
Nhat Nguyen
Julien Mailleret
Ryan Ernst
Przemko Robakowski
Lyudmila Fokina