The build version is made up of a few parts in non-release builds. Both
the snapshot and pre-release qualifiers are appended to it. These
qualifiers used to be part of Version, but in 7.0 the qualifiers were
made to be found only in the build info. The Build class retains these
qualifiers through the compile ES version extracted from the server jar
at runtime.
Build.qualifiedVersion() is suppose to provide the fully qualified
version, including snapshot and pre-release qualifiers. Yet
Build.version() also includes this information; there is no distinction
since the qualifier was moved to be only in the build info.
This commit separates the pre-release qualifier from the version. It
maintains bwc in talking to older nodes, passing the fully qualified
version there, but in current nodes splits out the pre-release qualifier
into a new member of Build.
The Plugin CLI can generally treat strings opaquely. We had some logic
comparing earlier versions, but what we really care about with most of our
plugins is whether or not they were built with the current version of
Elasticsearch, not whether they were built before or after. (This question
will be trickier with stable plugins, but none of that code is in the CLI.)
The CLI classes can be cleaned up even more once Version is removed from
PluginDescriptor. Some of the tests can't use opaque strings for versions
until PluginDescriptor can handle them.
* Remove Version from Install and List plugin actions
* Remove Version from SyncPluginsAction
Another round of automated fixes to this, marking things that can be
made static as static. Saves some JIT cycles but also turns some lambdas
from capturing to non-capturing and makes the "utilityness" of some
classes visible.
In order for build info to be pluggable for serverless, the current
build needs to be lazily determined. This commit moves the CURRENT
constant to a static method.
relates #96861
This commit updates the plugin cli and scanner components to use ASM 9.5.
The update is required to successfully test with JDK 21. Tests in this component programatically run the java source compiler, which generates class files with major version 65, then tries to parse those generated class files. Without this change the tests fail with java.lang.IllegalArgumentException: Unsupported class file major version 65.
stable plugins not build with ES's gradle plugin will not have named_components.json file.
To allow these plugins to expose their named components, a scan can be performed upon install.
relates #88980
Use local-independent `Strings.format` method instead of `String.format(Locale.ROOT, ...)`.
Inline `ESTestCase.forbidden` calls with `Strings.format` for the consistency sake.
Add `Strings.format` alias in `common.Strings`
Removing the custom dependency checksum functionality in favor of Gradle build-in dependency verification support.
- Use sha256 in favor of sha1 as sha1 is not considered safe these days.
Closes https://github.com/elastic/elasticsearch/issues/69736
The new stable plugin api will have a slightly different descriptor file
format. This commit prepares for stable plugins by adding support for
reading those new files. The basic info for a plugin is the same like
name and version info. Other stuff like classname are not necessary. The
one additional property specific to the new plugins is "modular", which
indicates whether the jars of the plugin should be loaded as named
modules (this is akin to setting the module path when running java).
The ingest attachment processor is currently available as a plugin. This
commit moves the processor to the default distribution so it is always
available.
Bootstrap plugins were an internal mechanism added to allow a
filesystemprovider for cloud with the quota-aware-fs plugin. Since that
was removed, bootstrap plugins no longer serve a purpose. They were
never officially documented because they were for internal use only.
This commit removes the bootstrap plugins infrastructure.
When running in Docker, the elasticsearch-plugins.yml allows configuring
plugins that should be installed in the system. Upon Elasticsearch
starting up, plugins are installed/removed to match the configured
plugins. However, this happens late in startup, and it would be nice to
keep the main Elasticsearch process from ever writing outside the
configured data directories. Now that the server cli has been moved to
Java, this is possible.
This commit moves invocation of the plugins sync command into the server
cli. Note that the sync plugins action should probably be reworked as it
can be implement Command directly now. However, this commit tries to be
the minimal change possible to remove plugin cli knowledge from server.
The class PluginInfo represents the plugin-descriptor.properties file
that each plugin must have. This commit renames the class to more
closely match what it represents: the plugin descriptor.
Our plugin loading code uses the PluginBundle class to describe a plugin on
disk. This should be an implementation detail of the plugin loading code, and
not part of the API shared with the plugin CLI.
* Refactor to hide internal code from Plugin CLI
* Add javadoc
* Move static plugin loading methods to util class
We had a set of utility methods in PluginsService that were public and
static so that the plugin cli tool could use them. Here, we move those
methods to a utility class so that the plugin CLI doesn't need to import
PluginsService. For the same reason, we also move
"PluginsService.Bundle" to a top-level class called "PluginBundle."
Finally, we move tests to the appropriate test classes.
Each Command subclass can implement close() so that resources will be
cleaned up on exceptional exit like SIGINT. This is implemented through
a shutdown hook added in the superclass constructor. However, this hook
makes testing difficult because the hook cannot be added in normal
tests, so a flag must be overriden when testing Command classes.
This commit moves the shutdown hook handling into the CliToolLauncher
that creates the command. It also adds non-evil tests that check how the
hook runs, in place of the old evil tests that actually registered a
real shutdown hook.
relates #85758
* Fix test for Windows line endings
* Make more tests line-separator agnostic
* Skip some tests where we need to fix production code
Co-authored-by: Rory Hunter <roryhunter2@gmail.com>
The EnvironmentAwareCommand is the most common base class for CLI tools.
Sometimes it is desirable for one tool to call into another. With the
new CliToolLauncher, this becomes easier to do. However, that only gives
access to the base Command class, which means the Environment would be
reparsed if the implementation is an EnvironmentAwareCommand. This
commit makes the execute method of EnvironmentAwareCommand public so
that other tools can pass the already parsed env.
relates #85758
The sysprops and envVars members of Command provide cli implementations
with information about the jvm process that is running. This is
convenient for runtime, but difficult for tests to mock because they
must subclass the cli class.
This commit adds a ProcessInfo record, and plumbs it through the
main and execute methods. The new record includes system properties,
environment variables and the working directory. By having this be a
single new parameter, additional information can be added in the future
without again needing to modify the method signatures.
relates #85758
The cli lib has the SuppressForbidden annotation, but so does core,
which cli depends on. This commit removes the SuppressForbidden from
cli, in favor of the one from core.
relates #85758
Terminal is the abstraction Elasticsearch uses for all input and output,
both character based and binary. In an interactive shell, this is backed
by Java's Console, and in non-interactive it is backed by
stdin/stdout/stderr. Over time, the Terminal class has been amended to
support several different use cases, which has made constructing
subclasses for testing or filter based implementations complex. This
commit reworks Terminal so that the readers/writers/streams are
constructor arguments, instead of overrides. This allows subclasses to
simply call super with what is neeeded, rather than overloading several
methods and adding the same boilerplate implementation as others.
Note that the majority of the modifications here are to tests because
MockTerminal now has a factory method instead of direct constructor.
relates #85758
Most classes under elasticsearch-core had been moved to the o.e.core
package. However, a couple io related classes remained in an "internal"
package. This commit moves Streams and IOUtils to the core package, as
they are no more "internal" than the rest of the classes in core.
Serveral mechanisms exist for intializing logging in cli tools. Some
base Command classes exist which initialize logging. But they do this
late, when they are constructed, which may be after static init has
occured for classes grabbing a Logger. Other CLIs like node tool
explicitly initialize logging to avoid that problem.
This commit removes all the of the LoggingAware classes, and
unifies logging configuration to occur at the very beginning of the cli
launcher.
relates #85758
Currently any code needing to access system properties or environment
variables does it with the static methods provided by Java. While this
is ok in production since these are instantiated for the entire jvm
once, it makes any code reading these properties difficult to test
without mucking with the test jvm.
This commit adds system properties and environment variables to the base
Command class that our CLI tools use. While it does not propagate the
properties and env down for all possible uses in the system, it is the
first step, and it makes CLI testing a bit easier.
CLI scripts have a common infrastructure in that they call to the shared
elasticsearch-cli shell script which launches them with the appropriate
java command line. However, each underlying Java class must implement
its own main method.
This commit introduces a single main method to be shared by CLIs. The
new CliToolLauncher takes in system properties to determine which tool
is being run, and a new CliToolProvider SPI allows defining and finding
the named tools.
relates #85758
Co-authored-by: William Brafford <william.brafford@elastic.co>
The main method of Command is not normally called by cli tests. Instead
they call the execute helper which calls mainWithoutErrorHandling.
Sometimes, though, it is desirable to test the top level behavior. This
commit cleans up the base class tests to not call main directly. This
will help with future refactorings to change the signature of main, so
that less uses need to be changed.
The default distribution is the only remaining build flavor, and has been for
quite a while now. This commit removes flavor from the internal Build
class. It keeps rest api compat for nodes info for now by hardcoding
`default`.
Plugin downloads can sometimes take unexpectedly
long because of slow random number generation on certain
operating system versions, e.g. Debian based OS
distros with kernels older than 5.6. This PR adds
a warning message for users explaining why plugin
installation from remote URLs is taking unexpectedly long.
Lucene issues that resulted in elasticsearch changes:
LUCENE-9820 Separate logic for reading the BKD index from logic to intersecting it.
LUCENE-10377: Replace 'sortPos' with 'enableSkipping' in SortField.getComparator()
LUCENE-10301: make the test-framework a proper module by moving all test
classes to org.apache.lucene.tests
LUCENE-10300: rewrite how resources are read in ukrainian morfologik analyzer:
LUCENE-10054 Make HnswGraph hierarchical
Try to represent immutable data with Java records introduced in JEP 395
Convert only existing immutable classes, no "POJO with setters to a record" refactorings.
Closes#81652.
Convert the `repository-azure`, `repository-gcs` and `repository-s3`
plugins into modules, so that they are always included in the
Elasticsearch distribution. Also change plugin installation, removal
and syncing so that attempting to add or remove these plugins still
succeeds but is now a no-op.
JEP 361[https://openjdk.java.net/jeps/361] added support for switch expressions
which can be much more terse and less error-prone than switch statements.
Another useful feature of switch expressions is exhaustiveness: we can make
sure that an enum switch expression covers all the cases at compile time.
The ES code base is quite JSON heavy. It uses a lot of multi-line JSON requests in tests which need to be escaped and concatenated which in turn makes them hard to read. Let's try to leverage Java 15 text blocks for representing them.
Fix the split package org.elasticsearch.cli, between server and the cli library. Move the server org.elasticsearch.cli package to org.elasticsearch.common.cli. Removing split packages is a prerequisite to modularization.
Closes#70219.
Introduce a declarative way for the Elasticsearch server to manage plugins,
which reads the `elasticsearch-plugins.yml` file and works which out
plugins need to be added and / or removed to match the configuration. Also
make it possible to configure a proxy in the config file, instead of
through the environment.
Most of the work of adding and removing is still done in the
`InstallPluginAction` and `RemovePluginAction` classes, so the
behaviour should be the same as with the `install` and `remove`
commands. However, these commands will now abort if the above config
file exists. The intent is to make it harder for the configuration
to drift.
This new method only applies to `docker` distribution types at the
moment.
Since this syncing mechanism declarative, rather than imperative,
the Cloud-specific plugin wrapper script is no longer required.
Instead, an environment variable informs `InstallPluginAction` to
install plugins from an archive directory instead of downloading
them, where possible.
* Fix split packages in plugin cli
The plugin CLI was born out of the server jar, to allow the code to be
independent, as well as have more isolated tests that use jimfs to test
any many filesystem implementations. Yet the cli classes continue to
reside in the same package as the plugin service. This commit renames
the package in the cli to be cli specific.
Most of the change is simple renames and adding imports. A few cases
required making plugin service specific classes public. In the future
these could be made scoped exports specific to a plugin-cli module, but
for now this fix the split packages. Also note the PluginSecurity class
was only used by the cli, but is also only tested in the evil-tests,
which are run without security manager. Since those will all eventually
go away, the test package is also renamed there.
* fix shell scripts
A MultiCommand requires that a (sub)command is provided for each
execution. Previously, the error handling for MultiCommand would
simply state "ERROR: Missing command" but give no assistance about
what commands were accepted. The user was required to pass "-help" in
order to get the command list.
This commit changes the behaviour so that the list of commands is
printed after the error message.
Part of #70219.
Split the install and remove plugin CLI commands into `Command` and
`Action` classes, so that the main implementations can be reused
in a later PR to create a `Sync` command.
Change the formatter config to sort / order imports, and reformat the
codebase. We already had a config file for Eclipse users, so Spotless now
uses that.
The "Eclipse Code Formatter" plugin ought to be able to use this file as
well for import ordering, but in my experiments the results were poor.
Instead, use IntelliJ's `.editorconfig` support to configure import
ordering.
I've also added a config file for the formatter plugin.
Other changes:
* I've quietly enabled the `toggleOnOff` option for Spotless. It was
already possible to disable formatting for sections using the markers
for docs snippets, so enabling this option just accepts this reality
and makes it possible via `formatter:off` and `formatter:on` without
the restrictions around line length. It should still only be used as
a very last resort and with good reason.
* I've removed mention of the `paddedCell` option from the contributing
guide, since I haven't had to use that option for a very long time. I
moved the docs to the spotless config.
Modularization of the JDK has been ongoing for several years. Recently
in Java 16 the JDK began enforcing module boundaries by default. While
Elasticsearch does not yet use the module system directly, there are
some side effects even for those projects not modularized (eg #73517).
Before we can even begin to think about how to modularize, we must
Prepare The Way by enforcing packages only exist in a single jar file,
since the module system does not allow packages to coexist in multiple
modules.
This commit adds a precommit check to the build which detects split
packages. The expectation is that we will add the existing split
packages to the ignore list so that any new classes will not exacerbate
the problem, and the work to cleanup these split packages can be
parallelized.
relates #73525
When libs/core was created, several classes were moved from server's
o.e.common package, but they were not moved to a new package. Split
packages need to go away long term, so that Elasticsearch can even think
about modularization. This commit moves all the classes under o.e.common
in core to o.e.core.
relates #73784
Ryan Ernst
William Brafford
Armin Braun
Simon Cooper
Rene Groeschke
Chris Hegarty
Craig Taverner
David Turner
Nikola Grcevski
Przemyslaw Gomulka
Artem Prigoda
Mayya Sharipova
Rory Hunter
Ryan Ernst
Tim Vernum