For Stateless autoscaling, we'd need a different Alpha to track the task
execution time EWMA. This change makes the EWMA Alpha configurable and
uses a different value for the Write executors.
Closes ES-6325
The log4j JUL bridge turned out to have issues because it relied on java
beans. This commit implements a custom bridge between JUL and Log4j.
closes#94613
This commit adds the Log4j JUL bridge so that messages using JUL are
more nicely converted to log4j messages. Currently these messages are
captured via the stdout logging stream. This commit also adds a log4j
filter to replace the logging stream filtering mechanism used to quiet
some Lucene log messages that may be confusing to users.
closes#94613
Fixes#82794. Upgrade the spotless plugin, which addresses the issue
around formatting `instanceof` expressions. Formatting of statements
including lambdas seems to have improved too.
Today the master's pending task queue is just the
`PriorityBlockingQueue<Runnable>` belonging to the underlying
`ThreadPoolExecutor`. The reasons for this date back a long way but it
doesn't really reflect the structure of the queue as it exists today. In
particular, we must keep track of batches independently of the queue
itself, and must do various bits of unchecked casting to process
multiple items of the same type at once.
This commit introduces an new queueing mechanism, independent of the
executor's queue, which better represents the conceptual structure of
the master's pending tasks:
* Today we use a priority queue to allow important tasks to preempt less-important ones. However there are only a small number of priority levels, so it is simpler to maintain a queue for each priority, effectively replacing the sorting within the priority queue with a radix sort.
* Today when a task is submitted we perform a map lookup to see if it can be added to an existing batch or not. With this change we allow client code to create its own dedicated queue of tasks. The entries in the per-priority-level queues are themselves queues, one for each executor, representing the batches to be run.
* Today each task in the queue holds a reference to its executor, but the executor used to run a task may belong to a different task in the same batch. In practice we know they're the same executor (that's how batches are defined) but we cannot express this knowledge in the type system so we have to do a bunch of unchecked casting to work around it. With this change we associate each per-executor queue directly with its executor, avoiding the need to do all this unchecked casting.
* Today the master service must block its thread while waiting for each task to complete, because otherwise the executor would start to process the next task in the queue. This makes testing using a `DeterministicTaskQueue` harder (see `FakeThreadPoolMasterService`). This change avoids enqueueing tasks on the `ThreadPoolExecutor` unless there is genuinely work to do, although it leaves the removal of the actual blocking to a followup.
Closes#81626
We built quite a bit of infrastructure to have one polling job
running via the `SchedulerEngine` and `ActiveSchedule`. This moves this
infrastructure outside x-pack to server so elasticsearch/modules can use
it and avoid re-implementing it using `threadPool.schedule`.
This pull request adds the necessary support, and implementation, for profiling queries in the Tracer.
In order to use the APM Agent's inferred spans functionality, the active span's context has to be open in the current thread. This PR adds context-sensitive methods to the Tracer interface, implements them in APMTracer, and makes use of them in the private SearchService.executeQueryPhase(), which is on the stack for a lot of our most critical operations.
This commit centralize the processor count concept into the Processors class.
With this change now all the places using a processor count rely on this new class,
such as desired nodes, `node.processors` setting and autoscaling deciders.
- Processor counts are rounded to up to 5 decimal places
- Processors can be represented as doubles
Desired nodes processors were stored as floats, this poses some challenges during
upgrades as once the value is casted to a double, the precision increases and therefore
the number is not the same. In order to allow idempotent desired nodes updates after
upgrades, this commit introduces `DesiredNode#equalsWithProcessorsCloseTo(DesiredNode that)`
which allows comparing two desired nodes that differ up to a max delta in their processor
specification as floats.
The initial implementation of the embedded class loader took a brute
force approach to supporting multi-release JARs - iterating over all
possible release versions when searching for classes and resources. This
change improves upon that approach by deriving and caching package and
version specific maps, so class and resource loading can go directly to
the class and resource bytes, respectively, rather than searching.
It's hard to get empirical numbers to quanify just how much this change
improves the performance of classes loaded by this loader, and there is
typically only a couple of hundred classes loaded, but the initial cli
seems observably much quicker, while the server startup has improved
just a bit (at least on my machine).
The log4j configuration file is shipped with all ES distributions. We
also check recursively for files possibly added by plugins. If no files
are found, we give a helpful startup error message. However, since the
log4j2 configuration file shipped with ES should always exist, we can
check upfront in the cli before even initializing logging.
This commit moves the validation of an existing log4j2 properties file
to the server cli.
Deleting the pidfile when Elasticsearch is shutting down was moved in #86934.
However, the delete SM permission is still needed, since SM is installed
by the time we shutdown. This commit adds back to the pidfile delete
permission, and also rethrows any io exception so it can be logged by
our uncaught exception handler.
This is a result of structural search/replace in intellij. This only affects log methods with a signature
logger.info(ParametrizedMessage)
logger.info(ParametrizedMessage, Throwable)
relates https://github.com/elastic/elasticsearch/issues/86549
The class PluginInfo represents the plugin-descriptor.properties file
that each plugin must have. This commit renames the class to more
closely match what it represents: the plugin descriptor.
Now that the server cli is in java, we can do more system level things
inside it. This commit moves validating and writing the pidfile into the
server cli. One benefit is we get validation of directory/file problems
up front before even trying to start the ES process.
The Elasticsearch test framework initializes a test security manager, so
that tests run in a similar environment to that which we run in
production. However, some tests need to run without security manager
because they do "evil" things, like setting adding and removing jvm wide
shutdown hooks. Currently these tests must exist in separate projects,
mostly under qa/evil-tests, so that the security manager can be disabled
for the entire jvm with the tests.security.manager system property. The
separation between these tests and the other tests for the server makes
development of these tests more difficult, especially the inability to run
these within IntelliJ without manually adding additional sysprops.
This commit adds a new mechanism to disable security manager for these
tests. A new `@NoSecurityManager` annotation can be added to a test
suite. ESTestCase detects this annotation and removes the security
manager for the duration of that test suite. We could potentially allow
this per test as well, but I started with the simple suite wide
strategy. One example test is converted. The rest will be converted in
followups.
As we are moving away from StandaloneRestTestPlugin (see https://github.com/elastic/elasticsearch/pull/85491) we want to reduce the surface of that plugin.
If rest tests are used a rest test related plugin should be applied
Each Command subclass can implement close() so that resources will be
cleaned up on exceptional exit like SIGINT. This is implemented through
a shutdown hook added in the superclass constructor. However, this hook
makes testing difficult because the hook cannot be added in normal
tests, so a flag must be overriden when testing Command classes.
This commit moves the shutdown hook handling into the CliToolLauncher
that creates the command. It also adds non-evil tests that check how the
hook runs, in place of the old evil tests that actually registered a
real shutdown hook.
relates #85758
The EnvironmentAwareCommand is the most common base class for CLI tools.
Sometimes it is desirable for one tool to call into another. With the
new CliToolLauncher, this becomes easier to do. However, that only gives
access to the base Command class, which means the Environment would be
reparsed if the implementation is an EnvironmentAwareCommand. This
commit makes the execute method of EnvironmentAwareCommand public so
that other tools can pass the already parsed env.
relates #85758
The sysprops and envVars members of Command provide cli implementations
with information about the jvm process that is running. This is
convenient for runtime, but difficult for tests to mock because they
must subclass the cli class.
This commit adds a ProcessInfo record, and plumbs it through the
main and execute methods. The new record includes system properties,
environment variables and the working directory. By having this be a
single new parameter, additional information can be added in the future
without again needing to modify the method signatures.
relates #85758
When bootstrapping Elasticsearch, the console appender is automatically
added by our log4j config. However, there are some cases where we want
to remove that appender. First, if --quiet is passed, we do not want to
log anything to it. Second, if we are daemonizing, then we want to close
the streams, so we need to remove it if it exists. Third, when huge
guice/startup exceptions occur, we log these only to the ES log file, so
we need to remove the appender temporarily.
This commit moves the logic for mucking with the console appender into
LogConfigurator. In the future this can be better isolated within
logging, perhaps even avoiding creating the console appender to begin
with when using --quiet, but for now this at least gets some log specific
logic out of bootstrap.
relates #85758
Terminal is the abstraction Elasticsearch uses for all input and output,
both character based and binary. In an interactive shell, this is backed
by Java's Console, and in non-interactive it is backed by
stdin/stdout/stderr. Over time, the Terminal class has been amended to
support several different use cases, which has made constructing
subclasses for testing or filter based implementations complex. This
commit reworks Terminal so that the readers/writers/streams are
constructor arguments, instead of overrides. This allows subclasses to
simply call super with what is neeeded, rather than overloading several
methods and adding the same boilerplate implementation as others.
Note that the majority of the modifications here are to tests because
MockTerminal now has a factory method instead of direct constructor.
relates #85758
Serveral mechanisms exist for intializing logging in cli tools. Some
base Command classes exist which initialize logging. But they do this
late, when they are constructed, which may be after static init has
occured for classes grabbing a Logger. Other CLIs like node tool
explicitly initialize logging to avoid that problem.
This commit removes all the of the LoggingAware classes, and
unifies logging configuration to occur at the very beginning of the cli
launcher.
relates #85758
Currently any code needing to access system properties or environment
variables does it with the static methods provided by Java. While this
is ok in production since these are instantiated for the entire jvm
once, it makes any code reading these properties difficult to test
without mucking with the test jvm.
This commit adds system properties and environment variables to the base
Command class that our CLI tools use. While it does not propagate the
properties and env down for all possible uses in the system, it is the
first step, and it makes CLI testing a bit easier.
Extract cold paths + build a more efficient permissions collection for the hot
data path permissions that don't need the generic collection because it's all
file permissions only.
Lucene issues that resulted in elasticsearch changes:
LUCENE-9820 Separate logic for reading the BKD index from logic to intersecting it.
LUCENE-10377: Replace 'sortPos' with 'enableSkipping' in SortField.getComparator()
LUCENE-10301: make the test-framework a proper module by moving all test
classes to org.apache.lucene.tests
LUCENE-10300: rewrite how resources are read in ukrainian morfologik analyzer:
LUCENE-10054 Make HnswGraph hierarchical
Try to represent immutable data with Java records introduced in JEP 395
Convert only existing immutable classes, no "POJO with setters to a record" refactorings.
Today scaling thread pools never reject tasks but always add them to the
queue of task the execute, even in the case the thread pool executor is
shutting down or terminated. This behaviour does not work great when a
task is blocked waiting for another task from another scaling thread
pool to complete an I/O operation which will never be executed if the
task was enqueued just before the scaling thread pool was shutting down.
This situation is more likely to happen with searchable snapshots in
which multiple threads can be blocked waiting for parts of Lucene files
to be fetched and made available in cache. We saw tests failures in CI
where Lucene 9 uses concurrent threads (to asynchronously checks
indices) that were blocked waiting for cache files to be available and
failing because of leaking files handles (see #77017, #77178).
This pull request changes the `ForceQueuePolicy` used by scaling thread
pools so that it now accepts a `rejectAfterShutdown` flag which can be
set on a per thread pool basis to indicate when tasks should just be
rejected once the thread pool is shut down. Because we rely on many
scaling thread pools to be black holes and never reject tasks, this flag
is set to `false` on most of them to keep the current behavior. In some
cases where the rejection logic was already implemented correctly this
flag has been set to `true`.
This pull request also reimplements the interface
`XRejectedExecutionHandler` into an abstract class
`EsRejectedExecutionHandler` that allows to share some logic for
rejections.
Scripts using deprecation logger can trigger log files rolling over.
Scripts also run with a very limited permissions and without
doPrivileged section would cause SM exception
closes#81708
MBeans are sometimes used by third party libraries, e.g. to report
metrics through JMX. This commit builds upon the initial set of MBean
permissions added in #76329, to further alllow a plugin register an
MBean, i.e to allow plugins to grant MBeanTrustPermission("register").
Fix the split package org.elasticsearch.cli, between server and the cli library. Move the server org.elasticsearch.cli package to org.elasticsearch.common.cli. Removing split packages is a prerequisite to modularization.
This reverts commit d933ecd26c.
The revert had two conflicts. The first was very minor in JoinHelper.
The second was several tests in PersistedClusterStateServiceTests.
relates #78525
relates #71205
* Fix split packages in plugin cli
The plugin CLI was born out of the server jar, to allow the code to be
independent, as well as have more isolated tests that use jimfs to test
any many filesystem implementations. Yet the cli classes continue to
reside in the same package as the plugin service. This commit renames
the package in the cli to be cli specific.
Most of the change is simple renames and adding imports. A few cases
required making plugin service specific classes public. In the future
these could be made scoped exports specific to a plugin-cli module, but
for now this fix the split packages. Also note the PluginSecurity class
was only used by the cli, but is also only tested in the evil-tests,
which are run without security manager. Since those will all eventually
go away, the test package is also renamed there.
* fix shell scripts
This commit changes default deprecation logger level to CRITICAL, where default means deprecations emitted by DeprecationLogger#critical method.
It also introduces WARN deprecations which are emitted by DeprecationLogger#warn Those log lines emitted at WARN are meant to indicate that a functionality is deprecated but will not break at next major version.
relates #76754
MBeans are sometimes used by third party libraries, for example to report
metrics through JMX. This commit changes plugin policies to allow most
mbean permissions, with the exception of getting classloaders.
Today the master service processes pending tasks in priority order. If
high-priority tasks arrive too frequently then low-priority tasks are
starved of access to the master service and are not executed. This can
cause certain tasks to appear to be stuck due to apparently-unrelated
overloads elsewhere.
With this commit we measure the interval between times when the pending
task queue is empty; if this interval exceeds a configurable threshold
then we log a warning.
Allows plugin developers of custom codecs to opt out of the assertion in LuceneFilesExtensionsTests that checks that all
encountered Lucene file extensions are known to this class. In the future, we would like to add a proper plugin extension
point for this.
Relates #74150
When libs/core was created, several classes were moved from server's
o.e.common package, but they were not moved to a new package. Split
packages need to go away long term, so that Elasticsearch can even think
about modularization. This commit moves all the classes under o.e.common
in core to o.e.core.
relates #73784
Extract usage of internal API from TestClustersPlugin and PluginBuildPlugin and related plugins and build logic
This includes a refactoring of ElasticsearchDistribution to handle types
better in a way we can differentiate between supported Elasticsearch
Distribution types supported in TestCkustersPlugin and types only supported
in internal plugins.
It also introduces a set of internal versions of public plugins.
As part of this we also generate the plugin descriptors now.
As a follow up on this we can actually move these public used classes into
an extra project (declared as included build)
We keep LoggedExec and VersionProperties effectively public And workaround for RestTestBase
The path.data setting is now a singular string, but the method
dataFiles() that gives access to the path is still an array. This commit
renames the method and makes the return type a single Path.
relates #71205
Since multiple data path support has been removed, the Setting no longer
needs to support multiple values. This commit converts the
PATH_DATA_SETTING to a String setting from List<String>.
relates #71205
ning pan
Jake Landis
David Turner
Pooya Salehi
Ryan Ernst
Przemyslaw Gomulka
Rory Hunter
Andrei Dan
Rick Boyd
Francisco Fernández Castaño
Chris Hegarty
Rene Groeschke
Armin Braun
Mayya Sharipova
James Rodewig
Artem Prigoda
Tanguy Leroux
Mary Gouseti
Mark Vieira
Ryan Ernst
Yannick Welsch