The preallocate module needs access to java.io internals. However, in
order to open java.io to a specific module, rather than the unnamed
module as was previously done, the said module must be in the boot
layer.
This commit moves the preallocate module to libs. It adds it to the main
lib dir, though it does not add it as a compile dependency of server.
We built quite a bit of infrastructure to have one polling job
running via the `SchedulerEngine` and `ActiveSchedule`. This moves this
infrastructure outside x-pack to server so elasticsearch/modules can use
it and avoid re-implementing it using `threadPool.schedule`.
Introduces:
* New action that can be used to broadcast to unpromotable shards of a given IndexShardRoutingTable.
* New hook in ReplicationOperation for custom logic when the primary operation completes. If there is a failure, this increases the shard failures of the replication operation.
* Refresh action now uses the new hook to broadcast the unpromotable refresh action to all unpromotable shards.
Fixes ES-5454
Fixes ES-5212
Refactoring that drops the api suffix from package name
This will have to be followed up by a plugins/examples fix in imports
Also set an artifact group name to `org.elasticsearch.plugin` in the plugin-api and plugin-analysis-api
This doc values format is meant to be used for TSDB. It provides
compression using delta encoding, gcd encoding and bit-packing
encoding. The expectation is that such encoding works well for
fields whose values are monotonically increasing, like the timestamp
field and counter metric fields. Encoding and decoding fields using
the new format is available behind a feature flag.
This PR introduces the collectors of the health API telemetry. Our
target telemetry has the following shape:
```
{
"invocations": {
"total": 22,
"verbose_true": 12,
"verbose_false": 10
},
"statuses": {
"green": 10,
"yellow": 4,
"red": 8,
"values": ["green", "yellow", "red"]
},
"indicators": {
"red" : {
"master_stability": 2,
"ilm":2,
"slm": 4,
"values": ["master_stability", "ilm", "slm"]
},
"yellow": {
"disk": 1,
"shards_availability": 1,
"master_stability": 2,
"values": ["disk", "shards_availability", "master_stability"]
}
},
"diagnoses": {
"red": {
"elasticsearch:health:shards_availability:primary_unassigned": 1,
"elasticsearch:health:disk:add_disk_capacity_master_nodes": 3,
"values": ["elasticsearch:health:shards_availability:primary_unassigned", "elasticsearch:health:disk:add_disk_capacity_master_nodes"]
},
"yellow": {
"elasticsearch:health:disk:add_disk_capacity_data_nodes": 1,
"values": [""elasticsearch:health:disk:add_disk_capacity_data_nodes"]
}
}
}
```
This PR introduces the thread safe `Counters` class and the
`HealthApiStats` which keeps keeps of the metrics above based on the
health api responses that it encounters. The `HealthApiStatsAction`
collects the `HealthApiStats` of all nodes.
Part of: #90877
This PR adds the first part of the Prevalidate Node Removal API. This
API allows checking whether attempting to remove some node(s) from the
cluster is likely to succeed or not. This check is useful when a node
needs to be removed from a RED cluster, without risking loosing the last
copy of some RED shards.
In this PR, we only check whether a RED index is a Searchable Snapshot
index or not, in which case the removal of any node is safe as the RED
index is backed by a snapshot.
Relates #87776
This also drops the TimeSeries ceremonial interface.
Note that this change also moves a search cancellation test to
aggregations module. It does so by creating a base search cancallation
base class that both server and this module share.
Relates to #90283 Relates to #82273
This commit introduces a new aggregation module
and moves the `adjacency_matrix` to this new module.
The new module name is `aggregations`.
The new module will use the `org.elasticsearch.aggregations.bucket` package for all bucket aggregations.
Relates to #90283
Stable plugins are using @ extensible and @ NamedComponents annotations
to mark components to be loaded.
This commit is loading extensible classNames from extensibles.json and
named components from named_components.json
The scanning mechanism that can generate these files will be done later in a gradle plugin/plugin installer
relates #88980
This PR adds BloomFilter to Elasticsearch and enables it for the _id
field of non-data stream indices. BloomFilter should speed up the
performance of mget and update requests at a small expense of refresh,
merge, and storage.
This PR introduces the local health monitoring functionality needed for
#84811 . The monitor uses the `NodeService` to get the disk usage stats
and determines the node's disk health.
When a change in the disk's is detected or when the health node changes,
this class would be responsible to send the node's health to the health
node. Currently this is simulated with a method that just logs the
current health.
The monitor keeps the last reported health, this way, if something fails
on the next check it will try to resend the new health state.
Part of #84369. Split out from #87696. Introduce tracing interfaces in
advance of adding APM support to Elasticsearch. The only implementation
at this point is a no-op class.
This PR moves kNN search and dense vector support out of an xpack plugin and
into server.
In #87625 we plan to integrate ANN search into the main `_search` endpoint as a
new top-level component called `knn`. So kNN will be a dedicated part of the
search request, and we'll have kNN logic within the search phases. The classes
and logic will live in server, matching the other search components like
suggesters, field collapsing, etc.
* Remove Desired Nodes API from NON_OPERATOR_ACTIONS
* Add desired nodes to the operator privileges test
* Add desired nodes privileges integration tests
Resolves#87777.
The health node task is a permanent task, for this reason it doesn't make sense to wait for it in a method that waits for pending tasks. This fixes that.
Introducing a stable logging API under libs/logging.
This change covers the most common use cases for logging: fetching a logger with LogManager, emitting a log messages with Logger and Level.
It is influenced by log4j2-api, but do not include Marker and LogBuilder methods.
Also methods using org.apache.logging.log4j.util.Supplier are replaced with java.util.Supplier
The basic implementation is present in server and injected statically in LogConfigurator
relates #84478
When running in Docker, the elasticsearch-plugins.yml allows configuring
plugins that should be installed in the system. Upon Elasticsearch
starting up, plugins are installed/removed to match the configured
plugins. However, this happens late in startup, and it would be nice to
keep the main Elasticsearch process from ever writing outside the
configured data directories. Now that the server cli has been moved to
Java, this is possible.
This commit moves invocation of the plugins sync command into the server
cli. Note that the sync plugins action should probably be reworked as it
can be implement Command directly now. However, this commit tries to be
the minimal change possible to remove plugin cli knowledge from server.
Temporarily provide SystemPropertiesPropertySource to workaround a bug in Log4J where it does not declare the provider implementations in its module-info.
This PR implements downsampling operation on time series indices.
The PR creates a _rollup endpoint that allows users to downsample an index and can be
accessed by the following call:
POST /<source_index>/_rollup/<rollup_index>
{
"fixed_interval": "1d"
}
Requirements
An index can be downsampled if all of the following requirements are met:
Must be a time series index (have the index.mode: time_series index setting)
Must not be writeable (have the index.blocks.write: true index setting)
Must have dimension fields marked with mapping parameter time_series_dimension: true
Must have metric fields marked with mapping parameter time_series_metric
Relates to #74660Fixes#65769Fixes#69799
Finally, this PR is based on the code written for #64900
This PR represents the initial phase of Modularizing Elasticsearch (with
Java Modules).
This initial phase modularizes the core of the Elasticsearch server
with Java Modules, which is then used to load and configure extension
components atop the server. Only a subset of extension components are
modularized at this stage (other components come in a later phase).
Components are loaded dynamically at runtime with custom class loaders
(same as is currently done). Components with a module-info.class are
defined to a module layer.
This architecture is somewhat akin to the Modular JDK, where
applications run on the classpath. In the analogy, the Elasticsearch
server modules are the platform (thus are always resolved and present),
while components without a module-info.class are non-modular code
running atop the Elasticsearch server modules. The extension components
cannot access types from non-exported packages of the server modules, in
the same way that classpath applications cannot access types from
non-exported packages of modules from the JDK. Broadly, the core
Elasticseach java modules simply "wrap" the existing packages and export
them. There are opportunites to export less, which is best done in more
narrowly focused follow-up PRs.
The Elasticsearch distribution startup scripts are updated to put jars
on the module path (the class path is empty), so the distribution will
run the core of the server as java modules. A number of key components
have been retrofitted with module-info.java's too, and the remaining
components can follow later. Unit and functional tests run as
non-modular (since they commonly require package-private access), while
higher-level integration tests, that run the distribution, run as
modular.
Co-authored-by: Chris Hegarty <christopher.hegarty@elastic.co>
Co-authored-by: Ryan Ernst <ryan@iernst.net>
Co-authored-by: Rene Groeschke <rene@elastic.co>
Martijn van Groningen
Ryan Ernst
Francisco Fernández Castaño
Andrei Dan
Iraklis Psaroudakis
Joe Gallo
Thomas Dullien
Przemyslaw Gomulka
Salvatore Campagna
David Turner
Mary Gouseti
Pooya Salehi
Jack Conradson
Nhat Nguyen
Rory Hunter
Nikola Grcevski
Chris Hegarty
Julie Tibshirani
Artem Prigoda
Christos Soulios
Ignacio Vera