synced flush is going to be replaced by flush. This commit allows to synced_flush api only in v7 compatibility mode.
Worth noting - sync_id is gone and won't be available in v7 responses from indices.stats
relates removal pr #50882
relates #51816
This introduces a new cmd line tool that generates the security configuration
for a new node in a new cluster (as opposed to joining an existing cluster).
The security configuration consists of TLS key and certificates, which
are stored in a directory inside the config path, as well as settings appended
to the elasticsearch.yml referencing the aforementioned certs.
Closes#74036. Since some orchestration platforms forbid periods in
environment variable names, allow Docker users to pass settings to ES
using an alternative name scheme. For example:
bootstrap.memory_lock
...becomes:
ES_BOOTSTRAP_MEMORY__LOCK
The setting name is uppercased, prefixed, all underscores are converted
to double underscores, and all periods are converted to underscores.
CCSDuelIT sometimes fails due to an unexpected number of shards returned in the refresh response. We create an index with a certain number of shards and no replicas, and sometimes the number of shards refresh is higher by one than the shards we created. This is not symptom of a problem though, as we are checking that there are no failures, but rather indicates a relocation happening, during which both copies of the same shard have been refreshed.
Closes#61258Closes#72637Closes#67754
Today the master service processes pending tasks in priority order. If
high-priority tasks arrive too frequently then low-priority tasks are
starved of access to the master service and are not executed. This can
cause certain tasks to appear to be stuck due to apparently-unrelated
overloads elsewhere.
With this commit we measure the interval between times when the pending
task queue is empty; if this interval exceeds a configurable threshold
then we log a warning.
This is related to #74755. The test to check that compression settings
are compatible after upgrade uses the cluster.remote prefix for remote
cluster settings. These settings did not exist in early versions of 6.x.
This commit adds an assume true to ensure that the test is not run
against older versions.
Found this to be the easiest fix, the alternative would have been to actually
wait for all snapshot meta threads to become blocked but that's kind of hacky.
closes#74743
With the overall theme of trying to configure and add less to the build instead of just disabling it later,
we're replacing standalone-test by standalone-rest tasks avoids creating the
unused test tasks.
Standalone rest test plugin and the other rest test plugins behave a little bit different in the sense how source sets and test tasks are wired.
The standalone rest test plugin assumes that all RestTestTasks are using the same sourceSet (test). The yaml, java Rest test plugins use one dedicated sourceSet per test task.
In the long run we probably will migrate standalone-rest-test usages to one of the other plugins and deprecate standalone-rest-test
This commit is related to #73497. It adds two new settings. The first setting
is transport.compression_scheme. This setting allows the user to
configure LZ4 or DEFLATE as the transport compression. Additionally, it
modifies transport.compress to support the value indexing_data. When
this setting is set to indexing_data only messages which are primarily
composed of raw source data will be compressed. This is bulk, operations
recovery, and shard changes messages.
This change updates the way we handle net new system indices, which are
those that have been newly introduced and do not require any BWC
guarantees around non-system access. These indices will not be included
in wildcard expansions for user searches and operations. Direct access
to these indices will also not be allowed for user searches.
The first index of this type is the GeoIp index, which this change sets
the new flag on.
Closes#72572
This PR returns the get snapshots API to the 7.x format (and transport client behavior) and enhances it for requests that ask for multiple repositories.
The changes for requests that target multiple repositories are:
* Add `repository` field to `SnapshotInfo` and REST response
* Add `failures` map alongside `snapshots` list instead of returning just an exception response as done for single repo requests
* Pagination now works across repositories instead of being per repository for multi-repository requests
closes#69108closes#43462
This adds just enough support for the traceparent header to be useful in es8.
Since Elasticsearch already logs in ECS format extending it with support for transaction.id and trace.id is a quick win.
This allows us to surface server/deprecation slow logs from an instrumented application using the Trace Logs feature.
Parsing `traceparent` in http layer and populating tasks with `trace_id` which is preserved in thread context.
Allows plugin developers of custom codecs to opt out of the assertion in LuceneFilesExtensionsTests that checks that all
encountered Lucene file extensions are known to this class. In the future, we would like to add a proper plugin extension
point for this.
Relates #74150
Pagination and snapshots for get snapshots API, build on top of the current implementation to enable work that needs this API for testing. A follow-up will leverage the changes to make things more efficient via pagination.
Relates https://github.com/elastic/elasticsearch/pull/73570 which does part of the under-the-hood changes required to efficiently implement this API on the repository layer.
Alpine Linux 3.14.0 is incompatible with older versions of Docker, so pin the
version that we use to 3.13. At some point in the future, it will
be possible to upgrade Alpine.
Also when compiling curl, if the configure step fails and a config.log
file exists, then dump it out before exiting to assist diagnosis.
Change the formatter config to sort / order imports, and reformat the
codebase. We already had a config file for Eclipse users, so Spotless now
uses that.
The "Eclipse Code Formatter" plugin ought to be able to use this file as
well for import ordering, but in my experiments the results were poor.
Instead, use IntelliJ's `.editorconfig` support to configure import
ordering.
I've also added a config file for the formatter plugin.
Other changes:
* I've quietly enabled the `toggleOnOff` option for Spotless. It was
already possible to disable formatting for sections using the markers
for docs snippets, so enabling this option just accepts this reality
and makes it possible via `formatter:off` and `formatter:on` without
the restrictions around line length. It should still only be used as
a very last resort and with good reason.
* I've removed mention of the `paddedCell` option from the contributing
guide, since I haven't had to use that option for a very long time. I
moved the docs to the spotless config.
Recursively remove write access from the bin, jdk, lib and
modules directories, since this access is not required, and removing
it makes it harder to exploit other issues in an ES distribution.
Modularization of the JDK has been ongoing for several years. Recently
in Java 16 the JDK began enforcing module boundaries by default. While
Elasticsearch does not yet use the module system directly, there are
some side effects even for those projects not modularized (eg #73517).
Before we can even begin to think about how to modularize, we must
Prepare The Way by enforcing packages only exist in a single jar file,
since the module system does not allow packages to coexist in multiple
modules.
This commit adds a precommit check to the build which detects split
packages. The expectation is that we will add the existing split
packages to the ignore list so that any new classes will not exacerbate
the problem, and the work to cleanup these split packages can be
parallelized.
relates #73525
ParseField is part of the x-content lib, yet it doesn't exist under the
same root package as the rest of the lib. This commit moves the class to
the appropriate package.
relates #73784
When libs/core was created, several classes were moved from server's
o.e.common package, but they were not moved to a new package. Split
packages need to go away long term, so that Elasticsearch can even think
about modularization. This commit moves all the classes under o.e.common
in core to o.e.core.
relates #73784
Same as #72644. This is a much longer running action than normal
get snapshots even so it should definitely be cancellable.
Parallelization for this action will be introduced in a separate PR.
Extract usage of internal API from TestClustersPlugin and PluginBuildPlugin and related plugins and build logic
This includes a refactoring of ElasticsearchDistribution to handle types
better in a way we can differentiate between supported Elasticsearch
Distribution types supported in TestCkustersPlugin and types only supported
in internal plugins.
It also introduces a set of internal versions of public plugins.
As part of this we also generate the plugin descriptors now.
As a follow up on this we can actually move these public used classes into
an extra project (declared as included build)
We keep LoggedExec and VersionProperties effectively public And workaround for RestTestBase
If this runs needlessly for large repositories (especially in timeout/retry situations)
it's a significant memory+cpu hit => made it cancellable like we recently did for many
other endpoints.
Script plugins cannot apply plugins and therefore wont work with porting
buildSrc to an included build as we plan. Therefore we take advantage
of moving our script plugins into precompiled script plugins.
As a limitation of this we ran into problems applying binary plugins
from script plugins and for now moved this out of those scripts.
Przemyslaw Gomulka
Mark Vieira
Rory Hunter
Albert Zaharovits
Jake Landis
Armin Braun
Ignacio Vera
Luca Cavanna
David Turner
Tim Brooks
Rene Groeschke
Yang Wang
Jay Modi
Martijn Laarman
Yannick Welsch
Ryan Ernst
Francisco Fernández Castaño
Nhat Nguyen
Dan Hermann