This will allow components to add custom metadata to deprecation issues.
This make extracting additional details about deprecations more robust,
otherwise these details need to be parsed from the deprecation message field.
Adjusted the ml model snapshot deprecation to use custom metadata, and
included the job id and snapshot id as custom metadata.
Closes#73089
Change the formatter config to sort / order imports, and reformat the
codebase. We already had a config file for Eclipse users, so Spotless now
uses that.
The "Eclipse Code Formatter" plugin ought to be able to use this file as
well for import ordering, but in my experiments the results were poor.
Instead, use IntelliJ's `.editorconfig` support to configure import
ordering.
I've also added a config file for the formatter plugin.
Other changes:
* I've quietly enabled the `toggleOnOff` option for Spotless. It was
already possible to disable formatting for sections using the markers
for docs snippets, so enabling this option just accepts this reality
and makes it possible via `formatter:off` and `formatter:on` without
the restrictions around line length. It should still only be used as
a very last resort and with good reason.
* I've removed mention of the `paddedCell` option from the contributing
guide, since I haven't had to use that option for a very long time. I
moved the docs to the spotless config.
When libs/core was created, several classes were moved from server's
o.e.common package, but they were not moved to a new package. Split
packages need to go away long term, so that Elasticsearch can even think
about modularization. This commit moves all the classes under o.e.common
in core to o.e.core.
relates #73784
So far when a deprecated route was executed it only emitted deprecation
warning once. All subsequent deprecated routes (even when path and
method were different) were throttled because the key was the same -
deprecated_route
This commit suffixes the deprecation key with path and method.
closes#73002
Extract usage of internal API from TestClustersPlugin and PluginBuildPlugin and related plugins and build logic
This includes a refactoring of ElasticsearchDistribution to handle types
better in a way we can differentiate between supported Elasticsearch
Distribution types supported in TestCkustersPlugin and types only supported
in internal plugins.
It also introduces a set of internal versions of public plugins.
As part of this we also generate the plugin descriptors now.
As a follow up on this we can actually move these public used classes into
an extra project (declared as included build)
We keep LoggedExec and VersionProperties effectively public And workaround for RestTestBase
Related to #71593 we move all build logic that is for elasticsearch build only into
the org.elasticsearch.gradle.internal* packages
This makes it clearer if build logic is considered to be used by external projects
Ultimately we want to only expose TestCluster and PluginBuildPlugin logic
to third party plugin authors.
This is a very first step towards that direction.
Related to #51816
Makes `Route`s `RestApiVersion` -aware (and `RestHandler`s `RestApiVersion` -agnostic). Refactors
how `Route`s are constructed in the case of deprecation or replacement of routes.
Extending deprecation logger to also log compatible API warnings.
These warnings will be written to deprecation log file, data stream and will emit response warning header - the same as deprecation logs. In order to comply with constant keyword type of data_stream.datatype the value will have to be the same for compatible and deprecation log - elasticsearch.deprecation.
The compatible and deprecation logs will differ by elasticsearch.event.category field which will always be DeprecationCategory.COMPATIBLE_API for all compatible logs
closes#52369
As per the new licensing change for Elasticsearch and Kibana this commit
moves existing Apache 2.0 licensed source code to the new dual license
SSPL+Elastic license 2.0. In addition, existing x-pack code now uses
the new version 2.0 of the Elastic license. Full changes include:
- Updating LICENSE and NOTICE files throughout the code base, as well
as those packaged in our published artifacts
- Update IDE integration to now use the new license header on newly
created source files
- Remove references to the "OSS" distribution from our documentation
- Update build time verification checks to no longer allow Apache 2.0
license header in Elasticsearch source code
- Replace all existing Apache 2.0 license headers for non-xpack code
with updated header (vendored code with Apache 2.0 headers obviously
remains the same).
- Replace all Elastic license 1.0 headers with new 2.0 header in xpack.
Closes#64824. Introduce the concept of categories to deprecation
logging. Every location where we log a deprecation message must now
include a deprecation category.
We decided to rename `QueryShardContext` to clarify that it supports all parts
of search request execution. Before there was confusion over whether it should
only be used for building queries, or maybe only used in the query phase. This
PR also updates the javadocs.
Closes#64740.
We were depending on the BouncyCastle FIPS own mechanics to set
itself in approved only mode since we run with the Security
Manager enabled. The check during startup seems to happen before we
set our restrictive SecurityManager though in
org.elasticsearch.bootstrap.Elasticsearch , and this means that
BCFIPS would not be in approved only mode, unless explicitly
configured so.
This commit sets the appropriate JVM property to explicitly set
BCFIPS in approved only mode in CI and adds tests to ensure that we
will be running with BCFIPS in approved only mode when we expect to.
It also sets xpack.security.fips_mode.enabled to true for all test clusters
used in fips mode and sets the distribution to the default one. It adds a
password to the elasticsearch keystore for all test clusters that run in fips
mode.
Moreover, it changes a few unit tests where we would use bcrypt even in
FIPS 140 mode. These would still pass since we are bundling our own
bcrypt implementation, but are now changed to use FIPS 140 approved
algorithms instead for better coverage.
It also addresses a number of tests that would fail in approved only mode
Mainly:
Tests that use PBKDF2 with a password less than 112 bits (14char). We
elected to change the passwords used everywhere to be at least 14
characters long instead of mandating
the use of pbkdf2_stretch because both pbkdf2 and
pbkdf2_stretch are supported and allowed in fips mode and it makes sense
to test with both. We could possibly figure out the password algorithm used
for each test and adjust password length accordingly only for pbkdf2 but
there is little value in that. It's good practice to use strong passwords so if
our docs and tests use longer passwords, then it's for the best. The approach
is brittle as there is no guarantee that the next test that will be added won't
use a short password, so we add some testing documentation too.
This leaves us with a possible coverage gap since we do support passwords
as short as 6 characters but we only test with > 14 chars but the
validation itself was not tested even before. Tests can be added in a followup,
outside of fips related context.
Tests that use a PKCS12 keystore and were not already muted.
Tests that depend on running test clusters with a basic license or
using the OSS distribution as FIPS 140 support is not available in
neither of these.
Finally, it adds some information around FIPS 140 testing in our testing
documentation reference so that developers can hopefully keep in
mind fips 140 related intricacies when writing/changing docs.
This adds checks that verify that machine learning anomaly job model snapshots support the required minimal version.
If any are not the required version, directions are given to either delete the model snapshot, or utilize the _upgrade API.
relates: https://github.com/elastic/elasticsearch/issues/64154
Closes#65589 hopefully. There is a test case that checks whether
deprecation logs have been indexed. It uses `assertBusy` so that it
keeps checking for a period, since it may take some time for the data to
become available. However, the test nonetheless occasionally still
fails, with no shards being available to search.
In an attempt to address this, add a `_refresh` call on the deprecation
data stream, and increase the `assertBusy` timeout to 30s.
This ports the majority of the rest integ tests tasks to use the task avoidance api.
- There are some edge cases left that we need to investigate, but we can do that separately.
The implementation for indexing deprecation logs to a data stream
(#58924) relied on the Stack template for `logs-*-*`. This meant
that if the user disabled the stack templates, the templates would
also be unavailable for the deprecation logs.
Change the implementation so that:
* There is a separate template for deprecation logging
* The data stream is marked as hidden * The data stream name is
prefixed with a period (`.`)
This commit address some build failures from the perspective of Intellij.
These changes include:
* changing an order of a dependency definition that seems to can cause Intellij build to fail.
* introduction of an abstract class out of the test source set (seems to be an issue sharing
classes cross projects with non-standard source sets.
* a couple of missing dependency definitions (not sure how the command line worked prior to this)
The bulk processor used to index deprecation logs was doing nothing to
output any errors encountered when writing documents to ES. These are
now detected.
Also expand the deprecation REST tests to check the shape of the indexed
documents.
Closes#46106. Implement a new log4j appender for deprecation logging, in
order to write logs to a dedicated data stream. This is controlled by a new
setting, `cluster.deprecation_indexing.enabled`.
Martijn van Groningen
Rory Hunter
Ryan Ernst
Przemyslaw Gomulka
Rene Groeschke
Mark Vieira
Joe Gallo
Julie Tibshirani
Ioannis Kakavas
Benjamin Trent
David Kyle
Jake Landis