* Give rollling upgrade tests more information
This passes the version that rolling upgrade tests are coming *from*
into the actual tests so they can reason about it. This is useful
because we have features that are not supported on early versions and
we want to write rolling upgrade tests for them. We can't run those
features or assert anything about them in when they don't exist. You'd
think we could use the minimum version of a node in the cluster, but
that only works in the unupgraded phases - once we've completed the
upgrade we need to have the version that we came from to know what we
did in the mixed version.
* Nope
With the overall theme of trying to configure and add less to the build instead of just disabling it later,
we're replacing standalone-test by standalone-rest tasks avoids creating the
unused test tasks.
Standalone rest test plugin and the other rest test plugins behave a little bit different in the sense how source sets and test tasks are wired.
The standalone rest test plugin assumes that all RestTestTasks are using the same sourceSet (test). The yaml, java Rest test plugins use one dedicated sourceSet per test task.
In the long run we probably will migrate standalone-rest-test usages to one of the other plugins and deprecate standalone-rest-test
When libs/core was created, several classes were moved from server's
o.e.common package, but they were not moved to a new package. Split
packages need to go away long term, so that Elasticsearch can even think
about modularization. This commit moves all the classes under o.e.common
in core to o.e.core.
relates #73784
Today we fail to allocate searchable snapshot shards if the repository
containing their underlying data is not registered with the cluster.
This failure is somewhat messy, we allocate them and let the recovery
fail, and furthermore we don't automatically retry the allocation if the
repository is subsequently registered.
This commit introduces an allocation decider to prevent the allocation
of such shards, and explain more clearly why, and also a cluster state
listener that performs a reroute when a new repository is registered.
Relates #73669
Relates #73714
This commit adds some clean up logic to ESRestTestCase so
that searchable snapshots indices are deleted after test case
executions, before the snapshot and repositories are wipe out.
Backport of #73555
Legacy index templates are deprecated but ML was still using
them for its hidden indices.
This PR switches the legacy ML index templates to use the new
composable index template framework.
The composable index templates get installed once the master
node is on a version that understands them. For templates
that need to be up-to-date in mixed version clusters where the
master might still be on a version that doesn't understand
composable index templates we still ship the legacy template
too, and install this if required in the mixed version cluster.
(The notifications index template falls into this category.)
This makes a couple of places in the code a little messy, as
the new style template definitions don't contain a dummy _doc
level (where the type used to be), but the legacy template
definitions do - hopefully we can tidy this up in master once
8.0 is released.
There is one more change of note in this PR that is not
strictly related to switching to composable templates, but
which was shown up during the testing. We used to wait for
all templates to be installed by the master node before running
tests in mixed version clusters. I do not believe we should
have been doing this, as other upgrade orchestration systems,
e.g. Cloud, will not be doing this. Our production code needs
to install templates and/or mappings before any operation that
requires them if there's a chance that the elected master won't
have done this in time.
Fixes#65437
.snapshot-blob-cache index is created today with a data tier
preference set to data_cold,data_warm,data_hot. This does
not works well with autoscaling and clusters that only have
a hot and a frozen tier: in such cases autoscaling adds a
cold tier just to host this system index.
This commit changes the tier preference to data_content,data_hot.
Extract usage of internal API from TestClustersPlugin and PluginBuildPlugin and related plugins and build logic
This includes a refactoring of ElasticsearchDistribution to handle types
better in a way we can differentiate between supported Elasticsearch
Distribution types supported in TestCkustersPlugin and types only supported
in internal plugins.
It also introduces a set of internal versions of public plugins.
As part of this we also generate the plugin descriptors now.
As a follow up on this we can actually move these public used classes into
an extra project (declared as included build)
We keep LoggedExec and VersionProperties effectively public And workaround for RestTestBase
Script plugins cannot apply plugins and therefore wont work with porting
buildSrc to an included build as we plan. Therefore we take advantage
of moving our script plugins into precompiled script plugins.
As a limitation of this we ran into problems applying binary plugins
from script plugins and for now moved this out of those scripts.
Related to #71593 we move all build logic that is for elasticsearch build only into
the org.elasticsearch.gradle.internal* packages
This makes it clearer if build logic is considered to be used by external projects
Ultimately we want to only expose TestCluster and PluginBuildPlugin logic
to third party plugin authors.
This is a very first step towards that direction.
* Warn users if security is implicitly disabled
Elasticsearch has security features implicitly disabled by default for
Basic and Trial licenses, unless explicitly set in the configuration
file.
This may be good for onboarding, but it also lead to unintended insecure
clusters.
This change introduces clear warnings when security features are
implicitly disabled.
- a warning header in each REST response if security is implicitly
disabled;
- a log message during cluster boot.
Check cluster prior to assert doc count in DataStreamsUpgradeIT#testDataStreamValidationDoesNotBreakUpgrade test.
This to avoid node connect issues during the search executions.
Closes#71329
This PR adds metadata support for API keys. Metadata are of type
Map<String, Object> and can be optionally provided at API key creation time.
It is returned as part of GetApiKey response. It is also stored as part of
the authentication object to transfer throw the wire.
Note that it is not yet searchable and not exposed to any ingest processors.
They will be handled by separate PRs.
This commit introduces the `HasFrozenCacheAllocationDecider` which
restricts allocation of partial shards only to those nodes with a
configured shared cache, and also the `FrozenCacheInfoService` which
tracks which nodes do/don't have a frozen cache configured.
Some test suites underneath `AbstractUpgradeTestCase` have a 70s timeout
when waiting for green health, which is longer than the usual socket
timeout of 60s, so a failure yields no useful information. The
longer-than-usual timeout is to allow time for delayed allocation.
This commit extends the timeout to 90s in line with other similar test
cases.
Relates #69704
The test SearchableSnapshotsRollingUpgradeIT caused CI failures
at least two times (#69705) since it was merged into master. After
looking at all node logs, I wasn't able to determine precisely the
cause of the test suite timeout.
After looking at the code I think it is preferable to use a dedicated
path.repo for this test so that it does not interfere with existing
and future tests that could wipe the repositories and snapshots
of the default path.repo at any time. I also fixed an issue with
the storage paramater that should only work starting 7.12.0.
Finally, #69704 has been merged and it should provide more
nformation if this test fails again.
Closes#69705
Today searchable snapshots IndexInput implementations use the
blob store cache to cache the first 4096 bytes of every Lucene files.
After some experiments we think that we could adjust the length of
the cached data depending of the Lucene file that is read, caching
up to 64KB for Lucene metadata files (ie files that are fully read
when a Directory is opened) and only 1KB for other files.
The files that are cached up to 64KB are the following extensions:
"cfe", // compound file's entry table
"dvm", // doc values metadata file
"fdm", // stored fields metadata file
"fnm", // field names metadata file
"kdm", // Lucene 8.6 point format metadata file
"nvm", // norms metadata file
"tmd", // Lucene 8.6 terms metadata file
"tvm", // terms vectors metadata file
"vem" // Lucene 9.0 indexed vectors metadata
The 64KB limit can be configured on a per index basis through a new
index setting. This change is extracted from #69283 and does not
address the caching of CFS files.
Forward port #69625 to master.
Today when upgrading from 7.9.x or 7.10.x version to 7.11.x or later and
if two data (or more) data streams exist that have a overlapping prefix and
one data stream name ends with the a date suffix that matches with backing index
date pattern (uuuu.MM.dd) then new upgraded nodes may refuse to join. Essentially
preventing upgrade of the cluster to continue.
In this case the validation logic in `Metadata#validateDataStreams(...)` confuses
backing indices of one data stream as regular indices and thinks these indices
collide with another data stream.
In this validation only incorrectly fails if {data-stream-name} and
{data-steam-name}-{uuuu.MM.dd} name exist and later has been rolled
over more than the former and then upgrade cluster to 7.11+.
A 7.10.2 cluster with:
Data stream 1: logs-foobar
Backing indices: logs-foobar-000001
Data stream 2: logs-foobar-2021.01.13
Backing indices: logs-foobar-2021.01.13-000001, logs-foobar-2021.01.13-000002
When upgrading, then the new node will not join, because it thinks that
'logs-foobar-2021.01.13-000002' index collides with the backing index space
of data stream 'logs-foobar'.
This change tries to address this.
The transform internal index now uses the special
functionality for protecting system indices from
unwanted modification.
The system index descriptor replaces the index
template that was previously used with the transform
internal index. In places where we used to check
that the template was installed we now proactively
create the index. Master node actions will apply
the desired mappings if the index is auto-created
as a result of any indexing that they do.
The ML system indices now use the special functionality for
applying the correct mappings on first use. This replaces
the index templates that used to do this job, but were
vulnerable to tampering.
A number of other changes have had to be made to utilise
the system index functionality:
1. All fields previously missed out of mappings have been
added to the system index mappings, with the types that
would have been assigned dynamically in previous
versions. This is necessary because dynamic mappings
updates are banned for system indices, yet some of our
mappings allow dynamic updates.
2. As a result of the contradiction regarding dynamic
mappings, we are now very well protected against failing
to add new fields to the mappings for those indices that
exhibit the contradiction (which are .ml-config and
.ml-meta). This means their mappings don't need to be
explicitly compared to expected mappings in upgrade
tests now. Instead, any usage of a new field during or
after upgrade will trigger an error in any test this occurs
in.
3. Reserved fields for the config index were unnecessary
(only used by tests) and just added extra complication,
so they have been removed. We have the concept of
reserved fields for our results indices because end user
fields get added to results and we need to ensure they
don't clash with fields we want to use ourselves. This
problem does not exist for the config index.
As per the new licensing change for Elasticsearch and Kibana this commit
moves existing Apache 2.0 licensed source code to the new dual license
SSPL+Elastic license 2.0. In addition, existing x-pack code now uses
the new version 2.0 of the Elastic license. Full changes include:
- Updating LICENSE and NOTICE files throughout the code base, as well
as those packaged in our published artifacts
- Update IDE integration to now use the new license header on newly
created source files
- Remove references to the "OSS" distribution from our documentation
- Update build time verification checks to no longer allow Apache 2.0
license header in Elasticsearch source code
- Replace all existing Apache 2.0 license headers for non-xpack code
with updated header (vendored code with Apache 2.0 headers obviously
remains the same).
- Replace all Elastic license 1.0 headers with new 2.0 header in xpack.
Nik Everett
Rene Groeschke
Benjamin Trent
Ryan Ernst
David Turner
Tanguy Leroux
David Roberts
Dan Hermann
Julie Tibshirani
Henning Andersen
Przemysław Witek
Lyudmila Fokina
Yang Wang
Martijn van Groningen
Mark Vieira
Hendrik Muhs
Joe Gallo
Dimitris Athanasiou
Valeriy Khakhutskyy