Commit Graph

352 Commits

Author SHA1 Message Date
Nik Everett a6fe766b7b
Give rollling upgrade tests more information (#76360)
* Give rollling upgrade tests more information

This passes the version that rolling upgrade tests are coming *from*
into the actual tests so they can reason about it. This is useful
because we have features that are not supported on early versions and
we want to write rolling upgrade tests for them. We can't run those
features or assert anything about them in when they don't exist. You'd
think we could use the minimum version of a node in the cluster, but
that only works in the unupgraded phases - once we've completed the
upgrade we need to have the version that we came from to know what we
did in the mixed version.

* Nope
2021-08-12 09:20:51 -04:00
Rene Groeschke d8e4e48a3b
Avoid creating unused test tasks (#74644)
With the overall theme of trying to configure and add less to the build instead of just disabling it later,
we're replacing standalone-test by standalone-rest tasks avoids creating the
unused test tasks.

Standalone rest test plugin and the other rest test plugins behave a little bit different in the sense how source sets and test tasks are wired.

The standalone rest test plugin assumes that all RestTestTasks are using the same sourceSet (test). The yaml, java Rest test plugins use one dedicated sourceSet per test task.

In the long run we probably will migrate standalone-rest-test usages to one of the other plugins and deprecate standalone-rest-test
2021-06-30 14:11:25 +02:00
Benjamin Trent d4251038e6
[ML] unmuting bwc tests for #73926 (#74012) 2021-06-10 12:34:53 -04:00
Benjamin Trent 6235ff2a5b
[ML] adjusting BWC for datafeed stats for #73926 (#74003) 2021-06-10 10:01:46 -04:00
Ryan Ernst 68817d7ca2
Rename o.e.common in libs/core to o.e.core (#73909)
When libs/core was created, several classes were moved from server's
o.e.common package, but they were not moved to a new package. Split
packages need to go away long term, so that Elasticsearch can even think
about modularization. This commit moves all the classes under o.e.common
in core to o.e.core.

relates #73784
2021-06-08 09:53:28 -07:00
David Turner 7a3a45a184
Reroute when new repository is registered (#73761)
Today we fail to allocate searchable snapshot shards if the repository
containing their underlying data is not registered with the cluster.
This failure is somewhat messy, we allocate them and let the recovery
fail, and furthermore we don't automatically retry the allocation if the
repository is subsequently registered.

This commit introduces an allocation decider to prevent the allocation
of such shards, and explain more clearly why, and also a cluster state
listener that performs a reroute when a new repository is registered.

Relates #73669
Relates #73714
2021-06-07 11:04:14 +01:00
Tanguy Leroux 4927b6917d
Delete mounted indices after test case in ESRestTestCase (#73650)
This commit adds some clean up logic to ESRestTestCase so 
that searchable snapshots indices are deleted after test case 
executions, before the snapshot and repositories are wipe out.

Backport of #73555
2021-06-02 15:06:44 +02:00
David Roberts 0216cf065b
[ML] Switch ML internal index templates to composable templates (#73232)
Legacy index templates are deprecated but ML was still using
them for its hidden indices.

This PR switches the legacy ML index templates to use the new
composable index template framework.

The composable index templates get installed once the master
node is on a version that understands them.  For templates
that need to be up-to-date in mixed version clusters where the
master might still be on a version that doesn't understand
composable index templates we still ship the legacy template
too, and install this if required in the mixed version cluster.
(The notifications index template falls into this category.)
This makes a couple of places in the code a little messy, as
the new style template definitions don't contain a dummy _doc
level (where the type used to be), but the legacy template
definitions do - hopefully we can tidy this up in master once
8.0 is released.

There is one more change of note in this PR that is not
strictly related to switching to composable templates, but
which was shown up during the testing.  We used to wait for
all templates to be installed by the master node before running
tests in mixed version clusters.  I do not believe we should
have been doing this, as other upgrade orchestration systems,
e.g. Cloud, will not be doing this.  Our production code needs
to install templates and/or mappings before any operation that
requires them if there's a chance that the elected master won't
have done this in time.

Fixes #65437
2021-05-24 11:13:24 +01:00
Rene Groeschke 2d42c26249
Keep StandaloneRestIntegTest task public (#72865)
This is used by hadoop
2021-05-09 11:36:16 +02:00
Rene Groeschke 78182686fc
Remove more usage of interal logic in public build api (#72858) 2021-05-08 23:06:23 +02:00
Tanguy Leroux 39c2ea2e9c
Adjust version in SearchableSnapshotsRollingUpgradeIT (#72847)
Now #72817 is merged in 7.13,7.x branches we can adjust the 
version in SearchableSnapshotsRollingUpgradeIT.
2021-05-07 13:40:52 +02:00
Tanguy Leroux dfbcb6b8ff
Change data tier preference of snapshot blob cache index to `data_content,data_hot` (#72817)
.snapshot-blob-cache index is created today with a data tier 
preference set to data_cold,data_warm,data_hot. This does 
not works well with autoscaling and clusters that only have 
a hot and a frozen tier: in such cases autoscaling adds a 
cold tier just to host this system index.

This commit changes the tier preference to data_content,data_hot.
2021-05-07 09:18:34 +02:00
Rene Groeschke e609e07cfe
Remove internal build logic from public build tool plugins (#72470)
Extract usage of internal API from TestClustersPlugin and PluginBuildPlugin and related plugins and build logic

This includes a refactoring of ElasticsearchDistribution to handle types
better in a way we can differentiate between supported Elasticsearch
Distribution types supported in TestCkustersPlugin and types only supported
in internal plugins.

It also introduces a set of internal versions of public plugins.

As part of this we also generate the plugin descriptors now.

As a follow up on this we can actually move these public used classes into 
an extra project (declared as included build)

We keep LoggedExec and VersionProperties effectively public And workaround for RestTestBase
2021-05-06 14:02:35 +02:00
Dan Hermann edf2f7d4d6
Unmute IndexingIT and TokenBackwardsCompatibilityIT tests (#72307) 2021-04-30 07:39:55 -05:00
Rene Groeschke 17da1038fe
Port gradle script plugins to precompiled script plugins (#72361)
Script plugins cannot apply plugins and therefore wont work with porting
buildSrc to an included build as we plan. Therefore we take advantage
of moving our script plugins into precompiled script plugins.

As a limitation of this we ran into problems applying binary plugins
from script plugins and for now moved this out of those scripts.
2021-04-29 09:33:28 +02:00
Rene Groeschke 5bcd02cb4d
Restructure build tools java packages (#72030)
Related to #71593 we move all build logic that is for elasticsearch build only into
the org.elasticsearch.gradle.internal* packages

This makes it clearer if build logic is considered to be used by external projects
Ultimately we want to only expose TestCluster and PluginBuildPlugin logic
to third party plugin authors.

This is a very first step towards that direction.
2021-04-26 14:53:55 +02:00
Julie Tibshirani 2d3f171a79 Mute TokenBackwardsCompatibilityIT while it awaits fix. 2021-04-21 11:50:27 -07:00
Henning Andersen 3c7c0c62f3
Move experimental frozen to frozen shard limit (#71781)
Frozen indices created on 7.12 would not belong to the frozen shard
limit group, now we convert them when last node is upgraded.

Relates #71392
2021-04-20 18:59:17 +02:00
Przemysław Witek 749120ed5b
[Transform] Add telemetry support for transform features (#71607) 2021-04-19 14:38:50 +02:00
Henning Andersen 6ad769548a
Mute testBlobStoreCacheWithPartialCopyInMixed... (#71782) 2021-04-16 21:34:40 +02:00
Lyudmila Fokina 3b0b7941ae
Warn users if security is implicitly disabled (#70114)
* Warn users if security is implicitly disabled

Elasticsearch has security features implicitly disabled by default for
Basic and Trial licenses, unless explicitly set in the configuration
file.
This may be good for onboarding, but it also lead to unintended insecure
 clusters.
 This change introduces clear warnings when security features are
 implicitly disabled.
 - a warning header in each REST response if security is implicitly
 disabled;
 - a log message during cluster boot.
2021-04-13 18:33:41 +02:00
Yang Wang c1257afaf6
Update version constants for API key metadata (#71603)
This PR updates the version constants for API key metadata to be 7.13.0
since #70956 is backported.
2021-04-13 10:53:49 +10:00
Martijn van Groningen a6c5c1c85c
Tweak DataStreamsUpgradeIT test (#71452)
Check cluster prior to assert doc count in DataStreamsUpgradeIT#testDataStreamValidationDoesNotBreakUpgrade test.
This to avoid node connect issues during the search executions.

Closes #71329
2021-04-08 14:28:47 +02:00
Dan Hermann 2c6ba92d46
Improve data stream rollover and simplify cluster metadata validation for data streams (#70934) 2021-03-29 07:36:44 -05:00
Yang Wang 3725cb53da
Support metadata on API keys (#70292)
This PR adds metadata support for API keys. Metadata are of type 
Map<String, Object> and can be optionally provided at API key creation time.
It is returned as part of GetApiKey response. It is also stored as part of 
the authentication object to transfer throw the wire.
Note that it is not yet searchable and not exposed to any ingest processors.
They will be handled by separate PRs.
2021-03-29 09:23:30 +11:00
Mark Vieira 6339691fe3
Consolidate REST API specifications and publish under Apache 2.0 license (#70036) 2021-03-26 16:20:14 -07:00
Dan Hermann 7c3ebe220f
Remove obsolete BWC checks for data streams (#70777) 2021-03-24 07:22:40 -05:00
Mark Vieira 21e1ce5377
Use single-quoted string so Windows file paths don't make YAML mad (#69946) 2021-03-04 07:30:27 -08:00
David Turner 1a432a26d1
Only allocate partial shards to nodes with cache (#69904)
This commit introduces the `HasFrozenCacheAllocationDecider` which
restricts allocation of partial shards only to those nodes with a
configured shared cache, and also the `FrozenCacheInfoService` which
tracks which nodes do/don't have a frozen cache configured.
2021-03-04 12:23:25 +00:00
David Turner 1fcc2cb21e
Extend socket timeout in AbstractUpgradeTestCase (#69800)
Some test suites underneath `AbstractUpgradeTestCase` have a 70s timeout
when waiting for green health, which is longer than the usual socket
timeout of 60s, so a failure yields no useful information. The
longer-than-usual timeout is to allow time for delayed allocation.

This commit extends the timeout to 90s in line with other similar test
cases.

Relates #69704
2021-03-04 09:48:48 +00:00
Hendrik Muhs 36f3ea96eb
[Transform] improve tracing for TransformSurvivesUpgradeIT (#69854)
report number of deleted state docs in order to help tracing issues

relates #68646
2021-03-04 08:13:24 +01:00
Joe Gallo be17497eee
Fix TransformSurvivesUpgradeIT (#69816) 2021-03-03 09:48:59 -05:00
Martijn van Groningen 7b668b4506
Move test for upgrade bug (#69625) to a separate test. (#69791) 2021-03-02 15:52:22 +01:00
Tanguy Leroux 1fe81e249a
Fix SearchableSnapshotsRollingUpgradeIT (#69798)
closes #69705
2021-03-02 14:34:49 +01:00
Tanguy Leroux 2ed8e16c4c
Reenable SearchableSnapshotsRollingUpgradeIT (#69788)
The test SearchableSnapshotsRollingUpgradeIT caused CI failures 
at least two times (#69705) since it was merged into master. After 
looking at all node logs, I wasn't able to determine precisely the 
cause of the test suite timeout.

After looking at the code I think it is preferable to use a dedicated 
path.repo for this test so that it does not interfere with existing 
and future tests that could wipe the repositories and snapshots 
of the default path.repo at any time. I also fixed an issue with 
the storage paramater that should only work starting 7.12.0. 
Finally, #69704 has been merged and it should provide more 
nformation if this test fails again.

Closes #69705
2021-03-02 12:33:22 +01:00
Tanguy Leroux 86dbc01596
Mute SearchableSnapshotsRollingUpgradeIT (#69706)
Relates #69705
2021-03-01 15:39:53 +01:00
Tanguy Leroux 5c2b15af72
Adjust the length of blob cache docs for Lucene metadata files (#69431)
Today searchable snapshots IndexInput implementations use the 
blob store cache to cache the first 4096 bytes of every Lucene files. 
After some experiments we think that we could adjust the length of 
the cached data depending of the Lucene file that is read, caching 
up to 64KB for Lucene metadata files (ie files that are fully read 
when a Directory is opened) and only 1KB for other files.

The files that are cached up to 64KB are the following extensions:

        "cfe", // compound file's entry table
        "dvm", // doc values metadata file
        "fdm", // stored fields metadata file
        "fnm", // field names metadata file
        "kdm", // Lucene 8.6 point format metadata file
        "nvm", // norms metadata file
        "tmd", // Lucene 8.6 terms metadata file
        "tvm", // terms vectors metadata file
        "vem"  // Lucene 9.0 indexed vectors metadata

The 64KB limit can be configured on a per index basis through a new 
index setting. This change is extracted from #69283 and does not 
address the caching of CFS files.
2021-03-01 11:28:43 +01:00
Martijn van Groningen 2f49109175
Don't mark backing indices of overlapping data streams as conflicts. (#69666)
Forward port #69625 to master.

Today when upgrading from 7.9.x or 7.10.x version to 7.11.x or later and
if two data (or more) data streams exist that have a overlapping prefix and
one data stream name ends with the a date suffix that matches with backing index
date pattern (uuuu.MM.dd) then new upgraded nodes may refuse to join. Essentially
preventing upgrade of the cluster to continue.

In this case the validation logic in `Metadata#validateDataStreams(...)` confuses
backing indices of one data stream as regular indices and thinks these indices
collide with another data stream.

In this validation only incorrectly fails if {data-stream-name} and
{data-steam-name}-{uuuu.MM.dd} name exist and later has been rolled
over more than the former and then upgrade cluster to 7.11+.

A 7.10.2 cluster with:

Data stream 1: logs-foobar
Backing indices: logs-foobar-000001

Data stream 2: logs-foobar-2021.01.13
Backing indices: logs-foobar-2021.01.13-000001, logs-foobar-2021.01.13-000002

When upgrading, then the new node will not join, because it thinks that
'logs-foobar-2021.01.13-000002' index collides with the backing index space
of data stream 'logs-foobar'.

This change tries to address this.
2021-03-01 09:15:36 +01:00
Dimitris Athanasiou 125655898e
[ML] Adjust versions and unmute DFA bwc tests (#69294)
... after backporting #69183
2021-02-20 14:29:52 +02:00
Dimitris Athanasiou b4cbd5b1d9
[ML] Mute data frame analytics BWC tests (#69275)
Until #69183 is backported to 7.x
2021-02-19 16:44:18 +02:00
Hendrik Muhs 967505c3d5
[Transform] add retention policy to documentation IT (#68728)
add retention policy and sync to documentation IT, add builders for sync
and retention policy

relates #67832
2021-02-11 15:59:58 +01:00
Hendrik Muhs 54ed2e37d9
[Transform] implement retention policy to delete data from a transform (#67832)
add a retention policy to transform to delete data that is considered outdated as part of a
transform checkpoint.

fixes #67916
2021-02-08 15:06:15 +01:00
David Roberts cc59612dd7
[Transform] Automatic management for transform system indices (#68590)
The transform internal index now uses the special
functionality for protecting system indices from
unwanted modification.

The system index descriptor replaces the index
template that was previously used with the transform
internal index.  In places where we used to check
that the template was installed we now proactively
create the index.  Master node actions will apply
the desired mappings if the index is auto-created
as a result of any indexing that they do.
2021-02-08 09:10:14 +00:00
David Roberts 5f5968b705
[ML] Automatic management for ML system indices (#68044)
The ML system indices now use the special functionality for
applying the correct mappings on first use. This replaces
the index templates that used to do this job, but were
vulnerable to tampering.

A number of other changes have had to be made to utilise
the system index functionality:

1. All fields previously missed out of mappings have been
   added to the system index mappings, with the types that
   would have been assigned dynamically in previous
   versions.  This is necessary because dynamic mappings
   updates are banned for system indices, yet some of our
   mappings allow dynamic updates.
2. As a result of the contradiction regarding dynamic
   mappings, we are now very well protected against failing
   to add new fields to the mappings for those indices that
   exhibit the contradiction (which are .ml-config and
   .ml-meta).  This means their mappings don't need to be
   explicitly compared to expected mappings in upgrade
   tests now.  Instead, any usage of a new field during or
   after upgrade will trigger an error in any test this occurs
   in.
3. Reserved fields for the config index were unnecessary
   (only used by tests) and just added extra complication,
   so they have been removed.  We have the concept of
   reserved fields for our results indices because end user
   fields get added to results and we need to ensure they
   don't clash with fields we want to use ourselves.  This
   problem does not exist for the config index.
2021-02-05 10:15:37 +00:00
Mark Vieira a92a647b9f Update sources with new SSPL+Elastic-2.0 license headers
As per the new licensing change for Elasticsearch and Kibana this commit
moves existing Apache 2.0 licensed source code to the new dual license
SSPL+Elastic license 2.0. In addition, existing x-pack code now uses
the new version 2.0 of the Elastic license. Full changes include:

 - Updating LICENSE and NOTICE files throughout the code base, as well
   as those packaged in our published artifacts
 - Update IDE integration to now use the new license header on newly
   created source files
 - Remove references to the "OSS" distribution from our documentation
 - Update build time verification checks to no longer allow Apache 2.0
   license header in Elasticsearch source code
 - Replace all existing Apache 2.0 license headers for non-xpack code
   with updated header (vendored code with Apache 2.0 headers obviously
   remains the same).
 - Replace all Elastic license 1.0 headers with new 2.0 header in xpack.
2021-02-02 16:10:53 -08:00
Valeriy Khakhutskyy 0743a8de71
[ML] Unmute BWC tests (#68273)
As a follow-up on #68099, #68269 and #68271, this PR removes the version checks and unmutes the BWC tests.
2021-02-01 16:11:11 +01:00
Valeriy Khakhutskyy fd68e7f130
[ML] Mute data frame analytics BWC tests (#68269)
Until #68099 is backported to 7.x.
2021-02-01 12:08:45 +01:00
Dimitris Athanasiou 3950cc95fa
[ML] Adjust versions and unmute DFA bwc tests (#67998)
Following backports of #67937 and #67950 in 7.x
2021-01-26 19:15:12 +02:00
Dimitris Athanasiou 875e233422
[ML] Mute data frame analytics BWC tests (#67970)
Until #67937 and #67950 are backported to 7.x
2021-01-26 13:10:54 +02:00
Benjamin Trent aec5d10881
[ML] fixing MlJobSnapshotUpgradeIT.testSnapshotUpgrader test (#67090)
* [ML] fixing MlJobSnapshotUpgradeIT.testSnapshotUpgrader test
fixes a minor typo that causes test flakiness.

closes #67059
2021-01-06 13:00:09 -05:00