Commit Graph

34 Commits

Author SHA1 Message Date
Rene Groeschke d8e4e48a3b
Avoid creating unused test tasks (#74644)
With the overall theme of trying to configure and add less to the build instead of just disabling it later,
we're replacing standalone-test by standalone-rest tasks avoids creating the
unused test tasks.

Standalone rest test plugin and the other rest test plugins behave a little bit different in the sense how source sets and test tasks are wired.

The standalone rest test plugin assumes that all RestTestTasks are using the same sourceSet (test). The yaml, java Rest test plugins use one dedicated sourceSet per test task.

In the long run we probably will migrate standalone-rest-test usages to one of the other plugins and deprecate standalone-rest-test
2021-06-30 14:11:25 +02:00
Armin Braun eca57352b5
Remove Outdated GetSnapshots BwC Handling in Rest Tests (#74734)
The group-by-repository response format is gone as a result of #74451
=> cleaning up its last remnants from REST tests.
2021-06-30 13:12:39 +02:00
Rene Groeschke 2d42c26249
Keep StandaloneRestIntegTest task public (#72865)
This is used by hadoop
2021-05-09 11:36:16 +02:00
Rene Groeschke 78182686fc
Remove more usage of interal logic in public build api (#72858) 2021-05-08 23:06:23 +02:00
Rene Groeschke e609e07cfe
Remove internal build logic from public build tool plugins (#72470)
Extract usage of internal API from TestClustersPlugin and PluginBuildPlugin and related plugins and build logic

This includes a refactoring of ElasticsearchDistribution to handle types
better in a way we can differentiate between supported Elasticsearch
Distribution types supported in TestCkustersPlugin and types only supported
in internal plugins.

It also introduces a set of internal versions of public plugins.

As part of this we also generate the plugin descriptors now.

As a follow up on this we can actually move these public used classes into 
an extra project (declared as included build)

We keep LoggedExec and VersionProperties effectively public And workaround for RestTestBase
2021-05-06 14:02:35 +02:00
Rene Groeschke 17da1038fe
Port gradle script plugins to precompiled script plugins (#72361)
Script plugins cannot apply plugins and therefore wont work with porting
buildSrc to an included build as we plan. Therefore we take advantage
of moving our script plugins into precompiled script plugins.

As a limitation of this we ran into problems applying binary plugins
from script plugins and for now moved this out of those scripts.
2021-04-29 09:33:28 +02:00
Armin Braun 2edbdbb6ce
Fix MultiVersionRepositoryAccessIT to Account for Plugins (#71809)
This test broken when the geoip processor created its index before we take the snapshot
as that would cause an unexpected number of shards in the restore. Rather than excluding
global state from the snapshot (the internal index of the plugin is snapshotted as part
of global state in 7.12+ so the index filtering we use doesn't apply to it) I opted to fix this
by making the restore selective instead to keep coverage of the global state.

closes #71763
2021-04-26 17:15:07 +02:00
Rene Groeschke 5bcd02cb4d
Restructure build tools java packages (#72030)
Related to #71593 we move all build logic that is for elasticsearch build only into
the org.elasticsearch.gradle.internal* packages

This makes it clearer if build logic is considered to be used by external projects
Ultimately we want to only expose TestCluster and PluginBuildPlugin logic
to third party plugin authors.

This is a very first step towards that direction.
2021-04-26 14:53:55 +02:00
Lyudmila Fokina 3b0b7941ae
Warn users if security is implicitly disabled (#70114)
* Warn users if security is implicitly disabled

Elasticsearch has security features implicitly disabled by default for
Basic and Trial licenses, unless explicitly set in the configuration
file.
This may be good for onboarding, but it also lead to unintended insecure
 clusters.
 This change introduces clear warnings when security features are
 implicitly disabled.
 - a warning header in each REST response if security is implicitly
 disabled;
 - a log message during cluster boot.
2021-04-13 18:33:41 +02:00
Armin Braun c86d9e3cf6
Simplify BwC for UUIDs in RepositoryData (#69335)
We don't need to separately handle BwC for these two fields since they
were both introduced in `7.12`.
2021-02-22 15:42:09 +01:00
Rene Groeschke bdf229a148
Introduce Internal Test Artifact Plugin (#68766)
This reduces the ceremony declaring test artifacts for a project.
It also solves an issue with usage of deprecated testRuntime that
testArtifacts extendsFrom which seems not required at all and would have
broke with Gradle 7.0 anyhow

Test artifact resolution is now variant aware which allows us a more adequate 
compile and runtime classpath for the consuming projects.

We also Introduce a convention method in the elasticsearch build to declare 
test artifact dependencies in an easy way close to how its done by the gradle build in 
test fixture plugin.

Furthermore we cleaned up some inconsistent test dependencies declarations when 
relying on a project and on its test artifacts
2021-02-16 14:36:17 +01:00
Mark Vieira a92a647b9f Update sources with new SSPL+Elastic-2.0 license headers
As per the new licensing change for Elasticsearch and Kibana this commit
moves existing Apache 2.0 licensed source code to the new dual license
SSPL+Elastic license 2.0. In addition, existing x-pack code now uses
the new version 2.0 of the Elastic license. Full changes include:

 - Updating LICENSE and NOTICE files throughout the code base, as well
   as those packaged in our published artifacts
 - Update IDE integration to now use the new license header on newly
   created source files
 - Remove references to the "OSS" distribution from our documentation
 - Update build time verification checks to no longer allow Apache 2.0
   license header in Elasticsearch source code
 - Replace all existing Apache 2.0 license headers for non-xpack code
   with updated header (vendored code with Apache 2.0 headers obviously
   remains the same).
 - Replace all Elastic license 1.0 headers with new 2.0 header in xpack.
2021-02-02 16:10:53 -08:00
David Turner dd519a9eba
Introduce string constant for readonly setting (#68291)
A blob store repository can be put in readonly mode by setting
`readonly: true` in its settings. In the codebase the setting key is
just the literal string `"readonly"` wherever it's used and it takes
some effort to determine what the right setting name is, in particular
to check each time that it's not spelled `"read_only"`.

This commit replaces those literal `"readonly"` strings with the
`BlobStoreRepository#READONLY_SETTING_KEY` constant to reduce this
trappiness.
2021-02-01 15:43:37 +00:00
Armin Braun 77162071f5
Add ClusterUUID to RepositoryData (#68002)
Record the clusterUUID of the last cluster to write
to a repository in the `RepositoryData` and use it for more
meaningful logging when running into a concurrent modification
issue.
2021-01-28 12:38:15 +01:00
David Turner 06e141888f
Reinstate BWC snapshot tests (#67938)
This commit mostly reverts #67934, except for the change to the version
constant `REPOSITORY_UUID_IN_REPO_DATA_VERSION`.

Completes the backport of #67829 via #67899
2021-01-25 18:36:12 +00:00
David Turner faed3e7199
Temporarily suppress BWC snapshot tests (#67934)
This commit suppresses any BWC tests related to snapshots in `master` so
that #67899 can be merged to `7.x`. It will mostly be reverted after the
merge of #67899 is complete.

Relates #66431
2021-01-25 17:48:47 +00:00
David Turner e5a15d4fcb
Introduce repository UUIDs (#67829)
Today a snapshot repository does not have a well-defined identity. It
can be reregistered with a different cluster under a different name, and
can even be registered with multiple clusters in readonly mode.

This presents problems for cases where we need to refer to a specific
snapshot in a globally-unique fashion. Today we rely on the repository
being registered under the same name on every cluster, but this is not a
safe assumption.

This commit adds a UUID that can be used to uniquely identify a
repository. The UUID is stored in the top-level index blob, represented
by `RepositoryData`, and is also usually copied into the
`RepositoryMetadata` that represents the repository in the cluster
state. The repository UUID is exposed in the get-repositories API; other
more meaningful consumers will be added in due course.
2021-01-25 12:17:52 +00:00
Rene Groeschke 810e7ff6b0
Move tasks in build scripts to task avoidance api (#64046)
- Some trivial cleanup on build scripts
- Change task referencing in build scripts to use task avoidance api
where replacement is trivial.
2020-11-12 12:04:15 +01:00
Rene Groeschke aa5d3159de
Use services for archive and file operations in tasks (#62968)
Referencing a project instance during task execution is discouraged by
Gradle and should be avoided. E.g. It is incompatible with Gradles
incubating configuration cache. Instead there are services available to handle
archive and filesystem operations in task actions.

Brings us one step closer to #57918
2020-10-01 10:47:41 +02:00
Rene Groeschke dd74be0f83
Merge test runner task into RestIntegTest (#60261)
* Merge test runner task into RestIntegTest
* Reorganizing Standalone runner and RestIntegTest task
* Rework general test task configuration and extension
2020-08-03 12:07:41 +02:00
Rene Groeschke a72760e55b
Replace immediate task creations by using task avoidance api (#60071)
- Replace immediate task creations by using task avoidance api
- One step closer to #56610
- Still many tasks are created during configuration phase. Tackled in separate steps
2020-07-31 11:29:14 +02:00
Rene Groeschke 680ea07f7f
Remove deprecated usage of testCompile configuration (#57921)
* Remove usage of deprecated testCompile configuration
* Replace testCompile usage by testImplementation
* Make testImplementation non transitive by default (as we did for testCompile)
* Update CONTRIBUTING about using testImplementation for test dependencies
* Fail on testCompile configuration usage
2020-06-12 13:34:53 +02:00
Armin Braun 37ab35156b
Deduplicate Index Metadata in BlobStore (#50278)
This PR introduces two new fields in to `RepositoryData` (index-N) to track the blob name of `IndexMetaData` blobs and their content via setting generations and uuids. This is used to deduplicate the `IndexMetaData` blobs (`meta-{uuid}.dat` in the indices folders under `/indices` so that new metadata for an index is only written to the repository during a snapshot if that same metadata can't be found in another snapshot.
This saves one write per index in the common case of unchanged metadata thus saving cost and making snapshot finalization drastically faster if many indices are being snapshotted at the same time.

The implementation is mostly analogous to that for shard generations in #46250 and piggy backs on the BwC mechanism introduced in that PR (which means this PR needs adjustments if it doesn't go into `7.6`). 

Relates to #45736 as it improves the efficiency of snapshotting unchanged indices
Relates to #49800 as it has the potential of loading the index metadata for multiple snapshots of the same index concurrently much more efficient speeding up future concurrent snapshot delete
2020-06-05 19:16:41 +02:00
Armin Braun a86fceebe6
Manually Craft CreateSnapshotRequest to fix BwC Test (#57661)
* Manually Craft CreateSnapshotRequest to fix BwC Test

We can't use the high level create snapshot request any longer
since we changed some of its default parameters in `8` and those
are not understood by older versions like `7.4`.

Closes #57650
2020-06-05 11:16:08 +02:00
Mark Vieira c9eed19f4c
Remove unused import 2020-06-04 11:52:26 -07:00
Mark Vieira 9ebfbaf6e0
Mute MultiVersionRepositoryAccessIT 2020-06-04 11:43:34 -07:00
Rene Groeschke fb86184dbb
Move bwcVersions extension property to BuildParams (#56206)
- resolved in GlobalBuildInfoPlugin
- propagate exception if version file cannot be resolved
- add null check when setting bwcVersions
2020-05-07 12:08:17 +02:00
Mark Vieira 33fc8e7ebf
Consolidate duplication of BWC testing task setup in script plugin (#53079) 2020-03-03 14:40:55 -08:00
Armin Braun f06f9fa5e4
Refactor Inflexible Snapshot Repository BwC (#52365)
Transport the version to use for  a snapshot instead of whether to use shard generations in the snapshots in progress entry. This allows making upcoming repository metadata changes in a flexible manner in an analogous way to how we handle serialization BwC elsewhere.
Also, exposing the version at the repository API level will make it easier to do BwC relevant changes in derived repositories like source only or encrypted.
2020-02-17 13:07:59 +01:00
Mark Vieira e8562a4e11
Always use bundled JDK for external cluster nodes when BWC testing (#51505) 2020-01-27 22:31:28 -08:00
Armin Braun 3eda5303d0
Fix X-Pack Indices Breaking Repository BwC Tests (#51120)
If some internal `.watcher` or so index gets created during these tests
then the shard counts on snapshot restores and creates won't match up with expectations.
Fixed by only creating the snapshot for the test index

Closes #50819
2020-01-16 20:38:19 +01:00
Armin Braun c30e05e5ab
Prevent Old Version Clusters From Corrupting Snapshot Repositories (#50853)
Follow up to #50692 that starts writing a `min_version` field to
the `RepositoryData` so that pre-7.6 ES versions can not read it
(and potentially corrupt it if they attempt to modify the repo contents)
after the repository moved to the new metadata format.
2020-01-13 13:32:22 +01:00
Armin Braun 15944c2a0d
Fix MultiVersionRepositoryAccessIT Tests (#50848)
We need the same fix we did in `7.x` (#50797) and only get snapshot status
for the current version or older. Otherwise these tests break for
e.g.`7.0.1` due to the same index metadata incompatibility.

Closes #50819
2020-01-10 13:01:28 +01:00
Armin Braun ee6fbcc4a7
Fix Snapshot Repository Corruption in Downgrade Scenarios (#50692)
This PR introduces test infrastructure for downgrading a cluster while interacting with a given repository.
It fixes the fact that repository metadata in the new format could be written while there's still older snapshots in the repository that require the old-format metadata to be restorable.
2020-01-09 15:31:21 +01:00