Commit Graph

281 Commits

Author SHA1 Message Date
Mayya Sharipova 26c3dd6857
Upgrade to lucene-9.1.0-snapshot-1336263051c (#83667)
Lucene issues that resulted in elasticsearch changes:

LUCENE-9820 Separate logic for reading the BKD index from logic to intersecting it.
LUCENE-10377: Replace 'sortPos' with 'enableSkipping' in SortField.getComparator()
LUCENE-10301: make the test-framework a proper module by moving all test
classes to org.apache.lucene.tests
LUCENE-10300: rewrite how resources are read in ukrainian morfologik analyzer:
LUCENE-10054 Make HnswGraph hierarchical
2022-02-22 09:53:20 +01:00
Mark Vieira 64929dc5df
Introduce explicit API for configure test cluster feature flags (#83876) 2022-02-14 15:22:33 -08:00
Francisco Fernández Castaño 3d4c3d2e33
Wait for relocations during the first mixed round in SnapshotBasedRecoveryIT (#81384)
This commits adds additional logging in order to debug the test failures

Closes #80939
2022-02-01 13:48:53 +01:00
Mark Vieira e6ba67f973
Limit full cluster restart upgrade tests to wire compatible versions (#82445) 2022-01-11 14:27:56 -08:00
James Rodewig b4c2b79c74
Mute testSnapshotBasedRecovery (#82436)
See #80939
2022-01-11 11:20:52 -05:00
Artem Prigoda 0699c9351f
Use Java 14 switch expressions (#82178)
JEP 361[https://openjdk.java.net/jeps/361] added support for switch expressions
which can be much more terse and less error-prone than switch statements.

Another useful feature of switch expressions is exhaustiveness: we can make
sure that an enum switch expression covers all the cases at compile time.
2022-01-10 09:53:35 +01:00
Artem Prigoda 763d6d510f
Use Java 15 text blocks for JSON and multiline strings (#80751)
The ES code base is quite JSON heavy. It uses a lot of multi-line JSON requests in tests which need to be escaped and concatenated which in turn makes them hard to read. Let's try to leverage Java 15 text blocks for representing them.
2021-12-15 18:01:28 +01:00
Nik Everett 5ce8b8ca99
Move MapMatcher into ES (#81593)
This donates the `MapMatcher` project to Elasticsearch. We want it in
our code base so we can more easilly maintain it and don't have to wait
for a release. It really isn't used outside of Elasticsearch so it isn't
really useful to keep it separate. We considered removing it entirely it
but still think that it's mostly and improvement on
`NotEqualMessageBuilder` and the build in `equalTo`.
2021-12-09 12:37:55 -05:00
weizijun 2f855aac88
TSDB: Make time boundaries settings required in tsdb indices (#81146)
Make `index.time_series.start_time` and `index.time_series.end_time` settings as required settings in tsdb indices.

It will change many tests, among which a time_series index is created, and it will add the `index.time_series.start_time` and `index.time_series.end_time` settings
2021-12-01 17:53:49 -05:00
Christos Soulios c0b4b6080d
TSDB: Add `_tsid` field to time_series indices (#80276)
This PR adds support for a field named _tsid that uniquely identifies the time series a document belongs to.

When a document is indexed in a time series index (IndexMode.TIME_SERIES), _tsid field is generated from the values of all dimension fields.
2021-11-29 12:44:28 +02:00
Mark Vieira fe15874352
Don't pass feature flag to non-snapshot artifacts (#80033)
When configuring BWC tests to use feature flags, we need to ensure we
don't inadvertently enable the flag on the old version nodes, as those
are NOT release builds and therefore fail if the feature flag is
present. This commit tweaks our config here to ensure we explicitly
enable the flag only on the upgraded nodes. Closes #78219
2021-11-08 20:04:27 -05:00
Przemyslaw Gomulka f53776389f
Enable bwc IndexingIT after backport (#80068)
testDateNanosFormatUpgrade was disable until
a revert #80003 was backported to 8.0
2021-10-29 11:21:18 +02:00
Przemyslaw Gomulka 7aaa2344e9
Revert "Deprecate resolution loss on date field (#78921)" (#79914)
This reverts commit d50c9e8.

relates #78921

reason https://github.com/elastic/elasticsearch/issues/37962#issuecomment-953700245
2021-10-28 12:05:35 +02:00
Mark Vieira 12ad399c48 Reformat Elasticsearch source 2021-10-27 08:19:51 -07:00
Gordon Brown 1a906897e6
Implement framework for migrating system indices (#78951)
This PR adds a framework for migrating system indices as necessary prior
to Elasticsearch upgrades. This framework uses REST APIs added in
another commit:
- GET _migration/system_features

This API, which gets the status of "features" (plugins which own system
indices) with regards to whether they need to be upgraded or not. As of
this PR, this API also reports errors encountered while migrating system
indices alongside the index that was being processed when this occurred.

As an example of this error reporting:

```json
{
    "feature_name": "logstash_management",
    "minimum_index_version": "8.0.0",
    "upgrade_status": "ERROR",
    "indices": [
        {
            "index": ".logstash",
            "version": "8.0.0",
            "failure_cause": {
                "error": {
                    "root_cause": [
                        {
                            "type": "runtime_exception",
                            "reason": "whoopsie",
                            "stack_trace": "<omitted for brevity>"
                        }
                    ],
                    "type": "runtime_exception",
                    "reason": "whoopsie",
                    "stack_trace": "<omitted for brevity>"
                }
            }
        }
    ]
}
```

- POST _migration/system_features

This API starts the migration process. The API for this has no changes,
but when called, any system indices which need to be migrated will be
migrated, with status information stored in the cluster state for later
use by the GET _migration/system_features API.
2021-10-19 18:52:22 -06:00
William Brafford 5c6a481cb7
Use 'migration' instead of 'upgrade' in GET system feature migration status responses (#79302)
* Change 'upgrade' to 'migration' in System Index Migration responses

* Mute rest compatibility test
2021-10-19 12:15:26 -04:00
Przemyslaw Gomulka d50c9e86ec
Deprecate resolution loss on date field (#78921)
When storing nanoseconds on a date field the nanosecond part is lost as
it cannot be stored. The date_nanos field should be used instead.
This commit emits a deprecation warning to notify users about this.

closes #37962
2021-10-18 14:38:36 +02:00
Nik Everett b6c61f4051
Route documents to the correct shards in tsdb (#77731)
* Route documents to the correct shards in tsdb

This causes elasticsearch to land documents from the same time series on
the same shard. It does so by adding a new index setting `routing_path`
which must be set when an index is in `mode: time_series` and may not be
set outside of that mode. That setting contains a list of patterns to
extract from the `_source` document that are hashed into the routing
value.

* Moar skip

* tsdb survives full cluster restart

* Remove auto generated id rejection

We do want to reject these documents but let's sae that for a follow up
change.

* Simplify

* Forbid routing_required

* Small

* Fork fork knife

* Let failures flow

* Fix full cluster

* Always fork

* Retry?

* Remove pressure test arm

* Add missing settings

* WIP

* Remove leftover

* More cleaning

* Fixup more tests

* Remove old skip

* New tests for Retry

* More tests

* Revert unrelated

* One dispatch please

* Stuff moved

* More moving

* Explain why fork

* Back to ActionRunnable

* Update comment

* Utility method

* Move routing_path under feature flag

* Imports
2021-10-15 17:16:48 -04:00
Chris Hegarty 20c9f756d2
Fix split package org.elasticsearch.common.xcontent (#78831)
Fix the split package org.elasticsearch.common.xcontent, between server and the x-content lib. Move the x-content lib exported package from org.elasticsearch.common.xcontent to org.elasticsearch.xcontent ( following the naming convention of similar libraries ). Removing split packages is a prerequisite to modularization.
2021-10-08 17:14:26 +01:00
William Brafford 0a28c7cb91
Implement GET API for System Feature Upgrades (#78642)
* Implement and test get feature upgrade status API
* Add integration test for feature upgrade endpoint
* Use constant enum for statuses
* Add unit tests for transport class methods
2021-10-07 15:18:47 -04:00
Rene Groeschke e684d5ef63
Introduce DSL for configuring BWC tests (#78597)
this makes configuring bwc Tests less errorprone due to strange groovy behaviour
2021-10-04 21:43:02 +02:00
Nik Everett 5de59aa988
TSDB: Add feature flag to more builds (#78619)
When we run the build in non-snapshot mode we don't enable feature flags
by default. This would cause the tests for tsdb to fail. So we enable
the tsdb feature flag in every build that needs it. There were a few
builds that needed it that didn't have it. This adds it to those builds.

Closes #78443
Closes #78598
2021-10-04 10:56:23 -04:00
Nik Everett 37a23d7ca4
TSDB: Basic rolling upgrade test (#78028)
Adds an index in time_series mode to the rolling upgrade tests.
2021-10-01 16:17:12 -04:00
Francisco Fernández Castaño 2fdb5a8b7e
Fix SnapshotBasedRecoveryIT#testSnapshotBasedRecovery (#77134)
Move the shard to a replica in an older version when the primary
is located in the upgraded node during the first rolling restart
round.

Closes #76595
2021-09-29 11:26:11 +02:00
Rene Groeschke 6ef13abe81
Do not create unused testCluster (#77581)
* Do not create unused testCluster

This avoids creating test clusters that are not required during the build.
We use lazy configuration here on testClusters and only instantiate them as theyre

* Do not fail on run task (debug)

* Create more test cluster lazy

* Make more test cluster lazy

* Avoid creating unused testcluster

* Fix PluginBuildPlugin

* Fix disabling geo db download

* Fix cluster setup in repository-multi-version

* Polishing

* Fix issue with irretic groovy ogic

* Fix bwc tests

* Fix more bwcTests

* Fix more bwc tests

* Fix more bwc tests

* Fix more bwc tests

* Fix typo

* Minor polishing

* Fix rolling upgrade tests

* Fix cluster config in sql qa mixedcluster project

* Fix more bwc tests

* Clean up before review

* Document test cluster usage

* Api polising after Review

provide useCluster(Provider) method to TestClusterAware

Ideally we take this a step further and realize those test clusters only on use.
But out of scope of this PR.

* Allow gradle provider as value for nonSystemProperties

* Some simplification on test configuration

* Fix typo in rest test config

* Fix more typos

* Fix another typo

* Fix more typos
2021-09-23 03:45:59 -04:00
Henning Andersen 2bfeab6165
Mute recover from snapshot rolling first round (#76601)
Selectively muting parts of the rolling upgrade test for recover from
snapshot.

Relates #76595
2021-08-17 13:51:37 +02:00
Francisco Fernández Castaño 2ebe5cd075
Add peer recoveries using snapshot files when possible (#76237)
This commit adds peer recoveries from snapshots. It allows establishing a replica by downloading file data from a snapshot rather than transferring the data from the primary. 

Enabling this feature is done on the repository definition. Repositories having the setting `use_for_peer_recovery=true` will be consulted to find a good snapshot when recovering a shard.

Relates #73496
2021-08-13 10:42:16 +02:00
Nik Everett a6fe766b7b
Give rollling upgrade tests more information (#76360)
* Give rollling upgrade tests more information

This passes the version that rolling upgrade tests are coming *from*
into the actual tests so they can reason about it. This is useful
because we have features that are not supported on early versions and
we want to write rolling upgrade tests for them. We can't run those
features or assert anything about them in when they don't exist. You'd
think we could use the minimum version of a node in the cluster, but
that only works in the unupgraded phases - once we've completed the
upgrade we need to have the version that we came from to know what we
did in the mixed version.

* Nope
2021-08-12 09:20:51 -04:00
Przemyslaw Gomulka c96139d006
[Rest Api Compatibility] Deprecate the use of synced flush (#75372)
synced flush is going to be replaced by flush. This commit allows to synced_flush api only in v7 compatibility mode.
Worth noting - sync_id is gone and won't be available in v7 responses from indices.stats

relates removal pr #50882
relates #51816
2021-07-28 14:17:49 +02:00
Rene Groeschke d8e4e48a3b
Avoid creating unused test tasks (#74644)
With the overall theme of trying to configure and add less to the build instead of just disabling it later,
we're replacing standalone-test by standalone-rest tasks avoids creating the
unused test tasks.

Standalone rest test plugin and the other rest test plugins behave a little bit different in the sense how source sets and test tasks are wired.

The standalone rest test plugin assumes that all RestTestTasks are using the same sourceSet (test). The yaml, java Rest test plugins use one dedicated sourceSet per test task.

In the long run we probably will migrate standalone-rest-test usages to one of the other plugins and deprecate standalone-rest-test
2021-06-30 14:11:25 +02:00
Ryan Ernst 68817d7ca2
Rename o.e.common in libs/core to o.e.core (#73909)
When libs/core was created, several classes were moved from server's
o.e.common package, but they were not moved to a new package. Split
packages need to go away long term, so that Elasticsearch can even think
about modularization. This commit moves all the classes under o.e.common
in core to o.e.core.

relates #73784
2021-06-08 09:53:28 -07:00
Rene Groeschke 2d42c26249
Keep StandaloneRestIntegTest task public (#72865)
This is used by hadoop
2021-05-09 11:36:16 +02:00
Rene Groeschke 78182686fc
Remove more usage of interal logic in public build api (#72858) 2021-05-08 23:06:23 +02:00
Rene Groeschke e609e07cfe
Remove internal build logic from public build tool plugins (#72470)
Extract usage of internal API from TestClustersPlugin and PluginBuildPlugin and related plugins and build logic

This includes a refactoring of ElasticsearchDistribution to handle types
better in a way we can differentiate between supported Elasticsearch
Distribution types supported in TestCkustersPlugin and types only supported
in internal plugins.

It also introduces a set of internal versions of public plugins.

As part of this we also generate the plugin descriptors now.

As a follow up on this we can actually move these public used classes into 
an extra project (declared as included build)

We keep LoggedExec and VersionProperties effectively public And workaround for RestTestBase
2021-05-06 14:02:35 +02:00
Dan Hermann edf2f7d4d6
Unmute IndexingIT and TokenBackwardsCompatibilityIT tests (#72307) 2021-04-30 07:39:55 -05:00
Rene Groeschke 17da1038fe
Port gradle script plugins to precompiled script plugins (#72361)
Script plugins cannot apply plugins and therefore wont work with porting
buildSrc to an included build as we plan. Therefore we take advantage
of moving our script plugins into precompiled script plugins.

As a limitation of this we ran into problems applying binary plugins
from script plugins and for now moved this out of those scripts.
2021-04-29 09:33:28 +02:00
Rene Groeschke 5bcd02cb4d
Restructure build tools java packages (#72030)
Related to #71593 we move all build logic that is for elasticsearch build only into
the org.elasticsearch.gradle.internal* packages

This makes it clearer if build logic is considered to be used by external projects
Ultimately we want to only expose TestCluster and PluginBuildPlugin logic
to third party plugin authors.

This is a very first step towards that direction.
2021-04-26 14:53:55 +02:00
Mark Vieira 02485b2fc3 Mute IndexingIT.testIndexing 2021-04-21 09:09:44 -07:00
Lyudmila Fokina 3b0b7941ae
Warn users if security is implicitly disabled (#70114)
* Warn users if security is implicitly disabled

Elasticsearch has security features implicitly disabled by default for
Basic and Trial licenses, unless explicitly set in the configuration
file.
This may be good for onboarding, but it also lead to unintended insecure
 clusters.
 This change introduces clear warnings when security features are
 implicitly disabled.
 - a warning header in each REST response if security is implicitly
 disabled;
 - a log message during cluster boot.
2021-04-13 18:33:41 +02:00
Jim Ferenczi 2468c41be9
Rewrite date_nanos rolling upgrade test (#70855)
This change rewrites the rolling upgrade yml test that was added for
date_nanos in Java. This is needed to allow extra logic when backporting to
7.x. date_nanos are not available in 6.x so we need to skip the upgraded test
based on the version of the old cluster.

Relates #70463
2021-03-26 09:50:57 +01:00
Jim Ferenczi 701abc6bea
Change default format for date_nanos field (#70463)
This commit updates the default format of date_nanos field
on existing and new indices to use `strict_date_optional_time_nanos` instead of
`strict_date_optional_time`.
Using `strict_date_optional_time` as the default format for date_nanos doesn't
make sense because it accepts and parses dates with nanosecond precision,
but when it formats it drops the nanoseconds.
The change should be transparent for users, these formats accept the same input.

Relates #69192
Closes #67063
2021-03-17 11:40:32 +01:00
David Turner 257a21630e
Fix ensureGreen() timeout in REST tests (#69704)
In 2a04118e88 we moved `ensureGreen()`
from `IndexingIT` to `ESRestTestCase`, including its `70s` timeout. This
timeout makes sense in the context of an `AbstractRollingTestCase` which
has a client timeout of `90s` (#26781) but general-purpose REST tests
only have a `60s` client timeout, so if `ensureGreen()` fails then it
fails with a `SocketTimeoutException`, bypassing the useful exception
handling that log the cluster state at time of failure.

This commit reduces the `ensureGreen()` timeout for most tests, leaving
it at `70s` only for `AbstractRollingTestCase`.
2021-03-01 16:13:27 +00:00
Mark Vieira a92a647b9f Update sources with new SSPL+Elastic-2.0 license headers
As per the new licensing change for Elasticsearch and Kibana this commit
moves existing Apache 2.0 licensed source code to the new dual license
SSPL+Elastic license 2.0. In addition, existing x-pack code now uses
the new version 2.0 of the Elastic license. Full changes include:

 - Updating LICENSE and NOTICE files throughout the code base, as well
   as those packaged in our published artifacts
 - Update IDE integration to now use the new license header on newly
   created source files
 - Remove references to the "OSS" distribution from our documentation
 - Update build time verification checks to no longer allow Apache 2.0
   license header in Elasticsearch source code
 - Replace all existing Apache 2.0 license headers for non-xpack code
   with updated header (vendored code with Apache 2.0 headers obviously
   remains the same).
 - Replace all Elastic license 1.0 headers with new 2.0 header in xpack.
2021-02-02 16:10:53 -08:00
David Turner 06e141888f
Reinstate BWC snapshot tests (#67938)
This commit mostly reverts #67934, except for the change to the version
constant `REPOSITORY_UUID_IN_REPO_DATA_VERSION`.

Completes the backport of #67829 via #67899
2021-01-25 18:36:12 +00:00
David Turner faed3e7199
Temporarily suppress BWC snapshot tests (#67934)
This commit suppresses any BWC tests related to snapshots in `master` so
that #67899 can be merged to `7.x`. It will mostly be reverted after the
merge of #67899 is complete.

Relates #66431
2021-01-25 17:48:47 +00:00
Przemyslaw Gomulka c2c50d5aed
Make scripted search templates work with new mediaType from XContentType.JSON (#67677)
Stored scripts can have content_type option set, however when empty they default to XContentType.JSON#mediaType(). Commit 5e74f79 has changed this in master (ES8) method to return application/json;charset=utf-8 (previously application/json; charset=UTF-8)
This means that when upgrading ES from version 7 to 8 stored script will fail when being used as the encoder is being matched with string equality (map key)

This commit address this by adding back (in addition) the old application/json; charset=UTF-8 into the encoders map.

closes #66986
2021-01-21 12:03:38 +01:00
Przemyslaw Gomulka 3312c2ed03
Mute failing tests due to mustache content_type (#66987)
relates #66986
2021-01-05 10:55:44 +01:00
Rene Groeschke 810e7ff6b0
Move tasks in build scripts to task avoidance api (#64046)
- Some trivial cleanup on build scripts
- Change task referencing in build scripts to use task avoidance api
where replacement is trivial.
2020-11-12 12:04:15 +01:00
Gordon Brown 27089e7460
Handle deprecation warnings in "old cluster" parts of BWC tests (#63363)
This PR adjusts the BWC tests in to handle the deprecation warnings that are now emitted in old clusters due to the backport of system index access deprecation warnings.
2020-10-06 15:39:20 -06:00
Gordon Brown 91f4b58bf7
Deprecate REST access to System Indices (#60945)
This PR adds deprecation warnings when accessing System Indices via the REST layer. At this time, these warnings are only enabled for Snapshot builds by default, to allow projects external to Elasticsearch additional time to adjust their access patterns.

Deprecation warnings will be triggered by all REST requests which access registered System Indices, except for purpose-specific APIs which access System Indices as an implementation detail a few specific APIs which will continue to allow access to system indices by default:

- `GET _cluster/health`
- `GET {index}/_recovery`
- `GET _cluster/allocation/explain`
- `GET _cluster/state`
- `POST _cluster/reroute`
- `GET {index}/_stats`
- `GET {index}/_segments`
- `GET {index}/_shard_stores`
- `GET _cat/[indices,aliases,health,recovery,shards,segments]`

Deprecation warnings for accessing system indices take the form:
```
this request accesses system indices: [.some_system_index], but in a future major version, direct access to system indices will be prevented by default
```
2020-10-06 11:13:48 -06:00
Rene Groeschke aa5d3159de
Use services for archive and file operations in tasks (#62968)
Referencing a project instance during task execution is discouraged by
Gradle and should be avoided. E.g. It is incompatible with Gradles
incubating configuration cache. Instead there are services available to handle
archive and filesystem operations in task actions.

Brings us one step closer to #57918
2020-10-01 10:47:41 +02:00
Lee Hinman 0c3599577e
Add index.routing.allocation.prefer._tier setting (#62589)
This commit adds the `index.routing.allocation.prefer._tier` setting to the
`DataTierAllocationDecider`. This special-purpose allocation setting lets a user specify a
preference-based list of tiers for an index to be assigned to. For example, if the setting were set
to:

```
"index.routing.allocation.prefer._tier": "data_hot,data_warm,data_content"
```

If the cluster contains any nodes with the `data_hot` role, the decider will only allow them to be
allocated on the `data_hot` node(s). If there are no `data_hot` nodes, but there are `data_warm` and
`data_content` nodes, then the index will be allowed to be allocated on `data_warm` nodes.

This allows us to specify an index's preference for tier(s) without causing the index to be
unassigned if no nodes of a preferred tier are available.

Subsequent work will change the ILM migration to make additional use of this setting.

Relates to #60848
2020-09-18 14:49:59 -06:00
Lee Hinman 28cec563b1
Allocate newly created indices on data_hot tier nodes (#61342)
This commit adds the functionality to allocate newly created indices on nodes in the "hot" tier by
default when they are created.

This does not break existing behavior, as nodes with the `data` role are considered to be part of
the hot tier. Users that separate their deployments by using the `data_hot` (and `data_warm`,
`data_cold`, `data_frozen`) roles will have their data allocated on the hot tier nodes now by
default.

This change is a little more complicated than changing the default value for
`index.routing.allocation.include._tier` from null to "data_hot". Instead, this adds the ability to
have a plugin inject a setting into the builder for a newly created index. This has the benefit of
allowing this setting to be visible as part of the settings when retrieving the index, for example:

```
// Create an index
PUT /eggplant

// Get an index
GET /eggplant?flat_settings
```

Returns the default settings now of:

```json
{
  "eggplant" : {
    "aliases" : { },
    "mappings" : { },
    "settings" : {
      "index.creation_date" : "1597855465598",
      "index.number_of_replicas" : "1",
      "index.number_of_shards" : "1",
      "index.provided_name" : "eggplant",
      "index.routing.allocation.include._tier" : "data_hot",
      "index.uuid" : "6ySG78s9RWGystRipoBFCA",
      "index.version.created" : "8000099"
    }
  }
}
```

After the initial setting of this setting, it can be treated like any other index level setting.

This new setting is *not* set on a new index if any of the following is true:

- The index is created with an `index.routing.allocation.include.<anything>` setting
- The index is created with an `index.routing.allocation.exclude.<anything>` setting
- The index is created with an `index.routing.allocation.require.<anything>` setting
- The index is created with a null `index.routing.allocation.include._tier` value
- The index was created from an existing source metadata (shrink, clone, split, etc)

Relates to #60848
2020-08-27 12:51:12 -06:00
Jay Modi ec1c2bd8dc
Remove content type required setting (#61043)
This change removes the HTTP content type required setting, which was
deprecated in 6.0 and only existed for users upgrading from 5.x so that
they did not need to remove the setting immediately. The setting has no
effect on behavior.
2020-08-12 12:42:26 -06:00
Jay Modi 8c51fc7e2d
System index reads in separate threadpool (#57936)
This commit introduces a new thread pool, `system_read`, which is
intended for use by system indices for all read operations (get and
search). The `system_read` pool is a fixed thread pool with a maximum
number of threads equal to lesser of half of the available processors
or 5. Given the combination of both get and read operations in this
thread pool, the queue size has been set to 2000. The motivation for
this change is to allow system read operations to be serviced in spite
of the number of user searches.

In order to avoid a significant performance hit due to pattern matching
on all search requests, a new metadata flag is added to mark indices
as system or non-system. Previously created system indices will have
flag added to their metadata upon upgrade to a version with this
capability.

Additionally, this change also introduces a new class, `SystemIndices`,
which encapsulates logic around system indices. Currently, the class
provides a method to check if an index is a system index and a method
to find a matching index descriptor given the name of an index.

Relates #50251
Relates #37867
2020-08-10 12:38:54 -06:00
Rene Groeschke dd74be0f83
Merge test runner task into RestIntegTest (#60261)
* Merge test runner task into RestIntegTest
* Reorganizing Standalone runner and RestIntegTest task
* Rework general test task configuration and extension
2020-08-03 12:07:41 +02:00
Dan Hermann a3314c1c4a
Ignore template warnings in IndexingIT (#59962) 2020-07-22 07:56:54 -05:00
Dan Hermann dd67392b66
Fix warning handler used in DataStreamsUpgradeIT (#59960) 2020-07-22 06:50:15 -05:00
Martijn van Groningen 448bcba7f9
Ignore multiple matching templates warning in specific tests. (#59692)
Closes #59679
2020-07-16 19:04:04 +02:00
Lee Hinman bb002cf0da
Add allowed warning in composable template upgrade test (#59180)
Resolves #58990
2020-07-08 08:59:09 -06:00
Jake Landis 333a5d8cdf
Create plugin for yamlTest task (#56841)
This commit creates a new Gradle plugin to provide a separate task name
and source set for running YAML based REST tests. The only project
converted to use the new plugin in this PR is distribution/archives/integ-test-zip.
For which the testing has been moved to :rest-api-spec since it makes the most
sense and it avoids a small but awkward change to the distribution plugin.

The remaining cases in modules, plugins, and x-pack will be handled in followups.

This plugin is distinctly different from the plugin introduced in #55896 since
the YAML REST tests are intended to be black box tests over HTTP. As such they
should not (by default) have access to the classpath for that which they are testing.

The YAML based REST tests will be moved to separate source sets (yamlRestTest).
The which source is the target for the test resources is dependent on if this
new plugin is applied. If it is not applied, it will default to the test source
set.

Further, this introduces a breaking change for plugin developers that
use the YAML testing framework. They will now need to either use the new source set
and matching task, or configure the rest resources to use the old "test" source set that
matches the old integTest task. (The former should be preferred).

As part of this change (which is also breaking for plugin developers) the
rest resources plugin has been removed from the build plugin and now requires
either explicit application or application via the new YAML REST test plugin.

Plugin developers should be able to fix the breaking changes to the YAML tests
by adding apply plugin: 'elasticsearch.yaml-rest-test' and moving the YAML tests
under a yamlRestTest folder (instead of test)
2020-07-06 12:13:01 -05:00
Lee Hinman 3ab3f48bd2
Add rolling upgrade tests for component and composable templates (#58867)
This adds rolling upgrade tests that component and composable templates can be read from older
versions of the cluster.

Relates to #58643
2020-07-02 14:13:35 -06:00
Nhat Nguyen bf910e9132
Fix recovery stage transition with sync_id (#57754)
If the recovery source is on an old node (before 7.2), then the recovery
target won't have the safe commit after phase1 because the recovery
source does not send the global checkpoint in the clean_files step. And
if the recovery fails and retries, then the recovery stage won't
transition properly. If a sync_id is used in peer recovery, then the
clean_files step won't be executed to move the stage to TRANSLOG.

This issue was addressed in #57187, but not forward-ported to 8.0. 
 
Closes #57708
2020-06-15 13:06:31 -04:00
Andrei Dan 6b410dfb78
ILM: add support for rolling over data streams (#57295)
As the datastream information is stored in the `ClusterState.Metadata` we exposed
the `Metadata` to the `AsyncWaitStep#evaluateCondition` method in order for
the steps to be able to identify when a managed index is part of a DataStream.

If a managed index is part of a DataStream the rollover target is the DataStream
name and the highest generation index is the write index (ie. the rolled index).
2020-06-02 10:25:22 +01:00
Hendrik Muhs 574aacc8b2
add a basic get index rolling upgrade test (#56322)
add a very basic rolling upgrade test for get index, post mortem action of #56274
2020-05-08 10:52:34 +02:00
Rene Groeschke fb86184dbb
Move bwcVersions extension property to BuildParams (#56206)
- resolved in GlobalBuildInfoPlugin
- propagate exception if version file cannot be resolved
- add null check when setting bwcVersions
2020-05-07 12:08:17 +02:00
David Turner f4d970a569
Fix up BWC for serialization of GetIndexResponse (#56274)
Completes the backport of #56022

Closes #56265
2020-05-06 15:51:27 +01:00
Ioannis Kakavas bb51f9894e
Mute failing test (#56266)
see https://github.com/elastic/elasticsearch/issues/56265
2020-05-06 12:40:50 +03:00
Igor Motov 1c80e85ad8
Re-enable nodes usage tests and update serialization version (#56051)
Updates the serialization version and re-enables bwc tests for nodes
usage api after backport of #55732
2020-04-30 13:44:30 -04:00
Igor Motov 0c0c289ccf
Suppress nodes usage tests for backport (#56046)
Suppresses bwc tests for backport of #55732
2020-04-30 12:51:14 -04:00
Igor Motov b909cee8e9
Expose agg usage in Feature Usage API (#55732)
* Expose agg usage in Feature Usage API

Counts usage of the aggs and exposes them on the _nodes/usage/.

Closes #53746

* Refactor to include non value sources aggregations

* Fix reported values source type for parent and children aggs

* Refactor SearchModule constructor

* Fix subtype in TTest and IPRanges

* Fix more subtypes in aggs that don't register themselves

* Fix doc tests

* Fix docs

* Fix ScriptedMetricAggregatorTests

* Fix compilation issues after merge

* Fix merge fallout

* This gets stale quickly...

* Address review comments

* Fix tests that were missing proper agg registration in the search module

* Fix ScriptedMetricAggregatorTests

* Address review comments

Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
2020-04-30 09:49:59 -04:00
William Brafford 38cd668ad0
Remove deprecated third-party methods from tests (#55255)
I've noticed that a lot of our tests are using deprecated static methods
from the Hamcrest matchers. While this is not a big deal in any
objective sense, it seems like a small good thing to reduce compilation
warnings and be ready for a new release of the matcher library if we
need to upgrade. I've also switched a few other methods in tests that
have drop-in replacements.
2020-04-15 16:31:51 -04:00
Jason Tedor 95a7eed9aa
Rename MetaData to Metadata in all of the places (#54519)
This is a simple naming change PR, to fix the fact that "metadata" is a
single English word, and for too long we have not followed general
naming conventions for it. We are also not consistent about it, for
example, METADATA instead of META_DATA if we were trying to be
consistent with MetaData (although METADATA is correct when considered
in the context of "metadata"). This was a simple find and replace across
the code base, only taking a few minutes to fix this naming issue
forever.
2020-03-31 15:52:01 -04:00
Mark Vieira 33fc8e7ebf
Consolidate duplication of BWC testing task setup in script plugin (#53079) 2020-03-03 14:40:55 -08:00
Jake Landis 810dc9fce3
Smarter copying of the rest specs and tests (#52114)
This PR addresses the unnecessary copying of the rest specs and allows
for better semantics for which specs and tests are copied. By default 
the rest specs will get copied if the project applies 
`elasticsearch.standalone-rest-test` or `esplugin` and the project 
has rest tests or you configure the custom extension `restResources`. 

This PR also removes the need for dozens of places where the x-pack 
specs were copied by supporting copying of the x-pack rest specs too. 

The plugin/task introduced here can also copy the rest tests to the 
local project through a similar configuration.

The new plugin/task allows a user to minimize the surface area of 
which rest specs are copied. Per project can be configured to include 
only a subset of the specs (or tests). Configuring a project to only 
copy the specs when actually needed should help with build cache hit 
rates since we can better define what is actually in use. 
However, project level optimizations for build cache hit rates are 
not included with this PR.

Also, with this PR you can no longer use the includePackaged flag on 
integTest task.  

The following items are included in this PR:
* new plugin: `elasticsearch.rest-resources` 
* new tasks: CopyRestApiTask and CopyRestTestsTask - performs the copy 
* new extension 'restResources'
```
restResources {
  restApi {
    includeCore 'foo' , 'bar' //will include the core specs that start with foo and bar
    includeXpack 'baz' //will include x-pack specs that start with baz
  }
  restTests {
    includeCore 'foo', 'bar' //will include the core tests that start with foo and bar
    includeXpack 'baz' //will include the x-pack tests that start with baz
  }
}

```
2020-02-25 18:46:32 -06:00
Nhat Nguyen f97b388d7b
Fix testRetentionLeasesEstablishedWhenRelocatingPrimary (#52445)
Replace the current assertion with a more robust assertion.

Closes #52364
2020-02-18 08:50:36 -05:00
Mark Vieira e8562a4e11
Always use bundled JDK for external cluster nodes when BWC testing (#51505) 2020-01-27 22:31:28 -08:00
Nhat Nguyen 6e2f7b4b08
Use Lucene index in peer recovery and resync (#51189)
We can use Lucene index exclusively in peer recoveries and
primary-replica resync in 8.0.

Relates #50775
2020-01-20 17:06:12 -05:00
Nhat Nguyen c893a3e495
Make soft-deletes mandatory in 8.0 (#51122)
Creating indices with soft deletes disabled is no longer supported in 8.0.
2020-01-17 17:34:22 -05:00
Nhat Nguyen 09b46c8646
Goodbye and thank you synced flush! (#50882)
Synced flush was a brilliant idea. It supports instant recoveries with a 
quite small implementation. However, with the presence of sequence
numbers and retention leases, it is no longer needed. This change
removes it from 8.0.

Relates #5077
2020-01-16 09:43:07 -05:00
Nhat Nguyen 08b8450b62 Deprecate synced flush (#50835)
A normal flush has the same effect as a synced flush on Elasticsearch
7.6 or later. It's deprecated in 7.6 and will be removed in 8.0.

Relates #50776
2020-01-13 19:54:23 -05:00
Nhat Nguyen 0c0f45538c
Deprecate indices without soft-deletes (#50502)
Soft-deletes will be enabled for all indices in 8.0. Hence, we should 
deprecate new indices without soft-deletes in 7.x.
2020-01-05 18:30:17 -05:00
Nhat Nguyen 5e0030e130 Adjust BWC for peer recovery retention leases (#50351)
Relates #50351
2019-12-24 09:08:54 -05:00
Yannick Welsch c17dcf3238 Fix testAutoExpandIndicesDuringRollingUpgrade (#50361)
Follow-up to #50361 that fixes the test that does not work against older ES versions
2019-12-20 11:49:07 +01:00
Yannick Welsch e6d2b2994e
Only auto-expand replicas with allocation filtering when all nodes upgraded (#50361)
Follow-up to #48974 that ensures that replicas are only auto-expanded according to allocation
filtering rules once all nodes are upgraded to a version that supports this. Helps with
orchestrating cluster upgrades.
2019-12-20 10:22:23 +01:00
Nhat Nguyen cec6678587
Use peer recovery retention leases for indices without soft-deletes (#50351)
Today, the replica allocator uses peer recovery retention leases to 
select the best-matched copies when allocating replicas of indices with
soft-deletes. We can employ this mechanism for indices without
soft-deletes because the retaining sequence number of a PRRL is the
persisted global checkpoint (plus one) of that copy. If the primary and 
replica have the same retaining sequence number, then we should be able
to perform a noop recovery. The reason is that we must be retaining
translog up to the local checkpoint of the safe commit, which is at most
the global checkpoint of either copy). The only limitation is that we
might not cancel ongoing file-based recoveries with PRRLs for noop
recoveries. We can't make the translog retention policy comply with
PRRLs. We also have this problem with soft-deletes if a PRRL is about to
expire.

Relates #45136
Relates #46959
2019-12-20 00:39:53 -05:00
Nhat Nguyen b9fbc8dc74
Migrate peer recovery from translog to retention lease (#49448)
Since 7.4, we switch from translog to Lucene as the source of history 
for peer recoveries. However, we reduce the likelihood of
operation-based recoveries when performing a full cluster restart from
pre-7.4 because existing copies do not have PPRL.

To remedy this issue, we fallback using translog in peer recoveries if 
the recovering replica does not have a peer recovery retention lease,
and the replication group hasn't fully migrated to PRRL.

Relates #45136
2019-12-13 13:56:50 -05:00
Jim Ferenczi c2deb287f1
Add a cluster setting to disallow loading fielddata on _id field (#49166)
This change adds a dynamic cluster setting named `indices.id_field_data.enabled`.
When set to `false` any attempt to load the fielddata for the `_id` field will fail
with an exception. The default value in this change is set to `false` in order to prevent
fielddata usage on this field for future versions but it will be set to `true` when backporting
to 7x. When the setting is set to true (manually or by default in 7x) the loading will also issue
a deprecation warning since we want to disallow fielddata entirely when https://github.com/elastic/elasticsearch/issues/26472
is implemented.

Closes #43599
2019-11-27 13:38:09 +01:00
Dimitris Athanasiou d0ea642dc7
Also expect RELOCATING in RecoveryIT.testRecoveryClosedIndex (#49090)
In addition to the fix in #48506, it seems we could also get the
`RELOCATING` state.

This fixes the failure in
 https://gradle-enterprise.elastic.co/s/svjmmvqk32cii/tests/ua7icdau7nz6y-2cwvhj3qg5qou?openStackTraces=WzBd
2019-11-15 14:20:10 +02:00
Rory Hunter 3a3e5f6176
Apply 2-space indent to all gradle scripts (#48849)
Closes #48724. Update `.editorconfig` to make the Java settings the default
for all files, and then apply a 2-space indent to all `*.gradle` files.
Then reformat all the files.
2019-11-13 10:14:04 +00:00
Mark Vieira af6af346f7
Introduce type-safe and consistent pattern for handling build globals (#48778)
This commit introduces a consistent, and type-safe manner for handling
global build parameters through out our build logic. Primarily this
replaces the existing usages of extra properties with static accessors.
It also introduces and explicit API for initialization and mutation of
any such parameters, as well as better error handling for uninitialized
or eager access of parameter values.

Closes #42042
2019-11-01 09:54:22 -07:00
Yannick Welsch 289db8ab12
Fix testRecoveryClosedIndex (#48506)
Fixes test failure: https://gradle-enterprise.elastic.co/s/lplb52nlth4re/tests/ftbzinkix5rme-2cwvhj3qg5qou
2019-10-31 16:24:16 +01:00
Alan Woodward 566e1b7d33
Remove type field from DocWriteRequest and associated Response objects (#47671)
This commit removes the type field from index, update and delete requests, and their
associated responses.

Relates to #41059
2019-10-11 10:23:55 +01:00
Alan Woodward 7a622f024f
Remove types from BulkRequest (#46983)
This commit removes types entirely from BulkRequest, both as a global
parameter and as individual entries on update/index/delete lines.

Relates to #41059
2019-10-07 13:29:12 +01:00
Yannick Welsch f52afa3cd4 Adapt version constants after backport (#47353) 2019-10-02 14:32:10 +02:00
Yannick Welsch c048c86351
Allow optype CREATE for append-only indexing operations (#47169)
Bulk requests currently do not allow adding "create" actions with auto-generated IDs.
This commit allows using the optype CREATE for append-only indexing operations. This is
mainly the user facing aspect of it.
2019-09-30 16:54:46 +02:00
Tanguy Leroux b1a03a137f
Remove unused private methods and fields (#47115)
This commit removes a bunch of unused private fields and 
unused private methods from the code base.
2019-09-26 09:35:57 +02:00
Nhat Nguyen 50fde6cfaa Strengthen testUpdate in rolling upgrade
We hit a bug where we can't partially update documents created in a
mixed cluster between 5.x and 6.x. Although this bug does not affect
7.0 or later, we should have a good test that catches this issue.

Relates #46198
2019-09-07 11:17:20 -04:00
Alpar Torok a238248706
Bwc testclusters all (#46265)
Convert all bwc projects to testclusters
2019-09-04 15:39:37 +03:00
David Turner 01287eacb2
Use index for peer recovery instead of translog (#45136)
Today we recover a replica by copying operations from the primary's translog.
However we also retain some historical operations in the index itself, as long
as soft-deletes are enabled. This commit adjusts peer recovery to use the
operations in the index for recovery rather than those in the translog, and
ensures that the replication group retains enough history for use in peer
recovery by means of retention leases.

Reverts #38904 and #42211
Relates #41536
2019-08-02 15:00:32 +01:00