Commit Graph

109 Commits

Author SHA1 Message Date
Lorenzo Dematté 77dac65761
Fix NodeInfo version parsing in bwc tests (#100838)
* Mixed cluster tests with string NodeInfo version

- Move version based feature comparison to a common, deprecated method (to be replaced with real features)
- Use string comparison against old cluster version to partition new/old cluster nodes
2023-10-25 12:59:10 +02:00
Lorenzo Dematté e5281bfb85
Exclude desired_nodes validation tests from mixed-cluster QA (#100682) 2023-10-13 15:30:37 +02:00
Kostas Krikellas ff6ef15fff
[TEST] Mute all tsdb tests in mixedClusterTests, for versions 8.7 - 8.10 (#100805)
* Don't print synthetic source in mapping for bwc tests

* Move comment.

* Don't print synthetic source in mapping for bwc tests #2

* Don't print synthetic source in mapping for bwc tests #2

* Revert "Don't print synthetic source in mapping for bwc tests #2"

This reverts commit 034262c5d2.

* Revert "Don't print synthetic source in mapping for bwc tests #2"

This reverts commit 44e815635e.

* Revert "Don't print synthetic source in mapping for bwc tests (#100572)"

This reverts commit 9322ab9b91.

* Exclude synthetic source test from mixedClusterTests

* Update comment.

* Mute all tsdb tests in mixedClusterTests

This is an interim step to stop sporadic test failures, while we try to
fix version skip for mixed cluster tests.

* Remove old exclusion

* Add aggregations too

* Mute tests for versions between 8.7-8.10

* Remove mute
2023-10-13 13:33:03 +03:00
Kostas Krikellas 24037d6ed6
Exclude synthetic source test for TSDB from mixedClusterTests (#100592)
* Don't print synthetic source in mapping for bwc tests

* Move comment.

* Don't print synthetic source in mapping for bwc tests #2

* Don't print synthetic source in mapping for bwc tests #2

* Revert "Don't print synthetic source in mapping for bwc tests #2"

This reverts commit 034262c5d2.

* Revert "Don't print synthetic source in mapping for bwc tests #2"

This reverts commit 44e815635e.

* Revert "Don't print synthetic source in mapping for bwc tests (#100572)"

This reverts commit 9322ab9b91.

* Exclude synthetic source test from mixedClusterTests

* Update comment.
2023-10-11 14:33:01 +03:00
David Turner 63b4ee128c
Increase timeout in MixedClusterClientYamlTestSuiteIT (#100585)
This suite now has a couple of thousand tests, some of which take a
couple of seconds, so it times out occasionally. Relaxing the timeout
further.
2023-10-10 22:12:35 +01:00
Luca Cavanna 83f99419d0 Revert "Add require data stream lifecycle feature flag (#98022) (#98026)"
This reverts commit 5406f96450.
2023-09-19 10:08:08 +02:00
Mary Gouseti 5406f96450 Add require data stream lifecycle feature flag (#98022) (#98026) 2023-09-19 09:52:54 +02:00
Martijn van Groningen a97915f0b0
Prepare aggregation yaml tests to be included in serverless build. (#99413)
* Remove setting the number of replicas in aggregation yaml tests.
* Stop waiting for yellow/green. The create index request by default return s when at least the primary shard has been assigned.
* Adjust assertions that are sensitive to the number of shard copies.
* Muted 4 aggregation yaml tests in mixed cluster qa ([link](https://github.com/elastic/elasticsearch/pull/99413/files#diff-033fa895a9866248015cbe7871deab0b3c88fa66a5e1bc9e789079d0d1a39f24R31)). These tests assert cache hits and with replicas we currently can't reliable assert this in a multi node test cluster. Like the mentioned comment suggest if the node selector can reliable select the same node, then we can unmute. I think muting is ok for now. These yaml tests are run in aggregation module and will soon be run in serverless. I think enforcing that many yaml tests run tests against indices with no replicas is worse for test coverage than muting 4 yaml tests in this qa module. Additionally I think that testing that the request cache work should not be tested in full blown integration tests, but more targeted unit/semi integration tests.
2023-09-13 11:45:26 -04:00
Andrei Dan 01ed7de99f
GA the data stream lifecycle (#98644)
This makes the data stream lifecycle generally available. This will allow
data streams to take advantage of a native simplified and resilient
lifecycle implementation.
2023-08-21 17:28:54 +01:00
Mary Gouseti 49ad2d5365
Add require data stream lifecycle feature flag (#98022) 2023-07-28 11:40:21 +03:00
Andrei Dan 614fa5d5cb Revert "Require DLM enabled in the mixed-cluster QA cluster (#96391)"
This reverts commit 4ca12c1849.
2023-06-07 10:54:11 +01:00
Andrei Dan 4ca12c1849
Require DLM enabled in the mixed-cluster QA cluster (#96391) 2023-05-30 08:53:04 +01:00
Salvatore Campagna abba374b08
Fix skip version (#95616)
Indexing in TSDB changed in 8.2.0 so we need to skip mixed cluster
tests on versions before 8.2.0.
2023-05-03 09:51:05 +02:00
Albert Zaharovits 8a32956b1b
Mute tsdb mixed cluster rest IT(#95609)
Muting  test {p0=tsdb/140_routing_path/missing routing path field} and
test {p0=tsdb/140_routing_path/multi-value routing path field}

Relates #95382
2023-04-27 13:50:03 +03:00
Rene Groeschke e735455431
Provide hints to Gradle Task.onlyIf declarations (#94953)
* Update gradle enterprise plugin to show skip reasons in build scans
* Unify OS specific handling in build logic
2023-03-31 23:57:55 +02:00
Artem Prigoda 2bc7398754
Use `Strings.format` instead of `String.format(Locale.ROOT, ...)` in tests (#92106)
Use local-independent `Strings.format` method instead of `String.format(Locale.ROOT, ...)`. 
Inline `ESTestCase.forbidden` calls with `Strings.format` for the consistency sake.
Add `Strings.format` alias in `common.Strings`
2023-01-03 19:28:27 +01:00
Nik Everett ca6e785cbd
Run aggs tests in mixed cluster (#91284)
As we move the aggs tests to a module, we still want to run the aggs
tests against a mixed version cluster. This adds the tests back to the
`qa:mixed-cluster` project.
2022-11-07 06:42:06 -05:00
Rene Groeschke 43a0377735
Update forbiddenapis to 3.4 (#90624)
Fix breaking changes to source validation after change in default jdk rule set
2022-10-06 16:52:06 +02:00
Ryan Ernst 0ec229050e
Move yaml rest test case to separate test lib (#84835)
The ESClientYamlSuiteTestCase is used to run yaml tests throughout
Elasticsearch. It utilizes the low level rest client in sniffing for
nodes, but the sniffer is not needed anywhere else in the test
framework.

This commit creates a new project, `:test:rest-runner` which is meant to
house the rest test running infrastructure. This has two purposes. First
is to remove the sniffer from the test framework dependencies, because
it transitively depends on Jackson. Second is to setup the runner for
future refactorings where it could be made to not depend on the entire
test framework, though how that could work is left for the future.
2022-03-11 10:51:11 -05:00
Benjamin Trent b592d2bf01
New random_sampler aggregation for sampling documents in aggregations (#84363)
This adds a new sampling aggregation that performs a background sampling over all documents in an index. 

The syntax is as follows:
```
{
  "aggregations": {
    "sampling": {
      "random_sampler": {
        "probability": 0.1
      },
      "aggs": {
        "price_percentiles": {
          "percentiles": {
            "field": "taxful_total_price"
          }
        }
      }
    }
  }
}
```

This aggregation provides fast random sampling over the entire document set in order to speed up costly aggregations.

Testing this over a variety of aggregations and data sets, the median speed up when sampling at `0.001` over millions of documents is around 70X speed improvement.

Relative error rate does rely on the size of the data and the aggregation kind. Here are some typically expected numbers when sampling over 10s of millions of documents. `p` is the configured probability and `n` is the number of documents matched by your provided filter query.
2022-03-02 14:32:30 -05:00
Mayya Sharipova 26c3dd6857
Upgrade to lucene-9.1.0-snapshot-1336263051c (#83667)
Lucene issues that resulted in elasticsearch changes:

LUCENE-9820 Separate logic for reading the BKD index from logic to intersecting it.
LUCENE-10377: Replace 'sortPos' with 'enableSkipping' in SortField.getComparator()
LUCENE-10301: make the test-framework a proper module by moving all test
classes to org.apache.lucene.tests
LUCENE-10300: rewrite how resources are read in ukrainian morfologik analyzer:
LUCENE-10054 Make HnswGraph hierarchical
2022-02-22 09:53:20 +01:00
Mark Vieira 64929dc5df
Introduce explicit API for configure test cluster feature flags (#83876) 2022-02-14 15:22:33 -08:00
Martijn van Groningen 0ddfad4cd7
Fix release build (#83720)
-  Add `es.index_mode_feature_flag_registered` feature flag to data-streams module's internalClusterTest task.
-  Add `es.random_sampler_feature_flag_registered` feature flag to xpack rest tests with security qa module.

Closes #83722
2022-02-09 18:30:15 -05:00
Artem Prigoda fc5a820da9
Migrate to Java 16 Records (part 1) (#82338)
Try to represent immutable data with Java records introduced in [JEP 395](https://openjdk.java.net/jeps/395)
2022-01-18 17:53:06 +01:00
Mark Vieira e6ba67f973
Limit full cluster restart upgrade tests to wire compatible versions (#82445) 2022-01-11 14:27:56 -08:00
Artem Prigoda 763d6d510f
Use Java 15 text blocks for JSON and multiline strings (#80751)
The ES code base is quite JSON heavy. It uses a lot of multi-line JSON requests in tests which need to be escaped and concatenated which in turn makes them hard to read. Let's try to leverage Java 15 text blocks for representing them.
2021-12-15 18:01:28 +01:00
Francisco Fernández Castaño 7a630b7b97
Skip IndexingIT#testSyncedFlushTransition on 8.0+ nodes (#81428)
Closes #79930
2021-12-08 17:15:57 +01:00
Mark Vieira 810226b7c8 Mute IndexingIT.testSyncedFlushTransition 2021-10-27 09:30:45 -07:00
Mark Vieira 12ad399c48 Reformat Elasticsearch source 2021-10-27 08:19:51 -07:00
Nikola Grcevski 3f0ea763bf
Reduce BWC version for HotThreads.sort (#79583)
The sort by change has been merged in 7.x now and therefore
we should make 8.0 use the same BWC version.

Relates to #79392 and #79507

Co-authored-by: Nhat Nguyen <nhat.nguyen@elastic.co>
2021-10-25 18:26:54 -04:00
Chris Hegarty 20c9f756d2
Fix split package org.elasticsearch.common.xcontent (#78831)
Fix the split package org.elasticsearch.common.xcontent, between server and the x-content lib. Move the x-content lib exported package from org.elasticsearch.common.xcontent to org.elasticsearch.xcontent ( following the naming convention of similar libraries ). Removing split packages is a prerequisite to modularization.
2021-10-08 17:14:26 +01:00
Rene Groeschke e684d5ef63
Introduce DSL for configuring BWC tests (#78597)
this makes configuring bwc Tests less errorprone due to strange groovy behaviour
2021-10-04 21:43:02 +02:00
Rene Groeschke 6ef13abe81
Do not create unused testCluster (#77581)
* Do not create unused testCluster

This avoids creating test clusters that are not required during the build.
We use lazy configuration here on testClusters and only instantiate them as theyre

* Do not fail on run task (debug)

* Create more test cluster lazy

* Make more test cluster lazy

* Avoid creating unused testcluster

* Fix PluginBuildPlugin

* Fix disabling geo db download

* Fix cluster setup in repository-multi-version

* Polishing

* Fix issue with irretic groovy ogic

* Fix bwc tests

* Fix more bwcTests

* Fix more bwc tests

* Fix more bwc tests

* Fix more bwc tests

* Fix typo

* Minor polishing

* Fix rolling upgrade tests

* Fix cluster config in sql qa mixedcluster project

* Fix more bwc tests

* Clean up before review

* Document test cluster usage

* Api polising after Review

provide useCluster(Provider) method to TestClusterAware

Ideally we take this a step further and realize those test clusters only on use.
But out of scope of this PR.

* Allow gradle provider as value for nonSystemProperties

* Some simplification on test configuration

* Fix typo in rest test config

* Fix more typos

* Fix another typo

* Fix more typos
2021-09-23 03:45:59 -04:00
Mark Vieira 0430de0a06
Fix mixed-cluster tests when built in release mode (#78224) 2021-09-22 16:46:06 -07:00
Nik Everett 962976b045
Add `index.mode: time_series` (#77626)
This adds a setting to enable time series mode that is hidden by the
`es.index_mode_feature_flag_registered` feature flag. All it does right
now is make sure you haven't configured index sorting and partitioning.
This gives us a place to "hang" all of our further work on time series
mode.

Time series mode will entirely take over index sorting. We don't believe
you'll ever be able to configure sorting yourself when you are in time
series mode. We don't expect time series mode to support index
partitioning when we first build it but we'd like to get there.
2021-09-14 12:47:44 -04:00
Przemyslaw Gomulka c96139d006
[Rest Api Compatibility] Deprecate the use of synced flush (#75372)
synced flush is going to be replaced by flush. This commit allows to synced_flush api only in v7 compatibility mode.
Worth noting - sync_id is gone and won't be available in v7 responses from indices.stats

relates removal pr #50882
relates #51816
2021-07-28 14:17:49 +02:00
Ignacio Vera 87a0711044
Unmute RareTermsIT (#75169) 2021-07-12 17:33:44 +02:00
Ignacio Vera 488829f454
mute RareTermsIT (#74986) 2021-07-06 17:02:07 +02:00
Ignacio Vera 24ff395769
Remove Lucene's PackedInt dependency from Cuckoo filter (#74736)
Forks the Lucene data structure into a class called PackedArray that can be serialised using
Elasticsearch streams.
2021-07-06 06:17:16 +02:00
Rene Groeschke d8e4e48a3b
Avoid creating unused test tasks (#74644)
With the overall theme of trying to configure and add less to the build instead of just disabling it later,
we're replacing standalone-test by standalone-rest tasks avoids creating the
unused test tasks.

Standalone rest test plugin and the other rest test plugins behave a little bit different in the sense how source sets and test tasks are wired.

The standalone rest test plugin assumes that all RestTestTasks are using the same sourceSet (test). The yaml, java Rest test plugins use one dedicated sourceSet per test task.

In the long run we probably will migrate standalone-rest-test usages to one of the other plugins and deprecate standalone-rest-test
2021-06-30 14:11:25 +02:00
Ryan Ernst 68817d7ca2
Rename o.e.common in libs/core to o.e.core (#73909)
When libs/core was created, several classes were moved from server's
o.e.common package, but they were not moved to a new package. Split
packages need to go away long term, so that Elasticsearch can even think
about modularization. This commit moves all the classes under o.e.common
in core to o.e.core.

relates #73784
2021-06-08 09:53:28 -07:00
Rene Groeschke 2d42c26249
Keep StandaloneRestIntegTest task public (#72865)
This is used by hadoop
2021-05-09 11:36:16 +02:00
Rene Groeschke 78182686fc
Remove more usage of interal logic in public build api (#72858) 2021-05-08 23:06:23 +02:00
Rene Groeschke e609e07cfe
Remove internal build logic from public build tool plugins (#72470)
Extract usage of internal API from TestClustersPlugin and PluginBuildPlugin and related plugins and build logic

This includes a refactoring of ElasticsearchDistribution to handle types
better in a way we can differentiate between supported Elasticsearch
Distribution types supported in TestCkustersPlugin and types only supported
in internal plugins.

It also introduces a set of internal versions of public plugins.

As part of this we also generate the plugin descriptors now.

As a follow up on this we can actually move these public used classes into 
an extra project (declared as included build)

We keep LoggedExec and VersionProperties effectively public And workaround for RestTestBase
2021-05-06 14:02:35 +02:00
Rene Groeschke 17da1038fe
Port gradle script plugins to precompiled script plugins (#72361)
Script plugins cannot apply plugins and therefore wont work with porting
buildSrc to an included build as we plan. Therefore we take advantage
of moving our script plugins into precompiled script plugins.

As a limitation of this we ran into problems applying binary plugins
from script plugins and for now moved this out of those scripts.
2021-04-29 09:33:28 +02:00
Rene Groeschke 5bcd02cb4d
Restructure build tools java packages (#72030)
Related to #71593 we move all build logic that is for elasticsearch build only into
the org.elasticsearch.gradle.internal* packages

This makes it clearer if build logic is considered to be used by external projects
Ultimately we want to only expose TestCluster and PluginBuildPlugin logic
to third party plugin authors.

This is a very first step towards that direction.
2021-04-26 14:53:55 +02:00
Lyudmila Fokina 3b0b7941ae
Warn users if security is implicitly disabled (#70114)
* Warn users if security is implicitly disabled

Elasticsearch has security features implicitly disabled by default for
Basic and Trial licenses, unless explicitly set in the configuration
file.
This may be good for onboarding, but it also lead to unintended insecure
 clusters.
 This change introduces clear warnings when security features are
 implicitly disabled.
 - a warning header in each REST response if security is implicitly
 disabled;
 - a log message during cluster boot.
2021-04-13 18:33:41 +02:00
Mark Vieira a92a647b9f Update sources with new SSPL+Elastic-2.0 license headers
As per the new licensing change for Elasticsearch and Kibana this commit
moves existing Apache 2.0 licensed source code to the new dual license
SSPL+Elastic license 2.0. In addition, existing x-pack code now uses
the new version 2.0 of the Elastic license. Full changes include:

 - Updating LICENSE and NOTICE files throughout the code base, as well
   as those packaged in our published artifacts
 - Update IDE integration to now use the new license header on newly
   created source files
 - Remove references to the "OSS" distribution from our documentation
 - Update build time verification checks to no longer allow Apache 2.0
   license header in Elasticsearch source code
 - Replace all existing Apache 2.0 license headers for non-xpack code
   with updated header (vendored code with Apache 2.0 headers obviously
   remains the same).
 - Replace all Elastic license 1.0 headers with new 2.0 header in xpack.
2021-02-02 16:10:53 -08:00
David Turner 06e141888f
Reinstate BWC snapshot tests (#67938)
This commit mostly reverts #67934, except for the change to the version
constant `REPOSITORY_UUID_IN_REPO_DATA_VERSION`.

Completes the backport of #67829 via #67899
2021-01-25 18:36:12 +00:00
David Turner faed3e7199
Temporarily suppress BWC snapshot tests (#67934)
This commit suppresses any BWC tests related to snapshots in `master` so
that #67899 can be merged to `7.x`. It will mostly be reverted after the
merge of #67899 is complete.

Relates #66431
2021-01-25 17:48:47 +00:00