Commit Graph

3214 Commits

Author SHA1 Message Date
Ryan Ernst b0fb121640
Make node client type setting a noop (#101214)
The node client type is a remnant of the transport client. This commit
cleans up some test reads and an unnecessary override of the setting. It
was already not read anywhere in production. Now it is only registered
in order to provide validation. In the future it should be deprecated
and removed.
2023-10-23 20:11:58 -04:00
Martijn van Groningen af08d28d38
Fix painless execute api and tsdb issue. (#101212)
Today using painless execute api with tsdb index can fail with a `_id must be unset or set to [cn4exTOUtxytuLkQAAABeRnR_mY] but was [_id] because [test_index] is in time_series mode` error.
This change addresses this.

The painless execute api shouldn't set use a static _id, but
let the TsidExtractingIdFieldMapper generate it.
Otherwise validation TsidExtractingIdFieldMapper fails.

Closes #101072
2023-10-23 16:40:21 +02:00
Kostas Krikellas aee4f5e087
Skip tsdb delete test for unsupported versions (#101124)
Yet another test affected by the fix for showing the synthetic source,
#98808. This can trigger an assert in older versions as the mapping they
produce (without synthetic source) doesn't match the one they may get
from the master, if the latter is in version 8.10+.

Fixes #101121
2023-10-19 08:00:25 -04:00
Benjamin Trent 65f91cfa71
Ensure mappings are fully updated before reading for all 60_dense_vector_dynamic_mapping (#101086)
other tests rely on getting the mapping and dynamically updated it, not
just ones creating for vectors (specifically there are dynamic `float`
mappings in this yaml test suite too).
2023-10-18 16:16:08 -04:00
Craig Taverner e001c9156f
Removed flaky test for unsupported metric type position (#100978)
This test failed about 1% of the time in mixed clusters. We never figured out why, but since it was a test that really only needed to be run a few times to verify that the correct error is raised in mixed clusters, it was seen as very low priority, and considered better to just delete it.
2023-10-18 18:20:16 +02:00
Benjamin Trent d3951a1472
Add cluster health check to tests & refactor DenseVectorFieldMapper (#101006)
I noticed that sometimes we read from the `fieldType()` and other times
we read from the mapper directly. It seems to me that we should only
ever read and update values from one of those, for sanity's sake. 

So, I removed all values that were part of the mapper directly and used
`fieldType().<value>` everywhere.

Additionally, David Turner suggested that we wait for cluster health
before verifying mappings in the yaml tests, so I added that as well.

related to: https://github.com/elastic/elasticsearch/issues/100502
2023-10-18 12:03:20 -04:00
David Turner 9e77bd7b2e AwaitsFix for #100502 2023-10-18 10:27:02 +01:00
Keith Massey 92ec9d6605
Add executed pipelines to bulk api response (#100031)
This change allows users to pass a new list_executed_pipelines parameter
to the bulk API, which results in an executed_pipelines list being returned.
2023-10-17 09:39:09 -05:00
Benjamin Trent 04db28631d
Adding more logging for diagnosing #100502 (#100961) 2023-10-17 09:35:25 -04:00
Kostas Krikellas a7a37f5196
[TEST] Replace flaky check on recovery stats with cluster health check in TSDS recovery test (#100963)
Recovery stats may contain additional entries, e.g. in case shards get 
relocated. When restoring an index from its snapshot, it suffices to 
check that the index is health and searchable.

Fixes #98746
2023-10-17 08:41:16 -04:00
Artem Prigoda 252926423f
Mute `pre_filter_shard_size with shards that have no hit` test (#100954)
Mutes https://github.com/elastic/elasticsearch/issues/92058
2023-10-17 12:35:35 +02:00
Kostas Krikellas 30ac8feb8f
Skip cat tsdb test for versions 8.7-8-10 (#100914)
Yet another test affected by the fix for showing the synthetic source,
#98808. This can trigger an assert in older versions as the mapping they
produce (without synthetic source) doesn't match the one they may get
from the master, if the latter is in version 8.10+.

Fixes #100913
2023-10-16 11:00:30 -04:00
Kostas Krikellas 3247accddf
[TEST] Assert that both time-series indexes are created (#100885)
* Assert that both time-series indexes are created

* Exclude from 8.7-8.10 mixedClusterTests

* Restore asserts

* Fix assert
2023-10-16 13:10:47 +03:00
Kostas Krikellas fe9995965f Revert "Assert that both time-series indexes are created"
This reverts commit baff9ae361.
2023-10-16 12:23:27 +03:00
Kostas Krikellas baff9ae361 Assert that both time-series indexes are created 2023-10-16 11:16:53 +03:00
Kostas Krikellas 1f6bffaa30
Restore version skipping for position fields (#100817)
* Don't print synthetic source in mapping for bwc tests

* Move comment.

* Don't print synthetic source in mapping for bwc tests #2

* Don't print synthetic source in mapping for bwc tests #2

* Revert "Don't print synthetic source in mapping for bwc tests #2"

This reverts commit 034262c5d2.

* Revert "Don't print synthetic source in mapping for bwc tests #2"

This reverts commit 44e815635e.

* Revert "Don't print synthetic source in mapping for bwc tests (#100572)"

This reverts commit 9322ab9b91.

* Exclude synthetic source test from mixedClusterTests

* Update comment.

* Mute all tsdb tests in mixedClusterTests

This is an interim step to stop sporadic test failures, while we try to
fix version skip for mixed cluster tests.

* Remove old exclusion

* Add aggregations too

* Mute tests for versions between 8.7-8.10

* Remove mute

* Restore version skipping for position fields

* Restore version skip for synthetic source
2023-10-13 15:08:50 +03:00
Kostas Krikellas ff6ef15fff
[TEST] Mute all tsdb tests in mixedClusterTests, for versions 8.7 - 8.10 (#100805)
* Don't print synthetic source in mapping for bwc tests

* Move comment.

* Don't print synthetic source in mapping for bwc tests #2

* Don't print synthetic source in mapping for bwc tests #2

* Revert "Don't print synthetic source in mapping for bwc tests #2"

This reverts commit 034262c5d2.

* Revert "Don't print synthetic source in mapping for bwc tests #2"

This reverts commit 44e815635e.

* Revert "Don't print synthetic source in mapping for bwc tests (#100572)"

This reverts commit 9322ab9b91.

* Exclude synthetic source test from mixedClusterTests

* Update comment.

* Mute all tsdb tests in mixedClusterTests

This is an interim step to stop sporadic test failures, while we try to
fix version skip for mixed cluster tests.

* Remove old exclusion

* Add aggregations too

* Mute tests for versions between 8.7-8.10

* Remove mute
2023-10-13 13:33:03 +03:00
David Kyle 4c41737f56
Update doc links for the inference API (#100690) 2023-10-12 07:26:44 +01:00
Benjamin Trent 64047234dc
Updating testing and logging for dense_vector dynamic dims (#100546)
This adds a test for dynamic dims update mapping merges.

Also, this adds logging to help investigate a periodically failing test.

related to: https://github.com/elastic/elasticsearch/issues/100502
2023-10-11 09:37:03 -04:00
Bogdan Pintea e723c7aacd
Switch visibility to public in ESQL REST spec (#100622)
This update the visibility field in ESQL's REST spec to public.

It also updates the types of quotes used for one the REST object
parameter to backticks, for consistency.
2023-10-10 19:33:30 +02:00
Yang Wang 35737a50d0
[Test] Explicitly set number_of_shards to 1 (#100171)
Some tests rely on the natural index order for test assertions. This
works when the index has a single primary shard but fails otherwise.
This PR adjusts the relevant tests so that they explicitly configure the
number of shards to 1.

Relates: ES-6540
2023-10-10 23:51:52 +11:00
Daniel Mitterdorfer 98756e0818
[Profiling] Add backwards-compatibility test (#99730)
With this commit we add a basic backwards-compatibility test that checks
whether indices and data streams can be properly upgraded to the latest
version.

Co-authored-by: Tim Rühsen <tim.ruehsen@gmx.de>

---------
Co-authored-by: Tim Rühsen <tim.ruehsen@gmx.de>
2023-10-09 14:30:37 +02:00
Panagiotis Bailis 24353a9e7c
Fixing exists query REST tests for sparse_vector_fields (#100030)
Removing (incorrectly) expected exceptions for `exists` queries `sparse_vectors`  in  < 8.0.0 versions.

Closes https://github.com/elastic/elasticsearch/issues/100003
2023-10-06 07:54:59 -04:00
Andrei Dan 839afdc331
Promote the Data stream lifecycle feature to Technical Preview (#100187)
This releases the Data stream lifecycle feature as a
Technical Preview feature.

Data stream lifecycle, albeit in technical preview, will allow data streams
to take advantage of a native simplified and resilient lifecycle implementation.
2023-10-03 17:12:35 +01:00
Luca Cavanna 689a1e490a Merge branch 'main' into lucene_snapshot_9_8 2023-10-02 13:56:12 +02:00
Benjamin Trent 92cea2797e
Add nested support for dense_vector fields and knn search (#99763)
* Nested dense_vector support

* Adjust nested support based on new lucene version

* fixing after rebase

* fixing some code

* fixing tests adding transport version

* spotless

* [Automated] Update Lucene snapshot to 9.9.0-snapshot-b3e67403aaf

* Adds new max_inner_product vector similarity function (#99527)

Adds new max_inner_product vector similarity function. This differs from dot_product in the following ways:

Doesn't require vectors to be normalized
Scales the similarity between vectors differently to prevent negative scores

* requiring top level filter to be parent filter

* adding docs & fixing tests

* adding and fixing docs

* adding changlog

* removing unnecessary file changes

* removing unused imports

* fixing test

* maybe fix doc tests

* continue tests in docs

* fixing more tests

* fixing tests

---------

Co-authored-by: Jim Ferenczi <jim.ferenczi@elastic.co>
Co-authored-by: elasticsearchmachine <infra-root+elasticsearchmachine@elastic.co>
2023-09-28 11:38:04 -04:00
Panagiotis Bailis 075e7c68bd
Muting failing test MixedClusterClientYamlTestSuiteIT test {p0=search.vectors/90_sparse_vector/Sparse vector in 7.x} (#100009) 2023-09-28 13:23:53 +03:00
Luca Cavanna 15c87b681c Merge branch 'main' into lucene_snapshot_9_8 2023-09-28 12:19:14 +02:00
Panagiotis Bailis e7fa8e8d1d
Support exists queries for sparse_vector fields (#99775)
This commit adds non-null `sparse_vector` fields to `_field_names`, so that we can support `exists` queries & also introduces a new IndexVersion to ensure backwards compatibility. 

Closes #99319
2023-09-28 10:31:18 +03:00
Francisco Fernández Castaño ae743e673d
Skip settings validation during desired nodes updates (#99946)
This commit skips settings validation during desired nodes updates.
The issue comes when a setting that needs to be validated depends
on a secure setting that cannot be read while the desired nodes are
updated. To avoid such issues, we'll skip the settings validations
completely.

Closes #96127
2023-09-27 16:51:04 +02:00
Luca Cavanna b3e769987d Merge branch 'main' into lucene_snapshot_9_8 2023-09-22 13:11:10 +02:00
Mary Gouseti dc8d84d2d5
Apply mixed tests after Data Stream Lifecycle went GA (#99795) 2023-09-22 10:51:24 +03:00
Mayya Sharipova ddf17e6be5
Increase the max vector dims to 4096 (#99682) 2023-09-20 15:43:40 -04:00
Benjamin Trent dee85de61c Adds new max_inner_product vector similarity function (#99527)
Adds new max_inner_product vector similarity function. This differs from dot_product in the following ways:

Doesn't require vectors to be normalized
Scales the similarity between vectors differently to prevent negative scores
2023-09-20 20:51:46 +02:00
Luca Cavanna 270de88ea0 Merge branch 'main' into lucene_snapshot_9_8 2023-09-20 20:41:55 +02:00
Quentin Pradet 7f4b9bbe8a
[DOCS] Stop linking to indices.templates.html page (#99618) 2023-09-19 16:28:43 +04:00
elasticsearchmachine 1fe6ca0a4b Merge remote-tracking branch 'origin/main' into lucene_snapshot 2023-09-18 10:06:35 +00:00
Andrei Dan 8768277112
DSL Disable yaml tests for <8.11 as the parser was not enabled (#99621) 2023-09-18 10:30:36 +01:00
elasticsearchmachine daaafffe1e Merge remote-tracking branch 'origin/main' into lucene_snapshot 2023-09-14 10:05:48 +00:00
Mark Vieira 7be3d2c191
Increase flexibility of test cluster execution environments (#99437) 2023-09-13 11:18:40 -07:00
Luca Cavanna b815a47e26 Merge branch 'main' into lucene_snapshot 2023-09-13 14:51:01 +02:00
eyalkoren af895549cd
Fix composable templates with `subobjects: false` (#97317) 2023-09-13 14:25:13 +03:00
David Kyle abee4d3ea8
Inference API (#99224)
Adds the _inference API for managing inference models and performing inference.

Inference is a new plugin in XPack that creates a new system index (.inference)
for storing the model configurations. Models configurations are managed with 
the standard PUT, GET, DELETE requests and POST to perform inference.
This PR creates an inference service for  deploying and inferring on the 
ELSER model.
2023-09-12 21:52:34 +01:00
elasticsearchmachine 3d186fda96 Merge remote-tracking branch 'origin/main' into lucene_snapshot 2023-09-12 10:05:06 +00:00
Benjamin Trent 83b70e37ef
Revert "Auto-normalize dot_product vectors at index & query (#98944)" (#99421)
This reverts commit 7b9c367aeb.
2023-09-11 09:33:17 -04:00
elasticsearchmachine 00299dd417 Merge remote-tracking branch 'origin/main' into lucene_snapshot 2023-09-09 10:04:18 +00:00
Carlos Delgado 648d30ba2a
Fix sparse vector tests (#99339)
* Fix sparse vector error so it can catch errors thrown before 8.11
2023-09-08 16:15:07 +02:00
elasticsearchmachine 7026d0711c Merge remote-tracking branch 'origin/main' into lucene_snapshot 2023-09-08 10:07:26 +00:00
David Turner a40b61a0d8 Mute YAML test for #99318 2023-09-08 08:58:23 +01:00
elasticsearchmachine 2bcf075403 Merge remote-tracking branch 'origin/main' into lucene_snapshot 2023-09-07 10:05:13 +00:00