Commit Graph

318 Commits

Author SHA1 Message Date
David Turner 9794c6e205
Use ESIntegTestCase#prepareSearch more (#101179)
The refactoring in #101175 only covered all the one-arg call sites. This
PR does the rest.
2023-10-20 18:33:00 +01:00
David Turner 1eda6ac74b
Extract ESIntegTestCase#prepareSearch (#101175)
Relates #101172
2023-10-20 06:18:58 -04:00
Armin Braun ca6295e582
Remove more explicit references to SearchResponse in tests (#101092)
Remove `assertSearchResponse` which was just an alias for
`assertNoFailures` and then cleanup many spots in the result
by combining the hit count and no failure assertion into a single
method.

follow-up to #100966
2023-10-19 17:53:13 +02:00
Armin Braun dcaba064dd
Remove more explicit SearchResponse references from test code (#100985)
Follow-up to #100966 adding more overrides to assertions that
consume a request builder.
2023-10-18 07:20:01 +02:00
Luca Cavanna db10810309
Expression script engine cleanups around _value usage (#99706)
We have found some inconsistencies as part of #99667, around the current usages of ReplaceableConstDoubleValueSource in ExpressionsScriptEngine. It looks like _value is exposed to the bindings of score scripts, but setValue is never called hence it will always be 0. That can be replaced with a constant double values source, but the next question is whether it even needs to be added to the bindings then.

Another cleanup discussed in #99667 is throwing UnsupportedOperationException from ReplaceableConstDoubleValueSource#explain as it should never be called. Implementing the method means we need to test it which makes little sense if the method is never called in production code.
2023-09-25 12:25:10 +02:00
Luca Cavanna 75b6f96453
Adjust ExpressionAggregationScript to support inter-segment concurrency (#99667)
Handling of _value in a script agg does not support search concurrency
when using the expression script engine. The reason is that the value gets set
globally assuming sequential execution. This commit addresses that by setting
the value to the values source associated with the correct leaf reader context,
while it was previosly being set on a shared data structure.

Closes #99156
2023-09-21 09:37:11 +02:00
David Kyle af4b2d98be
Mute MoreExpressionIT::testStringSpecialValueVariable (#99411)
For #99156
2023-09-11 10:47:20 +01:00
Armin Braun dd7d381922
Dry up getting cluster admin client in tests (#96952)
Drying this up further and adding the same short-cut for single node
tests. Dealing with most of the spots that I could grab via automatic
refactorings.
2023-06-22 14:27:23 +02:00
Armin Braun 3f8ee82ef8
Use indices admin client shortcut in most integration tests (#96946)
Replacing the remaining usages that I could automatically replace
and a couple that I did by hand in this PR.
Also, added the same shortcut to the single node tests to save some
duplication there.
2023-06-20 13:32:59 +02:00
Luca Cavanna edd7749164
Upgrade to lucene-9.5.0-snapshot-d19c3e2e0ed (#92957)
9.5 will include several changes related to vector search. An extensive list is available at https://github.com/apache/lucene/milestone/4 .
2023-01-19 14:07:33 +01:00
Alan Woodward c720cdbbbf
Replace SourceLookup with SourceProvider interface (#91540)
SourceLookup mixes up several concerns - lazy loading, map access to scripts,
different access providers - and duplicates logic (such as that choosing how to
apply filtering) that is better handled directly in the Source interface.

This commit removes SourceLookup entirely and replaces it with a new
SourceProvider interface, with a simple stored fields reader implementation.
SearchLookup implements this interface directly, and the fetch phase uses
a custom implementation to provide its separately loaded source to fetch-time
scripts.
2023-01-12 16:17:46 +00:00
Mark Vieira c2eda511de
Add JUnit rule based integration test cluster orchestration framework (#92379)
This commit adds a new test framework for configuring and orchestrating
test clusters for both Java and YAML REST testing. This will eventually
replace the existing "test-clusters" Gradle plugin and the build-time
cluster orchestration.
2022-12-21 15:33:46 -08:00
Nikola Grcevski a23999e700
Support DoubleValues expression scripts in lang-expression (#89895)
This allows for custom DoubleValues scripts to be made
with lang-expressions. The change also adds a way for plugin
integration and unit tests to be able to create class loaders,
after this permission was removed in 8.0.
2022-09-14 10:42:59 -04:00
Craig Taverner b68d62ba1e
Refactor GeoPoint and GeoShape with generics (#89388)
* Refactor GeoPoint and GeoShape with generics

In preparation for supporting CartesianPoint and CartesianShape
in aggregations, this PR adds a common interface between GeoPoint
and CartesianPoint, and then uses that to split out some key common
code that will be used in CartesianPoint and CartesianShape aggregations

* Simplify generics (by Ignacio)

Co-authored-by: Ignacio Vera <ivera@apache.org>

* Refactor ElasticPoint to SpatialPoint

* Rename ShapeValuesProvider to ShapeValuesSource

It extends ValuesSource, and is extended by GeoShapeValuesSource.
There is no reason for the suffix `Provider`.

* Code review, mostly AbstractShapeIndexFieldData

* Reverted trivial refactoring

* Removed unused Writable interface implementation

* Further generics refinements

Based on Ignacio's work in 050df953df,
we fix the BoundingBox generics, and also add a little more specificity
to the previous generics (replace <?> with <? extends SpatialPoint>).

* Removed some geo-specific code from BoundingBox

Co-authored-by: Ignacio Vera <ivera@apache.org>
2022-09-07 11:23:34 +02:00
tmgordeeva db5ddb321f
Synthetic source error on script loads (#88334)
* Source Lookup refactor with error on script synthetic source load

Refactors SourceLookup into a static source lookup used for most cases where we
access the source again after the fetch phase, and a re-loading lookup used by
scripts. The re-loading lookup now also fails with an error when we are using
synthetic source preventing silent failures or non-sensical behavior from
scripts.
2022-08-26 11:32:53 -07:00
Rene Groeschke 3909b5eaf9
Add verification metadata for dependencies (#88814)
Removing the custom dependency checksum functionality in favor of Gradle build-in dependency verification support. 

- Use sha256 in favor of sha1 as sha1 is not considered safe these days.

Closes https://github.com/elastic/elasticsearch/issues/69736
2022-08-04 09:51:16 +02:00
Ignacio Vera ed564f6e1d
Update lo lucene-9.3.0 (#88927) 2022-08-01 07:21:13 +02:00
Jack Conradson 5e0701f026
Add source fallback for keyword fields using operation (#88735)
This change adds an operation parameter to FieldDataContext that allows us to specialize the field data that are returned from fielddataBuilder in MappedFieldType. Keyword, integer, and geo point field types now support source fallback where we build a doc values wrapper using source if doc values doesn't exist for this field under the operation SCRIPT. This allows us to have source fallback in scripting for the scripting fields API.
2022-07-28 10:34:05 -07:00
Ignacio Vera 3b7f393a82
Upgrade to lucene snapshot lucene-9.3.0-snapshot-b8d1fcfd0ec (#88706) 2022-07-22 11:22:39 +02:00
Nhat Nguyen bd69f90fff
Upgrade to Lucene-9.3.0-snapshot-2d05f5c623e (#88284)
To include LUCENE-10620 - which passes Weight to Collector
2022-07-06 16:16:03 -04:00
Nhat Nguyen c2dc6e6ef4
Upgrade to new Lucene snapshot (#87932)
This PR uses Lucene-9.3 snapshot in Elasticsearch 8.4. Noticeable changes in this Lucene snapshot:

- Merge-on-refresh (disabled)
- No more pathological merging
- SortedSetDocValues#count for value_count aggs
2022-06-23 12:18:27 -04:00
Chris Hegarty aa170f1da5
Modularize the lang-expression component (#87790) 2022-06-17 14:11:56 +01:00
Chris Hegarty e27df984d5
Update lang-expression transitive dependencies (#87751) 2022-06-17 10:08:04 +01:00
Alan Woodward 048fa422c2
Update to public lucene 9.2.0 release (#87162) 2022-06-06 10:06:41 +01:00
Alan Woodward 205cfec52f
Upgrade to lucene 9.2.0-RC2 snapshot (#86931)
Only difference from last snapshot is a revert of a change in the behaviour
of PersianAnalyzer
2022-05-20 08:54:35 +01:00
Alan Woodward 0418e8a9d8
Upgrade to lucene snapshot 978eef5459c (#86852)
Final (hopefully!) snapshot before the 9.2.0 release

* Update test to expect persian tokenfilter - will be exposed later
* Fix KnnVectorQueryBuilderTests::doAssertLuceneQuery

Co-authored-by: Mayya Sharipova <mayya.sharipova@elastic.co>
2022-05-17 15:27:52 -07:00
Alan Woodward 4d076eee20
Upgrade to Lucene 9.2 snapshot efa5d6f4d43 (#86227)
Notable changes include:

count implementations for MultiRangeQuery and IndexSortedNumericDocValuesRangeQuery, which may speed up certain aggregations
more efficient decoding of docids in BKD reader
2022-05-05 15:48:13 +01:00
Ignacio Vera af2fe8ee33
Upgrade Lucene to 9.1.0 release (#85211) 2022-03-22 14:11:53 +01:00
Alan Woodward 0863fb83d5
Upgrade to lucene 9.1.0-snapshot-5b522487ba8 (#85025)
Specifically includes LUCENE-10469 which should address a performance 
regression in EQL.
2022-03-16 14:58:20 +00:00
Julie Tibshirani bba2dfac56
Upgrade Lucene to 9.1.0-snapshot-949752 (#84540)
This PR upgrades Lucene to a newer snapshot `9.1.0-snapshot-9497524cc2d`.

Changes:
* Adapt to `LeafReader#searchNearestVectors` signature change
* Adapt checks in `GeometryIndexerTests`, `SearchServiceTests`, `FiltersAggregatorTests`, `AggregationProfilerIT`
* Address highlighting failures in `MultiPhrasePrefixQuery` and `HasChildQueryBuilder.LateParsingQuery`
2022-03-04 08:36:26 -08:00
Mayya Sharipova 26c3dd6857
Upgrade to lucene-9.1.0-snapshot-1336263051c (#83667)
Lucene issues that resulted in elasticsearch changes:

LUCENE-9820 Separate logic for reading the BKD index from logic to intersecting it.
LUCENE-10377: Replace 'sortPos' with 'enableSkipping' in SortField.getComparator()
LUCENE-10301: make the test-framework a proper module by moving all test
classes to org.apache.lucene.tests
LUCENE-10300: rewrite how resources are read in ukrainian morfologik analyzer:
LUCENE-10054 Make HnswGraph hierarchical
2022-02-22 09:53:20 +01:00
Przemyslaw Gomulka 037261356e
Convert 'id' and '_id' values in REST API tests to strings (#82681)
Follow-up from #77144 (comment) with converting id/_id to always be strings instead of integers. This makes the type value in the Elasticsearch specification be only string instead of string | number.

this change was generated using following command on ubuntu
find . -type f -name "*.yml" -print0 | xargs -0 sed -i -r 's/([^a-zA-Z0-9_\.]id|[^a-zA-Z0-9_]_id):(\s*)([0-9]+)/\1:\2"\3"/g'
2022-02-10 09:14:17 +01:00
Artem Prigoda 0699c9351f
Use Java 14 switch expressions (#82178)
JEP 361[https://openjdk.java.net/jeps/361] added support for switch expressions
which can be much more terse and less error-prone than switch statements.

Another useful feature of switch expressions is exhaustiveness: we can make
sure that an enum switch expression covers all the cases at compile time.
2022-01-10 09:53:35 +01:00
Artem Prigoda 763d6d510f
Use Java 15 text blocks for JSON and multiline strings (#80751)
The ES code base is quite JSON heavy. It uses a lot of multi-line JSON requests in tests which need to be escaped and concatenated which in turn makes them hard to read. Let's try to leverage Java 15 text blocks for representing them.
2021-12-15 18:01:28 +01:00
Alan Woodward 33ef38e478
Upgrade to released lucene 9.0.0 (#81426)
This commit makes elasticsearch depend on the released artifacts
of lucene 9.0.0, rather than an internal snapshot.
2021-12-07 14:19:56 +00:00
Rory Hunter 754393d352
Fix shadowed vars pt4 (#80842)
Part of #19752. Fix more instances where local variable names were shadowing field names.
2021-11-18 19:59:33 +00:00
Mayya Sharipova db0b4ba08a
Upgrade Lucene 9 snapshot cc2a31f2be8 (#80213) 2021-11-02 15:50:33 -04:00
Rene Groeschke 92e8ba2e74
Check for multiple javadocs in java headers (#79603)
We also now enforce to have the license statement on the very top of the java file before 
the package declaration

Fixes #79235
2021-10-29 08:32:11 +02:00
Ryan Ernst c66a371f8e
Upgrade Mockito to 4.0.0 (#79949)
Mockito 4.0 removes several deprecated methods. This commit updates
usages of those deprecated methods and upgrades mockito. The changes
include: * Replace anyMapOf,anyListOf,anySetOf,anyCollectionOf with the
same   method name without `Of` and no longer taking any arguments. *
Replace anyObject with any * Removing argument from isNull * Replace
verifyZeroInteractions with verifyNoMoreInteractions The changes here
were completely mechanical, done entirely with forms of find/replace
within IntelliJ.
2021-10-27 16:16:18 -04:00
Mark Vieira 12ad399c48 Reformat Elasticsearch source 2021-10-27 08:19:51 -07:00
Ryan Ernst f8d8702b88
Convert uses of mockito Matchers to ArgumentMatchers (#79852)
Matchers is deprecated in Mockito, in favor of the newer
ArgumentMatchers class. In fact, internally Matchers just extends
ArgumentMatchers as all the methods there were moved. This commit
changes all imports of org.mockito.Matchers to
org.mockito.ArgumentMatchers.
2021-10-26 14:16:11 -07:00
Alan Woodward dd3227192a
Update lucene 9 snapshot (#79701)
This includes the following changes:

* LUCENE-10180: Avoid using lambdas in SegmentMerger
* LUCENE-10187: Reduce DirectWriter's padding
* LUCENE-10193: Cut over more array access to VarHandles
* LUCENE-10189: Optimize flush of doc-value fields that are effectively single-valued
* LUCENE-10165: Implement Lucene90DocValuesProducer#getMergeInstance
2021-10-25 12:59:43 +01:00
Mayya Sharipova 074b023be5
Upgrade lucene version 8b68bf60c98 (#79461)
This introduces a new 8.10.1 Lucene version for ES 7.16

Also Enable bwc that were disabled in PR#79385
2021-10-19 11:30:41 -04:00
Alan Woodward 13f05358a0
Update to new lucene snapshot (#79138)
Includes the following new commits of interest:

* LUCENE-10150: override readLongs() in ByteBuffersDataInput
* LUCENE-10146: Add VectorSimilarityFunction.COSINE
* LUCENE-10140: Correct minimizing iterator sub-matches
* LUCENE-10103 Make QueryCache respect Accountable queries
* LUCENE-10170: Restore compression speed for LZ4.
2021-10-14 12:25:09 +01:00
Ryan Ernst 208d575e43
Remove Joda dependency (#79007)
This commit removes the dependency on the Joda library. It removes
many remaining references to joda, though not all because some comments
are worthwhile for historical reasoning.
2021-10-13 17:37:31 -07:00
Chris Hegarty 20c9f756d2
Fix split package org.elasticsearch.common.xcontent (#78831)
Fix the split package org.elasticsearch.common.xcontent, between server and the x-content lib. Move the x-content lib exported package from org.elasticsearch.common.xcontent to org.elasticsearch.xcontent ( following the naming convention of similar libraries ). Removing split packages is a prerequisite to modularization.
2021-10-08 17:14:26 +01:00
Alan Woodward b56d72a468
Upgrade to lucene snapshot ba75dc5e6bf (#78817)
Includes the following lucene changes:

* LUCENE-10145: Speed up byte[] comparisons using VarHandles.
* LUCENE-10143: Delegate primitive writes in RateLimitedIndexOutput
* LUCENE-10182: No longer check dvGen.
* LUCENE-10153: Speed up BKDWriter using VarHandles.
* LUCENE-10150: override ByteBuffersDataInput readLong/readInt/readShort
2021-10-07 12:11:25 +01:00
Jack Conradson 7ba84696c8
Remove joda time from expressions (#78649)
This change replaces joda time with java time moving from ReadableDateTime to ZonedDateTime in the 
expressions scripting module.
2021-10-04 14:50:57 -07:00
Alan Woodward 92db6a62c0
Update to new lucene snapshot (#78548)
Includes the following changes:

- Updates to prevent package splits in ES: LUCENE-10118, LUCENE-10132
- Speedups in writing doc values: LUCENE-10127, LUCENE-10123, LUCENE-10125
- Speedups in writing primitives: LUCENE-10125
- Sort-after bugfixes: LUCENE-10126
2021-10-01 12:07:42 +01:00
Nhat Nguyen 7d82dbf9e3
Upgrade to Lucene9-snapshot-94c6e261274 (#78286)
Upgrade to a new Lucene 9 snapshot that includes LUCENE-10119 so we can 
re-enable the sort optimization with points for scroll and search_after
requests.

Relates #78230
2021-09-24 10:46:11 -04:00