Remove `assertSearchResponse` which was just an alias for
`assertNoFailures` and then cleanup many spots in the result
by combining the hit count and no failure assertion into a single
method.
follow-up to #100966
We have found some inconsistencies as part of #99667, around the current usages of ReplaceableConstDoubleValueSource in ExpressionsScriptEngine. It looks like _value is exposed to the bindings of score scripts, but setValue is never called hence it will always be 0. That can be replaced with a constant double values source, but the next question is whether it even needs to be added to the bindings then.
Another cleanup discussed in #99667 is throwing UnsupportedOperationException from ReplaceableConstDoubleValueSource#explain as it should never be called. Implementing the method means we need to test it which makes little sense if the method is never called in production code.
Handling of _value in a script agg does not support search concurrency
when using the expression script engine. The reason is that the value gets set
globally assuming sequential execution. This commit addresses that by setting
the value to the values source associated with the correct leaf reader context,
while it was previosly being set on a shared data structure.
Closes#99156
Drying this up further and adding the same short-cut for single node
tests. Dealing with most of the spots that I could grab via automatic
refactorings.
Replacing the remaining usages that I could automatically replace
and a couple that I did by hand in this PR.
Also, added the same shortcut to the single node tests to save some
duplication there.
SourceLookup mixes up several concerns - lazy loading, map access to scripts,
different access providers - and duplicates logic (such as that choosing how to
apply filtering) that is better handled directly in the Source interface.
This commit removes SourceLookup entirely and replaces it with a new
SourceProvider interface, with a simple stored fields reader implementation.
SearchLookup implements this interface directly, and the fetch phase uses
a custom implementation to provide its separately loaded source to fetch-time
scripts.
This commit adds a new test framework for configuring and orchestrating
test clusters for both Java and YAML REST testing. This will eventually
replace the existing "test-clusters" Gradle plugin and the build-time
cluster orchestration.
This allows for custom DoubleValues scripts to be made
with lang-expressions. The change also adds a way for plugin
integration and unit tests to be able to create class loaders,
after this permission was removed in 8.0.
* Refactor GeoPoint and GeoShape with generics
In preparation for supporting CartesianPoint and CartesianShape
in aggregations, this PR adds a common interface between GeoPoint
and CartesianPoint, and then uses that to split out some key common
code that will be used in CartesianPoint and CartesianShape aggregations
* Simplify generics (by Ignacio)
Co-authored-by: Ignacio Vera <ivera@apache.org>
* Refactor ElasticPoint to SpatialPoint
* Rename ShapeValuesProvider to ShapeValuesSource
It extends ValuesSource, and is extended by GeoShapeValuesSource.
There is no reason for the suffix `Provider`.
* Code review, mostly AbstractShapeIndexFieldData
* Reverted trivial refactoring
* Removed unused Writable interface implementation
* Further generics refinements
Based on Ignacio's work in 050df953df,
we fix the BoundingBox generics, and also add a little more specificity
to the previous generics (replace <?> with <? extends SpatialPoint>).
* Removed some geo-specific code from BoundingBox
Co-authored-by: Ignacio Vera <ivera@apache.org>
* Source Lookup refactor with error on script synthetic source load
Refactors SourceLookup into a static source lookup used for most cases where we
access the source again after the fetch phase, and a re-loading lookup used by
scripts. The re-loading lookup now also fails with an error when we are using
synthetic source preventing silent failures or non-sensical behavior from
scripts.
Removing the custom dependency checksum functionality in favor of Gradle build-in dependency verification support.
- Use sha256 in favor of sha1 as sha1 is not considered safe these days.
Closes https://github.com/elastic/elasticsearch/issues/69736
This change adds an operation parameter to FieldDataContext that allows us to specialize the field data that are returned from fielddataBuilder in MappedFieldType. Keyword, integer, and geo point field types now support source fallback where we build a doc values wrapper using source if doc values doesn't exist for this field under the operation SCRIPT. This allows us to have source fallback in scripting for the scripting fields API.
This PR uses Lucene-9.3 snapshot in Elasticsearch 8.4. Noticeable changes in this Lucene snapshot:
- Merge-on-refresh (disabled)
- No more pathological merging
- SortedSetDocValues#count for value_count aggs
Final (hopefully!) snapshot before the 9.2.0 release
* Update test to expect persian tokenfilter - will be exposed later
* Fix KnnVectorQueryBuilderTests::doAssertLuceneQuery
Co-authored-by: Mayya Sharipova <mayya.sharipova@elastic.co>
Notable changes include:
count implementations for MultiRangeQuery and IndexSortedNumericDocValuesRangeQuery, which may speed up certain aggregations
more efficient decoding of docids in BKD reader
This PR upgrades Lucene to a newer snapshot `9.1.0-snapshot-9497524cc2d`.
Changes:
* Adapt to `LeafReader#searchNearestVectors` signature change
* Adapt checks in `GeometryIndexerTests`, `SearchServiceTests`, `FiltersAggregatorTests`, `AggregationProfilerIT`
* Address highlighting failures in `MultiPhrasePrefixQuery` and `HasChildQueryBuilder.LateParsingQuery`
Lucene issues that resulted in elasticsearch changes:
LUCENE-9820 Separate logic for reading the BKD index from logic to intersecting it.
LUCENE-10377: Replace 'sortPos' with 'enableSkipping' in SortField.getComparator()
LUCENE-10301: make the test-framework a proper module by moving all test
classes to org.apache.lucene.tests
LUCENE-10300: rewrite how resources are read in ukrainian morfologik analyzer:
LUCENE-10054 Make HnswGraph hierarchical
Follow-up from #77144 (comment) with converting id/_id to always be strings instead of integers. This makes the type value in the Elasticsearch specification be only string instead of string | number.
this change was generated using following command on ubuntu
find . -type f -name "*.yml" -print0 | xargs -0 sed -i -r 's/([^a-zA-Z0-9_\.]id|[^a-zA-Z0-9_]_id):(\s*)([0-9]+)/\1:\2"\3"/g'
JEP 361[https://openjdk.java.net/jeps/361] added support for switch expressions
which can be much more terse and less error-prone than switch statements.
Another useful feature of switch expressions is exhaustiveness: we can make
sure that an enum switch expression covers all the cases at compile time.
The ES code base is quite JSON heavy. It uses a lot of multi-line JSON requests in tests which need to be escaped and concatenated which in turn makes them hard to read. Let's try to leverage Java 15 text blocks for representing them.
Mockito 4.0 removes several deprecated methods. This commit updates
usages of those deprecated methods and upgrades mockito. The changes
include: * Replace anyMapOf,anyListOf,anySetOf,anyCollectionOf with the
same method name without `Of` and no longer taking any arguments. *
Replace anyObject with any * Removing argument from isNull * Replace
verifyZeroInteractions with verifyNoMoreInteractions The changes here
were completely mechanical, done entirely with forms of find/replace
within IntelliJ.
Matchers is deprecated in Mockito, in favor of the newer
ArgumentMatchers class. In fact, internally Matchers just extends
ArgumentMatchers as all the methods there were moved. This commit
changes all imports of org.mockito.Matchers to
org.mockito.ArgumentMatchers.
This includes the following changes:
* LUCENE-10180: Avoid using lambdas in SegmentMerger
* LUCENE-10187: Reduce DirectWriter's padding
* LUCENE-10193: Cut over more array access to VarHandles
* LUCENE-10189: Optimize flush of doc-value fields that are effectively single-valued
* LUCENE-10165: Implement Lucene90DocValuesProducer#getMergeInstance
Includes the following new commits of interest:
* LUCENE-10150: override readLongs() in ByteBuffersDataInput
* LUCENE-10146: Add VectorSimilarityFunction.COSINE
* LUCENE-10140: Correct minimizing iterator sub-matches
* LUCENE-10103 Make QueryCache respect Accountable queries
* LUCENE-10170: Restore compression speed for LZ4.
This commit removes the dependency on the Joda library. It removes
many remaining references to joda, though not all because some comments
are worthwhile for historical reasoning.
Fix the split package org.elasticsearch.common.xcontent, between server and the x-content lib. Move the x-content lib exported package from org.elasticsearch.common.xcontent to org.elasticsearch.xcontent ( following the naming convention of similar libraries ). Removing split packages is a prerequisite to modularization.
Includes the following lucene changes:
* LUCENE-10145: Speed up byte[] comparisons using VarHandles.
* LUCENE-10143: Delegate primitive writes in RateLimitedIndexOutput
* LUCENE-10182: No longer check dvGen.
* LUCENE-10153: Speed up BKDWriter using VarHandles.
* LUCENE-10150: override ByteBuffersDataInput readLong/readInt/readShort
Includes the following changes:
- Updates to prevent package splits in ES: LUCENE-10118, LUCENE-10132
- Speedups in writing doc values: LUCENE-10127, LUCENE-10123, LUCENE-10125
- Speedups in writing primitives: LUCENE-10125
- Sort-after bugfixes: LUCENE-10126
Upgrade to a new Lucene 9 snapshot that includes LUCENE-10119 so we can
re-enable the sort optimization with points for scroll and search_after
requests.
Relates #78230
David Turner
Armin Braun
Luca Cavanna
David Kyle
Alan Woodward
Mark Vieira
Nikola Grcevski
Craig Taverner
tmgordeeva
Rene Groeschke
Ignacio Vera
Jack Conradson
Nhat Nguyen
Chris Hegarty
Alan Woodward
Julie Tibshirani
Mayya Sharipova
Przemyslaw Gomulka
Artem Prigoda
Rory Hunter
Ryan Ernst