This commit adds a new test framework for configuring and orchestrating
test clusters for both Java and YAML REST testing. This will eventually
replace the existing "test-clusters" Gradle plugin and the build-time
cluster orchestration.
No need for this extension, we don't make use of the settings or deprecation logger
in production any more. Also, this slows down CS operations that require a
temporary index service which builds quite a bit slower when the loggers
need to be set up via reflective calls.
This PR uses Lucene-9.3 snapshot in Elasticsearch 8.4. Noticeable changes in this Lucene snapshot:
- Merge-on-refresh (disabled)
- No more pathological merging
- SortedSetDocValues#count for value_count aggs
Final (hopefully!) snapshot before the 9.2.0 release
* Update test to expect persian tokenfilter - will be exposed later
* Fix KnnVectorQueryBuilderTests::doAssertLuceneQuery
Co-authored-by: Mayya Sharipova <mayya.sharipova@elastic.co>
Remove the inheritance here to make instances smaller and speed up many-shards benchmarks a little.
Did not remove the dead arguments from the constructors in this PR as that would have been a
very noisy change.
Notable changes include:
count implementations for MultiRangeQuery and IndexSortedNumericDocValuesRangeQuery, which may speed up certain aggregations
more efficient decoding of docids in BKD reader
This PR upgrades Lucene to a newer snapshot `9.1.0-snapshot-9497524cc2d`.
Changes:
* Adapt to `LeafReader#searchNearestVectors` signature change
* Adapt checks in `GeometryIndexerTests`, `SearchServiceTests`, `FiltersAggregatorTests`, `AggregationProfilerIT`
* Address highlighting failures in `MultiPhrasePrefixQuery` and `HasChildQueryBuilder.LateParsingQuery`
Lucene issues that resulted in elasticsearch changes:
LUCENE-9820 Separate logic for reading the BKD index from logic to intersecting it.
LUCENE-10377: Replace 'sortPos' with 'enableSkipping' in SortField.getComparator()
LUCENE-10301: make the test-framework a proper module by moving all test
classes to org.apache.lucene.tests
LUCENE-10300: rewrite how resources are read in ukrainian morfologik analyzer:
LUCENE-10054 Make HnswGraph hierarchical
Follow-up from #77144 (comment) with converting id/_id to always be strings instead of integers. This makes the type value in the Elasticsearch specification be only string instead of string | number.
this change was generated using following command on ubuntu
find . -type f -name "*.yml" -print0 | xargs -0 sed -i -r 's/([^a-zA-Z0-9_\.]id|[^a-zA-Z0-9_]_id):(\s*)([0-9]+)/\1:\2"\3"/g'
This includes the following changes:
* LUCENE-10180: Avoid using lambdas in SegmentMerger
* LUCENE-10187: Reduce DirectWriter's padding
* LUCENE-10193: Cut over more array access to VarHandles
* LUCENE-10189: Optimize flush of doc-value fields that are effectively single-valued
* LUCENE-10165: Implement Lucene90DocValuesProducer#getMergeInstance
Includes the following new commits of interest:
* LUCENE-10150: override readLongs() in ByteBuffersDataInput
* LUCENE-10146: Add VectorSimilarityFunction.COSINE
* LUCENE-10140: Correct minimizing iterator sub-matches
* LUCENE-10103 Make QueryCache respect Accountable queries
* LUCENE-10170: Restore compression speed for LZ4.
Includes the following lucene changes:
* LUCENE-10145: Speed up byte[] comparisons using VarHandles.
* LUCENE-10143: Delegate primitive writes in RateLimitedIndexOutput
* LUCENE-10182: No longer check dvGen.
* LUCENE-10153: Speed up BKDWriter using VarHandles.
* LUCENE-10150: override ByteBuffersDataInput readLong/readInt/readShort
Includes the following changes:
- Updates to prevent package splits in ES: LUCENE-10118, LUCENE-10132
- Speedups in writing doc values: LUCENE-10127, LUCENE-10123, LUCENE-10125
- Speedups in writing primitives: LUCENE-10125
- Sort-after bugfixes: LUCENE-10126
Upgrade to a new Lucene 9 snapshot that includes LUCENE-10119 so we can
re-enable the sort optimization with points for scroll and search_after
requests.
Relates #78230
The analysis-smartcn plugin reuses server the server package name for
analysis. This commit moves the plugin implementation to use a single
package name, o.e.p.analysis.smartcn
This introduces a basic public yaml rest test plugin that is supposed to be used by external
elasticsearch plugin authors. This is driven by #76215
- Rename yaml-rest-test to intern-yaml-rest-test
- Use public yaml plugin in example plugins
Co-authored-by: Mark Vieira <portugee@gmail.com>
Modularization of the JDK has been ongoing for several years. Recently
in Java 16 the JDK began enforcing module boundaries by default. While
Elasticsearch does not yet use the module system directly, there are
some side effects even for those projects not modularized (eg #73517).
Before we can even begin to think about how to modularize, we must
Prepare The Way by enforcing packages only exist in a single jar file,
since the module system does not allow packages to coexist in multiple
modules.
This commit adds a precommit check to the build which detects split
packages. The expectation is that we will add the existing split
packages to the ignore list so that any new classes will not exacerbate
the problem, and the work to cleanup these split packages can be
parallelized.
relates #73525
Upgrades to Lucene-8.9 snapshot which includes:
- LUCENE-9507: Custom order for leaves (/cc @mayya-sharipova)
- LUCENE-9935: Enable bulk merge for stored fields with index sort
As per the new licensing change for Elasticsearch and Kibana this commit
moves existing Apache 2.0 licensed source code to the new dual license
SSPL+Elastic license 2.0. In addition, existing x-pack code now uses
the new version 2.0 of the Elastic license. Full changes include:
- Updating LICENSE and NOTICE files throughout the code base, as well
as those packaged in our published artifacts
- Update IDE integration to now use the new license header on newly
created source files
- Remove references to the "OSS" distribution from our documentation
- Update build time verification checks to no longer allow Apache 2.0
license header in Elasticsearch source code
- Replace all existing Apache 2.0 license headers for non-xpack code
with updated header (vendored code with Apache 2.0 headers obviously
remains the same).
- Replace all Elastic license 1.0 headers with new 2.0 header in xpack.
This change upgrades to the latest Lucene 8.8.0 snapshot.
It also restores the compression on binary doc values that was lost in the last snapshot upgrade.
The compression is now configurable on binary doc values but we don't expose this functionality yet so this commit ensures that we pick the same compression mode as previous releases (BEST_COMPRESSION).
This new snapshot contains the following JIRAs that we're interested in:
- [LUCENE-9525](https://issues.apache.org/jira/browse/LUCENE-9525)
Better handling of small documents. This should improve retrieval times
when documents are less than ~1kB.
- [LUCENE-9510](https://issues.apache.org/jira/browse/LUCENE-9510)
Faster flushes when index sorting is enabled by not compressing the
temporary files that store stored fields and term vectors.
Mark Vieira
Ignacio Vera
Nhat Nguyen
Armin Braun
Alan Woodward
Alan Woodward
Julie Tibshirani
Mayya Sharipova
Przemyslaw Gomulka
Ryan Ernst
Rene Groeschke
Jake Landis
Ignacio Vera
Jim Ferenczi
Adrien Grand