The doc values codec iterates a few times over the doc value instance that needs to be written to disk. In case when merging and index sorting is enabled, this is much more expensive, as each time the doc values instance is iterated a merge sorting is performed (in order to get the doc ids of new segment in order of index sorting).
There are several reasons why the doc value instance is iterated multiple times:
* To compute stats (num values, number of docs with value) required for writing values to disk.
* To write bitset that indicate which documents have a value. (indexed disi, jump table)
* To write the actual values to disk.
* To write the addresses to disk (in case docs have multiple values)
This applies for numeric doc values, but also for the ordinals of sorted (set) doc values.
This PR addresses solving the first reason why doc value instance needs to be iterated. This is done only when in case of merging and when the segments to be merged with are also of type es87 doc values, codec version is the same and there are no deletes. Note this optimized merged is behind a feature flag for now.
This commit adds support for system data streams reindexing. The system data stream migration extends the existing system indices migration task and uses the data stream reindex API.
The system index migration task starts a reindex data stream task and tracks its status every second. Only one system index or system data stream is migrated at a time. If a data stream migration fails, the entire system index migration task will also fail.
Port of #123926
* [main] Move system indices migration to migrate plugin
It seems the best way to fix#122949 is to use existing data stream reindex API. However, this API is located in the migrate x-pack plugin. This commit moves the system indices migration logic (REST handlers, transport actions, and task) to the migrate plugin.
Port of #123551
* [CI] Auto commit changes from spotless
* Fix compilation
* Fix tests
* Fix test
---------
Co-authored-by: elasticsearchmachine <infra-root+elasticsearchmachine@elastic.co>
This action solely needs the cluster state, it can run on any node.
Since this is the last class/action that extends the `ClusterInfo`
abstract classes, we remove those classes too as they're not required
anymore.
Relates #101805
The change contains the following changes:
- The numDocsWithField field moved from SortedNumericEntry to NumericEntry. Making this statistic always available.
- Store jump table after values in ES87TSDBDocValuesConsumer#writeField(...). Currently it is stored before storing values. This will allow us later to iterate over the SortedNumericDocValues once. When merging, this is expensive as a merge sort on the fly is being executed.
This change will allow all the optimizations that are listed in #125403
A while ago we introduced a completion postings format extension to eventually
be able to customize how completion FSTs are loaded. See #111494.
We have never leveraged this extension, and meanwhile Lucene is moving
to always load FSTs off-heap, and no longer allow on-heap.
See https://github.com/apache/lucene/pull/14364 .
This commit removes the SPI extension as it is no longer needed.
Query interceptor is meant for internal modules to implement, not any
external plugin. Yet it is defined on SearchPlugin that is available to
all plugin authors. This commit creates an InternalSearchPlugin
interface and moves the query interceptor method to that.
Create a multi-project version of `FileSettingsService` to handle individual project settings files.
At the moment, there are no handlers for project-specific data, but it will create projects in the cluster based on the settings files that exist in the directory
The security codebase relies heavily on automata and caching these. The
Lucene 10 upgrade removed automaton minimization which can result in a
memory usage increase of >5x, esp. for roles with many application
privileges.
This PR brings back Automaton minimization to avoid the explosion in
roles cache size.
Relates: ES-10451
* Added query name to inference field metadata
* Fix build error
* Added query builder service
* Add query builder service to query rewrite context
* Updated match query to support querying semantic text fields
* Fix build error
* Fix NPE
* Update the POC to rewrite to a bool query when combined inference and non-inference fields
* Separate clause for each inference index (to avoid inference ID clashes)
* Simplify query builder service concept to a single default inference query
* Rename QueryBuilderService, remove query name from inference metadata
* Fix too many rewrite rounds error by injecting booleans in constructors for match query builder and semantic text
* Fix test compilation errors
* Fix tests
* Add yaml test for semantic match
* Add NodeFeature
* Fix license headers
* Spotless
* Updated getClass comparison in MatchQueryBuilder
* Cleanup
* Add Mock Inference Query Builder Service
* Spotless
* Cleanup
* Update docs/changelog/117839.yaml
* Update changelog
* Replace the default inference query builder with a query rewrite interceptor
* Cleanup
* Some more cleanup/renames
* Some more cleanup/renames
* Spotless
* Checkstyle
* Convert List<QueryRewriteInterceptor> to Map keyed on query name, error on query name collisions
* PR feedback - remove check on QueryRewriteContext class only
* PR feedback
* Remove intercept flag from MatchQueryBuilder and replace with wrapper
* Move feature to test feature
* Ensure interception happens only once
* Rename InterceptedQueryBuilderWrapper to AbstractQueryBuilderWrapper
* Add lenient field to SemanticQueryBuilder
* Clean up yaml test
* Add TODO comment
* Add comment
* Spotless
* Rename AbstractQueryBuilderWrapper back to InterceptedQueryBuilderWrapper
* Spotless
* Didn't mean to commit that
* Remove static class wrapping the InterceptedQueryBuilderWrapper
* Make InterceptedQueryBuilderWrapper part of QueryRewriteInterceptor
* Refactor the interceptor to be an internal plugin that cannot be used outside inference plugin
* Fix tests
* Spotless
* Minor cleanup
* C'mon spotless
* Test spotless
* Cleanup InternalQueryRewriter
* Change if statement to assert
* Simplify template of InterceptedQueryBuilderWrapper
* Change constructor of InterceptedQueryBuilderWrapper
* Refactor InterceptedQueryBuilderWrapper to extend QueryBuilder
* Cleanup
* Add test
* Spotless
* Rename rewrite to interceptAndRewrite in QueryRewriteInterceptor
* DOESN'T WORK - for testing
* Add comment
* Getting closer - match on single typed fields works now
* Deleted line by mistake
* Checkstyle
* Fix over-aggressive IntelliJ Refactor/Rename
* And another one
* Move SemanticMatchQueryRewriteInterceptor.SEMANTIC_MATCH_QUERY_REWRITE_INTERCEPTION_SUPPORTED to Test feature
* PR feedback
* Require query name with no default
* PR feedback & update test
* Add rewrite test
* Update server/src/main/java/org/elasticsearch/index/query/InnerHitContextBuilder.java
Co-authored-by: Mike Pellegrini <mike.pellegrini@elastic.co>
---------
Co-authored-by: Mike Pellegrini <mike.pellegrini@elastic.co>
This measurably improves BBQ by adjusting the underlying algorithm to an
optimized per vector scalar quantization.
This is a brand new way to quantize vectors. Instead of there being a
global set of upper and lower quantile bands, these are optimized and
calculated per individual vector. Additionally, vectors are centered on
a common centroid.
This allows for an almost 32x reduction in memory, and even better
recall than before at the cost of slightly increasing indexing time.
Additionally, this new approach is easily generalizable to various other
bit sizes (e.g. 2 bits, etc.). While not taken advantage of yet, we may
update our scalar quantized indices in the future to use this new
algorithm, giving significant boosts in recall.
The recall gains spread from 2% to almost 10% for certain datasets with
an additional 5-10% indexing cost when indexing with HNSW when compared
with current BBQ.
This will make `TransportLocalClusterStateAction` wait for a new state
that is not blocked. This means we need a timeout (again). For
consistency's sake, we're reusing the REST param `master_timeout` for
this timeout as well.
The only class that was using `TransportLocalClusterStateAction` was
`TransportGetAliasesAction`, so its request needed to accept a timeout
again as well.
Historical features are now trivially true on v9 - so we can remove the features, and the check.
Historical features do not affect cluster state, so this has no compatibility restrictions.
* Refactor: treat "maybe" JVM options uniformly
* WIP
* Get entitlement running with bridge all the way through, with qualified
exports
* Cosmetic changes to SystemJvmOptions
* Disable entitlements by default
* Bridge module comments
* Fixup forbidden APIs
* spotless
* Rename EntitlementChecker
* Fixup InstrumenterTests
* exclude recursive dep
* Fix some compliance stuff
* Rename asm-provider
* Stop using bridge in InstrumenterTests
* Generalize readme for asm-provider
* InstrumenterTests doesn't need EntitlementCheckerHandle
* Better javadoc
* Call parseBoolean
* Add entitlement to internal module list
* Docs as requested by Lorenzo
* Changes from Jack
* Rename ElasticsearchEntitlementChecker
* Remove logging javadoc
* exportInitializationToAgent should reference EntitlementInitialization, not EntitlementBootstrap.
They're currently in the same module, but if that ever changes, this code would have become wrong.
* Some suggestions from Mark
---------
Co-authored-by: Ryan Ernst <ryan@iernst.net>
* Adding API to get list of service configurations
* Update docs/changelog/114862.yaml
* Fixing some configurations
* PR feedback -> Stream.of
* PR feedback -> singleton
* Renaming ServiceConfiguration to SettingsConfiguration. Adding TaskSettingsConfiguration
* Adding task type settings configuration to response
* PR feedback
The most relevant ES changes that upgrading to Lucene 10 requires are:
- use the appropriate IOContext
- Scorer / ScorerSupplier breaking changes
- Regex automaton are no longer determinized by default
- minimize moved to test classes
- introduce Elasticsearch900Codec
- adjust slicing code according to the added support for intra-segment concurrency
- disable intra-segment concurrency in tests
- adjust accessor methods for many Lucene classes that became a record
- adapt to breaking changes in the analysis area
Co-authored-by: Christoph Büscher <christophbuescher@posteo.de>
Co-authored-by: Mayya Sharipova <mayya.sharipova@elastic.co>
Co-authored-by: ChrisHegarty <chegar999@gmail.com>
Co-authored-by: Brian Seeders <brian.seeders@elastic.co>
Co-authored-by: Armin Braun <me@obrown.io>
Co-authored-by: Panagiotis Bailis <pmpailis@gmail.com>
Co-authored-by: Benjamin Trent <4357155+benwtrent@users.noreply.github.com>
This removes the possibility for a plugin to provide factory retention settings. Factory retention settings have been deprecated and completely replaced by #111972.
Note: this feature is not in use. If someone wants to set global retention they can use the cluster settings as defined in #111972.
Including the cluster state in responses to the `POST _cluster/state`
API was deprecated in #90399 (v8.6.0) requiring callers to pass
`?metric=none` to avoid the deprecation warning. This commit adjusts the
behaviour as promised in v9 so that this API never returns the cluster
state, and deprecates the `?metric` parameter itself.
Closes#88978
Ryan Ernst
Martijn van Groningen
Alexey Ivanov
Niels Bauman
Luca Cavanna
Simon Cooper
Tim Vernum
Niels Bauman
Chris Hegarty
Kostas Krikellas
Nikolaj Volgushev
Yang Wang
Kathleen DeRusso
Benjamin Trent
John Verwolf
Alexis Charveriat
Patrick Doyle
Nhat Nguyen
Ying Mao
Mary Gouseti
David Turner
Albert Zaharovits