The security codebase relies heavily on automata and caching these. The
Lucene 10 upgrade removed automaton minimization which can result in a
memory usage increase of >5x, esp. for roles with many application
privileges.
This PR brings back Automaton minimization to avoid the explosion in
roles cache size.
Relates: ES-10451
* Added query name to inference field metadata
* Fix build error
* Added query builder service
* Add query builder service to query rewrite context
* Updated match query to support querying semantic text fields
* Fix build error
* Fix NPE
* Update the POC to rewrite to a bool query when combined inference and non-inference fields
* Separate clause for each inference index (to avoid inference ID clashes)
* Simplify query builder service concept to a single default inference query
* Rename QueryBuilderService, remove query name from inference metadata
* Fix too many rewrite rounds error by injecting booleans in constructors for match query builder and semantic text
* Fix test compilation errors
* Fix tests
* Add yaml test for semantic match
* Add NodeFeature
* Fix license headers
* Spotless
* Updated getClass comparison in MatchQueryBuilder
* Cleanup
* Add Mock Inference Query Builder Service
* Spotless
* Cleanup
* Update docs/changelog/117839.yaml
* Update changelog
* Replace the default inference query builder with a query rewrite interceptor
* Cleanup
* Some more cleanup/renames
* Some more cleanup/renames
* Spotless
* Checkstyle
* Convert List<QueryRewriteInterceptor> to Map keyed on query name, error on query name collisions
* PR feedback - remove check on QueryRewriteContext class only
* PR feedback
* Remove intercept flag from MatchQueryBuilder and replace with wrapper
* Move feature to test feature
* Ensure interception happens only once
* Rename InterceptedQueryBuilderWrapper to AbstractQueryBuilderWrapper
* Add lenient field to SemanticQueryBuilder
* Clean up yaml test
* Add TODO comment
* Add comment
* Spotless
* Rename AbstractQueryBuilderWrapper back to InterceptedQueryBuilderWrapper
* Spotless
* Didn't mean to commit that
* Remove static class wrapping the InterceptedQueryBuilderWrapper
* Make InterceptedQueryBuilderWrapper part of QueryRewriteInterceptor
* Refactor the interceptor to be an internal plugin that cannot be used outside inference plugin
* Fix tests
* Spotless
* Minor cleanup
* C'mon spotless
* Test spotless
* Cleanup InternalQueryRewriter
* Change if statement to assert
* Simplify template of InterceptedQueryBuilderWrapper
* Change constructor of InterceptedQueryBuilderWrapper
* Refactor InterceptedQueryBuilderWrapper to extend QueryBuilder
* Cleanup
* Add test
* Spotless
* Rename rewrite to interceptAndRewrite in QueryRewriteInterceptor
* DOESN'T WORK - for testing
* Add comment
* Getting closer - match on single typed fields works now
* Deleted line by mistake
* Checkstyle
* Fix over-aggressive IntelliJ Refactor/Rename
* And another one
* Move SemanticMatchQueryRewriteInterceptor.SEMANTIC_MATCH_QUERY_REWRITE_INTERCEPTION_SUPPORTED to Test feature
* PR feedback
* Require query name with no default
* PR feedback & update test
* Add rewrite test
* Update server/src/main/java/org/elasticsearch/index/query/InnerHitContextBuilder.java
Co-authored-by: Mike Pellegrini <mike.pellegrini@elastic.co>
---------
Co-authored-by: Mike Pellegrini <mike.pellegrini@elastic.co>
This measurably improves BBQ by adjusting the underlying algorithm to an
optimized per vector scalar quantization.
This is a brand new way to quantize vectors. Instead of there being a
global set of upper and lower quantile bands, these are optimized and
calculated per individual vector. Additionally, vectors are centered on
a common centroid.
This allows for an almost 32x reduction in memory, and even better
recall than before at the cost of slightly increasing indexing time.
Additionally, this new approach is easily generalizable to various other
bit sizes (e.g. 2 bits, etc.). While not taken advantage of yet, we may
update our scalar quantized indices in the future to use this new
algorithm, giving significant boosts in recall.
The recall gains spread from 2% to almost 10% for certain datasets with
an additional 5-10% indexing cost when indexing with HNSW when compared
with current BBQ.
This will make `TransportLocalClusterStateAction` wait for a new state
that is not blocked. This means we need a timeout (again). For
consistency's sake, we're reusing the REST param `master_timeout` for
this timeout as well.
The only class that was using `TransportLocalClusterStateAction` was
`TransportGetAliasesAction`, so its request needed to accept a timeout
again as well.
Historical features are now trivially true on v9 - so we can remove the features, and the check.
Historical features do not affect cluster state, so this has no compatibility restrictions.
* Refactor: treat "maybe" JVM options uniformly
* WIP
* Get entitlement running with bridge all the way through, with qualified
exports
* Cosmetic changes to SystemJvmOptions
* Disable entitlements by default
* Bridge module comments
* Fixup forbidden APIs
* spotless
* Rename EntitlementChecker
* Fixup InstrumenterTests
* exclude recursive dep
* Fix some compliance stuff
* Rename asm-provider
* Stop using bridge in InstrumenterTests
* Generalize readme for asm-provider
* InstrumenterTests doesn't need EntitlementCheckerHandle
* Better javadoc
* Call parseBoolean
* Add entitlement to internal module list
* Docs as requested by Lorenzo
* Changes from Jack
* Rename ElasticsearchEntitlementChecker
* Remove logging javadoc
* exportInitializationToAgent should reference EntitlementInitialization, not EntitlementBootstrap.
They're currently in the same module, but if that ever changes, this code would have become wrong.
* Some suggestions from Mark
---------
Co-authored-by: Ryan Ernst <ryan@iernst.net>
* Adding API to get list of service configurations
* Update docs/changelog/114862.yaml
* Fixing some configurations
* PR feedback -> Stream.of
* PR feedback -> singleton
* Renaming ServiceConfiguration to SettingsConfiguration. Adding TaskSettingsConfiguration
* Adding task type settings configuration to response
* PR feedback
The most relevant ES changes that upgrading to Lucene 10 requires are:
- use the appropriate IOContext
- Scorer / ScorerSupplier breaking changes
- Regex automaton are no longer determinized by default
- minimize moved to test classes
- introduce Elasticsearch900Codec
- adjust slicing code according to the added support for intra-segment concurrency
- disable intra-segment concurrency in tests
- adjust accessor methods for many Lucene classes that became a record
- adapt to breaking changes in the analysis area
Co-authored-by: Christoph Büscher <christophbuescher@posteo.de>
Co-authored-by: Mayya Sharipova <mayya.sharipova@elastic.co>
Co-authored-by: ChrisHegarty <chegar999@gmail.com>
Co-authored-by: Brian Seeders <brian.seeders@elastic.co>
Co-authored-by: Armin Braun <me@obrown.io>
Co-authored-by: Panagiotis Bailis <pmpailis@gmail.com>
Co-authored-by: Benjamin Trent <4357155+benwtrent@users.noreply.github.com>
This removes the possibility for a plugin to provide factory retention settings. Factory retention settings have been deprecated and completely replaced by #111972.
Note: this feature is not in use. If someone wants to set global retention they can use the cluster settings as defined in #111972.
Including the cluster state in responses to the `POST _cluster/state`
API was deprecated in #90399 (v8.6.0) requiring callers to pass
`?metric=none` to avoid the deprecation warning. This commit adjusts the
behaviour as promised in v9 so that this API never returns the cluster
state, and deprecates the `?metric` parameter itself.
Closes#88978
Regardless of JDK version, ES should always use CLDR locale database from 9.0.0.
This also removes IsoCalendarDataProvider used to override week-date calculations for the root locale only.
Extensible plugins use a custom classloader for other plugin jars. When
extensible plugins were first added, the transport client still existed,
and elasticsearch plugins did not exist in the transport client (at
least not the ones that create classloaders). Yet the transport client
still created a PluginsService. An indirection was used to avoid
creating separate classloaders when the transport client had created the
PluginsService.
The transport client was removed in 8.0, but the indirection still
exists. This commit removes that indirection layer.
* Initial new injector
* Allow createComponents to return classes
* Downsample injection
* Remove more vestiges of subtype handling
* Lowercase logger
* Respond to code review comments
* Only one object per class
* Some additional cleanup incl spotless
* PR feedback
* Missed one
* Rename workQueue
* Remove Injector.addRecordContents
* TelemetryProvider requires us to inject an object using a supertype
* Address Simon's comments
* Clarify the reason for SuppressForbidden
* Make log indentation code less intrusive
Adds to the `GET _cluster/stats` endpoint information about the snapshot
repositories in use, including their types, whether they are read-only
or read-write, and for Azure repositories the kind of credentials in
use.
The initial goal of this PR was to make the "put ILM" action go through the project resolver in order to resolve the project-scoped metadata, and hence avoid referring to the whole cluster state. This implies changing some methods to work on the project metadata rather than the whole cluster metadata.
It turns out, due to good code reuse, it is hard to only change one specific action to only refer to project-scoped metadata.
Tim Vernum
Nikolaj Volgushev
Yang Wang
Kathleen DeRusso
Niels Bauman
Benjamin Trent
Niels Bauman
Simon Cooper
John Verwolf
Martijn van Groningen
Alexis Charveriat
Patrick Doyle
Nhat Nguyen
Ying Mao
Luca Cavanna
Mary Gouseti
David Turner
Albert Zaharovits
Chris Hegarty
Ryan Ernst
Mark Vieira
Panagiotis Bailis
Keith Massey
Joe Gallo