XFeatureField was added in conjunction with a required update for
SparseVectorFields. However, with the upgrade to Lucene 10.1+, it has
outstayed its usefulness.
There are only a handful of relevant 5xx http error codes. Yet
distinguishing errors can be important for debugging large numbers of
errors. This commit adds a new `X-Elasticsearch-Exception` response
header to 5xx error responses which contains the class name of the Java
exception.
The ingest service dispatches to the provided executor. This work became
redundant several years ago when we moved the transport bulk action to
be dispatched from the transport threads for the action itself. With
this behavior, there is no reason to re-dispatch an ingest action.
The test submits merge tasks that support IO throttling, and asserts
that all the currently running merge tasks are indeed IO throttled after
the new one was submitted.
The test erroneously tried to assert a property on the set of currently
running merge tasks, which is very difficult to do since all merge tasks
are possibly backlogged and re-enqueued asynchronously multiple times
before they are run or aborted (so looking at the threadpool merge task
queue there's no telling which merge task will execute first).
Fixes https://github.com/elastic/elasticsearch/issues/129531
In #129825, we modified the dense_vector field type to delay setting index options until the field's dimensions are known. However, this introduced a discrepancy for indices created before that change, which would previously default to int8_hnsw even when dimensions were not set.
This discrepancy leads to an assertion failure in mixed-version clusters, where the serialized mappings differ between nodes:
```
[2025-07-02T20:37:29,852][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [v9.0.4-2] fatal error in thread [elasticsearch[v9.0.4-2][clusterApplierService#updateTask][T#1]], exiting java.lang.AssertionError: provided source [{"_doc":{"properties":{"vector":{"type":"dense_vector","index":true,"similarity":"cosine"}}}}] differs from mapping [{"_doc":{"properties":{"vector":{"type":"dense_vector","index":true,"similarity":"cosine","index_options":{"type":"int8_hnsw","m":16,"ef_construction":100}}}}}]
```
This commit resolves the issue by ensuring that indices created before the change continue to default to int8_hnsw index options, even if dimensions remain unset.
There isn't a '_remote_access' profile, it looks like the
author meant to write '_remote_cluster'. This commit changes
it to REMOTE_CLUSTER_PROFILE to match the code below and avoid
confusing the reader with a profile name that doesn't exist.
In Java 14 the `MessageDigest` specification was changed so that the
"MD5" hash function is no longer required. It is permissible for a JRE
to ship without support for MD5 hashes.
This commit modifies the ES|QL MD5 hash function implementation so
that if the specified `MessageDigest` object is not available on startup,
then the error is non-fatal, and the node will still boot successfully.
If an ES|QL query attempts to make use of md5, and it is unavailable,
then the query will fail with an ES|QL verification exception
Resolves: #129689
This PR ensures the node responses are copied and drained exclusively in
onCompletion so that they do not get concurrently modified by
cancellation.
Resolves: #128852
We used to have an assertion during transport version loading that
duplicate ids were not found, but it appears to have been lost in
refactorings. This commit adds a test to ensure duplicate ids do not
occur.
relates #130486
* Add Ibm Granite Completion and Chat Completion support
* Apply suggestions
* remove ibm watsonx transport version constant
* update transport version
Project deletions is used to update the Stateless lease blob, as a project deletion
is a notable event that that our stateless cluster consistency check should consider
before acknowledging writes.
Relates ES-11207
* Fix ExceptionSerializationTests to use getCodeSource instead of getResource.
Using getResource makes this sensitive to unrelated classpath entries,
such as the entitlement bridge library, that get prepended to the classpath.
* FIx logging tests to use org.elasticsearch.index instead of root logger.
Using the root logger makes this sensitive to unrelated
logging, such as from the entitlement library.
* Fix entitlement error message by stashing the module name in ModuleEntitlements.
Taking the actual module name from the class doesn't work in tests,
where those classes are loaded from the classpath and so their module
info is misleading.
* Ignore server locations whose representative class isn't loaded
* Partial initial implementation
* System properties: testOnlyClasspath and enableForTests
* Trivially allow some packages
* DEBUG: use TreeMap in TestScopeResolver for readability
* Special case bouncycastle for security plugin
* Add CONFIG to TestPathLookup
* Add the classpath to the source path list for every plugin
* Add @WithoutEntitlements to tests that run ES nodes
* Set es.entitlement.enableForTests for all libs
* Use @WithoutEntitlements on ingest plugin tests
* Substitute ALL-UNNAMED for module name in non-modular plugins
* Add missing entitlements found by unit tests
* Comment in TestScopeResolver
* Properly compute bridge jar location for patch-module
* Call out nonServerLibs
* Don't build two TestPathLookups
* More comments for meta-tests
* Remove redundant dependencies for bridgeJarConfig.
These are alread set in ElasticsearchJavaBasePlugin.
* Add bridge+agent dependencies only if those exist.
For serverless, those project dependencies don't exist, and we'll need
to add the dependencies differently, using Maven coordinates.
* [CI] Auto commit changes from spotless
* Pass testOnlyPath in environment instead of command line.
It's typically a very very long string, which made Windows angry.
* [CI] Auto commit changes from spotless
* Split testOnlyPathString at File.pathSeparator
* Use doFirst to delay setting testOnlyPath env var
* Trivially allow jimfs (??)
* Don't enforce entitlements on internalClusterTest for now
* Replace forbidden APIs
* Match testOnlyClasspath using URI instead of String.
We already get the "needle" in the form of a URI, so this skips
a step, and has the benefit of also working on Windows.
* [CI] Auto commit changes from spotless
* More forbidden APIs
* Disable configuration cache for LegacyYamlRestTestPluginFuncTest
* Strip carriage-return characters in expected output for ReleaseNotesGeneratorTest.
The template generator also strips these, so we need to do so to make this pass
on Windows.
Note that we use replace("\r", "") where the template generator uses
replace("\\r", ""). The latter didn't work for me when I tried it on Windows,
for reasons I'm not aware of.
* Move configureEntitlements to ElasticsearchTestBasePlugin as-is
* Use matching instead of if
* Remove requireNonNull
* Remove default configuration
* Set inputs instead of dependencies
* Use test.systemProperty
* Respond to PR comments
* Disable entitlement enforcement for ScopedSettingsTests.
This test works by altering the logging on the root logger.
With entitlements enabled, that will cause additional log statements to appear,
which interferes with the test.
* Address PR comments
* Moritz's configureJavaBaseModuleOptions
* Allow for entitlements not yet enforced in serverless
* fix entitlementBridge config after rename
* drop empty file collections
* Remove workaround in LegacyYamlRestTestPluginFuncTest
---------
Co-authored-by: elasticsearchmachine <infra-root+elasticsearchmachine@elastic.co>
Co-authored-by: Lorenzo Dematté <lorenzo.dematte@elastic.co>
Co-authored-by: Moritz Mack <mmack@apache.org>
Relates to #128621
This is a first step in making the ES|QL query planner aware of TSDB Dimensions and Metric field metadata. This is purposefully small to only touch the serialization change we'll need for this. The plan is get the TSDB metadata field type out of Field Caps and to store this information on EsField. This PR adds a place to store such a field, and adds it to the serialization for EsField and its sub-classes.
As of this PR, we don't do anything with this data. That's intentional, to minimize the footprint of the transport version change. Further PRs in this project will load and act on this data. I've added some constructors here to minimize the number of files I'm touching in this PR. I hope that as we begin loading this data (as opposed to just defaulting it right now) we can get rid of some of these default value constructors.
Previously, out of zealousness for testing efficiency, the mocked
filesystems were reused across the test suite class. But this makes
tests liable to interference wrt to filesystem stats calls. Moreover, if
one test fails, it can trigger failures in other test methods.
This PR recreates the mocked filesystems for every test method.
Fixes#129296#130205
optimize OptimizedScalarQuantizer#scalarQuantize when destination can optimize
OptimizedScalarQuantizer#scalarQuantize when destination can be an integer array
When return an error from Elasticsearch exceptions may contain values
written as http response headers. ElasticsearchException contains a map
of headers that are added to the response. But these values are also
written to a special "header" section of the response body.
This commit renames the existing "headers" in ElasticsearchException to
"body headers", which are both http headers and written to the response
body. A new "http headers" is added for headers that should only be
written as response headers.
This is three fixes:
- We should be doing actual sampling when doing kmeans clustering, taking the first N vectors creates some weird edge cases
- Having assignments initialized as `0` means that if a vector gets assigned to cluster ord `0`, that cluster centroid actually isn't updated later in the lloyd steps. So, this initializes assignments to -1
- If we actually don't sample the vectors for lloyd, don't bother with final pass to potentially update the centroids
this is a small refactor, laying ground work for more generalized bulk
writing.
I did some benchmarking and there was no significant performance
difference (as expected).
There was a bug in previous version where flattened fields would produce incorrect synthetic source with too few opening braces. This bug was fixed as a side effect of #129600. Adding this test to confirm. See #129600 for a full explanation.
Passes the minimum transport version down to expressions when
we convert them into queries that we'll use for can_match.
Right now all this is used for is skipping the can_match from
the wildcard like queries. The queries we make there aren't
serializable. We'll fix that - but this should give us the
levers that we need to do it in a backwards incompatible way.
When running under security manager an assumptionw as made that failing
to create a hard link due to security exception implied hard links were
supported. Now that security manager is gone, the code to create a hard
link in StoreRecoveryTests executes. But in the case of windows,
BasicFileAttributes.fileKey does not return a unique object that can be
used to verify a link exists. Yet the fact createLink returned is enough
to trust the jdk was able to create a link.
closes#124104
For dense vector fields using the `flat` index, we already know a brute-force search will be used—so there’s no need to go through the codec’s approximate KNN logic. This change skips that step and builds the brute-force query directly, making things faster and simpler.
I tested this on a setup with **10 million random vectors**, each with **1596 dimensions** and **17,500 partitions**, using the `random_vector` track.
The results:
### Performance Comparison
| Metric | Before | After | Change |
| ----------------- | --------- | ---------- | --------- |
| **Throughput** | 221 ops/s | 2762 ops/s | 🟢 +1149% |
| **Latency (p50)** | 29.2 ms | 1.6 ms | 🔻 -94.4% |
| **Latency (p99)** | 81.6 ms | 3.5 ms | 🔻 -95.7% |
Filtered KNN queries on flat vectors are now over 10x faster on my laptop!
The enqueued task can run before the code reaches the status update
line. When it happens, the status update can set the status backwards.
This PR fixes it by moving the status update before enqueuing the task.
Resolves: #129752
Pooya Salehi
Ignacio Vera
Benjamin Trent
Ryan Ernst
Tim Brooks
Albert Zaharovits
Jim Ferenczi
Jeremy Dahlgren
Tim Vernum
Yang Wang
Oleksandr Kolomiiets
Evgenii-Kazannik
Mark Tozzi
Patrick Doyle
Parker Timmins
Dimitris Rempapis
Jordan Powers
James Baiera
Keith Massey
Nik Everett
Tommaso Teofili
Martijn van Groningen
Ievgen Degtiarenko
Mike Pellegrini
Sam Xiao