Elasticsearch requires access to some native functions. Historically
this has been achieved with the JNA library. However, JNA is a
complicated, magical library, and has caused various problems booting
Elasticsearch over the years. The new Java Foreign Function and Memory
API allows access to call native functions directly from Java. It also
has the advantage of tight integration with hotspot which can improve
performance of these functions (though performance of Elasticsearch's
native calls has never been much of an issue since they are mostly at
boot time).
This commit adds a new native lib that is internal to Elasticsearch. It
is built to use the foreign function api starting with Java 21, and
continue using JNA with Java versions below that.
Only one function, checking whether Elasticsearch is running as root, is
migrated. Future changes will migrate other native functions.
Preallocate opens a FileInputStream in order to get a native file
desctiptor to pass to native functions. However, getting at the file
descriptor requires breaking modular access. This commit adds native
posix functions for opening/closing and retrieving stats on a file in
order to avoid requiring additional permissions.
Qualfied exports in the boot layer only work when they are to other boot
modules. Yet Elasticsearch has dynamically loaded modules as in plugins.
For this purpose we have ModuleQualifiedExportsService. This commit
moves loading of ModuleQualfiedExportService instances in the boot layer
into core so that it can be reused by ProviderLocator when a qualified
export applies to an embedded module.
We have some usages of SearchException that don't provide a cause exception and also don't define a status code. That means that the status code of such requests will default to 500 which is in many cases not a good choice. Normally, for internal server error a cause is associated with the wrapper exception.
This scenario is not very common, and looks like a leftover of shard validation that used to happen on shards, which can be moved to the coordinating node. This commit moves some of the exceptions thrown in SearchService#parseSource to SearchRequest#validate. This way we will fail before serializing the shard level request to all the shards, which is much better.
Note that for bw comp reasons, we need to keep on throwing the same exception from the data node, while intuitively this is now replaced by the same validation in the coord node. This is because in a mixed cluster scenario, an older node that does not perform the validation as coord node, could serialize shard level requests that need to be checked again on data nodes, to prevent unexpected situations.
This avoids keeping downsamplingInterval field around. Additionally, the
downsample interval is known when downsample interval is invoked and
doesn't change.
This PR extends the repository integrity health indicator to cover also unknown and invalid repositories. Because these errors are local to a node, we extend the `LocalHealthMonitor` to monitor the repositories and report the changes in their health regarding the unknown or invalid status.
To simplify this extension in the future, we introduce the `HealthTracker` abstract class that can be used to create new local health checks.
Furthermore, we change the severity of the health status when the repository integrity indicator reports unhealthy from `RED` to `YELLOW` because even though this is a serious issue, there is no user impact yet.
* Move doc-values classes needed by ST_INTERSECTS to server
This classes are needed by ESQL spatial queries, and are not licensed in a way that prevents this move.
Since they depend on lucene it is not possible to move them to a library.
Instead they are moved to be co-located with the GeoPoint doc-values classes that already exist in server.
* Moved to lucene package org.elasticsearch.lucene.spatial
* Moved Geo/ShapeDocValuesQuery to server because it is Lucene specific
And this gives us access to these classes from ESQL for lucene-pushdown of spatial queries.
With this commit we refactor the internal representation of stacktraces
to use plain arrays instead of lists for some of its properties. The
motivation behind this change is simplicity:
* It avoids unnecessary boxing
* We could eliminate a few redundant null checks because we use
primitive types now in some places
* We could slightly simplify runlength decoding
Improve downsampling by making the following changes:
- Avoid NPE and assert tripping when fetching the last processed tsid.
- If the write block has been set, then there is no reason to start the downsample persistent tasks, since shard level downsampling has completed. Not doing so also causes ILM/DSL to get stuck on downsampling. In this case shard level downsampling should be skipped.
- Sometimes the source index may not be allocated yet on the node performing shard level downsampling operation. This causes a NPE, with this PR, this now fails a shard level downsample with a less disturbing error.
Additionally unmute
DataStreamLifecycleDownsampleDisruptionIT#testDataStreamLifecycleDownsampleRollingRestart
Relates to #105068
The unconditional rollover that is a consequence of a lazy rollover command is triggered by the creation of a document. In many cases, the user triggering this rollover won't have sufficient privileges to ensure the successful execution of this rollover. For this reason, we introduce a dedicated rollover action and a dedicated internal user to cover this case and enable this functionality.
Without a change in behavior, we can remove the ununsed ListValues as well as most
of the allocations when serializing the values.
This still leaves the problem that the temporary buffer in `valueRef` could be massive
in size, that will be addressed in a short follow-up that this change sets up.
This is a followup to https://github.com/elastic/elasticsearch/pull/104320 which
adds validation during secure setting reload of a `bind_password`.
The reload of `secure_bind_password` will now fail with an exception instead
of logging a deprecation warning.
There's no need for this helper to take more than one argument. Almost
all the usages only passed in a single argument, and the few cases that
supplied more than one can be rewritten as a single argument to save
allocating all those extra lambdas.
In ESQL you can write a grouping `STATS` command without any any
functions - just `STATS BY foo, bar, baz` and we'll calculate all the
combinations of group keys without running any functions. We only have a
single example of that in our tests though! This adds two more that are
slightly more complex. Just out of paranoia.
Transform Checkpoints have a chance to log duplicate audits or drop
iterations. The volatile counters can be read and incremented in
multiple threads, potentially storing the same value back into memory.
Replacing volatile counters with a single Atomic counter, which counts
down the iterations until it reaches zero, then updates the counter to
the next audited checkpoint.
Closes#105106
There is a case where the mapper parser throws a MapperParsingException
instead of not consuming the index:false parameter. We missed that case
in the previous fix (see #98038). This PR hardens that check by
returning false when hitting a MapperParsingException.
Relates #98038
UTF16 represents some characters as surrogate pairs which are represented
by 2 UTF16 characters, often emojis are encoded as surrogate pairs. This PR
fixes an error in calculating the number of bytes required to convert a UTF16
string to UTF8 as surrogate pairs were not processed properly
This adds some docs to the top of `docs.csv-spec` and
`docs-IT_tests_only.csv-spec` telling folks not to add more stuff there
and instead put new examples into whatever files they line up with. It
also shifts some things out of the file to "prime the pump" on cleaning
it up.
We are building a list of InternalAggregations from a list of Buckets, therefore we can use an AbstractList to create the actual list and save some allocations.
Since these docs were originally written there have been a couple
of changes:
1. We now support aarch64 as well as x86_64, so the SSE4.2 guidance
needed clarification.
2. ML is more deeply embedded into Elasticsearch functionality
across nodes that are not ML nodes. For example, ingest pipelines
now routinely use ML, and, in the near future, index mappings
will too in the form of semantic text. Although we cannot mandate
that xpack.ml.enabled is set uniformly across the cluster, as
that would be a breaking change, we should say ever more strongly
that ML must be enabled on all nodes if all ML functionality is to
work correctly. The primary reason for wanting to disable ML is
hardware incompatibility, and if ML is disabled for that reason
then it should not be used at all.
Makes the task_type element of the _inference API optional so that
it is possible to GET, DELETE or POST to an inference entity without
providing the task type
Ryan Ernst
Ry Biesemeyer
Nhat Nguyen
Fabio Busatto
Niels Bauman
Keith Massey
Mary Gouseti
Tim Rühsen
Luca Cavanna
Liam Thompson
Martijn van Groningen
Craig Taverner
Daniel Mitterdorfer
David Turner
Pooya Salehi
Armin Braun
Slobodan Adamović
elasticsearchmachine
Luigi Dell'Aquila
Johannes Fredén
James Baiera
Jonathan Buttner
Fang Xing
Max Hniebergall
Nik Everett
Pat Whelan
David Kyle
Jedr Blaszyk
elasticsearchmachine
Przemysław Witek
Ignacio Vera
David Roberts
Joe Gallo