We have to account for queued up clones when dealing with nodes dropping out
and start them when they become ready to execute because of a node leaving the cluster.
Added test to reproduce the issue in #77101 and another test to verify that the more complex
case of clone queued after snapshot queued after clone still works correctly as well.
The solution here is the most direct fix I could think of and the by far easiest to backport.
That said, I added a TODO that asks for a follow-up that should allow for completely removing
the duplicate code across handling shard updates and external changes. The difference between
the two ways of updating the state is a left-over from the time before we had concurrent
operations and has become a needless complexity nowadays.
closes#77101
This adds the pattern into the error message returned when trying to
fetch fields. So this:
```
POST _search {
"fields": [ { "field": "*", "format": "date_time" } ]
}
```
Will return an error message like
```
error fetching [foo] which matches [*]: Field [foo] of type [keyword] doesn't support formats
```
Inference requests can be batched by adding more rows to the input tensor.
These batch calls are more performant than making multiple calls to forward()
with a single input when all the inputs are of a similar length. The expected
input is now a 2D array of tokens and 2D arrays of supporting arguments,
the output is a 3D array.
In a number of places, we read and write binary data into byte arrays using lucene's
DataInput and DataOutput abstractions. In lucene 9 these abstractions are changing
the endianness of their read/writeInt methods. To avoid dealing with this formatting
change, this commit changes things to use elasticsearch StreamInput/StreamOutput
abstractions instead, which have basically the same API but will preserve endianness.
Relates to #73324
On Windows, rendered Groovy templates contain carriage returns, which
breaks the unit tests and results in them sneaking into the output. Fix
this by rendering into a string and removing the carriage returns.
Today when testing dying with dignity, we simply throw an
OutOfMemoryError. We know this should not get caught by any intermediate
code and end up in the uncaught exception handler. This allows us to
test that this exception handler is able to successfully kill the
VM. However, it is on the table to no longer use the uncaught exception
handler, but instead the built-in support for ExitOnOutOfMemoryError. A
fake OutOfMemoryError would not be processed by this handler, so to
prepare the way, we switch to using a real OutOfMemoryError.
This adds two utility methods for to validate the parameters to the
`docValueFormat` method and replaces a pile of copy and pasted code with
calls to them. They just emit a standard error message if the any
unsupported parameters are provided.
Preview datafeed currently fails when the time field is of type
`date_nanos`. The failure contains the error message:
```
date [...] is after 2262-04-11T23:47:16.854775807 and cannot be
stored in nanosecond resolution
```
This commit fixes this failure. The cause of the issue was that
preview generates a search with a range query on the time field
whose upper bound is `Long.MAX_VALUE` in order to include all
available data in the preview. However, that value is parsed
with `DateUtils.toLong` when the time field is `date_nanos` and
it hits the limitation that values can't be larger than
`DateUtils.MAX_NANOSECOND_INSTANT`. The fix checks whether the
time field is `date_nanos` and uses `DateUtils.MAX_NANOSECOND_INSTANT`
as the upper bound instead of `Long.MAX_VALUE`.
This adds feature tracking for machine learning features.
Model Snapshot upgrader
Anomaly jobs
Data frame analytics jobs
Can all take advantage of the license state tracking built for persistent tasks.
The ModelLoadingService needed special handling to ensure that models cached and referenced
by pipelines are tracked.
License tracking is done per-node and allows for a simple view into when a feature was last used on a given node.
rate aggregation should support being a sub-aggregation
of a composite agg.
The catch is that the composite aggregation source
must be a date histogram. Other sources can be present
but their must be exactly one date histogram source
otherwise the rate aggregation does not know which
interval to compare its unit rate to.
closes https://github.com/elastic/elasticsearch/issues/76988
Closes#76812.
`OsProbe` was only capable of handle cgroup data in the v1 format.
However, Debian 11 uses cgroups v2 by default, and Elasticsearch isn't
capable of reporting any cgroup information. Therefore, add support for
the v2 layout.
This introduces a general document to track guidelines for working on and with
the elasticsearch gradle build
Co-authored-by: Rory Hunter <pugnascotia@users.noreply.github.com>
The changelog generation process currently relies on version
information being present in the changelog YAML descriptors. However,
this makes them difficult to update in some scenarios. For example,
if a PR is merged and subsequently labelled for backporting, our
automation won't update the versions in the changelog YAML.
We can make the process more flexible by removing version data from
the changelog YAML files, and instead inferring the versions from
each changelog YAML file's existence in the git tree at each tag
in the minor series.
This change makes the process more ergonomic for developers, but
harder to test, since I can't simply concoct YAML data for a range
of versions. Instead, I've added a number of unit tests, and tried
to exercise all the relevant parts.
It is now an error to include `versions` the YAML file.
This PR implements support for multiple validators to a FieldMapper.Parameter.
The Parameter#setValidator method was replaced by Parameter#addValidator that can be called multipled times
to add validation to a parameter.
All validators of a parameter will be executed in the same order as they have been added and if any of them fails all validation will failed.
* [DOCS] Add ES security principles
* Incorporating review feedback
* More changes from review feedback
* Fix cross-link to Painless guide
* Clarify callout text
* Add information about elasticsearch user
* Minor wording edits
* Consolidate Java Security Manager description, plus other edits
* Clarify not running as root
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
This commit cleans up some cruft left over from older versions of the
`ClusterApplierService`:
- `UpdateTask` doesn't need to implement lots of interfaces and give
access to its internals, it can just pass appropriate arguments to
`runTasks()`.
- No need for the `runOnApplierThread` override with a default priority,
just have callers be explicit about the priority.
- `submitStateUpdateTask` takes a config which never has a timeout, may
as well just pass the priority and remove the dead code
- `SafeClusterApplyListener` doesn't need to be a
`ClusterApplyListener`, may as well just be an `ActionListener<Void>`.
- No implementations of `ClusterApplyListener` care about the source
argument, may as well drop it.
- Adds assertions to prevent `ClusterApplyListener` implementations from
throwing exceptions since we just swallow them.
- No need to override getting the current time in the
`ClusterApplierService`, we can control this from the `ThreadPool`.
This change adds a method to the PainlessLookup used to find methods of all allow listed sub classes.
A method is specified for a specific super class, and then a list is built with all matching methods from
the all the allow listed sub classes. If no matches are found, null is returned which is consistent with
the behavior of the other look up methods. It is up to the caller to check.
The randomization of the repo version often wasn't used because of the repository cache.
Force re-creating the repository every time we manually mess with the versions.
The QueryStringQuery parser assumes that wildcard queries should use normalized values in queries.
The KeywordScriptFieldType did not support this so was throwing an error. Given there is currently no concept of normalisation in scripted fields I assume it is safe to just add support for this in the same way un-normalized wildcard queries are handled - it feels right that they should behave the same rather than throw an error.
Added a test too.
Closes#76838
When using the index action to index several documents at once,
simulation the action ended up in indexed documents because there was
not break in case the action should only be simulated.
This commits adds such an abortion condition together with a proper
simulation response.
Closes#74148#66735
This introduces a basic public yaml rest test plugin that is supposed to be used by external
elasticsearch plugin authors. This is driven by #76215
- Rename yaml-rest-test to intern-yaml-rest-test
- Use public yaml plugin in example plugins
Co-authored-by: Mark Vieira <portugee@gmail.com>
This PR adds the `REPLACE` shutdown type. As of this PR, `REPLACE` behaves identically to `REMOVE`.
Co-authored-by: Lee Hinman <lee@writequit.org>
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
`TransportFieldCapabilitiesAction` currently holds a lot of logic. This PR
breaks it up into smaller pieces and simplifies its large `doExecute` method.
Simplifying the class will help before we start to make field caps
optimizations.
Changes:
* Factor some methods out of `doExecute` to reduce its length
* Pull `AsyncShardAction` out into its own class to simplify and better match
the code structure in 7.x
Since the test is really for making sure the serialised authentication
header can work after cluster upgrade, it is sufficient to just assert
that the watcher execute successfully once regardless of the total
number of execution.
Armin Braun
Lisa Cawley
Martijn van Groningen
Nik Everett
Adam Locke
David Kyle
Alan Woodward
Mark Vieira
Rory Hunter
Rory Hunter
Jason Tedor
Benjamin Trent
Dimitris Athanasiou
Dan Hermann
Ignacio Vera
Rene Groeschke
Christos Soulios
Francois-Clement Brossard
David Turner
Lukas Wegmann
Jack Conradson
Mark Tozzi
markharwood
Alexander Reelsen
Gordon Brown
Julie Tibshirani
István Zoltán Szabó
Yang Wang
debadair