Test was failing because a previous change made
setting the task to `stopping` a asynchronous action.
It should occur in-line with the cluster even being handled.
Additionally, this commit cleans up the method call paths to
simplify the logic.
closes https://github.com/elastic/elasticsearch/issues/76347
* Give rollling upgrade tests more information
This passes the version that rolling upgrade tests are coming *from*
into the actual tests so they can reason about it. This is useful
because we have features that are not supported on early versions and
we want to write rolling upgrade tests for them. We can't run those
features or assert anything about them in when they don't exist. You'd
think we could use the minimum version of a node in the cluster, but
that only works in the unupgraded phases - once we've completed the
upgrade we need to have the version that we came from to know what we
did in the mixed version.
* Nope
Changes:
* Notes that you can't use cross-cluster search to run a terms lookup on a remote index.
* Removes a redundant sentence noting `_source` is enabled by default.
Closes#61364.
* [DOCS] Clarify where to complete security steps
* Remove config/ from HTTP keystore path
* Clarify the node where generating certs for the HTTP layer
A datafeed will not start running if no indices exist that match
the pattern it's configured to search. However, it is possible that
after the datafeed has started somebody could delete all the indices
that it is configured to search.
This change handles that situation more gracefully, avoiding
repeated logging of null pointer exceptions to the Elasticsearch log.
Move common ANTLR utility methods, such as extracting code or case
insensitive streams into QL.
Replace CaseInsensitiveStream (which relies on ANTLRInputStream which
has been deprecated) with a CharStream variant.
Introduce inference configs for NLP models.
When a PyTorch model is put, the config now expects
a different inference config per task type. Thus, we
have a `ner`, `fill_mask`, and `sentiment_analysis`
config. In addition, the tokenization parameters have
been grouped together and are now part of the relevant
inference config objects. Thus the vocabulary can now
be on a document on its own. A new vocabulary config
object allows the user to specify the location of the
vocabulary document.
Remove methods for creating ArrayLists from arrays with prefix elements.
They are used only in one place in Loggers to create a new array from an
element and an array. We can do it easily with the Streams API without
helpers methods.
The javadoc generation for painless api classes was broken by a recent
commit. This commit fixes it by correct the classpath used to run the
javadoc task. Additionally, the task is added to `check` to ensure it
runs with CI.
closes#76373
The quota aware filesystem was added as a means of allowing
Elasticsearch to track the used space of the underlying filesystem in
virtualized environments. However, the need for it was due to a bug in a
much earlier version of Elasticsearch that always found the underlying
mount and checked it directely for usage. That bug has already been
fixed, so the there is no longer a need for this plugin. This commit
removes the plugin. We should consider separately whether there is still
a need for bootstrap plugins.
closes#70309
This adds a new optional field to the allocate ILM action called "total_shards_per_node". If present, the
value of this field is set as the value of "index.routing.allocation.total_shards_per_node" before the allocation
takes place.
Relates to #44070
* QL: Upgrade ANTLR and move it into QL
Following #76288, upgrade ANTLR library to benefit from the various
improvements made in most recent releases in particular better
performance and error messages.
Looking at the changelog, since version 4.7.2 most changes in ANTLR seem
to have occurred in non-Java targets however this commit upgrades to
ANTLR 4.9.2 to benefit from the dependency updates (such as
StringTemplate).
Additionally move the library into QL to consolidate its use across QL
projects.
Relates #74448Fix#76354
* Fix matching on error message
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
In #76302 we removed a few warnings in a way that javac was happy with
but confused Eclipse. This makes eclipse happy by dropping our overly
strict bounds in `AggregationInspectionHelper`. We really weren't using
any of the type bounds we'd declared there at all.
Adds a test for setting the maximum number of dimensions setting and
tests the names and types of the metadata fields in the index.
Previously we just asserted the count of metadata fields. That made it
hard to read failures.
This stops `MapXContentParser` from throwing an
`UnsupportedOperationException` when passed as an argument to
`XContentBuilder#copyCurrentStructure`. This is mostly useful in tests
where `Map` is a convenient way to talk about structured configuration
but the production APIs need the map to be embedded into a larger blob
of `XContent`.
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
This makes the IP parsing code in `IpFieldMapper` a little easier to
read by removing the string/object juggling. It also pulls the value
parsing bit out into a very sensible and readable method. As a bonus,
this method is reusable and we *do* plan to reuse it.
This change updates the aggregation script, map script for aggregations, and field scripts to extend
DocBasedScript to give them access to the new fields api.
Failure count should not only be reset at checkpoints. Checkpoints could have many pages of data. Consequently, we should reset the failure count once we handle a single composite aggregation page.
This way, the transform won't mark itself as failed erroneously when it has actually succeeded searches + indexing results within the same checkpoint.
closes#76074
* Add support in plugins for a separate spi classloader
SPI is how plugins provide extension points for other plugins to
customize their behavior. Plugins may have a separate SPI jar, so as not
to expose the internals of the plugin implementation. In essense, this
system was built as a stopgap for Java modules, where implementation can
be protected in the same jar. However, currently the plugins service
loads both spi and plugin implementations into the same classloader.
This means that although another plugin may only compile against spi,
they could still get jar hell from a duplicate dependency.
This commit adds support for plugins to contain an "spi" subdirectory
which is loaded into a separate classloader. This spi classloader is a
parent to the plugin implementation, as well as any other plugins which
have extended it. Additionally as a demonstration of how this works in
practice, lang-painless is setup as the first plugin to provide an spi
jar, and sql's dependence on painless is changed to only spi, so that
it now has an independent version of antlr.
closes#74448
* Change eql to use painless spi
* checkstyle
* add licens files to eql for antlr
* Address review
We are adding a _meta field to many of our REST APIs so that users can attach whatever metadata they
want. The data in this field will not be used by Elasticsearch. This commit add the _meta field to ingest
pipelines.
Currently we compile the regex for each log string that we are
analyzing. This is extremely inefficient and may contribute to
instability seen in the logging IT. This commit compiles the regex a
single time.
Currently resync and bulk requests are not compressed when compression
level indexing_data is enabled. Since both of these messages propogate
raw source documents, these should be compressed.
The node shutdown work done in #75188 did not impose any
upper bound on the time allowed for ML jobs to shut down
gracefully.
This change imposes a cap of 10 minutes on shutdown time.
In reality closing ML jobs shouldn't take this long, but
we don't want ML to stall the shutdown process forever
due to a bug, and require user intervention to recover.
This PR fixes a bug where the process context for a given task ID isn't removed from the map.
This may cause a minor data leak as processes are started and stopped.
Additionally, it unmutes the pytorch integration tests and makes the calls multi-threaded to ensure correctness.
Today a force-merge will write out the newly-merged segments and then
flush them, but it does not automatically release the segments from
before the merge. This will retain extra data on disk until the next
refresh, which could be a long time away if the user has disabled
periodic refreshes or is not searching the index frequently.
With this commit we change the behaviour always to refresh after a
force-merge.
Closes#74649
It's in the title. It has been observed that the merge step can run for
non-trivial amounts of time for very large states so we should fork
it to guard the transport threads.
MBeans are sometimes used by third party libraries, for example to report
metrics through JMX. This commit changes plugin policies to allow most
mbean permissions, with the exception of getting classloaders.
* Only start re-assigning persistent tasks if they are not already being reassigned
* adding tests addressing PR comments
* addressing Pr COmments
* addressing PR comments + style"
* improving test rigor
Extracts the functionality in `AbstractHttpServerTransport` that keeps
track of the HTTP client stats so that it can be more easily tested in
isolation.
Benjamin Trent
Keith Massey
Nik Everett
James Rodewig
Adam Locke
David Roberts
Costin Leau
Dimitris Athanasiou
Artem Prigoda
Armin Braun
Ryan Ernst
Mark Vieira
Jack Conradson
Rory Hunter
Rory Hunter
Francisco Fernández Castaño
Tim Brooks
David Turner
Lukas Wegmann
Rene Groeschke
Joe Gallo
Dan Hermann