The current `ids` option doesn't allow pinning a specific document in a
single index when searching over multiple indices. This introduces a
`documents` option, which is an array of `_id` and `_index`
fields to allow index-specific pins.
Closes https://github.com/elastic/elasticsearch/issues/67855.
When performing incremental reductions, 0 value of docCountError may mean that
the error was not previously calculated, or that the error was indeed previously
calculated and its value was 0. We end up rejecting true values set to 0 this
way. This may lead to wrong upper bound of error in result. To fix it, this PR
makes docCountError nullable. null values mean that error was not calculated
yet.
Fixes#40005
Co-authored-by: Igor Motov <igor@motovs.org>
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
In 8.0, with security on by default, we store the HTTP
layer CA PrivateKeyEntry in the http.ssl keystore (along
with the node certificate) so that it is available in our
Enrollment API transport actions.
When loading a keystore, the current behavior is that the
X509ExtendedKeyManager will iterate through the PrivateKeyEntry
objects and will return the first key/certificate that satisfies
the requirements of the client and the server configuration,
and lacks any additional logic/filters.
We need the KeyManager to deterministically pick the node
certificate/key in all cases as this is the intended entry to be
used for TLS on the HTTP layer.
This change introduces filtering when creating the in-memory
keystore the KeyManager is loaded with, so that it will not
include PrivateKeyEntry objects when:
- there are more than 1 PrivateKeyEntry objects in the keystore
- The leaf certificate associated with the PrivateKeyEntry is a
CA certificate
Related: #75097
Co-authored-by: Ioannis Kakavas <ioannis@elastic.co>
Reindex and friends have tasks that start but are not ready to
rethrottle before they figured out if they are leader or worker
tasks. Now wait for the task to fully start before rethrottling.
Also added additional assertions to help see if the inability
to rethrottle is caused by some failure.
Closes#60811
This PR returns the get snapshots API to the 7.x format (and transport client behavior) and enhances it for requests that ask for multiple repositories.
The changes for requests that target multiple repositories are:
* Add `repository` field to `SnapshotInfo` and REST response
* Add `failures` map alongside `snapshots` list instead of returning just an exception response as done for single repo requests
* Pagination now works across repositories instead of being per repository for multi-repository requests
closes#69108closes#43462
During implementation we discovered that the clusters should not
necessarily have a unique name and thus we don't need to convey
this information in the response of the Enroll Node API.
The name token1 is used in both testGetServiceAccountCredentials and
testCreateServiceAccountToken. There are times when the two tests are not fully
isolated. This leads to test failure because indexing of the service token
document must be an op_create. This PR fixes the failure by using a different
token name in testGetServiceAccountCredentials.
This will allow components to add custom metadata to deprecation issues.
This make extracting additional details about deprecations more robust,
otherwise these details need to be parsed from the deprecation message field.
Adjusted the ml model snapshot deprecation to use custom metadata, and
included the job id and snapshot id as custom metadata.
Closes#73089
We barely test the correct handling of user metadata directly.
With upcoming changes to how `SnapshotInfo` is stored it would be nice
to have better test coverage. This PR adds randomized coverage of serializing
user metadata to a large number of tests that all user the shared infrastructure
that is adjusted here.
Follow-up to #73434
Ensures that High Level Rest Client is running against a verified
Elasticsearch. When the first request is send on HLRC, a request to the
info endpoint is made first to verify the product identification and
version.
The data_description of anomaly detection jobs used to accept
delimited data, although this was never documented.
This change removes the delimited option from the data_description,
and the associated functionality in post_data that handled it.
This is not a breaking change because it's removing functionality
that officially never existed. However, just in case somebody
was using it it is only removed from 8.0 and higher, so that at
least they won't find out during a patch install.
Modularization of the JDK has been ongoing for several years. Recently
in Java 16 the JDK began enforcing module boundaries by default. While
Elasticsearch does not yet use the module system directly, there are
some side effects even for those projects not modularized (eg #73517).
Before we can even begin to think about how to modularize, we must
Prepare The Way by enforcing packages only exist in a single jar file,
since the module system does not allow packages to coexist in multiple
modules.
This commit adds a precommit check to the build which detects split
packages. The expectation is that we will add the existing split
packages to the ignore list so that any new classes will not exacerbate
the problem, and the work to cleanup these split packages can be
parallelized.
relates #73525
ParseField is part of the x-content lib, yet it doesn't exist under the
same root package as the rest of the lib. This commit moves the class to
the appropriate package.
relates #73784
When libs/core was created, several classes were moved from server's
o.e.common package, but they were not moved to a new package. Split
packages need to go away long term, so that Elasticsearch can even think
about modularization. This commit moves all the classes under o.e.common
in core to o.e.core.
relates #73784
The recent upgrade of the Azure SDK has caused a few test failures that
have been difficult to debug and do not yet have a fix. In particular, a
change to the netty reactor resolving
(https://github.com/reactor/reactor-netty/issues/1655). We need to wait
for a fix for that issue, so this reverts commit
6c4c4a0ecb.
relates #73493
This commit adds the ability to specify exclusion patterns in Auto-Follow patterns. This allows excluding indices that match any of the inclusion patterns and also match some of the exclusion patterns giving more fine grained control in scenarios where this is important.
Related #67686
We changed the default joda behaviour in strict_date_optional_time to
max 4 digits in a year. Java.time implementation should behave the same way.
At the same time date_optional_time should have 9digits for year part.
closes#52396closes#72191
The feature branch contains changes to configure PyTorch models with a
TrainedModelConfig and defines a format to store the binary models.
The _start and _stop deployment actions control the model lifecycle
and the model can be directly evaluated with the _infer endpoint.
2 Types of NLP tasks are supported: Named Entity Recognition and Fill Mask.
The feature branch consists of these PRs: #73523, #72218, #71679#71323, #71035, #71177, #70713
Categorization jobs created once the entire cluster is upgraded to
version 7.14 or higher will default to using the new ml_standard
tokenizer rather than the previous default of the ml_classic
tokenizer, and will incorporate the new first_non_blank_line char
filter so that categorization is based purely on the first non-blank
line of each message.
The difference between the ml_classic and ml_standard tokenizers
is that ml_classic splits on slashes and colons, so creates multiple
tokens from URLs and filesystem paths, whereas ml_standard attempts
to keep URLs, email addresses and filesystem paths as single tokens.
It is still possible to config the ml_classic tokenizer if you
prefer: just provide a categorization_analyzer within your
analysis_config and whichever tokenizer you choose (which could be
ml_classic or any other Elasticsearch tokenizer) will be used.
To opt out of using first_non_blank_line as a default char filter,
you must explicitly specify a categorization_analyzer that does not
include it.
If no categorization_analyzer is specified but categorization_filters
are specified then the categorization filters are converted to char
filters applied that are applied after first_non_blank_line.
Closeselastic/ml-cpp#1724
This moves the public build api and plugins into a separete included build called 'build-tools'
and we removed the duplication of included buildSrc twice (2nd import as build-tools).
The elasticsearch internal build logic is kept in build-tools-internal as included build which allows us better handling of this project that its just being an buildSrc project (e.g. we can reference tasks directly from the root build etc.)
Convention logic applied to both projects will live in a new build-conventions project.
Instead of assertion that store stats has a non zero value,
just verify that a value is returned. Verifying whether a
non value or specific value is returned, isn't the purpose
of hlrc integration tests.
Closes#60461
This commit upgrades the Azure SDK to 12.11.0 and Jackson to 2.12.2. The
Jackson upgrade must happen at the same time due to Azure depending on
this new version of Jackson.
closes#66555closes#67214
Co-authored-by: Francisco Fernández Castaño <francisco.fernandez.castano@gmail.com>
This commit adds a `cancelled` flag to each cancellable task in the
response to the list tasks API, allowing users to see that a task has
been properly cancelled and will complete as soon as possible.
Closes#72907
Implements a V7 compatible typed endpoints for REST for search related apis
retrofits the REST layer change removed in #41640
relates main meta issue #51816
relates types removal issue #54160
Enroll node API can be used by new nodes in order to join an
existing cluster that has security features enabled. The response
of a call to this API contains all the necessary information that
the new node requires in order to configure itself and bootstrap
trust with the existing cluster.
This commit upgrades the Azure SDK to 12.11.0 and Jackson to 2.12.2. The
Jackson upgrade must happen at the same time due to Azure depending on
this new version of Jackson.
closes#66555closes#67214
Extract usage of internal API from TestClustersPlugin and PluginBuildPlugin and related plugins and build logic
This includes a refactoring of ElasticsearchDistribution to handle types
better in a way we can differentiate between supported Elasticsearch
Distribution types supported in TestCkustersPlugin and types only supported
in internal plugins.
It also introduces a set of internal versions of public plugins.
As part of this we also generate the plugin descriptors now.
As a follow up on this we can actually move these public used classes into
an extra project (declared as included build)
We keep LoggedExec and VersionProperties effectively public And workaround for RestTestBase
David Harsha
Nikita Glashenko
Lyudmila Fokina
Henning Andersen
Almog Tavor
Sylvain Wallez
Armin Braun
Ioannis Kakavas
Yang Wang
Rene Groeschke
Martijn van Groningen
David Turner
David Roberts
Nhat Nguyen
Ryan Ernst
Przemyslaw Gomulka
Francisco Fernández Castaño
David Kyle
Tanguy Leroux
Dan Hermann
Ignacio Vera
Greg Back
Yang Cheng
Ryan Ernst
William Brafford