This change has two main components.
The first is to have method/field resolution for compile-time and run-time use the same code path for
now. This removes copying of member methods between super and sub classes and instead does a
resolution through the class hierarchy. This allows us to correctly implement the next change.
The second is a data structure that allows for the lookup of direct sub classes for all allow listed
classes/interfaces within Painless.
* Add system index patterns to TestRestrictedIndices
A missing piece in #74212 was system index patterns in the tests for the
ReservedRolesStore. Without these patterns, the tests did not accurately
check whether a role was incorrectly accessing a system index that was
not previously a restricted index.
This commit adds all of the current system index patterns to the test
class and adds restricted index access to the system roles that need it
for tests to pass.
* Preserve existing Kibana data telemetry privileges
* Test that data telemetry can't access security and async indices
* QL: regenerate parser due to ANTLR upgrade
To avoid warnings and subtle change in behavior, regenerate the parser
classes using the upgraded ANTLR (4.9.2 vs 4.5.3).
Fix#76969
Continuous transform reduce the amount of data to query for by detecting what has been changed
since the last checkpoint. This information is used to inject queries that narrow the scope.
The query is send to all configured indices. This change reduces the indexes to call
using checkpoint information. The number of network calls go down which in addition to performance
reduces the probability of a failure.
This change mainly helps the transforms of type latest, pivot transform require additional
changes planned for later.
when rounding UTC timestamps we convert a timestamp from UTC to local, round this to closest midnight, then we convert back to UTC.
This means, that for a timestamp close to a DST we need to make sure we collected a transition that will be needed when converting back to UTC.
To do this, we decrease the minUtcMillis by 2* unit to sure that the additional transition that could affect the timestamp is also fetched and the correct minimum is used in further lookups
closes#73995
This change allows user to disable GeoIP downloader using elasticsearch.yml and it deletes .geoip_databases index if downloader is disabled.
Closes#76586
This PR fixes a BWC edge case: In a mixed cluster, e.g. rolling upgrade, API
keys can sometimes fail to validate due to mismatch of role descriptors
depending on where the request is initially authenticated.
We auto-wire the bwcTestSnapshots task to the check task. This dependOn
means we run the full wire compatible test suite on every build which
is not what we want since we run those explicitly in parallel CI jobs.
Running check should only test unreleased versions.
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
A sequence payload is constructed by providing one list of sequences and
one for the hits. When fetching the list of hits though, the list of
sequnces can be iterated in reverse order to build the hit references;
meaning that the original list of sequences provided to the sequence
payload needs reversing too.
While `verifyDataStream` correctly accounts for data changes during
rollover the rest of the test did not. Since we have the concrete
correct index names available in `verifyDataStream` already we might
as well just return them here and use them.
closes#76163
I noticed this recently when trying to reproduce a test failure. We're doing a lot of sleeping
when validating that the cluster formed if that process is slow randomly (which it tends to be
due to disk interaction on node starts and such.). By reusing the approach for waiting on a
cluster state we rarely if ever need to get into the busy assert loop and remove all these sleeps,
shaving of a few seconds here and there from running internal cluster tests.
* [TEST] Implement HotThreads unit tests
Add unit tests for the internal HotThreads logic for calculating and
sorting threads by CPU, Wait and Blocked "hotness". Also adds tests
for identifying certain threads as idle, as well as supported report
types (e.g. cpu, wait, blocked).
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
This commit adds support for distilbert pytorch models.
While the tokenization itself is exactly the same as bert, the parameters sent to the model are different.
DistilBERT does not require the segment mask or positional IDs to be sent. Only the input mask and token ids.
But, since the effective output of the tokenization sent to the model is different, I opted to consider it as a unique
tokenizer, inheriting from our bert implementation.
The API now looks like:
for BERT models
```js
"inference_config": {
"ner": {
"vocabulary": {/*...*/},
"tokenization": {
"bert": {/*...*/}
}
}
}
```
For DistilBERT models
```js
"inference_config": {
"ner": {
"vocabulary": {/*...*/},
"tokenization": {
"distil_bert": {/*...*/}
}
}
}
```
We can have a freak situation here where running the get snapshots
request concurrently with a delete produces a missing snapshot exception
if a snasphot is deleted just as its metadata is fetched.
This is a known issue and a fix is tricky (SLM etc. work around this issue
in tests and prod code by using the ignore-unavailable flag for example).
In this test we can easily fix the problem by just using the deterministic waiting
on cluster state before asserting that the snapshots are gone from the repo.
closes#76549
Low effort quick-fix to improve efficiency of this round trip and
push the boundary of what search responses we can still convert until
we have a real fix.
relates #74513
Adds the REST API /_ml/trained_models/{model_id}/deployment/_stats.
The API reports stats for live deployments -those that are started and running
- such as on which nodes the model(s) are deployed, avg inference
time and the inference count
Remove potential duplicates from results gathered with sorting
and search_after. Pagination does not use _doc anymore since it could
change between requests.
Resolves: #76542
Elasticsearch's keystore initial md5sum was added in #28928 with
the intention to allow us to remove the elasticsearch.keystore
file upon package removal, if this hadn't been altered after
installation. At that time this decision made perfect sense as
the elasticsearch keystore only contains transient data by
default ( keystore.seed ) that is meant to be useful for bootstrap
related actions, and doesn't need to survive re-installations.
With Security ON by default, we will be storing additional
settings in the keystore upon installation(namely, the passwords
for the PKCS#12 keystores used for TLS) and these have a more
persistent nature. Since `remove` doesn't delete the configuration
directories and files where said PKCS#12 keystores are stored, it
makes sense to also not delete the elasticsearch.keystore which
stores the passwords.
Our packaging tests set up dependencies so that the tests depend on the
artifact that is going to be tested. For Docker, the tests depend on an
export of the Docker image, partly to follow the pattern, but partly to
support loading the image when the tests are executed inside Vagrant.
There is a sub-project for each type of image that can be exported. The
main Docker build file adds an export task and an artifact to the
sub-project. However, it also makes the export task a dependency of the
sub-project's `assemble` task. This last step isn't necessary for the
tests to run, and causes extra and needless work to be done when running
`assemble` on the entire build. Therefore, removing the last bit of task
wiring.
Currently all of the coordinating action for a bulk request is performed
on the transport thread. With Elasticsearch increasing its usage of
compression, it is now possible that large bulk requests will be
compressed at the coordinating stage when routing to other nodes. This
commit moves the coordination of large bulk requsets to the write thread
to avoid blocking the transport threads on costly CPU operations.
Jack Conradson
Joe Gallo
Adam Locke
Howard
Stuart Tettemer
William Brafford
Costin Leau
Hendrik Muhs
Przemyslaw Gomulka
Przemko Robakowski
Yang Wang
Lisa Cawley
Mark Vieira
Rory Hunter
weensykim
Nhat Nguyen
István Zoltán Szabó
David Kyle
Bogdan Pintea
Armin Braun
Nikola Grcevski
Andrei Stefan
James Rodewig
Przemysław Witek
Dan Hermann
Benjamin Trent
Tim Vernum
Ioannis Kakavas
David Turner
Tim Brooks