Commit Graph

60468 Commits

Author SHA1 Message Date
Jack Conradson 718b1635e2
Add a direct sub classes data structure to the Painless lookup (#76955)
This change has two main components.

The first is to have method/field resolution for compile-time and run-time use the same code path for 
now. This removes copying of member methods between super and sub classes and instead does a 
resolution through the class hierarchy. This allows us to correctly implement the next change.

The second is a data structure that allows for the lookup of direct sub classes for all allow listed 
classes/interfaces within Painless.
2021-08-26 12:09:43 -07:00
Joe Gallo a127154042
[DOCS] Fix the terms enum API docs for search_after (#76991) 2021-08-26 14:07:06 -04:00
Adam Locke 1056c857ee
[DOCS] Update combined fields wording (#76893)
* [DOCS] Update combined fields wording

* Clarifications from review feedback
2021-08-26 13:16:55 -04:00
Howard 4432b39112
[DOCS] Fix formatting for `snapshot_meta` thread pool (#76973) 2021-08-26 10:36:26 -04:00
Stuart Tettemer 3c737341fe
Script: Fields API converter tests (#76900) 2021-08-26 09:18:42 -05:00
William Brafford 8759c85a68
Allow access to restricted system indices for reserved system roles (#76845)
* Add system index patterns to TestRestrictedIndices

A missing piece in #74212 was system index patterns in the tests for the
ReservedRolesStore. Without these patterns, the tests did not accurately
check whether a role was incorrectly accessing a system index that was
not previously a restricted index.

This commit adds all of the current system index patterns to the test
class and adds restricted index access to the system roles that need it
for tests to pass.

* Preserve existing Kibana data telemetry privileges
* Test that data telemetry can't access security and async indices
2021-08-26 08:56:13 -04:00
Costin Leau d349f25247
QL: regenerate parser due to ANTLR upgrade (#76970)
* QL: regenerate parser due to ANTLR upgrade

To avoid warnings and subtle change in behavior, regenerate the parser
classes using the upgraded ANTLR (4.9.2 vs 4.5.3).

Fix #76969
2021-08-26 15:41:57 +03:00
Hendrik Muhs 4974a7cd7b
[Transform] Reduce indexes to query based on checkpoints (#75839)
Continuous transform reduce the amount of data to query for by detecting what has been changed
since the last checkpoint. This information is used to inject queries that narrow the scope.
The query is send to all configured indices. This change reduces the indexes to call
using checkpoint information. The number of network calls go down which in addition to performance
reduces the probability of a failure.

This change mainly helps the transforms of type latest, pivot transform require additional
changes planned for later.
2021-08-26 09:08:10 +02:00
Przemyslaw Gomulka 0e50d2041c
Expand the minimum utc timestamp used in fetching timezone transitions (#75584)
when rounding UTC timestamps we convert a timestamp from UTC to local, round this to closest midnight, then we convert back to UTC.
This means, that for a timestamp close to a DST we need to make sure we collected a transition that will be needed when converting back to UTC.
To do this, we decrease the minUtcMillis by 2* unit to sure that the additional transition that could affect the timestamp is also fetched and the correct minimum is used in further lookups

closes #73995
2021-08-26 09:05:19 +02:00
Przemko Robakowski 99c85f8e70
Fix disabling GeoIP downloader through elasticsearch.yml (#76924)
This change allows user to disable GeoIP downloader using elasticsearch.yml and it deletes .geoip_databases index if downloader is disabled.

Closes #76586
2021-08-26 08:47:37 +02:00
Yang Wang 21230cb4bd
Handle a edge case for validation of API key role descriptors (#76959)
This PR fixes a BWC edge case: In a mixed cluster, e.g. rolling upgrade, API
keys can sometimes fail to validate due to mismatch of role descriptors
depending on where the request is initially authenticated.
2021-08-26 14:45:33 +10:00
Lisa Cawley d36f24fbc3
[DOCS] Update datafeed details in ML docs (#76854) 2021-08-25 11:35:21 -07:00
Mark Vieira 8a9ea85657
Remove no-jdk distributions (#76896) 2021-08-25 09:52:15 -07:00
Rory Hunter 5c00587edf
Update Azul JVM on Apple M1 (#76923)
Closes #76901. Bump the Azul JVM version for aarch64 M1 i.e. Apple silicon.
2021-08-25 17:07:04 +01:00
Mark Vieira 9389396183
Remove unnecessary test task dependency (#76938)
We auto-wire the bwcTestSnapshots task to the check task. This dependOn
means we run the full wire compatible test suite on every build which
is not what we want since we run those explicitly in parallel CI jobs.
Running check should only test unreleased versions.

Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
2021-08-25 12:01:21 -04:00
weensykim a50a8f9eb9
[DOCS] Fix typo (#76911) 2021-08-25 10:03:51 -04:00
Nhat Nguyen 25984d159d
Ensure norms is non-zero in IndexDiskUsageAnalyzerIT (#76894)
The test fails when norms have a singleton value and is loaded eagerly. 
This commit makes sure that we index more than one norm value.

Closes #76173
2021-08-25 09:43:37 -04:00
Nhat Nguyen 7b618a01d9
Relax disk usage test assertions (#76895)
This commit allows up to 2K data structures that are loaded
before we can track bytes read.

Closes #76643
2021-08-25 09:41:17 -04:00
István Zoltán Szabó 789368b38f
[DOCS] Fixes a syntax error in datafeed runtime field example. (#76917) 2021-08-25 12:04:32 +02:00
David Kyle 283e6b1d4a
[ML] Prefer American English spellings (#76915)
organisation -> organization
2021-08-25 10:55:11 +01:00
Bogdan Pintea 79ae386506
Fix join keys ordering in a sequence (#76699)
A sequence payload is constructed by providing one list of sequences and
one for the hits. When fetching the list of hits though, the list of
sequnces can be iterated in reverse order to build the hit references;
meaning that the original list of sequences provided to the sequence
payload needs reversing too.
2021-08-25 10:22:31 +02:00
Armin Braun 9fa8babb4a
Fix Unexpected DS Name in CCR Test around Midnight (#76890)
While `verifyDataStream` correctly accounts for data changes during
rollover the rest of the test did not. Since we have the concrete
correct index names available in `verifyDataStream` already we might
as well just return them here and use them.

closes #76163
2021-08-25 09:43:22 +02:00
Hendrik Muhs deac0030bf
[Test] TransformIndexerStateTests testStopAtCheckpoint fix listener count (#76880)
change check for open listeners, avoiding failures due to execution timing

fixes #76555
2021-08-25 08:04:33 +02:00
Armin Braun 706ccbd8b5
Remove Needless Sleeps on Node Configuration Changes in Internal Cluster Tests (#76884)
I noticed this recently when trying to reproduce a test failure. We're doing a lot of sleeping
when validating that the cluster formed if that process is slow randomly (which it tends to be
due to disk interaction on node starts and such.). By reusing the approach for waiting on a
cluster state we rarely if ever need to get into the busy assert loop and remove all these sleeps,
shaving of a few seconds here and there from running internal cluster tests.
2021-08-25 05:39:33 +02:00
Mark Vieira 80817d0fcd Add GitHub action for synchronizing commits from master to main branch 2021-08-24 15:51:33 -07:00
Nikola Grcevski 98f0f4b9ec
[TEST] Implement HotThreads unit tests (#76857)
* [TEST] Implement HotThreads unit tests

Add unit tests for the internal HotThreads logic for calculating and
sorting threads by CPU, Wait and Blocked "hotness". Also adds tests
for identifying certain threads as idle, as well as supported report
types (e.g. cpu, wait, blocked).

Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
2021-08-24 18:14:49 -04:00
Mark Vieira 705a8090af Give the nojdk packaging test builds a unique name 2021-08-24 12:49:58 -07:00
Mark Vieira db6f86f2b0 Disable testing of no-jdk distribution for 8.0 2021-08-24 11:50:27 -07:00
Adam Locke 56efbd83ce
[DOCS] Fixing field context examples (#76887) 2021-08-24 14:10:41 -04:00
David Kyle ae2d3e2c74
Mute PyTorchModelIT in preparation for result format changes (#76888) 2021-08-24 12:53:33 -04:00
Andrei Stefan b85d6872f9
SQL: Change the rolled over file name to consider the compression (#76866)
* Change the rolled over file name to consider the compression

* Use NIO instead of java.io.File
2021-08-24 08:50:33 -04:00
James Rodewig bb9587351e
[DOCS] Fix typos (#76875) 2021-08-24 08:47:12 -04:00
Przemysław Witek 676d4de3de
[Transform] Implement the ability to preview the existing transform (#76697) 2021-08-24 14:41:49 +02:00
Dan Hermann d648b768d8
Sync grok processor patterns with Logstash (#76752) 2021-08-24 07:34:27 -05:00
Benjamin Trent f913aaef5b
[ML] add support for distilbert pytorch models (#76679)
This commit adds support for distilbert pytorch models.

While the tokenization itself is exactly the same as bert, the parameters sent to the model are different. 

DistilBERT does not require the segment mask or positional IDs to be sent. Only the input mask and token ids. 

But, since the effective output of the tokenization sent to the model is different, I opted to consider it as a unique
tokenizer, inheriting from our bert implementation.

The API now looks like:
for BERT models
```js
"inference_config": {
  "ner": {
    "vocabulary": {/*...*/},
    "tokenization": {
      "bert": {/*...*/}
    }
  }
}
```
For DistilBERT models
```js
"inference_config": {
  "ner": {
    "vocabulary": {/*...*/},
    "tokenization": {
      "distil_bert": {/*...*/}
    }
  }
}
```
2021-08-24 08:19:03 -04:00
Armin Braun 98bf4eb3d5
Fix Test Failure in testMasterFailOverWithQueuedDeletes (#76865)
We can have a freak situation here where running the get snapshots
request concurrently with a delete produces a missing snapshot exception
if a snasphot is deleted just as its metadata is fetched.
This is a known issue and a fix is tricky (SLM etc. work around this issue
in tests and prod code by using the ignore-unavailable flag for example).
In this test we can easily fix the problem by just using the deterministic waiting
on cluster state before asserting that the snapshots are gone from the repo.

closes #76549
2021-08-24 13:48:53 +02:00
Armin Braun b3c1170627
Save some Memory in Watcher XContent -> Map Round Trip (#76864)
Low effort quick-fix to improve efficiency of this round trip and
push the  boundary of what search responses we can still convert until
we have a real fix.

relates #74513
2021-08-24 13:21:59 +02:00
David Kyle 587f061cc0
[ML] Get Trained Model Deployment Stats API (#75268)
Adds the REST API /_ml/trained_models/{model_id}/deployment/_stats.
The API reports stats for live deployments -those that are started and running
- such as on which nodes the model(s) are deployed, avg inference
time and the inference count
2021-08-24 09:54:20 +01:00
Andrei Stefan c9210072e8
Use a different query for the multiply function (#76846)
Use a different query for the multiply function due to the different behaviors in versions on or after 7.12.0
2021-08-24 11:12:00 +03:00
Yang Wang a5a1c6aa91
[Test] More robust assertions for sorting and pagination (#76654)
Remove potential duplicates from results gathered with sorting
and search_after. Pagination does not use _doc anymore since it could
change between requests.

Resolves: #76542
2021-08-24 11:42:19 +10:00
Tim Vernum 285c93002f
[Test] Fix filename check on Windows (#76807)
Replaces "/" with platform file separator character

Resolves: #76767
2021-08-24 11:10:36 +10:00
Mark Vieira 077602e498
Upgrade build scan plugin to 3.6.4 (#76784) 2021-08-23 13:39:23 -07:00
Ioannis Kakavas 3a1f40302b
Remove keystore initial_md5sum (#76835)
Elasticsearch's keystore initial md5sum was added in #28928 with
the intention to allow us to remove the elasticsearch.keystore
file upon package removal, if this hadn't been altered after
installation. At that time this decision made perfect sense as
the elasticsearch keystore only contains transient data by
default ( keystore.seed ) that is meant to be useful for bootstrap
related actions, and doesn't need to survive re-installations.

With Security ON by default, we will be storing additional
settings in the keystore upon installation(namely, the passwords
for the PKCS#12 keystores used for TLS) and these have a more
persistent nature. Since `remove` doesn't delete the configuration
directories and files where said PKCS#12 keystores are stored, it
makes sense to also not delete the elasticsearch.keystore which
stores the passwords.
2021-08-23 22:28:13 +03:00
Rory Hunter 51b0c0cb01
Don't export docker images on assemble (#76817)
Our packaging tests set up dependencies so that the tests depend on the
artifact that is going to be tested. For Docker, the tests depend on an
export of the Docker image, partly to follow the pattern, but partly to
support loading the image when the tests are executed inside Vagrant.

There is a sub-project for each type of image that can be exported. The
main Docker build file adds an export task and an artifact to the
sub-project. However, it also makes the export task a dependency of the
sub-project's `assemble` task. This last step isn't necessary for the
tests to run, and causes extra and needless work to be done when running
`assemble` on the entire build. Therefore, removing the last bit of task
wiring.
2021-08-23 20:11:17 +01:00
David Turner 007b5aec40
Fix testMasterStatsOnSuccessfulUpdate (#76844)
This test advances time in surprising places, so we have to allow a
little longer for publication or stabilisation to complete.

Closes #76840
2021-08-23 15:02:16 -04:00
David Turner 1ebc378e78 AwaitsFix for #76840 2021-08-23 19:07:05 +01:00
Armin Braun 57ba73c12c
Make Releasing Aggregation Buffers Safer (#76741)
Fixing a few possible leaks and making the logic more obviously correct.
2021-08-23 19:51:51 +02:00
David Turner 7094493b2b
Re-enable BWC tests after backport of #76771 (#76839)
Adjusts the wire compatibility constraints and reverts commit
18e657aed9.
2021-08-23 13:49:54 -04:00
Tim Brooks 0e3e722754
Dispatch large bulk requests to write thread (#76736)
Currently all of the coordinating action for a bulk request is performed
on the transport thread. With Elasticsearch increasing its usage of
compression, it is now possible that large bulk requests will be
compressed at the coordinating stage when routing to other nodes. This
commit moves the coordination of large bulk requsets to the write thread
to avoid blocking the transport threads on costly CPU operations.
2021-08-23 11:07:42 -06:00
David Turner 18e657aed9 Disable BWC tests for #76771 2021-08-23 17:52:18 +01:00