Commit Graph

60468 Commits

Author SHA1 Message Date
Benjamin Trent 17a4b20f85
[ML] fix test and unify model deployment task stopping paths (#76376)
Test was failing because a previous change made
setting the task to `stopping` a asynchronous action.

It should occur in-line with the cluster even being handled.

Additionally, this commit cleans up the method call paths to
simplify the logic.

closes https://github.com/elastic/elasticsearch/issues/76347
2021-08-12 09:34:14 -04:00
Keith Massey a9c883f7db
Updating supported version of _meta field on pipelines after backport (#76381)
Updating the supported version for the _meta field on ingest pipelines from 8.0 to 7.15 after backporting
support to 7.15.
Relates to #75799
2021-08-12 08:29:49 -05:00
Nik Everett a6fe766b7b
Give rollling upgrade tests more information (#76360)
* Give rollling upgrade tests more information

This passes the version that rolling upgrade tests are coming *from*
into the actual tests so they can reason about it. This is useful
because we have features that are not supported on early versions and
we want to write rolling upgrade tests for them. We can't run those
features or assert anything about them in when they don't exist. You'd
think we could use the minimum version of a node in the cluster, but
that only works in the unupgraded phases - once we've completed the
upgrade we need to have the version that we came from to know what we
did in the mixed version.

* Nope
2021-08-12 09:20:51 -04:00
James Rodewig 8eaf4043e4
[DOCS] Terms lookup doesn't support remote indices (#76371)
Changes:
* Notes that you can't use cross-cluster search to run a terms lookup on a remote index.
* Removes a redundant sentence noting `_source` is enabled by default.

Closes #61364.
2021-08-12 08:33:10 -04:00
Adam Locke f1e1492581
[DOCS] Clarify where to complete security steps (#76120)
* [DOCS] Clarify where to complete security steps

* Remove config/ from HTTP keystore path

* Clarify the node where generating certs for the HTTP layer
2021-08-12 08:11:30 -04:00
David Roberts a85e319579
[ML] Fix null pointer exception after all datafeed indices deleted (#76401)
A datafeed will not start running if no indices exist that match
the pattern it's configured to search. However, it is possible that
after the datafeed has started somebody could delete all the indices
that it is configured to search.

This change handles that situation more gracefully, avoiding
repeated logging of null pointer exceptions to the Elasticsearch log.
2021-08-12 12:40:40 +01:00
Costin Leau 5679e10f4a
QL: Introduce ParserUtils to consolidate code (#76399)
Move common ANTLR utility methods, such as extracting code or case
insensitive streams into QL.
Replace CaseInsensitiveStream (which relies on ANTLRInputStream which
has been deprecated) with a CharStream variant.
2021-08-12 05:44:53 -04:00
Dimitris Athanasiou adfa977c83
[ML] Inference configs for NLP models (#76350)
Introduce inference configs for NLP models.

When a PyTorch model is put, the config now expects
a different inference config per task type. Thus, we
have a `ner`, `fill_mask`, and `sentiment_analysis`
config. In addition, the tokenization parameters have
been grouped together and are now part of the relevant
inference config objects. Thus the vocabulary can now
be on a document on its own. A new vocabulary config
object allows the user to specify the location of the
vocabulary document.
2021-08-12 12:10:05 +03:00
Artem Prigoda e2b5eaf027
Remove unused CollectionUtils methods (#76387)
Remove methods for creating ArrayLists from arrays with prefix elements.
They are used only in one place in Loggers to create a new array from an
element and an array. We can do it easily with the Streams API without
helpers methods.
2021-08-12 09:31:30 +02:00
Armin Braun 847be51b64
Reenable BwC Rest Tests (#76226)
These tests should work now that we aligned 7.x and 8.x snapshot apis.
2021-08-12 09:06:29 +02:00
Ryan Ernst dfcb3416b8
Fix generation of javadocs (#76382)
The javadoc generation for painless api classes was broken by a recent
commit. This commit fixes it by correct the classpath used to run the
javadoc task. Additionally, the task is added to `check` to ensure it
runs with CI.

closes #76373
2021-08-11 18:33:19 -04:00
Ryan Ernst 96627dfa14
Remove quota-aware-fs plugin (#76352)
The quota aware filesystem was added as a means of allowing
Elasticsearch to track the used space of the underlying filesystem in
virtualized environments. However, the need for it was due to a bug in a
much earlier version of Elasticsearch that always found the underlying
mount and checked it directely for usage. That bug has already been
fixed, so the there is no longer a need for this plugin. This commit
removes the plugin. We should consider separately whether there is still
a need for bootstrap plugins.

closes #70309
2021-08-11 15:12:50 -07:00
Keith Massey 8610db674a
Allow for setting the total shards per node in the Allocate ILM action (#76134)
This adds a new optional field to the allocate ILM action called "total_shards_per_node". If present, the
value of this field is set as the value of "index.routing.allocation.total_shards_per_node" before the allocation
takes place.
Relates to #44070
2021-08-11 16:35:46 -05:00
Mark Vieira 81616c2fb6
Use version-specific documentation link in jvm.options file (#76323) 2021-08-11 14:31:11 -07:00
Armin Braun b01cd8ff87
Radomize BlobContainer Path in Retries Tests (#76303)
Follow up to #76273 adding some randomization across all retries tests.
2021-08-11 22:31:04 +02:00
Costin Leau 6d7b3458bd
QL: Upgrade ANTLR and move it into QL (#76358)
* QL: Upgrade ANTLR and move it into QL

Following #76288, upgrade ANTLR library to benefit from the various
improvements made in most recent releases in particular better
performance and error messages.

Looking at the changelog, since version 4.7.2 most changes in ANTLR seem
to have occurred in non-Java targets however this commit upgrades to
ANTLR 4.9.2 to benefit from the dependency updates (such as
StringTemplate).

Additionally move the library into QL to consolidate its use across QL
projects.

Relates #74448
Fix #76354

* Fix matching on error message

Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
2021-08-11 16:11:37 -04:00
Nik Everett 0661cf3cce
Fix eclipse compilation of hasValue (#76375)
In #76302 we removed a few warnings in a way that javac was happy with
but confused Eclipse. This makes eclipse happy by dropping our overly
strict bounds in `AggregationInspectionHelper`. We really weren't using
any of the type bounds we'd declared there at all.
2021-08-11 15:04:40 -04:00
Mark Vieira b58416cd60 Delete backport GitHub action 2021-08-11 11:46:00 -07:00
Nik Everett 9bde1d9007
Expand DocumentMapperTests (#76368)
Adds a test for setting the maximum number of dimensions setting and
tests the names and types of the metadata fields in the index.
Previously we just asserted the count of metadata fields. That made it
hard to read failures.
2021-08-11 14:22:14 -04:00
Nik Everett b86d526a35
Fix copyCurentStructure(MapXContentParser) (#76357)
This stops `MapXContentParser` from throwing an
`UnsupportedOperationException` when passed as an argument to
`XContentBuilder#copyCurrentStructure`. This is mostly useful in tests
where `Map` is a convenient way to talk about structured configuration
but the production APIs need the map to be embedded into a larger blob
of `XContent`.

Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
2021-08-11 13:30:37 -04:00
Nik Everett 72ba4cdccb
Make ip field parsing easier to read (#76363)
This makes the IP parsing code in `IpFieldMapper` a little easier to
read by removing the string/object juggling. It also pulls the value
parsing bit out into a very sensible and readable method. As a bonus,
this method is reusable and we *do* plan to reuse it.
2021-08-11 11:58:23 -04:00
Jack Conradson a48c7369a2
Add Fields API to aggregation scripts and field scripts (#76325)
This change updates the aggregation script, map script for aggregations, and field scripts to extend 
DocBasedScript to give them access to the new fields api.
2021-08-11 08:32:36 -07:00
Rory Hunter 4137d11079
Fix compiler warnings in :server - part 4 (#76302)
Closes #40366.

Fix the last remaining javac issues when linting is enforced in `server/`.
2021-08-11 16:15:30 +01:00
Rory Hunter 1800b5e3c4 Mute RetryableActionTests.testRetryableActionTimeout 2021-08-11 16:09:48 +01:00
Benjamin Trent 70d91d1d38
[ML][Transform] reset failure count when a transform aggregation page is handled successfully (#76355)
Failure count should not only be reset at checkpoints. Checkpoints could have many pages of data. Consequently, we should reset the failure count once we handle a single composite aggregation page.

This way, the transform won't mark itself as failed erroneously when it has actually succeeded searches + indexing results within the same checkpoint.

closes #76074
2021-08-11 10:59:19 -04:00
Mark Vieira 98ce2d2c0a Revert "Delete backport GitHub action"
This reverts commit 3d0b8efb78.
2021-08-11 07:42:47 -07:00
Francisco Fernández Castaño 7d65d83fa6
Fix IndexSnapshotsServiceIT and SnapshotsRecoveryPlannerServiceTests (#76351)
Closes #76343
2021-08-11 16:31:44 +02:00
Ryan Ernst 35980c8da6
Add support in plugins for a separate spi classloader (#76288)
* Add support in plugins for a separate spi classloader

SPI is how plugins provide extension points for other plugins to
customize their behavior. Plugins may have a separate SPI jar, so as not
to expose the internals of the plugin implementation. In essense, this
system was built as a stopgap for Java modules, where implementation can
be protected in the same jar. However, currently the plugins service
loads both spi and plugin implementations into the same classloader.
This means that although another plugin may only compile against spi,
they could still get jar hell from a duplicate dependency.

This commit adds support for plugins to contain an "spi" subdirectory
which is loaded into a separate classloader. This spi classloader is a
parent to the plugin implementation, as well as any other plugins which
have extended it. Additionally as a demonstration of how this works in
practice, lang-painless is setup as the first plugin to provide an spi
jar, and sql's dependence on painless is changed to only spi, so that
it now has an independent version of antlr.

closes #74448

* Change eql to use painless spi

* checkstyle

* add licens files to eql for antlr

* Address review
2021-08-11 09:59:29 -04:00
Keith Massey 498684a696
Add support for _meta field to ingest pipelines (#75905)
We are adding a _meta field to many of our REST APIs so that users can attach whatever metadata they
want. The data in this field will not be used by Elasticsearch. This commit add the _meta field to ingest
pipelines.
2021-08-11 08:30:36 -05:00
Tim Brooks a91be8282d
Precompile regex for PatternSeenEventExpectation (#76332)
Currently we compile the regex for each log string that we are
analyzing. This is extremely inefficient and may contribute to
instability seen in the logging IT. This commit compiles the regex a
single time.
2021-08-11 07:04:58 -06:00
Tim Brooks 3834972b3d
Ensure indexing_data is compressed appropriately (#76321)
Currently resync and bulk requests are not compressed when compression
level indexing_data is enabled. Since both of these messages propogate
raw source documents, these should be compressed.
2021-08-11 07:04:32 -06:00
Benjamin Trent d8e30cf123
[ML] muting org.elasticsearch.xpack.ml.inference.allocation.TrainedModelAllocationNodeServiceTests.testClusterChanged (#76348) 2021-08-11 07:54:50 -04:00
David Roberts 8f5e45755c
[ML] Cap graceful shutdown time (#76342)
The node shutdown work done in #75188 did not impose any
upper bound on the time allowed for ML jobs to shut down
gracefully.

This change imposes a cap of 10 minutes on shutdown time.
In reality closing ML jobs shouldn't take this long, but
we don't want ML to stall the shutdown process forever
due to a bug, and require user intervention to recover.
2021-08-11 12:44:51 +01:00
Benjamin Trent 97d8a151f7
[ML] remove process context on stop, fail, and crash (#76300)
This PR fixes a bug where the process context for a given task ID isn't removed from the map.

This may cause a minor data leak as processes are started and stopped.

Additionally, it unmutes the pytorch integration tests and makes the calls multi-threaded to ensure correctness.
2021-08-11 07:14:32 -04:00
David Turner 9bc30a37a6
Refresh after force-merge (#76221)
Today a force-merge will write out the newly-merged segments and then
flush them, but it does not automatically release the segments from
before the merge. This will retain extra data on disk until the next
refresh, which could be a long time away if the user has disabled
periodic refreshes or is not searching the index frequently.

With this commit we change the behaviour always to refresh after a
force-merge.

Closes #74649
2021-08-11 11:58:18 +01:00
Benjamin Trent 3cbc948d75
Make persistent task cluster service test more stable (#76318) 2021-08-11 12:06:01 +02:00
Armin Braun 6c86e9e6e9
Fork Building Aggregate Index Capabilities Response to Management Pool (#76333)
It's in the title. It has been observed that the merge step can run for
non-trivial amounts of time for very large states so we should fork
it to guard the transport threads.
2021-08-11 12:00:55 +02:00
Francisco Fernández Castaño ca1f4dd5c2
Compute latest snapshot directly in TransportGetShardSnapshotAction (#76254) 2021-08-11 09:57:24 +02:00
Lukas Wegmann 0fb10cc15a
SQL: Fix nested ORDER BY (#76277)
* introduce PushDownAndCombineOrderBy optimization

* move merging of order by clauses to QueryFolder

* address review comments

* fix iteration
2021-08-11 09:42:23 +02:00
Rene Groeschke da29bb028f
Avoid configuration time task creation (#76245)
Fix publish plugin
2021-08-11 09:30:51 +02:00
Ryan Ernst a3b04ad229
Allow MBean permissions in plugin policies (#76329)
MBeans are sometimes used by third party libraries, for example to report
metrics through JMX. This commit changes plugin policies to allow most
mbean permissions, with the exception of getting classloaders.
2021-08-11 01:05:37 -04:00
Mark Vieira 3fdb02c0ca
Remove unused build script extra property (#76324) 2021-08-10 15:02:53 -07:00
Mark Vieira 3d0b8efb78 Delete backport GitHub action 2021-08-10 13:49:54 -07:00
Joe Gallo e7295443eb
Re-enable deprecated _cluster/voting_config_exclusions/{node_name} route via REST compatibility (#75951) 2021-08-10 16:42:51 -04:00
Dan Hermann c81cf2f7fe
Configurable media_type for mustache template encoding on append processor (#76210) 2021-08-10 15:13:36 -05:00
Mark Vieira f03809dc10 Add packaging upgrade tests to packaging pull requests 2021-08-10 12:43:38 -07:00
James Rodewig 6b29dc07e5
[DOCS] Remove `.com` from `elasticsearch.host` setup snippet (#76200) (#76311)
Co-authored-by: Gema Liano Benavides <46450383+gemaliano@users.noreply.github.com>
2021-08-10 14:32:55 -04:00
Benjamin Trent e8a3b05fcf
Only start re-assigning persistent tasks if they are not already being reassigned (#76258)
* Only start re-assigning persistent tasks if they are not already being reassigned

* adding tests addressing PR comments

* addressing Pr COmments

* addressing PR comments + style"

* improving test rigor
2021-08-10 14:11:16 -04:00
David Turner 8cc8043a9c
Extract HttpClientStatsTracker (#76279)
Extracts the functionality in `AbstractHttpServerTransport` that keeps
track of the HTTP client stats so that it can be more easily tested in
isolation.
2021-08-10 18:33:27 +01:00
James Rodewig bd665103ed
[DOCS] Change `_routing` to `routing` in mget API docs (#76214) (#76304)
The request body parameter is `routing`, not `_routing`.

Co-authored-by: Daniel Imfeld <daniel@imfeld.dev>
2021-08-10 13:08:50 -04:00