Commit Graph

54750 Commits

Author SHA1 Message Date
James Rodewig 0e1aa14bc8
[DOCS] EQL: Remove support for single quote strings (#62479) 2020-09-17 09:19:04 -04:00
Luca Cavanna 6f688c4295
Minor cleanup of runtime fields classes (#62531)
This commit addresses some compiler warnings in the runtime fields classes
2020-09-17 14:45:55 +02:00
Tim Vernum 58e5418513
Unmute BasicLicenseUpgradeIT (#62349)
Failure was temporary, caused by an unmerged BWC PR

Relates: #62301, #62286
Resolves: #62299
2020-09-17 22:03:39 +10:00
David Kyle 92f930c080
[ML] Add datafeed run time fields integration test (#62535) 2020-09-17 12:58:17 +01:00
Andrei Dan c1746afffd
ILM migrate data between tiers (#61377)
This adds ILM support for automatically migrating the managed
indices between data tiers.

This proposal makes use of a MigrateAction that is injected
(similar to how the Unfollow action is injected) in phases that
don't define index allocation rules using the AllocateAction or
don't explicitly define the MigrateAction itself (regardless if it's
enabled or disabled).
2020-09-17 10:56:49 +01:00
Christoph Büscher 5cbad94aa0
Fix condition in ILM step that cannot be met (#62377)
ReplaceDataStreamBackingIndexStep#performAction seems to perform an equality
check on an original Index and the write indexes names, but because this
compares an Index instance to a String, the condition can never be met. This PR
changes this comparison.
2020-09-17 11:22:29 +02:00
Alan Woodward 41bce50e71
Introduce FetchContext (#62357)
We currently pass a SearchContext around to share configuration among
FetchSubPhases. With the introduction of runtime fields, it would be useful
to start storing some state on this context to be shared between different
subphases (for example, stored fields or search lookups can be loaded lazily
but referred to by many different subphases). However, SearchContext is a
very large and unwieldy class, and adding more methods or state here feels
like a bridge too far.

This commit introduces a new FetchContext class that exposes only those
methods on SearchContext that are required for fetch phases. This reduces
the API surface area for fetch phases considerably, and should give us some
leeway to add further state.
2020-09-17 09:46:03 +01:00
David Turner cdc175c195
Suppress stack in VersionConflictEngineException (#62433)
`VersionConflictEngineException` is thrown on the hot path for updates,
but stack traces are expensive to compute and transport and rarely
useful for this kind of exception. This commit avoids computing the
stack trace for these exceptions.
2020-09-17 09:25:22 +01:00
Tanguy Leroux e4be206ae9
Remove ShardClusterSnapshotRestoreIT and fold test in DataStreamsSnapshotsIT (#62470)
ShardClusterSnapshotRestoreIT is confusing as we already have a 
very complete SharedClusterSnapshotRestoreIT test suite. This 
commit removes ShardClusterSnapshotRestoreIT and folds its 
unique test in DataStreamsSnapshotsIT.
2020-09-17 09:34:04 +02:00
David Turner 1edb4f9919
Log if recovery affected by disconnect (#62437)
Today we only emit `DEBUG` logs if the source disconnects from the
target during a recovery. This deserves to be noisier by default since
it should be rare and may help users identify other problems with their
network or with their shard movements.

This commit promotes this message to `INFO`. There's no need for `WARN`
since these days we will normally resume the recovery where it left off.
2020-09-17 08:09:17 +01:00
Armin Braun cb5f104854
Add WARN Logging on Slow Transport Message Handling (#62444)
Add simple WARN logging on slow inbound TCP messages.
2020-09-17 08:46:46 +02:00
Ignacio Vera ebb381c6bc
Introduce a sparse HyperLogLogPlusPlus class for cloning and serializing low cardinality buckets (#62480)
Reduces the memory footprint of an HLL++ structure that uses Linear counting when cloning or deserialising the data structure.
2020-09-17 08:06:49 +02:00
James Rodewig ca11665c03
[DOCS] Fix broken link to EC hot warm profile (#62514) 2020-09-16 22:07:40 -04:00
Kelly Murphy fa5320a125
[DOCS] Fix outdated cross doc link (#62504) 2020-09-16 17:37:01 -04:00
Julie Tibshirani 3b9e18b9b3 Remove unused test search context for significant_terms. 2020-09-16 14:24:30 -07:00
Adam Locke b420caa723
[DOCS] Updating CCR setup to be more tutorial focused (#62256)
* Applying some initial changes.

* Updating intro and screenshots.

* Removing unnecessary links, streamlining content, and adding GIF.

* Adding what's next section.

* Removing what's next.

* Minor edits.

* Apply suggestions from code review

Co-authored-by: debadair <debadair@elastic.co>

* Incorporating review feedback.

* Moving CCR user privileges to another page, plus more edits.

* Apply suggestions from code review

Co-authored-by: debadair <debadair@elastic.co>

* Incorporating more review feedback.

* Adding TESTSETUP to fix build errors.

* Update docs/reference/ccr/getting-started.asciidoc

Co-authored-by: debadair <debadair@elastic.co>

* Swapping GIF for mp4 hosted on web team CMS.

* Removing GIF in favor of mp4.

Co-authored-by: debadair <debadair@elastic.co>
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
2020-09-16 16:41:11 -04:00
Marios Trivyzas 8be400b773
EQL: Forbid usage of ['] for string literals (#62458)
The usage of single quotes to wrap a string literal is forbidden
and an error encouraging the user to user double quotes is returned.

Tests are properly adjusted.

Relates to #61659
2020-09-16 21:21:58 +02:00
Luca Cavanna e1882cd9cd
Runtime fields: rename fielddata and mapped field type classes (#62483)
With this commit we rename all of the fielddata, doc_values and mapped field type classes for runtime fields to not start with the Script prefix but rather their runtime type (e.g. Boolean) and only then Script
2020-09-16 20:19:19 +02:00
Martijn van Groningen ecf370cd67
Ignore 404 when wiping data streams. (#62484)
It is possible in mixed version clusters (nodes prior to 7.10)
that a 404 is returned when wiping all data streams.

This is because there are no data streams and
the coordinator node is on a version that doesn't
mark the delete request for wildcard usage.
2020-09-16 19:15:00 +02:00
Costin Leau 1c2362fcf2
EQL: Fetch sequence documents using Point-In-Time (#62469)
To preserve the PIT semantics, the retrieval of results has moved from
using multi-get to using an idsQuery.
2020-09-16 20:01:16 +03:00
Jay Modi fb4e4ff00c
LocalNodeMasterListener is a regular listener (#62422)
This commit makes the LocalNodeMasterListener interface extend the
ClusterStateListener interface and use a default implementation for
detecting whether the local node master status changed.
2020-09-16 10:15:05 -06:00
James Rodewig 86a0f15733
[DOCS] EQL: Use consistent string notation (#62472) 2020-09-16 11:29:52 -04:00
Fernando Briano 0cfb3e9ce5
Adds quotes to timestamp values in runtime_fields/40_date YAML test (#62468) 2020-09-16 16:28:56 +01:00
Dimitris Athanasiou 513117ae6a
Generalize AsyncTwoPhaseIndexer first phase (#61739)
Current implementations of the indexer are using aggregations.
Thus each search step executes a search action. However,
we can generalize that to allow for any action that returns a `SearchResponse`.
This commit abstracts the search phase from the search action.
2020-09-16 18:27:18 +03:00
Adrien Grand 115c48fdda
Speed up merging when source is disabled. (#62443)
The CodecReader wrapper we use to remove the `_recovery_source` field
doesn't override `StoredFieldsreader#getMergeInstance`, which has the
undesired side-effect of preventing the wrapped stored fields reader
from optimizing merging.
2020-09-16 17:13:53 +02:00
Adrien Grand 0418e9c09b
Upgrade to lucene-8.7.0-snapshot-9cd3af50f80. (#62450)
This new snapshot contains the following JIRAs that we're interested in:
 - [LUCENE-9525](https://issues.apache.org/jira/browse/LUCENE-9525)
Better handling of small documents. This should improve retrieval times
when documents are less than ~1kB.
 - [LUCENE-9510](https://issues.apache.org/jira/browse/LUCENE-9510)
Faster flushes when index sorting is enabled by not compressing the
temporary files that store stored fields and term vectors.
2020-09-16 17:13:39 +02:00
Luca Cavanna acd2395eaf
Runtime fields: rename script classes (#62448)
With this commit we rename the script classes used for each mapped field type used for runtime fields. The new naming is a shorter version of the previous one: from e.g. BooleanScriptFieldScrip to BooleanScript . We also move such classes to the existing mapper package.
2020-09-16 17:08:03 +02:00
James Rodewig db52f8485b [DOCS] EQL: Clarify wildcard operator 2020-09-16 11:05:00 -04:00
James Rodewig 9e325bb810 [DOCS] EQL: Make operator refs consistent 2020-09-16 11:03:09 -04:00
James Rodewig 7274b42a14 [DOCS] EQL: Move comparison operator defs 2020-09-16 10:54:02 -04:00
James Rodewig 7630064a25 [DOCS] EQL: Add xrefs to EQL intro 2020-09-16 10:41:56 -04:00
Tanguy Leroux fd7f936c7e
Wait for relocations and disk threshold monitor in DiskThresholdDeciderIT (#62358)
Closes #62326
2020-09-16 16:14:06 +02:00
Lee Hinman 5263ca01d4
Remove data_frozen node role (tier) and frozen ILM phase (#62403)
With the differentiation between searchable snapshots on the cold phase and searchable snapshots on
the frozen phase not implemented, there is no need to have a separate phase/tier for now. This
commit removes the frozen phase and tier, which can be added back at a later time.

(this tier was never in a released version, so this is not a breaking change)

Relates to #60983
Relates to #60994
Relates to #60848
2020-09-16 08:10:16 -06:00
James Rodewig 4ba7c408f7
[DOCS] Document `toJSON` function for role query (#62257) 2020-09-16 09:51:36 -04:00
William Brafford 035dfce16e
Deprecate xpack.eql.enabled setting and make it a no-op (#61375)
* Deprecate xpack.eql.enabled and make it a no-op
* Remove uses of xpack.eql.enabled
2020-09-16 09:48:16 -04:00
James Rodewig 9b9889b17e
[DOCS] Document static/dynamic security settings (#62181) 2020-09-16 09:36:42 -04:00
Christoph Büscher f5f72648d4 Muting SimpleSecurityNetty4ServerTransportTests 2020-09-16 15:13:37 +02:00
Wylie Conlon 4be761fde4
[DOCS] Update range field type docs (#62112) 2020-09-16 09:07:51 -04:00
Dan Hermann 03d0e191c6
Fix failure in AppendProcessorTests.testAppendingToListWithDuplicatesDisallowed (#62385) 2020-09-16 08:04:20 -05:00
Benjamin Trent 4bbd15084d
[ML] fixes testWatchdog test verifying matcher is interrupted on timeout (#62391)
Constructing the timout checker FIRST and THEN registering the watcher allows the test to have a race condition.

The timeout value could be reached BEFORE the matcher is added. To prevent the matcher never being interrupted, a new timedOut value is added to the watcher thread entry. Then when a new matcher is registered, if the thread was previously timedout, we interrupt the matcher immediately.

closes #48861
2020-09-16 08:36:34 -04:00
Lyudmila Fokina 183533734b
Update authc failure headers on license change (#61734)
DefaultAuthenticationFailureHandler determines the set of response headers based on the active realms at startup.
However, the Realms list may be empty if the node thinks it has a TRIAL or BASIC license at startup (because we default security off in those cases).
The handler is never updated if the license state changes.

The fix is to add a call-back for license change to update the AuthenticationFailureHandler's Headers to reflect the new license property.
Resolves: #56318
2020-09-16 11:21:12 +02:00
Christoph Büscher 6485698750 Muting LogstashSystemIndexIT.testPipelineCRUD 2020-09-16 11:05:28 +02:00
Tim Vernum b8401e1f99 Reapply "[ML] Fix full cluster upgrade failure where index does not exist (#62430)"
This reverts commit 54f38cf007 which
incorrectly reverted 5c870df8fe.
2020-09-16 18:47:46 +10:00
BigPandaToo c9f3e38a8b Revert "Adding authentication information to access token create APIs"
This reverts commit 51dbd9e584.
2020-09-16 10:45:16 +02:00
BigPandaToo 54f38cf007 Revert "[ML] Fix full cluster upgrade failure where index does not exist (#62430)"
This reverts commit 5c870df8fe.
2020-09-16 10:44:37 +02:00
David Kyle 5c870df8fe
[ML] Fix full cluster upgrade failure where index does not exist (#62430)
Depending on the order in which the tests are run the .ml-notifications-000001
may or may not have been created.
2020-09-16 09:30:55 +01:00
BigPandaToo c016868ab9 Merge branch 'master' of https://github.com/elastic/elasticsearch into APIresponse 2020-09-16 09:07:19 +02:00
Armin Braun d433d2567a
Don't Start Redundant ConsistentSettingsService (#62283)
The consistent settings service is only used in tests so far. No need to start it
unless it's actually used.
2020-09-16 08:40:52 +02:00
Hendrik Muhs 394063cb64
[Transform] Make pivot validation sub-agg aware (#62381)
With the addition of sub aggregations like filter, the validation could fail if 2 sub aggs use the
same output name. This change makes validation sub-agg aware.

fixes #57814
2020-09-16 07:54:29 +02:00
Ignacio Vera ea19854598
Adds bucketOrd back to cardinality algorithms (#62389) 2020-09-16 07:48:13 +02:00