Commit Graph

60468 Commits

Author SHA1 Message Date
Igor Motov 189f6509fa
Reenable bwc tests for #76475 (#76576)
Reenabling bwc tests after merge of #76475
2021-08-16 11:42:15 -10:00
Przemysław Witek 7a4d320e7b
Set version to 7.15 in BWC code (#76577) 2021-08-16 22:07:57 +02:00
Lyudmila Fokina 34ea976ccd
Don't remove warning headers on all failure (#76434)
* Don't remove warning headers on all failure

We shouldn't remove warning when request is failing not because of
security reasons (syntax error for ex.).
Note, that security related failure could happen not only during
authentication (therefore we will check for the rest status), also all
failures happened during authentication will be considered security
related and warnings will be removed from the response.

Resolves: #75739
2021-08-16 21:19:37 +02:00
Igor Motov 97f6b0cfcd
Disable bwc tests for #76475 (#76541)
Disabling bwc tests to merge #76475
2021-08-16 08:05:13 -10:00
Ryan Ernst 797c6b5d82
Re-enable bwc tests (#76567)
This commit re-enables the bwc tests after the backport of #76476.

relates #76537
2021-08-16 09:44:33 -07:00
Francisco Fernández Castaño 2c132fe5f7
Keep track of data recovered from snapshots in RecoveryState (#76499)
Adds new field to recovery API to keep track of amount of data
recovered from snapshots.

The normal recovered_bytes field remains and is also increased for
recovery from snapshot but can go backwards in the unlikely case
that recovery from snapshot fails to download a file.

Relates #73496
2021-08-16 18:18:37 +02:00
Przemysław Witek f9d30adf6f
[Transform] Align transform checkpoint range with date_histogram interval for better performance (#74004) 2021-08-16 17:50:11 +02:00
Bogdan Pintea a388f70615
EQL: Remove "wildcard" function (#76099)
This removes "wildcard" as an available function in EQL. This has
already been replace with "like" and "regex" embedded syntax (and
respective case insensitive variants).
2021-08-16 17:48:35 +02:00
Seth Michael Larson 5424748840
Fix 'accept' and 'content_type' fields for search_mvt API 2021-08-16 10:35:04 -05:00
Ryan Ernst 7e4d2f9303
Add persistent licensed feature tracking (#76476)
Licensed feature tracking utilizes the existing license level checks to
track when a feature is used. However, some features check the license
level at the start of an operation or when enabling a feature, but then
the tracking only captures the beginning time.

This commit reworks the licensed feature framework to use a new
LicensedFeature class which will eventually replace
XPackLicenseState.Feature values. There are two LicensedFeature
implementations, one for "momentary" features that are tracked just at
the moment they are used, and "persistent" features that are considered
"on" until the feature is untracked. The usage map of tracked features
is cleaned up every hour, and those features that have not been used in
the last 24 hours are removed from tracking.

Not all features are converted to LicensedFeature yet. Instead, a few
features have been converted to demonstrate how it can be done, so that
the rest can be done in parallel at a future time.

Co-authored-by: Tim Vernum <tim@adjective.org>
2021-08-16 08:24:56 -07:00
William Brafford 81b6d10216
Add system data streams to feature state snapshots (#75902)
Add system data streams to the "snapshot feature state" code block, so that if
we're snapshotting a feature by name we grab that feature's system data streams
too. Handle these data streams on the restore side as well.

* Add system data streams to feature state snapshots
* Don't pass system data streams through index name resolution
* Don't add no-op features to snapshots
* Hook in system data streams for snapshot restoration
2021-08-16 11:10:39 -04:00
Jack Conradson 82919dab52
fix the error message for instance methods that don't exist (#76512)
This fixes the error message in Painless for when an instance-style method isn't found. It now prints the 
number of arguments instead of the memory location of the nodes data structure.
2021-08-16 08:08:47 -07:00
bellengao 58feb4e195
ILM: Add validation of the number_of_shards parameter in Shrink Action of ILM (#74219)
Add validation of the number_of_shards parameter in Shrink Action of ILM
2021-08-16 15:18:15 +01:00
Devon Thomson 5c53a66e8f
remove dashboard only reserved role (#76507)
Removes the deprecated kibana_dashboard_only_user from the set of reserved roles in Elasticsearch as this legacy functionality is being removed from Kibana.

Relates: elastic/kibana#54755.
2021-08-16 10:15:50 -04:00
hanbj 1d8bad6ffc
Fix Stack Overflow in UnassignedInfo in Corner Case (#76480)
We kept wrapping the collection over and over again which in extreme corner cases could lead to a SOE.

Closes #76490
2021-08-16 06:35:50 +02:00
sindhusp f4e3f33f77
Add (Extended)KeyUsage KeyUsage, CipherSuite & Protocol to SSL diagnostics (#65634)
This commit extends the SSL diagnostics message to include descriptions of the

- The KeyUsage and ExtendedKeyUsage of the peer's certificate
- The CipherSuite & Protocol (TLS/SSL version) of the current session 

These can be helpful in diagnosing SSL errors.

Co-authored-by: Tim Vernum <tim@adjective.org>
2021-08-16 13:19:37 +10:00
Henning Andersen 51d524bb2c
Add recovery from snapshot to tests (#76535)
Randomly add to use a snapshot for recovery to searchable snapshot and
snapshot tests to verify that recover from snapshot does not break other
features (those should not care about the flag).

Relates #76237
2021-08-15 19:32:30 +02:00
Armin Braun 58eefdd757
Reenable BwC Tests after #76532 (#76534)
Reenabling BwC tests now that #76532 has been merged.
2021-08-14 19:49:04 +02:00
Henning Andersen b138d600df
Fix memory/breaker leaks for outbound responses (#76474)
Outbound responses would not get the expected `decRef`, resulting in
memory and/or circuit breaker leaks. In particular, the
`GetCcrRestoreFileChunkResponse` expects this, causing a leak when
a follower bootstraps.

Relates #65921
2021-08-14 18:41:42 +02:00
Armin Braun 04dd27395c
Disable BwC Tests to Merge #76532 (#76533)
Disabling BwC tests so that #76532 can be merged.
2021-08-14 18:28:43 +02:00
Yang Wang 245ba384a1
Add sort and pagination support for QueryApiKey API (#76144)
This PR adds support for sort and pagination similar to those used with
regular search API. Similar to the query field, the sort field also
supports only a subset of what is available for regular search.
2021-08-14 08:57:15 +10:00
Tim Brooks 60d707b0df
Remove broken compression version assertion
A recent commit added an assertion to make sure that LZ4 is not
configured on an incompatible node. This assertion is currently broken
in tests. This commit removes it.
2021-08-13 14:20:48 -06:00
Tim Brooks e6fd459a6e
Respond with same compression scheme received (#76372)
This is related to #73497. Currently, we only use the configured
transport.compression_scheme setting when compressing a request or a
response. Additionally, the cluster.remote.*.compression_scheme
setting is ignored. This commit fixes this behavior by respecting the
per-cluster setting. Additionally, it resolves confusion around inbound
and outbound connections by always responding with the same scheme that
was received. This allows remote connections to have different schemes
than local connections.
2021-08-13 13:29:22 -06:00
Armin Braun 83afe8807a
Fix Mutable Data Structure in SnapshotsInProgress (#76505)
The feature info instances shouldn't be mutable to prevent accidental bugs down the line.
2021-08-13 19:16:56 +02:00
James Rodewig 7d70414c00
[DOCS] Update vector tile search API title (#76506) 2021-08-13 11:55:11 -04:00
Lyudmila Fokina c5b73fcaf8
Set elastic password and generate enrollment token (#75816)
Set elastic password and generate enrollment token for initial node

Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
2021-08-13 17:00:35 +02:00
Nik Everett f56d7fcc55
Fix eclipse compilation after plugin spi (#76437)
Painless extensions now compile against it's `spi` subproject. Gradle
and IntelliJ rig the dependencies correctly but Eclipse doesn't seem to
manage it. This temporarily declares the `spi` as a `compileOnly`
dependency when in Eclipse land so Eclipse can see the classes.
2021-08-13 10:50:21 -04:00
Ioannis Kakavas 59d75f213f
Add kibana-system service account (#76449)
This change introduces a Service Account for Kibana to use when
authenticating to Elasticsearch. The Service Account with
kibana service name under the elastic namespace,
uses the same RoleDescriptor as the existing kibana_system
built-in user and is its functional equivalent when it comes to
AuthZ.
2021-08-13 17:00:13 +03:00
Julie Tibshirani 2ddbd62291
Mention match_only_text in disk usage docs (#76416)
* Mention match_only_text in disk usage docs

Previously we explained how to manually disable norms, freqs, and positions. We
now have a ready-made solution in the new `match_only_text` field type.

* Fixing typo and minor grammar changes

Co-authored-by: Adam Locke <adam.locke@elastic.co>
2021-08-13 09:31:09 -04:00
Rene Groeschke ba7c7d783d
Update redline library to 1.2.10 (#76440)
* Update redline library to 1.2.10

The redline team just released version 1.2.10 of the redline
library which contains our fix of the rpm signatures / headers.

Also a PR to update that dependency in the ospackage plugin has been
raised at https://github.com/nebula-plugins/gradle-ospackage-plugin/pull/402

* Update common about enforcing redline 1.2.10
2021-08-13 09:29:37 -04:00
Francisco Fernández Castaño a6aa5998aa
Add third party integration tests for snapshot based recoveries (#76489)
This commit adds third party integration tests for snapshot based
recoveries in S3, Azure and GCS.

Relates #73496
2021-08-13 15:11:28 +02:00
James Rodewig 27d28e1976
[DOCS] Note assumptions for shard size and count recommendations (#76353)
On the "Size your shards" page, the shard size recommendation assumes a time
series use case. Similarly, users shouldn't count nearly empty and rarely used
Enterprise Search system indices against the recommended shard count limit.

Closes #76328.
2021-08-13 08:44:19 -04:00
Adam Locke 08865e31b2
[DOCS] Fix formatting for Watcher settings (#76491)
Adds missing description list tagging for two Watcher settings.

Resolves #76484
2021-08-13 08:33:10 -04:00
István Zoltán Szabó b9d875bf68
[DOCS] Updates description of model_prune_window property in ML shared (#76487) 2021-08-13 12:18:38 +02:00
Francisco Fernández Castaño 2ebe5cd075
Add peer recoveries using snapshot files when possible (#76237)
This commit adds peer recoveries from snapshots. It allows establishing a replica by downloading file data from a snapshot rather than transferring the data from the primary. 

Enabling this feature is done on the repository definition. Repositories having the setting `use_for_peer_recovery=true` will be consulted to find a good snapshot when recovering a shard.

Relates #73496
2021-08-13 10:42:16 +02:00
Tim Vernum ff5df64a42
[Test] Fix error message (path name) on Windows (#76478)
Closes: #76166
2021-08-13 16:45:15 +10:00
Tim Vernum c2760bebd2
Adjust logging levels for LDAP failures (#76058)
This change increases the logging level for LDAP failures in 3 cases:

1. When a search fails, the failure details are now logged at DEBUG
rather than TRACE. The errors are useful for diagnostic purposes and
it should not be necessary to turn on TRACE to see them

2. When a failure occurs when attempting to bind using the configured
"bind_dn" and password, this is now a WARN rather than DEBUG. Failures
for user supplied credentials (e.g. basic auth) are still logged as
DEBUG because these are typically not under the control of the cluster
administrator.

3. When a failure occurs while attempting to retrieve a connection
from an LDAP connection pool, this will be logged at a WARN level.
This is almost always an error that the cluster administrator should
be aware of and seek to resolve.

In some cases this may cause 2 sets of log messages (one from LDAP and
one from the authentication service) however it is not always the case
that both are logged, and even when they are they may have different
levels of detail (for example, the authentication service message does
not always include the DN of the bind user).
2021-08-13 13:16:37 +10:00
Tim Vernum 01f20a3343
Default hasher to PBKDF2_STRETCH on FIPS mode (#76274)
When running in FIPS mode, (fips_mode.enabled: true), the default
password hasher is now "pbkdf2_stretch"

In non-FIPS mode the default is still "bcrypt"

In 7.x and earlier, the default hasher was always "bcrypt"
In 8.0-alpha1, the default hasher on FIPS was "pbkdf2"

Resolves: #66840
2021-08-13 12:18:44 +10:00
Tim Brooks 425b7b280b
Add docs for production ready compression settings (#76441)
In 7.15, we intend for the indexing_data compression level and the
compression scheme lz4 to no longer be experimental. This commit
updates the documentation to reflect this. Additionally, it adds
missing docs for the cluster.remote.*.transport.compression_scheme
setting.

Relates to #73497.
2021-08-12 16:48:56 -06:00
Igor Motov e35021132e
Fix docCountError calculation for multiple reduces (#76391)
Fix docCountError calculation in case of multiple reduces. It fixes 2 mistakes
in #43874. The first error was introduced in the original PR, where unknown doc
count errors were initialized equal to 0, the second was introduced during in
order to fix the first one by ignoring these 0s, which essentially disabled the
original fix.

Fixes #75667
2021-08-12 11:50:17 -10:00
James Rodewig 5c954df521
[DOCS] Fix data type for create snapshot API's `metadata` param (#76465) 2021-08-12 16:38:56 -04:00
Armin Braun 1f080e3aa5
Fix Snapshot State Machine Issues around Failed Clones (#76419)
With recent fixes it is never correct to simply remove a snapshot from the cluster state without
updating other snapshot entries if an entry contains any successful shards due to possible dependencies.
This change reproduces two issues resulting from simply removing snapshot without regard for other queued
operations and fixes them by having all removal of snapshot from the cluster state go through the same
code path.
Also, this change moves the tracking of a snapshot as "ending" up a few lines to fix an assertion about finishing
snapshots that forces them to be in this collection.
2021-08-12 20:41:44 +02:00
Nik Everett e305a6bed7
Name `BulkItemResponse` ctors (#76439)
* Name `BulkItemResponse` ctors

`BulkItemResponse` can contain either a success or failure. This
replaces the two constructors used to build either case with named
static methods. So instead of
```
return new BulkItemResponse(0, OpType.CREATE, createResponse);
return new BulkItemResponse(0, OpType.CREATE, failure);
```
you now use
```
return BulkItemResponse.success(0, OpType.CREATE, createResponse);
return BulkItemResponse.failure(0, OpType.CREATE, failure);
```

This makes it marginally easier to read code building these things - you
don't have to know the type of the parameter to know if its a failure
or success.

* Consistent

* Mock response
2021-08-12 14:41:26 -04:00
James Rodewig c238296b7a
[DOCS] Add missing `timeout` param to create pipeline API docs (#76432) 2021-08-12 14:32:34 -04:00
Adam Locke 59aeb8552c
change a typo in first letter of a user query (#76394) (#76450)
Co-authored-by: Arseni Prokharchyk <2657789+arsen91@users.noreply.github.com>
2021-08-12 14:28:51 -04:00
James Rodewig 20f000e95a
[DOCS] Fix query params for create snapshot API (#76436)
Moves the `master_timeout` and `wait_for_completion` parameters to a query parameters section.
2021-08-12 13:11:12 -04:00
István Zoltán Szabó 9b0417f2df
[DOCS] Comments out links that points to regression loss functions (#76435)
* [DOCS] Comments out links that points to regression loss functions.

* Update docs/reference/ml/df-analytics/apis/get-trained-models.asciidoc
2021-08-12 18:33:42 +02:00
Jack Conradson 63425d50b2
Clean up whitelist loading for core contexts in painless plugin (#76392)
This change creates a standard for loading additional allow listed elements for the core contexts 
within the ScriptPlugin. For each core context we look for org.elasticsearch.script.<context-name>.txt 
and if it exists we load it automatically for that context.
2021-08-12 08:22:10 -07:00
Costin Leau 1b617f1d74
QL: Remove delegating method (#76418)
Remove typedParsing method in favor of ParserUtils

Follow-up to #76399
2021-08-12 10:32:41 -04:00
Benjamin Trent aaeecd7729
[ML][Transform] fixing testFailureCounterIsResetOnSuccess test failure #76397 (#76417)
There are two possible race conditions that were not
previously handled in this test.

- Since the syncconfig was null, it may be that the
  transform actually gets set to stopping/stopped
  and its unable to kick off another indexing pass
- It may also be that the indexer thread is still
  finishing up work when the second execution is
  requested, so it returns false.

Adding a sync config and assertBusy handles these
cases. Ran 1k+ times locally with this change
and it never failed. Without, it failed ~10 runs.

closes https://github.com/elastic/elasticsearch/issues/76397
2021-08-12 10:01:49 -04:00