Commit Graph

2277 Commits

Author SHA1 Message Date
Mark Vieira 92b59d994f
Continue to publish REST API specifications under Apache 2.0 license (#68488) 2021-02-03 13:46:20 -08:00
Joe Gallo 4d18334442
Add max_single_primary_size as a condition for the rollover index API (#67842) 2021-02-03 10:39:06 -05:00
Mark Vieira a92a647b9f Update sources with new SSPL+Elastic-2.0 license headers
As per the new licensing change for Elasticsearch and Kibana this commit
moves existing Apache 2.0 licensed source code to the new dual license
SSPL+Elastic license 2.0. In addition, existing x-pack code now uses
the new version 2.0 of the Elastic license. Full changes include:

 - Updating LICENSE and NOTICE files throughout the code base, as well
   as those packaged in our published artifacts
 - Update IDE integration to now use the new license header on newly
   created source files
 - Remove references to the "OSS" distribution from our documentation
 - Update build time verification checks to no longer allow Apache 2.0
   license header in Elasticsearch source code
 - Replace all existing Apache 2.0 license headers for non-xpack code
   with updated header (vendored code with Apache 2.0 headers obviously
   remains the same).
 - Replace all Elastic license 1.0 headers with new 2.0 header in xpack.
2021-02-02 16:10:53 -08:00
Hendrik Muhs 4cbe61467c
add possibility to mute yaml tests by operating system (#67681)
this change adds the possibility to mute yaml tests based on operating system to avoid muting whole
tests
2021-02-01 09:33:23 +01:00
William Brafford 42b748588d
Allow the "*,-*" ("match none") pattern for destructive actions when destructive_requires_name is true (#68021)
Since the "*,-*" pattern resolves to "no indices", it makes a normally
destructive action into a non-destructive one. Rather than throwing a
wildcards-not-allowed exception, we can allow this pattern to pass
without triggering an exception. This allows the security layer to
safely use a "*,-*" pattern to indicate a "no indices" result for its
index resolution step, which is important because otherwise we get
wildcards-not-allowed exceptions when trying to delete nonexistent
concrete indices. For simplicity, we require exactly "*,-*", rather than
any other wildcards that might be logically equivalent.
2021-01-28 14:08:29 -05:00
David Turner 06e141888f
Reinstate BWC snapshot tests (#67938)
This commit mostly reverts #67934, except for the change to the version
constant `REPOSITORY_UUID_IN_REPO_DATA_VERSION`.

Completes the backport of #67829 via #67899
2021-01-25 18:36:12 +00:00
David Turner faed3e7199
Temporarily suppress BWC snapshot tests (#67934)
This commit suppresses any BWC tests related to snapshots in `master` so
that #67899 can be merged to `7.x`. It will mostly be reverted after the
merge of #67899 is complete.

Relates #66431
2021-01-25 17:48:47 +00:00
David Turner e5a15d4fcb
Introduce repository UUIDs (#67829)
Today a snapshot repository does not have a well-defined identity. It
can be reregistered with a different cluster under a different name, and
can even be registered with multiple clusters in readonly mode.

This presents problems for cases where we need to refer to a specific
snapshot in a globally-unique fashion. Today we rely on the repository
being registered under the same name on every cluster, but this is not a
safe assumption.

This commit adds a UUID that can be used to uniquely identify a
repository. The UUID is stored in the top-level index blob, represented
by `RepositoryData`, and is also usually copied into the
`RepositoryMetadata` that represents the repository in the cluster
state. The repository UUID is exposed in the get-repositories API; other
more meaningful consumers will be added in due course.
2021-01-25 12:17:52 +00:00
David Turner bc1f50c523
Permit wait_for_active_shards warnings in master (#67498)
Part of the fixes for #66419, this commit permits nodes to emit the
deprecation warning regarding not specifying `?wait_for_active_shards`
when closing an index in 7.x versions for x ≥ 12. This change is
required on `master` too since the BWC tests encounter these warnings.

Relates #67246, which is the 7.x part of this change.
2021-01-14 15:55:43 +00:00
Andrei Stefan e3386e155c
Add minimum compatibility version to SearchRequest (#65896)
* Adds a minimum version request parameter to SearchRequest.
The minimum version helps failing a request if any shards
involved in the search do not meet the compatibility requirements
(all shards need to have a version equal or later than the minimum
version provided).
2021-01-13 00:50:30 +02:00
David Turner ec08f924c7
Introduce ?wait_for_active_shards=index-setting (#67158)
In 7.x the close indices API defaulted to `?wait_for_active_shards=0`
but from 8.0 it defaults to respecting the index settings instead.  This
commit introduces the `index-setting` value for this parameter on this
API allowing users to opt-in to the future behaviour today, and emits a
deprecation warning indicating that the default no longer needs to be
used and will be unsupported in future.

In 7.x a follow up PR will introduce support for the same
`index-setting` value for this parameter and will emit deprecation
warnings if users try and use the default instead.

Relates #66419
2021-01-11 08:33:16 +00:00
Nik Everett a9e8a6a31b
Update skip after backport of #67043 (#67191)
Now that #67043 has been backported we can update the skip so the bwc
tests don't complain.
2021-01-07 17:01:53 -05:00
Nik Everett f23e568948 Update skip before backport of #67043
When I merged #67043 it had an integration test for the thing it was
fixing but it still fails in the bwc tests. Yikes! I should know better
but life is life. Anyway, this updates the skip to ignore the test for
now. I'll reenable once the backport is in.
2021-01-07 10:57:39 -05:00
Nik Everett b0747c5a76
Fix bug with nested and filters agg (#67043)
Fixes a bug where nested documents that match a filter in the `filters`
agg will be counted as matching the filter. Usually nested documents
only match if you explicitly ask to match them. Worse, we only mach them
in the "filter by filter" mode that we wrote to speed up date_histogram.
The `filters` agg is fairly rare, but with #63643 we run
`date_histogram` and `range` aggregations using `filters.
2021-01-07 10:05:59 -05:00
Julie Tibshirani 1515f36f7f
Make sure shared source always represents the top-level root document. (#66725)
We started passing down the root document's _source when processing
nested hits, to avoid reloading and reparsing the root source for each hit.
Unfortunately the approach did not work when there are multiple layers of
`inner_hits`. In this case, the second-layer inner hit received its immediate
parent's source instead of the root source. This parent source is filtered to
just contain the parts corresponding to the nested document, but the source
parsing logic is designed to always operate on the top-level root source. This
caused failures when loading the second-layer inner hits.

This PR makes sure to always pass the root document's _source when processing
inner hits, even if there are multiple layers.
2021-01-05 08:17:26 -08:00
Nik Everett dd1ffe3900
Update skip after backport of #66295 (#66808)
Now that #66295 has landed in 7.11 we can run the bwc tests it adds
against that branch.
2020-12-23 16:33:34 -05:00
Ioannis Kakavas bd873698bc
Ensure CI is run in FIPS 140 approved only mode (#64024)
We were depending on the BouncyCastle FIPS own mechanics to set
itself in approved only mode since we run with the Security
Manager enabled. The check during startup seems to happen before we
set our restrictive SecurityManager though in
org.elasticsearch.bootstrap.Elasticsearch , and this means that
BCFIPS would not be in approved only mode, unless explicitly
configured so.

This commit sets the appropriate JVM property to explicitly set
BCFIPS in approved only mode in CI and adds tests to ensure that we
will be running with BCFIPS in approved only mode when we expect to.
It also sets xpack.security.fips_mode.enabled to true for all test clusters
used in fips mode and sets the distribution to the default one. It adds a
password to the elasticsearch keystore for all test clusters that run in fips
mode.
Moreover, it changes a few unit tests where we would use bcrypt even in
FIPS 140 mode. These would still pass since we are bundling our own
bcrypt implementation, but are now changed to use FIPS 140 approved
algorithms instead for better coverage.

It also addresses a number of tests that would fail in approved only mode
Mainly:

    Tests that use PBKDF2 with a password less than 112 bits (14char). We
    elected to change the passwords used everywhere to be at least 14
    characters long instead of mandating
    the use of pbkdf2_stretch because both pbkdf2 and
    pbkdf2_stretch are supported and allowed in fips mode and it makes sense
    to test with both. We could possibly figure out the password algorithm used
    for each test and adjust password length accordingly only for pbkdf2 but
    there is little value in that. It's good practice to use strong passwords so if
    our docs and tests use longer passwords, then it's for the best. The approach
    is brittle as there is no guarantee that the next test that will be added won't
    use a short password, so we add some testing documentation too.
    This leaves us with a possible coverage gap since we do support passwords
    as short as 6 characters but we only test with > 14 chars but the
    validation itself was not tested even before. Tests can be added in a followup,
    outside of fips related context.

    Tests that use a PKCS12 keystore and were not already muted.

    Tests that depend on running test clusters with a basic license or
    using the OSS distribution as FIPS 140 support is not available in
    neither of these.

Finally, it adds some information around FIPS 140 testing in our testing
documentation reference so that developers can hopefully keep in
mind fips 140 related intricacies when writing/changing docs.
2020-12-23 21:00:49 +02:00
Nik Everett 3e3152406a
Bust the request cache when the mapping changes (#66295)
This makes sure that we only serve a hit from the request cache if it
was build using the same mapping and that the same mapping is used for
the entire "query phase" of the search.

Closes #62033
2020-12-23 13:19:02 -05:00
Seth Michael Larson 11153fcb33
Use single backslash for nested paths (#66794) 2020-12-23 11:57:19 -06:00
Julie Tibshirani 652ff74adf Adjust test skips now that inner_hits fix is backported. 2020-12-21 15:08:34 -08:00
Julie Tibshirani 15f5758957
Fix regressions around nested hits and disabled _source. (#66572)
This PR fixes two bugs that can arise when _source is disabled and we fetch nested documents:
* Fix exception when highlighting `inner_hits` with disabled _source.
* Fix exception in nested `top_hits` with disabled _source.
* Add more tests for highlighting `inner_hits`.
2020-12-18 14:06:52 -08:00
Jim Ferenczi c756ce1acf
Sort field tiebreaker for PIT (point in time) readers (#66093)
This commit introduces a new sort field called `_shard_doc` that
can be used in conjunction with a PIT to consistently tiebreak
identical sort values. The sort value is a numeric long that is
composed of the ordinal of the shard (assigned by the coordinating node)
and the internal Lucene document ID. These two values are consistent within
a PIT so this sort criteria can be used as the tiebreaker of any search
requests.
Since this sort criteria is stable we'd like to add it automatically to any
sorted search requests that use a PIT but we also need to expose it explicitly
in order to be able to:
* Reverse the order of the tiebreaking, useful to search "before" `search_after`.
* Force the primary sort to use it in order to benefit from the `search_after` optimization when sorting by index order (to be released in Lucene 8.8.

I plan to add the documentation and the automatic configuration for PIT in a follow up since this change is already big.

Relates #56828
2020-12-18 12:13:12 +01:00
Steve Gordon 96555bfa33
Mark Cat Tasks API as experimental in rest-api-spec (#66536) 2020-12-17 17:54:21 +00:00
Rory Hunter 1eb50f876e Tweak version skip range in cat.plugin yml test 2020-12-17 10:06:32 +00:00
Rory Hunter 4ff612550e
Allow bootstrap plugins to appear in _cat/plugins (#66260)
Closes #66107.

Bootstrap plugins are not loaded in the main Elasticsearch process, but
instead take effect only when ES is starting. As such, these plugins are
skipped when ES loads all installed plugins.

As a result, it was impossible for the plugins _cat API to report
whether any bootstrap plugins are installed.

Fix this by adjusting how the loading process skips bootstrap plugins,
and then tweaking the plugins _cat API so that bootstrap plugins can
optionally be included in the response.
2020-12-17 09:30:16 +00:00
Julie Tibshirani 6bc56d18e5
Fix failure in fvh REST tests. (#66192)
In general, we can't guarantee that a match_all query will return documents in
the order they were indexed. This PR adds an ID to each document to avoid
relying on document order.
2020-12-15 11:28:51 -08:00
Jay Modi 7011cbac5d
Fix cat tasks api params in spec and handler (#66272)
This commit fixes the cat tasks api parameter specification and the
handler so that the parameters are consumed during request preparation.

Closes #59493
2020-12-14 12:29:25 -07:00
Jay Modi 410ae396bd
Remove suggest reference in some API specs (#66180)
This commit removes the reference to the suggest index metric in the
nodes stats and indices stats rest api spec files. Suggest has been
removed so it is no longer correct to have it in these files.

Closes #43407
2020-12-14 10:00:04 -07:00
Martijn Laarman e31e3dea32
Add `visibility` the to rest-spec-api (#56104) 2020-12-14 12:23:28 +01:00
Julie Tibshirani a40186a7f4 Adjust skip version in fvh REST tests.
We can expand the compatible versions now that the bug fix has been packported.
2020-12-10 13:39:47 -08:00
Fernando Briano 91595657a2
Wraps timestamp values in quotes in search.aggregation histogram YAML test (#66153) 2020-12-10 12:19:03 +00:00
Dimitrios Liappis 5b24829796
Mute fvh REST tests (#66149)
Relates #66147
2020-12-10 12:13:23 +02:00
Julie Tibshirani 3cbe0eadce Ensure consistent hit order in fvh REST tests. 2020-12-09 17:53:17 -08:00
Julie Tibshirani ddf1f4cdb8
Fix bug where fvh fragments could be loaded from wrong doc (#65641)
This PR fixes a regression where fvh fragments could be loaded from the wrong
document _source.

Some `FragmentsBuilder` implementations contain a `SourceLookup` to load from
_source. The lookup should be positioned to load from the current hit document.
However, since `FragmentsBuilder` are cached and shared across hits, the lookup
is never updated to load from the new documents. This means we accidentally
load _source from a different document.

The regression was introduced in #60179, which started storing `SourceLookup`
on `FragmentsBuilder`.

Fixes #65533.
2020-12-09 14:47:11 -08:00
Ioannis Kakavas 3cd93eef92
Skip REST YAML tests in FIPS 140 mode (#65735)
Currently we don't have a way to selectively mute REST YAML tests
in FIPS 140 mode.

This commit introduces a new feature (fips_140) that can be used
in skip blocks to allow that.
2020-12-09 15:45:31 +02:00
Martijn Laarman 8d3def3e1f
Add Accept & Content-Type headers to rest api spec (#53979)
Co-authored-by: Russ Cam <russ.cam@elastic.co>
2020-12-09 14:43:05 +01:00
Nik Everett 9e57a46c56
Update skip after backport (#66024)
After backporting #65707 we can now run it in our backwards
compatibility tests.
2020-12-08 12:01:56 -05:00
Nik Everett ecb4e16d35
Fix sneaky date_histogram bug (#65707)
`date_histogram` has a bug with `offset` and `extended_bounds` when it
needs to create an "empty" aggregation result: it includes the bounds
twice! Wooops!

I broke this a while back when I started trying to merge `offset` into
`Rounding`. I never finished that merge, sadly. Interestingly, we've
discovered that the merge is required to properly handle daylight
savings time (#56305) but it isn't really something we're looking to
solve today. For now, this just stops counting the offset twice.

Closes #65624
2020-12-07 15:39:52 -05:00
Nik Everett 92e803a06d
Fix test sending body as url parameter (#65779)
Our test framework randomly sends the body of requests over the `source`
parameter. We never send the body if it is more than 2000 bytes becuse
our HTTP receiver can't handle lines more the 4096 bytes. The thing is,
when we do that 2000 bytes check we do it against the reported length of
body, not the body after it has been url encoded. Than url encoding
strings can *vastly* increase their size. Which could cause us to send
some request over the URL that are longer than 4096 bytes.

This fixes that by checking the url encoded length as well. We keep the
2000 byte check of the unencoded length because it is a nice fast check,
even if it is a bit inaccurate.

Closes #65718
2020-12-07 12:26:34 -05:00
Seth Michael Larson 56a25bf08b
Mark Task APIs as experimental in rest-api-spec 2020-12-03 13:47:13 -06:00
Christoph Büscher 8c29e54bf3 Adapt test skip versions after backport 2020-12-03 16:10:55 +01:00
Christos Soulios 1c8a97784e
Lower minimum compatibility version of _doc_count field tests (#65790)
After merging _doc_count field type in v7.11.0 (#64594), this PR lowers the minimum compatibility
version from v8.0.0 to v7.11.0

Relates to #64503
2020-12-03 12:08:00 +02:00
Alan Woodward f0fc1b3dad
Use scriptless fields in CoreTestTranslator (#65599)
CoreTestTranslator re-writes some core search yaml tests into a tests
for runtime queries, and mimics source-only fields by building ad-hoc
painless scripts for each runtime type. We now have source-only fields
built in via scriptless mappings, so we can cut over to using these
instead.
2020-12-02 10:06:40 +00:00
Christoph Büscher 3c3a43249f
Support unmapped fields in search 'fields' option (#65386)
Currently, the 'fields' option only supports fetching mapped fields. Since
'fields' is meant to be the central place to retrieve document content, it
should allow for loading unmapped values. This change adds implementation and
tests for this feature.

Closes #63690
2020-12-01 21:40:27 +01:00
Christoph Büscher c327794ae8
Fix range query on date fields for number inputs (#63692)
Currently, if you write a date range query with numeric 'to' or 'from' bounds,
they can be interpreted as years if no format is provided. We use
"strict_date_optional_time||epoch_millis" in this case that can interpret inputs
like 1000 as the year 1000 for example. 
This PR change this to always interpret and parse numbers with the "epoch_millis"
parser if no other formatter was provided.

Closes #63680
2020-12-01 18:49:50 +01:00
Rene Groeschke 97749a3372
Port rest integ tests to use task avoidance api (#65011)
This ports the majority of the rest integ tests tasks to use the task avoidance api.

- There are some edge cases left that we need to investigate, but we can do that separately.
2020-11-26 10:30:06 +01:00
Christoph Büscher 6854974c1c
Reactivate deactivated integration test (#65388) 2020-11-24 10:48:14 +01:00
Mark Vieira 53b10dbb66 Mute "Tests catching other exceptions per item" 2020-11-23 10:16:01 -08:00
Christoph Büscher ae3880ed51
Better handling of item errors in _mtermvectors API (#65324)
Currently an error in a `_mtermvectors`, for example because querying through an
alias that has several indices assigned to it, fails the whole request. Instead
we should only fail the problematic item in the multi item request, like we e.g.
do in same situations in _mget.
2020-11-23 16:01:09 +01:00
Henning Andersen 21892817c4
Test version update for must_exist test (#65216)
Relates #65141
2020-11-19 07:45:01 +01:00