Commit Graph

2508 Commits

Author SHA1 Message Date
Jake Landis 26dfe02a0b
Update task names for rest compatiblity (#75267)
This commit updates two task names:
```
yamlRestCompatTest -> yamlRestTestV7CompatTest
transformV7RestTests -> yamlRestTestV7CompatTransform 
```

`7` is the N-1 version and calculated, such that when `8` is 
N-1 version the task names will be  `yamlRestTestV8CompatTest` and 
`yamlRestTestV8CompatTransform`

The motivation for `yamlRestCompatTest -> yamlRestTestV7CompatTest`  is that 
many projects have configured `yamlRestCompatTest` 
but that configuration is specific to the N-1 version. For example, 
if we blacklist tests when running compatibility with v7, we don't also
want to blacklist those tests when running compatibility with v8.

By introducing a version-specific identifier in the name, the task will not
even exist when bumping the version creating the need to (correctly) remove
the version-specific condition.

The motivation for `transformV7RestTests -> yamlRestTestV7CompatTransform` 
is to provide more consistent naming. 

The idea behind the naming is the main task people
are likely familiar with is :

`yamlRestTest` so we will use that as a base.
`yamlRestTestV7CompatTest` to run the version-specific compat tests
`yamlRestTestV7CompatTransform` to run the version-specific transformations for the compat tests

CI should be un-effected since since we introduced a lifecycle task 
name `checkRestCompat` which is what CI should be configured to use.
2021-09-03 11:26:11 -05:00
Nik Everett 574d6439d8
Update skip after backport (#77212)
Now that we've backported #76903 we can run backwards compatibility
tests against all versions that have it.
2021-09-03 09:11:46 -04:00
Benjamin Trent 02e17c3442
[ML] adding new defer_definition_decompression parameter to put trained model API (#77189)
This new parameter is a boolean parameter that allows
users to put in a compressed model without it having
to be inflated on the master node during the put
request

This is useful for system/module set up and then later
having the model validated and fully parsed when it
is being loaded on a node for usage
2021-09-03 09:07:54 -04:00
Seth Michael Larson eaf4ac26cc
Convert 'routing' values in REST API tests to strings
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
2021-09-03 07:33:26 -05:00
Philip Krauss 6d841043f5
Flip HTTP methods in _mvt rest-api-spec (#77228) 2021-09-03 09:52:28 +02:00
Henning Andersen f585a73238
Recovery stats test verify recovered from snapshot (#76974)
Fix test to verify against 7.15+
2021-09-02 15:31:45 +02:00
Jake Landis edcdd30b62
Compatible REST API - minor build clean up (#77145)
This commit allows the compatible REST API tests to execute on Windows.
They were previously excluded from Windows due to a command line limit
when defining a very large exclusion list. That exclusion list is much
smaller now and they will now execute properly on Windows.

Also, an empty exclusion list has been removed from the build config.
2021-09-01 15:51:14 -05:00
Nik Everett dde7d418a8
Add matching pattern to error in fields (#76903)
This adds the pattern into the error message returned when trying to
fetch fields. So this:
```
POST _search {
  "fields": [ { "field": "*", "format": "date_time" } ]
}
```

Will return an error message like
```
error fetching [foo] which matches [*]: Field [foo] of type [keyword] doesn't support formats
```
2021-09-01 12:15:54 -04:00
Rene Groeschke 35ec6f348c
Introduce simple public yaml-rest-test plugin (#76554)
This introduces a basic public yaml rest test plugin that is supposed to be used by external 
elasticsearch plugin authors. This is driven by #76215

- Rename yaml-rest-test to intern-yaml-rest-test
- Use public yaml plugin in example plugins

Co-authored-by: Mark Vieira <portugee@gmail.com>
2021-08-31 08:45:52 +02:00
Przemysław Witek 676d4de3de
[Transform] Implement the ability to preview the existing transform (#76697) 2021-08-24 14:41:49 +02:00
David Kyle 587f061cc0
[ML] Get Trained Model Deployment Stats API (#75268)
Adds the REST API /_ml/trained_models/{model_id}/deployment/_stats.
The API reports stats for live deployments -those that are started and running
- such as on which nodes the model(s) are deployed, avg inference
time and the inference count
2021-08-24 09:54:20 +01:00
David Turner 7094493b2b
Re-enable BWC tests after backport of #76771 (#76839)
Adjusts the wire compatibility constraints and reverts commit
18e657aed9.
2021-08-23 13:49:54 -04:00
David Turner 4a17847b85
Add timing stats to publication process (#76771)
This commit introduces into the node stats API various statistics to
track the time that the elected master spends in various phases of the
cluster state publication process.

Relates #76625
2021-08-23 17:38:32 +01:00
Benjamin Trent fbc982122c
Fixing bwc yaml test version after #76730 backport (#76746) 2021-08-20 09:14:11 -04:00
Benjamin Trent 55edc62b0d
Fix bug with significant terms background count (#76730)
When building empty responses for shards that don't have the term field in question,
significant terms ignored the background filter.

This commit fixes this bug by respecting the background filter count, even
when building empty results.

closes #76729
2021-08-20 07:20:55 -04:00
Przemyslaw Gomulka 127015e54c
[Rest Api Compatibility] Clean up blocklist (#76179)
v7compatibilityNotSupportedTests was introduced to make it easier to
track tests that have been identified as not needing compatible changes
and those that still need to be checked.
We have checked all tests now and the separate list is no longer needed.

relates #51816
relates #73912
2021-08-19 10:09:46 +02:00
Dimitris Athanasiou a9fd297dd0
[ML] Extend inference processor to work with allocated models (#76641)
This commit extends the `inference_processor` to support allocated
models. In particular, the internal infer action is now checking
if the model is allocated, and if so, it redirects the request
to an instance of the allocated model. If not, then it proceeds
as previously to load the model through the `ModelLoadingService`.

In addition, we now check that a model is not allocated when
the `StopTrainedModelDeploymentAction`.

Note that no new `InferenceConfigUpdate` objects are introduced
as there are no settings currently that can be set on inference time
for allocated models.
2021-08-18 15:17:46 +03:00
Nik Everett a84427dde2
Gracefully handle very large sizes on terms (#76578)
* Gracefully nandle very large sizes on terms

Folks often as for the `terms` agg to have *very* large `size`
parameters in an effort to get everything they can. But they never fill
that large size. They can't! There isn't really enough heap to return
two billion buckets.

After #74096 we try to pre-allocate an array of `size + 1` length,
regardless of how many results are returned. When folks ask for
`MAX_INT` buckets this helpfully fails off in lucene land with an error
message that fairly esoterit to someone reading it outside of
Elasticsearch.

This change handles that `MAX_INT` case by building a
`TopBucketsBuilder` designed to handle large max sizes that are rarely
filled. When you ask for 1024 or more buckets you get that one instead
and we don't preallocate the entire array for the reduction. You get the
old preallocated one when you have few buckets.

Closes #76492

* fixup skip

* Moar comment
2021-08-17 14:15:59 -04:00
Yang Wang 55cd7bb9d6
Query API key - Rest spec and yaml tests (#76238)
This PR adds Rest API spec and related yaml tests for the new Query API key API.

Relates: #71023
2021-08-17 10:56:40 +10:00
Francisco Fernández Castaño 2c132fe5f7
Keep track of data recovered from snapshots in RecoveryState (#76499)
Adds new field to recovery API to keep track of amount of data
recovered from snapshots.

The normal recovered_bytes field remains and is also increased for
recovery from snapshot but can go backwards in the unlikely case
that recovery from snapshot fails to download a file.

Relates #73496
2021-08-16 18:18:37 +02:00
Seth Michael Larson 5424748840
Fix 'accept' and 'content_type' fields for search_mvt API 2021-08-16 10:35:04 -05:00
Ioannis Kakavas 59d75f213f
Add kibana-system service account (#76449)
This change introduces a Service Account for Kibana to use when
authenticating to Elasticsearch. The Service Account with
kibana service name under the elastic namespace,
uses the same RoleDescriptor as the existing kibana_system
built-in user and is its functional equivalent when it comes to
AuthZ.
2021-08-13 17:00:13 +03:00
Armin Braun 847be51b64
Reenable BwC Rest Tests (#76226)
These tests should work now that we aligned 7.x and 8.x snapshot apis.
2021-08-12 09:06:29 +02:00
Joe Gallo e7295443eb
Re-enable deprecated _cluster/voting_config_exclusions/{node_name} route via REST compatibility (#75951) 2021-08-10 16:42:51 -04:00
James Rodewig 96c4ee3e5c
[DOCS] Document `_mvt` API (#75384)
* [DOCS] Document `_mvt` API

Documents the `_mvt` API endpoint added with #73872.

Relates to #75242.

* Reword

* Rename API

* Fix doc.url in JSON spec

* Reword

* Reword

* Add content type to JSON spec

* Edits

* Fix typo

* Reword

* Update docs after meeting

* Fix typos

* Fix `size` default

* Updates for #75522

* Fixes

* Clean up JSON spec

* Fix extent tag

* [DOCS] Add `<field>` constraints

* Minor clarification

* Update for #75697

* Reword

* Update for #75621

* Reword default sort

* Update for #75367

* Remove unneeded whitespace

* Add experimental admon and if flags

* [DOCS] Remove ifdefs

Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
2021-08-05 15:04:07 -04:00
Przemyslaw Gomulka 782e2c67f8
[Rest Api Compatibility] CommonTermsQuery and cutoff_frequency param (#75896)
Previously removed in #42654. The query and the parameter won't work under rest api compatibility and an exception with a message is returned advising that just use of match/multi_match is enough

relates #51816
2021-08-05 15:14:28 +02:00
Przemyslaw Gomulka d85acf48ae
[Rest Api Compatibility] Nested path and filter sort options (#76022)
previously removed in #42809. nested_path and nested_filter sort options will now throw exceptions suggesting to use nested option
relates #51816
2021-08-05 11:38:33 +02:00
Przemyslaw Gomulka f3392829d6
[Rest Api Compatibility] Typed query (#75453)
Type query support was removed in #47207. This query will throw an exception in v7 rest api compatibility indicating that the support was removed + deprecation warnings.
In v8 it will not be available and error about type query being not found will be returned.

relates main meta issue #51816
relates types removal meta #54160
2021-08-03 15:20:34 +02:00
David Turner 95edc6deb2
Clarify allocation explain if random shard chosen (#75670)
Today we often encounter users that are confused by the behaviour of
calling `GET _cluster/allocation/explain` without a body: it _seems_ to
work, but it explains a random shard, and if this isn't the shard
they're thinking of then it's unclear how to proceed.

With this commit we add a note to the response when a shard was randomly
chosen indicating that it is possible, and possibly useful, to explain a
different shard. We also adjust the exception message in the case when
all shards are assigned to indicate why it's an invalid request and what
to do to make it valid.
2021-08-02 15:14:09 +01:00
Keith Massey 0790107f9a
Re-enabling BWC tests, and updating supported versions for shard count feature after backport (#75789) 2021-07-28 09:43:49 -05:00
Przemyslaw Gomulka c96139d006
[Rest Api Compatibility] Deprecate the use of synced flush (#75372)
synced flush is going to be replaced by flush. This commit allows to synced_flush api only in v7 compatibility mode.
Worth noting - sync_id is gone and won't be available in v7 responses from indices.stats

relates removal pr #50882
relates #51816
2021-07-28 14:17:49 +02:00
Keith Massey ddc3b37580
Adding shard count to node stats api (#75433)
* Adding shard count to _nodes/stats api

Added a shards section to each node returned by the _nodes/stats api. Currently this new section only contains a total count of all shards on the node.
2021-07-27 10:39:53 -05:00
Lisa Cawley 02d851e50e
[DOCS] Drafts trained model deployment APIs (#75497) 2021-07-26 09:49:37 -07:00
Przemyslaw Gomulka 838e8d8aeb
[Rest Api Compatibility] Voting config exclusion exception message (#75406)
the exception message has changed in #55291. This is not covered by rest
api compatibility, so no need to return the old message for v7 requests.
This commit adds a transformation to allow for the 7.x test to pass with
a new exception message

relates #51816
2021-07-26 16:42:38 +02:00
Dan Hermann a882f61913
Fix failing HTTP client stats test (#75527) 2021-07-21 08:20:02 -05:00
James Rodewig ea635d67a6
[DOCS] SQL: Add formal API docs (#75506)
Adds formal API docs for the following APIs:

* Clear SQL cursor
* SQL search
* SQL translate

Other changes:

* Removes and redirects the "Supported REST parameters section." This is now covered in the SQL search API docs.
* Updates a few related xrefs.

Closes #75085
2021-07-21 08:04:21 -04:00
Mark Tozzi 7af39dbc35
Remove deprecated date histo interval (#75000)
Date histogram interval parameter was deprecated in 7.2, in favor of the more specific fixed_interval and calendar_interval parameters.  The old logic used some poorly understood guessing to decide if it should operate in fixed or calendar mode.  The new logic requires a specific choice by the user, which is more explicit.  In 7.x REST compatibility mode, we will parse the interval as calendar if possible, and otherwise interpret it as fixed.
2021-07-20 13:08:45 -04:00
Przemyslaw Gomulka 5485d3ebdd
[Rest Api Compatibility] Do not return _doc for empty mappings in template (#75448)
Previously the compatibility layer was always returning an _doc in mappings for get
template.
This commit does not return _doc in empty mappings.
Returning just {} empty object (v7 and v8 behaviour)

also moving term lookups tests which are already fixed (relates #74544)

relates #70966
relates main meta issue #51816
relates types removal meta #54160
2021-07-19 13:21:00 +02:00
Przemyslaw Gomulka 4af7150a40
[Rest Api Compatibility] Allow to use size -1 (#75342)
previously disallowed in #70209. However since now the recommendation is
to not set the field at all - and rely on a default - this would be a
shape change for users prefering to set -1 so far.
With request compatible with v7 they will still be allowed to do this
and will get a warning.

relates #69548
relates #51816
2021-07-16 12:46:05 +02:00
Steve Gordon 1e7e92dda7
Add missing X-Pack repositories REST specs (#65548)
* Add missing repositories API REST specs

A new experimental X-Pack API was added for repositories, but is missing a REST spec. The API was added in #60371

* Move this under the nodes namespace

* Rename max_version_to_clear parameter

* Update docs URL to use current
2021-07-16 10:55:13 +01:00
Lisa Cawley 345c00ceff
[DOCS] Update doc URLs for trained model deployment APIs (#75388) 2021-07-15 10:51:15 -07:00
James Rodewig b79e4a594d
[DOCS] Fix broken doc url values in JSON API spec (#75385)
Fixes a broken link in the `documentation.url` for the field usage stats API and
enroll Kibana API.

These broken links caused the build for the JS client docs to fail.
2021-07-15 13:46:00 -04:00
Steve Gordon 2083304253
Remove SET aliases from disk usage and field usage YML tests (#75381)
* Remove aliases from disk usage test

* Remove aliases from field usage test
2021-07-15 16:17:21 +01:00
Przemyslaw Gomulka 2320a38833
[Rest Api Compatibility] Allow transforming warnings per test (#75187)
Warning related transformations missed the possibility to apply per single test only.
Also a warning changed in #67158 for indices.close so this PR also applies the transformation for 7.x test

relates #51816
2021-07-15 10:50:42 +02:00
Julie Tibshirani a2b8b53014
Loosen checks in REST test for PIT slicing (#75261)
Since the index may have more than one shard, we can't always predict how many
documents will fall in each slice. This PR simply removes the checks, since we
already test the slicing logic extensively in an integration test. The REST test
is now just a sanity check for the API.

Fixes #75212.
2021-07-14 11:10:48 -07:00
Yannick Welsch 88d6079507 Adapt BWC after backport (#74227)
Adapts BWC conditions after backporting #74227 to 7.x branch (targeting 7.15).
2021-07-14 14:59:45 +02:00
Yannick Welsch db814f403b
Track Lucene field usage (#74227)
Adds a field usage API that reports shard-level statistics about which Lucene fields have been accessed, and which
parts of the Lucene data structures have been accessed.

Field usage statistics are automatically captured when queries are runnning on a cluster. A shard-level search request
that accesses a given field, even if multiple times during that request, is counted as a single use.
2021-07-14 13:21:11 +02:00
Przemyslaw Gomulka 71e05838a6
[Rest Api Compatibility] Enable tests after types and cat api fixed (#75179)
Some tests are fixed after typed api is available with compatible api.
Also cat api returning text fixed some tests
relates #51816
2021-07-14 08:37:38 +02:00
James Rodewig a9ed7dc1eb
[DOCS] Document async SQL APIs (#75078)
Adds formal API docs and JSON specs for the following APIs:

* Get async SQL search
* Get async SQL search status
* Delete async SQL search

Closes #74845
2021-07-12 10:12:02 -04:00
Przemyslaw Gomulka ff8aede63a
[Rest Api Compatibility] MoreLikeThisQuery with types (#75123)
previously removed in #42198
also adding a type for MultiTermVectorsResponse in failures

relates #54160
relates #51816
2021-07-12 12:08:45 +02:00