Commit Graph

2479 Commits

Author SHA1 Message Date
David Turner 83c64ded2d
Adjust BWC for tracking snapshot index details (#71913)
Re-enables BWC tests and completes the backport of #71754
2021-04-20 13:16:14 +01:00
Mark Tozzi 7b29cb31b4
Allow composite to be a sub-aggregation of filter (#71639) 2021-04-19 13:28:03 -04:00
Tanguy Leroux ceaa16eddc
Add node level cache stats for searchable snapshots (#71701)
This commit adds node-level statistics about the searchable 
snapshots shared cache that can be retrieved using the REST 
endpoint `GET /_searchable_snapshots/cache/stats`.

And the returned informations are:
{
  "nodes" : {
    "eerrtBMtQEisohZzxBLUSw" : {
      "shared_cache" : {
        "reads" : 6051,
        "bytes_read" : "5.1mb",
        "bytes_read_in_bytes" : 5448829,
        "writes" : 37,
        "bytes_written" : "1.1mb",
        "bytes_written_in_bytes" : 1208320,
        "evictions" : 5,
        "num_regions" : 32,
        "size" : "1mb",
        "size_in_bytes" : 1048576,
        "region_size" : "32kb",
        "region_size_in_bytes" : 32768
      }
    }
  }
}
2021-04-19 17:02:27 +02:00
David Turner c8fb9aad40
Track index details in SnapshotInfo (#71754)
This commit adds some per-index statistics to the `SnapshotInfo` blob:

- number of shards
- total size in bytes
- maximum number of segments per shard

It also exposes these statistics in the get snapshot API.
2021-04-19 14:57:32 +01:00
Dimitris Athanasiou b5a8ffd7ca
[ML] Mark data frame analytics API specs as GA (#71818) 2021-04-19 16:29:08 +03:00
Przemyslaw Gomulka 3ef5e4c6e7
[Rest Compatible Api] include_type_name parameter (#70966)
This commit allows to use the include_type_name parameter with the compatible rest api.
The support for include_type_name was previously removed in #48632

relates #51816
types removal meta issue #54160
2021-04-19 15:21:24 +02:00
Rick Boyd 53b0141a11
Fix cluster.get_settings documentation URL 2021-04-16 10:56:12 -05:00
Igor Motov 02eef40a45
Tests: add support for close_to assertion (#71590)
Adds support for close_to assertion to yaml tests. The assertion can be called
the following way:
```
  - close_to:   { get.fields._routing: { value: 5.1, error: 0.00001 } }
```
Closes #71303
2021-04-15 17:11:37 -10:00
Christoph Büscher 5e481b4b36 [Test] Adapt skip version after backport 2021-04-15 15:51:50 +02:00
Christoph Büscher 708947b4bd Skip flattend subfields rest test 2021-04-15 14:57:39 +02:00
Tanguy Leroux d1c3b71384
Reenable BWC tests after backport of #71725 (#71732)
This commit reenables BWC tests (disabled in #71728) and adjusts some versions.
2021-04-15 13:38:40 +02:00
Christoph Büscher 948d02e4d6
Support fetching flattened subfields (#70916)
Currently the `fields` API fetches the root flattened field and returns it in a
structured way in the response. In addition this change makes it possible to
directly query subfields. However, requesting flattened subfields via wildcard
patterns is not possible.

Closes #70605
2021-04-15 12:28:58 +02:00
Tanguy Leroux fe4c3e745c
Enhanced segment files sizes information in Nodes Stats/Indices Stats APIs (#71643)
Since #16661 it is possible to know the total sizes for some Lucene segment files 
by using the Node Stats or Indices Stats API with the include_segment_file_sizes 
parameter, and the list of file extensions has been extended in #71416.

This commit adds a bit more information about file sizes like the number of files 
(count), the min, max and average file sizes in bytes that share the same extension.

Here is a sample:
"cfs" : {
  "description" : "Compound Files",
  "size_in_bytes" : 2260,
  "min_size_in_bytes" : 2260,
  "max_size_in_bytes" : 2260,
  "average_size_in_bytes" : 2260,
  "count" : 1
}

This commit also simplifies how compound file sizes were computed: before 
compound segment files were extracted and sizes aggregated with regular 
non-compound files sizes (which can be confusing and out of the scope of 
the original issue #6728), now CFS/CFE files appears as distinct files.

These new information are provided to give a better view of the segment 
files and are useful in many cases, specially with frozen searchable snapshots 
whose segment stats can now be introspected thanks to the 
include_unloaded_segments parameter.
2021-04-15 10:15:26 +02:00
Julie Tibshirani 2318be0be2 Adjust skip versions for combined_fields tests.
We can run the REST tests against 7.13 now that it's been backported.
2021-04-14 16:06:20 -07:00
Nik Everett 2d6f8d1e0c
Add integration tests for filters (#69439)
Revamps the integration tests for the `filter` agg to be more clear and
builds integration tests for the `fitlers` agg. Both of these
integration tests are fairly basic but they do assert that the aggs
work.
2021-04-14 16:54:23 -04:00
Julie Tibshirani 318bf14126
Introduce `combined_fields` query (#71213)
This PR introduces a new query called `combined_fields` for searching multiple
text fields. It takes a term-centric view, first analyzing the query string
into individual terms, then searching for each term any of the fields as though
they were one combined field. It is based on Lucene's `CombinedFieldQuery`,
which takes a principled approach to scoring based on the BM25F formula.

This query provides an alternative to the `cross_fields` `multi_match` mode. It
has simpler behavior and a more robust approach to scoring.

Addresses #41106.
2021-04-14 13:33:19 -07:00
Tim Brooks d5b96a35d0
Add fleet polling API for global checkpoint (#71093)
Fleet server needs an API to access up to date global checkpoints for
indices. Additionally, it requires a mode of operation when fleet can
provide its current knowledge about the global checkpoints and poll for
advancements. This commit introduces this API in the fleet plugin.
2021-04-14 11:19:33 -06:00
Nik Everett b4bac7b769
Test: assert adjacency matrix cache (#69440)
Adds an assertion that the adjacency matrix aggregation can hit the
request cache even when the request contains a lookup style `terms`
filter.
2021-04-13 16:57:26 -04:00
Nik Everett 57e6c78a52
Fix profiled global agg (#71575)
This fixes the `global` aggregator when `profile` is enabled. It does so
by removing all of the special case handling for `global` aggs in
`AggregationPhase` and having the global aggregator itself perform the
scoped collection using the same trick that we use in filter-by-filter
mode of the `filters` aggregation.

Closes #71098
2021-04-13 08:36:51 -04:00
Mark Tozzi 3c198e2606
Tests for running composite under nested aggregation (#68243) 2021-04-12 10:52:01 -04:00
Nhat Nguyen 8bef36dac7
Adjust BWC version for dynamic templates in bulk requests (#71537)
Relates #69948
2021-04-10 16:01:31 -04:00
Nhat Nguyen 5c9969250d
Allow specify dynamic templates in bulk request (#69948)
This change allows users to specify dynamic templates in a bulk request.

```
PUT myindex
{
  "mappings": {
    "dynamic_templates": [{
      "time_histograms": {
        "mapping": {
          "type": "histogram",
          "meta": {
            "unit": "s"
          }
        }
      }
    }]
  }
}
```

```
POST myindex/_bulk
{ "index": { "dynamic_templates": { "response_times": "time_histograms" } } }
{ "@timestamp": "2020-08-12", "response_times": { "values": [1, 10], "counts": [5, 1] }}
```

Closes #61939
2021-04-08 12:44:36 -04:00
Nik Everett c50fd8f3f1 Drop flaky assertion
Drops an assertion that we can't be sure will always pass. If we're
unlucky all documents with `_doc_count` can end up on a single shard and
our assertion won't pass. In yaml we don't have the ability to assert
that *either* shard has `_doc_count`. It's ok! We have an assertion for
this in another place too.

Close #71088
2021-04-08 10:27:47 -04:00
Jason Tedor 8d716e83fd
Re-enable BWC tests after migrating ML roles
This commit reenables the BWC tests after the ML roles were migrated to
server. During the course of that work, the BWC tests were disabled
pending that work being backported to 7.x. Now that that work is not
going to be backported to 7.x, instead we apply some permanent
transformations to the 7.x assertions run against ES in the REST
compatibility tests.
2021-04-07 19:40:53 -04:00
Jason Tedor 241b653ae4
Move machine learning roles to server (#71412)
This commit moves the machine learning roles to server. We no longer
need to maintain these roles outside of server since we only produce a
single distribution, the default distribution, which includes all
roles. Therefore we can simplify the plugin architecture by removing the
plugin extension point for roles. This is one step in that, by moving
the machine learning roles to server.
2021-04-07 19:25:36 -04:00
Joe Gallo 4ff17c1b7a
[REST Compatible API] 'template' parameter and field on PUT index template (#71238) 2021-04-02 15:16:14 -04:00
Benjamin Trent b4b4994f5d
[Text Structure][ML] adjust spec to reflect accurate stability (#71236)
the text_structure/find_structure API is stable and GA as of 7.12.

This commit adjusts the spec to reflect that.
2021-04-02 11:56:27 -04:00
Jake Landis 26ed10b4df
fix REST compat test with test transform 2021-03-31 16:12:50 -05:00
Jake Landis 28fcaa3824
Revert "Mute ClientYamlTestSuiteIT node_info role test"
This reverts commit 78fe9cca46.
2021-03-31 16:08:30 -05:00
Jake Landis db948d8374
Revert "Mute correct test"
This reverts commit cac94707f6.
2021-03-31 16:03:18 -05:00
Mark Vieira cac94707f6 Mute correct test 2021-03-31 13:40:24 -07:00
Mark Vieira 78fe9cca46 Mute ClientYamlTestSuiteIT node_info role test 2021-03-31 13:28:20 -07:00
Jason Tedor e119ac60d4
Move data tier roles to server (#71084)
This commit moves the data tier roles to server. It is no longer
necessary to separate these roles from server as we no longer build
distributions that would not contain these roles. Moving these roles
will simplify many things. This is deliberately the smallest possible
commit that moves these roles. Other aspects related to the data tiers
can move in separate, also small, commits.
2021-03-31 15:13:02 -04:00
Henning Andersen 632d23db4a
StoreStats update serialization version (#71107)
Reenable bwc and serialize new field `totalDataSetSizeInBytes` to 7.13+ now that the backport of #70625 is done.
2021-03-31 17:04:00 +02:00
James Rodewig 693807a6d3
[DOCS] Fix double spaces (#71082) 2021-03-31 09:57:47 -04:00
Henning Andersen 0f28e97857
Total data set size in stats (#70625)
With shared cache searchable snapshots we have shards that have a size
in S3 that differs from the locally occupied disk space. This commit
introduces `store.total_data_set_size` to node and indices stats, allowing to
differ between the two.

Relates #69820
2021-03-30 15:23:29 +02:00
Nhat Nguyen 977ecd670a Adjust BWC for point in time yaml test 2021-03-28 17:26:49 -04:00
Mark Vieira 6339691fe3
Consolidate REST API specifications and publish under Apache 2.0 license (#70036) 2021-03-26 16:20:14 -07:00
Mayya Sharipova ccfdbb4d15
Fix binary docvalue_fields with padding (#70826)
Previously docvalue_fields for binary values with paddings did not
output padding. We consider it to be a bug because: 1) es would
not be able parse these values 2) output from source filtering
and fields API is different and does output padding.

This patches fixes this by outputing padding for binary
docvalue_fields where it is present.
2021-03-26 16:18:20 -04:00
Nhat Nguyen 5bb440cdca
Move point in time to server (#70704)
This change moves the implementation of point in time to the server package.
2021-03-24 14:29:20 -04:00
Przemko Robakowski f5b7aad8b7
Add stats endpoint to GeoIpDownloader (#70282)
This change adds _geoip/stats endpoint that can be used to collect basic data about geoip downloader (successful, failed and skipped downloads, current db count and total time spent downloading).
It also fixes missing/wrong origins for clients that will break if used with security.

Relates to #68920
2021-03-23 14:34:32 +01:00
Przemyslaw Gomulka e942873bd5
[REST Compatible API] Typed endpoints for Index and Get APIs (#69131)
The types removal effort has removed the type from Index API in #47671 and from Get API in #46587
This commit allows to use 'typed' endpoints for the both Index and Get APIs

relates compatible types-removal meta issue #54160
2021-03-23 10:59:21 +01:00
Nik Everett 96b49a96ec
Update skip after backport of #70493 (#70690) 2021-03-22 16:50:10 -04:00
Nik Everett 8e6d478c0e
Stop terms agg from losing buckets (#70493)
When the `terms` agg is at the top level it can run as a `filters` agg
instead because that is typically faster. This was added in #68871 and
we mistakely made it so that a bucket without any hits could take up a
slot on the way back to the coordinating node. You could trigger this by
having a fairly precise `size` on the terms agg and a top level filter.

This fixes the issue by properly mimicing the regular terms aggregator
in the "as filters" version: only send back buckets without any matching
documents if the min_doc_count is 0.

Closes #70449
2021-03-22 12:20:23 -04:00
Philip Krauss 74d3f583b2
fix doc url and api description (#70631) 2021-03-22 14:59:13 +01:00
Nhat Nguyen fa51d3f33c Adjust wire compact version
Relates #70357
2021-03-21 11:27:40 -04:00
William Brafford 624ee45a8e
Add API for resetting state of a `SystemIndexPlugin` (#69469)
When we disable access to system indices, plugins will still need
a way to erase their state. The obvious and most pressing use
case for this is in tests, which need to be able to clean up the
state of a cluster in between groups of tests.

* Use a HandledTransportAction for reset action

My initial cut used a TransportMasterNodeAction, which requires code
that carefully manipulates cluster state. At least for the first cut and
testing, it seems like it will be much easier to use a client within a
HandledTransportAction, which effectively makes the
TransportResetFeatureStateAction a class that dispatches other transport
actions to do the real work.

* Clean up code by using a GroupedActionListener

* ML feature state cleaner

* Implement Transform feature state reset

* Change _features/reset path to _features/_reset

Out of an abundance of caution, I think the "reset" part of this path
should have a leading underscore, so that if there's ever a reason to
implement "GET _features/<feature_id>" we won't have to worry about
distinguishing "reset" from a feature name.

Co-authored-by: Gordon Brown <gordon.brown@elastic.co>
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
2021-03-17 16:14:34 -04:00
Dan Hermann 8a9055a0ca
Adjust BWC after backport of #64561 (HTTP client stats) (#70508) 2021-03-17 12:32:22 -05:00
Dan Hermann 14cee55c88
Add info on each HTTP client to HTTP stats (#64561) 2021-03-17 07:51:54 -05:00
Nhat Nguyen 8b5aa84647
Allow format sort values of date fields (#70357)
If a search after request targets multiple indices and some of its sort 
field has type `date` in one index but `date_nanos` in other indices,
then Elasticsearch won't interpret the search_after parameter correctly
in every target index. The sort value of a date field by default is a
long of milliseconds since the epoch while a date_nanos field is a long
of nanoseconds.

This commit introduces the `format` parameter in the sort field so a 
sort value of a date or date_nanos will be formatted using a date format
in a search response.

The below example illustrates how to use this new parameter.

```js
{
    "query": {
        "match_all": {}
    },
    "sort": [
        {
            "timestamp": { 
                "order": "asc",
                "format": "strict_date_optional_time_nanos"
           }
        }
    ]
}
```

```js
{
    "query": {
        "match_all": {}
    },
    "sort": [
        {
            "timestamp": { 
                "order": "asc",
                "format": "strict_date_optional_time_nanos"
            }
        }
    ],
    "search_after": [
        "2015-01-01T12:10:30.123456789Z" // in `strict_date_optional_time_nanos` format
    ]
}
```

Closes #69192
2021-03-16 21:27:51 -04:00
Martijn van Groningen 715eb90fea
Support specifying multiple templates names in delete component template api (#70314)
Add support to delete component templates api to specify multiple template
names separated by a comma.

Change the cleanup template logic for rest tests to remove all component templates via a single delete component template request. This to optimize the cleanup logic. After each rest test we delete all templates. So deleting templates this via a single api call (and thus single cluster state update) saves a lot of time considering the number of rest tests.

Older versions don't support component / composable index templates
and/or data streams. Yet the test base class tries to remove objects
after each test, which adds a significant number of lines to the
log files (which slows the tests down). The ESRestTestCase will
now check whether all nodes have a specific version and then decide
whether data streams and component / composable index templates will
be deleted.

Also ensured that the logstash-index-template and security-index-template
aren't deleted between tests, these templates are builtin templates that
ES will install if missing. So if tests remove these templates between tests
then ES will add these template back almost immediately. These causes
many log lines and a lot of cluster state updates, which slow tests down.

Relates to #69973

Co-authored-by: Lee Hinman <dakrone@users.noreply.github.com>
2021-03-15 13:08:49 +01:00
Nhat Nguyen 19be066e2e Adjust BWC version for _shard_doc sort
Relates #66093
2021-03-13 13:59:43 -05:00
Julie Tibshirani e03ccaa621
Adjust version checks after moving flattened to core. (#70333)
Now that the PR to move flattened to core was backported, we can adjust the
skip version in REST tests. We can also remove FlattenedFeatureSetUsage, since
it is only necessary to communicate with pre-7.13 nodes.
2021-03-11 13:49:30 -08:00
Christoph Büscher c3a8674f7c
Completely disallow setting negative size in search (#70209)
We used to treat setting size to -1 in search request bodies or as a rest
parameter as a no-op, using the default search size of 10 in this case. This
lenient behaviour was deprecated in #69548 and is removed with this PR in 8.0.

Relates to #69548
2021-03-11 11:42:25 +01:00
Fernando Briano 0001cbee0a
Wraps YAML tests values with colon in quotes (#70181) 2021-03-10 17:18:40 +00:00
Gordon Brown 7c347406ed
Remove old test from REST compatibility blocklist (#70187)
This test was left on the blacklist, but no longer exists under this
name, as the API name was changed. This commit removes it from the list.
2021-03-09 17:12:15 -07:00
Christoph Büscher d8ef26ad26
Add earlier validation for some SearchSourceBuilder settings (#69548)
Currently we check several search parameters for illegal values in their
SearchSourceBuilder setters, e.g. negative values throw IAE for: `size`,
`terminateAfter` and `trackTotalHits`.

The validation in the builder setters are used when parsing the above as rest
request parameters, however we currently don't check values when parsing them
from the search request body. This leads to builders with invalid parameters
that sometimes get caucht later (e.g. a negative size is triggering an
IllegalArgumentException in TotalHitCountCollector), but we should validate and
throw errors early.

This PR changes the parsing in SearchSourceBuilder to use the setters, adds
tests and also adds a deprecation for allowing a size parameter of -1, currently
meaning an "unset" value.

Closes #54958
2021-03-09 17:15:27 +01:00
Nik Everett f3680b49f2
Update skip after backport of #69806 (#70153)
Now that we've backported #69806 we can test it in the bwc tests.
2021-03-09 10:51:27 -05:00
Christoph Büscher f7c382460e
Fix fetch_fields yaml test result order (#70149)
The test failing in #69985 does so because under rare circumstance the result
order for match_all can be different. If we want to make assertions on specific
entries in the result, we should sort by a field that imposes a fixed result
ordering.

Closes #69985
2021-03-09 16:05:41 +01:00
Przemyslaw Gomulka 9ad9c781de
Add compatible logging when parsing a compatible field (#69539)
A #68808 introduced a possibility to declare fields which will be only available to parsing when a compatible API was used.

This commit replaces deprecated log with compatible logging when a 'compatible only' field was used. Also includes a refactoring of LoggingDeprecationHandler method names

relates #51816
2021-03-09 12:29:40 +01:00
Julie Tibshirani 796284a190
Move flattened field to core. (#68780)
This field mapper only lived in its own module so it could be licensed as x-pack
basic. Now it can be moved to core, which matches its status as a core type.
2021-03-08 16:56:16 -08:00
Dan Hermann beebc1dc66
Adjust BWC versions after backport of #69756 (#70097) 2021-03-08 11:49:45 -06:00
Przemyslaw Gomulka 3114436329
Ignore a testcase until a compatible api is implemented (#70095)
'search/110_field_collapsing/field collapsing and inner_hits', is
enabled in 7.x (#69753 )and the compatible api is not implemented yet
2021-03-08 18:39:55 +01:00
Luca Cavanna 06f244ba8c
Adjust compatibility and tests of runtime fields telemetry (#70067)
Runtime fields telemetry has been entirely moved to be part of cluster stats API in 7.x and master. This commit removes the backwards compatibility layer that was needed before such change was backported.
2021-03-08 14:55:49 +01:00
Dan Hermann bec1cf8599
Summary option for listing ingest pipelines without their definitions (#69756) 2021-03-08 07:48:40 -06:00
Luca Cavanna ffe61fb097
Move runtime fields stats to server (#69487)
Runtime fields usage is currently reported as part of the xpack feature usage API. Now that runtime fields are part of server, their corresponding stats can be moved to be part of the ordinary mapping stats exposed by the cluster stats API.
2021-03-08 12:38:20 +01:00
Martijn van Groningen c2fee2bebe
Adjusted skip version for date math aliases yaml test. (#69950) 2021-03-04 10:11:29 +01:00
Nik Everett 10e2f90560
Speed up aggs with sub-aggregations (#69806)
This allows many of the optimizations added in #63643 and #68871 to run
on aggregations with sub-aggregations. This should:
* Speed up `terms` aggregations on fields with less than 1000 values that
  also have sub-aggregations. Locally I see 2 second searches run in 1.2
  seconds.
* Applies that same speedup to `range` and `date_histogram` aggregations but
  it feels less impressive because the point range queries are a little
  slower to get up and go.
* Massively speed up `filters` aggregations with sub-aggregations that
  don't have a `parent` aggregation or collect "other" buckets. Also
  save a ton of memory while collecting them.
2021-03-03 18:04:47 -05:00
Tommmster 9c45dbcb8e
Add date math support for aliases (#67226)
This commit adds date match support to aliases to the put alias, update aliases and create index APIs.

For example:

```
PUT %3Clogs-myapp-%7Bnow%2Fd%2B1d%7D-0%3E 
POST logs-myapp-2021.03.03-0/_alias/%3Clogs-myapp-%7Bnow%2B1d%7D%3E
```

Or via a single api call:

```
PUT %3Clogs-myapp-%7Bnow%2Fd%2B1d%7D-0%3E
{
   "aliases": {
       '<logs-myapp-{now+1d}> ': {}
   }
}
```

Closes #20367

Co-authored-by: Martijn van Groningen <martijn.v.groningen@gmail.com>
2021-03-03 17:23:29 +01:00
Gordon Brown ce8a0c0cea
Change Get Snapshottable Features endpoint to `_features` (#69755)
The endpoint `_snapshottable_features` is long and implies incorrect
things about this API - it is used not just for snapshots, but also for
the upcoming reset API. Following discussions on the team, this commit
changes the endpoint to `_features` and removes the connection between
this API and snapshots, as snapshots are not the only use for the output
of this API.
2021-03-02 11:30:02 -07:00
Jake Landis 3dc9dc01c7
avoid costly regex expression (#69768)
This commit changes part of a regular expression for some tests to
be more performant. While it is difficult articulate why this is change is much
faster, testing has shown for some inputs this match to be less then 1s,
where prior could take over 30s.

related: #69757
2021-03-02 07:35:27 -06:00
Yannick Welsch 4a94534bbb Adapt BWC conditions after backport of #69721 2021-03-02 08:42:52 +01:00
Jake Landis 13915bc8c1
Add support for regex in REST test warnings and allowed_warnings (#69501)
This commit adds support for two new REST test features.
warnings_regex and allowed_warnings_regex.

This is a near mirror of the warnings and allowed_warnings
warnings feature where the test can be instructed to allow
or require HTTP warnings. The difference with these new features
is that is allows the match to be based on a regular expression.
2021-03-01 15:40:39 -06:00
Nik Everett 8b8a20ba18
Update skip after backport (#69710)
Now that we've backported #69377 to 7.x we can run backwards
compatibility tests against it.
2021-03-01 11:43:44 -05:00
Yannick Welsch 529c6227fe
Support include_unloaded_segments in node stats (#69682)
Adds support for the include_unloaded_segments flag in node stats, which helps with understanding resource usage of
shared_cache-style searchable snapshots on a per-node basis.
2021-03-01 17:18:47 +01:00
Nik Everett 19eef6a258 Skip test before backport
Skip running the a particular filters aggregator test while we backport
the change that'll make it correct.
2021-02-25 14:50:08 -05:00
Nik Everett 4ffdad36d4
Speed up terms agg when alone (#69377)
This speeds up the `terms` agg in a very specific case:
1. It has no child aggregations
2. It has no parent aggregations
3. There are no deleted documents
4. You are not using document level security
5. There is no top level query
6. The field has global ordinals
7. There are less than one thousand distinct terms

That is a lot of restirctions! But the speed up pretty substantial because
in those cases we can serve the entire aggregation using metadata that
lucene precomputes while it builds the index. In a real rally track we
have we get a 92% speed improvement, but the index isn't *that* big:

```
| 90th percentile service time | keyword-terms-low-cardinality |     446.031 |     36.7677 | -409.263 |     ms |
```

In a rally track with a larger index I ran some tests by hand and the
aggregation went from 2200ms to 8ms.

Even though there are 7 restrictions on this, I expect it to come into
play enough to matter. Restriction 6 just means you are aggregating on
a `keyword` field. Or an `ip`. And its fairly common for `keyword`s to
have less than a thousand distinct values. Certainly not everywhere, but
some places.

I expect "cold tier" indices are very very likely not to have deleted
documents at all. And the optimization works segment by segment - so
it'll save some time on each segment without deleted documents. But more
time if the entire index doesn't have any.

The optimization builds on #68871 which translates `terms` aggregations
against low cardinality fields with global ordinals into a `filters`
aggregation. This teaches the `filters` aggregation to recognize when
it can get its results from the index metadata. Rather, it creates the
infrastructure to make that fairly simple and applies it in the case of
the queries generated by the terms aggregation.
2021-02-25 14:15:57 -05:00
Henning Andersen 8c69d0f247
Mute 110_field_collapsing/field collapsing and from (#69587) 2021-02-25 10:23:20 +01:00
Mark Vieira 30eca91183 Mute rest-api-spec compatibility tests on Windows 2021-02-24 12:34:51 -08:00
Nik Everett 577205a6f9 Test: Fix filter assertion
Forces a test to use only a single shard so the assertion about the
aggregation profiler results are correct. Without this the test fails
randomly but very rarely. We have to use multiple shards (seeded random
10% choice) and we have to land all of the documents on one shard
(unseeded random 3.2%) and that shard has to be the second shard in the
list (unseeded random 50%). That works out to about 1.6% chance with an
appropriate seed - .16% without it.
2021-02-24 13:37:49 -05:00
Joe Gallo f3aac00f5d
Apply yaml-rest-compat-test to rest-api-spec (#69462) 2021-02-23 13:15:53 -05:00
Igor Motov 6e0b1b6eae
Update rest tests skip version after backport (#69216)
Updates versions after fix backports

Relates to #66876 and #62130
2021-02-22 13:49:54 -05:00
Joe Gallo 7e7c5db74e
Rename max_single_primary_size to max_primary_shard_size (#69239) 2021-02-18 21:13:58 -05:00
Nik Everett db6b3b8cd4
Update skip after backport of #68930 (#69233) 2021-02-18 19:15:28 -05:00
Nhat Nguyen 94f16cd323
Avoid shard relocations in refresh yaml tests (#69095)
If shard relocations happen then a search can be executed by a relocated shard.

Closes #68562
2021-02-18 16:29:08 -05:00
Christoph Büscher e55e0f1792
Prevent field API NPEs from token_count fields inside nested (#69068)
Currently when a `token_count` field is defined inside a nested field, we get an
NPE because the underlying DocValueFetcher needs its formattedDocValues to be
loaded and the SourceLookup it sees needs to have a valid docId other than -1.
This change fixes those issues so the whole fields request doesn't error.
However this change doesn't solve the missing support for doc values lookup
under nested fields described in 68983. Fortunately `token_count` seems to be the only
mapping type currently affected.

Relates to #68983
2021-02-18 21:27:09 +01:00
Nik Everett 477d287f4a
Fix filter by filter execution with doc_count (#68930)
This fixed "filter by filter" execution order so it doesn't ignore
`doc_count`. The "filter by filter" execution is fairly performance
sensitive but when I reran performance numbers everything looked fine.
2021-02-17 10:10:34 -05:00
Alan Woodward 2c73387a14
Adjust YAML test skip value after backport (#69105)
Relates to #68738
2021-02-17 11:49:47 +00:00
Alan Woodward 8fba6e4a6d
Handle ignored fields directly in SourceValueFetcher (#68738)
Currently, the value fetcher framework handles ignored fields by reading
the stored values of the _ignored metadata field, and passing these through
on calls to fetchValues(). However, this means that if a document has multiple
values indexed for a field, and one malformed value, then the fields API will
ignore everything, including the valid values, and return an empty list for this
document.

If a document source contains a malformed value, then it must have been
ignored at index time. Therefore, we can safely assume that if we get an
exception parsing values from source at fetch time, they were also ignored
at index time and they can be skipped. This commit moves this exception
handling directly into SourceValueFetcher and ArraySourceValueFetcher,
removing the need to inspect the _ignored metadata and fixing the case
of mixed valid and invalid values.
2021-02-16 15:19:15 +00:00
Marios Trivyzas 3ba6e4f317
Update versions after backport of `max_analyzed_offset` (#69029)
Since #69016 is merged versions for serialisation/deserialisation
and YAML tests are updated.

Follows: #67325
2021-02-16 13:54:29 +01:00
Marios Trivyzas f9af60bf69
Add query param to limit highlighting to specified length (#67325)
Add a `max_analyzed_offset` query parameter to allow users
to limit the highlighting of text fields to a value less than or equal to the
`index.highlight.max_analyzed_offset`, thus avoiding an exception when
the length of the text field exceeds the limit. The highlighting still takes place,
but stops at the length defined by the new parameter.

Closes: #52155
2021-02-16 09:25:45 +01:00
Gordon Brown 3f6472de74
Introduce "Feature States" for managing snapshots of system indices (#63513)
This PR expands the meaning of `include_global_state` for snapshots to include system indices. If `include_global_state` is `true` on creation, system indices will be included in the snapshot regardless of the contents of the `indices` field. If `include_global_state` is `true` on restoration, system indices will be restored (if included in the snapshot), regardless of the contents of the `indices` field. Index renaming is not applied to system indices, as system indices rely on their names matching certain patterns. If restored system indices are already present, they are automatically deleted prior to restoration from the snapshot to avoid conflicts.

This behavior can be overridden to an extent by including a new field in the snapshot creation or restoration call, `feature_states`, which contains an array of strings indicating the "feature" for which system indices should be snapshotted or restored. For example, this call will only restore the `watcher` and `security` system indices (in addition to `index_1`):

```
POST /_snapshot/my_repository/snapshot_2/_restore
{
  "indices": "index_1",
  "include_global_state": true,
  "feature_states": ["watcher", "security"]
}
```

If `feature_states` is present, the system indices associated with those features will be snapshotted or restored regardless of the value of `include_global_state`. All system indices can be omitted by providing a special value of `none` (`"feature_states": ["none"]`), or included by omitting the field or explicitly providing an empty array (`"feature_states": []`), similar to the `indices` field.

The list of currently available features can be retrieved via a new "Get Snapshottable Features" API:
```
GET /_snapshottable_features
```

which returns a response of the form:
```
{
    "features": [
        {
            "name": "tasks",
            "description": "Manages task results"
        },
        {
            "name": "kibana",
            "description": "Manages Kibana configuration and reports"
        }
    ]
}
```

Features currently map one-to-one with `SystemIndexPlugin`s, but this should be considered an implementation detail. The Get Snapshottable Features API and snapshot creation rely upon all relevant plugins being installed on the master node.

Further, the list of feature states included in a given snapshot is exposed by the Get Snapshot API, which now includes a new field, `feature_states`, which contains a list of the feature states and their associated system indices which are included in the snapshot. All system indices in feature states are also included in the `indices` array for backwards compatibility, although explicitly requesting system indices included in a feature state is deprecated. For example, an excerpt from the Get Snapshot API showing `feature_states`:
```
"feature_states": [
    {
        "feature_name": "tasks",
        "indices": [
            ".tasks"
        ]
    }
],
"indices": [
    ".tasks",
    "test1",
    "test2"
]
```

Co-authored-by: William Brafford <william.brafford@elastic.co>
2021-02-11 11:55:14 -07:00
Christoph Büscher b213d8cbf9
Add additional fields API tests (#68897)
This change adds tests around the handling of mixed object and dot notation in
document source when using the `fields` API with nested fields left out
of #67432. After merging #68540, this test can now be added.

Relates to #67432
2021-02-11 18:38:39 +01:00
Igor Motov 7fe21e1c74
Mute "order by sub agg containing nested" test correctly (#68879)
Fat-fingered the skip statement in this one.

Relates to #66876
2021-02-10 22:15:29 -05:00
Igor Motov 0bbc6addd9
Revert "Remove aggregation's postCollect phase (#68615)
This partially reverts #64016 and  and adds #67839 and adds
additional tests that would have caught issues with the changes
in #64016. It's mostly Nik's code, I am just cleaning things up
a bit.

Co-authored-by: Nik Everett <nik9000@gmail.com>
2021-02-10 19:12:50 -05:00
Christoph Büscher 8b9fe12f1a Lower skip version for token_cound yaml test (#68583) 2021-02-08 11:27:31 +01:00
Lee Hinman 3f9f007545
Add the frozen tier node role and ILM phase (#68605)
This commit adds the `data_frozen` node role as part of the formalization of data tiers. It also
adds the `"frozen"` phase to ILM, currently allowing the same actions as the existing cold phase.

The frozen phase is intended to be used for data even less frequently searched than the cold phase,
and will eventually be loosely tied to data using partial searchable snapshots (as oppposed to full
searchable snapshots in the cold phase).

Relates to #60848
2021-02-05 14:38:13 -07:00
Julie Tibshirani af1cc495b2
Remove support for _type in searches (#68564)
Types are no longer allowed in requests in 8.0, so we can remove support for
using the `_type` field within a search request.

Relates to #41059.
Closes #68311.
2021-02-05 12:13:05 -08:00
Christoph Büscher 77c5bcd369 Adapt yaml test skip version after backport 2021-02-05 15:39:40 +01:00
Christoph Büscher e2d5183af0
Return structured nested data in ‘fields’ API
At the moment, the ‘fields’ API handles nested fields the same way I handles non-nested object arrays: it just returns them in a flat list. However, the relationship between nested fields is something we should try to preserve, since this is the main purpose of mapping something as “nested” instead of just using an object.

This PR changes this by returning grouped field values that are inside a nested object according to the nested object they initially appear in. Any further object structures inside a nested object are again returned as a flattened list. Fields inside nested fields don’t appear in the flattened response outside of the nested path any more. The grouping of fields inside nested objects is applied recursively if nested mappings are defined inside another nested mapping.

Closes #63709
2021-02-05 11:05:03 +01:00
Joe Gallo f444f64cd0
Adjust the MaxSinglePrimarySizeCondition version (#68553)
and re-enable BWC tests
2021-02-04 16:50:13 -05:00
Jason Tedor 01944627ed
Revert "Continue to publish REST API specifications under Apache 2.0 license (#68488)"
This reverts commit 92b59d994f.
2021-02-04 08:21:05 -05:00
Mark Vieira 92b59d994f
Continue to publish REST API specifications under Apache 2.0 license (#68488) 2021-02-03 13:46:20 -08:00
Joe Gallo 4d18334442
Add max_single_primary_size as a condition for the rollover index API (#67842) 2021-02-03 10:39:06 -05:00
Mark Vieira a92a647b9f Update sources with new SSPL+Elastic-2.0 license headers
As per the new licensing change for Elasticsearch and Kibana this commit
moves existing Apache 2.0 licensed source code to the new dual license
SSPL+Elastic license 2.0. In addition, existing x-pack code now uses
the new version 2.0 of the Elastic license. Full changes include:

 - Updating LICENSE and NOTICE files throughout the code base, as well
   as those packaged in our published artifacts
 - Update IDE integration to now use the new license header on newly
   created source files
 - Remove references to the "OSS" distribution from our documentation
 - Update build time verification checks to no longer allow Apache 2.0
   license header in Elasticsearch source code
 - Replace all existing Apache 2.0 license headers for non-xpack code
   with updated header (vendored code with Apache 2.0 headers obviously
   remains the same).
 - Replace all Elastic license 1.0 headers with new 2.0 header in xpack.
2021-02-02 16:10:53 -08:00
Hendrik Muhs 4cbe61467c
add possibility to mute yaml tests by operating system (#67681)
this change adds the possibility to mute yaml tests based on operating system to avoid muting whole
tests
2021-02-01 09:33:23 +01:00
William Brafford 42b748588d
Allow the "*,-*" ("match none") pattern for destructive actions when destructive_requires_name is true (#68021)
Since the "*,-*" pattern resolves to "no indices", it makes a normally
destructive action into a non-destructive one. Rather than throwing a
wildcards-not-allowed exception, we can allow this pattern to pass
without triggering an exception. This allows the security layer to
safely use a "*,-*" pattern to indicate a "no indices" result for its
index resolution step, which is important because otherwise we get
wildcards-not-allowed exceptions when trying to delete nonexistent
concrete indices. For simplicity, we require exactly "*,-*", rather than
any other wildcards that might be logically equivalent.
2021-01-28 14:08:29 -05:00
David Turner 06e141888f
Reinstate BWC snapshot tests (#67938)
This commit mostly reverts #67934, except for the change to the version
constant `REPOSITORY_UUID_IN_REPO_DATA_VERSION`.

Completes the backport of #67829 via #67899
2021-01-25 18:36:12 +00:00
David Turner faed3e7199
Temporarily suppress BWC snapshot tests (#67934)
This commit suppresses any BWC tests related to snapshots in `master` so
that #67899 can be merged to `7.x`. It will mostly be reverted after the
merge of #67899 is complete.

Relates #66431
2021-01-25 17:48:47 +00:00
David Turner e5a15d4fcb
Introduce repository UUIDs (#67829)
Today a snapshot repository does not have a well-defined identity. It
can be reregistered with a different cluster under a different name, and
can even be registered with multiple clusters in readonly mode.

This presents problems for cases where we need to refer to a specific
snapshot in a globally-unique fashion. Today we rely on the repository
being registered under the same name on every cluster, but this is not a
safe assumption.

This commit adds a UUID that can be used to uniquely identify a
repository. The UUID is stored in the top-level index blob, represented
by `RepositoryData`, and is also usually copied into the
`RepositoryMetadata` that represents the repository in the cluster
state. The repository UUID is exposed in the get-repositories API; other
more meaningful consumers will be added in due course.
2021-01-25 12:17:52 +00:00
David Turner bc1f50c523
Permit wait_for_active_shards warnings in master (#67498)
Part of the fixes for #66419, this commit permits nodes to emit the
deprecation warning regarding not specifying `?wait_for_active_shards`
when closing an index in 7.x versions for x ≥ 12. This change is
required on `master` too since the BWC tests encounter these warnings.

Relates #67246, which is the 7.x part of this change.
2021-01-14 15:55:43 +00:00
Andrei Stefan e3386e155c
Add minimum compatibility version to SearchRequest (#65896)
* Adds a minimum version request parameter to SearchRequest.
The minimum version helps failing a request if any shards
involved in the search do not meet the compatibility requirements
(all shards need to have a version equal or later than the minimum
version provided).
2021-01-13 00:50:30 +02:00
David Turner ec08f924c7
Introduce ?wait_for_active_shards=index-setting (#67158)
In 7.x the close indices API defaulted to `?wait_for_active_shards=0`
but from 8.0 it defaults to respecting the index settings instead.  This
commit introduces the `index-setting` value for this parameter on this
API allowing users to opt-in to the future behaviour today, and emits a
deprecation warning indicating that the default no longer needs to be
used and will be unsupported in future.

In 7.x a follow up PR will introduce support for the same
`index-setting` value for this parameter and will emit deprecation
warnings if users try and use the default instead.

Relates #66419
2021-01-11 08:33:16 +00:00
Nik Everett a9e8a6a31b
Update skip after backport of #67043 (#67191)
Now that #67043 has been backported we can update the skip so the bwc
tests don't complain.
2021-01-07 17:01:53 -05:00
Nik Everett f23e568948 Update skip before backport of #67043
When I merged #67043 it had an integration test for the thing it was
fixing but it still fails in the bwc tests. Yikes! I should know better
but life is life. Anyway, this updates the skip to ignore the test for
now. I'll reenable once the backport is in.
2021-01-07 10:57:39 -05:00
Nik Everett b0747c5a76
Fix bug with nested and filters agg (#67043)
Fixes a bug where nested documents that match a filter in the `filters`
agg will be counted as matching the filter. Usually nested documents
only match if you explicitly ask to match them. Worse, we only mach them
in the "filter by filter" mode that we wrote to speed up date_histogram.
The `filters` agg is fairly rare, but with #63643 we run
`date_histogram` and `range` aggregations using `filters.
2021-01-07 10:05:59 -05:00
Julie Tibshirani 1515f36f7f
Make sure shared source always represents the top-level root document. (#66725)
We started passing down the root document's _source when processing
nested hits, to avoid reloading and reparsing the root source for each hit.
Unfortunately the approach did not work when there are multiple layers of
`inner_hits`. In this case, the second-layer inner hit received its immediate
parent's source instead of the root source. This parent source is filtered to
just contain the parts corresponding to the nested document, but the source
parsing logic is designed to always operate on the top-level root source. This
caused failures when loading the second-layer inner hits.

This PR makes sure to always pass the root document's _source when processing
inner hits, even if there are multiple layers.
2021-01-05 08:17:26 -08:00
Nik Everett dd1ffe3900
Update skip after backport of #66295 (#66808)
Now that #66295 has landed in 7.11 we can run the bwc tests it adds
against that branch.
2020-12-23 16:33:34 -05:00
Ioannis Kakavas bd873698bc
Ensure CI is run in FIPS 140 approved only mode (#64024)
We were depending on the BouncyCastle FIPS own mechanics to set
itself in approved only mode since we run with the Security
Manager enabled. The check during startup seems to happen before we
set our restrictive SecurityManager though in
org.elasticsearch.bootstrap.Elasticsearch , and this means that
BCFIPS would not be in approved only mode, unless explicitly
configured so.

This commit sets the appropriate JVM property to explicitly set
BCFIPS in approved only mode in CI and adds tests to ensure that we
will be running with BCFIPS in approved only mode when we expect to.
It also sets xpack.security.fips_mode.enabled to true for all test clusters
used in fips mode and sets the distribution to the default one. It adds a
password to the elasticsearch keystore for all test clusters that run in fips
mode.
Moreover, it changes a few unit tests where we would use bcrypt even in
FIPS 140 mode. These would still pass since we are bundling our own
bcrypt implementation, but are now changed to use FIPS 140 approved
algorithms instead for better coverage.

It also addresses a number of tests that would fail in approved only mode
Mainly:

    Tests that use PBKDF2 with a password less than 112 bits (14char). We
    elected to change the passwords used everywhere to be at least 14
    characters long instead of mandating
    the use of pbkdf2_stretch because both pbkdf2 and
    pbkdf2_stretch are supported and allowed in fips mode and it makes sense
    to test with both. We could possibly figure out the password algorithm used
    for each test and adjust password length accordingly only for pbkdf2 but
    there is little value in that. It's good practice to use strong passwords so if
    our docs and tests use longer passwords, then it's for the best. The approach
    is brittle as there is no guarantee that the next test that will be added won't
    use a short password, so we add some testing documentation too.
    This leaves us with a possible coverage gap since we do support passwords
    as short as 6 characters but we only test with > 14 chars but the
    validation itself was not tested even before. Tests can be added in a followup,
    outside of fips related context.

    Tests that use a PKCS12 keystore and were not already muted.

    Tests that depend on running test clusters with a basic license or
    using the OSS distribution as FIPS 140 support is not available in
    neither of these.

Finally, it adds some information around FIPS 140 testing in our testing
documentation reference so that developers can hopefully keep in
mind fips 140 related intricacies when writing/changing docs.
2020-12-23 21:00:49 +02:00
Nik Everett 3e3152406a
Bust the request cache when the mapping changes (#66295)
This makes sure that we only serve a hit from the request cache if it
was build using the same mapping and that the same mapping is used for
the entire "query phase" of the search.

Closes #62033
2020-12-23 13:19:02 -05:00
Seth Michael Larson 11153fcb33
Use single backslash for nested paths (#66794) 2020-12-23 11:57:19 -06:00
Julie Tibshirani 652ff74adf Adjust test skips now that inner_hits fix is backported. 2020-12-21 15:08:34 -08:00
Julie Tibshirani 15f5758957
Fix regressions around nested hits and disabled _source. (#66572)
This PR fixes two bugs that can arise when _source is disabled and we fetch nested documents:
* Fix exception when highlighting `inner_hits` with disabled _source.
* Fix exception in nested `top_hits` with disabled _source.
* Add more tests for highlighting `inner_hits`.
2020-12-18 14:06:52 -08:00
Jim Ferenczi c756ce1acf
Sort field tiebreaker for PIT (point in time) readers (#66093)
This commit introduces a new sort field called `_shard_doc` that
can be used in conjunction with a PIT to consistently tiebreak
identical sort values. The sort value is a numeric long that is
composed of the ordinal of the shard (assigned by the coordinating node)
and the internal Lucene document ID. These two values are consistent within
a PIT so this sort criteria can be used as the tiebreaker of any search
requests.
Since this sort criteria is stable we'd like to add it automatically to any
sorted search requests that use a PIT but we also need to expose it explicitly
in order to be able to:
* Reverse the order of the tiebreaking, useful to search "before" `search_after`.
* Force the primary sort to use it in order to benefit from the `search_after` optimization when sorting by index order (to be released in Lucene 8.8.

I plan to add the documentation and the automatic configuration for PIT in a follow up since this change is already big.

Relates #56828
2020-12-18 12:13:12 +01:00
Steve Gordon 96555bfa33
Mark Cat Tasks API as experimental in rest-api-spec (#66536) 2020-12-17 17:54:21 +00:00
Rory Hunter 1eb50f876e Tweak version skip range in cat.plugin yml test 2020-12-17 10:06:32 +00:00
Rory Hunter 4ff612550e
Allow bootstrap plugins to appear in _cat/plugins (#66260)
Closes #66107.

Bootstrap plugins are not loaded in the main Elasticsearch process, but
instead take effect only when ES is starting. As such, these plugins are
skipped when ES loads all installed plugins.

As a result, it was impossible for the plugins _cat API to report
whether any bootstrap plugins are installed.

Fix this by adjusting how the loading process skips bootstrap plugins,
and then tweaking the plugins _cat API so that bootstrap plugins can
optionally be included in the response.
2020-12-17 09:30:16 +00:00
Julie Tibshirani 6bc56d18e5
Fix failure in fvh REST tests. (#66192)
In general, we can't guarantee that a match_all query will return documents in
the order they were indexed. This PR adds an ID to each document to avoid
relying on document order.
2020-12-15 11:28:51 -08:00
Jay Modi 7011cbac5d
Fix cat tasks api params in spec and handler (#66272)
This commit fixes the cat tasks api parameter specification and the
handler so that the parameters are consumed during request preparation.

Closes #59493
2020-12-14 12:29:25 -07:00
Jay Modi 410ae396bd
Remove suggest reference in some API specs (#66180)
This commit removes the reference to the suggest index metric in the
nodes stats and indices stats rest api spec files. Suggest has been
removed so it is no longer correct to have it in these files.

Closes #43407
2020-12-14 10:00:04 -07:00
Martijn Laarman e31e3dea32
Add `visibility` the to rest-spec-api (#56104) 2020-12-14 12:23:28 +01:00
Julie Tibshirani a40186a7f4 Adjust skip version in fvh REST tests.
We can expand the compatible versions now that the bug fix has been packported.
2020-12-10 13:39:47 -08:00
Fernando Briano 91595657a2
Wraps timestamp values in quotes in search.aggregation histogram YAML test (#66153) 2020-12-10 12:19:03 +00:00
Dimitrios Liappis 5b24829796
Mute fvh REST tests (#66149)
Relates #66147
2020-12-10 12:13:23 +02:00
Julie Tibshirani 3cbe0eadce Ensure consistent hit order in fvh REST tests. 2020-12-09 17:53:17 -08:00
Julie Tibshirani ddf1f4cdb8
Fix bug where fvh fragments could be loaded from wrong doc (#65641)
This PR fixes a regression where fvh fragments could be loaded from the wrong
document _source.

Some `FragmentsBuilder` implementations contain a `SourceLookup` to load from
_source. The lookup should be positioned to load from the current hit document.
However, since `FragmentsBuilder` are cached and shared across hits, the lookup
is never updated to load from the new documents. This means we accidentally
load _source from a different document.

The regression was introduced in #60179, which started storing `SourceLookup`
on `FragmentsBuilder`.

Fixes #65533.
2020-12-09 14:47:11 -08:00
Ioannis Kakavas 3cd93eef92
Skip REST YAML tests in FIPS 140 mode (#65735)
Currently we don't have a way to selectively mute REST YAML tests
in FIPS 140 mode.

This commit introduces a new feature (fips_140) that can be used
in skip blocks to allow that.
2020-12-09 15:45:31 +02:00
Martijn Laarman 8d3def3e1f
Add Accept & Content-Type headers to rest api spec (#53979)
Co-authored-by: Russ Cam <russ.cam@elastic.co>
2020-12-09 14:43:05 +01:00
Nik Everett 9e57a46c56
Update skip after backport (#66024)
After backporting #65707 we can now run it in our backwards
compatibility tests.
2020-12-08 12:01:56 -05:00
Nik Everett ecb4e16d35
Fix sneaky date_histogram bug (#65707)
`date_histogram` has a bug with `offset` and `extended_bounds` when it
needs to create an "empty" aggregation result: it includes the bounds
twice! Wooops!

I broke this a while back when I started trying to merge `offset` into
`Rounding`. I never finished that merge, sadly. Interestingly, we've
discovered that the merge is required to properly handle daylight
savings time (#56305) but it isn't really something we're looking to
solve today. For now, this just stops counting the offset twice.

Closes #65624
2020-12-07 15:39:52 -05:00
Nik Everett 92e803a06d
Fix test sending body as url parameter (#65779)
Our test framework randomly sends the body of requests over the `source`
parameter. We never send the body if it is more than 2000 bytes becuse
our HTTP receiver can't handle lines more the 4096 bytes. The thing is,
when we do that 2000 bytes check we do it against the reported length of
body, not the body after it has been url encoded. Than url encoding
strings can *vastly* increase their size. Which could cause us to send
some request over the URL that are longer than 4096 bytes.

This fixes that by checking the url encoded length as well. We keep the
2000 byte check of the unencoded length because it is a nice fast check,
even if it is a bit inaccurate.

Closes #65718
2020-12-07 12:26:34 -05:00
Seth Michael Larson 56a25bf08b
Mark Task APIs as experimental in rest-api-spec 2020-12-03 13:47:13 -06:00
Christoph Büscher 8c29e54bf3 Adapt test skip versions after backport 2020-12-03 16:10:55 +01:00
Christos Soulios 1c8a97784e
Lower minimum compatibility version of _doc_count field tests (#65790)
After merging _doc_count field type in v7.11.0 (#64594), this PR lowers the minimum compatibility
version from v8.0.0 to v7.11.0

Relates to #64503
2020-12-03 12:08:00 +02:00
Alan Woodward f0fc1b3dad
Use scriptless fields in CoreTestTranslator (#65599)
CoreTestTranslator re-writes some core search yaml tests into a tests
for runtime queries, and mimics source-only fields by building ad-hoc
painless scripts for each runtime type. We now have source-only fields
built in via scriptless mappings, so we can cut over to using these
instead.
2020-12-02 10:06:40 +00:00
Christoph Büscher 3c3a43249f
Support unmapped fields in search 'fields' option (#65386)
Currently, the 'fields' option only supports fetching mapped fields. Since
'fields' is meant to be the central place to retrieve document content, it
should allow for loading unmapped values. This change adds implementation and
tests for this feature.

Closes #63690
2020-12-01 21:40:27 +01:00
Christoph Büscher c327794ae8
Fix range query on date fields for number inputs (#63692)
Currently, if you write a date range query with numeric 'to' or 'from' bounds,
they can be interpreted as years if no format is provided. We use
"strict_date_optional_time||epoch_millis" in this case that can interpret inputs
like 1000 as the year 1000 for example. 
This PR change this to always interpret and parse numbers with the "epoch_millis"
parser if no other formatter was provided.

Closes #63680
2020-12-01 18:49:50 +01:00
Rene Groeschke 97749a3372
Port rest integ tests to use task avoidance api (#65011)
This ports the majority of the rest integ tests tasks to use the task avoidance api.

- There are some edge cases left that we need to investigate, but we can do that separately.
2020-11-26 10:30:06 +01:00
Christoph Büscher 6854974c1c
Reactivate deactivated integration test (#65388) 2020-11-24 10:48:14 +01:00
Mark Vieira 53b10dbb66 Mute "Tests catching other exceptions per item" 2020-11-23 10:16:01 -08:00