The `data-streams/downsampling.asciidoc` test was missing a teardown clean of the ILM policies created. Due to this tests *do not have* the string `ilm` in its name, the automatic teardown process that cleans up the resources (check `ESRestTestCase.java#L815` & `DocsClientYamlTestSuiteIT.java` lines 177 & 195) is not executed for this specific test. In the case this test runs right before the `get-lifecycle` test, the policy won't be automatically deleted hence the test checking the version will fail. Finally, the order of execution of the test is not guaranteed by the suite.
This change at a high level adds global ranking on the coordinating node at the end of query reduction
prior to the fetch phase. Individual rank methods are defined in plugins.
The first rank plugin added as part of this change is reciprocal rank fusion (RRF). RRF uses a relatively
simple formula for merging 1...n results sets together with sum(1/(k+d)) where k is a ranking constant
and d is a document's scored position within a result set from a query.
This change adds:
* Total global ordinal build time for all fields and per field.
* Max shard value count per field. The value count is per shard and of the shard with the highest count. Reporting value on index level or across indices is too expensive to report or keep track of.
This is added to common stats, which
is exposed in several stats APIs.
The following api call:
```
GET /_nodes/stats?filter_path=nodes.*.indices.fielddata&fields=key,key2
```
Returns:
```
{
"nodes": {
"pcMNy4GsQ8ef6Rw-bI2EFg": {
"indices": {
"fielddata": {
"memory_size_in_bytes": 2552,
"evictions": 0,
"fields": {
"key2": {
"memory_size_in_bytes": 1320
},
"key": {
"memory_size_in_bytes": 1232
}
},
"global_ordinals": {
"build_time_in_millis": 8,
"fields": {
"key2": {
"build_time_in_millis": 4,
"shard_max_value_count": 4
},
"key": {
"build_time_in_millis": 4,
"shard_max_value_count": 4
}
}
}
}
}
}
}
}
```
This introduces an endpoint to reset the desired balance.
It could be used if computed balance diverged from the actual one a lot
to start a new computation from the current state.
This adds support for an index's index.lifecycle.origination_date setting in DLM. If an index has a
value for index.lifecycle.origination_date then it is used instead of the creation date or rollover
date (except in the case of the write index when the write index has not been rolled over yet).
* [docs] Clarify that index template settings take precedence over component templates.
[docs] Clarify that index template settings take precedence over component templates.
* Update docs/reference/indices/index-templates.asciidoc
Co-authored-by: Adam Locke <adam.locke@chronosphere.io>
---------
Co-authored-by: Abdon Pijpelink <abdon.pijpelink@elastic.co>
Co-authored-by: Adam Locke <adam.locke@chronosphere.io>
* It adds the profiling index pattern profiling-* to the fleet server service privileges.
* And adds profiling-* to kibana system role privileges.
---------
Co-authored-by: Daniel Mitterdorfer <daniel.mitterdorfer@elastic.co>
Adds a new include flag definition_status to the GET trained models API.
When present the trained model configuration returned in the response
will have the new boolean field fully_defined if the full model definition
is exists.
Currently Lucene limits the max number of vector dimensions to 1024.
This commit overrides KnnFloatVectorField and KnnByteVectorField
classes to increase the limit to 2048 for indexed vectors in ES.
This changes the `GET _data_stream/ds_name/_lifecycle` endpoint to
return the data stream name even if it doesn't have a lifecycle
configured.
e.g.
```
{
"data_streams": [
{
"name": "logs-nginx"
}
]
}
```
* Fix xcontent and tests
* Update docs
---------
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
This removes a redundant `lifecycle` field in the `PUT _lifecycle`
request.
Before we had
```
PUT _data_stream/logs-nginx/_lifecycle
{
"lifecycle": {
"data_retention": "7d"
}
}
```
This changes the request to
```
PUT _data_stream/logs-nginx/_lifecycle
{
"data_retention": "7d"
}
```
Here we add synthetic source support for fields whose type is flattened.
Note that flattened fields and synthetic source have the following limitations,
all arising from the fact that in synthetic source we just see key/value pairs
when reconstructing the original object and have no type information in mappings:
* flattened fields use sorted set doc values of keywords, which means two things:
first we do not allow duplicate values, second we treat all values as keywords
* reconstructing array of objects results in nested objects (no array)
* reconstructing arrays with just one element results in a single-value field since we
have no way to distinguish single-valued from multi-values fields other then looking
at the count of values
* Update release notes to include 8.7.0
Release notes and migration guide from 8.7.0 release ported into main
as well as re-generating 8.8.0 release notes. This latter step will
be overwritten anyway, multiple times, by more up-to-date regeneration
of the 8.8.0 release notes during the release process.
* Remove coming 8.7.0 line
* Update docs/reference/migration/migrate_8_7.asciidoc
Co-authored-by: Abdon Pijpelink <abdon.pijpelink@elastic.co>
* Make same change to 8.8
---------
Co-authored-by: Abdon Pijpelink <abdon.pijpelink@elastic.co>
gmarouli
Martijn van Groningen
Pablo Alcantar Morales
Abdon Pijpelink
Sloane Perrault
Stef Nestor
Jack Conradson
Leaf-Lin
Mayya Sharipova
Salvatore Campagna
Ievgen Degtiarenko
Keith Massey
Toby Sutor
Felix Barnsteiner
Cauê Marcondes
Kathleen DeRusso
David Kyle
Andrei Dan
David Turner
Przemysław Witek
Joe Gallo
Aurélien FOUCRET
David Kilfoyle
Alessandro Stoltenberg
Jean-Fabrice Bobo
Matthias Wilhelm
QY
István Zoltán Szabó
Craig Taverner