Add support to delete component templates api to specify multiple template
names separated by a comma.
Change the cleanup template logic for rest tests to remove all component templates via a single delete component template request. This to optimize the cleanup logic. After each rest test we delete all templates. So deleting templates this via a single api call (and thus single cluster state update) saves a lot of time considering the number of rest tests.
Older versions don't support component / composable index templates
and/or data streams. Yet the test base class tries to remove objects
after each test, which adds a significant number of lines to the
log files (which slows the tests down). The ESRestTestCase will
now check whether all nodes have a specific version and then decide
whether data streams and component / composable index templates will
be deleted.
Also ensured that the logstash-index-template and security-index-template
aren't deleted between tests, these templates are builtin templates that
ES will install if missing. So if tests remove these templates between tests
then ES will add these template back almost immediately. These causes
many log lines and a lot of cluster state updates, which slow tests down.
Relates to #69973
Co-authored-by: Lee Hinman <dakrone@users.noreply.github.com>
Add support to delete composable index templates api to specify multiple template
names separated by a comma.
Change to cleanup template logic for rest tests to remove all composable index templates via a single delete composable index template request. This to optimize the cleanup logic. After each rest test we delete all templates. So deleting templates this via a single api call (and thus single cluster state update) saves a lot of time considering the number of rest tests.
If this pr is accepted then I will do the same change for the delete component template api.
Relates to #69973
This makes sure that we only serve a hit from the request cache if it
was build using the same mapping and that the same mapping is used for
the entire "query phase" of the search.
Closes#62033
Closes#20640.
This PR introduces a new parameter to v2 templates, `allow_auto_create`,
which allows templates to override the cluster setting `auto_create_index`.
Notes:
* `AutoCreateIndex` now looks for a matching v2 template, and if its
`allow_auto_create` setting is true, it overrides the usual logic.
* `TransportBulkAction` previously used `AutoCreateIndex` to check
whether missing indices should be created. We now rely on
`AutoCreateAction`, which was already differentiating between creating
indices and creating data streams. I've updated `AutoCreateAction` to
use `AutoCreateIndex`. Data streams are also influenced by
`allow_auto_create`, in that their default auto-create behaviour can
be disabled with this setting.
* Most of the Java file changes are due to introducing an extra
constructor parameter to `ComposableIndexTemplate`.
* I've added the new setting to various x-pack templates
* I added a YAML test to check that watches can be created even when
`auto_create_index` is `false`.
This PR adds deprecation warnings when accessing System Indices via the REST layer. At this time, these warnings are only enabled for Snapshot builds by default, to allow projects external to Elasticsearch additional time to adjust their access patterns.
Deprecation warnings will be triggered by all REST requests which access registered System Indices, except for purpose-specific APIs which access System Indices as an implementation detail a few specific APIs which will continue to allow access to system indices by default:
- `GET _cluster/health`
- `GET {index}/_recovery`
- `GET _cluster/allocation/explain`
- `GET _cluster/state`
- `POST _cluster/reroute`
- `GET {index}/_stats`
- `GET {index}/_segments`
- `GET {index}/_shard_stores`
- `GET _cat/[indices,aliases,health,recovery,shards,segments]`
Deprecation warnings for accessing system indices take the form:
```
this request accesses system indices: [.some_system_index], but in a future major version, direct access to system indices will be prevented by default
```
The underlying issue was fixed a while ago in Lucene:
https://issues.apache.org/jira/browse/LUCENE-9517
and went away when lucene snapshot version was upgraded.
Also the name of the index to rollover had to be slightly changed,
so that it doesn't collide with data stream template's namespace.
(a regular index can't be created in the namespace that is managed
by a template that creates data streams)
Closes#62043
* Add "synthetics-*-*" templates for synthetics fleet data
For the Elastic Agent we currently have `logs` and `metrics`, however, synthetic data doesn't belong
with those and thus we should have a place for it to live. This would be data reported from
heartbeat and under the 'monitoring' category.
This commit adds a composable index template for `synthetics-*-*` indices similar to the work in
#56709 and #57629.
Resolves#61665
Adds an important admonition for the built-in `metrics-*-*` and `logs-*-*` index
templates.
Updates several put index template snippets to include a priority.
Uses `my-data-stream` in place of `logs` for data stream examples.
This provides a more intuitive experience for users that copy/paste
their own values into snippets.
This commit fixes the list dangling indices response.
The dangling_indices array is an array of objects
that represent aggregated dangling index information
Changes:
* Adds the `number_of_routing_shards` index setting to index modules docs.
* Updates the split API docs to mention that `number_of_routing_shards`
is a static setting.
Removes the `@timestamp` field mapping from several data stream index
template snippets.
With #59317, the `@timestamp` field defaults to a `date` field data type
for data streams.
This makes the data_stream timestamp field specification optional when
defining a composable template.
When there isn't one specified it will default to `@timestamp`.
The commit makes the following changes:
* The timestamp field of a data stream definition in a composable
index template can only be set to '@timestamp'.
* Removed custom data stream timestamp field validation and reuse the validation from `TimestampFieldMapper` and
instead only check that the _timestamp field mapping has been defined on a backing index of a data stream.
* Moved code that injects _timestamp meta field mapping from `MetadataCreateIndexService#applyCreateIndexRequestWithV2Template58956(...)` method
to `MetadataIndexTemplateService#collectMappings(...)` method.
* Fixed a bug (#58956) that cases timestamp field validation to be performed
for each template and instead of the final mappings that is created.
* only apply _timestamp meta field if index is created as part of a data stream or data stream rollover,
this fixes a docs test, where a regular index creation matches (logs-*) with a template with a data stream definition.
Relates to #58642
Relates to #53100Closes#58956Closes#58583
Small edit highlighting the fact that atomic cluster state change does not guarantee lack of errors for in-flight requests.
Co-authored-by: James Rodewig <james.rodewig@elastic.co>
Part of #48366. Add documentation for the dangling indices
API added in #58176.
Co-authored-by: David Turner <david.turner@elastic.co>
Co-authored-by: Adam Locke <adam.locke@elastic.co>
Martijn van Groningen
James Rodewig
Tommmster
Adam Locke
Joe Gallo
David Turner
Pius
bellengao
Jake Landis
Nik Everett
Dan Hermann
DeDe Morton
Lee Hinman
Rudolf Meijering
Przemko Robakowski
Rory Hunter
Gordon Brown
David Kyle
Julie Tibshirani
Russ Cam
debadair
James Rodewig
Andrei Dan
Grzegorz Banasiak