It seems that for now we don't have a good use for the histogram and summary metric types.
They had been left as place holders for a while, but at this point there is no concrete plan forward for them.
This PR removes the histogram and summary metric types. We may add them back in the future.
Also, this PR completely removes the time_series_metric mapping parameter from the histogram field type and only allows the gauge metric type for aggregate_metric_double fields.
Added a requirement that index.look_ahead_time index setting
can't be lower than time_series.poll_interval setting.
Additional changes:
* Fixed a mistake in the docs that referenced indices.lifecycle.poll_interval
instead of time_series.poll_interval.
* Moved index.look_ahead_time setting to data stream module.
* Revert "Revert "[DOCS] Add TSDS docs (#86905)" (#87702)"
This reverts commit 0c86d7b9b2.
* First fix to tests
* Add data_stream object to index template
* small rewording
* Add enable data stream object in gradle example setup
* Add bullet about data stream must be enabled in template
* [DOCS] Add TSDB docs
* Update docs/build.gradle
Co-authored-by: Adam Locke <adam.locke@elastic.co>
* Address Nik's comments, part 1
* Address Nik's comments, part deux
* Reword write index
* Add feature flags
* Wrap one more section in feature flag
* Small fixes
* set index.routing_path to optional
* Update storage reduction value
* Update create index template code example
Co-authored-by: James Rodewig <40268737+jrodewig@users.noreply.github.com>
Co-authored-by: Adam Locke <adam.locke@elastic.co>
Internally we already kept track of whether a data stream is replicated by CCR.
It is part of the `DataStream` class. This just adds it to the xcontent serialization
of the get data stream api response class.
Relates to elastic/kibana#118899
This PR changes uses of transient cluster settings to
persistent cluster settings.
The PR also deprecates the transient settings usage.
Relates to #49540
To switch an index's lifecycle policy, you must first remove the existing
policy. Otherwise, phase execution for the index may silently fail.
Closes#70151
This commit adds support for system data streams and also the first use
of a system data stream with the fleet action results data stream. A
system data stream is one that is used to store system data that users
should not interact with directly. Elasticsearch will manage these data
streams. REST API access is available for external system data streams
so that other stack components can store system data within a system
data stream. System data streams will not use the system index read and
write threadpools.
Changes:
- Reworks the ILM tutorial to focus on the Elastic Agent and a built-in ILM policy
- Updates several screenshots in the docs for the new ILM UI
Co-authored-by: debadair <debadair@elastic.co>
This fixes the manage_follow_index builtin privilege so that it can be used
for managing data streams in a follower cluster. In order to successfully
unfollow a data stream the promote data stream and rollover APIs need to be
executed. (This is additional to the close and unfollow APIs).
* [DOCS] Updated data streams list screenshots and delete functionality description
* Update docs/reference/data-streams/set-up-a-data-stream.asciidoc
Co-authored-by: James Rodewig <40268737+jrodewig@users.noreply.github.com>
* Update set-up-a-data-stream.asciidoc
Co-authored-by: James Rodewig <40268737+jrodewig@users.noreply.github.com>
When a data stream is being auto followed then a rollover in a local cluster can break auto following,
if the local cluster performs a rollover then it creates a new write index and if then later the remote
cluster rolls over as well then that new write index can't be replicated, because it has the same name
as in the write index in the local cluster, which was created earlier.
If a data stream is managed by ccr, then the local cluster should not do a rollover for those data streams.
The data stream should be rolled over in the remote cluster and that change should replicate to the local
cluster. Performing a rollover in the local cluster is an operation that the data stream support in ccr should
perform.
To protect against rolling over a replicated data stream, this PR adds a replicate field to DataStream class.
The rollover api will fail with an error in case a data stream is being rolled over and the targeted data stream is
a replicated data stream. When the put follow api creates a data stream in the local cluster then the replicate flag
is set to true. There should be a way to turn a replicated data stream into a regular data stream when for example
during disaster recovery. The newly added api in this pr (promote data stream api) is doing that. After a replicated
data stream is promoted to a regular data stream then the local data stream can be rolled over, so that the new
write index is no longer a follower index. Also if the put follow api is attempting to update this data stream
(for example to attempt to resume auto following) then that with fail, because the data stream is no longer a
replicated data stream.
Today with time based indices behind an alias, the is_write_index property isn't replicated from remote cluster
to the local cluster, so when attempting to rollover the alias in the local cluster the rollover fails, because the
alias doesn't have a write index. The added replicated field in the DataStream class and added validation
achieve the same kind of protection, but in a more robust way.
A followup from #61993.
* Add "synthetics-*-*" templates for synthetics fleet data
For the Elastic Agent we currently have `logs` and `metrics`, however, synthetic data doesn't belong
with those and thus we should have a place for it to live. This would be data reported from
heartbeat and under the 'monitoring' category.
This commit adds a composable index template for `synthetics-*-*` indices similar to the work in
#56709 and #57629.
Resolves#61665
Adds an important admonition for the built-in `metrics-*-*` and `logs-*-*` index
templates.
Updates several put index template snippets to include a priority.
Uses `my-data-stream` in place of `logs` for data stream examples.
This provides a more intuitive experience for users that copy/paste
their own values into snippets.
Removes the `@timestamp` field mapping from several data stream index
template snippets.
With #59317, the `@timestamp` field defaults to a `date` field data type
for data streams.
This makes the data_stream timestamp field specification optional when
defining a composable template.
When there isn't one specified it will default to `@timestamp`.
This adds a low precendece mapping for the `@timestamp` field with
type `date`.
This will aid with the bootstrapping of data streams as a timestamp
mapping can be omitted when nanos precision is not needed.
Replaces `composable index template` and `composable template` with
`index template` throughout data stream-related docs.
`Composable index template` is only used to contrast with legacy index
templates.
With #58096, data streams now track the timestamp field mapping outside
of the template associated with the stream. This means you can no longer
update the timestamp field mapping using template changes.
This updates the associated data stream docs.
Changes:
* Adds additional examples to the `Search a data stream` section of
`Use a data stream`
* Updates existing search docs to make them aware of data streams
Adds corresponding steps for the get and delete data stream APIs to the
data stream setup tutorial. Also provides some guidance on how to
determine the current write index for a data stream.
David Kilfoyle
Christos Soulios
Martijn van Groningen
James Rodewig
Dan Hermann
James Rodewig
Nikola Grcevski
xiaoping
Joe Gallo
Jay Modi
Yulia Čech
DeDe Morton
Przemko Robakowski
Lee Hinman
Andrei Dan