68 lines
2.9 KiB
Plaintext
68 lines
2.9 KiB
Plaintext
|
[[data-streams]]
|
||
|
= Data streams
|
||
|
|
|
||
|
|
[partintro]
|
||
|
|
--
|
||
|
|
You can use data streams to index time-based data that's continuously generated.
|
||
|
|
A data stream groups indices from the same time-based data source.
|
||
|
|
A data stream tracks its indices, known as _backing indices_, using an ordered
|
||
|
|
list.
|
||
|
|
|
||
|
|
A data stream's backing indices are <<index-hidden,hidden>>.
|
||
|
|
While all backing indices handle read requests, the most recently created
|
||
|
|
backing index is the data stream's only write index. A data stream only
|
||
|
|
accepts <<docs-index_,index requests>> with `op_type` set to `create`. To update
|
||
|
|
or delete specific documents in a data stream, submit a <<docs-delete,delete>>
|
||
|
|
or <<docs-update,update>> API request to the backing index containing the
|
||
|
|
document.
|
||
|
|
|
||
|
|
To create a data stream, set up a <<indices-templates,composable index
|
||
|
|
template>> containing:
|
||
|
|
|
||
|
|
* A name or wildcard pattern for the data stream in the `index_patterns` property.
|
||
|
|
* A `data_stream` definition that contains the `timestamp_field` property.
|
||
|
The `timestamp_field` must be the primary timestamp field
|
||
|
|
for the data source. This field must be included in every
|
||
|
|
document indexed to the data stream.
|
||
|
|
|
||
|
|
When you index one or more documents to a not-yet-existent target matching
|
||
|
|
the template's name or pattern, {es} automatically creates the corresponding
|
||
|
|
data stream. You can also manually create a data stream using the
|
||
|
|
<<indices-create-data-stream,create data stream API>>. However, a composable
|
||
|
|
template for the stream is still required.
|
||
|
|
|
||
|
|
You can use the <<indices-rollover-index,rollover API>> to roll a data stream
|
||
|
|
over to a new index when the current write index meets specified criteria, such
|
||
|
|
as a maximum age or size. A rollover creates a new backing index and updates the
|
||
|
|
data stream's list of backing indices. This new index then becomes the stream's
|
||
|
|
new write index. See <<rollover-data-stream-ex>>.
|
||
|
|
|
||
|
|
Backing indices are generated with the naming convention
|
||
|
|
`.ds-<data-stream-name>-zzzzzz`, where `zzzzzz` is the six-digit, zero-padded
|
||
|
|
generation of the data stream. For example, a data stream named
|
||
|
|
`web-server-logs` with a generation of 34 would have a write index named
|
||
|
|
`.ds-web-server-logs-000034`. Data streams may have backing indices that do not
|
||
|
|
conform to this naming convention if operations such as
|
||
|
|
<<indices-shrink-index,shrink>> have been performed on them.
|
||
|
|
|
||
|
|
[discrete]
|
||
|
|
[[data-streams-apis]]
|
||
|
|
== Data stream APIs
|
||
|
|
|
||
|
|
The following APIs are available for managing data streams:
|
||
|
|
|
||
|
|
* To get information about data streams, use the <<indices-get-data-stream, get data stream API>>.
|
||
|
|
* To delete data streams, use the <<indices-delete-data-stream, delete data stream API>>.
|
||
|
|
* To manually create a data stream, use the <<indices-create-data-stream, create data stream API>>.
|
||
|
|
|
||
|
|
[discrete]
|
||
|
|
[[data-streams-toc]]
|
||
|
|
== In this section
|
||
|
|
|
||
|
|
* <<set-up-a-data-stream>>
|
||
|
|
* <<use-a-data-stream>>
|
||
|
|
--
|
||
|
|
|
||
|
|
include::set-up-a-data-stream.asciidoc[]
|
||
|
|
include::use-a-data-stream.asciidoc[]
|