Commit Graph

20 Commits

Author SHA1 Message Date
Mary Gouseti 91f4023e27
Expose global retention settings via data stream lifecycle API (#112210)
In this PR we expose the global retention via the `GET
_data_stream/{target}/_lifecycle` API.

Since the global retention is a main feature of the data stream
lifecycle we chose to expose it by default.

```
GET /_data_stream/my-data-stream/_lifecycle
{
 "global_retention": {
      "default_retention": "7d",
      "max_retention": "365d"
  }, 
  "data_streams": [...]
}
```
2024-09-02 18:40:08 +10:00
Mary Gouseti ed60470518
Display effective retention in the relevant data stream APIs (#112019) 2024-08-22 17:42:49 +03:00
Stef Nestor c1019d4c5d
(Doc+) Link API doc to parent object - part1 (#111951)
* (Doc+) Link API to parent Doc part1

---------

Co-authored-by: shainaraskas <shaina.raskas@elastic.co>
Co-authored-by: shainaraskas <58563081+shainaraskas@users.noreply.github.com>
2024-08-20 14:58:18 -06:00
Liam Thompson cadb3f9325
Remove typo put-lifecycle.asciidoc (#110875) (#110918) 2024-07-17 08:12:52 +01:00
Nick Tindall 3ecdd77e97
[DOCS] Align docs to implementation for timeout parameters (#108593)
* [DOCS] Fix documentation for timeout-related parameters

Closes #108224
2024-05-16 13:05:39 +10:00
Lee Hinman 87928b2303
GA data stream lifecycles in documentation (#107582)
This commit removes the warning about techincal preview for the data stream lifecycle documentation pages.
2024-04-18 09:09:44 -06:00
Mary Gouseti 732c7c4c30
[DSL] Remove REST APIs for global retention (#107565) 2024-04-17 21:36:26 +03:00
Liam Thompson 33a71e3289
[DOCS] Refactor book-scoped variables in `docs/reference/index.asciidoc` (#107413)
* Remove `es-test-dir` book-scoped variable

* Remove `plugins-examples-dir` book-scoped variable

* Remove `:dependencies-dir:` and `:xes-repo-dir:` book-scoped variables

- In `index.asciidoc`, two variables (`:dependencies-dir:` and `:xes-repo-dir:`) were removed.
- In `sql/index.asciidoc`, the `:sql-tests:` path was updated to fuller path
- In `esql/index.asciidoc`, the `:esql-tests:` path was updated idem

* Replace `es-repo-dir` with `es-ref-dir`

* Move `:include-xpack: true` to few files that use it, remove from index.asciidoc
2024-04-17 14:37:07 +02:00
Mary Gouseti 2122da31cd
[DSL] Introduce data stream global retention - Part 3 (#105682)
In this PR we introduce the API that will expose the global retention configuration and will allow users to take advantage of it.

These APIs are protected by the dedicated introduced privileges:

`manage_data_stream_global_retention` or higher, which allows all operations on the global retention configuration
`monitor_data_stream_retention` or higher, which allows the retrieval of the global retention configuration.

This PR is the final PR that makes the global retention available for our users.
2024-03-28 10:40:33 +02:00
Mary Gouseti 2988799079
[DSL Global Retention] Use data stream global retention metadata (#106221) 2024-03-20 20:27:08 +02:00
Andrei Dan 17811280c2
[DOCS] DSL downsampling docs (#103148) 2023-12-08 06:52:18 -05:00
Mary Gouseti 5a3409b7c5
ES-6566: [DSL] Introduce new endpoint to expose data stream lifecycle stats (#101845) 2023-11-20 10:38:41 +02:00
Andrei Dan 7b436bae2c
[DOCS] DSL: More visible tech preview tags (#101313) 2023-10-26 12:06:15 +01:00
Andrei Dan 74ea04fb2d
[DOCS] document tail merging and create tutorial for migrating to DSL (#101117)
This documents tail merging, the enabled flag, and
adds a tutorial to migrate a data stream from ILM to DSL.
2023-10-25 11:12:36 +01:00
Andrei Dan 839afdc331
Promote the Data stream lifecycle feature to Technical Preview (#100187)
This releases the Data stream lifecycle feature as a
Technical Preview feature.

Data stream lifecycle, albeit in technical preview, will allow data streams
to take advantage of a native simplified and resilient lifecycle implementation.
2023-10-03 17:12:35 +01:00
Mary Gouseti b9b818e28e
Allow explain data stream lifecycle to accept a data stream. (#98811)
Currently the `GET target/_lifecycle/explain` API only works for
indices. In this PR we extend this behaviour to allow the target to be a
data stream so we can get the overview lifecycle status for all the
backing indices of a data stream.
2023-08-24 06:29:09 -04:00
Andrei Dan 01ed7de99f
GA the data stream lifecycle (#98644)
This makes the data stream lifecycle generally available. This will allow
data streams to take advantage of a native simplified and resilient
lifecycle implementation.
2023-08-21 17:28:54 +01:00
Mary Gouseti e71ea6e6d7
Add data stream lifecycle by default (#97823)
In this PR we enable all new data streams to be managed by the data
stream lifecycle by default. This is implemented by adding an empty
`lifecycle: {}` upon new data stream creation. 

Opting out is represented by a the `enabled` flag:

```
{
  "lifecycle": {
    "enabled": false
  }
}
```

This change has the following implications on when is an index managed
and by which feature:

| Parent data stream lifecycle| ILM| `prefer_ilm`|Managed by|
|----------------------------|----|----------------|-| | default | yes|
true| ILM| | default | yes| false| data stream lifecycle| |default |
no|true/false|data stream lifecycle| |opt-out or
missing|yes|true/false|ILM| |opt-out or missing|no|true/false|unmanaged|

Data streams that have been created before the data stream lifecycle is
enabled will not have the default lifecycle.

Next steps: - We need to document this when the feature will be GA
(https://github.com/elastic/elasticsearch/issues/97973).
2023-08-11 06:28:37 -04:00
Mary Gouseti a432313ff3
Data stream lifecycle class names (#97381) 2023-07-05 12:28:32 +03:00
Mary Gouseti 1abd51b167
Start with data stream lifecycle documentation (#95326) 2023-06-28 16:18:05 +03:00