342 lines
9.7 KiB
Plaintext
342 lines
9.7 KiB
Plaintext
[role="xpack"]
|
|
[[set-up-lifecycle-policy]]
|
|
== Configure a lifecycle policy
|
|
|
|
For {ilm-init} to manage an index, a valid policy
|
|
must be specified in the `index.lifecycle.name` index setting.
|
|
|
|
To configure a lifecycle policy for <<index-rollover, rolling indices>>,
|
|
you create the policy and add it to the <<index-templates, index template>>.
|
|
|
|
To use a policy to manage an index that doesn't roll over,
|
|
you can specify a lifecycle policy when you create the index,
|
|
or apply a policy directly to an existing index.
|
|
|
|
{ilm-init} policies are stored in the global cluster state and can be included in snapshots
|
|
by setting `include_global_state` to `true` when you <<snapshots-take-snapshot, take the snapshot>>.
|
|
When the snapshot is restored, all of the policies in the global state are restored and
|
|
any local policies with the same names are overwritten.
|
|
|
|
IMPORTANT: When you enable {ilm} for {beats} or the {ls} {es} output plugin,
|
|
the necessary policies and configuration changes are applied automatically.
|
|
You can modify the default policies, but you do not need to explicitly configure a policy or
|
|
bootstrap an initial index.
|
|
|
|
[discrete]
|
|
[[ilm-create-policy]]
|
|
=== Create lifecycle policy
|
|
|
|
To create a lifecycle policy from {kib}, open the menu and go to *Stack
|
|
Management > Index Lifecycle Policies*. Click *Create policy*.
|
|
|
|
[role="screenshot"]
|
|
image:images/ilm/create-policy.png[Create policy page]
|
|
|
|
You specify the lifecycle phases for the policy and the actions to perform in each phase.
|
|
|
|
The <<ilm-put-lifecycle,create or update policy>> API is invoked to add the
|
|
policy to the {es} cluster.
|
|
|
|
.API example
|
|
[%collapsible]
|
|
====
|
|
[source,console]
|
|
------------------------
|
|
PUT _ilm/policy/my_policy
|
|
{
|
|
"policy": {
|
|
"phases": {
|
|
"hot": {
|
|
"actions": {
|
|
"rollover": {
|
|
"max_primary_shard_size": "25GB" <1>
|
|
}
|
|
}
|
|
},
|
|
"delete": {
|
|
"min_age": "30d",
|
|
"actions": {
|
|
"delete": {} <2>
|
|
}
|
|
}
|
|
}
|
|
}
|
|
}
|
|
------------------------
|
|
|
|
<1> Roll over the index when it reaches 25GB in size
|
|
<2> Delete the index 30 days after rollover
|
|
====
|
|
|
|
IMPORTANT: The rollover action implicitly always rolls over a data stream or alias if one or more shards contain
|
|
200000000 or more documents. Normally a shard will reach 25GB long before it reaches 200M documents,
|
|
but this isn't the case for space efficient data sets. Search performance will very likely suffer
|
|
if a shard contains more than 200M documents. This is the reason of the builtin limit.
|
|
|
|
[discrete]
|
|
[[apply-policy-template]]
|
|
=== Apply lifecycle policy with an index template
|
|
|
|
To use a policy that triggers the rollover action,
|
|
you need to configure the policy in the index template used to create each new index.
|
|
You specify the name of the policy and the alias used to reference the rolling indices.
|
|
|
|
You can use the {kib} Create template wizard to create a template. To access the
|
|
wizard, open the menu and go to *Stack Management > Index Management*. In the
|
|
*Index Templates* tab, click *Create template*.
|
|
|
|
[role="screenshot"]
|
|
image:images/ilm/create-template-wizard-my_template.png[Create template page]
|
|
|
|
The wizard invokes the <<indices-put-template,create or update index template
|
|
API>> to add templates to a cluster.
|
|
|
|
.API example
|
|
[%collapsible]
|
|
====
|
|
[source,console]
|
|
-----------------------
|
|
PUT _index_template/my_template
|
|
{
|
|
"index_patterns": ["test-*"], <1>
|
|
"template": {
|
|
"settings": {
|
|
"number_of_shards": 1,
|
|
"number_of_replicas": 1,
|
|
"index.lifecycle.name": "my_policy", <2>
|
|
"index.lifecycle.rollover_alias": "test-alias" <3>
|
|
}
|
|
}
|
|
}
|
|
-----------------------
|
|
|
|
<1> Use this template for all new indices whose names begin with `test-`
|
|
<2> Apply `my_policy` to new indices created with this template
|
|
<3> Define an index alias for referencing indices managed by `my_policy`
|
|
====
|
|
//////////////////////////
|
|
|
|
[source,console]
|
|
--------------------------------------------------
|
|
DELETE _index_template/my_template
|
|
--------------------------------------------------
|
|
// TEST[continued]
|
|
|
|
//////////////////////////
|
|
|
|
[discrete]
|
|
[[create-initial-index]]
|
|
==== Create an initial managed index
|
|
|
|
When you set up policies for your own rolling indices, you need to manually create the first index
|
|
managed by a policy and designate it as the write index.
|
|
|
|
IMPORTANT: When you enable {ilm} for {beats} or the {ls} {es} output plugin,
|
|
the necessary policies and configuration changes are applied automatically.
|
|
You can modify the default policies, but you do not need to explicitly configure a policy or
|
|
bootstrap an initial index.
|
|
|
|
The name of the index must match the pattern defined in the index template and end with a number.
|
|
This number is incremented to generate the name of indices created by the rollover action.
|
|
|
|
For example, the following request creates the `test-00001` index.
|
|
Because it matches the index pattern specified in `my_template`,
|
|
{es} automatically applies the settings from that template.
|
|
|
|
[source,console]
|
|
-----------------------
|
|
PUT test-000001
|
|
{
|
|
"aliases": {
|
|
"test-alias":{
|
|
"is_write_index": true <1>
|
|
}
|
|
}
|
|
}
|
|
-----------------------
|
|
|
|
<1> Set this initial index to be the write index for this alias.
|
|
|
|
Now you can start indexing data to the rollover alias specified in the lifecycle policy.
|
|
With the sample `my_policy` policy, the rollover action is triggered once the initial
|
|
index exceeds 25GB.
|
|
{ilm-init} then creates a new index that becomes the write index for the `test-alias`.
|
|
|
|
[discrete]
|
|
[[apply-policy-manually]]
|
|
=== Apply lifecycle policy manually
|
|
|
|
You can specify a policy when you create an index or
|
|
apply a policy to an existing index through {kib} Management or
|
|
the <<indices-update-settings, update settings API>>.
|
|
When you apply a policy, {ilm-init} immediately starts managing the index.
|
|
|
|
IMPORTANT: Do not manually apply a policy that uses the rollover action.
|
|
Policies that use rollover must be applied by the <<apply-policy-template, index template>>.
|
|
Otherwise, the policy is not carried forward when the rollover action creates a new index.
|
|
|
|
The `index.lifecycle.name` setting specifies an index's policy.
|
|
|
|
.API example
|
|
[%collapsible]
|
|
====
|
|
[source,console]
|
|
-----------------------
|
|
PUT test-index
|
|
{
|
|
"settings": {
|
|
"number_of_shards": 1,
|
|
"number_of_replicas": 1,
|
|
"index.lifecycle.name": "my_policy" <1>
|
|
}
|
|
}
|
|
-----------------------
|
|
<1> Sets the lifecycle policy for the index.
|
|
====
|
|
|
|
[discrete]
|
|
[[apply-policy-multiple]]
|
|
==== Apply a policy to multiple indices
|
|
|
|
You can apply the same policy to multiple indices by using wildcards in the index name
|
|
when you call the <<indices-update-settings,update settings>> API.
|
|
|
|
WARNING: Be careful that you don't inadvertently match indices that you don't want to modify.
|
|
|
|
//////////////////////////
|
|
[source,console]
|
|
-----------------------
|
|
PUT _index_template/mylogs_template
|
|
{
|
|
"index_patterns": [
|
|
"mylogs-*"
|
|
],
|
|
"template": {
|
|
"settings": {
|
|
"number_of_shards": 1,
|
|
"number_of_replicas": 1
|
|
},
|
|
"mappings": {
|
|
"properties": {
|
|
"message": {
|
|
"type": "text"
|
|
},
|
|
"@timestamp": {
|
|
"type": "date"
|
|
}
|
|
}
|
|
}
|
|
}
|
|
}
|
|
-----------------------
|
|
|
|
[source,console]
|
|
-----------------------
|
|
POST mylogs-pre-ilm-2019.06.24/_doc
|
|
{
|
|
"@timestamp": "2019-06-24T10:34:00",
|
|
"message": "this is one log message"
|
|
}
|
|
-----------------------
|
|
// TEST[continued]
|
|
|
|
[source,console]
|
|
-----------------------
|
|
POST mylogs-pre-ilm-2019.06.25/_doc
|
|
{
|
|
"@timestamp": "2019-06-25T17:42:00",
|
|
"message": "this is another log message"
|
|
}
|
|
-----------------------
|
|
// TEST[continued]
|
|
|
|
[source,console]
|
|
--------------------------------------------------
|
|
DELETE _index_template/mylogs_template
|
|
--------------------------------------------------
|
|
// TEST[continued]
|
|
|
|
//////////////////////////
|
|
|
|
[source,console]
|
|
-----------------------
|
|
PUT mylogs-pre-ilm*/_settings <1>
|
|
{
|
|
"index": {
|
|
"lifecycle": {
|
|
"name": "mylogs_policy_existing"
|
|
}
|
|
}
|
|
}
|
|
-----------------------
|
|
// TEST[continued]
|
|
|
|
<1> Updates all indices with names that start with `mylogs-pre-ilm`
|
|
|
|
[discrete]
|
|
[[switch-lifecycle-policies]]
|
|
==== Switch lifecycle policies
|
|
|
|
To switch an index's lifecycle policy, follow these steps:
|
|
|
|
. Remove the existing policy using the <<ilm-remove-policy,remove policy API>>.
|
|
Target a data stream or alias to remove the policies of all its indices.
|
|
+
|
|
[source,console]
|
|
----
|
|
POST logs-my_app-default/_ilm/remove
|
|
----
|
|
// TEST[continued]
|
|
// TEST[s/^/PUT _data_stream\/logs-my_app-default\n/]
|
|
|
|
. The remove policy API removes all {ilm-init} metadata from the index and
|
|
doesn't consider the index's lifecycle status. This can leave indices in an
|
|
undesired state.
|
|
+
|
|
--
|
|
For example, the <<ilm-forcemerge,`forcemerge`>> action temporarily closes an
|
|
index before reopening it. Removing an index's {ilm-init} policy during a
|
|
`forcemerge` can leave the index closed indefinitely.
|
|
|
|
After policy removal, use the <<indices-get-index,get index API>> to check an
|
|
index's state . Target a data stream or alias to get the state of all its
|
|
indices.
|
|
|
|
[source,console]
|
|
----
|
|
GET logs-my_app-default
|
|
----
|
|
// TEST[continued]
|
|
|
|
You can then change the index as needed. For example, you can re-open any
|
|
closed indices using the <<indices-open-close,open index API>>.
|
|
|
|
[source,console]
|
|
----
|
|
POST logs-my_app-default/_open
|
|
----
|
|
// TEST[continued]
|
|
--
|
|
|
|
. Assign a new policy using the <<indices-update-settings,update settings API>>.
|
|
Target a data stream or alias to assign a policy to all its indices.
|
|
+
|
|
--
|
|
WARNING: Don't assign a new policy without first removing the existing policy.
|
|
This can cause <<ilm-phase-execution,phase execution>> to silently fail.
|
|
|
|
[source,console]
|
|
----
|
|
PUT logs-my_app-default/_settings
|
|
{
|
|
"index": {
|
|
"lifecycle": {
|
|
"name": "new-lifecycle-policy"
|
|
}
|
|
}
|
|
}
|
|
----
|
|
// TEST[continued]
|
|
// TEST[s/new-lifecycle-policy/mylogs_policy_existing/]
|
|
--
|