2019-03-06 17:45:13 +08:00
2017-06-20 09:01:52 +08:00
[role="xpack"]
2017-04-18 02:53:31 +08:00
[[ml-settings]]
2018-09-29 00:41:14 +08:00
=== Machine learning settings in Elasticsearch
2017-08-12 01:00:35 +08:00
++++
2018-09-29 00:41:14 +08:00
<titleabbrev>Machine learning settings</titleabbrev>
2017-08-12 01:00:35 +08:00
++++
2020-07-03 02:13:06 +08:00
[[ml-settings-description]]
// tag::ml-settings-description-tag[]
2017-04-28 01:51:48 +08:00
You do not need to configure any settings to use {ml}. It is enabled by default.
2017-04-18 02:53:31 +08:00
2020-08-24 22:29:25 +08:00
IMPORTANT: {ml-cap} uses SSE4.2 instructions, so it works only on machines whose
CPUs {wikipedia}/SSE4#Supporting_CPUs[support] SSE4.2. If you run {es} on older
hardware, you must disable {ml} (by setting `xpack.ml.enabled` to `false`).
2018-09-29 00:41:14 +08:00
2020-07-03 02:13:06 +08:00
// end::ml-settings-description-tag[]
2020-07-23 23:48:22 +08:00
[discrete]
2017-04-18 02:53:31 +08:00
[[general-ml-settings]]
2018-09-29 00:41:14 +08:00
==== General machine learning settings
2017-04-18 02:53:31 +08:00
2020-08-18 17:23:13 +08:00
`node.roles: [ ml ]`::
2020-08-24 22:29:25 +08:00
(<<static-cluster-setting,Static>>) Set `node.roles` to contain `ml` to identify
2020-12-23 02:11:09 +08:00
the node as a _{ml} node_. If you want to run {ml} jobs, there must be at least
one {ml} node in your cluster.
2017-10-26 00:00:53 +08:00
+
2021-01-20 06:19:37 +08:00
If you set `node.roles`, you must explicitly specify all the required roles for
the node. To learn more, refer to <<modules-node>>.
2017-10-26 00:00:53 +08:00
+
2020-12-23 02:11:09 +08:00
[IMPORTANT]
====
* On dedicated coordinating nodes or dedicated master nodes, do not set
2020-08-18 17:23:13 +08:00
the `ml` role.
2020-12-23 02:11:09 +08:00
* It is strongly recommended that dedicated {ml} nodes also have the `remote_cluster_client` role; otherwise, {ccs} fails when used in {ml} jobs or {dfeeds}. See <<remote-node>>.
====
2017-10-26 00:00:53 +08:00
2017-04-18 02:53:31 +08:00
`xpack.ml.enabled`::
2020-08-24 22:29:25 +08:00
(<<static-cluster-setting,Static>>) Set to `true` (default) to enable {ml} APIs
on the node.
2017-04-18 02:53:31 +08:00
+
2020-04-03 06:34:37 +08:00
If set to `false`, the {ml} APIs are disabled on the node. Therefore the node
cannot open jobs, start {dfeeds}, or receive transport (internal) communication
requests related to {ml} APIs. If the node is a coordinating node, {ml} requests
from clients (including {kib}) also fail. For more information about disabling
{ml} in specific {kib} instances, see
{kibana-ref}/ml-settings-kb.html[{kib} {ml} settings].
2017-04-18 02:53:31 +08:00
+
2020-04-03 06:34:37 +08:00
IMPORTANT: If you want to use {ml-features} in your cluster, it is recommended
2020-10-06 22:01:40 +08:00
that you set `xpack.ml.enabled` to `true` on all nodes. This is the default
behavior. At a minimum, it must be enabled on all master-eligible nodes. If you
want to use {ml-features} in clients or {kib}, it must also be enabled on all
2020-08-18 17:23:13 +08:00
coordinating nodes.
2017-04-18 02:53:31 +08:00
2019-11-20 05:07:37 +08:00
`xpack.ml.inference_model.cache_size`::
2020-08-24 22:29:25 +08:00
(<<static-cluster-setting,Static>>) The maximum inference cache size allowed.
The inference cache exists in the JVM heap on each ingest node. The cache
affords faster processing times for the `inference` processor. The value can be
a static byte sized value (i.e. "2gb") or a percentage of total allocated heap.
The default is "40%". See also <<model-inference-circuit-breaker>>.
2019-11-20 05:07:37 +08:00
2020-07-03 02:13:06 +08:00
[[xpack-interference-model-ttl]]
// tag::interference-model-ttl-tag[]
`xpack.ml.inference_model.time_to_live` {ess-icon}::
2020-08-24 22:29:25 +08:00
(<<static-cluster-setting,Static>>) The time to live (TTL) for models in the
inference model cache. The TTL is calculated from last access. The `inference`
processor attempts to load the model from cache. If the `inference` processor
does not receive any documents for the duration of the TTL, the referenced model
is flagged for eviction from the cache. If a document is processed later, the
model is again loaded into the cache. Defaults to `5m`.
2020-07-03 02:13:06 +08:00
// end::interference-model-ttl-tag[]
2019-11-20 05:07:37 +08:00
2020-08-24 22:29:25 +08:00
`xpack.ml.max_inference_processors`::
(<<cluster-update-settings,Dynamic>>) The total number of `inference` type
processors allowed across all ingest pipelines. Once the limit is reached,
adding an `inference` processor to a pipeline is disallowed. Defaults to `50`.
`xpack.ml.max_machine_memory_percent`::
(<<cluster-update-settings,Dynamic>>) The maximum percentage of the machine's
memory that {ml} may use for running analytics processes. (These processes are
separate to the {es} JVM.) Defaults to `30` percent. The limit is based on the
total memory of the machine, not current free memory. Jobs are not allocated to
a node if doing so would cause the estimated memory use of {ml} jobs to exceed
2021-03-04 00:20:39 +08:00
the limit. When the {operator-feature} is enabled, this setting can be updated
only by operator users.
2020-08-24 22:29:25 +08:00
`xpack.ml.max_model_memory_limit`::
(<<cluster-update-settings,Dynamic>>) The maximum `model_memory_limit` property
value that can be set for any job on this node. If you try to create a job with
a `model_memory_limit` property value that is greater than this setting value,
an error occurs. Existing jobs are not affected when you update this setting.
For more information about the `model_memory_limit` property, see
<<put-analysislimits>>.
2017-12-16 03:19:11 +08:00
2020-02-03 21:33:02 +08:00
[[xpack.ml.max_open_jobs]]
2020-08-24 22:29:25 +08:00
`xpack.ml.max_open_jobs`::
2020-10-06 22:01:40 +08:00
(<<cluster-update-settings,Dynamic>>) The maximum number of jobs that can run
2020-08-24 22:29:25 +08:00
simultaneously on a node. Defaults to `20`. In this context, jobs include both
{anomaly-jobs} and {dfanalytics-jobs}. The maximum number of jobs is also
constrained by memory usage. Thus if the estimated memory usage of the jobs
would be higher than allowed, fewer jobs will run on a node. Prior to version
7.1, this setting was a per-node non-dynamic setting. It became a cluster-wide
dynamic setting in version 7.1. As a result, changes to its value after node
startup are used only after every node in the cluster is running version 7.1 or
higher. The maximum permitted value is `512`.
2020-10-06 22:01:40 +08:00
`xpack.ml.nightly_maintenance_requests_per_second`::
2021-03-02 09:14:21 +08:00
(<<cluster-update-settings,Dynamic>>) The rate at which the nightly maintenance
task deletes expired model snapshots and results. The setting is a proxy to the
<<docs-delete-by-query-throttle,requests_per_second>> parameter used in the
2021-03-04 00:20:39 +08:00
delete by query requests and controls throttling. When the {operator-feature} is
enabled, this setting can be updated only by operator users. Valid values must
be greater than `0.0` or equal to `-1.0` where `-1.0` means a default value is
used. Defaults to `-1.0`
2020-10-06 22:01:40 +08:00
2020-08-24 22:29:25 +08:00
`xpack.ml.node_concurrent_job_allocations`::
(<<cluster-update-settings,Dynamic>>) The maximum number of jobs that can
concurrently be in the `opening` state on each node. Typically, jobs spend a
small amount of time in this state before they move to `open` state. Jobs that
must restore large models when they are opening spend more time in the `opening`
2021-03-04 00:20:39 +08:00
state. When the {operator-feature} is enabled, this setting can be updated only
by operator users. Defaults to `2`.
2018-09-29 00:41:14 +08:00
2020-07-23 23:48:22 +08:00
[discrete]
2018-09-29 00:41:14 +08:00
[[advanced-ml-settings]]
==== Advanced machine learning settings
2020-07-03 02:13:06 +08:00
These settings are for advanced use cases; the default values are generally
2018-09-29 00:41:14 +08:00
sufficient:
2020-08-24 22:29:25 +08:00
`xpack.ml.enable_config_migration`::
2021-03-04 00:20:39 +08:00
(<<cluster-update-settings,Dynamic>>) Reserved. When the {operator-feature} is
enabled, this setting can be updated only by operator users.
2019-01-04 01:26:57 +08:00
2020-08-24 22:29:25 +08:00
`xpack.ml.max_anomaly_records`::
(<<cluster-update-settings,Dynamic>>) The maximum number of records that are
output per bucket. The default value is `500`.
2018-09-29 00:41:14 +08:00
2020-08-24 22:29:25 +08:00
`xpack.ml.max_lazy_ml_nodes`::
(<<cluster-update-settings,Dynamic>>) The number of lazily spun up {ml} nodes.
Useful in situations where {ml} nodes are not desired until the first {ml} job
opens. It defaults to `0` and has a maximum acceptable value of `3`. If the
current number of {ml} nodes is greater than or equal to this setting, it is
2018-10-19 05:11:36 +08:00
assumed that there are no more lazy nodes available as the desired number
2020-08-24 22:29:25 +08:00
of nodes have already been provisioned. If a job is opened and this setting has
a value greater than zero and there are no nodes that can accept the job, the
job stays in the `OPENING` state until a new {ml} node is added to the cluster
2021-03-04 00:20:39 +08:00
and the job is assigned to run on that node. When the {operator-feature} is
enabled, this setting can be updated only by operator users.
2018-10-19 05:11:36 +08:00
+
2020-08-24 22:29:25 +08:00
IMPORTANT: This setting assumes some external process is capable of adding {ml}
nodes to the cluster. This setting is only useful when used in conjunction with
2018-10-19 05:11:36 +08:00
such an external process.
2019-06-25 23:36:02 +08:00
2020-12-17 04:55:34 +08:00
`xpack.ml.max_ml_node_size`::
(<<cluster-update-settings,Dynamic>>)
The maximum node size for {ml} nodes in a deployment that supports automatic
cluster scaling. Defaults to `0b`, which means this value is ignored. If you set
it to the maximum possible size of future {ml} nodes, when a {ml} job is
assigned to a lazy node it can check (and fail quickly) when scaling cannot
2021-03-04 00:20:39 +08:00
support the size of the job. When the {operator-feature} is enabled, this
setting can be updated only by operator users.
`xpack.ml.persist_results_max_retries`::
(<<cluster-update-settings,Dynamic>>) The maximum number of times to retry bulk
indexing requests that fail while processing {ml} results. If the limit is
reached, the {ml} job stops processing data and its status is `failed`. When the
{operator-feature} is enabled, this setting can be updated only by operator
users. Defaults to `20`. The maximum value for this setting is `50`.
2020-12-17 04:55:34 +08:00
2020-08-24 22:29:25 +08:00
`xpack.ml.process_connect_timeout`::
(<<cluster-update-settings,Dynamic>>) The connection timeout for {ml} processes
that run separately from the {es} JVM. Defaults to `10s`. Some {ml} processing
is done by processes that run separately to the {es} JVM. When such processes
are started they must connect to the {es} JVM. If such a process does not
connect within the time period specified by this setting then the process is
2021-03-04 00:20:39 +08:00
assumed to have failed. When the {operator-feature} is enabled, this setting can
be updated only by operator users. Defaults to `10s`. The minimum value for this
setting is `5s`.
2020-06-09 00:55:25 +08:00
2020-10-22 00:50:55 +08:00
xpack.ml.use_auto_machine_memory_percent::
2020-12-17 00:32:57 +08:00
(<<cluster-update-settings,Dynamic>>) If this setting is `true`, the
2020-10-22 00:50:55 +08:00
`xpack.ml.max_machine_memory_percent` setting is ignored. Instead, the maximum
percentage of the machine's memory that can be used for running {ml} analytics
processes is calculated automatically and takes into account the total node size
and the size of the JVM on the node. The default value is `false`. If this
setting differs between nodes, the value on the current master node is heeded.
2021-03-04 00:20:39 +08:00
When the {operator-feature} is enabled, this setting can be updated only by
operator users.
2020-10-22 00:50:55 +08:00
+
TIP: If you do not have dedicated {ml} nodes (that is to say, the node has
multiple roles), do not enable this setting. Its calculations assume that {ml}
analytics are the main purpose of the node.
+
IMPORTANT: The calculation assumes that dedicated {ml} nodes have at least
`256MB` memory reserved outside of the JVM. If you have tiny {ml}
nodes in your cluster, you shouldn't use this setting.
2020-07-23 23:48:22 +08:00
[discrete]
2020-06-09 00:55:25 +08:00
[[model-inference-circuit-breaker]]
==== {ml-cap} circuit breaker settings
2020-08-24 22:29:25 +08:00
`breaker.model_inference.limit`::
(<<cluster-update-settings,Dynamic>>) Limit for the model inference breaker,
which defaults to 50% of the JVM heap. If the parent circuit breaker is less
than 50% of the JVM heap, it is bound to that limit instead. See
<<circuit-breaker>>.
2020-07-04 02:06:42 +08:00
2020-08-24 22:29:25 +08:00
`breaker.model_inference.overhead`::
(<<cluster-update-settings,Dynamic>>) A constant that all accounting estimations
are multiplied by to determine a final estimation. Defaults to 1. See
<<circuit-breaker>>.
2020-07-04 02:06:42 +08:00
`breaker.model_inference.type`::
2020-08-24 22:29:25 +08:00
(<<static-cluster-setting,Static>>) The underlying type of the circuit breaker.
There are two valid options: `noop` and `memory`. `noop` means the circuit
breaker does nothing to prevent too much memory usage. `memory` means the
circuit breaker tracks the memory used by inference models and can potentially
break and prevent `OutOfMemory` errors. The default is `memory`.
