Commit Graph

33 Commits

Author SHA1 Message Date
Lisa Cawley 9ab6808206
[DOCS] Clean up xpack.ml.enabled details (#74573) 2021-06-30 09:34:46 -07:00
Benjamin Trent 2ce4d175f0
[ML] increase the default value of xpack.ml.max_open_jobs from 20 to 512 for autoscaling improvements (#72487)
This commit increases the xpack.ml.max_open_jobs from 20 to 512. Additionally, it ignores nodes that cannot provide an accurate view into their native memory.

If a node does not have a view into its native memory, we ignore it for assignment.

This effectively fixes a bug with autoscaling. Autoscaling relies on jobs with adequate memory to assign jobs to nodes. If that is hampered by the xpack.ml.max_open_jobs scaling decisions are hampered.
2021-04-30 07:55:57 -04:00
István Zoltán Szabó 11c22ce23a
[DOCS] Adds backticks to xpack.ml.use_auto_machine_memory_percent. (#71814) 2021-04-19 11:12:27 +02:00
Lisa Cawley 2caba7b11f
[DOCS] Edits machine learning settings (#69947)
Co-authored-by: David Roberts <dave.roberts@elastic.co>
2021-03-09 10:59:12 -08:00
Lisa Cawley 8c0439f803
[DOCS] Add operator privileges to ML settings (#69766) 2021-03-03 08:20:39 -08:00
Lisa Cawley 4c39136837
[DOCS] Fix link in machine learning nightly maintenance setting (#69760) 2021-03-01 17:14:21 -08:00
Lisa Cawley 4d1abd1494
[DOCS] Clarifies default ML and transform node settings (#67671) 2021-01-19 14:19:37 -08:00
Lisa Cawley 6b463a7b7a
[DOCS] Clarify use of CCS on ML nodes (#66616)
Co-authored-by: David Roberts <dave.roberts@elastic.co>
2020-12-22 10:11:09 -08:00
Lisa Cawley 709068da7c
[DOCS] Adds xpack.ml.max_ml_node_size (#66285) 2020-12-16 12:55:34 -08:00
Lisa Cawley 35cc6bf8a0
[DOCS] Changes static ML setting to dynamic (#66286) 2020-12-16 08:32:57 -08:00
Benjamin Trent 165e063b50
[ML] add new setting xpack.ml.use_auto_machine_memory_percent for auto calculating native memory percentage allowed for machine learning jobs (#63887)
When running ML, sometimes it is best to automatically adjust the
memory allotted for machine learning based on the nodesize
and how much space is given to the JVM

This commit adds a new static setting xpack.ml.use_auto_machine_memory_percent for
allowing this dynamic calculation. The old setting remains as a backup
just in case the limit cannot be automatically determined due to
lack of information.

Closes #63795
2020-10-21 12:50:55 -04:00
David Kyle 6784e826a8
[ML] Audit message when nightly maintenance times out (#63252)
During deletion of old ml data set the delete by query timeout to 8 hours and 
audit a job message when the nightly maintenance task times out.
2020-10-06 15:01:40 +01:00
Lisa Cawley e12d6f168c
[DOCS] Document static machine learning settings (#61382) 2020-08-24 07:29:25 -07:00
István Zoltán Szabó f8e572fb54
[DOCS] Replaces ml.node with node.roles: [ ml ] in ML settings (#61017) 2020-08-18 11:23:13 +02:00
James Rodewig a94e5cb7c4
[DOCS] Replace Wikipedia links with attribute (#61171) 2020-08-17 09:44:24 -04:00
James Rodewig 2774cd6938
[DOCS] Swap `[float]` for `[discrete]` (#60124)
Changes instances of `[float]` in our docs for `[discrete]`.

Asciidoctor prefers the `[discrete]` tag for floating headings:
https://asciidoctor.org/docs/asciidoc-asciidoctor-diffs/#blocks
2020-07-23 11:48:22 -04:00
Lisa Cawley 1dd071049f
[DOCS] Deprecates node.ml (#59024) 2020-07-03 11:08:31 -07:00
Lisa Cawley bff59765a0
[DOCS] Edits ML circuit breaker settings (#59026) 2020-07-03 11:06:42 -07:00
Adam Locke 3a1258fe97
[DOCS] Add supported ESS settings to ES docs (#57953)
* Adding ESS icons to supported ES settings.

* Adding new file for supported ESS settings.

* Adding supported ESS settings for HTTP and disk-based shard allocation.

* Adding more supported settings for ESS.

* Adding descriptions for each Cloud section, plus additional settings.

* Adding new warehouse file for Cloud, plus additional settings.

* Adding node settings for Cloud.

* Adding audit settings for Cloud.

* Resolving merge conflict.

* Adding SAML settings (part 1).

* Adding SAML realm encryption and signing settings.

* Adding SAML SSL settings.

* Adding Kerberos realm settings.

* Adding OpenID Connect Realm settings.

* Adding OpenID Connect SSL settings.

* Resolving leftover Git merge markers.

* Removing Cloud settings page and link to it.

* Add link to mapping source

* Update docs/reference/docs/reindex.asciidoc

* Incorporate edit of HTTP settings

* Remove "cloud" from tag and ID

* Remove "cloud" from tag and update description

* Remove "cloud" from tag and ID

* Change "whitelists" to "specifies"

* Remove "cloud" from end tag

* Removing cloud from IDs and tags.

* Changing link reference to fix build issue.

* Adding index management page for missing settings.

* Removing warehouse file for Cloud and moving settings elsewhere.

* Clarifying true/false usage of http.detailed_errors.enabled.

* Changing underscore to dash in link to fix ci build.
2020-07-02 14:13:06 -04:00
Benjamin Trent 955afe3f81
[ML] add new circuit breaker for inference model caching (#57731)
This adds new plugin level circuit breaker for the ML plugin.

`model_inference` is the circuit breaker qualified name. 

Right now it simply adds to the breaker when the model is loaded (and possibly breaking) and removing from the breaker when the model is unloaded.
2020-06-08 12:55:25 -04:00
Lisa Cawley 5b85013976
[DOCS] Clarify ML and transform settings on coordinating nodes (#54676) 2020-04-02 15:34:37 -07:00
James Rodewig b9fafc7dc6
[DOCS] Document node stats response meta (#51263)
Documents several metadata-related parameters returned by the
`GET _nodes/stats` API.
2020-02-03 08:33:02 -05:00
Lisa Cawley 0f51bc2f72
[DOCS] Move anomaly detection job resource definitions into APIs (#49700)
Co-Authored-By: István Zoltán Szabó <istvan.szabo@elastic.co>
2019-12-06 15:32:07 -08:00
Benjamin Trent 6fbb1effdb
[ML][Inference] document new settings (#49309)
* [ML][Inference] document new settings

* [DOCS] Minor edits
2019-11-19 16:07:37 -05:00
István Zoltán Szabó a6c517a96e
[DOCS] Changes wording to move away from data frame terminology in the ES repo (#47093)
* [DOCS] Changes wording to move away from data frame terminology in the ES repo.
Co-Authored-By: Lisa Cawley <lcawley@elastic.co>
2019-10-01 08:04:06 +02:00
David Roberts 76ad7d8464
[ML] Introduce a setting for the process connect timeout (#43234)
This change introduces a new setting,
xpack.ml.process_connect_timeout, to enable
the timeout for one of the external ML processes
to connect to the ES JVM to be increased.

The timeout may need to be increased if many
processes are being started simultaneously on
the same machine. This is unlikely in clusters
with many ML nodes, as we balance the processes
across the ML nodes, but can happen in clusters
with a single ML node and a high value for
xpack.ml.node_concurrent_job_allocations.
2019-06-25 16:36:02 +01:00
David Roberts 37be0a164f [DOCS] Adding ML-specific prerequisites to setup docs (#42529) 2019-05-24 10:44:51 -07:00
David Roberts 6242beef7a
[ML] Use scaling thread pool and xpack.ml.max_open_jobs cluster-wide dynamic (#39320)
This change does the following:

1. Makes the per-node setting xpack.ml.max_open_jobs
   into a cluster-wide dynamic setting
2. Changes the job node selection to continue to use the
   per-node attributes storing the maximum number of open
   jobs if any node in the cluster is older than 7.1, and
   use the dynamic cluster-wide setting if all nodes are on
   7.1 or later
3. Changes the docs to reflect this
4. Changes the thread pools for native process communication
   from fixed size to scaling, to support the dynamic nature
   of xpack.ml.max_open_jobs
5. Renames the autodetect thread pool to the job comms
   thread pool to make clear that it will be used for other
   types of ML jobs (data frame analytics in particular)

Closes #29809
2019-03-06 09:45:13 +00:00
Lisa Cawley 366b6b5e11
[DOCS] Adds cluster setting to enable/disable config migration (#36948) 2019-01-03 09:26:57 -08:00
lcawl 781130c7ab [DOCS] Fixes formatting of dynamic settings links 2018-11-01 11:52:41 -07:00
Benjamin Trent fa973007b8
[DOCS] Adds new lazy ml node setting (#34600)
* Adding new xpack.ml.max_lazy_ml_nodes setting to docs

* Fixing docs, making it clearer what the setting does

* Adding note about external process need
2018-10-18 16:11:36 -05:00
Lisa Cawley f143e39a71
[DOCS] Adds new dynamic machine learning settings (#34094) 2018-09-28 09:41:14 -07:00
Lisa Cawley 45537c59e5
[DOCS] Moves X-Pack settings to docs folder (#31120) 2018-06-06 10:05:32 -07:00