root
/
elasticsearch
mirror of https://github.com/elastic/elasticsearch.git
1
0
Fork 0
Code Issues Actions 1 Packages Projects Releases Wiki Activity
elasticsearch/docs/reference/ml/anomaly-detection/apis/get-ml-info.asciidoc

Ignoring revisions in .git-blame-ignore-revs. Click here to bypass and see the normal blame view.

129 lines
3.3 KiB
Plaintext
Raw Normal View History

[ML] Add docs for ML info endpoint (#35783) This endpoint was not previously documented as it was not particularly useful to end users. However, since the HLRC will support the endpoint we need some documentation to link to. The purpose of the endpoint is to provide defaults and limits used by ML. These are needed to fully understand configurations that have missing values because the missing value means the default should be used. Relates #35777
2018-11-23 00:23:31 +08:00
[role="xpack"]
[testenv="platinum"]
[[get-ml-info]]
[DOCS] Changes level offset for anomaly detection APIs (#59920)
2020-07-21 03:38:09 +08:00
= Get machine learning info API
[DOCS] Allow attribute substitution in titleabbrevs for Asciidoctor migration (#41574) * [DOCS] Replace attributes in titleabbrevs for Asciidoctor migration * [DOCS] Add [subs="attributes"] so attributes render in Asciidoctor * Revert "[DOCS] Replace attributes in titleabbrevs for Asciidoctor migration" This reverts commit 98f130257a7c71e9f6cddf5157af7886418338d8. * [DOCS] Fix merge conflict
2019-05-01 01:46:13 +08:00
[subs="attributes"]
[ML] Add docs for ML info endpoint (#35783) This endpoint was not previously documented as it was not particularly useful to end users. However, since the HLRC will support the endpoint we need some documentation to link to. The purpose of the endpoint is to provide defaults and limits used by ML. These are needed to fully understand configurations that have missing values because the missing value means the default should be used. Relates #35777
2018-11-23 00:23:31 +08:00
++++
[DOCS] Synchs titles of X-Pack APIs
2018-12-21 02:23:28 +08:00
<titleabbrev>Get {ml} info</titleabbrev>
[ML] Add docs for ML info endpoint (#35783) This endpoint was not previously documented as it was not particularly useful to end users. However, since the HLRC will support the endpoint we need some documentation to link to. The purpose of the endpoint is to provide defaults and limits used by ML. These are needed to fully understand configurations that have missing values because the missing value means the default should be used. Relates #35777
2018-11-23 00:23:31 +08:00
++++
Returns defaults and limits used by machine learning.
[DOCS] Adds anchors and attributes to ML APIs
2019-06-28 00:42:47 +08:00
[[get-ml-info-request]]
[DOCS] Changes level offset for anomaly detection APIs (#59920)
2020-07-21 03:38:09 +08:00
== {api-request-title}
[ML] Add docs for ML info endpoint (#35783) This endpoint was not previously documented as it was not particularly useful to end users. However, since the HLRC will support the endpoint we need some documentation to link to. The purpose of the endpoint is to provide defaults and limits used by ML. These are needed to fully understand configurations that have missing values because the missing value means the default should be used. Relates #35777
2018-11-23 00:23:31 +08:00
[ML] Deprecate X-Pack centric ML endpoints (#36315) This commit is part of our plan to deprecate and ultimately remove the use of _xpack in the REST APIs. Relates #35958
2018-12-08 04:34:11 +08:00
`GET _ml/info`
[ML] Add docs for ML info endpoint (#35783) This endpoint was not previously documented as it was not particularly useful to end users. However, since the HLRC will support the endpoint we need some documentation to link to. The purpose of the endpoint is to provide defaults and limits used by ML. These are needed to fully understand configurations that have missing values because the missing value means the default should be used. Relates #35777
2018-11-23 00:23:31 +08:00
[DOCS] Updates ML APIs to use new API template (#43711)
2019-06-28 04:58:42 +08:00
[[get-ml-info-prereqs]]
[DOCS] Changes level offset for anomaly detection APIs (#59920)
2020-07-21 03:38:09 +08:00
== {api-prereq-title}
[DOCS] Updates ML APIs to use new API template (#43711)
2019-06-28 04:58:42 +08:00
[ML] add new default char filter `first_line_with_letters` for machine learning categorization (#77457) The char filter replaces the previous default of `first_non_blank_line`. `first_non_blank_line` worked well to figure out what line had characters at all, but log lines like the following were handled poorly: ``` -------------------------------------------------------------------------------- Alias 'foo' already exists and this prevents setting up ILM for logs -------------------------------------------------------------------------------- ``` When combined with the `ml_standard` tokenizer, the first line was used: ``` -------------------------------------------------------------------------------- ``` This has no valid tokens for our standard tokenizer. Consequently, no tokens were found by `ml_standard` tokenizer. The new filter, `first_line_with_letters`, returns the first line with any letter character (e.g. `Character#isLetter` returns true). Given the previously poorly handled log, when combining with our `ml_standard` tokenizer, we get the following, more appropriate, tokens: ``` "tokens" : ["Alias", "foo", "already", "exists", "and", "this", "prevents", "setting", "up", "ILM", "for", "logs"] ```
2021-09-09 22:09:57 +08:00
Requires the `monitor_ml` cluster privilege. This privilege is included in the
[DOCS] Revises required privileges info in Anomaly Detection API docs (#72483)
2021-05-03 16:20:14 +08:00
`machine_learning_user` built-in role.
[DOCS] Updates ML APIs to use new API template (#43711)
2019-06-28 04:58:42 +08:00
[DOCS] Adds anchors and attributes to ML APIs
2019-06-28 00:42:47 +08:00
[[get-ml-info-desc]]
[DOCS] Changes level offset for anomaly detection APIs (#59920)
2020-07-21 03:38:09 +08:00
== {api-description-title}
[ML] Add docs for ML info endpoint (#35783) This endpoint was not previously documented as it was not particularly useful to end users. However, since the HLRC will support the endpoint we need some documentation to link to. The purpose of the endpoint is to provide defaults and limits used by ML. These are needed to fully understand configurations that have missing values because the missing value means the default should be used. Relates #35777
2018-11-23 00:23:31 +08:00
This endpoint is designed to be used by a user interface that needs to fully
understand machine learning configurations where some options are not specified,
[DOCS] Fix double spaces (#71082)
2021-03-31 21:57:47 +08:00
meaning that the defaults should be used. This endpoint may be used to find out
what those defaults are. It also provides information about the maximum size
[ML] Add total ML memory to ML info (#65195) This change adds an extra piece of information, limits.total_ml_memory, to the ML info response. This returns the total amount of memory that ML is permitted to use for native processes across all ML nodes in the cluster. Some of this may already be in use; the value returned is total, not available ML memory.
2020-11-18 23:06:21 +08:00
of {ml} jobs that could run in the current cluster configuration.
[ML] Add docs for ML info endpoint (#35783) This endpoint was not previously documented as it was not particularly useful to end users. However, since the HLRC will support the endpoint we need some documentation to link to. The purpose of the endpoint is to provide defaults and limits used by ML. These are needed to fully understand configurations that have missing values because the missing value means the default should be used. Relates #35777
2018-11-23 00:23:31 +08:00
[DOCS] Adds anchors and attributes to ML APIs
2019-06-28 00:42:47 +08:00
[[get-ml-info-example]]
[DOCS] Changes level offset for anomaly detection APIs (#59920)
2020-07-21 03:38:09 +08:00
== {api-examples-title}
[ML] Add docs for ML info endpoint (#35783) This endpoint was not previously documented as it was not particularly useful to end users. However, since the HLRC will support the endpoint we need some documentation to link to. The purpose of the endpoint is to provide defaults and limits used by ML. These are needed to fully understand configurations that have missing values because the missing value means the default should be used. Relates #35777
2018-11-23 00:23:31 +08:00
The endpoint takes no arguments:
[DOCS] Change // CONSOLE comments to [source,console] (#46441)
2019-09-06 22:55:16 +08:00
[source,console]
[ML] Add docs for ML info endpoint (#35783) This endpoint was not previously documented as it was not particularly useful to end users. However, since the HLRC will support the endpoint we need some documentation to link to. The purpose of the endpoint is to provide defaults and limits used by ML. These are needed to fully understand configurations that have missing values because the missing value means the default should be used. Relates #35777
2018-11-23 00:23:31 +08:00
--------------------------------------------------
[ML] Deprecate X-Pack centric ML endpoints (#36315) This commit is part of our plan to deprecate and ultimately remove the use of _xpack in the REST APIs. Relates #35958
2018-12-08 04:34:11 +08:00
GET _ml/info
[ML] Add docs for ML info endpoint (#35783) This endpoint was not previously documented as it was not particularly useful to end users. However, since the HLRC will support the endpoint we need some documentation to link to. The purpose of the endpoint is to provide defaults and limits used by ML. These are needed to fully understand configurations that have missing values because the missing value means the default should be used. Relates #35777
2018-11-23 00:23:31 +08:00
--------------------------------------------------
// TEST
This is a possible response:
[DOCS] [5 of 5] Change // TESTRESPONSE comments to [source,console-results] (#46449)
2019-09-07 02:05:36 +08:00
[source,console-result]
[ML] Add docs for ML info endpoint (#35783) This endpoint was not previously documented as it was not particularly useful to end users. However, since the HLRC will support the endpoint we need some documentation to link to. The purpose of the endpoint is to provide defaults and limits used by ML. These are needed to fully understand configurations that have missing values because the missing value means the default should be used. Relates #35777
2018-11-23 00:23:31 +08:00
----
{
"defaults" : {
"anomaly_detectors" : {
[ML] Add default categorization analyzer definition to ML info (#49545) The categorization job wizard in the ML UI will use this information when showing the effect of the chosen categorization analyzer on a sample of input.
2019-11-25 21:20:12 +08:00
"categorization_analyzer" : {
[ML] Make ml_standard tokenizer the default for new categorization jobs (#72805) Categorization jobs created once the entire cluster is upgraded to version 7.14 or higher will default to using the new ml_standard tokenizer rather than the previous default of the ml_classic tokenizer, and will incorporate the new first_non_blank_line char filter so that categorization is based purely on the first non-blank line of each message. The difference between the ml_classic and ml_standard tokenizers is that ml_classic splits on slashes and colons, so creates multiple tokens from URLs and filesystem paths, whereas ml_standard attempts to keep URLs, email addresses and filesystem paths as single tokens. It is still possible to config the ml_classic tokenizer if you prefer: just provide a categorization_analyzer within your analysis_config and whichever tokenizer you choose (which could be ml_classic or any other Elasticsearch tokenizer) will be used. To opt out of using first_non_blank_line as a default char filter, you must explicitly specify a categorization_analyzer that does not include it. If no categorization_analyzer is specified but categorization_filters are specified then the categorization filters are converted to char filters applied that are applied after first_non_blank_line. Closes elastic/ml-cpp#1724
2021-06-01 22:11:32 +08:00
"char_filter" : [
[ML] add new default char filter `first_line_with_letters` for machine learning categorization (#77457) The char filter replaces the previous default of `first_non_blank_line`. `first_non_blank_line` worked well to figure out what line had characters at all, but log lines like the following were handled poorly: ``` -------------------------------------------------------------------------------- Alias 'foo' already exists and this prevents setting up ILM for logs -------------------------------------------------------------------------------- ``` When combined with the `ml_standard` tokenizer, the first line was used: ``` -------------------------------------------------------------------------------- ``` This has no valid tokens for our standard tokenizer. Consequently, no tokens were found by `ml_standard` tokenizer. The new filter, `first_line_with_letters`, returns the first line with any letter character (e.g. `Character#isLetter` returns true). Given the previously poorly handled log, when combining with our `ml_standard` tokenizer, we get the following, more appropriate, tokens: ``` "tokens" : ["Alias", "foo", "already", "exists", "and", "this", "prevents", "setting", "up", "ILM", "for", "logs"] ```
2021-09-09 22:09:57 +08:00
"first_line_with_letters"
[ML] Make ml_standard tokenizer the default for new categorization jobs (#72805) Categorization jobs created once the entire cluster is upgraded to version 7.14 or higher will default to using the new ml_standard tokenizer rather than the previous default of the ml_classic tokenizer, and will incorporate the new first_non_blank_line char filter so that categorization is based purely on the first non-blank line of each message. The difference between the ml_classic and ml_standard tokenizers is that ml_classic splits on slashes and colons, so creates multiple tokens from URLs and filesystem paths, whereas ml_standard attempts to keep URLs, email addresses and filesystem paths as single tokens. It is still possible to config the ml_classic tokenizer if you prefer: just provide a categorization_analyzer within your analysis_config and whichever tokenizer you choose (which could be ml_classic or any other Elasticsearch tokenizer) will be used. To opt out of using first_non_blank_line as a default char filter, you must explicitly specify a categorization_analyzer that does not include it. If no categorization_analyzer is specified but categorization_filters are specified then the categorization filters are converted to char filters applied that are applied after first_non_blank_line. Closes elastic/ml-cpp#1724
2021-06-01 22:11:32 +08:00
],
"tokenizer" : "ml_standard",
[ML] Add default categorization analyzer definition to ML info (#49545) The categorization job wizard in the ML UI will use this information when showing the effect of the chosen categorization analyzer on a sample of input.
2019-11-25 21:20:12 +08:00
"filter" : [
{
"type" : "stop",
"stopwords" : [
"Monday",
"Tuesday",
"Wednesday",
"Thursday",
"Friday",
"Saturday",
"Sunday",
"Mon",
"Tue",
"Wed",
"Thu",
"Fri",
"Sat",
"Sun",
"January",
"February",
"March",
"April",
"May",
"June",
"July",
"August",
"September",
"October",
"November",
"December",
"Jan",
"Feb",
"Mar",
"Apr",
"May",
"Jun",
"Jul",
"Aug",
"Sep",
"Oct",
"Nov",
"Dec",
"GMT",
"UTC"
]
}
]
},
[ML] Add docs for ML info endpoint (#35783) This endpoint was not previously documented as it was not particularly useful to end users. However, since the HLRC will support the endpoint we need some documentation to link to. The purpose of the endpoint is to provide defaults and limits used by ML. These are needed to fully understand configurations that have missing values because the missing value means the default should be used. Relates #35777
2018-11-23 00:23:31 +08:00
"model_memory_limit" : "1gb",
"categorization_examples_limit" : 4,
[ML] More advanced model snapshot retention options (#56125) This PR implements the following changes to make ML model snapshot retention more flexible in advance of adding a UI for the feature in an upcoming release. - The default for `model_snapshot_retention_days` for new jobs is now 10 instead of 1 - There is a new job setting, `daily_model_snapshot_retention_after_days`, that defaults to 1 for new jobs and `model_snapshot_retention_days` for pre-7.8 jobs - For days that are older than `model_snapshot_retention_days`, all model snapshots are deleted as before - For days that are in between `daily_model_snapshot_retention_after_days` and `model_snapshot_retention_days` all but the first model snapshot for that day are deleted - The `retain` setting of model snapshots is still respected to allow selected model snapshots to be retained indefinitely Closes #52150
2020-05-05 19:55:50 +08:00
"model_snapshot_retention_days" : 10,
"daily_model_snapshot_retention_after_days" : 1
[ML] Add docs for ML info endpoint (#35783) This endpoint was not previously documented as it was not particularly useful to end users. However, since the HLRC will support the endpoint we need some documentation to link to. The purpose of the endpoint is to provide defaults and limits used by ML. These are needed to fully understand configurations that have missing values because the missing value means the default should be used. Relates #35777
2018-11-23 00:23:31 +08:00
},
"datafeeds" : {
"scroll_size" : 1000
}
},
ML: Adds set_upgrade_mode API endpoint (#37837) * ML: Add MlMetadata.upgrade_mode and API * Adding tests * Adding wait conditionals for the upgrade_mode call to return * Adding tests * adjusting format and tests * Adjusting wait conditions for api return and msgs * adjusting doc tests * adding upgrade mode tests to black list
2019-01-28 23:07:30 +08:00
"upgrade_mode": false,
Add native code info to ML info api (#43172) The machine learning feature of xpack has native binaries with a different commit id than the rest of code. It is currently exposed in the xpack info api. This commit adds that commit information to the ML info api, so that it may be removed from the info api.
2019-06-14 02:38:29 +08:00
"native_code" : {
"version": "7.0.0",
"build_hash": "99a07c016d5a73"
},
[ML] Add effective max model memory limit to ML info (#55529) The ML info endpoint returns the max_model_memory_limit setting if one is configured. However, it is still possible to create a job that cannot run anywhere in the current cluster because no node in the cluster has enough memory to accommodate it. This change adds an extra piece of information, limits.effective_max_model_memory_limit, to the ML info response that returns the biggest model memory limit that could be run in the current cluster assuming no other jobs were running. The idea is that the ML UI will be able to warn users who try to create jobs with higher model memory limits that their jobs will not be able to start unless they add a bigger ML node to their cluster. Relates elastic/kibana#63942
2020-04-22 18:36:58 +08:00
"limits" : {
[ML] Add total ML memory to ML info (#65195) This change adds an extra piece of information, limits.total_ml_memory, to the ML info response. This returns the total amount of memory that ML is permitted to use for native processes across all ML nodes in the cluster. Some of this may already be in use; the value returned is total, not available ML memory.
2020-11-18 23:06:21 +08:00
"effective_max_model_memory_limit": "28961mb",
"total_ml_memory": "86883mb"
[ML] Add effective max model memory limit to ML info (#55529) The ML info endpoint returns the max_model_memory_limit setting if one is configured. However, it is still possible to create a job that cannot run anywhere in the current cluster because no node in the cluster has enough memory to accommodate it. This change adds an extra piece of information, limits.effective_max_model_memory_limit, to the ML info response that returns the biggest model memory limit that could be run in the current cluster assuming no other jobs were running. The idea is that the ML UI will be able to warn users who try to create jobs with higher model memory limits that their jobs will not be able to start unless they add a bigger ML node to their cluster. Relates elastic/kibana#63942
2020-04-22 18:36:58 +08:00
}
[ML] Add docs for ML info endpoint (#35783) This endpoint was not previously documented as it was not particularly useful to end users. However, since the HLRC will support the endpoint we need some documentation to link to. The purpose of the endpoint is to provide defaults and limits used by ML. These are needed to fully understand configurations that have missing values because the missing value means the default should be used. Relates #35777
2018-11-23 00:23:31 +08:00
}
----
relax ML Info Docs expected response (#38993) the get-ml-info API documentation tested that the response show that ML's `upgrade_mode` was false. For reasons that may be true due to other tests running in parallel or not cleaning themselves up, this may not be guaranteed. Since the actual value here is not of importance, this commit relaxes the requirement that upgrade_mode be static.
2019-02-16 08:26:41 +08:00
// TESTRESPONSE[s/"upgrade_mode": false/"upgrade_mode": $body.upgrade_mode/]
Add native code info to ML info api (#43172) The machine learning feature of xpack has native binaries with a different commit id than the rest of code. It is currently exposed in the xpack info api. This commit adds that commit information to the ML info api, so that it may be removed from the info api.
2019-06-14 02:38:29 +08:00
// TESTRESPONSE[s/"version": "7.0.0",/"version": "$body.native_code.version",/]
// TESTRESPONSE[s/"build_hash": "99a07c016d5a73"/"build_hash": "$body.native_code.build_hash"/]
[ML] Add effective max model memory limit to ML info (#55529) The ML info endpoint returns the max_model_memory_limit setting if one is configured. However, it is still possible to create a job that cannot run anywhere in the current cluster because no node in the cluster has enough memory to accommodate it. This change adds an extra piece of information, limits.effective_max_model_memory_limit, to the ML info response that returns the biggest model memory limit that could be run in the current cluster assuming no other jobs were running. The idea is that the ML UI will be able to warn users who try to create jobs with higher model memory limits that their jobs will not be able to start unless they add a bigger ML node to their cluster. Relates elastic/kibana#63942
2020-04-22 18:36:58 +08:00
// TESTRESPONSE[s/"effective_max_model_memory_limit": "28961mb"/"effective_max_model_memory_limit": "$body.limits.effective_max_model_memory_limit"/]
[ML] Add total ML memory to ML info (#65195) This change adds an extra piece of information, limits.total_ml_memory, to the ML info response. This returns the total amount of memory that ML is permitted to use for native processes across all ML nodes in the cluster. Some of this may already be in use; the value returned is total, not available ML memory.
2020-11-18 23:06:21 +08:00
// TESTRESPONSE[s/"total_ml_memory": "86883mb"/"total_ml_memory": "$body.limits.total_ml_memory"/]