Commit Graph

75 Commits

Author SHA1 Message Date
István Zoltán Szabó c994369893
[DOCS] Expands GET DFA stats API docs with new phases (#56407)
Co-authored-by: Lisa Cawley <lcawley@elastic.co>
2020-05-11 09:22:30 +02:00
Dimitris Athanasiou 6bf3834059
[ML] Add loss_function to regression (#56118)
Adds parameters `loss_function` and `loss_function_parameter`
to regression.
2020-05-05 12:36:05 +03:00
István Zoltán Szabó 86032ac56a
[DOCS] Simplifies footnote text in DFA APIs (#56105)
Co-authored-by: Lisa Cawley <lcawley@elastic.co>
2020-05-05 09:03:16 +02:00
Lisa Cawley 52a2f7689f
[DOCS] Synchs and links hyperparameter descriptions (#55827) 2020-05-04 07:37:14 -07:00
Lisa Cawley 5ef7aacbf7
[DOCS] Adds documentation for secondary authorization headers (#55365)
Co-authored-by: Tim Vernum <tim@adjective.org>
2020-04-29 08:28:42 -07:00
István Zoltán Szabó ca2f98382f
[DOCS] Changes feature importance links to point to the new page (#55531)
* [DOCS] Changes feature importance links to point to the new page.

* [DOCS] Fixes line breaks.
2020-04-28 09:02:14 +02:00
Lisa Cawley 7fafec0f8f
[DOCS] Update example and nesting in get data frame analytics job stats API (#55191)
Co-Authored-By: Valeriy Khakhutskyy <1292899+valeriy42@users.noreply.github.com>
2020-04-22 08:07:31 -07:00
David Roberts 8906e76079
[ML] Return assigned node in start/open job/datafeed response (#55473)
Adds a "node" field to the response from the following endpoints:

1. Open anomaly detection job
2. Start datafeed
3. Start data frame analytics job

If the job or datafeed is assigned to a node immediately then
this field will return the ID of that node.

In the case where a job or datafeed is opened or started lazily
the node field will contain an empty string.  Clients that want
to test whether a job or datafeed was opened or started lazily
can therefore check for this.

Fixes #54067
2020-04-22 08:44:57 +01:00
Benjamin Trent f72b2db29a
[ML] partitions model definitions into chunks (#55260)
This paves the data layer way so that exceptionally large models are partitioned across multiple documents.

This change means that nodes before 7.8.0 will not be able to use trained inference models created on nodes on or after 7.8.0.

I chose the definition document limit to be 100. This *SHOULD* be plenty for any large model. One of the largest models that I have created so far had the following stats:
~314MB of inflated JSON, ~66MB when compressed, ~177MB of heap. 
With the chunking sizes of `16 * 1024 * 1024` its compressed string could be partitioned to 5 documents. 
Supporting models 20 times this size (compressed) seems adequate for now.
2020-04-20 15:13:23 -04:00
Benjamin Trent c1afda4a23
[ML] adding prediction_field_type to inference config (#55128)
Data frame analytics dynamically determines the classification field type. This field type then dictates the encoded JSON that is written to Elasticsearch. 

Inference needs to know about this field type so that it may provide the EXACT SAME predicted values as analytics. 

Here is added a new field `prediction_field_type` which indicates the desired type. Options are: `string` (DEFAULT), `number`, `boolean` (where close_to(1.0) == true, false otherwise). 

Analytics provides the default `prediction_field_type` when the model is created from the process.
2020-04-15 08:32:48 -04:00
Lisa Cawley 1f0341db39
[DOCS] Removes unshared sections from ml-shared.asciidoc (#55129) 2020-04-14 15:19:31 -07:00
Lisa Cawley 998a085c14
[DOCS] Edits create data frame analytics job API (#54751) 2020-04-13 09:58:03 -07:00
István Zoltán Szabó b1b067c5ba
[DOCS] Adds link points to the data frame analytics supported fields (#55004)
Co-authored-by: lcawl <lcawley@elastic.co>
2020-04-09 11:16:13 -07:00
Lisa Cawley c355fea8f4
[DOCS] Remove text fields from classification dependent variables (#54849) 2020-04-07 10:43:15 -07:00
István Zoltán Szabó 1ae8bde756
[DOCS] Changes kibana_user to kibana_admin in DFA API prerequisites. (#54806) 2020-04-06 15:45:08 +02:00
István Zoltán Szabó a0662399c7
[DOCS] Makes PUT inference API docs collapsible (#54653)
Co-authored-by: lcawl <lcawley@elastic.co>
2020-04-03 09:45:42 +02:00
Benjamin Trent 4e1ff31c3c
[ML] add new inference_config field to trained model config (#54421)
A new field called `inference_config` is now added to the trained model config object. This new field allows for default inference settings from analytics or some external model builder. 

The inference processor can still override whatever is set as the default in the trained model config.
2020-04-02 10:34:17 -04:00
Benjamin Trent 1d24960ff8
[ML] prefer secondary authorization header for data[feed|frame] authz (#54121)
Secondary authorization headers are to be used to facilitate Kibana spaces support + ML jobs/datafeeds. 

Now on PUT/Update/Preview datafeed, and PUT data frame analytics the secondary authorization is preferred over the primary (if provided).

closes https://github.com/elastic/elasticsearch/issues/53801
2020-04-02 10:10:46 -04:00
Lisa Cawley b90e491f68
[DOCS] Collapses nested objects in data frame analytics APIs (#54472) 2020-03-31 10:56:48 -07:00
Dimitris Athanasiou 5a98fc20e1
[ML] Fix DF analytics explain API request in docs (#54510)
The explain API expects a data frame analytics config
as its request.
2020-03-31 18:37:19 +03:00
István Zoltán Szabó 85d9b34dc5
[DOCS] Adds description of analysis_stats object and its properties to GET DFA stats API docs (#53881)
Co-authored-by: Valeriy Khakhutskyy <1292899+valeriy42@users.noreply.github.com>
Co-authored-by: Lisa Cawley <lcawley@elastic.co>
2020-03-31 13:27:54 +02:00
Tom Veasey 58340c2dbe
[ML] Adds the class_assignment_objective parameter to classification (#52763)
Adds a new parameter for classification that enables choosing whether to assign labels to
maximise accuracy or to maximise the minimum class recall.

Fixes #52427.
2020-03-12 18:39:29 +00:00
Dimitris Athanasiou 5a32f50d18
[ML] Rename data frame analytics maximum_number_trees to max_trees (#53300)
Deprecates `maximum_number_trees` parameter of classification and
regression and replaces it with `max_trees`.
2020-03-11 10:33:53 +02:00
István Zoltán Szabó 24fe7e5899
[DOCS] Adds response body documentation to GET inference API (#53050) 2020-03-03 16:25:24 +01:00
Lisa Cawley f41ebe47e3
[DOCS] Clarifies description of num_top_feature_importance_values (#52246)
Co-Authored-By: Valeriy Khakhutskyy <1292899+valeriy42@users.noreply.github.com>
2020-02-18 08:48:24 -08:00
David Kyle f64c6359ed
[ML] Make Ensemble feature names optional (#51996)
The featureNames field is requisite in individual models but is not required by the Ensemble.
2020-02-07 10:07:18 +00:00
István Zoltán Szabó 67f14c3978
[DOCS] Adds PUT inference API docs (#51231)
Co-authored-by: Benjamin Trent <ben.w.trent@gmail.com>
Co-authored-by: Lisa Cawley <lcawley@elastic.co>
2020-01-31 13:12:24 +01:00
Benjamin Trent c9e285c1e6
[ML][Inference] add tags url param to GET (#51330)
Adds a new URL parameter, `tags` to the GET _ml/inference/<model_id> endpoint.

This parameter allows the list of models to be further reduced to those who contain all the provided tags.
2020-01-24 07:30:56 -05:00
Dimitris Athanasiou 24ce598239
[ML] DF Analytics _explain API should skip object fields (#51115)
Object fields cannot be used as features. At the moment _explain
API includes them and even worse it allows it does not error when
an object field is excluded. This creates the expectation to the
user that all children fields will also be excluded while it's not
the case.

This commit omits object fields from the _explain API and also
adds an error if an object field is included or excluded.
2020-01-17 12:24:17 +02:00
David Kyle 5ad1d0d2cc
Fix hardcoded version replacement in put-dfanalytics.asciidoc (#51056) 2020-01-16 10:06:45 +00:00
Przemysław Witek 999884d8fb
Add missing docs for new evaluation metrics (#50967) 2020-01-15 14:23:37 +01:00
Dimitris Athanasiou 4d2be9bd32
[ML] Add num_top_feature_importance_values param to regression and classi… (#50914)
Adds a new parameter to regression and classification that enables computation
of importance for the top most important features. The computation of the importance
is based on SHAP (SHapley Additive exPlanations) method.
2020-01-14 15:01:47 +02:00
István Zoltán Szabó b3457154a3
[DOCS] Fine-tunes data frame analytics API docs formatting. (#50799) 2020-01-09 16:21:01 +01:00
István Zoltán Szabó b683f96e23
[DOCS] Moves analysis resources to PUT DFA API docs (#50704)
Co-authored-by: Lisa Cawley <lcawley@elastic.co>
2020-01-09 13:57:11 +01:00
István Zoltán Szabó bc21500201
[DOCS] Forms role and privilege requirements as bulleted lists in DFA API docs (#50732)
Co-Authored-By: Lisa Cawley <lcawley@elastic.co>
2020-01-09 10:44:07 +01:00
Dimitris Athanasiou af0ce426cc
[ML] Implement force deleting a data frame analytics job (#50553)
Adds a `force` parameter to the delete data frame analytics
request. When `force` is `true`, the action force-stops the
jobs and then proceeds to the deletion. This can be used in
order to delete a non-stopped job with a single request.

Closes #48124
2020-01-03 12:01:41 +02:00
István Zoltán Szabó fd50169c74
[DOCS] Specifies the possible data types of classification dependent_variable (#50582) 2020-01-03 10:41:38 +01:00
István Zoltán Szabó 50e26d40a2
[DOCS] Adds GET, GET stats and DELETE inference APIs (#50224)
Co-Authored-By: Lisa Cawley <lcawley@elastic.co>
2019-12-18 09:10:12 +01:00
István Zoltán Szabó 3857e3d94f
[DOCS] Moves data frame analytics job resource definitions into APIs (#50021) 2019-12-12 10:59:37 +01:00
Dimitris Athanasiou 269425b54d
[ML] Introduce randomize_seed setting for regression and classification (#49990)
This adds a new `randomize_seed` for regression and classification.
When not explicitly set, the seed is randomly generated. One can
reuse the seed in a similar job in order to ensure the same docs
are picked for training.
2019-12-10 10:22:53 +02:00
Lisa Cawley 0f51bc2f72
[DOCS] Move anomaly detection job resource definitions into APIs (#49700)
Co-Authored-By: István Zoltán Szabó <istvan.szabo@elastic.co>
2019-12-06 15:32:07 -08:00
István Zoltán Szabó e5d512a8ed
[DOCS] Fixes classification evaluation example response. (#49905) 2019-12-06 13:24:22 +01:00
István Zoltán Szabó f7a5b73972
[DOCS] Adds an example of preprocessing actions to the PUT DFA API docs (#49831) 2019-12-05 14:15:19 +01:00
Dimitris Athanasiou bad07b76f7
[ML] Add optional source filtering during data frame reindexing (#49690)
This adds a `_source` setting under the `source` setting of a data
frame analytics config. The new `_source` is reusing the structure
of a `FetchSourceContext` like `analyzed_fields` does. Specifying
includes and excludes for source allows selecting which fields
will get reindexed and will be available in the destination index.

Closes #49531
2019-11-29 14:20:31 +02:00
lcawl 3b3f3ca925 [DOCS] Fixes typo in ML resources 2019-11-26 10:28:18 -08:00
lcawl 63b944c00f [DOCS] Fixes data type formatting 2019-11-26 08:21:39 -08:00
Dimitris Athanasiou 0390ec3627
[ML] Explain data frame analytics API (#49455)
This commit replaces the _estimate_memory_usage API with
a new API, the _explain API.

The API consolidates information that is useful before
creating a data frame analytics job.

It includes:

- memory estimation
- field selection explanation

Memory estimation is moved here from what was previously
calculated in the _estimate_memory_usage API.

Field selection is a new feature that explains to the user
whether each available field was selected to be included or
not in the analysis. In the case it was not included, it also
explains the reason why.
2019-11-22 20:08:14 +02:00
István Zoltán Szabó 7180b90646
[DOCS] Removes best practice about fields that are highly correlated to the dependent variable. (#48935) 2019-11-11 10:00:11 -05:00
István Zoltán Szabó e9cec6e1f7
[DOCS] Extends analyzed_fields description in PUT DFA API docs. (#48307) 2019-11-11 09:53:59 -05:00
István Zoltán Szabó 6c3fed8d4d
[DOCS] Adds classification type DFA API docs and ml-shared.asciidoc (#48241) 2019-11-06 07:40:27 -05:00