Commit Graph

592 Commits

Author SHA1 Message Date
David Kyle d38d1af242
[ML] GA the update trained model action (#108868)
Accidentally missed when the other trained model APIs went GA
2024-05-22 13:30:25 +01:00
Max Hniebergall a2008bd190
[ML] Add option to disable inference process cache by default (#108784)
* Add option to disable inference process cache by default

* Add test

* improve tests

* Update docs and improve code

---------

Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
2024-05-19 11:11:02 -04:00
Lisa Cawley a079cdc17d
[DOCS] Update transform and anomaly detection rule creation steps (#107975) 2024-05-07 07:52:45 -07:00
Liam Thompson 33a71e3289
[DOCS] Refactor book-scoped variables in `docs/reference/index.asciidoc` (#107413)
* Remove `es-test-dir` book-scoped variable

* Remove `plugins-examples-dir` book-scoped variable

* Remove `:dependencies-dir:` and `:xes-repo-dir:` book-scoped variables

- In `index.asciidoc`, two variables (`:dependencies-dir:` and `:xes-repo-dir:`) were removed.
- In `sql/index.asciidoc`, the `:sql-tests:` path was updated to fuller path
- In `esql/index.asciidoc`, the `:esql-tests:` path was updated idem

* Replace `es-repo-dir` with `es-ref-dir`

* Move `:include-xpack: true` to few files that use it, remove from index.asciidoc
2024-04-17 14:37:07 +02:00
Niels Bauman 0f3ac367ac
Rename values of `FailureStoreOptions` (#107062)
With these new values, there's a better match between selecting failure stores in read and write operations.
2024-04-06 08:46:38 +02:00
David Kyle 50dcfdc726
[ML] Document wait_for_completion parameter to PUT trained models (#106769) 2024-03-27 16:55:06 +00:00
István Zoltán Szabó de7e8980d1
[DOCS] Amends important note on delayed data detection. (#106610) 2024-03-21 16:41:31 +01:00
István Zoltán Szabó 5d7549c2c8
[DOCS] Documents scripted metric aggregation limitation in datafeeds (#106059) 2024-03-07 14:07:14 +01:00
Mary Gouseti 6fbf9892aa
Add the parameter `failure_store` to multi-target syntax APIs (#105386)
In this PR we introduce a new query parameter behind the failure store feature flag. The query param, `faliure_store` allows the multi-syntax supporting APIs to choose the failure store indices as well. If an API should not support failure store, the `allowFailureStore` flag should be `false`.
2024-03-06 14:04:18 +02:00
Lisa Cawley 1cfa86ee13
[DOCS] Update anomaly detection jobs health rule details (#105716) 2024-02-22 08:26:31 -08:00
István Zoltán Szabó cfa2b2a2e2
[DOCS] Rephrases sentence in data_description param of PUT job API docs (#104792)
* [DOCS] Rephrase sentence in data_description param of PUT job API docs.

* [DOCS] Further edits.
2024-01-26 14:27:02 +01:00
David Roberts 35aff34912
[ML] Add an important note about a gotcha with the delayed data check (#104725)
Recently a user saw spurious delayed data warnings. These turned
out to be due to accidentally setting `summary_count_field` to a
field that was always zero. This meant that every document was
considered delayed.
2024-01-25 14:31:24 +00:00
István Zoltán Szabó e48b549588
[DOCS] Fixes asciidoc syntax in PUT trained models API docs. (#104741) 2024-01-25 14:22:17 +01:00
István Zoltán Szabó 56be53adac
[DOCS] Fixes link in PUT datafeeds API docs. (#102684) 2023-11-28 10:47:24 +01:00
István Zoltán Szabó cb5a702174
[DOCS] Expands anomaly alerts docs with info on Anomaly Explorer integration. (#102477) 2023-11-23 14:33:06 +01:00
David Kyle 330e8b99bf
[ML] Add prefix strings option to trained models (#102089)
Certain NLP models such as multilingual-e5-large require a prefix 
string to be applied to the input text. For asymmetric tasks such as 
information retrieval the prefix can be different when ingesting the
data and when searching it. For example text embedding model can
have a one prefix applied when the model is evaluated as part of an
knn search and a different prefix when ingesting documents.
2023-11-14 13:02:02 +00:00
István Zoltán Szabó 481ebd2e21
[DOCS] Improves readability of PUT trained models API docs page (#101880)
* [DOCS] Improves readability of PUT trained models API docs page.

* [DOCS] Fixes URLs.
2023-11-08 17:57:57 +01:00
Abdon Pijpelink 70128f5b74
[DOCS] Mark 'ignore_throttled' deprecated in all docs (#101838) 2023-11-07 13:03:49 +01:00
David Roberts 9286716660
[ML] Include ML processor limits in `_ml/info` response (#101392)
The _ml/info response now includes two extra fields in its
`limits`:

1. `max_single_ml_node_processors`
2. `total_ml_processors`

These fields are _only_ included if they can be accurately
calculated. If autoscaling is enabled and the ML nodes are
not at their maximum size then these fields _cannot_
currently be accurately calculated. (This could potentially
be improved in the future with additional settings set by
the control plane.)
2023-10-27 13:52:03 +01:00
István Zoltán Szabó c34e0c0746
[DOCS] Clarifies that inference input must be single string (#101301) 2023-10-25 17:18:05 +02:00
Lisa Cawley 3418c6a2af
[DOCS] Alerts-as-data for anomaly detection rules (#100864) 2023-10-17 07:37:22 -07:00
Max Hniebergall 7c21ce3f1b
Platform specific models (#99584)
* Added platform architecture field to TrainedModelMetadata and users of TrainedModelMetadata

* Added TransportVersions guarding for TrainedModelMetadata

* Prevent platform-specific models from being deployed on the wrong architecture

* Added logic to only verify node architectures for models which are platform specific

* Handle null platform architecture

* Added logging for the detection of heterogeneous platform architectures among ML nodes and refactoring to support this

* Added platform architecture field to TrainedModelConfig

* Stop platform-speficic model when rebalance occurs and the cluster has a heterogeneous architecture among ML nodes

* Added logic to TransportPutTrainedModelAction to return a warning response header when the model is paltform-specific and cannot be depoloyed on the cluster at that time due to heterogenous architectures among ML nodes

* Added MlPlatformArchitecturesUtilTests

* Updated Create Trained Models API docs to describe the new platform_architecture optional field.

* Updated/incremented InferenceIndexConstants

* Added special override to make  models with linux-x86_64 in the model ID to be platform specific
2023-09-28 13:56:45 -04:00
DeDe Morton e3f1c15788
Update ml-delayed-data-detection.asciidoc (#99843) 2023-09-26 11:18:24 +02:00
James Rodewig 255c9a7f95
[DOCS] Move x-pack docs to `docs/reference` dir (#99209)
**Problem:**
For historical reasons, source files for the Elasticsearch Guide's security, watcher, and Logstash API docs are housed in the `x-pack/docs` directory. This can confuse new contributors who expect Elasticsearch Guide docs to be located in `docs/reference`. 

**Solution:**
- Move the security, watcher, and Logstash API doc source files to the `docs/reference` directory
- Update doc snippet tests to use security

Rel: https://github.com/elastic/platform-docs-team/issues/208
2023-09-12 14:53:41 -04:00
Jonathan Buttner 1ca66bde91
[ML] Safely drain deployment request queues before allowing node to shutdown (#98406)
* isSafeToShutdown checks routing table

* Rebalancer changes and tests

* Update docs/changelog/98406.yaml

* Forcing lifecycle tests to avoid over time case

* Changes and remaining tests

* Adding node service changes

* Finishing unit tests

* Adding wait for completion paramater

* Adding stop deployment integration tests

* Cleaning up code

* Fixing stop deployment test

* Fixing string formatter issue and timeout

* Investigating deadlock

* More testing

* More logging

* Prevent model reloading while stopping

* Fixing compile error

* More code clean up

* Adding test for loading model after stopping

* Addressing review feedback

* Fixing a couple shutdown -> shutdownNow tests

* Adding doc changes and refactoring
2023-08-31 15:37:11 -04:00
István Zoltán Szabó 947128e76d
[DOCS] Fixes NOTE display error. (#98783) 2023-08-23 12:18:54 +02:00
Ed Savage 3682a88199
[ML] Update documentation regarding versioning. (#98320)
Update the ml and transform reference documentation to provide information regarding the new versioning schemes independent from the product versions.

Co-authored-by: István Zoltán Szabó <istvan.szabo@elastic.co>
2023-08-10 11:20:58 +01:00
István Zoltán Szabó 3fd94678e6
[DOCS] Adds important admonition to handling delayed data page (#97753) 2023-07-19 13:32:06 +02:00
David Roberts e810d7b77b
[ML] `inference_config` is optional for the infer trained model API (#97464)
It was made optional in #92359 which was released in version 8.6.1,
but the docs weren't updated to reflect this.
2023-07-12 08:35:06 +01:00
Max Hniebergall 3a4113801c
[NLP] Support the different mask tokens used by NLP models for Fill Mask (#97453)
Add mask_token field to fill_mask of _ml/trained_models.

This change will enable users and Kibana to get the particular mask tokens needed for deployed models by adding a mask_token field to the GET _ml/trained_models API, as an enhancement to support kibana#159577.
2023-07-11 14:42:44 -04:00
István Zoltán Szabó 8d5b803bff
[DOCS] Adds API docs for bert_ja text embedding tokenizer option (#96873) 2023-06-26 11:36:08 +02:00
Benjamin Trent 14ca8fee20
[ML] add support for xlm_roberta tokenized models (#94089)
Many multi-lingual and newer models use a tokenization scheme similar to
sentence-piece. This PR adds support for one of those tokenization
schemes, XLMRoBERTa. 

The main changes are:  - Support for xlm_roberta tokenization
configuration  - Adding `scores` to the vocabulary document stored,
requiring that scores be the same size as the vocabulary  - Adding a new
flat text file to resources that is the spm char normalizer.
2023-06-13 08:40:55 -04:00
debadair 777598d602
[DOCS] Remove redirect pages (#88738)
* [DOCS] Remove manual redirects

* [DOCS] Removed refs to modules-discovery-hosts-providers

* [DOCS] Fixed broken internal refs

* Fixing bad cross links in ES book, and adding redirects.asciidoc[] back into docs/reference/index.asciidoc.

* Update docs/reference/search/point-in-time-api.asciidoc

Co-authored-by: James Rodewig <james.rodewig@elastic.co>

* Update docs/reference/setup/restart-cluster.asciidoc

Co-authored-by: James Rodewig <james.rodewig@elastic.co>

* Update docs/reference/sql/endpoints/translate.asciidoc

Co-authored-by: James Rodewig <james.rodewig@elastic.co>

* Update docs/reference/snapshot-restore/restore-snapshot.asciidoc

Co-authored-by: James Rodewig <james.rodewig@elastic.co>

* Update repository-azure.asciidoc

* Update node-tool.asciidoc

* Update repository-azure.asciidoc

---------

Co-authored-by: amyjtechwriter <61687663+amyjtechwriter@users.noreply.github.com>
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
Co-authored-by: Amy Jonsson <amy.jonsson@elastic.co>
Co-authored-by: James Rodewig <james.rodewig@elastic.co>
2023-05-24 12:32:46 +01:00
István Zoltán Szabó b164555072
[DOCS] Adds deployment ID param documentation to trained model APIs (#96174) 2023-05-17 15:56:58 +02:00
David Kyle 6de8469a51
[ML] Include model definition install status for Pytorch models (#95271)
Adds a new include flag definition_status to the GET trained models API.
When present the trained model configuration returned in the response 
will have the new boolean field fully_defined if the full model definition 
is exists.
2023-04-17 18:12:26 +01:00
David Kyle 7d90c519ef
[ML] Add embedding_size to text embedding config (#95176) 2023-04-17 11:49:35 +01:00
David Roberts 708730e27c
[ML] Add _meta field to data frame analytics config (#94529)
This PR adds a new field, `_meta`, to the data frame
analytics configuration.

The `_meta` field stores an arbitrary key-value map.
Keys are strings. Values are arbitrary objects
(possibly also maps).

The `_meta` field can be updated using the data frame
analytics `_update` endpoint.
2023-03-20 11:53:53 +00:00
Lisa Cawley f49bb09503
[DOCS] Refresh screenshots for machine learning rules (#93805) 2023-02-15 15:43:30 -08:00
István Zoltán Szabó c08c16e311
[DOCS] Removes semantic search reference docs (#93500) 2023-02-06 11:00:25 +01:00
István Zoltán Szabó 9229011000
[DOCS] Adds size parameter details to aggregating data for faster performance page (#93475) 2023-02-02 19:29:11 +01:00
István Zoltán Szabó 05c77534fe
[DOCS] Fixes markup for example in count function docs. (#93308) 2023-01-27 14:41:30 +01:00
Valeriy Khakhutskyy c24712bfa7
[ML] Add multimodal distribution field processing for anomaly score explanation (#92978)
The companion PR to elastic/ml-cpp#2440 adds processing of multimodal_distribution field in the anomaly score explanation. I added a changelog entry in the ml-cpp PR hence I mark this PR as a non-issue.
2023-01-17 21:16:12 +01:00
David Kyle 6acfbbcd8b
[ML] Utilise parallel allocations where the inference request contains multiple documents (#92359)
Divide work from the _infer API among all allocations
2023-01-11 12:38:35 +00:00
David Kyle fbb6abd2f4
[ML] Increase the default timeout for start trained model deployment (#92328)
A 30 second timeout is inline with the default value used in most ML APIs.
2022-12-14 13:32:23 +00:00
David Roberts 6fa3d73fd5
[ML] Make native inference generally available (#92213)
Previously this functionality was beta. This PR changes it to GA.
2022-12-12 15:43:30 +00:00
David Roberts 1be17d8c55
[ML] Correct the update datafeed docs (#92227)
These docs previously implied that you could update datafeed
properties while the datafeed was running, but then would have
to stop and restart it for the changes to take effect.

In fact datafeed updates can only be made while the datafeed is
stopped (and this has been the case for many years, if not forever).
2022-12-08 09:58:48 +00:00
Nik Everett 6481342466
Fix sneaky docs test failure (#91829)
This prevents docs files from *starting* with a "response" because when
that happens the response is converted to an assertion and appended
to the last snippet that was processed. If that last snipper was in a
different file then it's very hard to reason about the tests. That goes
double because the order we iterate files isn't defined....

Anyway! This adds a guard in the build, removes the offending
"response", and reenables the tests that we'd thought we failing here.

Closes #91081
2022-12-07 11:02:44 -05:00
István Zoltán Szabó f5b98b3182
[DOCS] Documents how aggregate_metric_double works in datafeeds (#92139) 2022-12-07 09:28:38 +01:00
Lisa Cawley 3f2f9de928
[DOCS] Refresh machine learning rule docs (#92013) 2022-12-05 07:47:42 -08:00
István Zoltán Szabó 99415818e2
[DOCS] Adds semantic search API to the trained model API list (#91815) 2022-11-22 18:08:06 +01:00