* Remove `es-test-dir` book-scoped variable
* Remove `plugins-examples-dir` book-scoped variable
* Remove `:dependencies-dir:` and `:xes-repo-dir:` book-scoped variables
- In `index.asciidoc`, two variables (`:dependencies-dir:` and `:xes-repo-dir:`) were removed.
- In `sql/index.asciidoc`, the `:sql-tests:` path was updated to fuller path
- In `esql/index.asciidoc`, the `:esql-tests:` path was updated idem
* Replace `es-repo-dir` with `es-ref-dir`
* Move `:include-xpack: true` to few files that use it, remove from index.asciidoc
In this PR we introduce a new query parameter behind the failure store feature flag. The query param, `faliure_store` allows the multi-syntax supporting APIs to choose the failure store indices as well. If an API should not support failure store, the `allowFailureStore` flag should be `false`.
Recently a user saw spurious delayed data warnings. These turned
out to be due to accidentally setting `summary_count_field` to a
field that was always zero. This meant that every document was
considered delayed.
Certain NLP models such as multilingual-e5-large require a prefix
string to be applied to the input text. For asymmetric tasks such as
information retrieval the prefix can be different when ingesting the
data and when searching it. For example text embedding model can
have a one prefix applied when the model is evaluated as part of an
knn search and a different prefix when ingesting documents.
The _ml/info response now includes two extra fields in its
`limits`:
1. `max_single_ml_node_processors`
2. `total_ml_processors`
These fields are _only_ included if they can be accurately
calculated. If autoscaling is enabled and the ML nodes are
not at their maximum size then these fields _cannot_
currently be accurately calculated. (This could potentially
be improved in the future with additional settings set by
the control plane.)
* Added platform architecture field to TrainedModelMetadata and users of TrainedModelMetadata
* Added TransportVersions guarding for TrainedModelMetadata
* Prevent platform-specific models from being deployed on the wrong architecture
* Added logic to only verify node architectures for models which are platform specific
* Handle null platform architecture
* Added logging for the detection of heterogeneous platform architectures among ML nodes and refactoring to support this
* Added platform architecture field to TrainedModelConfig
* Stop platform-speficic model when rebalance occurs and the cluster has a heterogeneous architecture among ML nodes
* Added logic to TransportPutTrainedModelAction to return a warning response header when the model is paltform-specific and cannot be depoloyed on the cluster at that time due to heterogenous architectures among ML nodes
* Added MlPlatformArchitecturesUtilTests
* Updated Create Trained Models API docs to describe the new platform_architecture optional field.
* Updated/incremented InferenceIndexConstants
* Added special override to make models with linux-x86_64 in the model ID to be platform specific
**Problem:**
For historical reasons, source files for the Elasticsearch Guide's security, watcher, and Logstash API docs are housed in the `x-pack/docs` directory. This can confuse new contributors who expect Elasticsearch Guide docs to be located in `docs/reference`.
**Solution:**
- Move the security, watcher, and Logstash API doc source files to the `docs/reference` directory
- Update doc snippet tests to use security
Rel: https://github.com/elastic/platform-docs-team/issues/208
* isSafeToShutdown checks routing table
* Rebalancer changes and tests
* Update docs/changelog/98406.yaml
* Forcing lifecycle tests to avoid over time case
* Changes and remaining tests
* Adding node service changes
* Finishing unit tests
* Adding wait for completion paramater
* Adding stop deployment integration tests
* Cleaning up code
* Fixing stop deployment test
* Fixing string formatter issue and timeout
* Investigating deadlock
* More testing
* More logging
* Prevent model reloading while stopping
* Fixing compile error
* More code clean up
* Adding test for loading model after stopping
* Addressing review feedback
* Fixing a couple shutdown -> shutdownNow tests
* Adding doc changes and refactoring
Update the ml and transform reference documentation to provide information regarding the new versioning schemes independent from the product versions.
Co-authored-by: István Zoltán Szabó <istvan.szabo@elastic.co>
Add mask_token field to fill_mask of _ml/trained_models.
This change will enable users and Kibana to get the particular mask tokens needed for deployed models by adding a mask_token field to the GET _ml/trained_models API, as an enhancement to support kibana#159577.
Many multi-lingual and newer models use a tokenization scheme similar to
sentence-piece. This PR adds support for one of those tokenization
schemes, XLMRoBERTa.
The main changes are: - Support for xlm_roberta tokenization
configuration - Adding `scores` to the vocabulary document stored,
requiring that scores be the same size as the vocabulary - Adding a new
flat text file to resources that is the spm char normalizer.
Adds a new include flag definition_status to the GET trained models API.
When present the trained model configuration returned in the response
will have the new boolean field fully_defined if the full model definition
is exists.
This PR adds a new field, `_meta`, to the data frame
analytics configuration.
The `_meta` field stores an arbitrary key-value map.
Keys are strings. Values are arbitrary objects
(possibly also maps).
The `_meta` field can be updated using the data frame
analytics `_update` endpoint.
The companion PR to elastic/ml-cpp#2440 adds processing of multimodal_distribution field in the anomaly score explanation. I added a changelog entry in the ml-cpp PR hence I mark this PR as a non-issue.
These docs previously implied that you could update datafeed
properties while the datafeed was running, but then would have
to stop and restart it for the changes to take effect.
In fact datafeed updates can only be made while the datafeed is
stopped (and this has been the case for many years, if not forever).
This prevents docs files from *starting* with a "response" because when
that happens the response is converted to an assertion and appended
to the last snippet that was processed. If that last snipper was in a
different file then it's very hard to reason about the tests. That goes
double because the order we iterate files isn't defined....
Anyway! This adds a guard in the build, removes the offending
"response", and reenables the tests that we'd thought we failing here.
Closes#91081
David Kyle
Max Hniebergall
Lisa Cawley
Liam Thompson
Niels Bauman
István Zoltán Szabó
Mary Gouseti
David Roberts
Abdon Pijpelink
DeDe Morton
James Rodewig
Jonathan Buttner
Ed Savage
Benjamin Trent
debadair
Valeriy Khakhutskyy
Nik Everett