APIs which perform cluster state updates typically accept the
`?master_timeout=` and `?timeout=` parameters to respectively set the
pending task queue timeout and the acking timeout for the cluster state
update. Both of these parameters accept the value `-1`, but
`?master_timeout=-1` means to wait indefinitely whereas `?timeout=-1`
means the same thing as `?timeout=0`, namely that acking times out
immediately on commit.
There are some situations where it makes sense to wait for as long as
possible for nodes to ack a cluster state update. In practice this wait
is bounded by other mechanisms (e.g. the lag detector will remove the
node from the cluster after a couple of minutes of failing to apply
cluster state updates) but these are not really the concern of clients.
Therefore with this commit we change the meaning of `?timeout=-1` to
mean that the acking timeout is infinite.
* Remove `es-test-dir` book-scoped variable
* Remove `plugins-examples-dir` book-scoped variable
* Remove `:dependencies-dir:` and `:xes-repo-dir:` book-scoped variables
- In `index.asciidoc`, two variables (`:dependencies-dir:` and `:xes-repo-dir:`) were removed.
- In `sql/index.asciidoc`, the `:sql-tests:` path was updated to fuller path
- In `esql/index.asciidoc`, the `:esql-tests:` path was updated idem
* Replace `es-repo-dir` with `es-ref-dir`
* Move `:include-xpack: true` to few files that use it, remove from index.asciidoc
Specifying `?master_timeout=-1` on an API which performs a cluster state
update means that the cluster state update task will never time out
while waiting in the pending tasks queue. However this parameter is also
re-used in a few places where a timeout of `-1` means something else,
typically to timeout immediately. This commit fixes those places so that
`?master_timeout=-1` consistently means to wait forever.
The Query API Key Information endpoint supports aggs since #104895.
But some lang clients actually use the `aggregations` keyword in requests,
as the preferred synonym to `aggs`.
This PR adds support for the `aggregations` request keyword as a synonym
for the existing `aggs` term.
Closes#106839
The typed_keys request parameter is the canonical parameter,
that's also used in the regular index _search enpoint, in order to
return the types of aggregations in the response.
This is required by typed language clients of the _security/_query/api_key
endpoint that are using aggregations.
Closes#106817
In this PR we introduce the API that will expose the global retention configuration and will allow users to take advantage of it.
These APIs are protected by the dedicated introduced privileges:
`manage_data_stream_global_retention` or higher, which allows all operations on the global retention configuration
`monitor_data_stream_retention` or higher, which allows the retrieval of the global retention configuration.
This PR is the final PR that makes the global retention available for our users.
Add new optional request option, `with_profile_uid`,
to the Get and Query API Key Information endpoints,
to return the API keys owner users' profile uid.
Closes#98939
Defines new inference_user and inference_admin roles with the
related cluster privileges manage_inference and monitor_inference.
inference_user can list the models and preform inference,
inference_admin can do the same plus create and delete models
* Add modelId and modelText to KnnVectorQueryBuilder
Use QueryVectorBuilder within KnnVectorQueryBuilder to make it
possible to perform knn queries also when a query vector is not
immediately available. Supplying a text_embedding query_vector_builder
with model_text and model_id instead of the query_vector will result
in the generation of a query_vector by calling inference on the
specified model_id with the supplied model_text (during query
rewrite). This is consistent with the way query vectors are built
from model_id / model_text in KnnSearchBuilder (DFS phase).
This enhancement adds a new abstraction to the _search API called "retriever." A
retriever is something that returns top hits. This adds three initial retrievers called
"standard", "knn", and "rrf". The retrievers use a parser-only approach where they
are parsed and then translated into a SearchSourceBuilder to execute the actual
search.
---------
Co-authored-by: Mayya Sharipova <mayya.sharipova@elastic.co>
When querying or getting API key information, ES returns the key owner's
username and realm (i.e. the realm name that authenticated the username
that last updated the API key).
This PR adds the realm_type to the information on the key's owner.
Adds support for the aggs request body parameter to the Query API Key Information API.
This parameter works identically to the well known eponymous parameter of the _search endpoint,
but the set of allowed aggregation types as well as the field names allowed is restricted.
This adds support for the `match` query type to the Query API key Information API.
Note that since string values associated to API Keys are mapped as `keywords`,
a `match` query with no analyzer parameter is effectively equivalent to a `term` query
for such fields (e.g. `name`, `username`, `realm_name`).
Relates: #101691
- Introduce new internal system index called .connector-secrets
- Add GET and POST requests for connector secrets
- Add permission sets for read and write connector secrets
- Introduce new internal system index called .connector-secrets
- Add GET and POST requests for connector secrets
- Create read_connector_secrets and write_connector_secrets role permissions
* Add extract match ranges functionality to Grok.
* TestGrokPatternAction and Request
* TestGrokPattern response
* Update docs/changelog/104394.yaml
* Polish validation error message
* Improve test_grok_pattern API
* Add explicit CharSet
* Add endpoint to operator constants
* Add TransportTestGrokPatternActionTests
* REST API spec
* One more TransportTestGrokPatternActionTest
* Fix API spec
* Refactor REST API spec
* Polish code
* Replace TransportTestGrokPatternActionTests by a YAML REST test
* Add ecs_compatibility
* Always return arrays in the API
* Documentation
* YAML test for ecs_compatibility
* Rename doc fileø
* serverless scope
* Fix docs (hopefully)
* Update docs/reference/rest-api/index.asciidoc
Co-authored-by: István Zoltán Szabó <istvan.szabo@elastic.co>
* Add "text structure APIs" header in docs TOC
* Move file
* Remove test grok from main index
* typo
* Nested APIs underneath text structure
---------
Co-authored-by: István Zoltán Szabó <istvan.szabo@elastic.co>
This adds support for the simple_query_string query type to the Query API key Information API.
In addition, this also adds support for querying all the API Key metadata fields simultaneously,
rather than requiring each to be specified, such as metadata.x, metadata.y, etc.
Relates: #101691
This adds support for the type parameter to the Query API key API.
The type for an API Key can currently be either rest or cross_cluster.
Relates: #101691
This commit adds detailed API docs for the ES|QL async apis, as well as moving the narrative to the ES|QL specific REST docs. Additionally, a few top-level lists are reflowed to reflect that there are now multiple APIs.
Adding equations to the docs around how to best calculate similarity & score. The similarity parameter for search was added in 8.8.
The max-inner-product mentions will be removed for all versions before 8.11 when backporting.
closes: https://github.com/elastic/elasticsearch/issues/102924
Introduces support for JWTs to the grant API Key API.
Callers can now pass-in a JWT in the request, like:
POST /_security/api_key/grant
{
"grant_type": "access_token",
"access_token" : "some.signed.JWT",
"client_authentication": { // optional
"scheme": "SharedSecret",
"value": "ES-Client-Authentication header value after scheme"
}
}
The JWT will be authenticated by a backing JWT realm and
a new API Key will be returned for the authenticated user.
* Add inference counts by NLP model to the machine learning usage stats.
* Update docs/changelog/101915.yaml
* Add inference_counts_by_model to yamlRestTest.
* Strip leading dot from internal model IDs.
* Add last access and task type to the stats by model.
* Change stats_by_model for map to list
* Simplify code.
* Fix style
With this commit we remove the `auto_configure` privilege for the Fleet
service account that targets profiling-related indices. This privilege
was needed to automatically create indices and data streams in the past
but as this managed by the Elasticsearch plugin, there is no need to
grant this privilege to Fleet-managed components.
* Break out 'Limitations' into separate page
* Add REST API docs
* Restructure commands, functions, and operators refs
* Add placeholder for getting started guide
* Group 'Syntax', 'Metafields', and 'MV fields' under 'Language'
* Add placeholder for Kibana page
* Add link from landing page
* Apply uniform formatting to ACOS, CASE, and DATE_PARSE function refs
* Reword default LIMIT
* Add support for COUNT(*)
* Move 'Commands' and 'Functions and operators' to individual pages
---------
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
* [DOCS] Creates documentation structure.
* [DOCS] Adds PUT inference API docs and part of GET inference API docs.
* [DOCS] Fixes complaining CI.
* [DOCS] Adds GET and DELETE API docs for inference API.
* [DOCS] Adds POST inference API docs.
* Apply suggestions from code review
This change introduces a new privilege monitor_enrich. Users are
required to have this privilege in order to use the enrich functionality
in ESQL. Additionally, it eliminates the need to use the enrich_origin
when executing enrich lookups. The enrich_origin will only be used when
resolving enrich policies to prevent warnings when accessing system
indices directly.
Closes#98482
**Problem:**
For historical reasons, source files for the Elasticsearch Guide's security, watcher, and Logstash API docs are housed in the `x-pack/docs` directory. This can confuse new contributors who expect Elasticsearch Guide docs to be located in `docs/reference`.
**Solution:**
- Move the security, watcher, and Logstash API doc source files to the `docs/reference` directory
- Update doc snippet tests to use security
Rel: https://github.com/elastic/platform-docs-team/issues/208
This adds a `data_lifecycle` section to the _xpack/usage API, giving basic information about data lifecycles in the cluster. The data looks something like:
```
"data_lifecycle": {
"available": true,
"enabled": true,
"lifecycle": {
"count": 1,
"default_rollover_used": true,
"retention": {
"minimum_millis": 360000,
"maximum_millis": 360000,
"average_millis": 360000.0
}
}
}
```
The previous fix (#95565) didn't work since the section was misplaced.
Note that this test runs only on snapshot build so I tested manually and the failure is now related to remote_clusters section missing.
Closes#95603
With PR we introduce CRUD endpoints which update/delete the data lifecycle on the data stream level. When this is updated it will apply at the next DLM run to all the backing indices that are managed by DLM.
This adds a new parameter to `knn` that allows filtering nearest neighbor results that are outside a given similarity.
`num_candidates` and `k` are still required as this controls the nearest-neighbor vector search accuracy and exploration. For each shard the query will search `num_candidates` and only keep those that are within the provided `similarity` boundary, and then finally reduce to only the global top `k` as normal.
For example, when using the `l2_norm` indexed similarity value, this could be considered a `radius` post-filter on `knn`.
relates to: https://github.com/elastic/elasticsearch/issues/84929 && https://github.com/elastic/elasticsearch/pull/93574
Every node (post `8.7`) collects stats from every health-api request it receives. We extend the `_xpack/usage` endpoint to expose these stats. When a node receives the request it will fan out to collect data from all other nodes, merge them and expose them. If the cluster is not fully upgraded, it will signal it with the `available` flag set to`false`.
Added Cartesian support for centroid aggregation
* First draft of cartesian-centroid docs
However, this is largely a duplicate of geo-centroid docs since they are essentially identical behaviour. We should consider merging them.
* Work on isAggregatable caused a minor logic conflict. When that work was done, Point and Shape were not aggregatable, but now they are.
This removes "data streams" from the docs for the `index`, `delete`,
and `update` actions because data streams only support the `update`
action.
Closes#87231
The current `ignore_unavailable` definition is a bit misleading. The parameter primarily determines if a request that targets a missing or closed index returns an error.
florent-leborgne
David Turner
Liam Thompson
Albert Zaharovits
Parker Timmins
Mary Gouseti
David Kyle
Bogdan Pintea
Stef Nestor
Tommaso Teofili
Panagiotis Bailis
Jack Conradson
István Zoltán Szabó
Johannes Fredén
Navarone Feekery
Jan Kuipers
Chris Hegarty
Benjamin Trent
Jedr Blaszyk
Andrei Stefan
Fabio Busatto
Abdon Pijpelink
Daniel Mitterdorfer
Keith Massey
Nhat Nguyen
James Rodewig
ChrisHegarty
Yang Wang
elasticsearchmachine
Kathleen DeRusso
Carlos Delgado
Marantidis Kiriakos
Sloane Perrault
Andrei Stefan
debadair
Jim Ferenczi
Jean-Pierre Matsumoto
Przemysław Witek
Andrei Dan
Craig Taverner
Andrey
Nik Everett
Lisa Cawley
Yannick Welsch