Adding a link tag for inner hits behavior and kNN search. Additionally
adding a note that if you are using multiple knn clauses, that the inner
hit name should be provided.
Adding equations to the docs around how to best calculate similarity & score. The similarity parameter for search was added in 8.8.
The max-inner-product mentions will be removed for all versions before 8.11 when backporting.
closes: https://github.com/elastic/elasticsearch/issues/102924
The only docs for this _search param were mentioned in the bool query docs. While it makes contextual sense to have it there, we should also add it as a _search parameter in the search API docs.
It was introduced in 8.8.
Adds new `quantization_options` to `dense_vector`. This allows for
vectors to be automatically quantized to `byte` when indexed.
Example:
```
PUT vectors
{
"mappings": {
"properties": {
"my_vector": {
"type": "dense_vector",
"index": true,
"index_options": {
"type": "int8_hnsw"
}
}
}
}
}
```
When querying, the query vector is automatically quantized and used when
querying the HNSW graph. This reduces the memory required to only `25%`
of what was previously required for `float` vectors at a slight loss of
accuracy.
This is currently only available when `index: true` and when using
`hnsw`
The open point in time API accepts a list of indices and opens a point in time view against those indices.
Like we do already for field caps, this commit allows users to provide an index_filter parameter as part of
the request body, that will be used to execute the can match phase and exclude the indices that can't possibly
match such filter.
Closes#99740
This introduced a new knn query:
- knn query is executed during the Query phase similar to all other queries.
- No k parameter, k defaults to size
- num_candidates is a size of queue for candidates to consider while
search a graph on each shard
- For aggregations: "size" results are collected with total = size * shards.
Aggregations will see size * shards results.
- All filters from DSL are applied as post-filters, except: 1) alias filter
is applied as pre-filter or 2) a filter provided as a parameter
inside knn query.
If a mustache script that outputs badly-formed json is referred to in a render
template request, then the error returned will be a 500 server error, rather than
a 400 json parsing error. This is because rendering templates skips json parsing,
and so the error ends up being caught in the REST layer instead.
This commit changes the template rendering logic to always parse the output of
the script, catching json errors higher in the stack and allowing us to return
the correct status code. This also means that errors are correctly detected and
returned as part of multi search template requests.
Fixes#101477
* Nested dense_vector support
* Adjust nested support based on new lucene version
* fixing after rebase
* fixing some code
* fixing tests adding transport version
* spotless
* [Automated] Update Lucene snapshot to 9.9.0-snapshot-b3e67403aaf
* Adds new max_inner_product vector similarity function (#99527)
Adds new max_inner_product vector similarity function. This differs from dot_product in the following ways:
Doesn't require vectors to be normalized
Scales the similarity between vectors differently to prevent negative scores
* requiring top level filter to be parent filter
* adding docs & fixing tests
* adding and fixing docs
* adding changlog
* removing unnecessary file changes
* removing unused imports
* fixing test
* maybe fix doc tests
* continue tests in docs
* fixing more tests
* fixing tests
---------
Co-authored-by: Jim Ferenczi <jim.ferenczi@elastic.co>
Co-authored-by: elasticsearchmachine <infra-root+elasticsearchmachine@elastic.co>
To help the user know what the possible cluster states are and to
provide an accurate accounting, we added counters summarising
`running`, `partial` and `failed` clusters to the `_clusters` section.
Changes:
- Now in the response is present the number of `running` clusters.
- We split up `partial` and `successful` (before was summed up in the
`successful` counter).
- We now have a counter for `failed` clusters.
- Now `total` is always equal to `running` + `skipped` + `failed` +
`partial` + `successful`.
This commit introduces an AggregatorCollector that contains a finish method which performs aggregation
post-collection and builds the internal aggregation for this collector. This method is called on the worker
thread at the end of the collection phase.
This commit tracks progress for each shard search by cluster alias
using a new SearchProgressListener (CCSSingleCoordinatorSearchProgressListener).
Both sync and async CCS searches use this new progress listener when
minimize_roundtrips=false.
Two of the SearchProgressListener method had to be extended to allow tracking
per-cluster took values (TransportSearchAction.SearchTimeProvider) and
whether searches timed out (by passing in QuerySearchResult to the onQueryResult
listener method).
This commit brings parity between minimize_roundtrips=true and false to have
the same _cluster/details sections in CCS search responses.
Note that there are still a few differences between minimize_roundtrips=true and false.
1. The per-cluster took value for minimize_roundtrips=true is accurate, but the
for 'false' it is only measured at the granualarity of each partial reduce,
so the per cluster took time is overestimated in basically all cases.
2. For minimize_roundtrips=true, a skip_unavailable=false cluster that disconnects
during the search or has all searches on all shards fail, will cause the entire
search to fail. This is (still) not true for minimize_roundtrips=false. The search
is only failed if the skip_unavailable=false cluster cannot be connected to at the
start of the search. (This will likely be changed in a follow up ticket that implements
fail-fast logic for in-progress searches that should fail due to a skip_unavailable=true
cluster failing.)
3. The shard accounting for minimize_roundtrips=false is always accurate (total shard counts
are known at the start of the search). For minimize_roundtrips=true, the shard accounting
is only accurate per cluster unless all clusters have successful (or partially successful)
searches. For clusters that have failures we do not have shard count info.
* Add link to Elasticsearch labs ELSER Python notebook
* Fix typos
* Use {es} variable
Co-authored-by: István Zoltán Szabó <istvan.szabo@elastic.co>
---------
Co-authored-by: István Zoltán Szabó <istvan.szabo@elastic.co>
* New docs structure for remote clusters
* Fix broken cross-book link errors
* More broken cross-book link errors
* Remove redirects for new pages
* Link to generic remote cluster docs instead
* Drop 'API' from the abbreviated title
* Add 'Establish trust with a remote cluster' section
* Restructure 'Establish trust' section into Prprequisite/local/remote instructions
* Add 'Configure roles and users' section
* Add 'Connect to a remote cluster' section
* Move version compatibility to prerequisites
* Fix test errors
* Incorporate review feedback
* Mention version 8.10 or later in the intro for API keys
* Add license prerequisite
* Change example field in rule query guide
* Change fuzzy to contains to get tests to work
---------
Co-authored-by: Kathleen DeRusso <kathleen.derusso@elastic.co>
* Add tests for query rules
* More tests
* Fix search app tests
* Fix tests
* Add teardown to tests
* Add tests for list search apps call
* Update test in get search application
* Tweak stack trace
* Make response match in test
---------
Co-authored-by: carlosdelest <carlos.delgado@elastic.co>
For CCS searches with ccs_minimize_roundtrips=true, when an error is returned, it is unclear which cluster
caused the problem. This commit adds additional accounting and error information to the search response
for each cluster involved in a cross-cluster search.
The _clusters section of the SearchResponse has a new details section added with an entry for each cluster
(remote and local). It includes status info, shard accounting counters and error information that are added
incrementally as the search happens.
The search on each cluster can be in one of 5 states:
RUNNING
SUCCESSFUL - all shards were successfully searched (successful or skipped)
PARTIAL - some shard searches failed, but at least one succeeded and partial data has been returned
SKIPPED - no shards were successfully searched (all failed or cluster unavailable) when skip_unavailable=true
FAILED - no shards were successfully searched (all failed or cluster unavailable) when skip_unavailable=false
A new SearchResponse.Cluster object has been added. Each TransportSearchAction.CCSActionListener
(one for each cluster) has a reference to a separate Cluster instance and updates once it gets back
information from its cluster.
The SearchResponse.Clusters object only uses the new Cluster object for CCS minimize_roundtrips=true.
For local-only searches and CCS minimize_roundtrips=false, it uses the current Clusters object as before.
Follow on work will change CCS minimize_roundtrips=false to also use the new Cluster model and update
state in the _cluster/details section.
The Cluster objects are immutable, so a CAS operation is required to swap in new state to the
map of Cluster objects held by the `SearchResponse.Clusters` class. This concurrency model is
a little bit of overkill for the minimize_roundtrips=true use case, but it will be necessary for
supporting minimize_roundtrips=false, since updates there will be done per shard, not per cluster.
The completion_time is set as the start_time (already present) plus the 'took'
time that is set in the SearchResponse object and only if the isRunning status == false
since took is set even for in-progress searches.
We use the 'took' field because it is based on relative time, not absolute wall clock time
which can go backwards due to NTP issues. See the comments in TransportSearchAction about
the SearchTimeProvider for details.
Closes#88640
Before we used to track max_score in collapse when requested (track_scores=true)
or when there is no sort in collapse (see PR#27122). But this feature
was lost through refactoring and changes.
This PR restores this feature.
Closes#97653
Benjamin Trent
Kathleen DeRusso
István Zoltán Szabó
Abdon Pijpelink
Luca Cavanna
Saikat Sarkar
Mayya Sharipova
James Rodewig
Alan Woodward
Liam Thompson
Carlos Delgado
Matteo Piergiovanni
Ignacio Vera
David Pilato
Michael Peterson
Craig Taverner
Nick Chow
Jack Conradson