Sometimes our fancy "run this agg as a Query" optimizations end up
slower than running the aggregation in the old way. We know that and use
heuristics to dissable the optimization in that case. But it turns out
that the process of running the heuristics itself can be slow, depending
on the query. Worse, changing the heuristics requires an upgrade, which
means waiting. If the heurisics make a terrible choice folks need a
quick way out. This adds such a way: a cluster level setting that
contains a list of queries that are considered "too expensive" to try
and optimize. If the top level query contains any of those queries we'll
disable the "run as Query" optimization.
The default for this settings is wildcard and term-in-set queries, which
is fairly conservative. There are certainly wildcard and term-in-set
queries that the optimization works well with, but there are other queries
of that type that it works very badly with. So we're being careful.
Better, you can modify this setting in a running cluster to disable the
optimization if we find a new type of query that doesn't work well.
Closes#73426
This moves the public build api and plugins into a separete included build called 'build-tools'
and we removed the duplication of included buildSrc twice (2nd import as build-tools).
The elasticsearch internal build logic is kept in build-tools-internal as included build which allows us better handling of this project that its just being an buildSrc project (e.g. we can reference tasks directly from the root build etc.)
Convention logic applied to both projects will live in a new build-conventions project.
The repository analyzer API spec was incorrectly stored in the plugin
directory rather than in the main `rest-api-spec` directory. This commit
fixes that.
This change adds support for using `search_after` with field collapsing. When
using these in conjunction, the same field must be used for both sorting and
field collapsing. This helps keep the behavior simple and predictable.
Otherwise it would be possible for a group to appear on multiple pages of
results.
Currently search after is handled directly in `CollapsingTopDocsCollector`. As
a follow-up, we could generalize the logic and move support to the Lucene
grouping framework.
Closes#53115.
Implements a V7 compatible typed endpoints for REST for search related apis
retrofits the REST layer change removed in #41640
relates main meta issue #51816
relates types removal issue #54160
Enroll node API can be used by new nodes in order to join an
existing cluster that has security features enabled. The response
of a call to this API contains all the necessary information that
the new node requires in order to configure itself and bootstrap
trust with the existing cluster.
Adds some extra debugging information to make it clear that you are
running `significant_text`. Also adds some using timing information
around the `_source` fetch and the `terms` accumulation. This lets you
calculate a third useful timing number: the analysis time. It is
`collect_ns - fetch_ns - accumulation_ns`.
This also adds a half dozen extra REST tests to get a *fairly*
comprehensive set of the operations this supports. It doesn't cover all
of the significance heuristic parsing, but its certainly much better
than what we had.
Implements a V7 compatible typed endpoints for REST put and get mapping endpoints. Also for Get Field Mappings endpoints.
retrofits the REST layer change removed in #41676
relates main meta issue #51816
relates types removal issue #54160
New api designed for use by apps like Kibana for auto-complete use cases.
A search string is supplied which is used as prefix for matching terms found in a given field in the index.
Supported field types are keyword, constant_keyword and flattened.
A timeout can limit the amount of time spent looking for matches (default 1s) and an `index_filter` query can limit indices e.g. those in the hot or warm tier by querying the `_tier` field
Closes#59137
This prevents the `date_histogram` from running out of memory allocating
empty buckets when you set the interval to something tiny like `seconds`
and aggregate over a very wide date range. Without this change we'd
allocate memory very quickly and throw and out of memory error, taking
down the node. With it we instead throw the standard "too many buckets"
error.
Relates to #71758
The test failing in #71685 does so because under rare circumstance the result
order for match_all can be different. If we want to make assertions on specific
entries in the result, we should sort by a field that imposes a fixed result
ordering.
Closes#71685
I broke composite early termination when reworking how aggregations'
contact for `getLeafCollector` around early termination in #70320. We
didn't see it in our tests because we weren't properly emulating the
aggregation collection stage. This fixes early termination by adhering
to the new contract and adds more tests.
Closes#72078
Co-authored-by: Benjamin Trent <4357155+benwtrent@users.noreply.github.com>
This prevents the `histogram` aggregation from allocating tons of empty
buckets when you set the `interval` to something tiny. Instead, we
reject the request. We're not in a place where we can aggregate over
huge ranges with tiny intervals, but we should fail gracefully when you
ask us to do so rather than OOM.
Closes#71744
Currently when the fleet global checkpoints API returns immediately if
the index is not ready or shards are not ready. This commit modifies the
API to wait on the index and primary shards active up until the timeout
period.
Related to #71449.
This commit revives the documentation of the "Clear Cache" and
"Shard Stats" APIs of Searchable Snapshots that was removed
in #62217. This is a partial revert of the commit b545c55 with
some light wording changes.
Co-authored-by: James Rodewig <40268737+jrodewig@users.noreply.github.com>
This commit adds some per-index statistics to the `SnapshotInfo` blob:
- number of shards
- total size in bytes
- maximum number of segments per shard
It also exposes these statistics in the get snapshot API.
This commit allows to use the include_type_name parameter with the compatible rest api.
The support for include_type_name was previously removed in #48632
relates #51816
types removal meta issue #54160
Adds support for close_to assertion to yaml tests. The assertion can be called
the following way:
```
- close_to: { get.fields._routing: { value: 5.1, error: 0.00001 } }
```
Closes#71303
Currently the `fields` API fetches the root flattened field and returns it in a
structured way in the response. In addition this change makes it possible to
directly query subfields. However, requesting flattened subfields via wildcard
patterns is not possible.
Closes#70605
Since #16661 it is possible to know the total sizes for some Lucene segment files
by using the Node Stats or Indices Stats API with the include_segment_file_sizes
parameter, and the list of file extensions has been extended in #71416.
This commit adds a bit more information about file sizes like the number of files
(count), the min, max and average file sizes in bytes that share the same extension.
Here is a sample:
"cfs" : {
"description" : "Compound Files",
"size_in_bytes" : 2260,
"min_size_in_bytes" : 2260,
"max_size_in_bytes" : 2260,
"average_size_in_bytes" : 2260,
"count" : 1
}
This commit also simplifies how compound file sizes were computed: before
compound segment files were extracted and sizes aggregated with regular
non-compound files sizes (which can be confusing and out of the scope of
the original issue #6728), now CFS/CFE files appears as distinct files.
These new information are provided to give a better view of the segment
files and are useful in many cases, specially with frozen searchable snapshots
whose segment stats can now be introspected thanks to the
include_unloaded_segments parameter.
Revamps the integration tests for the `filter` agg to be more clear and
builds integration tests for the `fitlers` agg. Both of these
integration tests are fairly basic but they do assert that the aggs
work.
This PR introduces a new query called `combined_fields` for searching multiple
text fields. It takes a term-centric view, first analyzing the query string
into individual terms, then searching for each term any of the fields as though
they were one combined field. It is based on Lucene's `CombinedFieldQuery`,
which takes a principled approach to scoring based on the BM25F formula.
This query provides an alternative to the `cross_fields` `multi_match` mode. It
has simpler behavior and a more robust approach to scoring.
Addresses #41106.
Nik Everett
Rene Groeschke
Yang Wang
Julie Tibshirani
David Turner
David Roberts
Julie Tibshirani
István Zoltán Szabó
Przemyslaw Gomulka
Seth Michael Larson
James Rodewig
Ioannis Kakavas
markharwood
Joe Gallo
Christos Soulios
Christoph Büscher
Mark Tozzi
Tomas Della Vedova
Tim Brooks
Tanguy Leroux
Dimitris Athanasiou
Rick Boyd
Igor Motov