This adds tests, supported types, and a signature image for `to_string`
and `to_version`. It also fixes the resolution of functions who's names
contain an `_`
Finally, it updates the docs for `ceil` to render the image more nicely.
Add the 'right' function, which extracts a substring beginning from its
right end (opposite function of 'left').
---------
Co-authored-by: Alexander Spies <alexander.spies@elastic.co>
* [DOCS] Add 'Troubleshooting an unstable cluster' to nav
* Adjust docs links in code
* Revert "Adjust docs links in code"
This reverts commit f3846b1d78.
---------
Co-authored-by: David Turner <david.turner@elastic.co>
This change adds a `index.look_back_time` index setting that sets the `index.time_series.start_time` setting for the first backing index when a data stream is created.
This allows accepting data that is older for initial indexing without changing the `index.look_ahead_time` setting. This setting also controls the `index.time_series.end_time` setting and would affect rollovers as well.
The default for the `index.look_back_time` is `2h`, which means documents with `@timestamp` up to 2 hours after creation of the data stream are allowed to be indexed. This is the same as is without this change, because `index.look_ahead_time` is used to set `index.time_series.start_time` of the first backing index.
Closes#98463
👋 howdy, team! Expanding reference to [internal](https://github.com/elastic/cloud/pull/118105) update, we've just confirmed ILM requires the repository name to be the same among migrating clusters. This is a hard block for Searchable Snapshots which requires un-Searchable-Snapshotting or redoing migration to resolve.
**Problem:**
- The `elasticsearch-reset-password` commands in the ES Docker install docs are missing the required `-u` flag
- The `ifeval` blocks in the Kibana section of the ES Docker install docs aren't rendering correctly in released docs.
**Solution:**
- Add the `-u` flag to `elasticsearch-reset-password` examples
- Fix the Asciidoc syntax to correctly render the `ifeval` blocks. Example:
Rel: https://github.com/elastic/elasticsearch/pull/99112
Currently pinned queries require either the `ids` or `docs` parameter.
`docs` allows pinning documents from specific indices. However for
`docs` the `_index` field is always required:
```
GET test/_search
{
"query": {
"pinned": {
"organic": {
"query_string": {
"query": "something"
}
},
"docs": [
{ "_id": "1" }
]
}
}
}
```
returns an error:
```
{
"error": {
"root_cause": [
{
"type": "parsing_exception",
"reason": "[10:22] [pinned] failed to parse field [docs]",
"line": 10,
"col": 22
}
],
"type": "parsing_exception",
"reason": "[10:22] [pinned] failed to parse field [docs]",
"line": 10,
"col": 22,
"caused_by": {
"type": "x_content_parse_exception",
"reason": "[10:22] [pinned] failed to parse field [docs]",
"caused_by": {
"type": "illegal_argument_exception",
"reason": "Required [_index]"
}
}
},
"status": 400
}
```
The proposal here is to make `_index` optional. I don't think we have a
strong requirement for making `_index` required, when it was initially
introduced in https://github.com/elastic/elasticsearch/pull/74873, we
mostly wanted the ability to pin docs from specific indices.
Making `_index` optional can give more flexibility to use a combination
of pinned documents from specific indices or just document ids. This
change can also help with pinned query rules. Currently pinned query
rules can accept either `ids` or `docs`. If multiple pinned query rules
match and they use a combination of `ids` and `docs`, we cannot build a
pinned query and we would need to return an error. This is because a
pinned query cannot accept both `ids` and `docs`. By making `_index`
optional we would no longer need to return an error when pinned query
rules use a combination of `ids` and `docs`, because we can easily
translate `ids` in `docs`.
The following pinned queries would be equivalent:
```
GET test/_search
{
"query": {
"pinned": {
"organic": {
"query_string": {
"query": "something"
}
},
"docs": [
{ "_id": "1" }
]
}
}
}
GET test/_search
{
"query": {
"pinned": {
"organic": {
"query_string": {
"query": "something"
}
},
"ids": [1]
}
}
}
```
The scores should be consistent when using a combination of _docs that
might use _index or not - see example
<details> <summary>Example </summary>
```
PUT test-1/_doc/1 { "title": "doc 1" }
PUT test-1/_doc/2 { "title": "doc 2" }
PUT test-2/_doc/1 { "title": "doc 1" }
PUT test-2/_doc/3 { "title": "lalala" }
POST test-1,test-2/_search { "query": { "pinned": {
"organic": { "query_string": { "query": "lalala"
} }, "docs": [ { "_id": "2", "_index": "test-1" },
{ "_id": "1" } ] } } }
```
response:
```
{ "took": 1, "timed_out": false, "_shards": { "total": 2,
"successful": 2, "skipped": 0, "failed": 0 }, "hits": {
"total": { "value": 4, "relation": "eq" },
"max_score": 1.7014124e+38, "hits": [ { "_index":
"test-1", "_id": "2", "_score": 1.7014124e+38,
"_source": { "title": "doc 2" } }, {
"_index": "test-1", "_id": "1", "_score": 1.7014122e+38,
// same score as doc with id 1 from test-2 "_source": {
"title": "doc 1" } }, { "_index": "test-2",
"_id": "1", "_score": 1.7014122e+38, // same score as doc with
id 1 from test-1 "_source": { "title": "doc 1"
} }, { "_index": "test-2", "_id": "3",
"_score": 0.8025915, // organic result "_source": {
"title": "lalala" } } ] } }
```
</details>
For query rules, if we have two query rules that both match and use a
combination of `ids` and `pinned`:
```
PUT _query_rules/test-ruleset
{
"ruleset_id": "test-ruleset",
"rules": [
{
"rule_id": "1",
"type": "pinned",
"criteria": [
{
"type": "exact",
"metadata": "query_string",
"value": "country"
}
],
"actions": {
"docs": [
{ "_index": "singers", "_id": "1" }
]
}
},
{
"rule_id": "2",
"type": "pinned",
"criteria": [
{
"type": "exact",
"metadata": "query_string",
"value": "country"
}
],
"actions": {
"ids": [
2
]
}
}
]
}
```
and the following query:
```
POST singers/_search
{
"query": {
"rule_query": {
"organic": {
"query_string": {
"default_field": "name",
"query": "country"
}
},
"match_criteria": {
"query_string": "country"
},
"ruleset_id": "test-ruleset"
}
}
}
```
then this would get translated into the following pinned query:
```
POST singers/_search
{
"query": {
"pinned": {
"organic": {
"query_string": {
"default_field": "name",
"query": "country"
}
},
"docs": [
{ "_index": "singers", "_id": "1" },
{"_id": 2 }
]
}
}
}
```
I think we can also simplify the pinned query rule so that it only
receives `docs`:
```
PUT _query_rules/test-ruleset
{
"ruleset_id": "test-ruleset",
"rules": [
{
"rule_id": "1",
"type": "pinned",
"criteria": [
{
"type": "exact",
"metadata": "query_string",
"value": "country"
}
],
"actions": {
"docs": [
{ "_id": "1" },
{ "_id": "2", "_index": "singers" }
]
}
}
]
}
```
**Problem:**
The [Kibana Docker install docs](https://www.elastic.co/guide/en/kibana/master/docker.html) duplicate the Elasticsearch Docker instructions. This makes the two doc sets harder to maintain. For example, the docs currently use different container names, which makes them incompatible.
**Solution:**
Adds Kibana setup instructions to the Elasticsearch Docker install docs. This will let us eventually merge the two doc sets.
**Issues:**
Rel: https://github.com/elastic/platform-docs-team/issues/182
* [DOCS] Remote cluster troubleshooting guide
* Fix test failures
* Apply suggestions from code review
Co-authored-by: Yang Wang <ywangd@gmail.com>
* Review feedback
* Group issues under 'common' and 'API key'
* Apply suggestions from code review
Co-authored-by: Yang Wang <ywangd@gmail.com>
---------
Co-authored-by: Yang Wang <ywangd@gmail.com>
* Add indices to GET search_application endpoint
* Update x-pack/plugin/ent-search/src/main/java/org/elasticsearch/xpack/application/search/SearchApplication.java
Co-authored-by: Carlos Delgado <6339205+carlosdelest@users.noreply.github.com>
* Update x-pack/plugin/ent-search/src/main/java/org/elasticsearch/xpack/application/search/SearchApplication.java
Co-authored-by: Carlos Delgado <6339205+carlosdelest@users.noreply.github.com>
* Update x-pack/plugin/ent-search/src/main/java/org/elasticsearch/xpack/application/search/SearchApplication.java
Co-authored-by: Carlos Delgado <6339205+carlosdelest@users.noreply.github.com>
* Add indices to GET search_application endpoint
---------
Co-authored-by: Carlos Delgado <6339205+carlosdelest@users.noreply.github.com>
* [DOCS] Expand the step that enables the remote cluster server
* Update docs/reference/modules/cluster/remote-clusters-api-key.asciidoc
* Reword
* Reword
* isSafeToShutdown checks routing table
* Rebalancer changes and tests
* Update docs/changelog/98406.yaml
* Forcing lifecycle tests to avoid over time case
* Changes and remaining tests
* Adding node service changes
* Finishing unit tests
* Adding wait for completion paramater
* Adding stop deployment integration tests
* Cleaning up code
* Fixing stop deployment test
* Fixing string formatter issue and timeout
* Investigating deadlock
* More testing
* More logging
* Prevent model reloading while stopping
* Fixing compile error
* More code clean up
* Adding test for loading model after stopping
* Addressing review feedback
* Fixing a couple shutdown -> shutdownNow tests
* Adding doc changes and refactoring
This commit tracks progress for each shard search by cluster alias
using a new SearchProgressListener (CCSSingleCoordinatorSearchProgressListener).
Both sync and async CCS searches use this new progress listener when
minimize_roundtrips=false.
Two of the SearchProgressListener method had to be extended to allow tracking
per-cluster took values (TransportSearchAction.SearchTimeProvider) and
whether searches timed out (by passing in QuerySearchResult to the onQueryResult
listener method).
This commit brings parity between minimize_roundtrips=true and false to have
the same _cluster/details sections in CCS search responses.
Note that there are still a few differences between minimize_roundtrips=true and false.
1. The per-cluster took value for minimize_roundtrips=true is accurate, but the
for 'false' it is only measured at the granualarity of each partial reduce,
so the per cluster took time is overestimated in basically all cases.
2. For minimize_roundtrips=true, a skip_unavailable=false cluster that disconnects
during the search or has all searches on all shards fail, will cause the entire
search to fail. This is (still) not true for minimize_roundtrips=false. The search
is only failed if the skip_unavailable=false cluster cannot be connected to at the
start of the search. (This will likely be changed in a follow up ticket that implements
fail-fast logic for in-progress searches that should fail due to a skip_unavailable=true
cluster failing.)
3. The shard accounting for minimize_roundtrips=false is always accurate (total shard counts
are known at the start of the search). For minimize_roundtrips=true, the shard accounting
is only accurate per cluster unless all clusters have successful (or partially successful)
searches. For clusters that have failures we do not have shard count info.
* [DOC+] snapshot-restore single index example
👋🏼 howdy, team! I'd like to append an example to snapshot-restore a single index. Support usually points users to [this page](https://www.elastic.co/guide/en/elasticsearch/reference/master/restore-snapshot-api.html) but then users attempt the `rename_pattern` example (which makes sense!). I'd like to point them to a more literal "close index > restore on that index" example in the future.
* Fix test failure and reword
---------
Co-authored-by: Abdon Pijpelink <abdon.pijpelink@elastic.co>
- Deduplicates the JVM sizing instructions
- Links the `-m` tag tip section to the JVM sizing instructions
- Replaces the fully typed out container registry to {docker-image}
Co-authored-by: James Rodewig <james.rodewig@elastic.co>
* [DOCS] Remote cluster migration guide
* Review feedback
* Clarify that any extra local privileges will be suppressed by the cross-cluster API key’s privileges
CI will skip building them. Lot's of CI machines don't have font support
so they can't generate these. But all local machine have a GUI so they
can.
Also, super-lazy initialize the font so CI don't bump into it by
accident.
Closes#99018
`dot_product` requires vectors to be unit-length. Previously, we would
check that vectors were unit-length and throw if they were not.
Instead, we will now auto-normalize vectors as they are indexed.
`cosine` will continue to behave as usual, not normalizing the vectors.
closes: https://github.com/elastic/elasticsearch/issues/98935
* Add link to Elasticsearch labs ELSER Python notebook
* Fix typos
* Use {es} variable
Co-authored-by: István Zoltán Szabó <istvan.szabo@elastic.co>
---------
Co-authored-by: István Zoltán Szabó <istvan.szabo@elastic.co>
Problem: The current install docs contain several `ifeval` statements that hide commands and code snippets based on the branch's release state. These statements make the Asciidoc hard to read and maintain. It also makes doc changes difficult to preview.
Solution: Remove `ifeval` statements that hide commands or code snippets. Leave in any `ifeval` statements used to add warnings.
@nik9000 Recheck out the main branch. Refactor the 'left' function to
cut the prefix string in place. But I meet a adversity that left failed
the test case 'testEvaluateInManyThreads'. I find that in multiple
thread situation, ` EvalOperator.ExpressionEvaluator eval =
evalSupplier.get(); for (int c = 0; c < count; c++) {
assertThat(toJavaObject(eval.eval(page), 0), testCase.getMatcher()); } `
toJavaObject function return a BytesRef with length=2, content is
[81,89]. However, assertThat function in junit4 receive the BytesRef
parameters that its length is 10. Can you give me some clues? I can't
find which variable is mutual.
Rerun failed test case's command: `gradlew ':x-pack:plugin:esql:test'
--tests
"org.elasticsearch.xpack.esql.expression.function.scalar.string.LeftTests.testEvaluateInManyThreads
{TestCase=Left basic test}" -Dtests.seed=44459C172243712
-Dtests.locale=lv-LV -Dtests.timezone=Asia/Irkutsk -Druntime.java=20`
Problem:
The current [single-node Docker instructions](https://www.elastic.co/guide/en/elasticsearch/reference/current/docker.html#docker-cli-run-dev-mode) are overly verbose and contain a lot of unneeded info about security internals.
Solution:
- Restructure the above docs to focus primarily on actionable steps.
- Test the docs across operating systems (Mac, Linux, Windows) to ensure they work.
- Removes duplicated security autoconfiguration output from the docs. This is difficult to keep updated and makes the docs longer.
- Encourages the user to store the `elastic` password as an environment variable. Users don't need to rely on curl's password prompts.
- Removes unused `api-call-widget` files. These aren't published anywhere in the docs currently.
Add the unary scalar function CEIL.
Analogously to FLOOR, it rounds up its argument.
- Implement CEIL, add it to the function registry and make sure it is serializable.
- Add csv tests, unit tests and docs.
- Add additional csv tests with different data types and some edge cases for both CEIL and FLOOR
- Add unit tests and update docs for FLOOR.
Locks the railroad diagrams to always use the same font, this one named
`roboto mono`. This makes sure that when we render the railroad diagrams
we always size them the same way. Because everyone has a copy of roboto
mono. Because gradle resolves that dependency.
This generates a "railroad diagram" svg image that can be embedded into
the docs for any function to explain it's syntax. It's basic, but it's
something we can iterate on.
It also generates a table of supported types from the list of types that
we test. It can be included in the docs for reference as well.
* New docs structure for remote clusters
* Fix broken cross-book link errors
* More broken cross-book link errors
* Remove redirects for new pages
* Link to generic remote cluster docs instead
* Drop 'API' from the abbreviated title
* Add 'Establish trust with a remote cluster' section
* Restructure 'Establish trust' section into Prprequisite/local/remote instructions
* Add 'Configure roles and users' section
* Add 'Connect to a remote cluster' section
* Move version compatibility to prerequisites
* Fix test errors
* Incorporate review feedback
* Mention version 8.10 or later in the intro for API keys
* Add license prerequisite
Currently the `GET target/_lifecycle/explain` API only works for
indices. In this PR we extend this behaviour to allow the target to be a
data stream so we can get the overview lifecycle status for all the
backing indices of a data stream.
Report node "roles" in the /_cluster/allocation/explain response.
Nodes with limited sets of roles may affect shard distribution in ways
users did not originally consider, so it is helpful to surface this
information along with node allocation decision explanations.
* Add 'dataset' size to cat indices and cat shards
This adds the `dataset` computed size for the `/_cat/indices` and `/_cat/shards` APIs. This new
column is reported by default.
Resolves#95092
This makes the data stream lifecycle generally available. This will allow
data streams to take advantage of a native simplified and resilient
lifecycle implementation.
Here we add support for the following two ESQL functions:
* LTRIM: remove leading spaces from a string
* RTRIM: remove trailing spaces from a string
We also fix an issue with the handling of unicode white spaces. We
make use of unicode code points to identify unicode whitespace
characters instead of relying on ASCII codes.
Moreover, iterating bytes in a Unicode string needs to consider
that some Unicode characters are encoded using multiple bytes.
* First version
* Spotless, I liked my version better
* Fix param default values
* Add a supplier for default value to ensure it's calculated correctly
* Can't improve this without breaking tests
* Added checks for not specifying a body in PUT requests
* Fix default provider for enum params
* Added yaml test
* Changed docs and fix TODO
* Removing synonyms changes
* Added separate methods for providing default value as suppliers in enums
* Fixed test
* Add a supplier for default value to ensure it's calculated correctly
* Added checks for not specifying a body in PUT requests
* Remove synonyms changes
* Remove some supplier changes
* Better call enumParam with supplier version
* Fix compiler error on supplier
* Apply validators or requires depending on index version
* Solved BWC tests that involved using validators instead of requiresParameters
* Add tests
* Spotless
* Update docs/changelog/98268.yaml
* Update changelog
* Update docs/changelog/98268.yaml
* PR comments
* PR feedback
* Serialize index only for new index versions
---------
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
* Change example field in rule query guide
* Change fuzzy to contains to get tests to work
---------
Co-authored-by: Kathleen DeRusso <kathleen.derusso@elastic.co>
* Sqrt function for ESQL
Introduces a unary scalar function for square root, which is a thin
wrapper over the Java.Math implementation.
* Fix area for ESQL integration changelog.
* Restore changelog.
* Restore area in changelog.
This PR extends the assumptions we make about database file availability to all database file
names instead of the default ones we host at Elastic. When creating a geo ip processor with
a database name that is not recognized we unilaterally convert the processor to one that
tags documents with a missing database message until the database file requested is
downloaded or provided via the manual configuration route. This allows a pipeline to be
created and for the download service to be started, potentially sourcing the needed files.
---------
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
This commit corrects the following issues with JWT and OIDC `jwkset_path` documentation:
* only https is supported for the JWT realm (OIDC support both https and http)
* JWT realm does not use a file watcher to reload the file every 5 seconds
* simplify "path" to "file name" ..technically it is resolved path, but 99% of the time it will be just
a file name in the config directory and "path" is ambiguous
* remove special mention of using the absolute path in cloud. .. this is an unnecessary implementation
detail and the only setting (of many) that calls out the cloud config directly by absolute path
* ensure the 2 different JWT documentations are the same
* make mention of when the JWT file will be reloaded (it is not backed by the file watcher, only OIDC is)
* Update post-analytics-collection-event.asciidoc
Update API endpoint. Instead of `click`, it should be `search_click`.
Expanded on request body in the API as well.
* Update post-analytics-collection-event.asciidoc
* Update post-analytics-collection-event.asciidoc
* Update post-analytics-collection-event.asciidoc
This change adds the total dense vector count to the output of the indices stats.
This is useful for observability in order to track the number of indexed vectors
in a cluster.
---------
Co-authored-by: Benjamin Trent <ben.w.trent@gmail.com>
In this PR we enable all new data streams to be managed by the data
stream lifecycle by default. This is implemented by adding an empty
`lifecycle: {}` upon new data stream creation.
Opting out is represented by a the `enabled` flag:
```
{
"lifecycle": {
"enabled": false
}
}
```
This change has the following implications on when is an index managed
and by which feature:
| Parent data stream lifecycle| ILM| `prefer_ilm`|Managed by|
|----------------------------|----|----------------|-| | default | yes|
true| ILM| | default | yes| false| data stream lifecycle| |default |
no|true/false|data stream lifecycle| |opt-out or
missing|yes|true/false|ILM| |opt-out or missing|no|true/false|unmanaged|
Data streams that have been created before the data stream lifecycle is
enabled will not have the default lifecycle.
Next steps: - We need to document this when the feature will be GA
(https://github.com/elastic/elasticsearch/issues/97973).
This commit enables concurrent search execution in the DFS phase, which is going to improve resource usage as well as performance of knn queries which benefit from both concurrent rewrite and collection.
We will enable concurrent execution for the query phase in a subsequent commit. While this commit does not introduce parallelism for the query phase, it introduces offloading sequential computation to the newly introduced executor. This is true both for situations where a single slice needs to be searched, as well as scenarios where a specific request does not support concurrency (currently only DFS phase does regardless of the request). Sequential collection is not offloaded only if the request includes aggregations that don't support offloading: composite, nested and cardinality as their post collection method must be executed in the same thread as the collection or we'll trip a lucene assertion that verifies that doc_values are pulled and consumed from the same thread.
## Technical details
This commit introduces a secondary executor, used exclusively to execute the concurrent bits of search. The search threads are still the ones that coordinate the search (where the caller search will originate from), but the actual work will be offloaded to the newly introduced executor.
We are offloading not only parallel execution but also sequential execution, to make the workload more predictable, as it would be surprising to have bits of search executed in either of the two thread pools. Also, that would introduce the possibility to suddenly run a higher amount of heavy operations overall (some in the caller thread and some in the separate threads), which could overload the system as well as make sizing of thread pools more difficult.
Note that fetch, together with other actions, is still executed in the search thread pool. This commit does not make the search thread pool merely a coordinating only thread pool, It does so only for what concerns the IndexSearcher#search operation itself, which is though a big portion of the different phases of search API execution.
Given that the searcher blocks waiting for all tasks to be completed, we take a simple approach of introducing a thread pool executor that has the same size as the existing search thread pool but relies on an unbounded queue. This simplifies handling of thread pool queue and rejections. In fact, we'd like to guarantee that the secondary thread pool won't reject, and delegate queuing entirely to the search thread pool which is the entry point for every search operation anyway. The principle behind this is that if you got a slot in the search thread pool, you should be able to complete your search, and rather quickly.
As part of this commit we are also introducing the ability to cancel tasks that have not started yet, so that if any task throws an exception, other tasks are prevented from starting needless computation.
Relates to #80693
Relates to #90700
Update the ml and transform reference documentation to provide information regarding the new versioning schemes independent from the product versions.
Co-authored-by: István Zoltán Szabó <istvan.szabo@elastic.co>
Fleet is currently hard coded to set index.codec to best_compression (deflate compression). This is good for most data streams, except for data streams were tsdb is enabled. Ideally Fleet doesn't need to set this setting at all and Elasticsearch's default would be good. But unfortunately this isn't the case. It default to default (lz4 - optimised for speed), which in would mean much higher disk space usage. Ideally the default would be default when synthetic source is enabled and otherwise best_compression. Changing this now, would mean a breaking change.
Instead Fleet like to depend on Elasticsearch's internal component templates. To at least abstract some of the internal details away. The metrics-settings is ok for non tsdb, but there is no component template for tsdb metrics. This PR adds this.
Relates to elastic/kibana#160288
* Update field-mapping.asciidoc that Epoch format is not supported as dynamic date format
Update field-mapping.asciidoc that Epoch format is not supported as dynamic date format
* Update docs/reference/mapping/dynamic/field-mapping.asciidoc
Co-authored-by: Abdon Pijpelink <abdon.pijpelink@elastic.co>
---------
Co-authored-by: Abdon Pijpelink <abdon.pijpelink@elastic.co>
* [DOC+] ILM min_age interpretation
👋 hiya, team!
From the time y'all helped me write this [ILM Troubleshooting blog](https://www.elastic.co/blog/troubleshooting-elasticsearch-ilm-common-issues-and-fixes) in 2021 and we later ported errors to [this doc](https://www.elastic.co/guide/en/elasticsearch/reference/master/index-lifecycle-error-handling.html) via https://github.com/elastic/elasticsearch/issues/75849, the remaining top-gotcha user's raise is "Common issue 3" that ILM's `min_age` calculates off rollover time fallback index creation time.
This PR cross-pollinates the blog quote into the docs so that Support can link it to users and so it becomes Google-able.
> Common issue 3: min_age calculation clarification
> When working with customers, I have seen confusion about how min_age works. The min_age must increase between subsequent phases. If rollover is used, min_age is calculated off the rollover date. This is because rollover generates a new index and the new index’s creation date is used in the calculation. Otherwise, min_age is calculated off the original index’s creation date.
* Apply suggestions from code review
---------
Co-authored-by: Abdon Pijpelink <abdon.pijpelink@elastic.co>
* Add tests for query rules
* More tests
* Fix search app tests
* Fix tests
* Add teardown to tests
* Add tests for list search apps call
* Update test in get search application
* Tweak stack trace
* Make response match in test
---------
Co-authored-by: carlosdelest <carlos.delgado@elastic.co>
Here we enable aggregations previously not allowed on fields of type counter.
The decision of enabling such aggregations even if the result is "meaningless"
for counters has been taken to favour TSDB adoption.
Aggregations now allowed, other than the existing ones, include:
* avg
* box plot
* cardinality
* extended stats
* median absolute deviation
* percentile ranks
* percentiles
* stats
* sum
* value count
I included tests for the weighted average and matrix stats aggregations too.
Resolves#97882
Today by default the `SEARCH_COORDINATION` pool is sized at half the
allocated processors, or five if there are more than ten CPUs. Yet, if
we scale up a node to have more than ten CPUs, we probably want to scale
up the number of search coordination threads to match. This commit
removes the limit of five threads.
For CCS searches with ccs_minimize_roundtrips=true, when an error is returned, it is unclear which cluster
caused the problem. This commit adds additional accounting and error information to the search response
for each cluster involved in a cross-cluster search.
The _clusters section of the SearchResponse has a new details section added with an entry for each cluster
(remote and local). It includes status info, shard accounting counters and error information that are added
incrementally as the search happens.
The search on each cluster can be in one of 5 states:
RUNNING
SUCCESSFUL - all shards were successfully searched (successful or skipped)
PARTIAL - some shard searches failed, but at least one succeeded and partial data has been returned
SKIPPED - no shards were successfully searched (all failed or cluster unavailable) when skip_unavailable=true
FAILED - no shards were successfully searched (all failed or cluster unavailable) when skip_unavailable=false
A new SearchResponse.Cluster object has been added. Each TransportSearchAction.CCSActionListener
(one for each cluster) has a reference to a separate Cluster instance and updates once it gets back
information from its cluster.
The SearchResponse.Clusters object only uses the new Cluster object for CCS minimize_roundtrips=true.
For local-only searches and CCS minimize_roundtrips=false, it uses the current Clusters object as before.
Follow on work will change CCS minimize_roundtrips=false to also use the new Cluster model and update
state in the _cluster/details section.
The Cluster objects are immutable, so a CAS operation is required to swap in new state to the
map of Cluster objects held by the `SearchResponse.Clusters` class. This concurrency model is
a little bit of overkill for the minimize_roundtrips=true use case, but it will be necessary for
supporting minimize_roundtrips=false, since updates there will be done per shard, not per cluster.
* [DOCS] Fix formatting issue in cardinality-aggregation.asciidoc
Fixes a header not rendering properly because of a missing newline.
* Update docs/reference/aggregations/metrics/cardinality-aggregation.asciidoc
* More issues
* More issues
* Update docs/reference/aggregations/metrics/cardinality-aggregation.asciidoc
Co-authored-by: James Rodewig <james.rodewig@elastic.co>
---------
Co-authored-by: James Rodewig <james.rodewig@elastic.co>
* [DOCS] Update manual downsampling documentation to use TSDS
* Swap manual and ILM downsampling examples in nav
* Typo
* Update prerequisites based on review feedback
* Warn against deleting the old backing index.
* Clarify counter/gauge results
* Mention that the downsampled type is 'aggregate_metric_double'
This adds the `to_degrees` and `to_radians` functions. It uses the
"convert" function framework because that just felt right - these
convert between radians and degrees after all.
* Stop returning the indices list for GET search app
* Stop returning the indices list for the list search app API
* Stop storing indices list
* Remove indices from system index mapping
* Check for alias in PUT rest tests
* Documentation changes
* Do not check for alias existence since we are already doing get alias
This adds the `query` to the main ESQL task so you can see long running
queries. And adds some docs about it including an example of cancelling
a query.
---------
Co-authored-by: Abdon Pijpelink <abdon.pijpelink@elastic.co>
Abdon Pijpelink
Paweł Krześniak
Joe Gallo
Nik Everett
James Rodewig
Benjamin Trent
Johannes Mahne
dreamquster
Martijn van Groningen
Stef Nestor
Marius Iversen
Craig Taverner
Ioana Tagirta
Kathleen DeRusso
Reza Torabi
Saikat Sarkar
István Zoltán Szabó
Yang Wang
Jonathan Buttner
Michael Peterson
Leandro de Lima Camargo
Tim
Kyle Carter
Bogdan Pintea
Liam Thompson
Alexander Spies
Mary Gouseti
Dianna Hohensee
Peter Dyson
Lee Hinman
Andrei Dan
Roberto Seldner
Carlos Delgado
ChrisHegarty
Simon Cooper
Nick Chow
Andrei Stefan
Kostas Krikellas
James Baiera
Jake Landis
amyjtechwriter
Leaf-Lin
Keith Massey
Jim Ferenczi
Felix Barnsteiner
Chris Hegarty
Luca Cavanna
Ed Savage
Kuni Sen
elasticsearchmachine
Salvatore Campagna
David Turner
David Kilfoyle
Mayya Sharipova