This proposes introducing two new settings we configure when
downsampling: `index.downsample.origin.name`
`index.downsample.origin.uuid`
These settings will carry on the name and uuid of the first source index
we downsample (like the source.name and source.uuid settings behaved
before this PR).
This also changes the behaviour of `index.downsample.source.name` and
`index.downsample.source.uuid` to always reflect the source of the
current downsampling round.
If an index is downsampled once, both the source and origin settings
will have the same value.
However, for subsequent downsampling operations, a later downsampling
round will have the actual index name and uuid that were used as the
source of the downsampling operation configured e.g.
`downsample-350s-.ds-metrics-foo-2023.08.17-000001` will have the
`index.downsample.source.name` configured to
`downsample-1s-.ds-metrics-foo-2023.08.17-000001` and
`index.downsample.origin.name` configured to
`.ds-metrics-foo-2023.08.17-000001`.
Note that this will help us simplify things in data stream lifecycle as
with subsequent downsampling configuration we currently have to store
the true source of the downsampling operation in the downsample index
metadata (
https://github.com/elastic/elasticsearch/blob/main/modules/data-streams/src/main/java/org/elasticsearch/datastreams/lifecycle/downsampling/ReplaceSourceWithDownsampleIndexTask.java#L173
)
If you agree with this proposal we'll have to manually backport to 8.10
as well to make sure the `generateDownsampleIndexName` method is
consistent across releases
https://github.com/elastic/elasticsearch/blob/main/server/src/main/java/org/elasticsearch/action/downsample/DownsampleConfig.java#L249
Note that the `index.provided_name` setting which is set on indices when
rolling over cannot act like a source (initially I thought we can use
that instead of introducing a new setting, the origin one) because it
contains date-math patterns.
The `StreamOutput` and `StreamInput` APIs are designed so that code
which serializes objects to the transport protocol aligns closely with
the corresponding deserialization code. However today
`StreamOutput#writeCollection` pairs up with a variety of methods on
`StreamInput`, including `readList`, `readSet`, and so on. These methods
are not obviously compatible with `writeCollection` unless you look at
the implementation, and that makes verifying transport protocol code
harder than it needs to be.
This commit renames these methods to `readCollectionAsList`,
`readCollectionAsSet`, and so on, to clarify that they are compatible
with `writeCollection`.
Relates
https://github.com/elastic/elasticsearch/pull/98971#issuecomment-1697289815
We can dry up and shorten a bunch of spots by using the overloads
we already have in the code. Also, added a new overload for writing a
map with string keys.
The doc values in the `GlobalOrdCardinalityAggregator` are shared
among multiple search threads, `search` and `search_worker`.
The search thread also runs the aggregation phase. When an
executor is used the 'search' thread is running `postCollection`, which
uses doc values, while other methods are executed by the `search_worker`
thread, using doc values too. As a result, doc values are concurrently
accessed by different threads. Using doc values concurrently from multiple
threads is not correct since multiple threads end up updating the doc values
state. This breaks access to doc values resulting in different issue depending
on how threads end up being scheduled (prematurely exhausting doc values,
accessing incorrect documents as a result of trying to access docIds not
in the thread owned leaf/segment,...).
The solution here is to:
1. make sure we executed `postCollection in the same thread as other
methods, which is `search` or `search_worker`.
2. make sure we do not call `postCollection` in case the `TimeSeriesIndexSearcher`
is used. In that case `postCollection` is called by `TimeSeriesIndexSearcher`.
Data stream lifecycle runs periodically and we don't want
to spam the logs with error logging on each run, unless
the type of error encountered for an index changes between
runs (we had this logic applied for some operations and
implemented ad-hoc).
This moves the logic that records the error and logs an
error message (if needed) into the ErrorRecordingActionListener.
Note that if listener.onFailure is called the ErrorRecordingActionListener
will handle the logging so no additional logging is needed.
Previous PR #87841 introduces a change where disabling the disk
allocation decider makes a single check to remove existing blocks.
This created a race condition with the test that puts a block that it
expects to not be removed after disabling the disk allocation decider.
A simple fix is to execute the single check before putting the block.
Closes#98855
By default in 8.11, vectors will be indexed by default. However, various
tests that relied on the previous behavior were not updated.
This PR updates those tests.
Related: https://github.com/elastic/elasticsearch/pull/98268
Noticed this when benchmarking FieldCaps transport messages.
The `writeList` alias just adds more lines to the code and makes
profiling more annoying to read, lets remove it.
An optimization introduced in:
https://github.com/elastic/elasticsearch/pull/81985 changed percolator
query behavior.
Users can specify a percolator query which expands fields based on a
wildcard pattern. Just one example is `simple_query_string`, which
allows field names like `"text_*"`. The user expects that this field
name will expand to relevant mapped fields (e.g. "text_foo"). However,
if there are no documents indexed in those fields at the time when the
percolator query is indexed, it doesn't expand to the relevant fields.
Additionally at query time, we may skip expanding fields and not match
the relevant mapped fields if they are considered "empty" (e.g. has no
values in the shard). We should instead allow expansion by indicating
that the field may exist in the shard.
closes: https://github.com/elastic/elasticsearch/issues/98819
Hide the creation of the index searcher from the implementers by changing the signature of
AggregatorTestCase#searchAndReduce and AggregatorTestCase#createAggregationContext to take
an IndexReader instead of an IndexSearcher.
This adds data stream lifecycle service implementation support
for downsampling.
Time series backing indices for a data stream with a lifecycle
that configures downsampling will be marked as read-only,
downsampled, removed from the data stream, replaced with the
corresponding downsample index, and deleted.
Multiple rounds can be configured for a data stream, and the
latest matching round will be the first one to be executed.
If one downsampling operation is in progress, we wait until it's
finished and then we start the next downsampling operation.
Note that in this scenario a data stream could have the following
backing indices:
```
[.ds-metrics-2023.08.22-000002, downsample-10s-.ds-metrics-2023.08.22-000001]
```
If this data stream has multiple rounds of downsampling configured,
the first generation index will subsequently be downsampled again
(and again).
We've seen very large generated exceptions (>5MB). As the Data Stream
Lifecycle error store is in-memory this caps the recorded error message
for each index to 1000 chars.
And in the fetch phase synthesize _id on the fly.
The _id is composed out of a hash of routing fields, tsid and timestamp. These are all properties that can be retrieved from doc values and used to generate the _id on the fly.
Currently the `GET target/_lifecycle/explain` API only works for
indices. In this PR we extend this behaviour to allow the target to be a
data stream so we can get the overview lifecycle status for all the
backing indices of a data stream.
* Kibana system index does not allow user templates to affect it
* Spotless and compilation fix
---------
Co-authored-by: Athena Brown <athena.brown@elastic.co>
Compare-and-swap operations on a S3 repository are implemented using
multipart uploads. Today to try and avoid collisions we refuse to
perform a compare-and-swap if there are other concurrent uploads in
process. However this means that a node which crashes partway through a
compare-and-swap will block all future register operations.
With this commit we introduce a time-to-live on S3 multipart uploads,
such that uploads older than the TTL now do not block future
compare-and-swap attempts.
Today we assert that every chunked response contains at least one chunk.
In practice it is sometimes useful to yield no chunks too. This commit
permits a zero-chunk chunked response, by converting it into a regular
unchunked response at the start.
Today we assert that every chunked response contains at least one chunk.
In practice it is sometimes useful to yield no chunks too. This commit
permits a zero-chunk chunked response, by converting it into a regular
unchunked response at the start.
When the subobject property is set to false and we encounter an object
while parsing we need a way to understand if its FieldMapper is able to
parse an object. If that's the case we can provide the entire object to
the FieldMapper otherwise its name becomes the part of the dotted field
name of each internal value.
This has being achieved by adding the `supportsParsingObject()` method
to the `FieldMapper` class. This method defaults to `false` since the
majority of FieldMappers do not support parsing objects and is
overwritten to return `true` by the ones that do support objects.
This change swaps test code that directly creates IndexSearcher instances with LuceneTestCase#newSearcher calls
that have the advantage of randomly using concurrency and also randomly use assertion wrappers internally.
While this doesn't guarantee testing the concurrent code path, it should generally increase the likelihood of doing so.
This makes the data stream lifecycle generally available. This will allow
data streams to take advantage of a native simplified and resilient
lifecycle implementation.
This PR adds a mappings version number to system index descriptor, which is intended to replace the use of Version to signal changes in the mappings for the descriptor. This value is required for managed system indices.
Previously, most of the system index descriptors automatically incremented their mapping version with each release, which meant that the mappings would stay up-to-date with any additive changes. Now, developers will need to increment the mapping version whenever there's a change.
* Add MappingsVersion inner class to SystemIndexDescriptor
* Add mappings version to metadata in all system index mappings
* Rename version meta key ('system' -> 'managed')
* Update mappings for ML indices if required
* Trigger ML index mappings updates based on new index mappings version.
---------
Co-authored-by: Ed Savage <ed.savage@elastic.co>
This PR extends the assumptions we make about database file availability to all database file
names instead of the default ones we host at Elastic. When creating a geo ip processor with
a database name that is not recognized we unilaterally convert the processor to one that
tags documents with a missing database message until the database file requested is
downloaded or provided via the manual configuration route. This allows a pipeline to be
created and for the download service to be started, potentially sourcing the needed files.
---------
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
* Mapped field types searchable with doc values
When using TSDB, time series metrics aren't indexed but do have doc values.
Field caps should report those fields as searchable.
In this PR we enable all new data streams to be managed by the data
stream lifecycle by default. This is implemented by adding an empty
`lifecycle: {}` upon new data stream creation.
Opting out is represented by a the `enabled` flag:
```
{
"lifecycle": {
"enabled": false
}
}
```
This change has the following implications on when is an index managed
and by which feature:
| Parent data stream lifecycle| ILM| `prefer_ilm`|Managed by|
|----------------------------|----|----------------|-| | default | yes|
true| ILM| | default | yes| false| data stream lifecycle| |default |
no|true/false|data stream lifecycle| |opt-out or
missing|yes|true/false|ILM| |opt-out or missing|no|true/false|unmanaged|
Data streams that have been created before the data stream lifecycle is
enabled will not have the default lifecycle.
Next steps: - We need to document this when the feature will be GA
(https://github.com/elastic/elasticsearch/issues/97973).
This makes the node ID an explicit part of the TaskInfo object in the response of GetTaskAction. This way it can be filtered out of the response without having to alter the taskId.
This adds a new parameter to LocalClusterSpecBuilder.user(..) to
indicate whether the user should be an operator or not.
Users added with the simplified user(username, password) method are
operators (and have the "_es_test_root" role).
This PR adapts the unified highlighter to use the Weight#matches mode by default when possible.
This is the default mode in Lucene for some time now. For cases where the matches mode won't work (nested and parent-child queries),
the matches mode is disabled automatically.
I didn't expose an option to explicitly disable this mode because that should be seen as an internal implementation detail.
With this change, matches that span multiple terms are highlighted together (something that users asked for years) and the clauses that don't match the document are ignored.
Some queries need to be rewritten before they support Weights; TermsQuery
has recently changed to require this, for example. The GlobalAggregator was
not calling rewrite() on possible alias filters, meaning that an index alias with
a terms filter would cause an exception when combined with a global agg.
Fixes#98076
In serverless we will like to report (meter and bill) upon a document ingestion. The metering should be agnostic to a document format (document structure should be normalised) hence we should allow to create XContentParsers which will keep track of parsed fields and values.
There are 2 places where the parsing of the ingested document happens:
1. upon the 'raw bulk' a request is sent without the pipelines
2. upon the 'ingest service' when a request is sent with pipelines
(parsing can occur twice when a dynamic mappings are calculated, this PR takes this into account and prevent double billing)
We also want to make sure, that the metering logic is not unnecessarily executed when a document was already reported. That is if a document was reported in IngestService, there is no point wrapping the XContentParser again.
This commit introduces a `DocumentReporterPlugin` an internal plugin that will be implemented in serverless. This plugin should return a `DocumentParsingObserver` supplier which will create a `DocumentParsingObserver`. A DocumentParsingObserver is used to wrap an `XContentParser` with an implementation that keeps track of parsed fields and values (performs a metering) and allows to send that information along with an index name to a MeteringReporter.
Today all responses to remote-cluster requests are deserialized and
handled on the transport worker thread. Some of these responses can be
sizeable, so with this commit we add the facility for callers to specify
a different executor to handle this work. It also adjusts several
callers to use more appropriate threads, including:
- responses from CCR-related admin actions are handled on `ccr`
- responses from field caps actions are handled on `search_coordination`
This is a common pattern, particularly in chunked XContent encoding, but
working with streams is surprisingly expensive. This commit replaces it
with a much simpler utility that works directly on iterators.
It also simplifies the cases where we avoid creating a stream by using
`Iterators.flatMap(..., x -> Iterators.single(...))`.
Build represents information about the running build of ES. It has
some methods that assume some structure to the existing version.
Additionally MainResponse contains Version just to be able to retrieve
the min compat versions.
This commit makes all of these bits of info part of the state of Build.
It also removes Version from MainResponse as it is no longer necessary.
Build flavor was added back in #97770, but the main endpoint response
is still hardcoded to "default". This commit updates the response to
return the flavor from the build info.
The Build class keeps information about the running build of
Elasticsearch. There was previously a build flavor, which has since been
narrowed to just return the "default" flavor. This commit adds flavor
back to Build so that a plugged in Build can include a change to flavor.
With the expansion of the data stream lifecycle configuration that is
coming (for example, downsampling). It will be helpful to reduce the
number of constructors and mainly use the builder to initialise the
DataStreamLifecycle object.
The index and transport versions are low level details of how a node
behaves in a cluster. They were recently added to the main endpoint
response, but they are too low level and should be moved to another
endpoint TBD.
This commit removes those versions from the main endpoint response. Due
to the fact lucene version is now derived from index version, this
commit also adds an explicit lucene version member to the main response.
Fix for #97334 where incorrect feature name was provided.
Correct more instances of synonyms_feature_flag_enabled for synonyms_api_feature_flag_enabled
Closes#96641, #97177
Added a clusterAlias to the Painless execute Request object, so that index
expressions in the request of the form "myremote:myindex" will be parsed to
set clusterAlias to "myremote" and the index to "myindex".
If clusterAlias is null, then it is executed against a shard on the local cluster, as before.
If clusterAlias is non-null, then the SingleShardTransportAction is sent to the remote cluster,
where it will run the full request (doing remote coordination). Note that the new clusterAlias
field is not Writeable so that when it is sent to the remote cluster it will only see the index
name, not the clusterAlias (which it wouldn't know how to handle correctly).
Added PainlessExecuteIT test that tests cross-cluster calls
Updated painless-execute-script end user docs to indicate support for cross-cluster executions
ActionListener.delegateFailure is not the optimal choice when the
response handler always invokes delegate.onResponse, it's just
needlessly verbose, error-prone and sometimes less efficient when
multiple map/delegation calls are chained. ActionListener.wrap is just a
less efficient version of ActionListener.delegateAndWrap when the error
handler is `listener::onFailure` -> fixup a couple of these as well.
The FetchPhase sets source and field providers on its SearchLookup instance
to ensure that we only have to load stored fields once per document. However,
these were being set after subprocessors were constructed, and the
FetchFieldsPhase subprocessor pulls a reference to the SearchLookup at
construction time. This meant that the FieldsFetcher was not using these
providers, instead using the standard source and fields lookup, in effect meaning
that fields access would load source and fields separately.
This would mostly just be an issue with performance; however, the top hits
aggregation calls the fetch phase multiple times, with the result that the second
and subsequent buckets in a top hits aggregation would see the source lookup
from the previous bucket when building their FetchFieldsPhase subprocessor.
This commit moves the call to SearchExecutionContext#setLookupProviders()
in FetchPhase#buildSearchHits() before the call to build fetch phase subprocessors,
ensuring that the fetch fields phase sees the correct SearchLookup.
Fixes#96284
This change adds the ability for reloadable search analysers to adapt their loading based on
the index creation context. It is useful for reloadable search analysers that need to load
expensive resources from indices or disk. In such case they can defer the loading of the
resource during the shard recovery and avoid blocking a master or a create index thread.
---------
Co-authored-by: Mayya Sharipova <mayya.sharipova@elastic.co>
In order for build info to be pluggable for serverless, the current
build needs to be lazily determined. This commit moves the CURRENT
constant to a static method.
relates #96861
This code works on one cluster state version. We read the `DataStream`
from the same cluster state twice. The write index will always be the
same in this case. (as we're not reading the `DataStream` from a new
cluster state version)
This PR drops a redundant (and confusing) reference the same write index
of the data stream.
Lots of spots where we did weird things around streams like redundant stream creation, redundant collecting
before adding all the collected elements to another collection or so, redundant streams for joining strings
and using less efficient `Collectors.toList` and in a few cases also incorrectly relying on the result being mutable.
We can just inline all usage of this thing for now and keep the assertion
about the field name for parsing BwC.
-> saves some code and questions about the timestamp field
Follow up to #96856, turns out this wasn't safe to remove after all.
We can't go back to the previous solution though since that had double invocation issues
=> use notify-once for now until this is fixed in Netty.
This PR introduces downsampling configuration to the data stream lifecycle. Keep in mind downsampling implementation will come in a follow up PR. Configuration looks like this:
```
{
"lifecycle": {
"data_retention": "90d",
"downsampling": [
{
"after": "1d",
"fixed_interval": "2h"
},
{ "after": "15d", "fixed_interval": "1d" },
{ "after": "30d", "fixed_interval": "1w" }
]
}
}
```
We will also support using `null` to unset downsampling configuration during template composition:
```
{
"lifecycle": {
"data_retention": "90d",
"downsampling": null
}
}
```
The code to have the better error message is already present.
But the current tests are not set up in a way to reveal that.
This adjusts the testing FailBeforeCurrentQueryBuilder to demonstrate it.
To show this via manual testing, I changed the MatchAllQueryBuilder to return
a TransportVersion in the future from its getMinimalSupportedVersion method.
When I start elasticsearch with -E search.check_ccs_compatibility=true and
do a match_all query, the response is:
```
{
"error": {
"root_cause": [
{
"type": "illegal_argument_exception",
"reason": "[class org.elasticsearch.action.search.SearchRequest] is not compatible with version 8500020 and the 'search.check_ccs_compatibility' setting is enabled. YY"
}
],
"type": "illegal_argument_exception",
"reason": "[class org.elasticsearch.action.search.SearchRequest] is not compatible with version 8500020 and the 'search.check_ccs_compatibility' setting is enabled. YY",
"caused_by": {
"type": "illegal_argument_exception",
"reason": "[match_all] was released first in version 8511132, failed compatibility check trying to send it to node with version 8500020"
}
},
"status": 400
}
```
The underlying caused_by shows the exact query type (match_all) that was incompatible.
* Update Guava dependency version
* Update gradle verification metadata
* fix :repository-gcs:thirdPartyAudit
* fix :discovery-gce:thirdPartyAudit
* Use single sha256 hash in allowed list
Plugins may supply a `SettingsUpgrader` which defines adjustments to
settings in an upgrade. The lifecycle of these adjustments is kinda
trappy: they run on a full-cluster-restart upgrade, but not in a rolling
upgrade, and they also apply upgrades to settings given in requests to
the cluster-settings update API.
In practice we have never needed these things very much, and all the
upgraders we've ever written involve versions which have been EOL for a
very long time. We have better ways to handle this kind of breaking
change today, so it'd be best to remove this feature.
This adds support for tail forcemerging in data stream lifecycle.
Before issuing the forcemerge (without the target segment specified) the
lifecycle service will aim to configure the floor segment merge policy
to `100mb` (configurable via a new cluster setting -
`data_streams.lifecycle.target.merge.policy.floor_segment`) and the merge
factor to `16` (configurable via a new cluster setting
`data_streams.lifecycle.target.merge.policy.merge_factor`).
In Java 21 List now extends SequencedCollection, instead of Collection
directly. When resolving methods Painless starts at the defined type,
and iterates up through super classes and interfaces. Unfortunately if a
superinterface was not known, as it is for SequencedCollection since it
is not in the allowed list of classes, method resolution would give up.
This commit adjusts the superinterface interation to continue traversing
until the method is found or no more superinterfaces are found.
fixes#97022
If 'now' is between midnight and 2 in the night, going back 2 hours
(twoHoursAgo) results in a date whose day is the day before. Here we use
a fixed instant so to avoid the test failing depending on the value of
`now`.
We also use `twoHoursAgo` instead of `now` to generate the correct
filename even if, for the selected instant, this is not strictly
necessary (using `now` or `twoHoursAgo` results in the same day in the
filename).
Resolves#96672
This PR adds a new optional parameter "resource" for ReloadAnalyzersRequest.
If used, only analyzers that use this specific "resource" will be reload.
This parameter is not documented, for internal use only.
PR #96886 introduced auto-reload of analyzers on synonyms index change. The problem
was that reloading was applied broadly for all indices that contained reloadable
analyzers. This PR improves this, so when a particular synonyms set changes,
only analyzers that use this synonyms set will auto-reloaded. Note that shard
requests will still be sent to all indices shards, as only on a shard we can
decide if analyzers need to be reloaded.
Drying this up further and adding the same short-cut for single node
tests. Dealing with most of the spots that I could grab via automatic
refactorings.
In #96900 a new transport version was added for additional information
in the main response. However, due to upstream conflicts, this transport
version had to be bumped a few times. In the process, the the version
was bumped, but the condition was never updated. This commit fixes the
condition to use the version that was added.
Setting `experiment_hint` to `high_accuracy` leads to using a slower but
more accurate implementation for TDigest.
Add YAML tests to cover for settings this param properly and for parsing
errors.
Related to #95903
The root endpoint for Elasticsearch contains diagnostic information
about what Elasticsearch is running. This commit adds TransportVersion
to this diagnostic information, alongside the existing version
information.
Replacing the remaining usages that I could automatically replace
and a couple that I did by hand in this PR.
Also, added the same shortcut to the single node tests to save some
duplication there.
Synonym Management API project
On changes of synonyms in a synonym set, auto-reload analyzers.
Note that currently all updateable analyzers will be reloaded, even
those that are not relevant for a synonyms set being updated.
* Initial import for TDigest forking.
* Fix MedianTest.
More work needed for TDigestPercentile*Tests and the TDigestTest (and
the rest of the tests) in the tdigest lib to pass.
* Fix Dist.
* Fix AVLTreeDigest.quantile to match Dist for uniform centroids.
* Update docs/changelog/96086.yaml
* Fix `MergingDigest.quantile` to match `Dist` on uniform distribution.
* Add merging to TDigestState.hashCode and .equals.
Remove wrong asserts from tests and MergingDigest.
* Fix style violations for tdigest library.
* Fix typo.
* Fix more style violations.
* Fix more style violations.
* Fix remaining style violations in tdigest library.
* Update results in docs based on the forked tdigest.
* Fix YAML tests in aggs module.
* Fix YAML tests in x-pack/plugin.
* Skip failing V7 compat tests in modules/aggregations.
* Fix TDigest library unittests.
Remove redundant serializing interfaces from the library.
* Remove YAML test versions for older releases.
These tests don't address compatibility issues in mixed cluster tests as
the latter contain a mix of older and newer nodes, so the output depends
on which node is picked as a data node since the forked TDigest library
is not backwards compatible (produces slightly different results).
* Fix test failures in docs and mixed cluster.
* Reduce buffer sizes in MergingDigest to avoid oom.
* Exclude more failing V7 compatibility tests.
* Update results for JdbcCsvSpecIT tests.
* Update results for JdbcDocCsvSpecIT tests.
* Revert unrelated change.
* More test fixes.
* Use version skips instead of blacklisting in mixed cluster tests.
* Switch TDigestState back to AVLTreeDigest.
* Update docs and tests with AVLTreeDigest output.
* Update flaky test.
* Remove dead code, esp around tracking of incoming data.
* Update docs/changelog/96086.yaml
* Delete docs/changelog/96086.yaml
* Remove explicit compression calls.
This was added to prevent concurrency tests from failing, but it leads
to reduces precision. Submit this to see if the concurrency tests are
still failing.
* Revert "Remove explicit compression calls."
This reverts commit 5352c96f65.
* Remove explicit compression calls to MedianAbsoluteDeviation input.
* Add unittests for AVL and merging digest accuracy.
* Fix spotless violations.
* Delete redundant tests and benchmarks.
* Fix spotless violation.
* Use the old implementation of AVLTreeDigest.
The latest library version is 50% slower and less accurate, as verified
by ComparisonTests.
* Update docs with latest percentile results.
* Update docs with latest percentile results.
* Remove repeated compression calls.
* Update more percentile results.
* Use approximate percentile values in integration tests.
This helps with mixed cluster tests, where some of the tests where
blocked.
* Fix expected percentile value in test.
* Revert in-place node updates in AVL tree.
Update quantile calculations between centroids and min/max values to
match v.3.2.
* Add SortingDigest and HybridDigest.
The SortingDigest tracks all samples in an ArrayList that
gets sorted for quantile calculations. This approach
provides perfectly accurate results and is the most
efficient implementation for up to millions of samples,
at the cost of bloated memory footprint.
The HybridDigest uses a SortingDigest for small sample
populations, then switches to a MergingDigest. This
approach combines to the best performance and results for
small sample counts with very good performance and
acceptable accuracy for effectively unbounded sample
counts.
* Remove deps to the 3.2 library.
* Remove unused licenses for tdigest.
* Revert changes for SortingDigest and HybridDigest.
These will be submitted in a follow-up PR for enabling MergingDigest.
* Remove unused Histogram classes and unit tests.
Delete dead and commented out code, make the remaining tests run
reasonably fast. Remove unused annotations, esp. SuppressWarnings.
* Remove Comparison class, not used.
* Revert "Revert changes for SortingDigest and HybridDigest."
This reverts commit 2336b11598.
* Use HybridDigest as default tdigest implementation
Add SortingDigest as a simple structure for percentile calculations that
tracks all data points in a sorted array. This is a fast and perfectly
accurate solution that leads to bloated memory allocation.
Add HybridDigest that uses SortingDigest for small sample counts, then
switches to MergingDigest. This approach delivers extreme
performance and accuracy for small populations while scaling
indefinitely and maintaining acceptable performance and accuracy with
constant memory allocation (15kB by default).
Provide knobs to switch back to AVLTreeDigest, either per query or
through ClusterSettings.
* Small fixes.
* Add javadoc and tests.
* Add javadoc and tests.
* Remove special logic for singletons in the boundaries.
While this helps with the case where the digest contains only
singletons (perfect accuracy), it has a major issue problem
(non-monotonic quantile function) when the first singleton is followed
by a non-singleton centroid. It's preferable to revert to the old
version from 3.2; inaccuracies in a singleton-only digest should be
mitigated by using a sorted array for small sample counts.
* Revert changes to expected values in tests.
This is due to restoring quantile functions to match head.
* Revert changes to expected values in tests.
This is due to restoring quantile functions to match head.
* Tentatively restore percentile rank expected results.
* Use cdf version from 3.2
Update Dist.cdf to use interpolation, use the same cdf
version in AVLTreeDigest and MergingDigest.
* Revert "Tentatively restore percentile rank expected results."
This reverts commit 7718dbba59.
* Revert remaining changes compared to main.
* Revert excluded V7 compat tests.
* Exclude V7 compat tests still failing.
* Exclude V7 compat tests still failing.
* Remove ClusterSettings tentatively.
* Initial import for TDigest forking.
* Fix MedianTest.
More work needed for TDigestPercentile*Tests and the TDigestTest (and
the rest of the tests) in the tdigest lib to pass.
* Fix Dist.
* Fix AVLTreeDigest.quantile to match Dist for uniform centroids.
* Update docs/changelog/96086.yaml
* Fix `MergingDigest.quantile` to match `Dist` on uniform distribution.
* Add merging to TDigestState.hashCode and .equals.
Remove wrong asserts from tests and MergingDigest.
* Fix style violations for tdigest library.
* Fix typo.
* Fix more style violations.
* Fix more style violations.
* Fix remaining style violations in tdigest library.
* Update results in docs based on the forked tdigest.
* Fix YAML tests in aggs module.
* Fix YAML tests in x-pack/plugin.
* Skip failing V7 compat tests in modules/aggregations.
* Fix TDigest library unittests.
Remove redundant serializing interfaces from the library.
* Remove YAML test versions for older releases.
These tests don't address compatibility issues in mixed cluster tests as
the latter contain a mix of older and newer nodes, so the output depends
on which node is picked as a data node since the forked TDigest library
is not backwards compatible (produces slightly different results).
* Fix test failures in docs and mixed cluster.
* Reduce buffer sizes in MergingDigest to avoid oom.
* Exclude more failing V7 compatibility tests.
* Update results for JdbcCsvSpecIT tests.
* Update results for JdbcDocCsvSpecIT tests.
* Revert unrelated change.
* More test fixes.
* Use version skips instead of blacklisting in mixed cluster tests.
* Switch TDigestState back to AVLTreeDigest.
* Update docs and tests with AVLTreeDigest output.
* Update flaky test.
* Remove dead code, esp around tracking of incoming data.
* Remove explicit compression calls.
This was added to prevent concurrency tests from failing, but it leads
to reduces precision. Submit this to see if the concurrency tests are
still failing.
* Update docs/changelog/96086.yaml
* Delete docs/changelog/96086.yaml
* Revert "Remove explicit compression calls."
This reverts commit 5352c96f65.
* Remove explicit compression calls to MedianAbsoluteDeviation input.
* Add unittests for AVL and merging digest accuracy.
* Fix spotless violations.
* Delete redundant tests and benchmarks.
* Fix spotless violation.
* Use the old implementation of AVLTreeDigest.
The latest library version is 50% slower and less accurate, as verified
by ComparisonTests.
* Update docs with latest percentile results.
* Update docs with latest percentile results.
* Remove repeated compression calls.
* Update more percentile results.
* Use approximate percentile values in integration tests.
This helps with mixed cluster tests, where some of the tests where
blocked.
* Fix expected percentile value in test.
* Revert in-place node updates in AVL tree.
Update quantile calculations between centroids and min/max values to
match v.3.2.
* Add SortingDigest and HybridDigest.
The SortingDigest tracks all samples in an ArrayList that
gets sorted for quantile calculations. This approach
provides perfectly accurate results and is the most
efficient implementation for up to millions of samples,
at the cost of bloated memory footprint.
The HybridDigest uses a SortingDigest for small sample
populations, then switches to a MergingDigest. This
approach combines to the best performance and results for
small sample counts with very good performance and
acceptable accuracy for effectively unbounded sample
counts.
* Remove deps to the 3.2 library.
* Remove unused licenses for tdigest.
* Revert changes for SortingDigest and HybridDigest.
These will be submitted in a follow-up PR for enabling MergingDigest.
* Remove unused Histogram classes and unit tests.
Delete dead and commented out code, make the remaining tests run
reasonably fast. Remove unused annotations, esp. SuppressWarnings.
* Remove Comparison class, not used.
* Revert "Revert changes for SortingDigest and HybridDigest."
This reverts commit 2336b11598.
* Use HybridDigest as default tdigest implementation
Add SortingDigest as a simple structure for percentile calculations that
tracks all data points in a sorted array. This is a fast and perfectly
accurate solution that leads to bloated memory allocation.
Add HybridDigest that uses SortingDigest for small sample counts, then
switches to MergingDigest. This approach delivers extreme
performance and accuracy for small populations while scaling
indefinitely and maintaining acceptable performance and accuracy with
constant memory allocation (15kB by default).
Provide knobs to switch back to AVLTreeDigest, either per query or
through ClusterSettings.
* Add javadoc and tests.
* Remove ClusterSettings tentatively.
* Restore bySize function in TDigest and subclasses.
* Update Dist.cdf to match the rest.
Update tests.
* Revert outdated test changes.
* Revert outdated changes.
* Small fixes.
* Update docs/changelog/96794.yaml
* TDigestState uses MergingDigest by default.
* Make HybridDigest the default implementation.
* Update boxplot documentation.
* Use HybridDigest for real.
* Restore AVLTreeDigest as the default in TDigestState.
TDigest.createHybridDigest nw returns the right type.
The switch in TDigestState will happen in a separate PR
as it requires many test updates.
* Use execution_hint in tdigest spec.
* Restore expected test values.
* Fix Dist.cdf for empty digest.
* Bump up TransportVersion.
* More test updates.
* Bump up TransportVersion for real.
* Restore V7 compat blacklisting.
* HybridDigest uses its final implementation during deserialization.
* Restore the right TransportVersion in TDigestState.read
* More test fixes.
* More test updates.
* Use TDigestExecutionHint instead of strings.
* Add link to TDigest javadoc.
* Spotless fix.
* Small fixes.
* Bump up TransportVersion.
* Bump up the TransportVersion, again.
* Update docs/changelog/96904.yaml
* Delete 96794.yaml
Delete existing changelog to get a new one.
* Restore previous changelog.
* Rename 96794.yaml to 96794.yaml
* Update breaking change notes in changelog.
* Remove mapping value from changelog.
* Set a valid breaking area.
* Use HybridDigest as default TDigest impl.
* Update docs/changelog/96904.yaml
* Use TDigestExecutionHint in MedianAbsoluteDeviationAggregator.
* Update changelog and comment in blacklisted V7 compat tests.
* Update breaking area in changelog.
This check isn't necessary any longer. We safely close all channels
before shutting down the even loop groups so we can not run into
the situation that a listener passed to `writeAndFlush` isn't completed.
We also don't have this check on the http channel sending.
-> remove the check that can cause a bug by double invoking the listener
closes#95759
This change adds a new top-level element to the search endpoint called sub_searches. This top-level
element allows for a list of additional searches where each "sub search" will have a query executed
separately as part of ranking and later combined into a final single set of documents based on the
ranking algorithm.
* When using a managed pipeline GeoIpDownloader is triggered only when an index exists for the pipeline.
* When using a managed pipeline GeoIpDownloader is triggered only when an index exists for the pipeline.
* Adding the geoip processor back
* Adding tags to the events mapping.
* Fix a forbidden API call into tests.
* lint
* Adding an integration tests for managed pipelines.
* lint
* Add a geoip_database_lazy_download param to pipelines and use it instead of managed.
* Fix a edge case: pipeline can be set after index is created.
* lint.
* Update docs/changelog/96624.yaml
* Update 96624.yaml
* Uses a processor setting (download_database_on_pipeline_creation) to decide database download strategy.
* Removing debug instruction.
* Improved documentation.
* Improved the way to check for referenced pipelines.
* Fixing an error in test.
* Improved integration tests.
* Lint.
* Fix failing tests.
* Fix failing tests (2).
* Adding javadoc.
* lint javadoc.
* Using a set instead of a list to store checked pipelines.
Mostly the keys we read are strings, lets add an overload for that
to save some code and maybe help the compiler make better decisions.
Also readMapOfLists an be way simplified, no point in duplicating
the map reading code here just to save one capturing lambda, there's
not hot code that benefits from this.
Just like the readImmutableList method, adding this avoids some copying
and needless wrapping relative to the replace usages. The few plain
usages of the mutable set read that were replaced are safe to replace
since their non-stream-input constructors use immutable set copies.
This commit changes access to the latest TransportVersion constant to
use a static method instead of a public static field. By encapsulating
the field we will be able to (in a followup) lazily determine what the
latest is, outside of clinit.
* Initial import for TDigest forking.
* Fix MedianTest.
More work needed for TDigestPercentile*Tests and the TDigestTest (and
the rest of the tests) in the tdigest lib to pass.
* Fix Dist.
* Fix AVLTreeDigest.quantile to match Dist for uniform centroids.
* Update docs/changelog/96086.yaml
* Fix `MergingDigest.quantile` to match `Dist` on uniform distribution.
* Add merging to TDigestState.hashCode and .equals.
Remove wrong asserts from tests and MergingDigest.
* Fix style violations for tdigest library.
* Fix typo.
* Fix more style violations.
* Fix more style violations.
* Fix remaining style violations in tdigest library.
* Update results in docs based on the forked tdigest.
* Fix YAML tests in aggs module.
* Fix YAML tests in x-pack/plugin.
* Skip failing V7 compat tests in modules/aggregations.
* Fix TDigest library unittests.
Remove redundant serializing interfaces from the library.
* Remove YAML test versions for older releases.
These tests don't address compatibility issues in mixed cluster tests as
the latter contain a mix of older and newer nodes, so the output depends
on which node is picked as a data node since the forked TDigest library
is not backwards compatible (produces slightly different results).
* Fix test failures in docs and mixed cluster.
* Reduce buffer sizes in MergingDigest to avoid oom.
* Exclude more failing V7 compatibility tests.
* Update results for JdbcCsvSpecIT tests.
* Update results for JdbcDocCsvSpecIT tests.
* Revert unrelated change.
* More test fixes.
* Use version skips instead of blacklisting in mixed cluster tests.
* Switch TDigestState back to AVLTreeDigest.
* Update docs and tests with AVLTreeDigest output.
* Update flaky test.
* Remove dead code, esp around tracking of incoming data.
* Update docs/changelog/96086.yaml
* Delete docs/changelog/96086.yaml
* Remove explicit compression calls.
This was added to prevent concurrency tests from failing, but it leads
to reduces precision. Submit this to see if the concurrency tests are
still failing.
* Revert "Remove explicit compression calls."
This reverts commit 5352c96f65.
* Remove explicit compression calls to MedianAbsoluteDeviation input.
* Add unittests for AVL and merging digest accuracy.
* Fix spotless violations.
* Delete redundant tests and benchmarks.
* Fix spotless violation.
* Use the old implementation of AVLTreeDigest.
The latest library version is 50% slower and less accurate, as verified
by ComparisonTests.
* Update docs with latest percentile results.
* Update docs with latest percentile results.
* Remove repeated compression calls.
* Update more percentile results.
* Use approximate percentile values in integration tests.
This helps with mixed cluster tests, where some of the tests where
blocked.
* Fix expected percentile value in test.
* Revert in-place node updates in AVL tree.
Update quantile calculations between centroids and min/max values to
match v.3.2.
* Add SortingDigest and HybridDigest.
The SortingDigest tracks all samples in an ArrayList that
gets sorted for quantile calculations. This approach
provides perfectly accurate results and is the most
efficient implementation for up to millions of samples,
at the cost of bloated memory footprint.
The HybridDigest uses a SortingDigest for small sample
populations, then switches to a MergingDigest. This
approach combines to the best performance and results for
small sample counts with very good performance and
acceptable accuracy for effectively unbounded sample
counts.
* Remove deps to the 3.2 library.
* Remove unused licenses for tdigest.
* Revert changes for SortingDigest and HybridDigest.
These will be submitted in a follow-up PR for enabling MergingDigest.
* Remove unused Histogram classes and unit tests.
Delete dead and commented out code, make the remaining tests run
reasonably fast. Remove unused annotations, esp. SuppressWarnings.
* Remove Comparison class, not used.
* Small fixes.
* Add javadoc and tests.
* Remove special logic for singletons in the boundaries.
While this helps with the case where the digest contains only
singletons (perfect accuracy), it has a major issue problem
(non-monotonic quantile function) when the first singleton is followed
by a non-singleton centroid. It's preferable to revert to the old
version from 3.2; inaccuracies in a singleton-only digest should be
mitigated by using a sorted array for small sample counts.
* Revert changes to expected values in tests.
This is due to restoring quantile functions to match head.
* Revert changes to expected values in tests.
This is due to restoring quantile functions to match head.
* Tentatively restore percentile rank expected results.
* Use cdf version from 3.2
Update Dist.cdf to use interpolation, use the same cdf
version in AVLTreeDigest and MergingDigest.
* Revert "Tentatively restore percentile rank expected results."
This reverts commit 7718dbba59.
* Revert remaining changes compared to main.
* Revert excluded V7 compat tests.
* Exclude V7 compat tests still failing.
* Exclude V7 compat tests still failing.
* Restore bySize function in TDigest and subclasses.
Notable changes:
* more efficient backwards reads in NIOFSDirectory
* faster merging when using soft deletes
* workaround security manager when using vector API
- Create a new option of "synonyms_set" for synonym set filter that
specifies which synonyms set to be loaded from the system ".synonyms" index
- On index creation for this option load synonyms set from index
- If synonyms set doesn't exist, index creation request still succeeds, but
shards are not allocated, so the cluster state will be read.
Note: this is a temporary solution, as:
- No check is done on master node, as fake synonyms are provided
- On shard on index creation we use a blocking operation in the cluster applier thread
Invoking the get data stream api on an upgrade tsdb data stream (which has normal and tsbd indices) can result in a warning log incorrectly be logged. This warning is logged as part of computing the temporal ranges of a tsdb data stream. This warning log indicates that tsdb backing indices are overlapping, but this isn't true. For normal indices, this api picked -9999-01-01T00:00:00Z as start time and 9999-12-31T23:59:59.999Z as end time. This will overlap with any tsdb backing index. Instead, the normal indices should be skipped for computing the temporal ranges.
The following can incorrectlu be logged for upgraded tsdb data streams: [instance-0000000074] previous backing index [-9999-01-01T00:00:00Z/9999-12-31T23:59:59.999Z] range is colliding with current backing index range [-9999-01-01T00:00:00Z/9999-12-31T23:59:59.999Z] This change addresses this. Note that in tests an assertion would trip before the log gets printed.
Motivated by looking into allocations of listeners in detail for shared cache benchmarking.
Wrapping a listener and using `listener::onFailure` as the failure callback means that we
have a reference to the listener from both the failure and the response handler.
If we use the approach used by the `.deleteGate*` methods, we can often save allocating
a response handler lambda or at least make the response handler cheaper.
We also save allocating the failure handler lambda.
If during the coordinator rewrite can't find the requested field we
should assume that the query matches. So
`MappedFieldType.Relation.INTERSECTS` should be returned instead of
`MappedFieldType.Relation.DISJOINT`.
Closes#96487 and #96501
(marking as non-issue, this is a bug in unreleased code)
This PR allows a user of an index template to explicitly reset to `null` either the whole data lifecycle config or just the retention of a component template and overwrite what previous component templates have set.
Add junit matchers allowing other matchers to be transformed using a
function. This can make checking properties of lists/arrays a lot more
fluent using nested matchers, rather than declaratively checking
individual items. These replace the custom `ElasticsearchMatchers` with
more generic ones.
As a basic example, it allows you to turn this:
assertThat(list.get(0).getName(), equalTo("foo"));
assertThat(list.get(1).getName(), equalTo("bar"));
assertThat(list.get(2).getName(), equalTo("quux"));
into this:
assertThat(list, transformedItems(Item::getName, contains("foo", "bar", "quux")));
Doing this 'properly' without these helpers requires defining your own
matchers, which is very cumbersome.
I've applied the new methods to `ElasticsearchAssertions` and a few
other classes to show various use cases.
JDK 20 added a new javac lint warning for possible lossy conversion in compound assignments - because of implicit type casts, e.g.
warning: [lossy-conversions] implicit cast from int to byte in compound assignment is possibly lossy
The change resolves all such warnings, by either widening the type of the left-hand operand, or explicitly casting the type of the right-hand operand.
Porting the DLM permissions REST test to the new style of cluster tests.
This has the usual nice perks, and also allows us to remove the separate
`qa/with-security` package.
No functional or test logic changes.
I didn't suggest this as part of the PR review for
https://github.com/elastic/elasticsearch/pull/95512 so as not to block
that PR further, and also because I wasn't sure about the overhead of
making this change (it did end up taking some battling with gradle).
Most relevant changes:
- add api to allow concurrent query rewrite (GITHUB-11838 Add api to allow concurrent query rewrite apache/lucene#11840)
- knn query rewrite (Concurrent rewrite for KnnVectorQuery apache/lucene#12160)
- Integrate the incubating Panama Vector API (Integrate the Incubating Panama Vector API apache/lucene#12311)
As part of this commit I moved the ES codebase off of overriding or relying on the deprecated rewrite(IndexReader) method in favour of using rewrite(IndexSearcher) instead. For score functions, I went for not breaking existing plugins and create a new IndexSearcher whenever we rewrite a filter, otherwise we'd need to change the ScoreFunction#rewrite signature to take a searcher instead of a reader.
Co-authored-by: ChrisHegarty <christopher.hegarty@elastic.co>
It turns out that it is even harder than I had thought to detect that
forcemerge has run. So I have changed
DataLifecycleServiceIT.testAutomaticForceMerge() to use a custom
StubbableTransport.SendRequestBehavior to detect that a ForceMergeAction
has been sent, rather than checking whether the number of segments has
changed. Closes#96084
This adds a new index privilege called `manage_dlm`. The `manage_dlm`
has permission to perform all DLM actions on an index, including put and
delete. It also adds the ability to call DLM get and explain to the
`view_index_metadata` existing index privilege.
This PR introduces a new internal user `_dlm`. `_dlm` will run all DLM
runtime tasks, which include index deletion (due to retention),
rollover, and force merges.
Technically, there is a BWC concern here: in a mixed cluster with nodes
on 8.8 and 8.9, 8.8 nodes would not be able to deserialize the `_dlm`
internal user (as we're only adding it in 8.9). However, DLM is still
behind a feature flag so I don't think we need to address this.
Currently we have a number of input streams that specifically override
the skip() method disabling the ability to skip bytes. In each case the
skip implementation works as we have properly implemented the
read(byte[]) methods used to discard bytes. However, we appear to have
disabled it as it would be possible to retry from the end of a skip if
there is a failure in the middle. At this time, that optimization is not
really necessary, however, we sporadically used skip so it would be nice
for the IS to support the method. This commit enables the super.skip()
and adds a comment about future optimizations.
Expand index.time_series.start_time and end_time to nanoseconds if
timestamp field's resolution is set to nanoseconds. When creating
coordinator rewrite context.
Closes#96030
If an integration test brings up a 1-node cluster, and if DLM runs
forcemerge, then forcemerge will report an error. This PR updates 2
DataLifecycleServiceIT tests to account for that, similar to #96047.
Closes#96070
The node-level task is always cancellable, so we can expose it as such
to implementations. Also we can use a `ChannelActionListener` rather
than a hand-crafted equivalent.
Following the changes in #95112, which relocated the calls
into the AuthenticationService that authenticate HTTP
requests, the authentication duration was no longer
comprised in between the Tracer#startTrace and
Tracer#stopTrace. Consequently, the span records
didn't cover the authentication duration any longer.
This PR remedies that by changing the Tracer
implementation, i.e. APMTracer, to look for the trace start
time instant in the transient thread context and use that
when starting traces (overriding the now default).
The trace start time is set in the thread context when
the request-wise thread context is first populated
(with HTTP request headers).