When every index that a transform is configured to search has
remained completely unchanged between checkpoints the transform
should not do a search at all.
Following #75839 there was a problem where the scenario of all
indices being unchanged between checkpoints could cause an empty
list of indices to be searched, which Elasticsearch treats as
meaning _all_ indices. This change should prevent that happening
in future.
Fixes#77137
Continuous transform reduce the amount of data to query for by detecting what has been changed
since the last checkpoint. This information is used to inject queries that narrow the scope.
The query is send to all configured indices. This change reduces the indexes to call
using checkpoint information. The number of network calls go down which in addition to performance
reduces the probability of a failure.
This change mainly helps the transforms of type latest, pivot transform require additional
changes planned for later.
* Name `BulkItemResponse` ctors
`BulkItemResponse` can contain either a success or failure. This
replaces the two constructors used to build either case with named
static methods. So instead of
```
return new BulkItemResponse(0, OpType.CREATE, createResponse);
return new BulkItemResponse(0, OpType.CREATE, failure);
```
you now use
```
return BulkItemResponse.success(0, OpType.CREATE, createResponse);
return BulkItemResponse.failure(0, OpType.CREATE, failure);
```
This makes it marginally easier to read code building these things - you
don't have to know the type of the parameter to know if its a failure
or success.
* Consistent
* Mock response
There are two possible race conditions that were not
previously handled in this test.
- Since the syncconfig was null, it may be that the
transform actually gets set to stopping/stopped
and its unable to kick off another indexing pass
- It may also be that the indexer thread is still
finishing up work when the second execution is
requested, so it returns false.
Adding a sync config and assertBusy handles these
cases. Ran 1k+ times locally with this change
and it never failed. Without, it failed ~10 runs.
closes https://github.com/elastic/elasticsearch/issues/76397
Failure count should not only be reset at checkpoints. Checkpoints could have many pages of data. Consequently, we should reset the failure count once we handle a single composite aggregation page.
This way, the transform won't mark itself as failed erroneously when it has actually succeeded searches + indexing results within the same checkpoint.
closes#76074
This change sets the default value for `xpack.security.enabled` to true
for all licenses. As such the value of the settings is read directly
from the node's settings and not from XPackLicenseState which
doesn't need to keep track of it depending on potential license changes
any more.
When _stop gets called with stop_at_checkpoint=true and at the same time a transform got triggered internally or externally a race condition could lead to a deadlock of 5s. The change fixes the situation where 2 lock objects could lock one another.
fixes#75846
Remove an unnecessary indirection and refactor progress tracking. Both rollup and transform
process documents as stream, however in the AsyncTwoPhaseIndexer takes a List of index
requests. This change removes the unnecessary temporary container and makes upcoming
transform enhancements easier.
Automatically reorder group_by for composite aggs, ensuring date histogram
group by comes first. The order is only changed for execution, the provided
config remains unchanged.
In case of 2 group_by's of the same order type, the configuration order is
respected. Script and runtime field based group_by's are penalized.
Use point in time API for every checkpoint in transform. Using point in time reduces pressure
on the source indexes, e.g. less refreshes. In case, pit isn't supported (e.g. when searching
remote clusters) it falls back to ordinary search requests as before.
closes#73481
- avoid eagerly created test cluster
- remove duplicate superflous configuration
- resolve system properties via provider factory
- move common test configuration / setup into rest test base plugin
With the overall theme of trying to configure and add less to the build instead of just disabling it later,
we're replacing standalone-test by standalone-rest tasks avoids creating the
unused test tasks.
Standalone rest test plugin and the other rest test plugins behave a little bit different in the sense how source sets and test tasks are wired.
The standalone rest test plugin assumes that all RestTestTasks are using the same sourceSet (test). The yaml, java Rest test plugins use one dedicated sourceSet per test task.
In the long run we probably will migrate standalone-rest-test usages to one of the other plugins and deprecate standalone-rest-test
We only need the javaRestTest sourceSet and can avoid main and test sourceSet by
just using the new introduced ElasticsearchJavaBase instead of ElasticsearchJava plugin
Aiming for configuring less during the build,
this removes non required configuration from qa build scripts that do not
contain any sources. We also remove a few non required afterEvaluate hooks
implement a simple change optimization for histograms using min and max aggregations. The
optimization is not applied if the range cutoff would be too small compared to the overall
range from previous checkpoints. At least 20% must be cut compared to former checkpoints.
fixes#63801
ParseField is part of the x-content lib, yet it doesn't exist under the
same root package as the rest of the lib. This commit moves the class to
the appropriate package.
relates #73784
When libs/core was created, several classes were moved from server's
o.e.common package, but they were not moved to a new package. Split
packages need to go away long term, so that Elasticsearch can even think
about modularization. This commit moves all the classes under o.e.common
in core to o.e.core.
relates #73784
When associated index patterns contained wildcards and
action.destructive_requires_name was set to true, feature resets were
failing. In order to avoid this, we want to resolve associated index
names, then pass the concrete index names to the delete request.
For normal system indices, the SystemIndexDescriptor provides a method
that searches cluster metadata for indices that match the the system
index pattern. This commit introduces an AssociatedIndexDescriptor that
provides the same mechanism. Although we could use an
IndexNameExpressionResolver for the associated indices, it makes sense
to me to keep things consistent across the various feature index pattern
collections.
This change has the effect of allowing the same regex-like syntax that
system index patterns can use, rather than just wildcards, in associated
index patterns.
Legacy index templates are deprecated but ML was still using
them for its hidden indices.
This PR switches the legacy ML index templates to use the new
composable index template framework.
The composable index templates get installed once the master
node is on a version that understands them. For templates
that need to be up-to-date in mixed version clusters where the
master might still be on a version that doesn't understand
composable index templates we still ship the legacy template
too, and install this if required in the mixed version cluster.
(The notifications index template falls into this category.)
This makes a couple of places in the code a little messy, as
the new style template definitions don't contain a dummy _doc
level (where the type used to be), but the legacy template
definitions do - hopefully we can tidy this up in master once
8.0 is released.
There is one more change of note in this PR that is not
strictly related to switching to composable templates, but
which was shown up during the testing. We used to wait for
all templates to be installed by the master node before running
tests in mixed version clusters. I do not believe we should
have been doing this, as other upgrade orchestration systems,
e.g. Cloud, will not be doing this. Our production code needs
to install templates and/or mappings before any operation that
requires them if there's a chance that the elected master won't
have done this in time.
Fixes#65437
Extract usage of internal API from TestClustersPlugin and PluginBuildPlugin and related plugins and build logic
This includes a refactoring of ElasticsearchDistribution to handle types
better in a way we can differentiate between supported Elasticsearch
Distribution types supported in TestCkustersPlugin and types only supported
in internal plugins.
It also introduces a set of internal versions of public plugins.
As part of this we also generate the plugin descriptors now.
As a follow up on this we can actually move these public used classes into
an extra project (declared as included build)
We keep LoggedExec and VersionProperties effectively public And workaround for RestTestBase
unmute continuous transform testing on sorted indexes. These extra
test randomness has been disabled due to triggered lucene assertions.
The upstream issue seems to have been fixed.
#72533 introduced a regression, causing transforms to timeout/fail.
With this change transform only waits for 1 active shard(primary) as waiting for all can block during
rolling upgrade
fixes#72617
relates #72533
This commit changes the `PersistentTasksClusterService` to limit nodes for a task to a subset of
nodes (candidates) that are not currently shutting down.
It does not yet cancel tasks that may already be running on the nodes that are shut down, that will
be added in a subsequent request.
Relates to #70338
Previously, the ResetFeatureStateStatus object captured its status in a
String, which meant that if we wanted to know if something succeeded or
failed, we'd have to parse information out of the string. This isn't a
good way of doing things.
I've introduced a SUCCESS/FAILURE enum for status constants, and added a
check for failures in the transport action. We return a 207 if some but not all
reset actions fail, and for every failure, we also return information about the
exception or error that caused it.
Co-authored-by: Jay Modi <jaymode@users.noreply.github.com>
add support for the stats and top metrics aggregation in transform. With this change it became
easier to add more multi value aggregations to transform
Limitations:
- only the 1st element of top_metrics gets consumed by transform[*].
- all values of stats will be mapped to double if mapping deduction is used, including count,
sum, min, max
fixes#52236
relates #51925
Related to #71593 we move all build logic that is for elasticsearch build only into
the org.elasticsearch.gradle.internal* packages
This makes it clearer if build logic is considered to be used by external projects
Ultimately we want to only expose TestCluster and PluginBuildPlugin logic
to third party plugin authors.
This is a very first step towards that direction.
David Roberts
Hendrik Muhs
Przemysław Witek
Nik Everett
Benjamin Trent
Ioannis Kakavas
Rene Groeschke
Ryan Ernst
William Brafford
Ignacio Vera
Lee Hinman
Ayushman Singh Chauhan
Dan Hermann