Commit Graph

137 Commits

Author SHA1 Message Date
Przemysław Witek 676d4de3de
[Transform] Implement the ability to preview the existing transform (#76697) 2021-08-24 14:41:49 +02:00
Przemysław Witek ec07e4213e
[Transform] Rename interim_results to align_checkpoints (#76609) 2021-08-18 13:58:50 +02:00
Hendrik Muhs 383fbd8e07
[Transform] Optimize composite agg execution using ordered groupings (#75424)
Automatically reorder group_by for composite aggs, ensuring date histogram
group by comes first. The order is only changed for execution, the provided
config remains unchanged.

In case of 2 group_by's of the same order type, the configuration order is
respected. Script and runtime field based group_by's are penalized.
2021-07-20 09:19:28 +02:00
Rene Groeschke 1db75f75ca
Polish and cleanup test related plugins (#74673)
- avoid eagerly created test cluster
- remove duplicate superflous configuration
- resolve system properties via provider factory
- move common test configuration / setup into rest test base plugin
2021-07-05 10:34:36 +02:00
Rene Groeschke d8e4e48a3b
Avoid creating unused test tasks (#74644)
With the overall theme of trying to configure and add less to the build instead of just disabling it later,
we're replacing standalone-test by standalone-rest tasks avoids creating the
unused test tasks.

Standalone rest test plugin and the other rest test plugins behave a little bit different in the sense how source sets and test tasks are wired.

The standalone rest test plugin assumes that all RestTestTasks are using the same sourceSet (test). The yaml, java Rest test plugins use one dedicated sourceSet per test task.

In the long run we probably will migrate standalone-rest-test usages to one of the other plugins and deprecate standalone-rest-test
2021-06-30 14:11:25 +02:00
Rene Groeschke 54738fc067
Avoid unused sourcesets and tasks in javaRestTest plugin (#74492)
We only need the javaRestTest sourceSet and can avoid main and test sourceSet by
just using the new introduced ElasticsearchJavaBase instead of ElasticsearchJava plugin
2021-06-28 09:54:59 +02:00
Rene Groeschke b79dd52c1b
Cleanup QA projects build scripts (#74428)
Aiming for configuring less during the build,
this removes non required configuration from qa build scripts that do not
contain any sources. We also remove a few non required afterEvaluate hooks
2021-06-23 11:35:47 +02:00
Hendrik Muhs d51b995f3a
[Transform] optmize histogam group_by change detection (#74031)
implement a simple change optimization for histograms using min and max aggregations. The
optimization is not applied if the range cutoff would be too small compared to the overall
range from previous checkpoints. At least 20% must be cut compared to former checkpoints.

fixes #63801
2021-06-14 16:44:36 +02:00
Ryan Ernst 68817d7ca2
Rename o.e.common in libs/core to o.e.core (#73909)
When libs/core was created, several classes were moved from server's
o.e.common package, but they were not moved to a new package. Split
packages need to go away long term, so that Elasticsearch can even think
about modularization. This commit moves all the classes under o.e.common
in core to o.e.core.

relates #73784
2021-06-08 09:53:28 -07:00
Przemysław Witek 51181d5d52
[Transform] Improve error message when user lacks privilege in _preview endpoint (#72002) 2021-05-12 15:06:03 +02:00
Hendrik Muhs ec92261293
[Transform] test with minimal privileges (#72816)
limit the privileges to the minimal required ones

relates #72715
2021-05-10 13:01:07 +02:00
Rene Groeschke e609e07cfe
Remove internal build logic from public build tool plugins (#72470)
Extract usage of internal API from TestClustersPlugin and PluginBuildPlugin and related plugins and build logic

This includes a refactoring of ElasticsearchDistribution to handle types
better in a way we can differentiate between supported Elasticsearch
Distribution types supported in TestCkustersPlugin and types only supported
in internal plugins.

It also introduces a set of internal versions of public plugins.

As part of this we also generate the plugin descriptors now.

As a follow up on this we can actually move these public used classes into 
an extra project (declared as included build)

We keep LoggedExec and VersionProperties effectively public And workaround for RestTestBase
2021-05-06 14:02:35 +02:00
Hendrik Muhs ca63643525
[Transform] fix 2 issues with index sort in integration test (#72742)
fix 2 corner cases in test setup: unsigned_long not support as index sort,do not overlay a runtime field with index sort

fixes #72733
relates #72692
2021-05-05 11:58:53 +02:00
Ignacio Vera 41241f69a2
mute TransformContinuousIT#testContinousEvents (#72734) 2021-05-05 09:24:11 +02:00
Hendrik Muhs 98db34907d
[Transform] unmute continuous transform testing on sorted indexes (#72692)
unmute continuous transform testing on sorted indexes. These extra
test randomness has been disabled due to triggered lucene assertions.
The upstream issue seems to have been fixed.
2021-05-04 17:55:57 +02:00
Hendrik Muhs 7fff5df7a3
[Transform] add support for top metrics (#71850)
add support for the stats and top metrics aggregation in transform. With this change it became
easier to add more multi value aggregations to transform

Limitations:
 - only the 1st element of top_metrics gets consumed by transform[*].
 - all values of stats will be mapped to double if mapping deduction is used, including count,
   sum, min, max

fixes #52236
relates #51925
2021-04-27 13:45:53 +02:00
Rene Groeschke 5bcd02cb4d
Restructure build tools java packages (#72030)
Related to #71593 we move all build logic that is for elasticsearch build only into
the org.elasticsearch.gradle.internal* packages

This makes it clearer if build logic is considered to be used by external projects
Ultimately we want to only expose TestCluster and PluginBuildPlugin logic
to third party plugin authors.

This is a very first step towards that direction.
2021-04-26 14:53:55 +02:00
Benjamin Trent 5e3d54b908
[ML] [Transform] use feature reset API for transform test cleanup (#72044)
This moves all transform cleanup logic to use the feature reset API.
2021-04-22 13:11:16 -04:00
Hendrik Muhs f3c175cc60
[Transform] enhance geobounds test for sparse data case (#72023)
use sparse data for geobounds agg, verifies the fix of #71874, adding debug
logging of index requests send by transform
2021-04-22 12:57:54 +02:00
Hendrik Muhs c7fb400b3f
[Transform] remove deprecated endpoint from tests (#71891)
remove the use of the deprecated _data_frame endpoint from rest test cases, because it can prevent detecting problems in PR builds

relates #71792
2021-04-20 10:35:23 +02:00
Przemysław Witek 319548c80c
Redirect transform actions to transform&remote_cluster_client node when needed (#70125) 2021-04-19 17:22:09 +02:00
Przemysław Witek 749120ed5b
[Transform] Add telemetry support for transform features (#71607) 2021-04-19 14:38:50 +02:00
Dan Hermann c831d887cb
Fix failure in TransformPivotRestSpecialCasesIT::testIndexTemplateMappingClash (#71778) 2021-04-19 07:34:12 -05:00
Henning Andersen 567c56e5fa
Mute testIndexTemplateMappingClash (#71810) 2021-04-19 08:14:10 +02:00
Lyudmila Fokina 3b0b7941ae
Warn users if security is implicitly disabled (#70114)
* Warn users if security is implicitly disabled

Elasticsearch has security features implicitly disabled by default for
Basic and Trial licenses, unless explicitly set in the configuration
file.
This may be good for onboarding, but it also lead to unintended insecure
 clusters.
 This change introduces clear warnings when security features are
 implicitly disabled.
 - a warning header in each REST response if security is implicitly
 disabled;
 - a log message during cluster boot.
2021-04-13 18:33:41 +02:00
David Roberts c4364584aa
[ML] Use feature reset API in ML REST test cleanup (#71552)
Now that we have a feature reset API, we should use
this for cleaning up in between tests instead of running
lots of bespoke cleanup code.

During testing of this change we found we need to
delete custom cluster state as part of the reset process,
so this PR also implements that.

Additionally we no longer assign persistent tasks
during feature reset.
2021-04-13 16:04:47 +01:00
Rene Groeschke 0f40889879
Update build to Gradle 7.0 (#68506)
- Update gradle wrapper to gradle 7.0
- Remove deprecated usages to make build 7.0 compatible
- Fix excludes in docs snippet tasks (See https://github.com/gradle/gradle/issues/16160 for details)
- Fix deprecation warnings in 7.0
- Add explicit dependencies that have been missed
- Make extract native licenses tasks output dir more explicit
- Use a snapshot of the ospackage plugin that includes a fix for 7.0 already
- fix test runtime classpath setup in repository-hdfs
- Make task dependency explicit to fix further deprecation warnings
- Remove manual check for http repo usages that has been deprecated in gradle 7.0
- Update spock to latest 2.0 milestone required for groovy 3
2021-04-13 09:15:08 +02:00
Benjamin Trent 55c7cbcb4a
[Transform] finalize feature reset integration (#71133)
This commit updates transform feature reset to:

- wait for transform tasks to complete
- wait for all indexing actions to transform indices to complete
- and prevents transform audit messages from being written while the reset is being processed

related to https://github.com/elastic/elasticsearch/issues/70008 & https://github.com/elastic/elasticsearch/issues/69581
2021-04-07 11:00:32 -04:00
Hendrik Muhs 7088bbf9e0
[Transform] remove index refresh workardounds from test cases (#70296)
When using wait_for_checkpoint the destination index should be searchable when the call returns.
This issue has been fixed as part of #67832. This change removes the index refresh workarounds still
present.

fixes #51154
2021-03-11 13:46:31 +01:00
Benjamin Trent 1438434b6c
[Transform] add support for geo_line aggregation in pivot function (#69299)
This commit adds support for the Gold+ licensed `geo_line` aggregation.

This aggregation takes a collection of `geo_point` values and constructs a line
according to some sort value. Adding to transforms allows users to create these
potentially expensive lines out of band of visualizations and then do additional aggs/queries
against the pivoted data. 

Examples would be:

"Do these daily user paths ever intersect?"
"Does this path enter and leave this area?"
2021-02-23 14:53:36 -05:00
Rene Groeschke bdf229a148
Introduce Internal Test Artifact Plugin (#68766)
This reduces the ceremony declaring test artifacts for a project.
It also solves an issue with usage of deprecated testRuntime that
testArtifacts extendsFrom which seems not required at all and would have
broke with Gradle 7.0 anyhow

Test artifact resolution is now variant aware which allows us a more adequate 
compile and runtime classpath for the consuming projects.

We also Introduce a convention method in the elasticsearch build to declare 
test artifact dependencies in an easy way close to how its done by the gradle build in 
test fixture plugin.

Furthermore we cleaned up some inconsistent test dependencies declarations when 
relying on a project and on its test artifacts
2021-02-16 14:36:17 +01:00
Hendrik Muhs 967505c3d5
[Transform] add retention policy to documentation IT (#68728)
add retention policy and sync to documentation IT, add builders for sync
and retention policy

relates #67832
2021-02-11 15:59:58 +01:00
Przemysław Witek 747198ffed
[Transform] Take runtime mappings into account when deducing dest index mappings (#68530) 2021-02-11 12:50:15 +01:00
Hendrik Muhs 54ed2e37d9
[Transform] implement retention policy to delete data from a transform (#67832)
add a retention policy to transform to delete data that is considered outdated as part of a
transform checkpoint.

fixes #67916
2021-02-08 15:06:15 +01:00
David Roberts cc59612dd7
[Transform] Automatic management for transform system indices (#68590)
The transform internal index now uses the special
functionality for protecting system indices from
unwanted modification.

The system index descriptor replaces the index
template that was previously used with the transform
internal index.  In places where we used to check
that the template was installed we now proactively
create the index.  Master node actions will apply
the desired mappings if the index is auto-created
as a result of any indexing that they do.
2021-02-08 09:10:14 +00:00
Rene Groeschke 5dfa6f46ac
Remove deprecated usage of default configuration (#68575)
This has been deprecated in gradle before but we havnt been warned.

Gradle 7.0 will likely introduce a change in behaviour here that we
should fix the usage of this configuration upfront.

See https://github.com/gradle/gradle/issues/16027 for further information
about the change in Gradle 7.0
2021-02-07 12:08:02 +01:00
Przemysław Witek 8073a49a86
[Transform] Set runtime mappings on search request in LatestContinuousIT (#68429) 2021-02-03 10:49:47 +01:00
Mark Vieira a92a647b9f Update sources with new SSPL+Elastic-2.0 license headers
As per the new licensing change for Elasticsearch and Kibana this commit
moves existing Apache 2.0 licensed source code to the new dual license
SSPL+Elastic license 2.0. In addition, existing x-pack code now uses
the new version 2.0 of the Elastic license. Full changes include:

 - Updating LICENSE and NOTICE files throughout the code base, as well
   as those packaged in our published artifacts
 - Update IDE integration to now use the new license header on newly
   created source files
 - Remove references to the "OSS" distribution from our documentation
 - Update build time verification checks to no longer allow Apache 2.0
   license header in Elasticsearch source code
 - Replace all existing Apache 2.0 license headers for non-xpack code
   with updated header (vendored code with Apache 2.0 headers obviously
   remains the same).
 - Replace all Elastic license 1.0 headers with new 2.0 header in xpack.
2021-02-02 16:10:53 -08:00
Mark Vieira 564d7575fc Mute LatestContinuousIT 2021-02-02 13:45:25 -08:00
Przemysław Witek 21b3946110
[Transform] Log and audit a warning if all the group-by fields are runtime fields (#68050) 2021-02-02 15:19:16 +01:00
Valeriy Khakhutskyy 82c566e95d
[TEST] Mute HistogramGroupByIT (#67897) 2021-01-25 14:10:58 +01:00
Valeriy Khakhutskyy 9bac56b911
Revert "[TEST] Mute HistogramGroupByIT (#67888)" (#67894)
This reverts commit 8b88568cd5.
2021-01-25 12:58:48 +01:00
Valeriy Khakhutskyy 8b88568cd5
[TEST] Mute HistogramGroupByIT (#67888) 2021-01-25 11:52:09 +01:00
Przemysław Witek d78f39c2ad
[Transform] Add support for search-time runtime fields. (#67643) 2021-01-21 13:27:42 +01:00
Hendrik Muhs 951d822604
[Transform] report last search time in transform stats (#66718)
transforms reports the the last time changes where detected with changes_last_detected_at, however
that doesn't tell a user it searched for changes, this PR adds a field last_search_time to report
when transform searched for changes the last time.

fixes #66410
relates #66367
2021-01-20 15:00:12 +01:00
Hendrik Muhs 8901ac9fc8
[Transform] Improve error handling of state persistence after a version conflict (#67302)
improve error handling on version conflict and ensure it will never fail
endlessly. For tests assert and log more details.

relates: #67121
2021-01-14 08:56:15 +01:00
Przemysław Witek 78690ab825
Mute HistogramGroupByIT (#66598) 2020-12-18 15:31:14 +01:00
Przemysław Witek b8b0fd1f94
Do not deduce mappings in latest transform function (#66523) 2020-12-18 11:34:16 +01:00
Przemysław Witek 9b9deaf986
Unmute TransformContinuousIT (#66496) 2020-12-17 15:18:17 +01:00
Jim Ferenczi cb486a1996
[CI] Mute TransformContinuousIT.testContinousEvents (#66438)
Relates #66410
2020-12-16 12:42:29 +01:00