Commit Graph

1881 Commits

Author SHA1 Message Date
Francisco Fernández Castaño a8101f346f
Add support for index pattern exclusion in CCR AutoFollow (#72935)
This commit adds the ability to specify exclusion patterns in Auto-Follow patterns. This allows excluding indices that match any of the inclusion patterns and also match some of the exclusion patterns giving more fine grained control in scenarios where this is important.

Related #67686
2021-06-04 13:53:30 +02:00
Przemyslaw Gomulka aba2282511
Change year max digits for strict_date_optional_time and date_optional_time (#73034)
We changed the default joda behaviour in strict_date_optional_time to
max 4 digits in a year. Java.time implementation should behave the same way.
At the same time date_optional_time should have 9digits for year part.

closes #52396
closes #72191
2021-06-04 09:35:07 +02:00
David Kyle 94adaa55c0
[ML] Merge the pytorch-inference feature branch (#73660)
The feature branch contains changes to configure PyTorch models with a 
TrainedModelConfig and defines a format to store the binary models. 
The _start and _stop deployment actions control the model lifecycle 
and the model can be directly evaluated with the _infer endpoint. 
2 Types of NLP tasks are supported: Named Entity Recognition and Fill Mask.

The feature branch consists of these PRs: #73523, #72218, #71679
#71323, #71035, #71177, #70713
2021-06-03 12:43:06 +01:00
David Roberts 0059c59e25
[ML] Make ml_standard tokenizer the default for new categorization jobs (#72805)
Categorization jobs created once the entire cluster is upgraded to
version 7.14 or higher will default to using the new ml_standard
tokenizer rather than the previous default of the ml_classic
tokenizer, and will incorporate the new first_non_blank_line char
filter so that categorization is based purely on the first non-blank
line of each message.

The difference between the ml_classic and ml_standard tokenizers
is that ml_classic splits on slashes and colons, so creates multiple
tokens from URLs and filesystem paths, whereas ml_standard attempts
to keep URLs, email addresses and filesystem paths as single tokens.

It is still possible to config the ml_classic tokenizer if you
prefer: just provide a categorization_analyzer within your
analysis_config and whichever tokenizer you choose (which could be
ml_classic or any other Elasticsearch tokenizer) will be used.

To opt out of using first_non_blank_line as a default char filter,
you must explicitly specify a categorization_analyzer that does not
include it.

If no categorization_analyzer is specified but categorization_filters
are specified then the categorization filters are converted to char
filters applied that are applied after first_non_blank_line.

Closes elastic/ml-cpp#1724
2021-06-01 15:11:32 +01:00
Tanguy Leroux d499061c30
Mute RestClientMultipleHostsIntegTests.testCancelAsyncRequests (#73592)
Relates #45577
2021-06-01 14:57:51 +02:00
Rene Groeschke b2a183baf0
Move public build api into included build (#72861)
This moves the public build api and plugins into a separete included build called 'build-tools' 
and we removed the duplication of included buildSrc twice (2nd import as build-tools).

The elasticsearch internal build logic is kept in build-tools-internal as included build which allows us better handling of this project that its just being an buildSrc project (e.g. we can reference tasks directly from the root build etc.)

Convention logic applied to both projects will live in a new build-conventions project.
2021-06-01 11:19:30 +02:00
Martijn van Groningen 28d69de040
Make assertion less flaky in IndicesClientIT.testDataStreams(). (#73556)
Instead of assertion that store stats has a non zero value,
just verify that a value is returned. Verifying whether a
non value or specific value is returned, isn't the purpose
of hlrc integration tests.

Closes #60461
2021-05-31 15:30:13 +02:00
Dan Hermann 40a029f5ba
Deprecate freeze index API (#72618) 2021-05-27 15:14:50 -05:00
Ryan Ernst 6c4c4a0ecb
Upgrade Azure SDK and Jackson (#72833) (#72995)
This commit upgrades the Azure SDK to 12.11.0 and Jackson to 2.12.2. The
Jackson upgrade must happen at the same time due to Azure depending on
this new version of Jackson.

closes #66555
closes #67214

Co-authored-by: Francisco Fernández Castaño <francisco.fernandez.castano@gmail.com>
2021-05-27 07:55:18 -07:00
Ignacio Vera 409b6cefe3
Add painless script support for geo_shape field (#72886)
Users can access the centroid, bounding box and dimensional type of the shape.
2021-05-26 18:55:45 +02:00
David Roberts ffa38331c9 Muting ILMDocumentationIT.testAddSnapshotLifecyclePolicy
Due to https://github.com/elastic/elasticsearch/issues/73317
2021-05-24 14:54:17 +01:00
Greg Back 6f46ae43e3
Fix comment about number of roles (#73163)
This comment is out of date since #71710.
2021-05-17 11:05:03 -04:00
David Turner b2956b3ae7
Identify cancelled tasks in list tasks API (#72931)
This commit adds a `cancelled` flag to each cancellable task in the
response to the list tasks API, allowing users to see that a task has
been properly cancelled and will complete as soon as possible.

Closes #72907
2021-05-17 11:02:50 +01:00
Nhat Nguyen 44fc661835
Add point in time to HLRC (#72167)
Closes #70593
2021-05-12 17:59:25 -04:00
Przemyslaw Gomulka 85ed9100df
[Rest Api Compatibility] Typed endpoints for search and related endpoints (#72155)
Implements a V7 compatible typed endpoints for REST for search related apis
retrofits the REST layer change removed in #41640

relates main meta issue #51816
relates types removal issue #54160
2021-05-12 21:00:19 +02:00
Ioannis Kakavas b826703e21
Enroll node API (#72129)
Enroll node API can be used by new nodes in order to join an
existing cluster that has security features enabled. The response
of a call to this API contains all the necessary information that
the new node requires in order to configure itself and bootstrap
trust with the existing cluster.
2021-05-12 08:45:02 +03:00
Yang Cheng d33d41d916
[HLRC] change GET method to POST in Searchtemplate (#72851)
Fix #72073
Close #72397
2021-05-11 08:12:01 -06:00
Ryan Ernst 8cd3944a0a
Revert "Upgrade Azure SDK and Jackson (#72833)"
This reverts commit dca0e92bef.
2021-05-06 20:51:31 -07:00
Ryan Ernst dca0e92bef
Upgrade Azure SDK and Jackson (#72833)
This commit upgrades the Azure SDK to 12.11.0 and Jackson to 2.12.2. The
Jackson upgrade must happen at the same time due to Azure depending on
this new version of Jackson.

closes #66555
closes #67214
2021-05-06 20:36:42 -07:00
Rene Groeschke e609e07cfe
Remove internal build logic from public build tool plugins (#72470)
Extract usage of internal API from TestClustersPlugin and PluginBuildPlugin and related plugins and build logic

This includes a refactoring of ElasticsearchDistribution to handle types
better in a way we can differentiate between supported Elasticsearch
Distribution types supported in TestCkustersPlugin and types only supported
in internal plugins.

It also introduces a set of internal versions of public plugins.

As part of this we also generate the plugin descriptors now.

As a follow up on this we can actually move these public used classes into 
an extra project (declared as included build)

We keep LoggedExec and VersionProperties effectively public And workaround for RestTestBase
2021-05-06 14:02:35 +02:00
Tanguy Leroux 2fee5e0990
Mute ReindexIT.testDeleteByQuery (#72546)
Relates #60811
2021-04-30 14:09:28 +02:00
William Brafford 8f0e5dcadd
Feature reset integration test should tolerate failed resets (#72326)
* Remove redundant section of test
* Mention in javadoc that we check task index specifically
2021-04-27 15:51:57 -04:00
William Brafford fc7c06d8a1
Make feature reset API response more informative (#71240)
Previously, the ResetFeatureStateStatus object captured its status in a
String, which meant that if we wanted to know if something succeeded or
failed, we'd have to parse information out of the string. This isn't a
good way of doing things.

I've introduced a SUCCESS/FAILURE enum for status constants, and added a
check for failures in the transport action. We return a 207 if some but not all
reset actions fail, and for every failure, we also return information about the
exception or error that caused it.

Co-authored-by: Jay Modi <jaymode@users.noreply.github.com>
2021-04-27 13:47:10 -04:00
Rene Groeschke 5bcd02cb4d
Restructure build tools java packages (#72030)
Related to #71593 we move all build logic that is for elasticsearch build only into
the org.elasticsearch.gradle.internal* packages

This makes it clearer if build logic is considered to be used by external projects
Ultimately we want to only expose TestCluster and PluginBuildPlugin logic
to third party plugin authors.

This is a very first step towards that direction.
2021-04-26 14:53:55 +02:00
Lisa Cawley 52c88a763e
[DOCS] Add runtime_mappings to update datafeed API in HLRC (#71772)
Co-authored-by: David Kyle <david.kyle@elastic.co>
2021-04-22 08:22:13 -07:00
Tamara Braun e5518e07f1
Fix Suppressing Interrupted Flag in Client (#68999) 2021-04-20 17:23:00 -05:00
Jay Modi a7dbb31765
Add Fleet action results system data stream (#71667)
This commit adds support for system data streams and also the first use
of a system data stream with the fleet action results data stream. A
system data stream is one that is used to store system data that users
should not interact with directly. Elasticsearch will manage these data
streams. REST API access is available for external system data streams
so that other stack components can store system data within a system
data stream. System data streams will not use the system index read and
write threadpools.
2021-04-20 13:33:12 -06:00
Tanguy Leroux 571acfe3e3
Add Searchable Snapshots Cache Stats API to HLRC (#71858)
In #71701 we added a new REST API that provides statistics 
about the searchable snapshots cache on Frozen Tier.

This commit adds the necessary plumbing to expose this API 
in the High Level REST Client. It also exposes the documentation 
of the Mount Snapshot API that was created in #68949 but not 
made accessible.
2021-04-20 13:35:21 +02:00
bellengao 7cd0bdcd9c
Add support for require_alias parameter to high level rest client (#67865)
The main changes are:
* Set the require_alias parameter in the reindex, index and update methods in RequestConverters class.
* Test that the parameter can be parsed correctly.

Closes #67819.
2021-04-20 13:08:44 +02:00
Ioannis Kakavas 5a7b1c6c35
Add new predefined reserved roles (#71710)
This change adds two new reserved roles in Elasticsearch, viewer
and editor
Solutions will use these roles in order to provide,out of the box,
a role for full access to data and configuration with no access to
cluster management(editor) and a role for read only access to data
again with no access to cluster management (viewer).
2021-04-20 11:33:57 +03:00
Dan Hermann eb345b2a8f
Deprecate legacy index template API endpoints (#71309) 2021-04-16 08:07:28 -05:00
Benjamin Trent 01fc8ed246
[ML] adding ability to update runtime_mappings via datafeed config update API (#71707)
Adds runtime_mappings as an updatable field via datafeed config update.

closes: #71702
2021-04-15 09:44:34 -04:00
Przemko Robakowski 39eb12a972
Enable GeoIP downloader by default (#71505)
This change enables GeoIP downloader by default.
It removes feature flag but adds flag that is used by tests to disable it again (as we don't want to hammer GeoIP database service with every test cluster we spin up).

Relates to #68920
2021-04-15 12:28:37 +02:00
David Roberts c4364584aa
[ML] Use feature reset API in ML REST test cleanup (#71552)
Now that we have a feature reset API, we should use
this for cleaning up in between tests instead of running
lots of bespoke cleanup code.

During testing of this change we found we need to
delete custom cluster state as part of the reset process,
so this PR also implements that.

Additionally we no longer assign persistent tasks
during feature reset.
2021-04-13 16:04:47 +01:00
Andrei Stefan b8266e5fb2
Adds `runtime_mappings` to EQL and SQL requests (#71356)
* Adds `runtime_mappings` to EQL and SQL requests allowing users to
define search time runtime fields which will be used in queries
2021-04-07 19:37:57 +03:00
Benjamin Trent 75e74918c2
[ML] updating rest and HLRC ML tests to wait for no initializing shards & necessary indices (#71146)
In the high level rest client tests, we should clean up trained models, same as we do for datafeeds, jobs, etc.

But, to delete a trained model, it needs to not be used in a pipeline, so we grab the model stats. 

This commit does:
- waiting for no more initializing shards when ML integration tests are cleaning up before attempting to delete things
- The specific inference stats test now waits for the `.ml-stats-00001` index to exist before continuing 
- And we make sure that config index is green before attempting to run the `pipeline_inference.yml` tests

closes https://github.com/elastic/elasticsearch/issues/71139
2021-04-02 11:59:26 -04:00
Mark Vieira af54635bb2 Mute MachineLearningIT.testGetTrainedModelsStats 2021-03-31 12:31:46 -07:00
Benjamin Trent ec9d0624c9
[ML] fixing feature reset integration tests (#71081)
previously created pipelines referencing ML models were not being appropriately deleted in upstream tests.

This commit ensures that machine learning removes relevant pipelines from cluster state after tests complete

closes #71072
2021-03-31 07:17:53 -04:00
Benjamin Trent e78cf3c58d
Test mute for ml reset change #71072 (#71075) 2021-03-30 14:09:44 -04:00
Jim Ferenczi fa88a46b9c
Expose if a field is a metadata field in the field capabilities response (#69977)
This change exposes for each field in the _field_caps response if the field is a metadata field.
This is needed for consumers of this API that want to filter these fields. Currently ML keeps a static list
and QL checks that the family type starts with `_`. In order to ease the addition of new metadata fields, this
change reworks the strategy in this solution and now only checks for the new flag.
Note that the new flag is also applied at the coordinator level in a best-effort to apply the logic on older nodes
in a mixed-version cluster.
2021-03-30 12:13:03 +02:00
Dan Hermann 2c6ba92d46
Improve data stream rollover and simplify cluster metadata validation for data streams (#70934) 2021-03-29 07:36:44 -05:00
Yang Wang 3725cb53da
Support metadata on API keys (#70292)
This PR adds metadata support for API keys. Metadata are of type 
Map<String, Object> and can be optionally provided at API key creation time.
It is returned as part of GetApiKey response. It is also stored as part of 
the authentication object to transfer throw the wire.
Note that it is not yet searchable and not exposed to any ingest processors.
They will be handled by separate PRs.
2021-03-29 09:23:30 +11:00
Mark Vieira 6339691fe3
Consolidate REST API specifications and publish under Apache 2.0 license (#70036) 2021-03-26 16:20:14 -07:00
Benjamin Trent b796632582
[ML] Allow datafeed and job configs for datafeed preview API (#70836)
Previously, a datafeed and job must already exist for the `_preview` API to work.

With this change, users can get an accurate preview of the data that will be sent to the anomaly detection job
without creating either of them. 

closes https://github.com/elastic/elasticsearch/issues/70264
2021-03-26 12:52:23 -04:00
Andrei Dan 03e2cec5a2
Fix flakiness in testRetryPolicy (#70769)
Retrying a retryable step when the ILM loop is configured to `1s` (as we do in 
the integration tests) is a recipe for flakiness - as ILM keeps moving a failed
step from `ERROR` back into the failed step for it to be retried, and the retry
step API fails if executed when ILM is not in the `ERROR` step.

This makes the documentation integration test more lenient.
2021-03-25 09:41:57 +00:00
Nhat Nguyen 5bb440cdca
Move point in time to server (#70704)
This change moves the implementation of point in time to the server package.
2021-03-24 14:29:20 -04:00
Tanguy Leroux 222f2a0f6c
Add documentation for Clone Snapshot Java API (#70720)
This commit adds some missing documentation about 
the Clone Snapshot Java API.

Relates #63863
2021-03-24 15:33:09 +01:00
Henning Andersen 8091fd22a1
Fix HLRC MigrateAction equals (#70360)
HLRC MigrateAction.equals did not consider the `enabled` flag, fixed
2021-03-24 14:07:23 +01:00
Przemko Robakowski f5b7aad8b7
Add stats endpoint to GeoIpDownloader (#70282)
This change adds _geoip/stats endpoint that can be used to collect basic data about geoip downloader (successful, failed and skipped downloads, current db count and total time spent downloading).
It also fixes missing/wrong origins for clients that will break if used with security.

Relates to #68920
2021-03-23 14:34:32 +01:00
Dimitris Athanasiou 8392d8734c
[ML] Ensure auc_roc curve is monotonic (#70628)
As we collect points for our auc roc curve from two
different percentiles aggregations, the result may
contain points with equal threshold that are not
monotonic. This is because the percentiles aggregation
is an approximation.

This commit ensures the fina auc roc curve we calculate
is monotonic by collapsing points of equal threshold into
a single point that is the average of the equal threshold
points it represents.
2021-03-23 09:55:59 +02:00
Adam Locke 9f2aaf907a Mute ILMDocumentationIT.testRetryPolicy 2021-03-18 17:00:11 -07:00
William Brafford 624ee45a8e
Add API for resetting state of a `SystemIndexPlugin` (#69469)
When we disable access to system indices, plugins will still need
a way to erase their state. The obvious and most pressing use
case for this is in tests, which need to be able to clean up the
state of a cluster in between groups of tests.

* Use a HandledTransportAction for reset action

My initial cut used a TransportMasterNodeAction, which requires code
that carefully manipulates cluster state. At least for the first cut and
testing, it seems like it will be much easier to use a client within a
HandledTransportAction, which effectively makes the
TransportResetFeatureStateAction a class that dispatches other transport
actions to do the real work.

* Clean up code by using a GroupedActionListener

* ML feature state cleaner

* Implement Transform feature state reset

* Change _features/reset path to _features/_reset

Out of an abundance of caution, I think the "reset" part of this path
should have a leading underscore, so that if there's ever a reason to
implement "GET _features/<feature_id>" we won't have to worry about
distinguishing "reset" from a feature name.

Co-authored-by: Gordon Brown <gordon.brown@elastic.co>
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
2021-03-17 16:14:34 -04:00
Andrei Dan 4b02e84058
Relax the assertion (#70308)
This relaxes the assertion to account for the case when the client
executes the request successfully before we cancel the underlying http
request.
2021-03-11 17:04:53 +00:00
Hendrik Muhs 61f0c4749d
[Transform] Enhance transform role checks (#70139)
improve robustness and ux in case of a missing transform node:

 - warn if cluster lacks a transform node in all API's (except DELETE)
 - report waiting state in stats if transform waits for assignment
 - cancel p-task on stop transform even if config has been deleted

relates #69518
2021-03-10 10:17:10 +01:00
Christoph Büscher d8ef26ad26
Add earlier validation for some SearchSourceBuilder settings (#69548)
Currently we check several search parameters for illegal values in their
SearchSourceBuilder setters, e.g. negative values throw IAE for: `size`,
`terminateAfter` and `trackTotalHits`.

The validation in the builder setters are used when parsing the above as rest
request parameters, however we currently don't check values when parsing them
from the search request body. This leads to builders with invalid parameters
that sometimes get caucht later (e.g. a negative size is triggering an
IllegalArgumentException in TotalHitCountCollector), but we should validate and
throw errors early.

This PR changes the parsing in SearchSourceBuilder to use the setters, adds
tests and also adds a deprecation for allowing a size parameter of -1, currently
meaning an "unset" value.

Closes #54958
2021-03-09 17:15:27 +01:00
Lee Hinman 5df763fc66
Reject creating ILM policies with phase timings are not >= previous phase (#70089)
It can be confusing to configure policies with phase timings that get smaller, because phase timings
are absolute. To make things a little clearer, this commit now rejects policies where a configured
min_age is less than a previous phase's min_age.

This validation is added only to the PutLifecycleAction.Request instead of the
TimeseriesLifecycleType class because we cannot do this validation every time a lifecycle is
created or else we will block cluster state from being recoverable for existing clusters that may
have invalid policies.

Resolves #70032
2021-03-08 12:40:20 -07:00
Przemko Robakowski 6ab35978f2
Fix RestClientMultipleHostsIntegTests.testCancelAsyncRequests (#70003)
Instead of using the same handler with new latches each time, test now registers completely new handler for every request. This prevents race condition that led to deadlock and timeout.
Local testing shows around 500k iterations without a failure.

Closes #45577
2021-03-05 18:52:06 +01:00
David Turner 864ff66f68
Unique names for bulk processor scheduler threads (#69432)
Today every `BulkProcessor` creates two scheduler threads, both called
`[node-name][scheduler][T#1]`, which is also the name of the main
scheduler thread for the node. The duplicated thread names make it
harder to interpret a thread dump.

This commit makes the names of these threads distinct.

Closes #68470
2021-03-04 12:05:24 +00:00
Jake Landis 63e9378dc8
LLRC - support new data roles (#66947)
This commit introduces data_content, data_hot, data_warm,
data_cold, and data_frozen to the low level REST client (LLRC).
Since the LLRC only cares about dedicated masters this change
simply makes the LLRC aware of the new roles and does not have
any functional impact. The tests have been adjusted accordingly.
2021-03-02 18:25:06 -06:00
Gordon Brown ce8a0c0cea
Change Get Snapshottable Features endpoint to `_features` (#69755)
The endpoint `_snapshottable_features` is long and implies incorrect
things about this API - it is used not just for snapshots, but also for
the upcoming reset API. Following discussions on the team, this commit
changes the endpoint to `_features` and removes the connection between
this API and snapshots, as snapshots are not the only use for the output
of this API.
2021-03-02 11:30:02 -07:00
Benjamin Trent f8de51b6ba
[ML] fix using deprecated get trained models options (#69597)
There were still some small places where we were utilizing deprecated options for getting
trained models.

This commit addresses it and utilizes the new API format.
2021-02-25 09:09:31 -05:00
Andrei Stefan df74efb9ef
Fix the bwc serialization tests (#69534) 2021-02-24 17:42:11 +02:00
Andrei Stefan 22c880e641
EQL: Adds "fields" request field to the eql request (#68962) 2021-02-23 16:57:30 +02:00
Benjamin Trent 2ee6dc37b6
[ML][HLRC] adds put and delete trained model alias APIs to rest high-level client (#69214)
adds put (and reassign) and delete trained model alias APIs to the rest high-level client.

This adds some serialization objects and request wrappers.
2021-02-19 14:18:26 -05:00
Dimitris Athanasiou 7fb98c0d3c
[ML] Add runtime mappings to data frame analytics source config (#69183)
Users can now specify runtime mappings as part of the source config
of a data frame analytics job. Those runtime mappings become part of
the mapping of the destination index. This ensures the fields are
accessible in the destination index even if the relevant data frame
analytics job gets deleted.

Closes #65056
2021-02-19 16:29:19 +02:00
Joe Gallo 7e7c5db74e
Rename max_single_primary_size to max_primary_shard_size (#69239) 2021-02-18 21:13:58 -05:00
Joe Gallo d24f5cbc91
Add max_single_primary_size as a condition for the ILM rollover action (#68917) 2021-02-18 09:28:58 -05:00
Mark Vieira 5341a8c419
Separate task output directories for YAML REST testing (#68943) 2021-02-17 15:16:42 -08:00
Lisa Cawley f37bd12735
Adds retention policy to HLRC put transform API (#68768)
Co-authored-by: Hendrik Muhs <hendrik.muhs@elastic.co>
2021-02-17 09:53:37 -08:00
Christoph Büscher 45b1c46f1e
Relaxing score comparisons for rank_eval tests (#68976)
The currently allowed delta in this tests is to strict to account for slight floating point
differences across different platforms, e.g. ARM.

Closes #68936
2021-02-16 12:47:14 +01:00
Jason Tedor 985ae9a1ae
Add Searchable Snapshots APIs to HLRC (#68949)
This commit adds the Searchable Snapshots APIs (the mount snapshot API)
to the high-level REST client.
2021-02-12 07:53:30 -05:00
Gordon Brown 3f6472de74
Introduce "Feature States" for managing snapshots of system indices (#63513)
This PR expands the meaning of `include_global_state` for snapshots to include system indices. If `include_global_state` is `true` on creation, system indices will be included in the snapshot regardless of the contents of the `indices` field. If `include_global_state` is `true` on restoration, system indices will be restored (if included in the snapshot), regardless of the contents of the `indices` field. Index renaming is not applied to system indices, as system indices rely on their names matching certain patterns. If restored system indices are already present, they are automatically deleted prior to restoration from the snapshot to avoid conflicts.

This behavior can be overridden to an extent by including a new field in the snapshot creation or restoration call, `feature_states`, which contains an array of strings indicating the "feature" for which system indices should be snapshotted or restored. For example, this call will only restore the `watcher` and `security` system indices (in addition to `index_1`):

```
POST /_snapshot/my_repository/snapshot_2/_restore
{
  "indices": "index_1",
  "include_global_state": true,
  "feature_states": ["watcher", "security"]
}
```

If `feature_states` is present, the system indices associated with those features will be snapshotted or restored regardless of the value of `include_global_state`. All system indices can be omitted by providing a special value of `none` (`"feature_states": ["none"]`), or included by omitting the field or explicitly providing an empty array (`"feature_states": []`), similar to the `indices` field.

The list of currently available features can be retrieved via a new "Get Snapshottable Features" API:
```
GET /_snapshottable_features
```

which returns a response of the form:
```
{
    "features": [
        {
            "name": "tasks",
            "description": "Manages task results"
        },
        {
            "name": "kibana",
            "description": "Manages Kibana configuration and reports"
        }
    ]
}
```

Features currently map one-to-one with `SystemIndexPlugin`s, but this should be considered an implementation detail. The Get Snapshottable Features API and snapshot creation rely upon all relevant plugins being installed on the master node.

Further, the list of feature states included in a given snapshot is exposed by the Get Snapshot API, which now includes a new field, `feature_states`, which contains a list of the feature states and their associated system indices which are included in the snapshot. All system indices in feature states are also included in the `indices` array for backwards compatibility, although explicitly requesting system indices included in a feature state is deprecated. For example, an excerpt from the Get Snapshot API showing `feature_states`:
```
"feature_states": [
    {
        "feature_name": "tasks",
        "indices": [
            ".tasks"
        ]
    }
],
"indices": [
    ".tasks",
    "test1",
    "test2"
]
```

Co-authored-by: William Brafford <william.brafford@elastic.co>
2021-02-11 11:55:14 -07:00
Hendrik Muhs 967505c3d5
[Transform] add retention policy to documentation IT (#68728)
add retention policy and sync to documentation IT, add builders for sync
and retention policy

relates #67832
2021-02-11 15:59:58 +01:00
Tim Vernum 528d029524
Add grant-api-key to HLRC (#68190)
This adds support for "Grant API Key" to the Java High Level Rest
Client.

This API was added in Elasticsearch 7.7 but did not have explicit support
in the HLRC.
2021-02-09 16:51:03 +11:00
Mark Vieira 71c0821ffc
Keep low level rest client under Apache 2 software license (#68694) 2021-02-08 11:13:31 -08:00
Rory Hunter 2d44cce31e
Replace NOT operator with explicit `false` check - part 9 (#68645)
Part 9.

We have an in-house rule to compare explicitly against `false` instead
of using the logical not operator (`!`). However, this hasn't
historically been enforced, meaning that there are many violations in
the source at present.

We now have a Checkstyle rule that can detect these cases, but before we
can turn it on, we need to fix the existing violations. This is being
done over a series of PRs, since there are a lot to fix.
2021-02-08 15:28:57 +00:00
Rory Hunter 780f273067
Replace NOT operator with explicit `false` check - part 8 (#68625)
Part 8.

We have an in-house rule to compare explicitly against `false` instead
of using the logical not operator (`!`). However, this hasn't
historically been enforced, meaning that there are many violations in
the source at present.

We now have a Checkstyle rule that can detect these cases, but before we
can turn it on, we need to fix the existing violations. This is being
done over a series of PRs, since there are a lot to fix.
2021-02-08 15:20:34 +00:00
Hendrik Muhs 54ed2e37d9
[Transform] implement retention policy to delete data from a transform (#67832)
add a retention policy to transform to delete data that is considered outdated as part of a
transform checkpoint.

fixes #67916
2021-02-08 15:06:15 +01:00
Joe Gallo 4d18334442
Add max_single_primary_size as a condition for the rollover index API (#67842) 2021-02-03 10:39:06 -05:00
Mark Vieira a92a647b9f Update sources with new SSPL+Elastic-2.0 license headers
As per the new licensing change for Elasticsearch and Kibana this commit
moves existing Apache 2.0 licensed source code to the new dual license
SSPL+Elastic license 2.0. In addition, existing x-pack code now uses
the new version 2.0 of the Elastic license. Full changes include:

 - Updating LICENSE and NOTICE files throughout the code base, as well
   as those packaged in our published artifacts
 - Update IDE integration to now use the new license header on newly
   created source files
 - Remove references to the "OSS" distribution from our documentation
 - Update build time verification checks to no longer allow Apache 2.0
   license header in Elasticsearch source code
 - Replace all existing Apache 2.0 license headers for non-xpack code
   with updated header (vendored code with Apache 2.0 headers obviously
   remains the same).
 - Replace all Elastic license 1.0 headers with new 2.0 header in xpack.
2021-02-02 16:10:53 -08:00
bellengao 8d81fe1679
Fix wrong totalIterationCount when running BulkIndexBenchmark (#64515) 2021-02-02 16:55:51 +01:00
bellengao fb0fcda5d4
Fix recording sample for formal run iterations in BulkBenchmarkTask (#64514) 2021-02-02 15:50:28 +01:00
Hakky54 937c1c73f9
Added support for passing http query parameters (#67802) 2021-02-02 14:56:35 +01:00
Sylvain Wallez 1dcacfeabf
[client] Add client metadata header on RestClient requests (#66303)
Adds a X-Elastic-Client-Meta header to http requests sent by RestClient. This
header contains information about the runtime environment that is meant to
allow analyzing usage context by collecting this information on the receiving
side of requests, like a proxy server in front of ES.

Using a custom header allows client applications to change the User-Agent
header for their own purpose without losing this information.
2021-02-02 11:09:51 +01:00
Valeriy Khakhutskyy 78368428b3
[ML] Add early stopping DFA configuration parameter (#68099)
The PR adds early_stopping_enabled optional data frame analysis configuration parameter. The enhancement was already described in elastic/ml-cpp#1676 and so I mark it here as non-issue.
2021-02-01 11:41:28 +01:00
bellengao d69c03359f
Support max_single_primary_size in Resize Action and exposed in ILM (#67705) 2021-01-29 15:29:45 -05:00
Rory Hunter ad1f876daa
Replace NOT operator with explicit `false` check (#67817)
We have an in-house rule to compare explicitly against `false` instead
of using the logical not operator (`!`). However, this hasn't
historically been enforced, meaning that there are many violations in
the source at present.

We now have a Checkstyle rule that can detect these cases, but before we
can turn it on, we need to fix the existing violations. This is being
done over a series of PRs, since there are a lot to fix.
2021-01-26 14:47:09 +00:00
Dimitris Athanasiou 5c961c1c81
[ML] Expand regression/classification hyperparameters (#67950)
Expands data frame analytics regression and classification
analyses with the followin hyperparameters:

- alpha
- downsample_factor
- eta_growth_rate_per_tree
- max_optimization_rounds_per_hyperparameter
- soft_tree_depth_limit
- soft_tree_depth_tolerance
2021-01-26 12:56:41 +02:00
Mayya Sharipova ef24722b11
Remove QueryDSLDocumentationTests.java (#67665)
This removes unused QueryDSLDocumentationTests.

It was used before only in transport client docs, but
PR #42483 removed these docs.

Relates to #42483
2021-01-21 17:25:18 -05:00
Przemysław Witek d78f39c2ad
[Transform] Add support for search-time runtime fields. (#67643) 2021-01-21 13:27:42 +01:00
Hendrik Muhs 951d822604
[Transform] report last search time in transform stats (#66718)
transforms reports the the last time changes where detected with changes_last_detected_at, however
that doesn't tell a user it searched for changes, this PR adds a field last_search_time to report
when transform searched for changes the last time.

fixes #66410
relates #66367
2021-01-20 15:00:12 +01:00
Benjamin Trent 149c397345
[ML] fix multi_encoding test failure (#67603) 2021-01-15 14:57:16 -05:00
Benjamin Trent cb34ca601c
[ML] add new multi custom processor for data frame analytics and model inference (#67362)
This adds the multi custom feature processor to data frame analytics and inference.

The `multi_encoding` processor allows custom processors to be chained together and use the outputs from one processor as the inputs to another.
2021-01-15 12:43:14 -05:00
Benjamin Trent 5cf569ffff
[ML] move find file structure finder in Rest high Level client to its new endpoint and plugin (#67290)
Find file structure finder is now its own plugin, and separated from the ml plugin.

This commit updates the rest high level client to reflect this.

Additionally, this adjusts the internal and client object names from `FileStructure` to the more general `TextStructure`
2021-01-14 08:16:52 -05:00
Benjamin Trent a437af3406
[ML] rename the text structure action and update required permissions (#67388)
This renames the text structure finder action to match the plugin name.
Also, this adds a new reserved role name so that adding specific permissions for this API is simple.
2021-01-13 14:11:25 -05:00
David Turner c20b92ff5f
Respect CloseIndexRequest#waitForActiveShards in HLRC (#67374)
Also includes some changes for `master` that make #67246 a bit smaller.

Relates #67158
2021-01-13 14:34:07 +00:00
Benjamin Trent af179ab2f5
[ML] move find file structure to a new API endpoint (#67123)
This introduces a new `text-structure` plugin. This is the new home of the find file structure API. 

The old REST URL is still available but is deprecated.

The new URL is: `_text_structure/find_structure`. All parameters and behavior are unchanged.

Changes to the high-level REST client and docs will be in separate commit.

related to: https://github.com/elastic/elasticsearch/issues/67001
2021-01-11 08:56:02 -05:00
Przemyslaw Gomulka dea6bdfa36
VND content types comparisons with corresponding Non VND types (#67139)
VND types like application/vnd.elasticsearch+json
(XContentType.VND_JSON) when compared with
application/json (XContentType.JSON) should use `canonnical`

closes #67064
2021-01-07 12:15:19 +01:00
Andrei Dan 8b158ad6b9
Test: fix testValidatePhases for delete phase (#67082)
The delete phase can no longer be defined as empty, it requires at least
one action to be defined. This fixes the test that used an empty collection
of actions for the delete phase to include the delete action.
2021-01-06 12:20:20 +00:00
Daniel Baskal 7f39514690
[ILM] check if delete phase has actions (#65847) (#66664)
Validate the delete phase to define actions.
2021-01-06 10:19:30 +00:00