Commit Graph

517 Commits

Author SHA1 Message Date
Samidh 199ecbbf77 [Docs] Fix typo in start-watch-service.asciidoc (#57182) 2020-05-27 15:51:07 +02:00
Benjamin Trent 8fed077b0a
[ML] relax throttling on expired data cleanup (#56711)
Throttling nightly cleanup as much as we do has been over cautious.

Night cleanup should be more lenient in its throttling. We still
keep the same batch size, but now the requests per second scale
with the number of data nodes. If we have more than 5 data nodes,
we don't throttle at all.

Additionally, the API now has `requests_per_second` and `timeout` set.
So users calling the API directly can set the throttling.

This commit also adds a new setting `xpack.ml.nightly_maintenance_requests_per_second`.
This will allow users to adjust throttling of the nightly maintenance.
2020-05-18 07:21:06 -04:00
Kamyar Ghajar e9f1896e3d [Docs] Update multi-search.asciidoc (#56284)
The documentation shows the wrong command for a multi-search async call.
2020-05-06 16:56:58 +02:00
Andrei Dan 9bcf89b1e2
HLRC: document index template v2 and component template APIs (#56136)
This documents the index template v2 and component template APIs in the
high level rest client.
2020-05-05 16:43:16 +01:00
Dimitris Athanasiou 6bf3834059
[ML] Add loss_function to regression (#56118)
Adds parameters `loss_function` and `loss_function_parameter`
to regression.
2020-05-05 12:36:05 +03:00
Hendrik Muhs 33f134f556
[Transform] add throttling (#56007)
add throttling to transform, throttling will slow down search requests by delaying the execution based on a documents per second metric. 

fixes #54862
2020-05-05 11:34:54 +02:00
David Roberts 8906e76079
[ML] Return assigned node in start/open job/datafeed response (#55473)
Adds a "node" field to the response from the following endpoints:

1. Open anomaly detection job
2. Start datafeed
3. Start data frame analytics job

If the job or datafeed is assigned to a node immediately then
this field will return the ID of that node.

In the case where a job or datafeed is opened or started lazily
the node field will contain an empty string.  Clients that want
to test whether a job or datafeed was opened or started lazily
can therefore check for this.

Fixes #54067
2020-04-22 08:44:57 +01:00
Yang Wang 92427d3758
Remove local parameter for get field mapping API (#55100)
The local parameter of get field mapping API is marked as deprecated in 7.x.
This PR removes it for v8.0
2020-04-15 12:02:10 +10:00
Yang Wang 6ce88038f2
Deprecate local parameter for get field mapping request (#55014)
The usage of local parameter for GetFieldMappingRequest has been removed from the underlying transport action since v2.0.

This PR deprecates the parameter from rest layer. It will be removed in next major version.
2020-04-12 12:34:44 +10:00
Benjamin Trent 4e1ff31c3c
[ML] add new inference_config field to trained model config (#54421)
A new field called `inference_config` is now added to the trained model config object. This new field allows for default inference settings from analytics or some external model builder. 

The inference processor can still override whatever is set as the default in the trained model config.
2020-04-02 10:34:17 -04:00
Nhat Nguyen ee3d40320a
Broadcast cancellation to only nodes have outstanding child tasks (#54312)
Today when canceling a task we broadcast ban/unban requests to all nodes 
in the cluster. This strategy does not scale well for hierarchical
cancellation. With this change, we will track outstanding child requests
and broadcast the cancellation to only nodes that have outstanding child
tasks. This change also prevents a parent task from sending child
requests once it got canceled.

Relates #50990
Supersedes #51157

Co-authored-by: Igor Motov <igor@motovs.org>
Co-authored-by: Yannick Welsch <yannick@welsch.lu>
2020-04-01 11:22:13 -04:00
Christoph Büscher 610b7bdcbf
[Docs] Add HLRC Async Search API documentation (#54353)
Adds documentation and a corresponding test case containing typical API usage
for the Async Search API to the High Level Rest Client.
2020-03-30 15:32:45 +02:00
David Roberts 8ee770560a
[ML] Add a model memory estimation endpoint for anomaly detection (#53507)
A new endpoint for estimating anomaly detection job
model memory requirements:

POST _ml/anomaly_detectors/estimate_model_memory

Closes #53219
2020-03-24 21:38:19 +00:00
Tom Veasey 58340c2dbe
[ML] Adds the class_assignment_objective parameter to classification (#52763)
Adds a new parameter for classification that enables choosing whether to assign labels to
maximise accuracy or to maximise the minimum class recall.

Fixes #52427.
2020-03-12 18:39:29 +00:00
Maxim b706239145
Add docs for errors in GetAlias API (#51850)
Closes #31499
2020-02-24 17:16:05 +01:00
Benjamin Trent 1c1d45130c
[ML][Inference] don't return inflated definition when storing trained models (#52573)
When `PUT` is called to store a trained model, it is useful to return the newly create model config. But, it is NOT useful to return the inflated definition.

These definitions can be large and returning the inflated definition causes undo work on the server and client side.
2020-02-20 11:25:34 -05:00
OriGlassman d16f92bbae
[DOCS] Fixed "SeachRequest" -> "SearchRequest" typo in HLRC docs (#52144) 2020-02-14 13:41:51 -05:00
Nik Everett 5b2266601b
Implement top_metrics agg (#51155)
The `top_metrics` agg is kind of like `top_hits` but it only works on
doc values so it *should* be faster.

At this point it is fairly limited in that it only supports a single,
numeric sort and a single, numeric metric. And it only fetches the "very
topest" document worth of metric. We plan to support returning a
configurable number of top metrics, requesting more than one metric and
more than one sort. And, eventually, non-numeric sorts and metrics. The
trick is doing those things fairly efficiently.

Co-Authored by: Zachary Tong <zach@elastic.co>
2020-02-14 07:13:52 -05:00
Nik Everett 75d83db8c7
HLRC support for string_stats (#52163)
This adds a builder and parsed results for the `string_stats`
aggregation directly to the high level rest client. Without this the
HLRC can't access the `string_stats` API without the elastic licensed
`analytics` module.

While I'm in there this adds a few of our usual unit tests and
modernizes the parsing.
2020-02-12 18:01:14 -05:00
Raidok 74a5683875 [DOCS] Fix "Asynchronous usage" title in HLRC docs (#52017) 2020-02-07 09:41:58 -05:00
Benjamin Trent c9e285c1e6
[ML][Inference] add tags url param to GET (#51330)
Adds a new URL parameter, `tags` to the GET _ml/inference/<model_id> endpoint.

This parameter allows the list of models to be further reduced to those who contain all the provided tags.
2020-01-24 07:30:56 -05:00
Maxim c01923043d Add Get Source API to the HLRC (#50885)
Relates to #47678
2020-01-23 09:32:30 +01:00
Nhat Nguyen 09b46c8646
Goodbye and thank you synced flush! (#50882)
Synced flush was a brilliant idea. It supports instant recoveries with a 
quite small implementation. However, with the presence of sequence
numbers and retention leases, it is no longer needed. This change
removes it from 8.0.

Relates #5077
2020-01-16 09:43:07 -05:00
Dimitris Athanasiou 4d2be9bd32
[ML] Add num_top_feature_importance_values param to regression and classi… (#50914)
Adds a new parameter to regression and classification that enables computation
of importance for the top most important features. The computation of the importance
is based on SHAP (SHapley Additive exPlanations) method.
2020-01-14 15:01:47 +02:00
Benjamin Trent 4cecb7a5be
[ML][Inference] PUT API (#50852)
This adds the `PUT` API for creating trained models that support our format. 

This includes

* HLRC change for the API
* API creation
* Validations of model format and call
2020-01-11 16:02:56 -05:00
Dimitris Athanasiou af0ce426cc
[ML] Implement force deleting a data frame analytics job (#50553)
Adds a `force` parameter to the delete data frame analytics
request. When `force` is `true`, the action force-stops the
jobs and then proceeds to the deletion. This can be used in
order to delete a non-stopped job with a single request.

Closes #48124
2020-01-03 12:01:41 +02:00
Martijn van Groningen a48d19d73a
Add remote info to the HLRC (#50482)
Unreverts the commit that added the remote info api to HLRC (#49657).

The additional change to the original PR, is that `org.elasticsearch.client.cluster.RemoteConnectionInfo` now parses the initial_connect_timeout field as a string instead of a TimeValue instance.

The reason that this is needed is because that the initial_connect_timeout field in the remote connection api is serialized for human consumption, but not for parsing purposes.
Therefore the HLRC can't parse it correctly (which caused test failures in CI, but not in the PR CI
:( ). The way this field is serialized needs to be changed in the remote connection api, but that is a breaking change. We should wait making this change until rest api versioning is introduced.

Co-Authored-By: j-bean anton.shuvaev91@gmail.com
2019-12-24 13:20:39 +01:00
Martijn van Groningen b7ac7324d2
Revert "Add remote info to the HLRC (#49657)"
This reverts commit fa1a7c57b8.
2019-12-24 11:40:44 +01:00
j-bean fa1a7c57b8 Add remote info to the HLRC (#49657)
Relates to #47678
2019-12-24 10:49:21 +01:00
Przemysław Witek 786ead630a
Implement `precision` and `recall` metrics for classification evaluation (#49671) 2019-12-19 16:07:09 +01:00
Dimitris Athanasiou 269425b54d
[ML] Introduce randomize_seed setting for regression and classification (#49990)
This adds a new `randomize_seed` for regression and classification.
When not explicitly set, the seed is randomly generated. One can
reuse the seed in a similar job in order to ensure the same docs
are picked for training.
2019-12-10 10:22:53 +02:00
Henning Andersen 5b56a990b0
Deprecate sorting in reindex (#49458)
Reindex sort never gave a guarantee about the order of documents being
indexed into the destination, though it could give a sense of locality
of source data.

It prevents us from doing resilient reindex and other optimizations and
it has therefore been deprecated.

Related to #47567
2019-11-29 17:46:44 +01:00
Dimitris Athanasiou bad07b76f7
[ML] Add optional source filtering during data frame reindexing (#49690)
This adds a `_source` setting under the `source` setting of a data
frame analytics config. The new `_source` is reusing the structure
of a `FetchSourceContext` like `analyzed_fields` does. Specifying
includes and excludes for source allows selecting which fields
will get reindexed and will be available in the destination index.

Closes #49531
2019-11-29 14:20:31 +02:00
Benjamin Trent ba914453be
[ML][Inference][HLRC] add GET _stats (#49562) 2019-11-26 09:26:31 -05:00
Benjamin Trent fc7df300a2
[ML][Inference][HLRC] Delete trained model API (#49567) 2019-11-26 07:13:02 -05:00
Dimitris Athanasiou 0390ec3627
[ML] Explain data frame analytics API (#49455)
This commit replaces the _estimate_memory_usage API with
a new API, the _explain API.

The API consolidates information that is useful before
creating a data frame analytics job.

It includes:

- memory estimation
- field selection explanation

Memory estimation is moved here from what was previously
calculated in the _estimate_memory_usage API.

Field selection is a new feature that explains to the user
whether each available field was selected to be included or
not in the analysis. In the case it was not included, it also
explains the reason why.
2019-11-22 20:08:14 +02:00
Lisa Cawley a4efab6ab4
[DOCS] Merge rollup config details into API (#49412) 2019-11-22 08:31:30 -08:00
Benjamin Trent 9006926a15
[ML][Inference][HLRC] GET trained models (#49464) 2019-11-22 07:31:30 -05:00
Przemysław Witek 94ee36d61e
Implement accuracy metric for multiclass classification (#47772) 2019-11-21 13:07:14 +01:00
Michael Basnight 7115d3e370
Add delete alias to the HLRC (#48819)
The delete alias call is a rest only API call, but should still be added
to the rest client. This commit adds it as well as relevant tests.

Ref #47678
2019-11-11 16:47:22 -05:00
James Rodewig fb5667cdb4
[DOCS] Replace `_uid` refs in reindex slicing docs (#48649)
PR #25543 removed the `_uid` field in favor of the `_id` field,
including for use in slicing.

This removes an outdated reference to `_uid` in our reindex docs.
2019-10-29 16:40:39 -04:00
Michael Basnight db6adf8577
[Docs] add missing snapshot restore reference (#45256) 2019-10-28 09:54:39 -05:00
Alexandre Fonseca ef240d8b0c [Docs] Fix opType options in IndexRequest API example. (#48290) 2019-10-22 13:48:24 +02:00
Przemysław Witek 99c912b79f
Make num_top_classes parameter's default value equal to 2 (#48119) 2019-10-17 17:59:22 +02:00
Martijn van Groningen d941e1b3cb
fixed invalid reference 2019-10-15 10:29:47 +02:00
Martijn van Groningen 85ad27e1e7
Merge remote-tracking branch 'es/master' into enrich 2019-10-15 07:19:25 +02:00
debadair a4df0711f3
[DOCS] Updating heading for consistency. (#47619) 2019-10-14 15:59:42 -07:00
Gordon Brown ef7fdf1800
SLM Start/Stop HLRC and docs (#47966)
This commit adds HLRC support and documentation for the SLM Start and
Stop APIs, as well as updating existing documentation where appropriate.

This commit also ensures that the SLM APIs are properly included in the
HLRC documentation.
2019-10-14 15:19:49 -06:00
Martijn van Groningen 6ed7d691b4
Add HLRC support for enrich execute policy API (#47991)
This PR also includes HLRC docs for the enrich stats api.

Relates to #32789
2019-10-14 19:44:47 +02:00
Tanguy Leroux 2891783976
Add Pause/Resume Auto-Follower APIs to High Level REST Client (#47989)
This commit adds support for Pause/Resume Auto-Follower APIs 
to the HLRC, with the documentation.

Relates #47510
2019-10-14 16:23:58 +02:00
Martijn van Groningen e06598ba56
Merge remote-tracking branch 'es/master' into enrich 2019-10-14 10:17:18 +02:00
Yogesh Gaikwad 4ac25f3016
Remove uniqueness constraint for API key name and make it optional (#47549)
Since we cannot guarantee the uniqueness of the API key `name` this commit removes the constraint and makes this field optional.

Closes #46646
2019-10-12 16:58:39 +11:00
Przemysław Witek 9b5770da0e
Add MlClientDocumentationIT tests for classification. (#47569) 2019-10-11 08:21:45 +02:00
Martijn van Groningen 0caca2f73f
Add HLRC support for enrich stats API (#47306)
This PR also includes HLRC docs for the enrich stats api.

Relates to #32789
2019-10-10 08:46:30 +02:00
Martijn van Groningen 957f0fad0f
Merge remote-tracking branch 'es/master' into enrich 2019-10-09 08:48:28 +02:00
Dimitris Athanasiou e99435a7f6
[ML] Additional outlier detection parameters (#47600)
Adds the following parameters to `outlier_detection`:

- `compute_feature_influence` (boolean): whether to compute or not
   feature influence scores
- `outlier_fraction` (double): the proportion of the data set assumed
   to be outlying prior to running outlier detection
- `standardization_enabled` (boolean): whether to apply standardization
   to the feature values
2019-10-07 15:28:21 +03:00
Yogesh Gaikwad 01f91771b4
Add support to retrieve all API keys if user has privilege (#47274)
This commit adds support to retrieve all API keys if the authenticated
user is authorized to do so.
This removes the restriction of specifying one of the
parameters (like id, name, username and/or realm name)
when the `owner` is set to `false`.

Closes #46887
2019-10-07 20:21:10 +11:00
Tal Levy 4d3f6816a7 Merge remote-tracking branch 'elastic/master' into enrich 2019-10-04 13:30:57 -07:00
Lee Hinman 17dc095606
Add API to execute SLM retention on-demand (#47405)
* Add API to execute SLM retention on-demand

This commit adds the `/_slm/_execute_retention` API endpoint. This
endpoint kicks off SLM retention and then returns immediately.

This in particular allows us to run retention without scheduling it
(for entirely manual invocation) or perform a one-off cleanup.

This commit also includes HLRC for the new API, and fixes an issue
in SLMSnapshotBlockingIntegTests where retention invoked prior to the
test completing could resurrect an index the internal test cluster
cleanup had already deleted.

Resolves #46508
Relates to #43663
2019-10-02 10:28:39 -06:00
James Rodewig 7583c07fa8
[DOCS] Reorder index APIs alphabetically (#46981) 2019-10-01 15:13:27 -04:00
István Zoltán Szabó a6c517a96e
[DOCS] Changes wording to move away from data frame terminology in the ES repo (#47093)
* [DOCS] Changes wording to move away from data frame terminology in the ES repo.
Co-Authored-By: Lisa Cawley <lcawley@elastic.co>
2019-10-01 08:04:06 +02:00
Martijn van Groningen 197c1d59d4
Merge remote-tracking branch 'es/master' into enrich 2019-09-30 08:12:07 +02:00
Lisa Cawley 91992a805f
[DOCS] Moves Watcher content into Elasticsearch book (#47147)
Co-Authored-By: James Rodewig <james.rodewig@elastic.co>
2019-09-27 16:05:44 -07:00
Martijn van Groningen f676d9730d
Merge remote-tracking branch 'es/master' into enrich 2019-09-27 13:51:17 +02:00
Hendrik Muhs fd3dc4da77
[Transform] rename data frame transform to transform for hlrc client (#46933)
rename data frame transform to transform for hlrc
2019-09-25 07:38:17 +02:00
Martijn van Groningen afc16ba518
Merge remote-tracking branch 'es/master' into enrich 2019-09-23 09:34:53 +02:00
Lisa Cawley 4da98c9e46
[DOCS] Update data frame transform URLs (#46940) 2019-09-20 13:26:57 -07:00
Lisa Cawley b1bbed84eb
[DOCS] Fixes data frame analytics job terminology in HLRC (#46758) 2019-09-16 10:00:44 -07:00
Lisa Cawley b3dfd6e6d0
[DOCS] Updates dataframe transform terminology (#46642) 2019-09-16 08:28:19 -07:00
Lisa Cawley 7f1e500512
[DOCS] Adds missing icons to Watcher HLRC APIs (#46626) 2019-09-11 16:32:47 -07:00
Lisa Cawley 5105db4eed
[DOCS] Adds missing icons to ILM HLRC APIs (#46633) 2019-09-11 15:45:07 -07:00
Lisa Cawley e3945231d3
[DOCS] Adds missing icons to CCR HLRC APIs (#46631) 2019-09-11 15:35:42 -07:00
Lisa Cawley 7f5353bdcb
[DOCS] Adds missing icons to Graph HLRC APIs (#46630) 2019-09-11 15:20:36 -07:00
Lisa Cawley e304815af2
[DOCS] Add missing icons to security HLRC APIs (#46619) 2019-09-11 13:19:13 -07:00
Lisa Cawley 03116c0f8c
[DOCS] Add missing icons to rollup HLRC APIs (#46617) 2019-09-11 11:48:32 -07:00
Lisa Cawley 7d0beb0b53
[DOCS] Add missing icons to transform HLRC APIs (#46616) 2019-09-11 11:21:45 -07:00
Martijn van Groningen 5d76d2d1e5
Add HLRC support for enrich get policy API. (#45970)
Changed the signature of AbstractResponseTestCase#createServerTestInstance(...)
to include the randomly selected xcontent type. This is needed for the
creating a server response instance with a query which is represented as BytesReference.
Maybe this should go into a different change?

This PR also includes HLRC docs for the get policy api.

Relates to #32789
2019-09-11 14:26:42 +02:00
Lisa Cawley 1e63105e30
[DOCS] Adds missing icons to ML HLRC APIs (#46515) 2019-09-10 08:26:56 -07:00
Lee Hinman 56aabcdd69
Add retention to Snapshot Lifecycle Management (#46407)
This commit adds retention to the existing Snapshot Lifecycle Management feature (#38461) as described in #43663. This allows a user to configure SLM to automatically delete older snapshots based on a number of criteria.

An example policy would look like:

```
PUT /_slm/policy/snapshot-every-day
{
  "schedule": "0 30 2 * * ?",
  "name": "<production-snap-{now/d}>",
  "repository": "my-s3-repository",
  "config": {
    "indices": ["foo-*", "important"]
  },
  // Newly configured retention options
  "retention": {
    // Snapshots should be deleted after 14 days
    "expire_after": "14d",
    // Keep a maximum of thirty snapshots
    "max_count": 30,
    // Keep a minimum of the four most recent snapshots
    "min_count": 4
  }
}
```

SLM Retention is run on a scheduled configurable with the `slm.retention_schedule` setting, which supports cron expressions. Deletions are run for a configurable time bounded by the `slm.retention_duration` setting, which defaults to 1 hour.

Included in this work is a new SLM stats API endpoint available through

``` json
GET /_slm/stats
```

That returns statistics about snapshot taken and deleted, as well as successful retention runs, failures, and the time spent deleting snapshots. #45362 has more information as well as an example of the output. These stats are also included when retrieving SLM policies via the API.

* Add base framework for snapshot retention (#43605)

* Add base framework for snapshot retention

This adds a basic `SnapshotRetentionService` and `SnapshotRetentionTask`
to start as the basis for SLM's retention implementation.

Relates to #38461

* Remove extraneous 'public'

* Use a local var instead of reading class var repeatedly

* Add SnapshotRetentionConfiguration for retention configuration (#43777)

* Add SnapshotRetentionConfiguration for retention configuration

This commit adds the `SnapshotRetentionConfiguration` class and its HLRC
counterpart to encapsulate the configuration for SLM retention.
Currently only a single parameter is supported as an example (we still
need to discuss the different options we want to support and their
names) to keep the size of the PR down. It also does not yet include version serialization checks
since the original SLM branch has not yet been merged.

Relates to #43663

* Fix REST tests

* Fix more documentation

* Use Objects.equals to avoid NPE

* Put `randomSnapshotLifecyclePolicy` in only one place

* Occasionally return retention with no configuration

* Implement SnapshotRetentionTask's snapshot filtering and delet… (#44764)

* Implement SnapshotRetentionTask's snapshot filtering and deletion

This commit implements the snapshot filtering and deletion for
`SnapshotRetentionTask`. Currently only the expire-after age is used for
determining whether a snapshot is eligible for deletion.

Relates to #43663

* Fix deletes running on the wrong thread

* Handle missing or null policy in snap metadata differently

* Convert Tuple<String, List<SnapshotInfo>> to Map<String, List<SnapshotInfo>>

* Use the `OriginSettingClient` to work with security, enhance logging

* Prevent NPE in test by mocking Client

* Allow empty/missing SLM retention configuration (#45018)

Semi-related to #44465, this allows the `"retention"` configuration map
to be missing.

Relates to #43663

* Add min_count and max_count as SLM retention predicates (#44926)

This adds the configuration options for `min_count` and `max_count` as
well as the logic for determining whether a snapshot meets this criteria
to SLM's retention feature.

These options are optional and one, two, or all three can be specified
in an SLM policy.

Relates to #43663

* Time-bound deletion of snapshots in retention delete function (#45065)

* Time-bound deletion of snapshots in retention delete function

With a cluster that has a large number of snapshots, it's possible that
snapshot deletion can take a very long time (especially since deletes
currently have to happen in a serial fashion). To prevent snapshot
deletion from taking forever in a cluster and blocking other operations,
this commit adds a setting to allow configuring a maximum time to spend
deletion snapshots during retention. This dynamic setting defaults to 1
hour and is best-effort, meaning that it doesn't hard stop a deletion
at an hour mark, but ensures that once the time has passed, all
subsequent deletions are deferred until the next retention cycle.

Relates to #43663

* Wow snapshots suuuure can take a long time.

* Use a LongSupplier instead of actually sleeping

* Remove TestLogging annotation

* Remove rate limiting

* Add SLM metrics gathering and endpoint (#45362)

* Add SLM metrics gathering and endpoint

This commit adds the infrastructure to gather metrics about the different SLM actions that a cluster
takes. These actions are stored in `SnapshotLifecycleStats` and perpetuated in cluster state. The
stats stored include the number of snapshots taken, failed, deleted, the number of retention runs,
as well as per-policy counts for snapshots taken, failed, and deleted. It also includes the amount
of time spent deleting snapshots from SLM retention.

This commit also adds an endpoint for retrieving all stats (further commits will expose this in the
SLM get-policy API) that looks like:

```
GET /_slm/stats
{
  "retention_runs" : 13,
  "retention_failed" : 0,
  "retention_timed_out" : 0,
  "retention_deletion_time" : "1.4s",
  "retention_deletion_time_millis" : 1404,
  "policy_metrics" : {
    "daily-snapshots2" : {
      "snapshots_taken" : 7,
      "snapshots_failed" : 0,
      "snapshots_deleted" : 6,
      "snapshot_deletion_failures" : 0
    },
    "daily-snapshots" : {
      "snapshots_taken" : 12,
      "snapshots_failed" : 0,
      "snapshots_deleted" : 12,
      "snapshot_deletion_failures" : 6
    }
  },
  "total_snapshots_taken" : 19,
  "total_snapshots_failed" : 0,
  "total_snapshots_deleted" : 18,
  "total_snapshot_deletion_failures" : 6
}
```

This does not yet include HLRC for this, as this commit is quite large on its own. That will be
added in a subsequent commit.

Relates to #43663

* Version qualify serialization

* Initialize counters outside constructor

* Use computeIfAbsent instead of being too verbose

* Move part of XContent generation into subclass

* Fix REST action for master merge

* Unused import

*  Record history of SLM retention actions (#45513)

This commit records the deletion of snapshots by the retention component
of SLM into the SLM history index for the purposes of reviewing operations
taken by SLM and alerting.

* Retry SLM retention after currently running snapshot completes (#45802)

* Retry SLM retention after currently running snapshot completes

This commit adds a ClusterStateObserver to wait until the currently
running snapshot is complete before proceeding with snapshot deletion.
SLM retention waits for the maximum allowed deletion time for the
snapshot to complete, however, the waiting time is not factored into
the limit on actual deletions.

Relates to #43663

* Increase timeout waiting for snapshot completion

* Apply patch

From 2374316f0d.patch

* Rename test variables

* [TEST] Be less strict for stats checking

* Skip SLM retention if ILM is STOPPING or STOPPED (#45869)

This adds a check to ensure we take no action during SLM retention if
ILM is currently stopped or in the process of stopping.

Relates to #43663

* Check all actions preventing snapshot delete during retention (#45992)

* Check all actions preventing snapshot delete during retention run

Previously we only checked to see if a snapshot was currently running,
but it turns out that more things can block snapshot deletion. This
changes the check to be a check for:

- a snapshot currently running
- a deletion already in progress
- a repo cleanup in progress
- a restore currently running

This was found by CI where a third party delete in a test caused SLM
retention deletion to throw an exception.

Relates to #43663

* Add unit test for okayToDeleteSnapshots

* Fix bug where SLM retention task would be scheduled on every node

* Enhance test logging

* Ignore if snapshot is already deleted

* Missing import

* Fix SnapshotRetentionServiceTests

* Expose SLM policy stats in get SLM policy API (#45989)

This also adds support for the SLM stats endpoint to the high level rest client.

Retrieving a policy now looks like:

```json
{
  "daily-snapshots" : {
    "version": 1,
    "modified_date": "2019-04-23T01:30:00.000Z",
    "modified_date_millis": 1556048137314,
    "policy" : {
      "schedule": "0 30 1 * * ?",
      "name": "<daily-snap-{now/d}>",
      "repository": "my_repository",
      "config": {
        "indices": ["data-*", "important"],
        "ignore_unavailable": false,
        "include_global_state": false
      },
      "retention": {}
    },
    "stats": {
      "snapshots_taken": 0,
      "snapshots_failed": 0,
      "snapshots_deleted": 0,
      "snapshot_deletion_failures": 0
    },
    "next_execution": "2019-04-24T01:30:00.000Z",
    "next_execution_millis": 1556048160000
  }
}
```

Relates to #43663

* Rewrite SnapshotLifecycleIT as as ESIntegTestCase (#46356)

* Rewrite SnapshotLifecycleIT as as ESIntegTestCase

This commit splits `SnapshotLifecycleIT` into two different tests.
`SnapshotLifecycleRestIT` which includes the tests that do not require
slow repositories, and `SLMSnapshotBlockingIntegTests` which is now an
integration test using `MockRepository` to simulate a snapshot being in
progress.

Relates to #43663
Resolves #46205

* Add error logging when exceptions are thrown
2019-09-09 09:55:34 -06:00
Martijn van Groningen f97cc7f355
Merge remote-tracking branch 'es/master' into enrich 2019-09-09 08:38:37 +02:00
Lisa Cawley 210b592f62
[DOCS] Synchs Watcher API titles with better HLRC titles (#46328) 2019-09-04 17:03:05 -07:00
Martijn van Groningen 63fe69fea4
Merge remote-tracking branch 'es/master' into enrich 2019-09-02 08:45:43 +02:00
Jilles van Gurp e40be722cc Add a few notes on Cancellable to the LLRC and HLRC docs. (#45912)
Add a section to both the low level and high level client documentation on asynchronous usage and `Cancellable` added for #44802 

Co-Authored-By: Lee Hinman <dakrone@users.noreply.github.com>
2019-08-28 10:59:33 +02:00
Martijn van Groningen c8436a7a36
Merge remote-tracking branch 'es/master' into enrich 2019-08-28 10:05:14 +02:00
Dimitris Athanasiou eab64250eb
[ML][HLRC] Add data frame analytics regression analysis (#46024) 2019-08-28 08:12:10 +03:00
Yogesh Gaikwad 5761b0a79c
Add `manage_own_api_key` cluster privilege (#45897)
The existing privilege model for API keys with privileges like
`manage_api_key`, `manage_security` etc. are too permissive and
we would want finer-grained control over the cluster privileges
for API keys. Previously APIs created would also need these
privileges to get its own information.

This commit adds support for `manage_own_api_key` cluster privilege
which only allows api key cluster actions on API keys owned by the
currently authenticated user. Also adds support for retrieval of
the API key self-information when authenticating via API key
without the need for the additional API key privileges.
To support this privilege, we are introducing additional
authentication context along with the request context such that
it can be used to authorize cluster actions based on the current
user authentication.

The API key get and invalidate APIs introduce an `owner` flag
that can be set to true if the API key request (Get or Invalidate)
is for the API keys owned by the currently authenticated user only.
In that case, `realm` and `username` cannot be set as they are
assumed to be the currently authenticated ones.

The changes cover HLRC changes, documentation for the API changes.

Closes #40031
2019-08-27 19:48:21 +10:00
Albert Zaharovits 715f7e9e01
PKI realm authentication delegation (#45906)
This commit introduces PKI realm delegation. This feature
supports the PKI authentication feature in Kibana.

In essence, this creates a new API endpoint which Kibana must
call to authenticate clients that use certificates in their TLS
connection to Kibana. The API call passes to Elasticsearch the client's
certificate chain. The response contains an access token to be further
used to authenticate as the client. The client's certificates are validated
by the PKI realms that have been explicitly configured to permit
certificates from the proxy (Kibana). The user calling the delegation
API must have the delegate_pki privilege.

Closes #34396
2019-08-26 18:53:10 +03:00
Martijn van Groningen a1e8194a57
Add HLRC support for delete policy api (#45833)
This PR also adds HLRC docs.

Relates to #32789
2019-08-26 09:54:25 +02:00
Martijn van Groningen b48784f5c1
Merge remote-tracking branch 'es/master' into enrich 2019-08-23 11:11:57 +02:00
markharwood d1e00e3cf5
Search - added HLRC support for PinnedQueryBuilder (#45779)
* Added HLRC support for PinnedQueryBuilder

Related  #44074
2019-08-22 16:32:42 +01:00
Przemysław Witek 31f6e78acd
Allow the user to specify 'query' in Evaluate Data Frame request (#45775) 2019-08-22 08:27:38 +02:00
Dimitris Athanasiou 8af319481e
[ML] Add description to DF analytics (#45774) 2019-08-21 19:58:09 +03:00
Martijn van Groningen a6917a1572
Merge remote-tracking branch 'es/master' into enrich 2019-08-21 14:17:16 +02:00
Przemysław Witek c6a25a818d
Add docs for HLRC for Estimate memory usage API (#45538) 2019-08-21 12:52:17 +02:00
Martijn van Groningen 5707bc7f5d
Merge remote-tracking branch 'es/master' into enrich 2019-08-16 09:42:36 +02:00
Jim Ferenczi 38f9e52c3e
Add mapper-extras and the RankFeatureQuery in the hlrc (#43713)
This change adds the support for the RankFeatureQuery in the HLRC by
providing an extra dependency on mapper-extras-client. It also removes
the dependency on lang-painless in mapper-extras which is not needed
anymore since the move of the vector field into a dedicated module.

Closes #43634
2019-08-14 09:52:49 +02:00
Martijn van Groningen 43b23aa505
Added HLRC support for enrich put policy API. (#45183)
This PR also adds HLRC docs.

Relates to #32789
2019-08-09 09:12:03 +02:00
David Roberts 65b502079a
[ML-DataFrame] Combine task_state and indexer_state in _stats (#45276)
This commit replaces task_state and indexer_state in the
data frame _stats output with a single top level state
that combines the two. It is defined as:

- failed if what's currently reported as task_state is failed
- stopped if there is no persistent task
- Otherwise what's currently reported as indexer_state

Closes #45201
2019-08-07 16:39:56 +01:00
Benjamin Trent 1da7c591c5
[ML][Data Frame] Add update transform api endpoint (#45154)
This adds the ability to `_update` stored data frame transforms. All mutable fields are applied when the next checkpoint starts. The exception being `description`. 

This PR contains all that is necessary for this addition:
* HLRC
* Docs
* Server side
2019-08-07 07:28:09 -05:00
Lisa Cawley 46912c8f3d
[DOCS] Reformats ML update APIs (#45253) 2019-08-06 11:05:01 -07:00