Commit Graph

8209 Commits

Author SHA1 Message Date
Martijn van Groningen 52afaf2060
Protect replicated data streams against local rollovers (#64710)
When a data stream is being auto followed then a rollover in a local cluster can break auto following,
if the local cluster performs a rollover then it creates a new write index and if then later the remote
cluster rolls over as well then that new write index can't be replicated, because it has the same name
as in the write index in the local cluster, which was created earlier.

If a data stream is managed by ccr, then the local cluster should not do a rollover for those data streams.
The data stream should be rolled over in the remote cluster and that change should replicate to the local
cluster. Performing a rollover in the local cluster is an operation that the data stream support in ccr should
perform.

To protect against rolling over a replicated data stream, this PR adds a replicate field to DataStream class.
The rollover api will fail with an error in case a data stream is being rolled over and the targeted data stream is
a replicated data stream. When the put follow api creates a data stream in the local cluster then the replicate flag
is set to true. There should be a way to turn a replicated data stream into a regular data stream when for example
during disaster recovery. The newly added api in this pr (promote data stream api) is doing that. After a replicated
data stream is promoted to a regular data stream then the local data stream can be rolled over, so that the new
write index is no longer a follower index. Also if the put follow api is attempting to update this data stream
(for example to attempt to resume auto following) then that with fail, because the data stream is no longer a
replicated data stream.

Today with time based indices behind an alias, the is_write_index property isn't replicated from remote cluster
to the local cluster, so when attempting to rollover the alias in the local cluster the rollover fails, because the
alias doesn't have a write index. The added replicated field in the DataStream class and added validation
achieve the same kind of protection, but in a more robust way.

A followup from #61993.
2020-12-08 08:34:24 +01:00
James Rodewig e3f6adf2d1
[DOCS] Fix wording for HTTP settings (#65964) 2020-12-07 12:18:55 -05:00
István Zoltán Szabó 55d2f86b5b
[DOCS] Adds Working with transforms at scale to docs (#65726) 2020-12-07 18:00:42 +01:00
David Turner aa4ab0bc26
Expand docs on disk-based shard allocation (#65668)
Today we document the settings used to control rebalancing and
disk-based shard allocation but there isn't really any discussion around
what these processes do so it's hard to know what, if any, adjustments
to make.

This commit adds some words to help folk understand this area better.
2020-12-07 14:51:26 +00:00
Hendrik Muhs 9b47889153
[Transform] use ISO dates in output instead of epoch millis (#65584)
Transform writes dates as epoch millis, this does not work for historic data in some cases or is
unsupported. Dates should be written as such. With this PR transform starts writing dates in ISO
format, but as existing transform might rely on the format it provides backwards compatibility for
old jobs as well as a setting to write dates as epoch millis.

fixes #63787
2020-12-07 15:34:28 +01:00
James Rodewig 3a2065b4ce
[DOCS] Make data stream names consistent (#65920) 2020-12-07 08:51:07 -05:00
James Rodewig 6a09df8520
[DOCS] EQL: Add diagrams for sequence matching (#65898) 2020-12-07 07:55:38 -05:00
Toast 966189fa6a
[DOCS] Fix typo (#65912) 2020-12-05 10:05:13 -05:00
bellengao e198bb233e
[DOCS] Correct the default value of `wait_for_completion` query param (#65800)
Co-authored-by: James Rodewig <40268737+jrodewig@users.noreply.github.com>
2020-12-04 15:52:35 -05:00
James Rodewig ef6fb59ec3
[DOCS] EQL: Document how sequence queries handle matches (#65794)
Co-authored-by: Ross Wolf <31489089+rw-access@users.noreply.github.com>
2020-12-04 09:34:38 -05:00
Yang Wang 201b25e92e
Fix xpack info and usage reports for operator privileges (#65867)
This is a follow-up PR for #65256 to fix the xpack info and usage reports for
operator privilegs. In summary, this PR ensures:

* _xpack does not report operator privileges because it is categorised under
security 
* _xpack/usage reports operator privileges status under the security
section 
* _license/feature_usage reports last used time of operator privileges.
It is up to the downstream to filter out this report if necessary.
2020-12-04 18:02:38 +11:00
Bogdan Pintea 638402c387
Abort sorting in case of local agg sort queue overflow (#65687)
In case the local agg sorter queue gets full and no limit has been provided,
the local sorter will now erroneously call the failure callback for every
single row in the original rowset that's left over the local queue limit
(instead for just the first one).  The failure response is dispatched in any
case, so this is relatively harmless.  The sorter continues iterating on the
original response fetching subsequent pages. In case of correct Elasticsearch
behaviour, this is also harmless, it'll just trigger a number of internal
exceptions. However, in case of a pagination defect in Elasticsearch (like
GH#65685, where the same search_after is returned), this will result in an
effective spin loop, potentially rendering eventually the node unresponsive.

This PR simply breaks both the inner loop iterating over the current unsorted
rowset, as well as the outer one, iterating over the left pages.

It also fixes an outdated documentation limitation.
2020-12-03 19:19:15 +01:00
David Roberts 49e492f313
[ML] Adding assignment_memory_basis to model_size_stats (#65561)
At present the Java code makes a decision on whether to
use current model memory or model memory limit to calculate
how much memory a job requires to be assigned.

The plan is to move this decision to the C++ code, which will
report it via a new field in the model size stats.  An
additional change will be that once we have made the switch
from using model memory limit to using current model memory
we will never switch back, as this causes large fluctuations
up and down in memory requirement which will be much more
noticeable when autoscaling is in use.

Although the only two options at present are model memory
limit and current model memory, the new enum includes a
third possibility, peak model memory.  To switch to this
now would be tricky, as there have been two bugs in the
implementation of peak model memory which render its value
unreliable in 7.x.  However, in 8.x it might make sense to
switch to using peak model memory instead of current model
memory and it's much easier from a BWC perspective if the
enum contains all the values from the start.

Relates #63163
2020-12-03 17:18:08 +00:00
James Rodewig 2044caa667
[DOCS] EQL: Document ? wildcard (#65698) 2020-12-03 12:14:38 -05:00
Mike Barretta 12c9ee4d80
Update inference-bucket-aggregation.asciidoc
tiny change to properly align the first code example and to add a missing word
2020-12-03 11:48:45 -05:00
James Rodewig e955f7752b
[DOCS] Fix typo in histogram agg docs (#65822) 2020-12-03 09:55:47 -05:00
Yang Wang def2f2777b
Phase 1 support for operator privileges (#65256)
In some Elastic Stack environments, there is a distinction between the operator
of the cluster infrastructure and the administrator of the cluster. This
distinction cannot be supported currently because the "administrator" often has
the superuser role which grants each and every privilege of the cluster.

This PR adds a new feature to protect a fixed set of APIs from the
"administrator" even when it is a highly privileged user such as superuser. It
enhances the Elasticsearch security model to have an additional layer of
restriction in addition to the RBAC.

Co-authored-by: Tim Vernum <tim@adjective.org>
2020-12-03 16:49:33 +11:00
David Turner ff5cb90cc9
Remove escape hatch permitting incompatible builds (#65753)
Today in `7.x` there is a deprecated system property that bypasses the
check that prevents nodes of incompatible builds from communicating.
This commit removes the system property in `master` so that the check is
always enforced.

Relates #65601, #65249
2020-12-02 19:30:55 +00:00
James Rodewig a1b6be045c
[DOCS] Fix docs integ tests for release builds (#65761) 2020-12-02 12:09:53 -05:00
Przemyslaw Gomulka 3b859f56a9
Compress audit logs (#64472)
audit logs should be compressed when rolling over due to size based
triggering policy breaching 1GB.
Files are not being deleted.

closes #63843
2020-12-02 17:36:31 +01:00
James Rodewig 0f406f1734
[DOCS] Add cluster get settings API example (#65754) 2020-12-02 10:37:01 -05:00
James Rodewig 8e3cc3007c
[DOCS] Remove inert component template file (#65749) 2020-12-02 09:46:02 -05:00
James Rodewig 72621873fd
[DOCS] Remove erroneous `flat_settings` query param (#65670) (#65745)
Co-authored-by: Thiago Souza <thiago@elastic.co>
2020-12-02 09:42:35 -05:00
Howard bcea87f3a3
[DOCS] Fix EQL syntax formatting (#65711) 2020-12-02 08:51:39 -05:00
Dan Hermann 90d25d6755
[DOCS] URI parts processor (#65695) 2020-12-02 07:22:40 -06:00
James Rodewig 6f11752ffe
[DOCS] Fix `_doc_count` field title (#65704) 2020-12-02 08:13:43 -05:00
István Zoltán Szabó 6f323ad0b1
[DOCS] Changes wording of pivot parameter in PUT transforms API docs. (#65731) 2020-12-02 14:07:17 +01:00
Christoph Büscher 3c3a43249f
Support unmapped fields in search 'fields' option (#65386)
Currently, the 'fields' option only supports fetching mapped fields. Since
'fields' is meant to be the central place to retrieve document content, it
should allow for loading unmapped values. This change adds implementation and
tests for this feature.

Closes #63690
2020-12-01 21:40:27 +01:00
Gil Raphaelli b4233f0cd4
[DOCS] Fix _doc_count example typo (#65686) 2020-12-01 13:53:33 -05:00
James Rodewig 1c3ddf8ff1
[DOCS] EQL: Flatten EQL syntax headings (#65693) 2020-12-01 12:56:12 -05:00
Christoph Büscher c327794ae8
Fix range query on date fields for number inputs (#63692)
Currently, if you write a date range query with numeric 'to' or 'from' bounds,
they can be interpreted as years if no format is provided. We use
"strict_date_optional_time||epoch_millis" in this case that can interpret inputs
like 1000 as the year 1000 for example. 
This PR change this to always interpret and parse numbers with the "epoch_millis"
parser if no other formatter was provided.

Closes #63680
2020-12-01 18:49:50 +01:00
James Rodewig ac1dbb7ffd
[DOCS] EQL: Remove outdated wildcard ref (#65684) 2020-12-01 11:30:17 -05:00
James Rodewig eac210436b
[DOCS] Label legacy rollup APIs (#65518) 2020-12-01 09:00:45 -05:00
James Rodewig 581e5c82b4
[DOCS] Update rollup glossary item (#65519)
Co-authored-by: Lisa Cawley <lcawley@elastic.co>
2020-12-01 08:57:07 -05:00
bellengao 80df5fe2f3
[DOCS] Correct restore snapshot API request example (#65525) 2020-11-30 13:55:37 -05:00
James Rodewig d3613ab1b5
[DOCS] Replace "time-series" with "time series" (#65620) 2020-11-30 12:39:24 -05:00
Henning Andersen 8fa1eea6f6
Searchable snapshot terminology (#65549)
We chose to use searchable snapshot index over snapshot-backed index, so
changed terminology towards this in a couple places.
2020-11-30 17:14:47 +01:00
David Turner 60442b8ec4
Clarify snapshot incrementality (#65587)
Today we describe snapshots as "incremental" but their incrementality is
rather different beast from e.g. incremental filesystem backups. With
traditional backups you take a large and relatively infrequent "full"
backup and then a sequence of smaller "incremental" ones, and this whole
sequence of backups is required for a restore so it must be kept around
until at least the next full backup. In contrast, Elasticsearch
snapshots are logically independent and each can be deleted without
affecting the integrity of the others.

This distinction frequently causes confusion amongst newer users, so
this commit clarifies what we mean by "incremental" in the docs.
2020-11-30 14:49:41 +00:00
James Rodewig f727f8eb21
[DOCS] Clarify `metrics` is array of strings (#65611) 2020-11-30 09:46:51 -05:00
James Rodewig 630cb18b79
[DOCS] Add `require_alias` query param to reindex API (#65608) 2020-11-30 09:30:40 -05:00
Henning Andersen 9d55cbd110
Cold tier time-range should not be specified (#65546)
Whether the cold tier can handle years depends a lot on the use case and
for instance our BWC guarantees. This would need to be part of a
specific sizing exercise, so in the spirit of not over-promising, the
description of the cold tier has been changed to not mention years.
2020-11-30 15:04:41 +01:00
Dimitris Athanasiou d1485eb3b1
[DOCS] Add missing "with" in remote reindex doc (#65532) 2020-11-30 08:32:18 -05:00
David Kyle be86dd5625
Mute rollup docs test (#65585)
For #65544
2020-11-30 10:43:13 +00:00
Yang Wang bdd99b250f
Deprecate cert gen without a CA and add self-signed option (#64037)
Generating a CA on the fly is an attempt at workflow optimisation that was
inherited from certgen. There are potential pitfalls with this approach. Overall
it is recommended to separate the step of CA creation and mandate a CA to be
specified when generating certificate.

This PR add a deprecation message if the cert command is used without specifying
a CA. A follow up PR will throw error for this usage in 8.0.

For use case where we explicitly trust a certificate without needing a CA, e.g.
SAML message signing, the PR adds a --self-signed option to the cert sub-command
to generate self-signed certificate.
2020-11-30 08:46:02 +11:00
Henning Andersen 302e5761fc
Clarify searchable snapshot cost trade-offs (#65384)
Clarify that searchable snapshots only result in cost savings for less
frequently accessed data and that the savings do not apply to the entire
cluster.
2020-11-26 13:40:08 +01:00
Lyudmila Fokina c758dc7f4a
Introduce an additional hasher (PBKDF2_STRETCH) (#65328)
* Introduce an additional hasher that is PBKDF2 but pads the input to > 14 chars before hashing to comply with FIPS Approve Only mode

* Introduce an additional hasher that is PBKDF2 but pads the input to > 14 chars before hashing to comply with FIPS Approve Only mode

* Addressing the PR feedback
adding doc changes

* Renaming the hash function + rephrasing the doc descriptions

* Removing leftover from the doc

* Return HexCharArray instead of Base64 encoding and avoid intermediate
String

Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
2020-11-26 13:29:19 +01:00
James Rodewig 167600c824 [DOCS] Remove rollup index ref 2020-11-25 17:29:13 -05:00
James Rodewig 427930d30a
[DOCS] Add rollup V2 security privileges (#65512) 2020-11-25 15:48:51 -05:00
James Rodewig b8c6ba2afd
[DOCS] Add ILM action redirects (#65479) (#65506) 2020-11-25 11:25:09 -05:00
James Rodewig a941c7147f
[DOCS] Add rollup V2 API docs (#65398)
Changes:

* Documents new `<index>/_rollup` API
* Updates rollup APIs overview page
* Enables rollups V2 in docs integ tests
2020-11-25 11:16:06 -05:00
Julie Tibshirani abe5e1707e
Remove note on marking copy_to fields as stored. (#65459)
During highlighting, we now load all values that were copied into the field
through copy_to. So there's no longer a reason to set 'store: true' to account
for fields not available in _source.
2020-11-25 08:01:36 -08:00
James Rodewig a18b87ddc1
[DOCS] Flatten EQL syntax headings (#65497) 2020-11-25 10:30:24 -05:00
Igor Motov a065b6d8da
Return an error when a rate aggregation cannot calculate bucket sizes (#65429)
In some cases when the rate aggregation is not a child of a date histogram
aggregation, it is not possible to determine the actual size of the date
histogram bucket. In this case the rate aggregation now throws an exception.

Closes #63703
2020-11-25 10:05:51 -05:00
James Rodewig 0805ff17a5
[DOCS] Fix redirects and anchors (#65472) (#65474) 2020-11-24 19:59:52 -05:00
James Rodewig 8e1081662c [DOCS] Fix xref 2020-11-24 19:32:39 -05:00
James Rodewig a6a2e417b5 [DOCS] Fix typo 2020-11-24 19:29:28 -05:00
Tal Levy a6755c3be8
Add mention of geo_shape support in geotile and geohash grid agg docs (#61129)
Previously, geo_shape support was only mentioned in a dedicated x-pack
section. This may be misleading, as the introductory paragraph only
mentions geo_point.

Co-authored-by: James Rodewig <40268737+jrodewig@users.noreply.github.com>
2020-11-24 13:57:42 -08:00
Andras Palinkas e242eb6a43
SQL: Add method args to PERCENTILE/PERCENTILE_RANK (#65026)
* Adds the capability to have functions with two optional arguments
* Adds two new optional arguments to `PERCENTILE()` and
  `PERCENTILE_RANK()` functions, namely the method and
  method_parameter which can be: 1) `tdigest` and a double `compression`
  parameter or 2) `hdr` and an integer representing the
  `number_of_digits` parameter.
* Integration tests
* Documentation updates

Closes #63567
2020-11-24 14:17:56 -05:00
James Rodewig b9ee0b3b48
[DOCS] EQL: Add lookup support to `:` operator (#65262) 2020-11-24 10:48:41 -05:00
bellengao 273b7bfe04
[DOCS] Correct the url in delete-component-template doc (#65349) 2020-11-24 09:48:17 -05:00
David Roberts fc72b39a17
[ML] Adjusting soft_limit description (#65383)
This PR adds detail to the explanation of the soft_limit
memory_status in ML job stats. A consequence that was not
mentioned before is that examples are not added to category
definitions.

Relates elastic/ml-cpp#1590
2020-11-24 09:35:07 +00:00
Tal Levy b514d9bf2e
Add geo_line aggregation (#41612)
A metric aggregation that aggregates a set of points as 
a GeoJSON LineString ordered by some sort parameter.

#### specifics

A `geo_line` aggregation request would specify a `geo_point` field, as well
as a `sort` field. `geo_point` represents the values used in the LineString, 
while the `sort` values will be used as the total ordering of the points.

the `sort` field would support any numeric field, including date.

#### sample usage

```
{
	"query": {
		"bool": {
			"must": [
				{ "term": { "person": "004" } },
				{ "term": { "trajectory": "20090131002206.plt" } }
			]
		}
	},
	"aggs": {
		"make_line": {
			"geo_line": {
				"point": {"field": "location"},
				"sort": { "field": "timestamp" },
                                "include_sort": true,
                                "sort_order": "desc",
                                "size": 15
			}
		}
	}
}
```

#### sample response

```
{
    "took": 21,
    "timed_out": false,
    "_shards": {...},
    "hits": {...},
    "aggregations": {
        "make_line": {
            "type": "LineString",
            "coordinates": [
                [
                    121.52926194481552,
                    38.92878997139633
                ],
                [
                    121.52922699227929,
                    38.92876998055726
                ],
             ]
        }
    }
}
```

#### visual response

<img width="540" alt="Screen Shot 2019-04-26 at 9 40 07 AM" src="https://user-images.githubusercontent.com/388837/56834977-cf278e00-6827-11e9-9c93-005ed48433cc.png">

#### limitations

Due to the cardinality of points, an initial max of 10k points 
will be used. This should support many use-cases.

One solution to overcome this limitation is to keep a PriorityQueue of
points, and simplifying the line once it hits this max. If simplifying
makes sense, it may be a nice option, in general. The ability to use a parameter
to specify how aggressive one wants to simplify. This parameter could be 
the number of points. Example algorithm one could use with a PriorityQueue:
https://bost.ocks.org/mike/simplify/. This would still require O(m) space, where m
is the number of points returned. And would also require heapifying triangles
sorted by their areas, which would be O(log(m)) operations. Since sorting is done, 
anyways, simplifying would still be a O(n log(m)) operation, where n is the total number 
of points to filter........... something to explore


closes #41649
2020-11-23 10:26:27 -08:00
David Turner 41e9ae51b0
Add snapshots to important config list (#65338)
The _Important Elasticsearch configuration_ docs lists a number of items
that you should consider before moving to production. Today this list
does not include configuring snapshots, even though they're very
important to have in production. This commit addresses that omission,
removes some repetition from the introductory paragraphs, and notes that
this config is handled for you on Cloud.

Co-authored-by: James Rodewig <40268737+jrodewig@users.noreply.github.com>
2020-11-23 14:26:07 +00:00
István Zoltán Szabó a85fb5534a
[DOCS] Fixes typo in Aggregating data for faster performance. (#65354) 2020-11-23 12:44:59 +01:00
Wylie Conlon 10ee0f2878
Clarify field data cache behavior in docs (#64375)
* Clarify that field data cache includes global ordinals
* Describe that the cache should be cleared once the limit is reached
* Clarify that the `_id` field does not supported aggregations anymore
* Fold the `fielddata` mapping parameter page into the `text field docs
* Improve cross-linking
2020-11-20 13:53:23 -08:00
István Zoltán Szabó f1e54a63a1
[DOCS] Adds UI related limitation to configuring aggs docs (#65184)
Co-authored-by: Lisa Cawley <lcawley@elastic.co>
2020-11-20 19:03:18 +01:00
bellengao 2d9b83e32c
[DOCS] Fix formatting issue in search explain docs (#65303) 2020-11-20 09:01:01 -05:00
István Zoltán Szabó 1e045da339
[DOCS] Makes the screenshot larger on the custom URLs page. (#65269) 2020-11-20 09:29:39 +01:00
James Rodewig f83ed81cb3
[DOCS] Document `xpack.http.proxy.scheme` setting (#65264) 2020-11-19 15:31:24 -05:00
James Rodewig a8ddf76526
[DOCS] ILM: Add alloc ex with multiple node attributes (#65266) 2020-11-19 12:17:10 -05:00
James Rodewig a51107a845
[DOCS] Document get data stream API's _meta prop (#65221) 2020-11-19 09:51:03 -05:00
James Rodewig ce644909dc
[DOCS] EQL: Add wildcard support to `:` operator (#65237) 2020-11-19 08:26:13 -05:00
DeDe Morton fb53dc3167
Fix links to Fleet overview (#65174) 2020-11-18 10:26:04 -08:00
James Rodewig cc75bda411
[DOCS] Remote reindex is not fwd compatible (#65207) 2020-11-18 12:29:05 -05:00
Armin Braun cb657c2629
Fix Allocation `include` Filter Docs (#65202)
Fix documentation to match actual behavior of `include` type filters.

Closes #65113
2020-11-18 17:59:59 +01:00
James Rodewig 661409a5a0
[DOCS] Clarify logger-package relationship (#65169)
Updates the logging level docs to better clarify the relationship
between loggers and their Java packages.
2020-11-18 10:28:06 -05:00
David Roberts e4ce39845b
[ML] Add total ML memory to ML info (#65195)
This change adds an extra piece of information,
limits.total_ml_memory, to the ML info response.
This returns the total amount of memory that ML
is permitted to use for native processes across
all ML nodes in the cluster.  Some of this may
already be in use; the value returned is total,
not available ML memory.
2020-11-18 15:06:21 +00:00
bellengao 585fd73bee
[DOCS] Format the ilm-migrate doc (#65182) 2020-11-18 09:02:07 -05:00
Rafi Estrada 5e4db2291f
[DOCS] Add note about NFS repositories (#65149)
Co-authored-by: James Rodewig <40268737+jrodewig@users.noreply.github.com>
2020-11-17 14:21:18 -05:00
James Rodewig 36d308bc23
[DOCS] EQL: Update docs for null tiebreakers (#65078) 2020-11-17 09:31:49 -05:00
James Rodewig 254807956f
[DOCS] EQL: Document result_position param (#65075) 2020-11-17 09:07:51 -05:00
Lisa Cawley 9fef6e7b7e
[DOCS] Adds new snapshot upgrade API (#65095) 2020-11-16 09:48:07 -08:00
Adam Locke b0a98f9515
[DOCS] Adding authorization_realms setting for OIDC (#64877)
* Adding authorization_realms setting for OIDC.

* Fix typo

* Splitting note into separate notes.
2020-11-16 11:22:46 -05:00
István Zoltán Szabó 95a0ed4304
[DOCS] Adds recommendation about when to use chunking_config in manual mode. (#65060) 2020-11-16 16:12:07 +01:00
Dan Hermann 923b2b90c5
Remove the deprecated local parameter for _cat/indices (#64868) 2020-11-16 07:53:16 -06:00
Dan Hermann 0a2891e1e2
[DOCS] Data stream migration API (#65017) 2020-11-16 07:50:51 -06:00
James Rodewig 1e13b11213
[DOCS] Document reloadable Watcher settings (#64998) 2020-11-16 08:34:45 -05:00
James Rodewig f95a52f280
[DOCS] Clarify diff between shards per node settings (#64875)
Clarifies differences between the
`cluster.routing.allocation.total_shards_per_node` and
`cluster.max_shards_per_node` cluster settings.

Closes #51839

Co-authored-by: Gordon Brown <arcsech@gmail.com>
2020-11-16 08:33:04 -05:00
Dan Hermann f63a3b5cdc
Remove the deprecated local parameter for _cat/shards (#64867) 2020-11-13 07:34:15 -06:00
Lee Hinman bf63edde8d
Mark component and composable index template APIs as stable (#65013)
These were previously marked as experimental, but as we have not had any changes made or needed, we
are marking these as stable.
2020-11-12 14:16:57 -07:00
Adam Locke 9fdcd79927
Explicitly defining types for sources parameter (#65006) 2020-11-12 16:09:04 -05:00
James Rodewig 68cf6b64b7 [DOCS] Fix redirect syntax 2020-11-12 12:13:28 -05:00
Dan Hermann c829f8edd1
Remove deprecated _upgrade API (#64732) 2020-11-12 11:09:56 -06:00
Benjamin Trent 33de89d94c
[ML] add new snapshot upgrader API for upgrading older snapshots (#64665)
This new API provides a way for users to upgrade their own anomaly job
model snapshots.

To upgrade a snapshot the following is done:
- Open a native process given the job id and the desired snapshot id
- load the snapshot to the process
- write the snapshot again from the native task (now updated via the
  native process)

relates #64154
2020-11-12 10:45:56 -05:00
István Zoltán Szabó db15c4d6b9
[DOCS] Adds scroll_size maximum value to datafeeds API docs (#64986) 2020-11-12 15:53:53 +01:00
Martijn van Groningen 6f9f3e5054
Improve error message in case of invalid dynamic templates (#60870)
Include the attempted 'match_mapping_type' into the message,
so that it is clearer that multiple validation attempts have occurred.

Dynamic template validation was recently added via #51233 and
there was some confusion over the deprecation message itself.
(in 7.x only deprecation warning will be omitted and from 8.0
 an error will be returned)
2020-11-12 15:03:08 +01:00
James Rodewig 1f8d1e9ee1
[DOCS] Fix indentation (#64941) (#64962)
Co-authored-by: kemalizing <36174398+kemalizing@users.noreply.github.com>
2020-11-12 08:47:20 -05:00
Andrei Dan 7d45355604
ILM: Add support for the searchable_snapshot action in the hot phase (#64883)
This adds support for the searchable_snapshot ILM action in the hot phase.

We define a series of actions that cannot be executed after the index has been
mounted as a searchable snapshot. Namely: freeze, forcemerge, shrink,
and searchable_snapshot (also available in the cold phase).

If by virtue of snapshot/restoring a managed index or updating an ILM policy while it
is executing for an index, these actions could get to be executed on an index that was
mounted as searchable snapshot in the hot phase. If this happens the actions will
skip entirely. ILM will not move into the ERROR step.
2020-11-12 10:44:04 +00:00
debadair 92efa3a0f3
[DOCS] Make system & hidden index info more prominent. (#64839)
* [DOCS] Make system & hidden index info more prominent.

* Update docs/reference/api-conventions.asciidoc
2020-11-11 10:35:02 -08:00
István Zoltán Szabó 9ed907bc75
[DOCS] Fixes example aggregation syntax in datafeed aggregations. (#64936) 2020-11-11 16:33:36 +01:00
James Rodewig 44456ed777
[DOCS] Add heading and anchor to reindex with ingest pipeline section (#64835) (#64894)
Co-authored-by: James Rodewig <40268737+jrodewig@users.noreply.github.com>

Co-authored-by: Mark Walkom <markw@elastic.co>
2020-11-10 17:24:01 -05:00
debadair fd11233ed7
[DOCS] Add system & hidden indices definitions. (#64834)
* [DOCS] Add system & hidden indices definitions.

* Update docs/reference/glossary.asciidoc

* Update docs/reference/glossary.asciidoc

Co-authored-by: Lisa Cawley <lcawley@elastic.co>

* Update docs/reference/glossary.asciidoc

* Update docs/reference/glossary.asciidoc

Co-authored-by: Lisa Cawley <lcawley@elastic.co>
2020-11-10 13:03:14 -08:00
debadair 7bf35ffe86
[DOCS] Add data tier defs (#64838)
* [DOCS] Add data tier defs

* Add hot tier def
2020-11-10 10:14:26 -08:00
Christos Soulios 66b5e4ec89
Implement aggregations on aggregate metric fields (#56745)
In the process of developing a new implementation for the Elasticsearch Rollups functionality we came up with the concept of the aggregate metric field type.

The aggregate_metric_double field type can store the results of aggregations (currently min, max, sum, value_count and avg are supported - more to come).

This field allows us to run (min, max, sum, value_count, avg) aggregations on the container field and the field will return the correct metric depending on the aggregation that is computed.
2020-11-10 19:31:59 +02:00
James Rodewig cb26c542ce
[DOCS] Document get pipeline API as multi-target (#64816) 2020-11-09 16:11:13 -05:00
Dan Hermann fae9b06cd5
Adjust deprecation version after backport (#64794)
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
2020-11-09 13:43:47 -06:00
Dan Hermann 82242f7c3f
Adjust deprecation version after backport (#64789) 2020-11-09 13:43:24 -06:00
Leaf-Lin 2bf3e36144 remove node.ingest setting in the documentation (#64456)
I'm not sure if this setting was left here deliberately? or by accident?
With all other node role definition has changed syntax from `node.xxx` to `node.roles: [ ]`, the ingest one is the only one left behind.
2020-11-09 12:21:43 -07:00
James Rodewig b31a8ff244
[DOCS] Fix put repository API docs (#64811) 2020-11-09 14:20:08 -05:00
James Rodewig 7ed7a9584e
[DOCS] Fix typo (#64675) (#64799)
Co-authored-by: Ashish Jayan <58534490+chasexd@users.noreply.github.com>
2020-11-09 11:12:46 -05:00
James Rodewig b31c235f58
[master] [DOCS] Remove unneeded period (#64687) (#64790)
Co-authored-by: Johannes Mahne <johannes.mahne@elastic.co>
2020-11-09 11:07:36 -05:00
James Rodewig 357ab0073e
[DOCS] Improve docs for Windows DOS/UNC paths in `path.*` settings (#64668) 2020-11-09 10:36:48 -05:00
bellengao e712243a24
[DOCS] Format the data tier allocation doc (#64722) 2020-11-09 10:19:43 -05:00
Henning Andersen be9725245d
Autoscaling delete policy by simple pattern (#64739)
Added the capability to delete autoscaling policies by pattern, allowing
to for instance do:
```
DELETE _autoscaling/policy/*
```
to delete all autoscaling policies. If a wildcard is involved, no
matches are required.
2020-11-09 15:44:33 +01:00
Bogdan Pintea 6a92b3ec7c
SQL: Remove constant_keyword from SQL docs (#64709)
* Remove constant_keyword from SQL docs

`constant_keyword` removed as distinct type from SQL in #60524.

Co-authored-by: James Rodewig <40268737+jrodewig@users.noreply.github.com>
2020-11-06 20:07:30 +01:00
Lisa Cawley 919c79b745
[DOCS] Add custom feature processor example (#64681) 2020-11-06 09:24:01 -08:00
James Rodewig fb1936bed1
[DOCS] EQL: Fix tiebreaker field docs (#64671)
Corrects the EQL docs to remove `event.sequence` as the default `tiebreaker_field` value.
2020-11-06 09:05:18 -05:00
James Rodewig 7621b4c48c
[DOCS] Document off-heap swaps under mlocks (#64667) 2020-11-06 07:38:13 -05:00
James Rodewig d845aa05cd
[DOCS] Clarify args for geoshape `orientation` (#64649) 2020-11-06 07:37:16 -05:00
Alex Chan 320cab9bb9 [Docs] Note that wildcard field supports `null_value` 2020-11-05 19:07:59 +01:00
James Rodewig c7a11b5719 [DOCS] Fix `remove_index` callout 2020-11-05 09:09:16 -05:00
Hendrik Muhs 208b69ccaa
[Transform] add support for median absolute deviation (#64634)
add median_absolute_deviation to the list of supported aggs in transform
2020-11-05 14:38:15 +01:00
Rudolf Meijering 33a38d461f
[DOCS] POST /_aliases remove_index action only works on concrete indices (#64616) 2020-11-05 08:14:19 -05:00
Julie Tibshirani 79c267f4fc
Document breaking change around _id field data. (#64610) 2020-11-04 13:45:07 -08:00
Mark Tozzi f666ccb3bc
Add supports for upper and lower values on boxplot based on the IQR value (#63617) 2020-11-04 14:39:05 -05:00
Christos Soulios 666f4acab2
Fix typo in fields doc (#64600) 2020-11-04 19:51:14 +02:00
James Rodewig 947de836e1
[DOCS] Add tab widget for default log locations (#64510) 2020-11-04 08:19:02 -05:00
James Rodewig b2b676d7d6 [DOCS] Remove italics formatting 2020-11-03 15:49:52 -05:00
Mayya Sharipova 0ffbcd3b3c
Disable using unsigned_long in scripts (#64523)
Relates to #64361
2020-11-03 14:20:46 -05:00
James Rodewig 3cca4b54f0
[DOCS] Clarify index size in ILM overview (#64544) 2020-11-03 13:08:09 -05:00
Armin Braun 6924236118
Limit the Number of Snapshots in a BlobStoreRepository (#64461)
Adds a limit to the maximum number of snapshots that are allowed
to be added to a snapshot repository as a safety measure of last resort
against repositories that grow to an unmanagable size due to e.g. incorrect SLM
settings.

Co-authored-by: David Turner <david.turner@elastic.co>
2020-11-03 18:02:11 +01:00
Mayya Sharipova 074f7d2e8a
Async search status (#62947)
Introduce async search status API

GET /_async_search/status/<id>

The API is restricted to the monitoring_user role.

For a running async search, the response is:

```js
{
  "id" : <id>,
  "is_running" : true,
  "is_partial" : true,
  "start_time_in_millis" : 1583945890986,
  "expiration_time_in_millis" : 1584377890986,
  "_shards" : {
      "total" : 562,
      "successful" : 188,
      "skipped" : 0,
      "failed" : 0
  }
}
```

For a completed async search, an additional
`completion_status` fields is added.

```js
{
  "id" : <id>,
  "is_running" : false,
  "is_partial" : false,
  "start_time_in_millis" : 1583945890986,
  "expiration_time_in_millis" : 1584377890986,
  "_shards" : {
      "total" : 562,
      "successful" : 562,
      "skipped" : 0,
      "failed" : 0
  },
 "completion_status" : 200
}
```

Closes #57537
2020-11-03 11:35:28 -05:00
Christos Soulios 4dc833fa44
Add doc_count field mapper (#64503)
Bucket aggregations compute bucket doc_count values by incrementing the doc_count by 1 for every document collected in the bucket.

When using summary fields (such as aggregate_metric_double) one field may represent more than one document. To provide this functionality we have implemented a new field mapper (named doc_count field mapper). This field is a positive integer representing the number of documents aggregated in a single summary field.

Bucket aggregations will check if a field of type doc_count exists in a document and will take this value into consideration when computing doc counts.
2020-11-03 17:47:17 +02:00
Martijn van Groningen c4c3c8b422
Add data stream support to CCR (#61993)
This commit adds support data stream support to CCR's auto following by making the following changes:
* When the auto follow coordinator iterates over the candidate indices to follow,
  the auto follow coordinator also checks whether the index is part of a data stream and
  if the name of data stream also matches with the auto follow pattern then the index
  will be auto followed.
* When following an index, the put follow api also checks whether that index is part
  of a data stream and if so then also replicates the data stream definition to the
  local cluster.
* In order for the follow index api to determine whether an index is part of a data
  stream, the cluster state api was modified to also fetch the data stream definition
  of the cluster state if only the state is queried for specific indices.

When a data stream is auto followed, only new backing indices are auto followed.
This is in line with how time based indices patterns are replicated today. This
means that the data stream isn't copied 1 to 1 into the local cluster. The local
cluster's data stream definition contains the same name, timestamp field and
generation, but the list of backing indices may be different (depending on when
a data stream was auto followed).

Closes #56259
2020-11-03 14:01:14 +01:00
James Rodewig 99dac25a20
[DOCS] Add redirect for heap size (#64507) 2020-11-02 16:44:01 -05:00
bellengao b17ce85f13
Add copy_from parameter for set ingest processor (#63540) 2020-11-02 10:40:05 -06:00
Armin Braun 1990e24e8e
Document Recommended Maximum Repository Size in SLM Docs (#64485)
Just adding a short note on reasonable sizing limits.
2020-11-02 17:20:20 +01:00
James Rodewig 0ec79adfb6
[DOCS] Note heap size must be set to same min and max (#64090) (#64474)
Co-authored-by: C.J. Jameson <cjcjameson@users.noreply.github.com>
2020-11-02 09:21:58 -05:00
Andrei Dan d061c11899
[DOCS]: ILM rollover max_age condition disregards origination date (#64404) 2020-11-02 09:30:48 +00:00
Henning Andersen 54911ace97
Autoscaling policy roles specification (#64222)
Add a roles specification to autoscaling policies. This is used to map
the policy to a set of nodes governed by the policy. The list of roles
is mandatory when adding a policy, optional on updates.

This commit also removes the outer level "policy" element from autoscaling
policy PUT and GET requests.
2020-11-02 07:30:31 +01:00
James Rodewig 8bc922512c
[DOCS] Redirect moving avg aggregation (#64435) 2020-10-30 14:12:09 -04:00
James Rodewig 2e9f95aa73
[DOCS] Change agg titles to sentence case (#64425) 2020-10-30 13:25:21 -04:00
Joe Gallo fea3a88017
Allow readonly in the hot phase for ILM policies (#64381) 2020-10-30 09:53:33 -04:00
Rory Hunter a32a0986c3
Make ES files inside Docker container world readable (#64274)
Running the Elasticsearch Docker image with a different GID is
possible but trappy, since at present all the ES files are only
readable by the user and group. This PR documents a Docker CLI flag
that fixes this situation, by ensuring the container user is added
to the default group (which is `root`, GID 0).

I also added a test for this case, and refactored the Docker tests
to use a builder pattern for constructing the `docker run` command.
The existing code was becoming unwieldy and hard to change.
2020-10-30 13:26:44 +00:00
James Rodewig 7492cc97e5
[DOCS] Move Kibana index mgmt docs to ES (#64380) 2020-10-30 09:14:52 -04:00
James Rodewig 37b6adaf91
[DOCS] Rewrite aggs overview (#64318)
- Replaces more abstract docs about object structure and values source with task-based examples.
- Relocates several sections from the current `misc.asciidoc` file.
- Alphabetically sorts agg categories in the nav.
- Removes the matrix agg family. Moves the stats matrix agg under the metric agg family

Co-authored-by: debadair <debadair@elastic.co>
2020-10-30 08:39:38 -04:00
Przemko Robakowski 0e81fc641a
Hidden data streams (#63987)
* Hidden data streams

* whitespace reverted

* stricter ds name

* Revert "stricter ds name"

This reverts commit 100dba5f3c.

* String.format removed

* fix test

* fix GetDataStream action

* fix test

* fix test

* rest test

* rest test

* spotless

* tests

* Delete a.json

* added expand_wildcards for GetDataStream and DeleteDataStream

* unused imports

* add hidden setting to data stream template

* fix expand_wildcards

* spotless

* fix compilation

* unused import

* yaml test

* fix test

* fix cleanup

* review

* compilation fix

* fix javadoc

* fix javadoc
2020-10-30 11:41:58 +01:00
István Zoltán Szabó f6b0216958
[DOCS] Fixes 'the the' typo in transforms docs. (#64393) 2020-10-30 10:52:35 +01:00
Joe Gallo 9986cb80ab
Allow shrink in the hot phase for ILM policies (#64008) 2020-10-29 14:21:50 -04:00
Jason Tedor 0d4494f121
Clarify ingest-geoip database_file docs (#64340)
The docs for the geoip processor database_file option appear to indicate
that all geoip databases are in the config directory. This is leftover
legacy from when this was the case when ingest-geoip was a plugin, but
it is no longer true as the built-in databases now ship inside the
ingest-geoip module that is bundled by default. This commit clarifies
those docs.

Co-authored-by: Jakob Reiter <jakommo@users.noreply.github.com>
2020-10-29 13:27:17 -04:00
István Zoltán Szabó 46df9dd75e
[DOCS] Adds note box about mappings to Transforms Painless examples (#64285) 2020-10-29 17:02:37 +01:00
István Zoltán Szabó 7c4c098f76
[DOCS] Fixes typos in Transform Painless examples. (#64352) 2020-10-29 16:06:14 +01:00
James Rodewig 4f82e667ae
[DOCS] Fix "the the" typos (#64344) 2020-10-29 10:11:58 -04:00
James Rodewig fa7c63e6c4
[DOCS] Fix whitespace in pattern replace token filter docs (#64345) 2020-10-29 10:07:10 -04:00
James Rodewig 1ea83359bb
[DOCS] Fix case for 'Boolean' (#64299) 2020-10-29 09:04:43 -04:00
Boice Huang 41fbc52743
Deprecate the 'local' parameter of /_cat/indices (#62198) 2020-10-29 06:59:05 -05:00
Boice Huang 7d65278642
Deprecate the 'local' parameter of /_cat/shards (#62197) 2020-10-29 06:58:32 -05:00
Ignacio Vera b72be253a9
Enable geo_distance and geo_bounding_box queries on geo_shape field type (#64224) 2020-10-29 07:30:53 +01:00
debadair 43a503f3dc
[DOCS] Add top-level Data management section. (#64185)
* [DOCS] Add top-level Data management section.

* Edits

* Edits

* Fixed xrefs

* Apply suggestions from code review

Co-authored-by: Andrei Dan <andrei.dan@elastic.co>
Co-authored-by: Lee Hinman <dakrone@users.noreply.github.com>

* Update docs/reference/datatiers.asciidoc

* Update docs/reference/datatiers.asciidoc

Co-authored-by: Andrei Dan <andrei.dan@elastic.co>
Co-authored-by: Lee Hinman <dakrone@users.noreply.github.com>
2020-10-28 15:38:22 -07:00
Nik Everett 3af540b50d
Remove aggregation's postCollect phase (#64016)
After #63811 it became clear to me that `postCollect` is kind of
dangerous and not all that useful. So this removes it.

The trouble with `postCollect` is that it all happened right after we
finished calling `collect` on the `LeafBucketCollectors` but before we
built the aggregation results. But in #63811 we found out that we can't
call `postCollect` on the children of `parent` or `child` aggregators
until we know which *which* aggregation results we're building.

So this removes `postCollect` and moves all of the things we did at
post-collect phase into `buildAggregations` or into hooks called in
those methods.
2020-10-28 17:33:27 -04:00
Andrew Kroh 24cae6d7f8
[DOCS] Sort field data types in docs (#64288)
Co-authored-by: James Rodewig <40268737+jrodewig@users.noreply.github.com>
2020-10-28 12:13:01 -04:00
Rudolf Meijering 29d2a9e3c3
[DOCS] Fix `refresh` def in `update_by_query` docs (#64277)
Co-authored-by: James Rodewig <40268737+jrodewig@users.noreply.github.com>
2020-10-28 11:25:20 -04:00
James Rodewig 1c0380dc21
[DOCS] EQL: Fix operator docs (#64286) 2020-10-28 10:27:17 -04:00
Jason Tedor d6c6563837
Clarify preferred method for setting heap size (#64256)
This commit clarifies that the preferred method for setting the heap
size is via jvm.options.d and that using the ES_JAVA_OPTS environment
variable is discouraged for production deployments.
2020-10-28 08:49:31 -04:00
Jason Tedor 117d79b5e9
Adjust defaults for tiered data roles (#64015)
This commit adjusts the defaults for the tiered data roles so that they
are enabled by default, or if the node has the legacy data role. This
ensures that the default experience is that the tiered data roles are
enabled.

To fully specifiy the behavior for the tiered data roles then:
 - starting a new node with the defaults: enabled
 - starting a new node with node.roles configured: enabled if and only
   if the tiered data roles are explicitly configured, independently
   of the node having the data role
 - starting a new node with node.data enabled: enabled unless the
   tiered data roles are explicitly disabled
 - starting a new node with node.data disabled: disabled unless the
   tiered data roles are explicitly enabled
2020-10-27 12:47:14 -04:00
Adam Locke 65ba0d4d32
Fixing typo in file name. (#64226) 2020-10-27 10:48:31 -04:00
Elasticsearch addict 32c7e08c6d
[DOCS] Fix pattern replace token filter intro (#64189)
Removes an incorrect statement about anchoring regex patterns on tokens.
2020-10-27 09:33:03 -04:00
bellengao 719d4086aa
[DOCS] Fix the data stream stats api url in the doc (#64200) 2020-10-27 07:49:49 -05:00
István Zoltán Szabó 6093518f4a
[DOCS] Changes experimental flag to beta in DFA related docs (#63992) 2020-10-26 17:02:46 +01:00
Rory Hunter dc855add49
Add a template parameter to override auto_create_index value (#61858)
Closes #20640.

This PR introduces a new parameter to v2 templates, `allow_auto_create`,
which allows templates to override the cluster setting `auto_create_index`.
Notes:

   * `AutoCreateIndex` now looks for a matching v2 template, and if its
     `allow_auto_create` setting is true, it overrides the usual logic.
   * `TransportBulkAction` previously used `AutoCreateIndex` to check
     whether missing indices should be created. We now rely on
     `AutoCreateAction`, which was already differentiating between creating
     indices and creating data streams.  I've updated `AutoCreateAction` to
     use `AutoCreateIndex`. Data streams are also influenced by
     `allow_auto_create`, in that their default auto-create behaviour can
     be disabled with this setting.
   * Most of the Java file changes are due to introducing an extra
     constructor parameter to `ComposableIndexTemplate`.
   * I've added the new setting to various x-pack templates
   * I added a YAML test to check that watches can be created even when
     `auto_create_index` is `false`.
2020-10-26 12:35:36 +00:00
James Rodewig 5953a90505 [DOCS] Remove unneeded words in EQL docs 2020-10-24 20:27:34 -04:00
James Rodewig 97bdb9ba75 [DOCS] Minor data stream docs fixes 2020-10-24 19:49:30 -04:00
James Rodewig 6a16bfd9b3
[DOCS] Remove unneeded link 2020-10-24 17:08:30 -04:00
James Rodewig ea3107827f
[DOCS] Tighten data streams copy (#64085) 2020-10-24 14:24:01 -04:00
James Rodewig 4c22ca3eed
[DOCS] Tighten async EQL copy (#64106) 2020-10-24 14:14:30 -04:00
James Rodewig f6bce6194f
[DOCS] Tighten EQL copy (#64081) 2020-10-24 10:49:05 -04:00
debadair b95d9c4aa4
[DOCS] Add searchable snapshots topic. (#63040)
* [DOCS] Add searchable snapshots topic.

* [DOCS] Add definitions & remove fully-remote storage.

* [DOCS] Fixed duplicate anchor.

* Expand conceptual docs for searchable snapshots

* Rewordings

* Glossary tidy-up

* Beta

* Reword

* More performance idea to a TIP

* use -> manage

* red -> not green

* Missing space?

* Update docs/reference/glossary.asciidoc

* Fix beta label

* Use more attributes, fix link titles

* Apply suggestions from code review

Co-authored-by: debadair <debadair@elastic.co>

* Reformat

* Minor rewordings

* More minor rewordings

* Address Henning's comments

Co-authored-by: David Turner <david.turner@elastic.co>
Co-authored-by: James Rodewig <40268737+jrodewig@users.noreply.github.com>
2020-10-22 14:34:32 -07:00
James Rodewig 159d2a33ee
[DOCS] Fix `ignore_unavailable` param in get index and get alias APIs (#64075) 2020-10-22 15:10:37 -04:00
Alan Woodward bfaf3040c6
Convert geo field mappers to Parametrized form (#63836)
Relates to #62988
2020-10-22 09:11:41 +01:00
Benjamin Trent 165e063b50
[ML] add new setting xpack.ml.use_auto_machine_memory_percent for auto calculating native memory percentage allowed for machine learning jobs (#63887)
When running ML, sometimes it is best to automatically adjust the
memory allotted for machine learning based on the nodesize
and how much space is given to the JVM

This commit adds a new static setting xpack.ml.use_auto_machine_memory_percent for
allowing this dynamic calculation. The old setting remains as a backup
just in case the limit cannot be automatically determined due to
lack of information.

Closes #63795
2020-10-21 12:50:55 -04:00
Lisa Cawley a00c7a2b6c
[DOCS] Add tips for num_top_classes classification parameter (#63781) 2020-10-21 09:27:13 -07:00
Hugo Chargois ff736f078b
Allow mixing set-based and regexp-based include and exclude (#63325)
* Allow mixing set-based and regexp-based include and exclude

* Coding style

* Disallow having both set and regexp include (resp. exclude)

* Test correctness of every combination of include/exclude
2020-10-21 10:26:42 -04:00
István Zoltán Szabó 9defe10616
[DOCS] Expands DFA evaluation API docs with the default set of metrics (#63971) 2020-10-21 14:30:33 +02:00
Henning Andersen d573fdefd3
Autoscaling capacity API and structure (#63905)
Renamed decision API to capacity. Responses now prefer objects/maps over
arrays. Removed mention of tier, using policies as the outer map and
total for the policy-wide total capacity.
2020-10-21 11:57:11 +02:00
markharwood 81b334913a
Search - make term/prefix/wildcard/regex query parsing more lenient (#63926)
* Remove errors when case_insensitive flag set to false

Closes #63893
2020-10-21 10:51:22 +01:00
Benjamin Trent ebcac2d864
[Transform] add new exclude_generated flag to GET transform (#63093)
This adds a new flag `exclude_generated` for GET transform API.

This flag is useful for when a transform needs to be cloned within a cluster or exported/imported between clusters.

It removes certain fields that are not able to be set via the PUT api (e.g. version, create_time).

relates https://github.com/elastic/elasticsearch/issues/63055
2020-10-20 11:38:40 -04:00
Benjamin Trent c1de07fa83
[ML] adding new flag exclude_generated that removes generated fields in GET config APIs (#63899)
When exporting and cloning ml configurations in a cluster it can be
frustrating to remove all the fields that were generated by
the plugin. Especially as the number of these fields change
from version to version.

This flag, exclude_generated, allows the GET config APIs to return
configurations with these generated fields removed.

APIs supporting this flag: 
- GET _ml/anomaly_detection/<job_id>
- GET _ml/datafeeds/<datafeed_id>
- GET _ml/data_frame/analytics/<analytics_id>

The following fields are not returned in the objects:

- any field that is not user settable (e.g. version, create_time)
- any field that is a calculated default value (e.g. datafeed chunking_config)
- any field that is automatically set via another Elastic stack process (e.g. anomaly job custom_settings.created_by)

relates to #63055
2020-10-20 11:28:29 -04:00
Adam Locke 954d6eeeba
Move clone snapshot API page. (#63902) 2020-10-20 08:45:38 -04:00
Dimitris Athanasiou 03ed7de6c1
[ML] Rename evaluation metric result fields to value (#63809)
Renames data frame analytics _evaluate API results as follows:

  - per class accuracy renamed from `accuracy` to `value`
  - per class precision renamed from `precision` to `value`
  - per class recall renamed from `recall` to `value`
  - auc_roc `score` renamed to `value` for both outlier detection and classification
2020-10-20 10:30:50 +03:00
James Rodewig 3deebc2804 [DOCS] Fix typo 2020-10-19 14:44:12 -04:00
James Rodewig 71aaa4ae0a
[DOCS] EQL: Update `allow_no_indices` default (#63748)
Co-authored-by: Adam Locke <adam.locke@elastic.co>
2020-10-19 12:14:23 -04:00
Adam Locke 789ee2d73e
[DOCS] Combining important config settings into a single page (#63849)
* Combining important config settings into a single page.

* Updating ids for two pages causing link errors and implementing redirects.
2020-10-19 10:02:22 -04:00
Aref Razavi 245663e5b7 Remove useless parentheses in bucket_key formula (#63868) 2020-10-19 11:54:21 +02:00
David Roberts 977a4ad3f9
[ML] Change docs test mute comment (#63866)
The original comment mentioned issue #48583, but issue #48941
is specifically open for this mute.  However, this is
inappropriate, as the underlying reason the test cannot be
unmuted is the same as for all the other tests skipped with the
comment "Kibana sample data": issues #51572, #51576 and #51678.

Closes #48941
2020-10-19 10:17:27 +01:00
Julie Tibshirani a98fd8b6e1
Mark the search 'fields' option as beta. (#63699)
We've identified two important enhancements that may affect the API. We expect
any API changes from these enhancements to be minor, but want to leave open the
possibility for small breaks. For example, we may end up returning unmapped
fields by default, or omitting nested fields from the root hit. The impact to
users should be quite small.

We're tracking the issues we need to resolve before removing the 'beta' label
here: #60985.
2020-10-16 17:13:29 -07:00
Jay Greenberg 16019a7b55 [DOCS] Add monitoring note (#61793) 2020-10-16 13:56:54 -07:00
István Zoltán Szabó ccc820a972
[DOCS] Clarifies destination index mappings for Transforms. (#63794) 2020-10-16 15:24:33 +02:00
Igor Motov e6c70f6811
Add value_count mode to rate agg (#63687)
Adds a new value count mode to the rate aggregation.

Closes #63575
2020-10-15 18:00:44 -04:00
James Rodewig 9b8826c581
[DOCS] Document `discovery.seed_hosts` only supports YAML sequences and arrays (#63746) 2020-10-15 16:28:35 -04:00
Josh Devins 9b8b20a32b
[DOCS] Clarifies the effect of per-field boosting (#63733)
The original description of per-field boosting is incorrect. Boosting a
field does not imply that it is more important relative to other fields.
It simply means that the score is multiplied by the supplied boost
value. Due to the differences in each field's term and document
statistics, it's not possible to imply relative importance of fields
based on the per-field boost value alone.
2020-10-15 09:24:32 -04:00
Hendrik Muhs d1e39e8269
[Transform] add support for "missing" aggregation (#63651)
add support for the missing (bucket) aggregation (counts docs with a configured missing field value)
in transform. The output is mapped to name:count, the mapping type is long.
2020-10-15 08:41:33 +02:00
Christoph Büscher b44a03d837
[Docs] Correct default scroll_size for update by query (#63656)
The current _update_by_query documentation mentions a scroll_size default of 100 and later another default of 1000.
We use the default of 1000 defined in AbstractBulkByScrollRequest and this PR changes the documentation accordingly.

Closes #63637
2020-10-14 16:42:21 +02:00
James Rodewig 505b03768a [DOCS] Reword EQL intro 2020-10-14 10:02:45 -04:00
James Rodewig c6a13d1cee
[DOCS] EQL: Remove `match` fn (#63271) 2020-10-14 09:57:29 -04:00
James Rodewig 857c2d1cd4
[DOCS] Update `ignore_unavailable` default for EQL search API (#63210) 2020-10-14 09:36:11 -04:00
James Rodewig f41de1bdce
[DOCS] EQL: Add `:` operator, remove wildcard operator (#63195) 2020-10-14 09:06:37 -04:00
James Rodewig 8527183f91
[DOCS] EQL: Remove Endgame EQL refs (#63636) 2020-10-14 08:34:11 -04:00
James Rodewig d7c5d37697 [DOCS] Remove unneeded word in EQL docs 2020-10-13 13:56:56 -04:00
James Rodewig e0cc841a60
[DOCS] EQL: Document multi-value field support (#63622) 2020-10-13 12:26:07 -04:00
Andras Palinkas 190d9fe3de
SQL: Escaped wildcard (*) not accepted in LIKE (#63428)
For a query like `SELECT name FROM test WHERE name LIKE ''%c*'` ES SQL
generates an error. `*` is not a special character in a `LIKE` construct
and it's expected to not needing to be escaped, so the previous query
should work as is.
In the LIKE pattern any `*` character was treated as invalid character
and the usage of `%` or `_` was suggested instead. But `*` is a valid,
acceptable non-wildcard on the right side of the `LIKE` operator.

Fix: #55108
2020-10-13 09:38:35 -04:00
Przemysław Witek d9e7d88f08
[ML] Allow setting num_top_classes to a special value -1 (#63587) 2020-10-13 13:14:17 +02:00
István Zoltán Szabó e8930a44a4
[DOCS] Adds AUC ROC classification metric to the API examples (#63563) 2020-10-13 11:03:20 +02:00
James Rodewig 2fd11f58a0
[DOCS] Fix typo in search profile docs (#63522) (#63557)
Co-authored-by: ondrejbardon <70531797+ondrejbardon@users.noreply.github.com>
2020-10-12 10:12:45 -04:00
Rory Hunter ec350157ea
Use a tiny base image for Docker builds (#52519)
Closes #51670, closes #50838.

Introduce a tiny base image for Docker builds. It aims to create a basic filesystem with as little as possible, which is mostly glibc, busybox and bash. A statically-built curl is also provided.

We still use CentOS 8 as a base. All the fun stuff happens in the Dockerfile.
2020-10-12 14:58:01 +01:00
James Rodewig 04c8ad3ced
[DOCS] EQL: Move to beta (#63284) 2020-10-12 08:55:16 -04:00
Adrien Grand 62348b6a8a
Document standard metadata entries. (#61941)
We standardize on some metadata entries that we plan to later leverage
in Kibana in order to provide a better out-of-the-box experience, e.g.
different visualizations make sense on gauges and counters.
2020-10-12 09:49:39 +02:00
Igor Motov 34bff3f776
Add support for histogram fields to rate aggregation (#63289)
The rate aggregation now supports histogram fields. At the moment only sum
is supported. 

Closes #62939
2020-10-08 16:54:25 -04:00
István Zoltán Szabó b517d4d9b5
[DOCS] Adds huber and msle metrics to Evaluate API example calls (#63414) 2020-10-08 17:05:04 +02:00
James Rodewig 1b0350bb80
[DOCS] Move searchable snapshots to beta (#63436) 2020-10-08 08:52:44 -04:00
Przemysław Witek b0019bd0a6
[ML] Validate that AucRoc has the data necessary to be calculated (#63302) 2020-10-08 08:19:43 +02:00
Lisa Cawley 5804a37f60
[DOCS] Add props for ILM searchable snapshot links (#63430) 2020-10-07 10:22:41 -07:00
James Rodewig 2ae0d2cb6e
[DOCS] Add xref to data tiers content (#63426) 2020-10-07 13:10:42 -04:00
Lisa Cawley 033dd5b89a
[DOCS] Add props for searchable snapshots API links (#63420) 2020-10-07 09:16:07 -07:00
Andrei Dan d588cab747
DOCS: general overview of data tiers and roles (#63086)
This adds general overview documentation for data tiers, 
the data tiers specific node roles, and their application in
ILM.

Co-authored-by: Lee Hinman <dakrone@users.noreply.github.com>
Co-authored-by: debadair <debadair@elastic.co>
2020-10-07 17:06:54 +01:00
Tim Brooks 09d1bee07d
Add ess marker to indexing_pressure.memory.limit (#61127)
Adds marker indicating this setting is supported on Cloud.
2020-10-06 13:57:32 -06:00
James Rodewig 39d064d668
[DOCS] Update snowball links (#63351) 2020-10-06 15:29:57 -04:00
James Rodewig 80a828c15f
[DOCS] Update link to Snowball documentation (#63305) (#63347)
The current link points to an obsolete site, which is no longer maintained.

Co-authored-by: Stefan Walter <67258699+rd-stefan-walter@users.noreply.github.com>
2020-10-06 13:40:51 -04:00
Gordon Brown 91f4b58bf7
Deprecate REST access to System Indices (#60945)
This PR adds deprecation warnings when accessing System Indices via the REST layer. At this time, these warnings are only enabled for Snapshot builds by default, to allow projects external to Elasticsearch additional time to adjust their access patterns.

Deprecation warnings will be triggered by all REST requests which access registered System Indices, except for purpose-specific APIs which access System Indices as an implementation detail a few specific APIs which will continue to allow access to system indices by default:

- `GET _cluster/health`
- `GET {index}/_recovery`
- `GET _cluster/allocation/explain`
- `GET _cluster/state`
- `POST _cluster/reroute`
- `GET {index}/_stats`
- `GET {index}/_segments`
- `GET {index}/_shard_stores`
- `GET _cat/[indices,aliases,health,recovery,shards,segments]`

Deprecation warnings for accessing system indices take the form:
```
this request accesses system indices: [.some_system_index], but in a future major version, direct access to system indices will be prevented by default
```
2020-10-06 11:13:48 -06:00
Adam Locke 266ac76fe0
[DOCS] Updating permissions language for RPM install packages (#63277)
* Updating permissions language for RPM install packages.

* Fix typo
2020-10-06 12:35:22 -04:00
lcawl 2177b46289 [DOCS] Fixes typo 2020-10-06 09:19:43 -07:00
Lisa Cawley 49ab8f8688
[DOCS] Add feature_importance_baseline to get trained model API (#63279)
Co-authored-by: Benjamin Trent <ben.w.trent@gmail.com>
2020-10-06 07:56:55 -07:00
István Zoltán Szabó de3ce8bc39
[DOCS] Adds delta and offset parameters to Evaluate DFA API docs (#63317) 2020-10-06 16:06:35 +02:00
David Kyle 6784e826a8
[ML] Audit message when nightly maintenance times out (#63252)
During deletion of old ml data set the delete by query timeout to 8 hours and 
audit a job message when the nightly maintenance task times out.
2020-10-06 15:01:40 +01:00
Mayya Sharipova c45724079c
Fix fields retrieval on unsinged_long field (#63119)
This fixes fields retrieval on unsigned_long field

1) For docvalue_fields a custom UnsignedLongLeafFieldData::getLeafValueFetcher
is implemented that correctly retrieves doc values.

2) For stored fields, an error was fixed in UnsignedLongFieldMapper
 how stored values were stored. Before they were incorrectly
stored in the shifted format, now they are stored as original
values in String format.

Relates to #60050
2020-10-06 05:44:50 -04:00
James Rodewig b7b7e12f3c
[DOCS] Document static/dynamic watcher settings (#62218) 2020-10-05 15:34:47 -04:00
James Rodewig 0aa0811aba
[DOCS] Make EQL case-sensitive by default (#63270) 2020-10-05 15:29:48 -04:00
James Rodewig 5c5acfddef
[DOCS] Clarify `allow_no_indices` def (#63209) 2020-10-05 12:31:44 -04:00
Andrei Stefan 7550e0664c
Remove case_sensitive request option (#63218)
Make EQL case sensitive by default and adapt some of the string functions
Remove the case sensitive option from Between string function
Add case_insensitive option to term and wildcard queries usage
2020-10-05 16:53:25 +03:00
Lisa Cawley 0013d3d7e8
[DOCS] Clarify BWC of monitoring clusters (#63151) 2020-10-02 14:06:20 -07:00
Armin Braun f7f239d39a
Clone Snapshot API (#61839)
Adds clone snapshot API to clone part of a snapshot into a new snapshot.
2020-10-02 21:28:44 +02:00
James Rodewig cb9e61fae5
[DOCS] EQL: Update grammary for escaped event categories (#63202) 2020-10-02 15:03:29 -04:00
Lisa Cawley 51f9bf657d
[DOCS] Fix titles for ML APIs (#63152) 2020-10-02 11:53:49 -07:00
István Zoltán Szabó baffdd1ec0
[DOCS] Updates trained models API docs titles. (#63165) 2020-10-02 10:15:14 -07:00
James Rodewig daef606de7
[DOCS] EQL: Replace ?"..." with """...""" for raw strings (#63191) 2020-10-02 11:20:24 -04:00
James Rodewig 1b878c8775
[DOCS] EQL: Reorganize EQL syntax sections (#63179) 2020-10-02 09:46:27 -04:00
James Rodewig 382c454c8b
[DOCS] Corrected track_total_hits def (#62830) (#63180)
Co-authored-by: John Berryman <jnbrymn@github.com>
2020-10-02 09:46:09 -04:00
Benjamin Trent 7bd6e78dae
[ML] adding for_export flag for ml plugin GET resource APIs (#63092)
This adds the new `for_export` flag to the following APIs:

- GET _ml/anomaly_detection/<job_id>
- GET _ml/datafeeds/<datafeed_id>
- GET _ml/data_frame/analytics/<analytics_id>

The flag is designed for cloning or exporting configuration objects to later be put into the same cluster or a separate cluster. 

The following fields are not returned in the objects:

- any field that is not user settable (e.g. version, create_time)
- any field that is a calculated default value (e.g. datafeed chunking_config)
- any field that would effectively require changing to be of use (e.g. datafeed job_id)
- any field that is automatically set via another Elastic stack process (e.g. anomaly job custom_settings.created_by)


closes https://github.com/elastic/elasticsearch/issues/63055
2020-10-02 08:29:19 -04:00
Rafi Estrada 907d0eb696
[Docs] Correct typo (#63102) 2020-10-02 10:16:26 +02:00
Przemyslaw Gomulka b38eaae47f
[doc] Rounding range query rules (#63109)
a documentation explaining defaulting of missing fields when using date math parser.
relates #62268
2020-10-02 08:59:27 +02:00
Benjamin Trent 1084aaf18a
[ML] renames */inference* apis to */trained_models* (#63097)
This commit renames all `inference` CRUD APIs to `trained_models`.

This aligns with internal terminology, documentation, and use-cases.
2020-10-01 12:13:49 -04:00