Commit Graph

7762 Commits

Author SHA1 Message Date
Rafi Estrada 907d0eb696
[Docs] Correct typo (#63102) 2020-10-02 10:16:26 +02:00
Przemyslaw Gomulka b38eaae47f
[doc] Rounding range query rules (#63109)
a documentation explaining defaulting of missing fields when using date math parser.
relates #62268
2020-10-02 08:59:27 +02:00
Benjamin Trent 1084aaf18a
[ML] renames */inference* apis to */trained_models* (#63097)
This commit renames all `inference` CRUD APIs to `trained_models`.

This aligns with internal terminology, documentation, and use-cases.
2020-10-01 12:13:49 -04:00
Lee Hinman 5fca68a155
Add telemetry for data tiers (#63031)
This commit adds telemetry for our data tier formalization. This telemetry helps determine the
topology of the cluster with regard to the content, hot, warm, & cold tiers/roles.

An example of the telemetry looks like:

```
GET /_xpack/usage?human
{
  ...
  "data_tiers" : {
    "available" : true,
    "enabled" : true,
    "data_warm" : {
      ...
    },
    "data_cold" : {
      ...
    },
    "data_content" : {
      "node_count" : 1,
      "index_count" : 6,
      "total_shard_count" : 6,
      "primary_shard_count" : 6,
      "doc_count" : 71,
      "total_size" : "59.6kb",
      "total_size_bytes" : 61110,
      "primary_size" : "59.6kb",
      "primary_size_bytes" : 61110,
      "primary_shard_size_avg" : "9.9kb",
      "primary_shard_size_avg_bytes" : 10185,
      "primary_shard_size_median" : "8kb",
      "primary_shard_size_median_bytes" : 8254,
      "primary_shard_size_mad" : "7.2kb",
      "primary_shard_size_mad_bytes" : 7391
    },
    "data_hot" : {
       ...
    }
  }
}
```

The fields are as follows:

- node_count :: number of nodes with this tier/role
- index_count :: number of indices on this tier
- total_shard_count :: total number of shards for all nodes in this tier
- primary_shard_count :: number of primary shards for all nodes in this tier
- doc_count :: number of documents for all nodes in this tier
- total_size_bytes :: total number of bytes for all shards for all nodes in this tier
- primary_size_bytes :: number of bytes for all primary shards on all nodes in this tier
- primary_shard_size_avg_bytes :: average shard size for primary shard in this tier
- primary_shard_size_median_bytes :: median shard size for primary shard in this tier
- primary_shard_size_mad_bytes :: [median absolute deviation](https://en.wikipedia.org/wiki/Median_absolute_deviation) of shard size for primary shard in this tier

Relates to #60848
2020-10-01 08:35:10 -06:00
James Rodewig 15d4d9597c
[DOCS] EQL: date_nanos timestamp is not supported (#63101) 2020-09-30 17:31:24 -04:00
bellengao 0c88c19c1d
Add country_name to the default properties of geoip ingest processor (#62915) 2020-09-30 14:06:51 -05:00
James Rodewig d8cfd569e6
[DOCS] Document escaped backticks for identifiers (#63079) 2020-09-30 11:56:23 -04:00
Alan Woodward 981258b02b
Remove TypeFieldMapper (#62838)
We don't need a special TypeFieldMapper for anything in particular; all access
to the type field can be done via a TypeFieldType that issues appropriate
deprecation warnings.

Relates to #41059
2020-09-30 15:47:29 +01:00
Lisa Cawley ecf9e929ba
[DOCS] Add experimental tag to inference processor and bucket aggregation (#63023) 2020-09-30 07:20:38 -07:00
James Rodewig b3e8767a35
[DOCS] Clarify that v2.0+ hyphenation files aren't supported (#60579) (#63072)
Co-authored-by: James Rodewig <40268737+jrodewig@users.noreply.github.com>

Co-authored-by: jgkirschbaum <juergen.kirschbaum@gmail.com>
2020-09-30 09:28:23 -04:00
István Zoltán Szabó 7f24c4df24
[DOCS] Adds limitation item about using scripts in transforms (#63021)
* [DOCS] Adds limitation item about using scripts in transforms.

* [DOCS] Adds scripts related limitation item to transforms docs.

* [DOCS] Merges two bullets, adds a new one, and removes last sentences.

* [DOCS] Refines last bullet.

* [DOCS] Addresses feedback.

* [DOCS] Removes low info content.
2020-09-30 15:16:10 +02:00
Jakob Reiter 534b179c33
[DOCS] Updated target_field description of the json ingest processor (#61968)
Co-authored-by: Dan Hermann <danhermann@users.noreply.github.com>
2020-09-30 08:43:29 -04:00
István Zoltán Szabó f0bfefe917
[DOCS] Adds data stream and ILM related limitation items to transforms (#63016) 2020-09-30 10:10:57 +02:00
Przemysław Witek cd1a27f273
[ML] Implement AucRoc metric for classification (#60502) 2020-09-30 08:56:23 +02:00
James Rodewig 277709004e
[DOCS] Fix elasticsearch-croneval chunking (#63008) 2020-09-29 09:53:20 -04:00
Howard e50799bc7e
[DOCS] Remove duplicate disk.threshold_enabled setting. (#62924) 2020-09-29 08:58:46 -04:00
James Rodewig 844558069b
[DOCS] EQL: Clarify EQL docs (#62961) 2020-09-28 15:29:35 -04:00
Nik Everett 2c93cd38e4
Docs: Support """ in console-response (#62876)
We support `"""` in `console` snippets to emulate kibana's CONSOLE.
CONSOLE also spits out `"""` when a json field contains a new line or a
double quote. This adds support for those sorts of responses to the
handling of `console-response` snippets.
2020-09-28 10:49:12 -04:00
James Rodewig 8bdf36d18b
[DOCS] Document shard sizing guide (#61942)
Revises the current 'How to avoid oversharding' docs to incorporate
information from our [shard sizing blog post][0].

Changes:

* Streamlines introduction
* Adds "Things to remember" section to describe how shards work
* Adds "Guidelines" section based on blog tips
* Creates a "Fix an oversharded cluster" section

[0]: https://www.elastic.co/blog/how-many-shards-should-i-have-in-my-elasticsearch-cluster
2020-09-28 09:24:40 -04:00
Inbar Shimshon 3938392204
Update ilm-settings.asciidoc 2020-09-25 16:25:00 +03:00
Peter Ansell b40bdd3093
Add network from MaxMind Geo ASN database (#61676)
This adds the network property from the MaxMind Geo ASN database. 
This enables analysis of IP data based on the subnets that MaxMind have 
previously identified for ASN networks.

closes #60942
2020-09-24 11:51:50 -05:00
James Rodewig a0149576ef
[DOCS] Correct the documented behaviour of `track_total_hits` (#62837) (#62866)
If `track_total_hits=true` is used, the exact value of the number of hits is returned - i.e. the value is effectively limitless, and not the default value of 10,000

Co-authored-by: AndyHunt66 <andrew.hunt@elastic.co>
2020-09-24 09:16:56 -04:00
Mayya Sharipova ff55296f7a
Introduce 64-bit unsigned long field type (#60050)
This field type supports
- indexing of integer values from [0, 18446744073709551615]
- precise queries (term, range)
- precise sort and terms aggregations
- other aggregations are based on conversion of long values
  to double and can be imprecise for large values.

Closes #32434
2020-09-23 12:06:21 -04:00
Alan Woodward b1d6d42a68
Remove mapping boost parameter entirely (#62639)
Follow up to #62623, this commit removes support in 8x for index-time boosts.
There is no longer a boost field on MappedFieldType. Indexes created in 8x
and after will throw exceptions if a boost parameter is included in mappings,
and indexes created in 7x will emit warnings.
2020-09-23 14:28:59 +01:00
Jay Modi 242083a36e
Dedicated threadpool for system index writes (#61655)
This commit adds a dedicated threadpool for system index write
operations. The dedicated resources for system index writes serves as
a means to ensure that user activity does not block important system
operations from occurring such as the management of users and roles.
2020-09-22 12:14:45 -06:00
James Rodewig acac14a35f [DOCS] EQL: Note = is not an equality operator 2020-09-22 13:54:19 -04:00
Lisa Cawley 4813b9b5dc
[DOCS] Add realm limitations for monitoring clusters (#62714) 2020-09-22 09:35:18 -07:00
Adam Locke 71b24db8f2
[DOCS] Add remote node as a node role (#62730)
* Adding remote node as a node role.

* Incorporating reviewer feedback.
2020-09-22 11:39:58 -04:00
Marios Trivyzas ad5ae4d887
EQL: Remove support for `=` for comparisons (#62756)
Since `=` is rarely used and is undocumented we its support for
equality comparisons keeping `==` as the only option. `=` is now only
used for assignments like in `maxspan=10m`.

Closes: #62650
2020-09-22 17:37:37 +02:00
James Rodewig 74ffbe7dcc [DOCS] EQL: Style fixes 2020-09-21 19:43:19 -04:00
James Rodewig 79a0a6406a [DOCS] EQL: Style fixes 2020-09-21 18:41:21 -04:00
James Rodewig 543919cea7
[DOCS] EQL: Improve regsvr32 misuse explanation (#62722)
Expands the introduction to better explain what regsvr32 misuse is and
how it works at a high level.
2020-09-21 18:36:35 -04:00
Lisa Cawley e48eab95e9
[DOCS] Formatting fix in get trained model API (#62643) 2020-09-21 08:19:37 -07:00
Binu R J da511f4e03
SQL: Implement FORMAT function (#55454)
Implement FORMAT according to the SQL Server spec: https://docs.microsoft.com/en-us/sql/t-sql/functions/format-transact-sql?view=sql-server-ver15#ExampleD by translating to the java.time patterns used in DATETIME_FORMAT.

Closes: #54965

Co-authored-by: Marios Trivyzas <matriv@users.noreply.github.com>
Co-authored-by: Bogdan Pintea <bogdan.pintea@elastic.co>
Co-authored-by: Andrei Stefan <astefan@users.noreply.github.com>
2020-09-21 15:32:11 +02:00
Bogdan Pintea 2bc4d7eb19
SQL: Document Maven Central as a JDBC repository (#62618)
* Add Maven Central as a JDBC repository

Document Maven Central as a JDBC repository.
2020-09-21 12:55:04 +02:00
Christoph Büscher ea2dbd93b4
Add field type for version strings (#59773)
This PR adds a new 'version' field type that allows indexing string values
representing software versions similar to the ones defined in the Semantic
Versioning definition (semver.org). The field behaves very similar to a
'keyword' field but allows efficient sorting and range queries that take into
accound the special ordering needed for version strings. For example, the main
version parts are sorted numerically (ie 2.0.0 < 11.0.0) whereas this wouldn't
be possible with 'keyword' fields today.

Valid version values are similar to the Semantic Versioning definition, with the
notable exception that in addition to the "main" version consiting of
major.minor.patch, we allow less or more than three numeric identifiers, i.e.
"1.2" or "1.4.6.123.12" are treated as valid too.

Relates to #48878
2020-09-21 11:04:22 +02:00
Christos Soulios b857768bb5
Histogram field type support for min/max aggregations (#62532)
Implement min/max aggregations for histogram fields.

Closes #60951
2020-09-19 23:34:43 +03:00
Lee Hinman 0c3599577e
Add index.routing.allocation.prefer._tier setting (#62589)
This commit adds the `index.routing.allocation.prefer._tier` setting to the
`DataTierAllocationDecider`. This special-purpose allocation setting lets a user specify a
preference-based list of tiers for an index to be assigned to. For example, if the setting were set
to:

```
"index.routing.allocation.prefer._tier": "data_hot,data_warm,data_content"
```

If the cluster contains any nodes with the `data_hot` role, the decider will only allow them to be
allocated on the `data_hot` node(s). If there are no `data_hot` nodes, but there are `data_warm` and
`data_content` nodes, then the index will be allowed to be allocated on `data_warm` nodes.

This allows us to specify an index's preference for tier(s) without causing the index to be
unassigned if no nodes of a preferred tier are available.

Subsequent work will change the ILM migration to make additional use of this setting.

Relates to #60848
2020-09-18 14:49:59 -06:00
James Rodewig 786a6202ee
[DOCS] Fix issue link in redirects (#62654) 2020-09-18 15:02:06 -04:00
Henning Andersen d8dd6e38a7
Autoscaling decision return absolute capacity (#61575)
The autoscaling decision API now returns an absolute capacity,
and leaves the actual decision of whether a scale up or down
is needed to the orchestration system.

The decision API now returns both a tier and node level required
and current capacity as wells as a decider level breakdown of the
same though with in particular current memory still not populated.
2020-09-18 20:44:20 +02:00
Leaf-Lin 0a716ec7d5
[DOCS] Fix typo in SLM docs (#62591) 2020-09-18 12:31:05 -04:00
markharwood fe9145fa5e
Search - add case insensitive flag for "term" family of queries (#61596)
Adds  case insensitive flag for term, prefix, and wildcard queries

Closes #61546
2020-09-18 17:17:08 +01:00
Alan Woodward 2b0418df5f
Emit deprecation warnings when boosts are defined in mappings (#62623)
We removed index-time boosting back in 5x, and we no longer document the 'boost'
parameter on any of our mapping types. However, it is still possible to define an
index-time boost on a field mapper for a surprisingly large number of field types, and
they even have an effect (sometimes, on some queries).

As a first step in finally removing all traces of index time boosting, this comment emits
a deprecation warning whenever a boost parameter is found on a mapping definition.
2020-09-18 15:34:54 +01:00
Benjamin Trent a653a1cbb8
[ML] all multiple wildcard values for GET Calendars, Events, and DELETE forecasts (#62563)
This commit adjusts the following APIs so now they not only support an `_all` case, but wildcard patterned Ids as well.

- `GET _ml/calendars/<calendar_id>/events`
- `GET _ml/calendars/<calendar_id>`
- `GET _ml/anomaly_detectors/<job_id>/model_snapshots/<snapshot_id>`
- `DELETE _ml/anomaly_detectors/<job_id>/_forecast/<forecast_id>`
2020-09-18 09:39:40 -04:00
James Rodewig 6b36be281a
[DOCS] EQL: Disallow chained comparisons (#62570) 2020-09-18 08:26:48 -04:00
Benjamin Trent fdb7b6d3b5
[ML] Add new include flag to GET inference/<model_id> API for model training metadata (#61922)
Adds new flag include to the get trained models API
The flag initially has two valid values: definition, total_feature_importance.
Consequently, the old include_model_definition flag is now deprecated.
When total_feature_importance is included, the total_feature_importance field is included in the model metadata object.
Including definition is the same as previously setting include_model_definition=true.
2020-09-18 07:11:38 -04:00
Martijn van Groningen e0caf3f72e
Unmute rollover docs test (#62603)
The underlying issue was fixed a while ago in Lucene:
https://issues.apache.org/jira/browse/LUCENE-9517

and went away when lucene snapshot version was upgraded.

Also the name of the index to rollover had to be slightly changed,
so that it doesn't collide with data stream template's namespace.
(a regular index can't be created in the namespace that is managed
by a template that creates data streams)

Closes #62043
2020-09-18 10:51:44 +02:00
James Rodewig f8c013d0fb
[DOCS] Clarify `http.max_content_length` def (#62562) 2020-09-17 12:49:18 -04:00
Adam Locke 9de8a2f695
Removing erroneous screen capture. (#62561) 2020-09-17 12:22:37 -04:00
Lee Hinman 22d54656a2
Allocate new indices on "hot" or "content" tier depending on data stream inclusion (#62338)
This commit changes the default allocation on the "hot" tier to allocating the newly created index
to the "hot" tier if it is part of a new or existing data stream, and to the "content" tier if it is
not part of a data stream.

Overriding any of the index.routing.allocation.(include|exclude|require).* settings continues to
cause the initial allocation not to be set (no change in behavior).

Relates to #60848
2020-09-17 08:52:55 -06:00