Commit Graph

7773 Commits

Author SHA1 Message Date
James Rodewig 5c5acfddef
[DOCS] Clarify `allow_no_indices` def (#63209) 2020-10-05 12:31:44 -04:00
Andrei Stefan 7550e0664c
Remove case_sensitive request option (#63218)
Make EQL case sensitive by default and adapt some of the string functions
Remove the case sensitive option from Between string function
Add case_insensitive option to term and wildcard queries usage
2020-10-05 16:53:25 +03:00
Lisa Cawley 0013d3d7e8
[DOCS] Clarify BWC of monitoring clusters (#63151) 2020-10-02 14:06:20 -07:00
Armin Braun f7f239d39a
Clone Snapshot API (#61839)
Adds clone snapshot API to clone part of a snapshot into a new snapshot.
2020-10-02 21:28:44 +02:00
James Rodewig cb9e61fae5
[DOCS] EQL: Update grammary for escaped event categories (#63202) 2020-10-02 15:03:29 -04:00
Lisa Cawley 51f9bf657d
[DOCS] Fix titles for ML APIs (#63152) 2020-10-02 11:53:49 -07:00
István Zoltán Szabó baffdd1ec0
[DOCS] Updates trained models API docs titles. (#63165) 2020-10-02 10:15:14 -07:00
James Rodewig daef606de7
[DOCS] EQL: Replace ?"..." with """...""" for raw strings (#63191) 2020-10-02 11:20:24 -04:00
James Rodewig 1b878c8775
[DOCS] EQL: Reorganize EQL syntax sections (#63179) 2020-10-02 09:46:27 -04:00
James Rodewig 382c454c8b
[DOCS] Corrected track_total_hits def (#62830) (#63180)
Co-authored-by: John Berryman <jnbrymn@github.com>
2020-10-02 09:46:09 -04:00
Benjamin Trent 7bd6e78dae
[ML] adding for_export flag for ml plugin GET resource APIs (#63092)
This adds the new `for_export` flag to the following APIs:

- GET _ml/anomaly_detection/<job_id>
- GET _ml/datafeeds/<datafeed_id>
- GET _ml/data_frame/analytics/<analytics_id>

The flag is designed for cloning or exporting configuration objects to later be put into the same cluster or a separate cluster. 

The following fields are not returned in the objects:

- any field that is not user settable (e.g. version, create_time)
- any field that is a calculated default value (e.g. datafeed chunking_config)
- any field that would effectively require changing to be of use (e.g. datafeed job_id)
- any field that is automatically set via another Elastic stack process (e.g. anomaly job custom_settings.created_by)


closes https://github.com/elastic/elasticsearch/issues/63055
2020-10-02 08:29:19 -04:00
Rafi Estrada 907d0eb696
[Docs] Correct typo (#63102) 2020-10-02 10:16:26 +02:00
Przemyslaw Gomulka b38eaae47f
[doc] Rounding range query rules (#63109)
a documentation explaining defaulting of missing fields when using date math parser.
relates #62268
2020-10-02 08:59:27 +02:00
Benjamin Trent 1084aaf18a
[ML] renames */inference* apis to */trained_models* (#63097)
This commit renames all `inference` CRUD APIs to `trained_models`.

This aligns with internal terminology, documentation, and use-cases.
2020-10-01 12:13:49 -04:00
Lee Hinman 5fca68a155
Add telemetry for data tiers (#63031)
This commit adds telemetry for our data tier formalization. This telemetry helps determine the
topology of the cluster with regard to the content, hot, warm, & cold tiers/roles.

An example of the telemetry looks like:

```
GET /_xpack/usage?human
{
  ...
  "data_tiers" : {
    "available" : true,
    "enabled" : true,
    "data_warm" : {
      ...
    },
    "data_cold" : {
      ...
    },
    "data_content" : {
      "node_count" : 1,
      "index_count" : 6,
      "total_shard_count" : 6,
      "primary_shard_count" : 6,
      "doc_count" : 71,
      "total_size" : "59.6kb",
      "total_size_bytes" : 61110,
      "primary_size" : "59.6kb",
      "primary_size_bytes" : 61110,
      "primary_shard_size_avg" : "9.9kb",
      "primary_shard_size_avg_bytes" : 10185,
      "primary_shard_size_median" : "8kb",
      "primary_shard_size_median_bytes" : 8254,
      "primary_shard_size_mad" : "7.2kb",
      "primary_shard_size_mad_bytes" : 7391
    },
    "data_hot" : {
       ...
    }
  }
}
```

The fields are as follows:

- node_count :: number of nodes with this tier/role
- index_count :: number of indices on this tier
- total_shard_count :: total number of shards for all nodes in this tier
- primary_shard_count :: number of primary shards for all nodes in this tier
- doc_count :: number of documents for all nodes in this tier
- total_size_bytes :: total number of bytes for all shards for all nodes in this tier
- primary_size_bytes :: number of bytes for all primary shards on all nodes in this tier
- primary_shard_size_avg_bytes :: average shard size for primary shard in this tier
- primary_shard_size_median_bytes :: median shard size for primary shard in this tier
- primary_shard_size_mad_bytes :: [median absolute deviation](https://en.wikipedia.org/wiki/Median_absolute_deviation) of shard size for primary shard in this tier

Relates to #60848
2020-10-01 08:35:10 -06:00
James Rodewig 15d4d9597c
[DOCS] EQL: date_nanos timestamp is not supported (#63101) 2020-09-30 17:31:24 -04:00
bellengao 0c88c19c1d
Add country_name to the default properties of geoip ingest processor (#62915) 2020-09-30 14:06:51 -05:00
James Rodewig d8cfd569e6
[DOCS] Document escaped backticks for identifiers (#63079) 2020-09-30 11:56:23 -04:00
Alan Woodward 981258b02b
Remove TypeFieldMapper (#62838)
We don't need a special TypeFieldMapper for anything in particular; all access
to the type field can be done via a TypeFieldType that issues appropriate
deprecation warnings.

Relates to #41059
2020-09-30 15:47:29 +01:00
Lisa Cawley ecf9e929ba
[DOCS] Add experimental tag to inference processor and bucket aggregation (#63023) 2020-09-30 07:20:38 -07:00
James Rodewig b3e8767a35
[DOCS] Clarify that v2.0+ hyphenation files aren't supported (#60579) (#63072)
Co-authored-by: James Rodewig <40268737+jrodewig@users.noreply.github.com>

Co-authored-by: jgkirschbaum <juergen.kirschbaum@gmail.com>
2020-09-30 09:28:23 -04:00
István Zoltán Szabó 7f24c4df24
[DOCS] Adds limitation item about using scripts in transforms (#63021)
* [DOCS] Adds limitation item about using scripts in transforms.

* [DOCS] Adds scripts related limitation item to transforms docs.

* [DOCS] Merges two bullets, adds a new one, and removes last sentences.

* [DOCS] Refines last bullet.

* [DOCS] Addresses feedback.

* [DOCS] Removes low info content.
2020-09-30 15:16:10 +02:00
Jakob Reiter 534b179c33
[DOCS] Updated target_field description of the json ingest processor (#61968)
Co-authored-by: Dan Hermann <danhermann@users.noreply.github.com>
2020-09-30 08:43:29 -04:00
István Zoltán Szabó f0bfefe917
[DOCS] Adds data stream and ILM related limitation items to transforms (#63016) 2020-09-30 10:10:57 +02:00
Przemysław Witek cd1a27f273
[ML] Implement AucRoc metric for classification (#60502) 2020-09-30 08:56:23 +02:00
James Rodewig 277709004e
[DOCS] Fix elasticsearch-croneval chunking (#63008) 2020-09-29 09:53:20 -04:00
Howard e50799bc7e
[DOCS] Remove duplicate disk.threshold_enabled setting. (#62924) 2020-09-29 08:58:46 -04:00
James Rodewig 844558069b
[DOCS] EQL: Clarify EQL docs (#62961) 2020-09-28 15:29:35 -04:00
Nik Everett 2c93cd38e4
Docs: Support """ in console-response (#62876)
We support `"""` in `console` snippets to emulate kibana's CONSOLE.
CONSOLE also spits out `"""` when a json field contains a new line or a
double quote. This adds support for those sorts of responses to the
handling of `console-response` snippets.
2020-09-28 10:49:12 -04:00
James Rodewig 8bdf36d18b
[DOCS] Document shard sizing guide (#61942)
Revises the current 'How to avoid oversharding' docs to incorporate
information from our [shard sizing blog post][0].

Changes:

* Streamlines introduction
* Adds "Things to remember" section to describe how shards work
* Adds "Guidelines" section based on blog tips
* Creates a "Fix an oversharded cluster" section

[0]: https://www.elastic.co/blog/how-many-shards-should-i-have-in-my-elasticsearch-cluster
2020-09-28 09:24:40 -04:00
Inbar Shimshon 3938392204
Update ilm-settings.asciidoc 2020-09-25 16:25:00 +03:00
Peter Ansell b40bdd3093
Add network from MaxMind Geo ASN database (#61676)
This adds the network property from the MaxMind Geo ASN database. 
This enables analysis of IP data based on the subnets that MaxMind have 
previously identified for ASN networks.

closes #60942
2020-09-24 11:51:50 -05:00
James Rodewig a0149576ef
[DOCS] Correct the documented behaviour of `track_total_hits` (#62837) (#62866)
If `track_total_hits=true` is used, the exact value of the number of hits is returned - i.e. the value is effectively limitless, and not the default value of 10,000

Co-authored-by: AndyHunt66 <andrew.hunt@elastic.co>
2020-09-24 09:16:56 -04:00
Mayya Sharipova ff55296f7a
Introduce 64-bit unsigned long field type (#60050)
This field type supports
- indexing of integer values from [0, 18446744073709551615]
- precise queries (term, range)
- precise sort and terms aggregations
- other aggregations are based on conversion of long values
  to double and can be imprecise for large values.

Closes #32434
2020-09-23 12:06:21 -04:00
Alan Woodward b1d6d42a68
Remove mapping boost parameter entirely (#62639)
Follow up to #62623, this commit removes support in 8x for index-time boosts.
There is no longer a boost field on MappedFieldType. Indexes created in 8x
and after will throw exceptions if a boost parameter is included in mappings,
and indexes created in 7x will emit warnings.
2020-09-23 14:28:59 +01:00
Jay Modi 242083a36e
Dedicated threadpool for system index writes (#61655)
This commit adds a dedicated threadpool for system index write
operations. The dedicated resources for system index writes serves as
a means to ensure that user activity does not block important system
operations from occurring such as the management of users and roles.
2020-09-22 12:14:45 -06:00
James Rodewig acac14a35f [DOCS] EQL: Note = is not an equality operator 2020-09-22 13:54:19 -04:00
Lisa Cawley 4813b9b5dc
[DOCS] Add realm limitations for monitoring clusters (#62714) 2020-09-22 09:35:18 -07:00
Adam Locke 71b24db8f2
[DOCS] Add remote node as a node role (#62730)
* Adding remote node as a node role.

* Incorporating reviewer feedback.
2020-09-22 11:39:58 -04:00
Marios Trivyzas ad5ae4d887
EQL: Remove support for `=` for comparisons (#62756)
Since `=` is rarely used and is undocumented we its support for
equality comparisons keeping `==` as the only option. `=` is now only
used for assignments like in `maxspan=10m`.

Closes: #62650
2020-09-22 17:37:37 +02:00
James Rodewig 74ffbe7dcc [DOCS] EQL: Style fixes 2020-09-21 19:43:19 -04:00
James Rodewig 79a0a6406a [DOCS] EQL: Style fixes 2020-09-21 18:41:21 -04:00
James Rodewig 543919cea7
[DOCS] EQL: Improve regsvr32 misuse explanation (#62722)
Expands the introduction to better explain what regsvr32 misuse is and
how it works at a high level.
2020-09-21 18:36:35 -04:00
Lisa Cawley e48eab95e9
[DOCS] Formatting fix in get trained model API (#62643) 2020-09-21 08:19:37 -07:00
Binu R J da511f4e03
SQL: Implement FORMAT function (#55454)
Implement FORMAT according to the SQL Server spec: https://docs.microsoft.com/en-us/sql/t-sql/functions/format-transact-sql?view=sql-server-ver15#ExampleD by translating to the java.time patterns used in DATETIME_FORMAT.

Closes: #54965

Co-authored-by: Marios Trivyzas <matriv@users.noreply.github.com>
Co-authored-by: Bogdan Pintea <bogdan.pintea@elastic.co>
Co-authored-by: Andrei Stefan <astefan@users.noreply.github.com>
2020-09-21 15:32:11 +02:00
Bogdan Pintea 2bc4d7eb19
SQL: Document Maven Central as a JDBC repository (#62618)
* Add Maven Central as a JDBC repository

Document Maven Central as a JDBC repository.
2020-09-21 12:55:04 +02:00
Christoph Büscher ea2dbd93b4
Add field type for version strings (#59773)
This PR adds a new 'version' field type that allows indexing string values
representing software versions similar to the ones defined in the Semantic
Versioning definition (semver.org). The field behaves very similar to a
'keyword' field but allows efficient sorting and range queries that take into
accound the special ordering needed for version strings. For example, the main
version parts are sorted numerically (ie 2.0.0 < 11.0.0) whereas this wouldn't
be possible with 'keyword' fields today.

Valid version values are similar to the Semantic Versioning definition, with the
notable exception that in addition to the "main" version consiting of
major.minor.patch, we allow less or more than three numeric identifiers, i.e.
"1.2" or "1.4.6.123.12" are treated as valid too.

Relates to #48878
2020-09-21 11:04:22 +02:00
Christos Soulios b857768bb5
Histogram field type support for min/max aggregations (#62532)
Implement min/max aggregations for histogram fields.

Closes #60951
2020-09-19 23:34:43 +03:00
Lee Hinman 0c3599577e
Add index.routing.allocation.prefer._tier setting (#62589)
This commit adds the `index.routing.allocation.prefer._tier` setting to the
`DataTierAllocationDecider`. This special-purpose allocation setting lets a user specify a
preference-based list of tiers for an index to be assigned to. For example, if the setting were set
to:

```
"index.routing.allocation.prefer._tier": "data_hot,data_warm,data_content"
```

If the cluster contains any nodes with the `data_hot` role, the decider will only allow them to be
allocated on the `data_hot` node(s). If there are no `data_hot` nodes, but there are `data_warm` and
`data_content` nodes, then the index will be allowed to be allocated on `data_warm` nodes.

This allows us to specify an index's preference for tier(s) without causing the index to be
unassigned if no nodes of a preferred tier are available.

Subsequent work will change the ILM migration to make additional use of this setting.

Relates to #60848
2020-09-18 14:49:59 -06:00
James Rodewig 786a6202ee
[DOCS] Fix issue link in redirects (#62654) 2020-09-18 15:02:06 -04:00