Commit Graph

206 Commits

Author SHA1 Message Date
James Rodewig 904c866060 [DOCS] Fix EQL syntax admon 2020-08-26 13:39:23 -04:00
James Rodewig f79d70225b [DOCS] Remove dupe EQl fn/pipe TOC 2020-08-26 12:44:51 -04:00
James Rodewig 35b35148b9
[DOCS] Remove response params for #61428 (#61524) 2020-08-25 09:30:38 -04:00
Costin Leau 997376fbe6
EQL: Replace SearchHit in response with Event (#61428)
The building block of the eql response is currently the SearchHit. This
is a problem since it is tied to an actual search, and thus has scoring,
highlighting, shard information and a lot of other things that are not
relevant for EQL.
This becomes a problem when doing sequence queries since the response is
not generated from one search query and thus there are no SearchHits to
speak of.
Emulating one is not just conceptually incorrect but also problematic
since most of the data is missed or made-up.

As such this PR introduces a simple class, Event, that maps nicely to
the terminology while hiding the ES internals (the use of SearchHit or
GetResult/GetResponse depending on the API used).

Fix #59764
Fix #59779

Co-authored-by: Igor Motov <igor@motovs.org>
2020-08-25 14:27:56 +03:00
James Rodewig a7d4e8b148
[DOCS] Remove collapsible sections in EQL fn docs (#61498) 2020-08-24 14:19:29 -04:00
James Rodewig c688cb6bfd
[DOCS] Fix hyphenation for "time series" (#61472) 2020-08-24 10:34:41 -04:00
James Rodewig 77bb7320dd
[DOCS] Fix EQL threat detection example (#61367) 2020-08-20 09:55:49 -04:00
Andrei Stefan d54957d61f
EQL: Return sequence join keys in the original type (#61268) 2020-08-18 18:20:43 +03:00
James Rodewig a94e5cb7c4
[DOCS] Replace Wikipedia links with attribute (#61171) 2020-08-17 09:44:24 -04:00
James Rodewig 36f02c7869 [DOCS] Reword in EQL threat detection example 2020-08-14 15:50:31 -04:00
James Rodewig 46c438f16b [DOCS] Reword EQL example 2020-08-14 13:24:05 -04:00
James Rodewig e8a907e34a
[DOCS] EQL: Add threat detection example (#59105) 2020-08-14 13:00:34 -04:00
James Rodewig fcc3b6c80c [DOCS] Fix EQL required fields language 2020-08-12 09:47:37 -04:00
James Rodewig 7c494434d6 [DOCS] Remove unneeded word in EQL docs 2020-08-11 12:18:46 -04:00
James Rodewig d09a6cfc7c
[DOCS] Make EQL example snippets more realistic (#60971) 2020-08-11 11:38:46 -04:00
James Rodewig dca46c29ff
[DOCS] Refactor EQL docs (#60700)
Changes:

* Moves sample data to reusable rest test
* Combines EQL index, requirements, and run a search pages
* Combines EQL syntax and limitations pages
* Adds related redirects
2020-08-05 10:11:02 -04:00
James Rodewig ae01606785
[DOCS] Replace `twitter` dataset in docs (#60604) 2020-08-03 12:49:56 -04:00
James Rodewig 441c3a21b1
[DOCS] Update my-index examples (#60132)
Changes the following example index names to `my-index-000001` for consistency:

* `my-index`
* `my_index`
* `myindex`
2020-07-27 14:46:39 -04:00
James Rodewig 2774cd6938
[DOCS] Swap `[float]` for `[discrete]` (#60124)
Changes instances of `[float]` in our docs for `[discrete]`.

Asciidoctor prefers the `[discrete]` tag for floating headings:
https://asciidoctor.org/docs/asciidoc-asciidoctor-diffs/#blocks
2020-07-23 11:48:22 -04:00
James Rodewig 861892add4
[DOCS] EQL: Remove collapsible sections from EQL search docs (#59819) 2020-07-20 08:50:19 -04:00
James Rodewig 5be36b41d4
[DOCS] EQL: Update EQL search response format (#59554) 2020-07-15 16:52:32 -04:00
James Rodewig d250f94374
[DOCS] Fix syntax and wording in EQL docs (#59623) 2020-07-15 14:27:02 -04:00
James Rodewig adc520b7c2 [DOCS] Note that EQL timestamp field can also be date_nanos 2020-07-15 09:53:43 -04:00
Costin Leau bccfbcd81f
EQL: Improve retrieval of results (#59552)
Instead of retrieving an entire SearchHit, get just a reference and 
postpone the document retrieval when assembling the final results.
Remove sort information from results to make them consistent.
Move TumblingWindow under the sequence package.

Co-authored-by: James Rodewig <james.rodewig@elastic.co>
2020-07-14 23:26:25 +03:00
James Rodewig 25c6a125c5
[DOCS] EQL: Document `until` keyword support (#59320) 2020-07-13 08:42:27 -04:00
James Rodewig 747e61508a
[DOCS] EQL: Prepare docs for release (#59259)
Changes:

* Swaps the `dev` admonitions for `experimental` admonitions
* Removes `ifdef` statements preventing the docs from appearing in
  released branches
2020-07-13 08:40:38 -04:00
James Rodewig 284ee85efd
[DOCS] Add data streams to EQL search docs (#58611) 2020-07-13 08:38:01 -04:00
Andrei Stefan 6ede6c59ef
Remove search_after and implicit_join_key_field (#59232) 2020-07-09 11:17:37 +03:00
James Rodewig 52bfe9eb9a
[DOCS] EQL: Document `size` limit for pipes (#59085)
Changes:
* Documents the `size` default as `10`.
* Updates `size` param def to note its relation to pipes.
* Updates the `head` and `tail` pipe docs to modify sequences.
* Documents the `fetch_size` parameter.

Relates to #59014 and #59063
2020-07-08 11:52:45 -04:00
James Rodewig c5df35eba1
[DOCS] EQL: Document unsupported var comparison (#58941)
ES EQL queries do not support the comparison of a variable, such as
a field value, to another variable.

This adds a related para and example to the EQL syntax docs.
2020-07-08 08:54:22 -04:00
James Rodewig 7c23933ec7
[DOCS] EQL: Document `maxspan` keyword (#58931) 2020-07-08 08:52:36 -04:00
James Rodewig 2be9db01c8
[DOCS] Replace `datatype` with `data type` (#58972) 2020-07-07 13:52:10 -04:00
DeDe Morton b5e374d958
[DOCS] Change Beats links to refactored getting started docs (#58790) 2020-07-02 17:10:09 -07:00
James Rodewig f18e136400 [DOCS] Fix xref format in async EQL search docs 2020-06-30 09:36:08 -04:00
James Rodewig cc3bd3974f
[DOCS] EQL: Document `head` and `tail` pipes (#58673) 2020-06-30 08:35:37 -04:00
James Rodewig 29da275b0a
[DOCS] EQL: Remove fields from EQL search response (#58667) 2020-06-29 09:19:07 -04:00
Costin Leau d6731d659d Update JSON results in EQL docs 2020-06-27 09:45:50 +03:00
Costin Leau 4521ca3367
EQL: Add Head/Tail pipe support (#58536)
Introduce pipe support, in particular head and tail
(which can also be chained).
2020-06-27 09:08:03 +03:00
James Rodewig d14b7d5399
[DOCS] EQL: Remove references to partial async EQL results (#58548)
Removes references to partial results from the async EQL search docs.
If an EQL search does not complete during the `wait_for_completion_timeout`
timeout period, it returns no results.
2020-06-26 10:27:30 -04:00
James Rodewig 662cf81bbc
[DOCS] Fix EQL search snippet for tiebreaker example (#58545) 2020-06-25 09:23:50 -04:00
James Rodewig 07874ec357
[DOCS] EQL: Document search API's `tiebreaker_field` param (#57935) 2020-06-25 08:44:34 -04:00
James Rodewig 7f5b72741e [DOCS] EQL: Correct EQL search API's `size` param def
The `size` parameter can be used to limit matching events or sequences.
2020-06-10 10:13:18 -04:00
James Rodewig 6d7acd0d94
[DOCS] EQL: Document delete async search API (#57732) 2020-06-05 12:45:09 -04:00
Igor Motov d197a85ee5 Merge remote-tracking branch 'elastic/master' into feature/async-eql 2020-06-04 15:50:40 -04:00
James Rodewig b30cc2b399
[DOCS] EQL: Add `dev` admonition to EQL pages (#57531) (#57534)
Adds the `dev` admonition to EQL features, which are in development
under a feature flag.
2020-06-02 11:04:56 -04:00
James Rodewig 982f168fd8
[DOCS] EQL: Add `dev` admonition to EQL pages (#57531)
Adds the `dev` admonition to EQL features, which are in development
under a feature flag.
2020-06-02 10:47:53 -04:00
James Rodewig 34c4505a2f
[DOCS] EQL: Fix hits param for sequences (#57410) (#57525) 2020-06-02 09:38:21 -04:00
James Rodewig f1b8df93cd
[DOCS] EQL: Fix hits param for sequences (#57410) 2020-06-02 09:22:14 -04:00
Lisa Cawley 8b9293b3bf
[DOCS] Replace docdir attribute with es-repo-dir (#57489) 2020-06-01 15:55:05 -07:00
James Rodewig 78146bbca9
[DOCS] EQL: Document get async EQL search API (#57366) 2020-05-30 08:42:30 -04:00
Igor Motov 39df45e156 Fix EQL doc tests after master merge 2020-05-27 09:19:50 -04:00
Igor Motov a301eab85b Merge remote-tracking branch 'elastic/master' into feature/async-eql 2020-05-27 08:55:02 -04:00
James Rodewig 8a086ba05d [DOCS] EQL: Fix whitespace in EQL snippet 2020-05-19 17:04:20 -04:00
James Rodewig c13c7aa681
[DOCS] EQL: Add sequence example to tutorial (#56965)
Adds an example using the sequence syntax to the 'Run an EQL search'
tutorial.

Supplements other examples added with #56721
2020-05-19 15:59:18 -04:00
James Rodewig 27cab68912 [DOCS] Add leading slashes to EQL API examples 2020-05-19 15:38:04 -04:00
James Rodewig a3b55d477b [DOCS] EQL: Fix merge conflict in search API docs 2020-05-19 12:54:41 -04:00
James Rodewig 0b557e4c93 [DOCS] EQL: Fix API example headings 2020-05-18 16:28:57 -04:00
Igor Motov dd2ac8ea04 Merge remote-tracking branch 'elastic/master' into feature/async-eql 2020-05-15 15:32:55 -04:00
James Rodewig 19699af81e
[DOCS] EQL: Document `case_sensitive` param (#56697) 2020-05-15 09:21:11 -04:00
James Rodewig 7c679614cd [DOCS] EQL: Align comments in `between` fn examples 2020-05-15 09:20:27 -04:00
James Rodewig aa6c4928e8
[DOCS] EQL: Remove references to arrays/multi-value fields (#56772) 2020-05-15 09:08:02 -04:00
James Rodewig 949a2927ed
[DOCS] EQL: Document `number` function (#56770)
Co-authored-by: Ross Wolf <31489089+rw-access@users.noreply.github.com>
2020-05-14 15:22:04 -04:00
James Rodewig aee5618001
[DOCS] EQL: Document async search submits (#56704) 2020-05-14 11:54:15 -04:00
James Rodewig 15431f2447
[DOCS] EQL: Document sequences (#56721)
Co-authored-by: Ross Wolf <31489089+rw-access@users.noreply.github.com>
2020-05-14 11:01:07 -04:00
Igor Motov fcebd4fd02
EQL: Adds an ability to start an asynchronous EQL search (#56631)
Adds support for async searches to eql search API. This commit is limited to
only submitting search API requests and doesn't provide APIs to get results
nor delete the results. These functions will be added in follow up PRs.

Relates to #49638
2020-05-13 09:50:15 -04:00
James Rodewig 918ef65c67 [DOCS] Sort EQL search API params alphabetically 2020-05-12 13:51:53 -04:00
James Rodewig 883bb29152
[DOCS] EQL: Document math functions (#55810)
Documents the following EQL functions:

* `add`
* `divide`
* `module`
* `multiply`
* `subtract`
2020-05-07 08:53:08 -04:00
James Rodewig c7ac7e005c
[DOCS] EQL: Document `concat` function (#56239)
Co-authored-by: Ross Wolf <31489089+rw-access@users.noreply.github.com>
2020-05-05 16:41:59 -04:00
James Rodewig 038c20b256
[DOCS] EQL: Add collapsible sections to EQL tutorial docs (#56235)
Adds collapsible sections to the snippet examples of the EQL tutorial
docs.

Also adds a leading slash to EQL API snippet examples.
2020-05-05 16:29:11 -04:00
James Rodewig 80f503257b
[DOCS] EQL: Add collapsible sections to EQL search API response (#56232)
Add collapsible sections to the response parameter docs
of the EQL search API.

Also clarifies some language regarding documents and
events.
2020-05-05 15:59:19 -04:00
James Rodewig 7156f40d46
[DOCS] EQL: Document `match` function (#56134) 2020-05-05 11:48:40 -04:00
James Rodewig e12419b276
[DOCS] EQL: Document nested field support (#56138)
Notes that you cannot use EQL in ES to search the values of `nested`
fields or their sub-fields. However, indices containing `nested` field
mappings are otherwise supported.
2020-05-05 11:26:20 -04:00
James Rodewig a7729c8e31
[DOCS] EQL: Remove case sensitivity from function docs (#55063)
Per #54411, we plan to handle case sensitivity via a parameter for the
EQL search API (with the possible exception of the `between` function).

This removes references and examples related to case sensitivity from
the EQL functions docs.
2020-05-05 09:25:55 -04:00
James Rodewig 991899ed47
[DOCS] EQL: Add advantages to overview (#53452)
Adds a concise list of EQL advantages, based on the "EQL Advantages"
section in the [EQL for the masses][0] blog post.

The intent is to inform users how EQL could benefit at a high level.

[0]: https://www.elastic.co/blog/eql-for-the-masses

Co-Authored-By: Ross Wolf <31489089+rw-access@users.noreply.github.com>
2020-04-30 12:57:32 -04:00
James Rodewig 6c26c4b768
[DOCS] EQL: Correct `cidrMatch` function heading (#55935) 2020-04-29 10:01:20 -04:00
James Rodewig 8918eefa9e
[DOCS] Update attribute for multi arg footnotes (#55860) 2020-04-29 08:57:21 -04:00
James Rodewig c69eda2d6a [DOCS] EQL: Fix whitespace in `stringContains` docs 2020-04-27 15:52:16 -04:00
James Rodewig cde5fc1ac5
[DOCS] EQL: Document `stringContains` function (#54968) 2020-04-24 14:53:29 -04:00
James Rodewig 32317a6910
[DOCS] Document EQL search REST API (#52384) 2020-04-24 14:25:33 -04:00
James Rodewig d22240443c
[DOCS] EQL: Document `cidrMatch` function (#54216) 2020-04-24 13:34:14 -04:00
James Rodewig b58e95d25c
[DOCS] Add admonition for EQL exact matches on text fields (#53402)
Adds a important admonition to the EQL syntax page noting that
the equal (`==`) operator should not be used to match `text` field
values.

Relates to #52709 and #53020
2020-04-23 09:53:02 -04:00
James Rodewig 881b214619
[DOCS] EQL: Document `indexOf` function (#55071) 2020-04-15 11:28:33 -04:00
James Rodewig 9bb621be9d [DOCS] Use consistent line breaks in EQL function docs 2020-04-14 10:15:49 -04:00
James Rodewig 28ff719787
[DOCS] EQL: Document `string` function (#55086) 2020-04-13 11:23:01 -04:00
James Rodewig fa138ed1e5
[DOCS] EQL: Reword field support for EQL functions (#55074)
Changes boilerplate sentence of "If using a field as the argument, this
parameter only supports..." to "...this parameter supports only...".

The latter is a bit more clear and readable.
2020-04-10 15:31:20 -04:00
James Rodewig 33dc417bd0
[DOCS] EQL: Document `wildcard` function (#54086) 2020-04-10 09:17:41 -04:00
James Rodewig 7aef7b3ebc
[DOCS] EQL: Document `between` function (#54950) 2020-04-08 13:30:50 -04:00
James Rodewig 7738ed40ff
[DOCS] EQL: Document `length` function (#54225) 2020-04-01 11:17:14 -04:00
James Rodewig e86e148ee4
[DOCS] EQL: Document `endsWith` function (#54521) 2020-04-01 10:13:47 -04:00
James Rodewig d614b7f358
[DOCS] EQL: Document `startsWith` function (#54518) 2020-04-01 09:15:53 -04:00
James Rodewig 658a331245
[DOCS] EQL: Add search/index speed tip for functions (#54346)
EQL functions are an easy way for users to transform indexed data
at search time. However, using multiple functions can make
queries difficult to write and slows search speeds.

Users can circumvent this by indexing fields containing the transformed
data, but that usually slows index speeds.

This adds a related tip and example covering these tradeoffs.
2020-04-01 08:21:35 -04:00
James Rodewig f91d8ee3e7
[DOCS] EQL: Document `substring` function (#53867)
Adds documentation for the EQL `substring` function.

Supporting changes:

* Creates a new "EQL function reference" page
* Updates the title of the "EQL syntax reference" page for consistency
* Adds a brief "Functions" section to the EQL syntax docs
* Updates EQL limitations docs to state that only array functions are
  unsupported
2020-03-25 12:07:20 -04:00
James Rodewig 61182c62ad
[DOCS] Adds Beats tip to EQL search docs (#53292)
Adds a tip admonition to the basic example in the EQL search docs.

This tip lets users know they can set up a Beat to automatically
index data in ES, rather than manually indexing using the bulk or index
APIs.
2020-03-10 05:14:55 -04:00
James Rodewig c6334ee4b7
[DOCS] Document `any` keyword in EQL syntax (#52821)
* [DOCS] Document `any` keyword in EQL syntax

Adds documentation for the `any` keyword to the EQL syntax docs.

Includes:

* Definition of an event type and its relationship to the event type
  field.
* Example matching all event types using `any` keyword
* Example matching event types beginning with a digit
* Example using `any` with `where true`

* Remove references to `event_type_field` default

* Reuse "Events starting with digits" section

* Updates for #53073
2020-03-05 04:44:35 -05:00
James Rodewig 9306c8dd5e
[DOCS] Update EQL default event category and timestamp values (#53102)
Updates the documented default `event_category_field` and `timestamp_field`
values for the EQL search API. Also updates related guidance in the
EQL requirement docs.

Relates to #53073.
2020-03-04 09:16:27 -05:00
Aleksandr Maus 82c645e3ef
EQL: consistent naming for event type vs event category (#53073)
Related to https://github.com/elastic/elasticsearch/issues/52941
2020-03-03 19:44:12 -05:00
James Rodewig 427c9a0568 [DOCS] Reorganize EQL requirements page 2020-03-03 07:01:19 -05:00
James Rodewig 4e58ddea0c [DOCS] Fix typo in EQL docs 2020-03-02 16:07:42 -05:00
Costin Leau 798015b7bb
EQL: Add implicit ordering on timestamp (#53004)
QL: Move Sort base class from SQL to QL
2020-03-02 22:40:05 +02:00
James Rodewig 06225353de
[DOCS] Add parameter examples to EQL search tutorial (#52953)
Makes the following updates to the EQL search tutorial:

* Adds an API response to the basic tutorial
* Adds an example using the `event_type_field` parm
* Adds an example using the `timestamp_field`parm
* Adds an example using the `query` parm
* Updates example dataset to support more EQL query variety
2020-03-02 09:30:51 -05:00
Aleksandr Maus d49a9aac53
EQL: Change request parameter query to filter and rule to query (#52971)
* EQL: Change request parameter query to filter and rule to query

Related to https://github.com/elastic/elasticsearch/issues/52911

* Adjust doc test
2020-03-02 08:03:39 -05:00
Costin Leau 3e039282bf
EQL: Hook engine to Elasticsearch (#52828)
Add query execution and return actual results returned from
Elasticsearch inside the tests
2020-02-27 11:16:26 +02:00
James Rodewig 6b62ec53db
[DOCS] Add EQL limitations page (#52001)
Documents limitations for EQL in Elasticsearch.
2020-02-12 08:45:15 -05:00
James Rodewig be8ae972bc
[DOCS] Add basic EQL search tutorial docs (#51574)
I plan to add additional sections to this page with future PRs:

* Specify timestamp and event type fields
* Specify a join key field
* Filter using query DSL
* Paginate a large response

See #51057.
2020-02-12 08:40:10 -05:00
James Rodewig 99c1ee88e8
[DOCS] Add EQL syntax page (#51821)
Adds documentation for basic EQL syntax.

Joins, sequences, and other syntax to be added as its supported
in future development.

Co-Authored-By: Ross Wolf <31489089+rw-access@users.noreply.github.com>
2020-02-05 08:12:09 -05:00
James Rodewig 0a31b67cc5
[DOCS] Add top-level EQL docs page. Adds EQL requirements page. (#51334)
* Creates a top-level page for EQL in the ES reference.
   This page contains a high-level introduction and will include a nav for other EQL docs pages as they're built.

* Creates a requirements page.
  This page outlines the fields needed to use EQL in ES.
2020-01-27 16:03:23 -05:00