Commit Graph

294 Commits

Author SHA1 Message Date
Sylvain Wallez e78bdc953a
ESQL: add Arrow dataframes output format (#109873)
Initial support for Apache Arrow's streaming format as a response for ES|QL. It triggers based on the Accept header or the format request parameter.

Arrow has implementations in every mainstream language and is a backend of the Python Pandas library, which is extremely popular among data scientists and data analysts. Arrow's streaming format has also become the de facto standard for dataframe interchange. It is an efficient binary format that allows zero-cost deserialization by adding data access wrappers on top of memory buffers received from the network.

This PR builds on the experiment made by @nik9000 in PR #104877

Features/limitations:
- all ES|QL data types are supported
- multi-valued fields are not supported
- fields of type _source are output as JSON text in a varchar array. In a future iteration we may want to offer the choice of the more efficient CBOR and SMILE formats.

Technical details:

Arrow comes with its own memory management to handle vectors with direct memory, reference counting, etc. We don't want to use this as it conflicts with Elasticsearch's own memory management.

We therefore use the Arrow library only for the metadata objects describing the dataframe schema and the structure of the streaming format. The Arrow vector data is produced directly from ES|QL blocks.

---------

Co-authored-by: Nik Everett <nik9000@gmail.com>
2024-07-03 10:29:57 +02:00
Fang Xing 8abc8857f2
[ES|QL] weighted_avg (#109993)
* weighted_avg
2024-07-02 18:29:02 -04:00
Nik Everett 6fbc52d170
ESQL docs: Push down needs index and doc_values (#110353)
This adds a `NOTE` to each comparison saying that pushing the comparison
to the search index requires that the field have an `index` and
`doc_values`. This is unique compared to the rest of Elasticsearch which
only requires an `index` and it's caused by our insistence that
comparisons only return true for single-valued fields. We can in future
accelerate comparisons without `doc_values`, but we just haven't written
that code yet.
2024-07-02 14:22:50 -04:00
Iván Cea Fontenla c89ee3b648
ESQL: Renamed TopList to Top (#110347)
Rename TopList aggregation to Top, after internal discussions
2024-07-02 03:52:24 +10:00
Costin Leau b906ce3d66
ESQL: change from quoting from backtick to quote (#108395)
* ESQL: change from quoting from backtick to quote

For historical reasons, the source declaration inside FROM command is
 treated as an identifier, using backticks (`) for escaping the value.
This is inconsistent since the source is not an identifier (field name)
 but an index name which has different semantics.
 `index` means a field name index while "index" means a literal with
 said value.

In case of FROM, the index name/location is more like a literal (also in
 unquoted form) than an identifier (that is a reference to a value).

This PR tweaks the grammar and plugs in the quoted string logic so that
 both the single quote (") and triple quote (""") are allowed.

* Update grammar

* Add more tests

* Add a few more tests

* Add extra test

* Update docs/changelog/108395.yaml

* Adress review comments

* Add doc note

* Revert test rename

* Fix quoting with remote cluster

* Update docs/reference/esql/source-commands/from.asciidoc

Co-authored-by: marciw <333176+marciw@users.noreply.github.com>

---------

Co-authored-by: Bogdan Pintea <bogdan.pintea@elastic.co>
Co-authored-by: Bogdan Pintea <pintea@mailbox.org>
Co-authored-by: marciw <333176+marciw@users.noreply.github.com>
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
2024-06-30 20:01:31 +03:00
Iván Cea Fontenla fc0313f429
ESQL: Add aggregations testing base and docs (#110042)
- Added a new `AbstractAggregationTestCase` base class for tests, that shares most of the code of function tests, adapted for aggregations. Including both testing and docs generation.
  - Reused the `AbstractFunctionTestCase` class to also let us test evaluators if the aggregation is foldable
- Added a `TopListTests` example
  - This includes the docs for Top_list _(Also added a missing include of Ip_prefix docs)_
- Adapted Kibana docs to use `type: "agg"` (@drewdaemon)

The current tests are very basic: Consume a page, generate an output,
all in Single aggregation mode (No intermediates, no grouping). More
complex testing will be added in future PRs

Initial PR of https://github.com/elastic/elasticsearch/issues/109917
2024-06-27 21:21:55 +10:00
Craig Taverner 536d614694
ES|QL ST_DISTANCE Function (#108764)
* WIP Started refactoring in preparation for ST_DISTANCE

* Initial evaluators for ST_DISTANCE

* Update docs/changelog/108764.yaml

* Fix invalid changelog generated by CI

* Register function and get unit tests working

* Fixed failing meta function description tests, and refined descriptions

* Added initial CsvTests and calculate Geo differently to Cartesian

* Added more csv-spec tests and changed to arcDistance for accuracy

* Added generated docs files

* Link to generated docs

* Fix examples tag for linking from generated docs

* Skip wrapper function

And note that we might want to include instead some of the related intelligence from Circle2D::HaversineDistance class

* Added ST_DWITHIN and more tests for ST_DISTANCE and ST_DWITHIN

* Code style

* Added more tests, this time for sorting on distance

* Fixes after rebase on main

* The ST_DWITHIN cannot use BinarySpatialFunction because it is ternary

So we moved the common code to a separate SpatialTypeResolver, and made a simpler TernarySpatialFunction based on a simple TernaryScalarFunction. This had additional consequences, simplifying the points-only cases.

The main reason for this change was to support StDWithinTests which need to test a lot of things that involve varying all three input types, generating expected error strings, etc. The original hack of just adding to BinarySpatialFunction worked for the actual integration tests, but clearly did not satisfy all the use cases tested by the unit tests.

We also restricted ST_DWITHIN to take only a double as the third argument, because otherwise the number of evaluators would explode, since we need a separate evaluator for each Block type, and Integer and Double use different block types.

* Fixed function count after rebasing on main

* Update docs/changelog/108764.yaml

* Added generated docs for ST_DWITHIN

* Connect docs for ST_DWITHIN

* Add back issue link

* Remove support for ST_DWITHIN

* Update docs/changelog/108764.yaml

* Bring back link to issue in changelog

* Update x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/expression/function/scalar/spatial/StDistance.java

Co-authored-by: Ignacio Vera <iverase@gmail.com>

* Revert reformatting of function descriptions

We should put this into a separate PR

* Github merged commit with incorrectly formatted whitespace

---------

Co-authored-by: Ignacio Vera <iverase@gmail.com>
2024-06-21 11:59:44 +02:00
Nik Everett b35f0ed48d
ESQL: Make a table of all inline casts (#109713)
This adds a test that generates
`docs/reference/esql/functions/kibana/inline_cast.json` which is a json
object who's keys are the names of valid inline casts and who's values
are the resulting data types.

I also moved one of the maps we use to make the inline casts to
`DataType`, which is a place where we want it.
2024-06-18 06:23:11 -04:00
Nik Everett 2aade9dd66
ESQL: Warn about division (#109716)
When you divide two integers or two longs we round towards 0. Like
Postgres or Java or Rust or C. Other systems, like MySQL or SPL or
Javascript or Python always produce a floating point number. We should
warn folks about this. It's genuinely unexpected for some folks. OTOH,
converting into a floating point number would be unexpected for other
folks. Oh well, let's document what we've got.
2024-06-14 08:36:27 -04:00
Luigi Dell'Aquila 47edae4fbd
ES|QL: reduce memory footprint for MvAppendTests with shapes (#109517)
Fixing MvAppendTests CB exceptions by generating smaller geometries: the
test generates a lot of documents and the CB is too small for multiple
big shapes.

Fixes https://github.com/elastic/elasticsearch/issues/109409
2024-06-13 02:44:49 +10:00
Liam Thompson 394d2b09a6
Revert "[DOCS] Remove ESQL demo env link from 8.14+ (#109562)" (#109579)
This reverts commit 0480c1acba.
2024-06-11 17:04:37 +02:00
Nik Everett c888e5f4cd
ESQL: Run LOOKUP docs test only in SNAPSHOT (#109493)
LOOKUP is only registered on SNAPSHOT builds.

closes #109478
2024-06-11 23:27:22 +10:00
Nik Everett c6fe3c3efe
ESQL: Improve syntax for LOOKUP tables (#109489)
Replace the syntax for `tables` with something a little more natural.

Now it is:

```
$ curl -uelastic:password -HContent-Type:application/json -XPOST \
    'localhost:9200/_query?error_trace&pretty&format=txt' \
-d'{
    "query": "ROW a=1::LONG | LOOKUP t ON a",
    "tables": {
        "t": {
            "a": {"long":     [    1,     4,     2]},
            "v1": {"integer": [   10,    11,    12]},
            "v2": {"keyword": ["cat", "dog", "wow"]}
        }
    }
}'
      v1       |      v2       |       a
---------------+---------------+---------------
10             |cat            |1
```
2024-06-11 23:26:04 +10:00
Liam Thompson 0480c1acba
[DOCS] Remove ESQL demo env link from 8.14+ (#109562) 2024-06-11 11:24:52 +02:00
Luigi Dell'Aquila 3d0c65d0c5
ES|QL: add tests for COALESCE() function on VERSION type (#109468) 2024-06-07 18:01:42 +02:00
Nik Everett 7916e6a231
ESQL: Implement LOOKUP, an "inline" enrich (#107987)
This adds support for `LOOKUP`, a command that implements a sort of
inline `ENRICH`, using data that is passed in the request:

```
$ curl -uelastic:password -HContent-Type:application/json -XPOST \
    'localhost:9200/_query?error_trace&pretty&format=txt' \
-d'{
    "query": "ROW a=1::LONG | LOOKUP t ON a",
    "tables": {
        "t": {
            "a:long":     [    1,     4,     2],
            "v1:integer": [   10,    11,    12],
            "v2:keyword": ["cat", "dog", "wow"]
        }
    },
    "version": "2024.04.01"
}'
      v1       |      v2       |       a       
---------------+---------------+---------------
10             |cat            |1
```

This required these PRs: * #107624 * #107634 * #107701 * #107762 *
#107923 * #107894 * #107982 * #108012 * #108020 * #108169 * #108191 *
#108334 * #108482 * #108696 * #109040 * #109045

Closes #107306
2024-06-07 11:38:51 +10:00
Parker Timmins bb3ff8e924
ESQL: add REPEAT string function (#109220)
Add support for the string manipulation function REPEAT(string, number). This function concatenates the string argument with itself the specified number of times. If number is 0 an empty string is returned. If number is less than 0, null is returned and a warning is logged. If number is less than 0 and is a constant, the query will fail without executing.
2024-06-04 16:32:43 -05:00
Luigi Dell'Aquila 5f6e8f687b
ES|QL: add MV_APPEND function (#107001)
Adding `MV_APPEND(value1, value2)` function, that appends two values
creating a single multi-value. If one or both the inputs are
multi-values, the result is the concatenation of all the values, eg.

```
MV_APPEND([a, b], [c, d]) -> [a, b, c, d]
```

~I think for this specific case it makes sense to consider `null` values
as empty arrays, so that~ ~MV_APPEND(value, null) -> value~ ~It is
pretty uncommon for ESQL (all the other functions, apart from
`COALESCE`, short-circuit to `null` when one of the values is null), so
let's discuss this behavior.~

[EDIT] considering the feedback from Andrei, I changed this logic and
made it consistent with the other functions: now if one of the
parameters is null, the function returns null
2024-06-05 03:42:29 +10:00
Luigi Dell'Aquila 21952c7e36
ES|QL: add geo tests for mv_dedupe (#109342)
Adding more unit tests for MV_DEDUPE function, covering geo_point,
geo_shape, cartesian_point and cartesian_shape. This also adds docs for
Kibana.

Fixes https://github.com/elastic/elasticsearch/issues/108982
2024-06-05 03:33:14 +10:00
Liam Thompson 2268e383e8
[DOCS][ESQL][8.14] Add API key based security model info for ESQL CCS (#109155)
Co-authored-by: Jake Landis <jake.landis@elastic.co>
2024-06-03 18:44:33 +02:00
Iván Cea Fontenla f16f71e2a2
ESQL: Add ip_prefix function (#109070)
Added ESQL function to get the prefix of an IP. It works now with both
IPv4 and IPv6. For users planning to use it with mixed IPs, we may need
to add a function like "is_ipv4()" first.

**About the skipped test:** There's currently a "bug" in the
evaluators//functions that return null. Evaluators can't handle them.
We'll work on support for that in another PR. It affects other
functions, like `substring()`. In this function, however, it only
affects in "wrong" cases (Like an invalid prefix), so it has no impact.

Fixes https://github.com/elastic/elasticsearch/issues/99064
2024-05-29 10:23:45 -04:00
Luigi Dell'Aquila a5b1848c14
ES|QL: more tests for coalesce() function (#109032)
Adding more unit tests for `coalesce()` function, in particular adding
tests for `ip`, `date` and spatial data types.

This also generates the right signatures for Kibana.

Related to https://github.com/elastic/elasticsearch/issues/108982
2024-05-27 04:36:06 -04:00
Alexander Spies c5ac06a70c
Remove esql version from docs (#108933)
Follow-up to https://github.com/elastic/elasticsearch/pull/108919. The
latter needs to be merged first to have the docs tests pass, as it makes
the version parameter optional in requests.
2024-05-23 10:36:15 -04:00
Iván Cea Fontenla 212fc05808
Reapply "ESQL: Expose "_ignored" metadata field" (#108864) (#108871)
Expose "_ignored" metadata field in ESQL queries.

This is the same code merged here:
https://github.com/elastic/elasticsearch/pull/108770 Which got reverted
here: https://github.com/elastic/elasticsearch/pull/108864

It was reverted because of a test failure:
https://gradle-enterprise.elastic.co/s/dpi2eib2x2fj2
2024-05-22 07:06:04 -04:00
Alexander Spies 16a5d248b7
ESQL: Clone ql for esql (#108773)
Part of https://github.com/elastic/elasticsearch/issues/106679

* Copy the `ql` project into a different project _just for esql_, call it `esql-core`.
* Make `esql` depend only on the latter.
* Fix `EsqlNodeSubclassTests`; I'm confused why this didn't bite us earlier.
* Update the warning regexes in some csv tests as the exceptions have other package names now.

**Note to reviewers:** Exclude the first commit when viewing the diff,
as that contains only the actual copying of `ql`. The remaining commits
are the actually meaningful ones. _The `build.gradle` files probably
require the most attention._
2024-05-22 04:35:17 -04:00
Iván Cea Fontenla 04f1fcee51
Revert "ESQL: Expose "_ignored" metadata field" (#108864)
Reverting after tests failure:
https://gradle-enterprise.elastic.co/s/dpi2eib2x2fj2

Reverts elastic/elasticsearch#108770
2024-05-21 11:22:27 -04:00
Iván Cea Fontenla 47370a15e6
ESQL: Expose "_ignored" metadata field (#108770)
Expose "_ignored" metadata field in ESQL queries
2024-05-21 07:32:32 -04:00
Liam Thompson 4e04bf19ce
[DOCS] Update Using ESQL in Kibana doc (#108715)
* [DOCS] Update Using ESQL in Kibana doc

* Add dashboard panel instructions,screenshots

* Add query history info, images

* Update enabling ESQL info
2024-05-17 12:36:04 +02:00
Bogdan Pintea 3b21ec8818
ESQL: Remove OPTIONS clause in FROM command (#108692)
This remove the `OPTIONS` clause of the `FROM` command.
2024-05-15 18:15:02 -04:00
Iván Cea Fontenla 62b372b4dc
ESQL: CBRT function (#108574)
- Added the cube root function to ESQL (`CBRT(x)`). Nearly identical to SQRT, but without the negative numbers exception
- Added docs generation support for Windows end lines (CRLF), as within the examples, it was writing the "\r" without the "\n" (Which was being converted to "\\n"), and some other inconsistencies
- Some updates to `package-info.java` documentation over how to create functions
- Fixes https://github.com/elastic/elasticsearch/issues/108675

Functions issue: https://github.com/elastic/elasticsearch/issues/98545
2024-05-15 16:50:15 +02:00
Fang Xing 172c05918c
[DOCS] ES|QL implicit casting (#108618)
* implicit casting doc
2024-05-15 09:07:09 -04:00
Fang Xing 11de886346
[ES|QL] Add/Modify annotations for spatial and conditional functions for better doc generation (#107722)
* annotation for spatial functions and conditional functions
2024-05-10 14:49:25 -04:00
Luigi Dell'Aquila fed808850d
ES|QL: Add unit tests for now() function (#108498) 2024-05-10 14:28:19 +02:00
Nik Everett 5a612d4100
ESQL: Remove remaining IT_tests_only (#108434)
This moves examples from files marked to run in integration tests only
to the files where they belong and disables this pattern matching. We
now use supported features.
2024-05-09 09:32:46 -04:00
Bogdan Pintea 8864058f83
ESQL: Add more time span units (#108300)
This adds `nanosecond`, `microsecond` and `quarter` to the set of
supported time spans. It also adds a few standard and common
abbreviations to some existing ones.
2024-05-08 08:51:02 -04:00
Liam Thompson 9b7e9b5d59
[DOCS] ESQL goes GA (#108342) 2024-05-07 14:12:50 +02:00
Bogdan Pintea de725aef80
Add docs clarifications on DATE_DIFF args (#108301)
This adds some clarifications on the time unit strings the function
takes as arguments, noting the differences between these and the time
span literals, as well as the abbreviations' source.
2024-05-07 12:59:01 +02:00
Nik Everett 089fd7d7da
ESQL: Rework integration-only csv testing (#108313)
This reworks the integration-test-only csv testing for `metadata` to use
the `required_feature:` syntax instead of the `-IT_tests_only`
extension. This is a little more flexible and way nicer on the eyes.
2024-05-06 11:06:50 -04:00
Bogdan Pintea b26d7d3e14
Introduce an IP functions group (#108304)
This takes the CIDR_MATCH out of the operators group and adds it to a
new `IP functions` group.
The change also re-aranges the groups, grouping together the
type-specific functions and ordering them alphabetically.
2024-05-06 13:43:30 +02:00
Fang Xing 4daac77e3b
[ES|QL] Add/Modify annotations for operators for better doc generation (#108220)
* annotation for operators
2024-05-03 22:59:51 -04:00
Bogdan Pintea 5f4ef87c47
Fix docs generation of signatures for variadic functions (#107865)
This fixes the generation of the signatures for variadic functions,
except for those that take a list as last argument; i.e.  functions with
optional arguments (like ROUND) or functions with overloading-like
signatures (like BUCKET).
2024-05-03 15:37:22 +02:00
Fang Xing 7ae08306a0
mv functions (#107839)
Add annotations for MV functions for better doc generation.
2024-05-01 10:47:22 -04:00
Bogdan Pintea 4b5c5e2ded
Update BUCKET docs in source (#108005)
This applies a review proposed changes to the source, so that they're
synchronized to the generated output.
2024-04-29 14:27:20 +02:00
Nhat Nguyen 22aad7b201
Support metrics counter types in ESQL (#107877)
This commit adds support for numeric metrics counter fields in ES|QL. 
These counter types, including counter_long, counter_integer, and
counter_double, are different from their parent types. Users will have
limited interaction with these counter types, restricted to:

- Retrieving values without any processing
- Casting to their root type (e.g., to_long(a_long_counter))
- Using them in the metrics rate aggregation

These restrictions are intentional to prevent misuse. If users want to 
use them as numeric values, explicit casting to their root types is
required.
2024-04-26 12:15:48 -07:00
Bogdan Pintea a21242054b
ESQL: Document BUCKET as a grouping function (#107864)
This adds the documentation for BUCKET as a grouping function and the
addition of the "direct" invocation mode providing a span (in addition
to the auto mode).
2024-04-25 12:38:12 -04:00
Bogdan Pintea 7af45cc52e
ESQL: Document the cast operator (::) (#107871)
This documents the cast operator, `::`.
2024-04-25 10:10:59 -04:00
Bogdan Pintea 31f2fb85df
Docs: move STARTS/ENDS_WITH under string functions in the docs (#107867)
This moves the STARTS_WITH and ENDS_with under the strings functions
section (as they're not operators).
2024-04-25 09:41:11 -04:00
Bogdan Pintea 9482673fbe
Docs: move base64 functions under string functions (#107866)
This moves the TO_BASE64 and FROM_BASE64 from the type conversion
functions under string functions (they take a string as input and output
another string).
2024-04-25 13:57:45 +02:00
Fang Xing ad15d50863
[ES|QL] more doc generation via annotations (#107541)
Annotations for math functions, datetime functions, string functions, type conversion functions.
2024-04-22 14:43:36 -04:00
Mark Tozzi f620961812
[ESQL] Add in the autogenerated docs for a bunch of functions (#107633) 2024-04-18 14:09:30 -04:00