Commit Graph

36 Commits

Author SHA1 Message Date
Nik Everett a1305373f2
ESQL: Use generated docs for abs and acos (#106510)
ESQL: Use generated docs for abs and acos
2024-03-20 16:32:12 -04:00
Fang Xing 5d05d81854
[ES|QL] Remove variadic functions' optional args from the output of show functions (#106454)
* remove optional args from the output of show functions optionalArgs, argNames and argTypes for variadic functions
* consistent names for arguments
2024-03-20 10:15:49 -04:00
Craig Taverner e14dd54ae9
Support ST_INTERSECTS between two geometry columns (#104907)
* Support ST_INTERSECTS between geometry column and other geometry or string

* Pushdown to lucene for ST_INTERSECTS on GEO_POINT

* Get geo_shape working in ST_INTERSECTS bypassing SingleValueQuery

* Initial work to support cartesian shape queries in ESQL

* Fixed CSV tests for combined ST_INTERSECTS and ST_CENTROID

* Fixed bug in point-in-shape query for CARTESIAN_POINT

* Added unit tests for SpatialIntersects and fixed a few bugs found

* Added comments to public ShapeQueryBuilder class

* Move calls to random() later to avoid security exception

* Refined type checking support in ST_INTERSECTS

Improved the combinations supported as preparation for removing the uly try/catch way of detecting the difference between WKT and WKB in some code.

* Fixed bugs in incorrect use of doc-values in parameter type matching

Also made a few reminfments, including removing one try/catch approach to differentiating between WKT and WKB.

* Removed second place where we used try/catch to differentiate WKT from WKB

This was a workaround for a mistake in the planning, where we incorrectly mapped incoming types to the wrong FieldEvaluators. We fixed that mistake in an earlier commit.

* Fixed flaky tests were GEO was treated as CARTSIAN

We assumed if the incoming types were constants, they had no CRS, even when they did, which was wrong. For shapes crossing the dateline this lead to different (incorrect) behaviour.

* Fixed a flaky test by removing some point==point optimizations

* Moved spatial intersects to 'spatial' package

When we developed the ST_CENTROID work, this was requested, so let's do it here too.

* Use normal switch on enums

* Cleanup some static utility methods

Now all code paths that can convert a constant string to a geometry use the same code.

* Fixed bugs with non-quantized coordinates, and cleaned up code a little

* Fixed failing test after change to evaluator class names

* Refactored SpatialRelatesFunction into three files, and made evaluatorRules static

This was a general cleanup, making the code more organized, but did also achieve static evaluator rules so we don't re-created these on every query parsing.

* Fixed compile error after rebase

* Removed ConstantAndConstant support, using fold() correctly instead

* better error on circles

* Make sure compound predicates are supported in use-doc-values pushdown

* Testing ENRICH with ST_INTERSECTS

This required adding new data for an ENRICH index, and this data could be tested with a few other related tests, which were also added.

* Added missing mixed-cluster rules for testing only with 8.14

* Fixed some mixed-cluster issues where we failed to mark test for only 8.14

Also added an interesting polygon-polygon intersection case from real data.

* Fix flaky test where cartesian polygons were generated from geo

* Remove support for string literals in ST_INTERSECTS

* Fix failing tests after removing string support

* Removed unused code from previous string literal support (WKT parsing)

* Support case where both fields are points and doc-values

If we have an ST_INTERSECTS and an ST_CENTROID, the centroid asks to load the points as doc-values, and the ST_INTERSECTS needs to therefor support two doc-values points.

* Disallow more than one field from doc-values for ST_INTERSECTS

* Remove unused evaluator classes

* Add tests for multiple doc-values if not in same intersects

* Fix errors after rebase on main

* Fixed bug in missing support for spatial function expressions in EVAL

When a spatial aggregate expects doc-values, this was not being communicated to spatial functions in EVAL, only in WHERE.

* Reduce flaky tests when reading directly from enrich source indices

The test framework does not expect enrich source indices to be used directly in queries, leading to duplicated results on multi-node clusters, so we edit the queries to be less sensitive to this case.

* Fixed failing test

* Code style

* Fixed test file name and added function name annotation

* Added documentation for st_intersects

* Fixed failing show functions test

* Code review changes, notably simplifying the type resolution

* Fixed broken docs link
2024-03-19 17:58:37 +01:00
Fang Xing e6d1c90501
[ES|QL] Add mv_sort (#106095)
* add mv_sort
2024-03-13 12:04:12 -04:00
Fang Xing 8d839e3b52
add mv_slice and mv_zip (#106147) 2024-03-11 23:02:57 -04:00
Craig Taverner 8d93a934f6
Add two new OGC functions ST_X and ST_Y (#105768)
* Add two new OGC functions ST_X and ST_Y

Recently Nik did work that involved extracting the X and Y coordinates from geo_point data using `to_string(field)` followed by a DISSECT command to re-parse the string to get the X and Y coordinates.

This is much more efficiently achieved using existing known OGC functions `ST_X` and `ST_Y`.

* Update docs/changelog/105768.yaml

* Fixed invalid changelog yaml

* Fixed mixed cluster tests

* Fixed tests and added docs

* Removed false impression that these functions were different for geo/cartesian

With the use of WKB as the core type in the compute engine, many spatial functions are actually the same between these two types, so we should not give the impression they are different.

* Code review comments and reduced object creation.

* Revert temporary StringUtils hack, and fix bug in x/y extraction from WKB

* Revert object creation reduction

* Fixed mistakes in documentation
2024-03-06 11:19:54 +01:00
Craig Taverner 85cd204317
Fix automatic generation of spatial function types files (#105766)
* Fix automatic generation of spatial function types files

The automatic mapping of spatial function names from class names was not working for spatial types, so the automatic generation of these files did not happen, and in fact existing files were deleted.

In addition, the generation of aggregation functions types does not yet exist at all, so the st_centroid.asciidoc file was always deleted. Until such support exists, this files contents will be moved back into the function definition file.

The railroad diagrams for syntax are now also created, however, not all functions in the documentation actually use these, and certainly none of the `TO_*` type-casting functions do, so we'll not include links to them from the docs, and leave that to the docs team to decide. Personally, while these diagrams are pretty, they contain no additional informational content, and in fact give a cluttered impression to the documentation visual appeal.

* Refined to use an annotation which is more generic
2024-03-05 18:04:09 +01:00
Fang Xing 0fb5dee75b
[ES|QL] Add function log(base, value) (#104913)
Add a new scalar function log
2024-02-06 13:03:53 -05:00
Luigi Dell'Aquila ad28dc9a6c
ESQL: Add TO_UPPER and TO_LOWER functions (#104309) 2024-01-15 11:58:57 +01:00
Marco Liberati 3145e9f9fc
[ES|QL] Annotate all ESQL functions (for SHOW FUNCTIONS) (#103686)
* 🔧 Initial annotations

* ✏️ Add substring doc

* 🏷️ Fix imports

* ✏️ Add annotations for string fns

* ♻️ Fix issues

* :rotatin_light: Fix linting issues

* ✏️ Add more annotations

* :rotatin_light: Fix linting issues

* ♻️ Add missing import

* ✏️ Add more annotations

* 🚨 Fix linting

*  Fix many tests for new annotations

*  Fix more tests

* 🐛 Fix missing flag

*  Annotate new functions

*  fix more tests

*  Fix signature issue

*  Make all csv tests pass

*  Fix remaining tests

*  New assets from annotations

*  Refactor test

*  Fix updated signature
2024-01-11 10:15:30 +01:00
Mark Tozzi 282f0f0a00
[ESQL] Remove is_nan, is_finite, and is_infinite (#104091)
A while ago we decided we weren't going to support NaN or infinite values in the language. Removing these functions is part of that effort.
2024-01-09 14:51:32 -05:00
Nik Everett 5ef5dca334
ESQL: `MV_FIRST` and `MV_LAST` (#103928)
This creates the `MV_FIRST` and `MV_LAST` functions that return the
first and last values from a multivalue field. They are noops from a
single valued field. They are quite similar to `MV_MIN` and `MV_MAX`
except they work on positional data rather than relative size. That
sounds like a large distinction, but in practice our multivalued fields
are often sorted. And when they operate on sorted arrays `MV_MIN` does
*the same* thing as `MV_FIRST`.

But there are some cases where it really does matter - say you are
`SPLIT`ing something - so `MV_FIRST(SPLIT("foo;bar;baz", ";"))` gets you
`foo` like you'd expect. No sorting needed.

Relates to #103879
2024-01-09 08:46:34 -05:00
Luigi Dell'Aquila 770fc19b14
ESQL: add date_diff function (#104118)
Same as https://github.com/elastic/elasticsearch/pull/103208

Fixes #101942

We had to revert it after a Checkstyle failure (strange it didn't pop up
in the CI before merging)
2024-01-09 07:03:58 -05:00
David Turner 943b2eae70 Revert "Esql/create DATE_DIFF function (#103208)"
This reverts commit ec2e18536d.
2024-01-09 09:50:48 +00:00
Nicolas Gras ec2e18536d
Esql/create DATE_DIFF function (#103208) 2024-01-09 10:37:42 +01:00
Bogdan Pintea 4e2389fb2c
ESQL: Improve docs for the floating points is_xxx() funtions (#103691)
Improve the docs for is_nan, is_finite, is_infinite functions.
This also adjusts the CamelCase to snake_case conversion, to not
consider the last capital letter (like in `IsNaN`).
2024-01-03 12:18:15 +01:00
Nik Everett 16e9bbe2d3 ESQL: Update generated docs 2023-12-27 15:58:02 -05:00
Nik Everett 3b738905e1
ESQL: Add type tables for operators to docs (#103206)
This adds a tiny blurb for each operator to the docs with a railroad
diagram of the operator's syntax and a table of the input and output
types. This also fixes the tests to correctly generate the tables for
operators.
2023-12-11 10:51:38 -05:00
Nik Everett 6e0c031342
ESQL: Generate railroad diagrams for operators (#103143)
This enables the generation of railroad diagrams for unary minus and a
bunch of binary operators like `+`, `-`, `%`, and `>=`.

Relates to #100558
2023-12-08 09:24:34 -05:00
Bogdan Pintea d3fefde0a3
ESQL: add unit tests for conversion functions. Extend `TEXT` type support for them (#102746)
This adds the missing unit tests for the conversion functions.
It also extends the type support by adding the `TEXT` type to those functions that support `KEYWORD` already (which also simplifies the testing, actually). Some functions did have it, some didn't; they now all do.

The change also fixes two defects resulting from better testing coverage: `ToInteger` and `ToUnsignedLong` had some missing necessary exceptions declarations in the decorators for the evaluators.
It also updates `ToInteger`'s `fromDouble()` conversion to use a newly added utility, so that the failed conversions contain the right message (`out of [integer] range`, instead of the confusing `out of [long] range`).

Related: #102488, #102552.
2023-12-04 13:58:23 +01:00
Nik Everett e39fdba02e ESQL: Add newly generated signature
We're generating it now, we should commit it so the docs can have it.
2023-11-27 13:29:03 -05:00
Nhat Nguyen 774a05cc0a
Update doc fields for pow and date_extract function (#102554)
These two generated files have not been updated and committed.
2023-11-23 17:33:38 -08:00
Nik Everett d46c386c86
ESQL: Extra tests for trim/ltrim/rtrim (#102308)
This adds some extra tests, docs, and descriptions for the `trim`,
`ltrim`, and `rtrim` functions.
2023-11-16 14:44:56 -05:00
Luigi Dell'Aquila d53c0cbf82
ESQL: annotate trigonometric functions and auto_bucket() (SHOW FUNCTIONS) (#101460) 2023-10-27 16:35:44 +02:00
Nik Everett 46f95a67b4
ESQL: More MV_* tests (#100564)
This adds more tests for some of the `MV_` functions and updates their
docs now that the railroad diagram and table generated by the tests
covers all of the types.
2023-10-24 16:55:17 -04:00
Nik Everett 9620512a89
ESQL: Tests for large concat and many evals (#100159) 2023-10-03 14:41:40 -04:00
AlexB 2ccdae6745
Eval REPLACE function (#98909)
Co-authored-by: Alexandros Batsakis <abatsakis@splunk.com>
Co-authored-by: Andrei Stefan <andrei@elastic.co>
2023-09-29 17:41:20 +03:00
Nik Everett 5e3ab06151
ESQL: Prevent `CONCAT` from using a ton of memory (#99716)
This prevents `CONCAT` from using an unbounded amount of memory by
hooking it's temporary value into the circuit breaker. To do so, it
makes *all* `ExpressionEvaluator`s `Releasable`. Most of the changes in
this PR just plumb that through to every evaluator. The rest of the
changes correctly release evaluators after their use.

I considered another tactic but didn't like it as much, even though the
number of changes would be smaller - I could have created a fresh,
`Releasable` temporary value for every `Page`. It would be pretty
contained keep the releasable there. But I wanted to share the temporary
state across runs to avoid a bunch of allocations.

Here's a script that used to crash before this PR but is fine after:
```
curl -uelastic:password -XDELETE localhost:9200/test
curl -HContent-Type:application/json -uelastic:password -XPUT localhost:9200/test -d'{
   "mappings": {
      "properties": {
         "short": {
            "type": "keyword"
         }
      }
   }
}'
curl -HContent-Type:application/json -uelastic:password -XPUT localhost:9200/test/_doc/1?refresh -d'{"short": "short"}'

echo -n '{"query": "FROM test ' > /tmp/evil
for i in {0..9}; do
   echo -n '| EVAL short = CONCAT(short' >> /tmp/evil
   for j in {1..9}; do
      echo -n ', short' >> /tmp/evil
   done
   echo -n ')' >> /tmp/evil
done
echo '| EVAL len = LENGTH(short) | KEEP len"}'>> /tmp/evil
curl -HContent-Type:application/json -uelastic:password -XPOST localhost:9200/_query?pretty --data-binary @/tmp/evil
```
2023-09-22 11:27:13 -04:00
gheorghepucea d58b9ea87d
Added esql ends_with implementation (#99613)
Added an implementation for `ends_with` function in esql.  `ends_with` -
Returns a boolean that indicates whether a keyword string ends with
another string. Also made sure that the docs look alright: 

<img width="1677" alt="Screenshot 2023-09-16 at 18 10 46"
src="https://github.com/elastic/elasticsearch/assets/91881042/eccd81e1-40a2-4a66-a514-cf3e4205f9da">
2023-09-18 11:29:20 -04:00
Nik Everett 936e69ddd5
ESQL: Yet more function tests and docs (#99009)
This adds tests, supported types, and a signature image for `to_string`
and `to_version`. It also fixes the resolution of functions who's names
contain an `_`

Finally, it updates the docs for `ceil` to render the image more nicely.
2023-09-11 14:10:17 -04:00
dreamquster 04381664c1
ESQL: Implement 'right' function (#98974)
Add the 'right' function, which extracts a substring beginning from its
right end (opposite function of 'left').
---------

Co-authored-by: Alexander Spies <alexander.spies@elastic.co>
2023-09-08 17:27:59 +02:00
Nik Everett b73cc0c529
ESQL: Only generate syntax diagrams locally (#99059)
CI will skip building them. Lot's of CI machines don't have font support
so they can't generate these. But all local machine have a GUI so they
can.

Also, super-lazy initialize the font so CI don't bump into it by
accident.

Closes #99018
2023-08-30 14:44:14 -04:00
Nik Everett 4cd7f40712
ESQL: More docs (#98890)
Adds some more typed docs to the ESQL functions.
2023-08-28 11:17:04 -04:00
Alexander Spies ca3dc3a882
ESQL: Add `CEIL` function (#98847)
Add the unary scalar function CEIL.

Analogously to FLOOR,  it rounds up its argument.

- Implement CEIL, add it to the function registry and make sure it is serializable.
- Add csv tests, unit tests and docs.
- Add additional csv tests with different data types and some edge cases for both CEIL and FLOOR
- Add unit tests and update docs for FLOOR.
2023-08-28 12:31:56 +02:00
Nik Everett ff01fb680b
ESQL: Standardize font used in railroad diagrams (#98897)
Locks the railroad diagrams to always use the same font, this one named
`roboto mono`. This makes sure that when we render the railroad diagrams
we always size them the same way. Because everyone has a copy of roboto
mono. Because gradle resolves that dependency.
2023-08-26 14:19:47 -04:00
Nik Everett 649ceb74ab
ESQL docs: generate references for functions (#98856)
This generates a "railroad diagram" svg image that can be embedded into
the docs for any function to explain it's syntax. It's basic, but it's
something we can iterate on.

It also generates a table of supported types from the list of types that
we test. It can be included in the docs for reference as well.
2023-08-25 09:07:25 -04:00