This moves the TO_BASE64 and FROM_BASE64 from the type conversion
functions under string functions (they take a string as input and output
another string).
* Remove `es-test-dir` book-scoped variable
* Remove `plugins-examples-dir` book-scoped variable
* Remove `:dependencies-dir:` and `:xes-repo-dir:` book-scoped variables
- In `index.asciidoc`, two variables (`:dependencies-dir:` and `:xes-repo-dir:`) were removed.
- In `sql/index.asciidoc`, the `:sql-tests:` path was updated to fuller path
- In `esql/index.asciidoc`, the `:esql-tests:` path was updated idem
* Replace `es-repo-dir` with `es-ref-dir`
* Move `:include-xpack: true` to few files that use it, remove from index.asciidoc
This extends `BUCKET` function to accept a two-parameters-only
invocation: the first parameter remains as is, while the second is a
span. It can be a numeric (floating point) span, if the first argument
is numeric, or a date period or time duration, if the first argument is
a date.
Also, the function can now be invoked with the alias BIN.
Additionally, the function has been turned into a grouping-only function
and thus can only be used within a `STATS` command.
This improves the tests and docs for a few functions, specifically `E`,
`FLOOR`, `PI`, `POW`, and `ROUND`. The examples and tested signatures
will get copied into the docs and kibana signatures.
Co-authored-by: Liam Thompson <32779855+leemthompo@users.noreply.github.com>
This renames the function AUTO_BUCKET to just BUCKET.
It also removes the experimental tagging of the function in the docs, making it generally available.
This takes a stab at generating the markdown files that Kibana uses for
its inline help. It doesn't include all of the examples because the
`@Example` annotation is not filled in - we're tracking that in
https://github.com/elastic/elasticsearch/issues/104247#issuecomment-2018944371
There are some links in the output and they are in markdown syntax. We
should figure out how to make them work for kibana.
This improves the tests for AUTO_BUCKET marginally, specifically so that
it tests all valid combinations of arguments and generates a correct
types table. This'll combine nicely with #106782 to generate the
signatures that kibana needs for it's editor.
* WIP Started developing ST_DISJOINT
Initially based on ST_INTERSECTS
* Fix functions list and add spatial point integration tests
* Update docs/changelog/107007.yaml
* More tests for shapes and cartesian-multigeoms
* Some more tests to highlight issues with DISJOINT on cartesian point indices
* Disable Lucene push-down for DISJOINT on cartesian point indices
* Added docs for ST_DISJOINT
* Support DISJOINT in the lucene-pushdown code for cartesian point indexes
* Re-enable push-to-source for DISJOINT on cartesian_point indices
* Fix docs example
* Try fix internal docs links which are not being rendered
* Fixed disjoint on empty geometry
* Added tests on empty linestring, and changed lucene push-down to exception
In lucene code only LineString can be empty, but in Elasticsearch even that is not allowed, resulting in parsing errors. So we cannot get to this code in the lucene push-down and now throw an error instead. The tests now assert on the warnings.
Note that for any predicate DISJOINT and INTERSECTS alike, the predicate fails, because the parsing error results in null, the function returns null, the predicate interprets this as false, and no documents match. This null-in-null-out rule means that DISJOINT and INTERSECTS give the same answer on invalid geometries.
* Add ES|QL signum function
* Update docs/changelog/106866.yaml
* Skip csv tests for versions older than 8.14
* Reference layout docs file and fix instructions for adding functions
* Break csv specs by param type
* More tests
This merges all of the hand written docs for `LOG` and `LOG10` into the
annotations which updates the `META FUNCTIONS` - now it'll always be the
same as the docs. This also deletes the hand maintained docs and let's
the documentation generation process rebuild it.
* WIP Started adding ST_CONTAINS
* Add generated evaluators
* Reduced warnings and use correct evaluators
* Refactored tests to remove duplicate code, and fixed Contains/multi-components
* Gradle build disallows using getDeclaredField
* Fixed cases where rectangles cross the dateline
* Fixed meta function tests
* Added ST_WITHIN to support inverting ST_CONTAINS
If the ST_CONTAINS is called with the constant on the left, we either have to create a lot more Evaluators to cover that case, or we have to invert it to ST_WITHIN. This inversion was a much easier option.
* Simplify inversion logic
* Add comment on choice of surrogate approach
* Add unit tests and missing fold() function
* Simple code cleanup
* Add integration tests for literals
* Add more integration tests based on actual data
* Generated documentation files
* Add documentation
* Fixed failing function count test
* Add tests that push-to-source works for ST_CONTAINS and ST_WITHIN
* Test more combinations of WITH/CONTAINS and literal on right and left
This also verifies that the re-writing of CONTAINS to WITHIN or vice versa occurs when the literal is on the left.
* test that physical planning also handles doc-values from STATS
* Added more tests for WITHIN/CONTAINS together with CENTROID
This should test the doc-values for points.
* Add cartesian_point tests
* Add cartesian_shape tests
* Disable Lucene-push-down for CARTESIAN data
This is a limitation in Lucene, which we could address as a performance optimization in a future PR, but since it probably requires Lucene changes, it cannot be done in this work.
* Fix doc links
* Added test data and tests for cartesian multi-polygons
Testing INTERSECTS, CONTAINS and WITHIN with multi-polydon fields
* Use required features for spatial points, shapes and centroid
* 8.13.0 is not yet historical version
This needs to be reverted as soon as 8.13.0 is released
* Added st_intersects and st_contains_within 'features'
* Code review updates
* Re-enable lucene push-down
* Added more required_features
* Fix point contains non-point
* Fix point contains point
* Re-enable lucene push-down in tests too
Forgot to change the physical planner unit tests after re-enabling lucene push-down
* Generate automatic docs
* Use generated examples docs
* Generated examples use '-result' prefix (singular)
* Mark spatial functions as preview/experimental
This updates the in-code docs on the trig functions to line up with the
docs, removes the docs, and uses the now mostly identical generated
docs. This means we only need to document these functions in one place -
right next to the code.
This creates the `VALUES` aggregation function which buffers all field
values it receives and emits them as a multivalued field. It can use a
significant amount of memory and will circuit break if it uses too much
memory, but it's really useful for putting together self-join-like
behavior. It sort of functions as a stop-gap measure until we have more
self-join style things.
In the future we'll have spill-to-disk for aggregations and, likely,
some kind of self-join command for aggregations at least so this will be
able to grow beyond memory. But for now, memory it is.
Example:
```
FROM employees
| EVAL first_letter = SUBSTRING(first_name, 0, 1)
| STATS first_name=VALUES(first_name) BY first_letter
| SORT first_letter
;
first_name:keyword | first_letter:keyword
[Anneke, Alejandro, Anoosh, Amabile, Arumugam] | A
[Bezalel, Berni, Bojan, Basil, Brendon, Berhard, Breannda] | B
[Chirstian, Cristinel, Claudi, Charlene] | C
[Duangkaew, Divier, Domenick, Danel] | D
```
I made this work for everything but `geo_point` and `cartesian_point`
because I'm not 100% sure how to integrate with those. We can grab those
in a follow up.
Closes#103600
This modifies the ESQL test infrastructure to generate more of the
documentation for functions. It generates the *Description* section, the
*Examples* section, and the *Parameters* section as separate files so we
can use them as needed. It also generates a `layout` file that's just
a guess as to how to render the whole thing. In some cases it'll work
and we can use that instead of hand maintaining a "top level"
description file for the function.
Most newly generated files are unused. We have to chose to pick them up
by replacing the sections we were manually maintaining with an include
of the generated section. Or by replacing the entire hand maintained
file with the generated top level file.
Relates to #104247
* Support ST_INTERSECTS between geometry column and other geometry or string
* Pushdown to lucene for ST_INTERSECTS on GEO_POINT
* Get geo_shape working in ST_INTERSECTS bypassing SingleValueQuery
* Initial work to support cartesian shape queries in ESQL
* Fixed CSV tests for combined ST_INTERSECTS and ST_CENTROID
* Fixed bug in point-in-shape query for CARTESIAN_POINT
* Added unit tests for SpatialIntersects and fixed a few bugs found
* Added comments to public ShapeQueryBuilder class
* Move calls to random() later to avoid security exception
* Refined type checking support in ST_INTERSECTS
Improved the combinations supported as preparation for removing the uly try/catch way of detecting the difference between WKT and WKB in some code.
* Fixed bugs in incorrect use of doc-values in parameter type matching
Also made a few reminfments, including removing one try/catch approach to differentiating between WKT and WKB.
* Removed second place where we used try/catch to differentiate WKT from WKB
This was a workaround for a mistake in the planning, where we incorrectly mapped incoming types to the wrong FieldEvaluators. We fixed that mistake in an earlier commit.
* Fixed flaky tests were GEO was treated as CARTSIAN
We assumed if the incoming types were constants, they had no CRS, even when they did, which was wrong. For shapes crossing the dateline this lead to different (incorrect) behaviour.
* Fixed a flaky test by removing some point==point optimizations
* Moved spatial intersects to 'spatial' package
When we developed the ST_CENTROID work, this was requested, so let's do it here too.
* Use normal switch on enums
* Cleanup some static utility methods
Now all code paths that can convert a constant string to a geometry use the same code.
* Fixed bugs with non-quantized coordinates, and cleaned up code a little
* Fixed failing test after change to evaluator class names
* Refactored SpatialRelatesFunction into three files, and made evaluatorRules static
This was a general cleanup, making the code more organized, but did also achieve static evaluator rules so we don't re-created these on every query parsing.
* Fixed compile error after rebase
* Removed ConstantAndConstant support, using fold() correctly instead
* better error on circles
* Make sure compound predicates are supported in use-doc-values pushdown
* Testing ENRICH with ST_INTERSECTS
This required adding new data for an ENRICH index, and this data could be tested with a few other related tests, which were also added.
* Added missing mixed-cluster rules for testing only with 8.14
* Fixed some mixed-cluster issues where we failed to mark test for only 8.14
Also added an interesting polygon-polygon intersection case from real data.
* Fix flaky test where cartesian polygons were generated from geo
* Remove support for string literals in ST_INTERSECTS
* Fix failing tests after removing string support
* Removed unused code from previous string literal support (WKT parsing)
* Support case where both fields are points and doc-values
If we have an ST_INTERSECTS and an ST_CENTROID, the centroid asks to load the points as doc-values, and the ST_INTERSECTS needs to therefor support two doc-values points.
* Disallow more than one field from doc-values for ST_INTERSECTS
* Remove unused evaluator classes
* Add tests for multiple doc-values if not in same intersects
* Fix errors after rebase on main
* Fixed bug in missing support for spatial function expressions in EVAL
When a spatial aggregate expects doc-values, this was not being communicated to spatial functions in EVAL, only in WHERE.
* Reduce flaky tests when reading directly from enrich source indices
The test framework does not expect enrich source indices to be used directly in queries, leading to duplicated results on multi-node clusters, so we edit the queries to be less sensitive to this case.
* Fixed failing test
* Code style
* Fixed test file name and added function name annotation
* Added documentation for st_intersects
* Fixed failing show functions test
* Code review changes, notably simplifying the type resolution
* Fixed broken docs link
* Fix error on sorting unsortable geo_point and cartesian_point
Without a LIMIT the correct error worked, but with LIMIT it did not. This fix mimics the same error with LIMIT and adds tests for all three scenarios:
* Without limit
* With Limit
* From row with limit
* Update docs/changelog/106351.yaml
* Add tests for geo_shape and cartesian_shape also
* Updated changelog
* Separate point and shape error messages
* Move error to later so we get it only if geo field is actually used in sort.
* Implemented planner check in Verifier instead
This is a much better solution.
* Revert previous solution
* Also check non-field attributes so the same error is provided for ROW
* Changed "can't" to "cannot"
* Add unit tests for verifier error
* Added sort limitations to documentation
* Added unit tests for spatial fields in VerifierTests
* Don't run the new yaml tests on older versions
These tests mostly test the validation errors which were changed only in 8.14.0, so should not be tested in earlier versions.
* Simplify check based on code review, skip duplicate forEachDown
* Add two new OGC functions ST_X and ST_Y
Recently Nik did work that involved extracting the X and Y coordinates from geo_point data using `to_string(field)` followed by a DISSECT command to re-parse the string to get the X and Y coordinates.
This is much more efficiently achieved using existing known OGC functions `ST_X` and `ST_Y`.
* Update docs/changelog/105768.yaml
* Fixed invalid changelog yaml
* Fixed mixed cluster tests
* Fixed tests and added docs
* Removed false impression that these functions were different for geo/cartesian
With the use of WKB as the core type in the compute engine, many spatial functions are actually the same between these two types, so we should not give the impression they are different.
* Code review comments and reduced object creation.
* Revert temporary StringUtils hack, and fix bug in x/y extraction from WKB
* Revert object creation reduction
* Fixed mistakes in documentation
* Fix automatic generation of spatial function types files
The automatic mapping of spatial function names from class names was not working for spatial types, so the automatic generation of these files did not happen, and in fact existing files were deleted.
In addition, the generation of aggregation functions types does not yet exist at all, so the st_centroid.asciidoc file was always deleted. Until such support exists, this files contents will be moved back into the function definition file.
The railroad diagrams for syntax are now also created, however, not all functions in the documentation actually use these, and certainly none of the `TO_*` type-casting functions do, so we'll not include links to them from the docs, and leave that to the docs team to decide. Personally, while these diagrams are pretty, they contain no additional informational content, and in fact give a cluttered impression to the documentation visual appeal.
* Refined to use an annotation which is more generic
This adds some docs to the top of `docs.csv-spec` and
`docs-IT_tests_only.csv-spec` telling folks not to add more stuff there
and instead put new examples into whatever files they line up with. It
also shifts some things out of the file to "prime the pump" on cleaning
it up.
* Document nested expressions for stats
* More docs
* Apply suggestions from review
- count-distinct.asciidoc
- Content restructured, moving the section about approximate counts to end of doc.
- count.asciidoc
- Clarified that omitting the `expression` parameter in `COUNT` is equivalent to `COUNT(*)`, which counts the number of rows.
- percentile.asciidoc
- Moved the note about `PERCENTILE` being approximate and non-deterministic to end of doc.
- stats.asciidoc
- Clarified the `STATS` command
- Added a note indicating that individual `null` values are skipped during aggregation
* Comment out mentioning a buggy behavior
* Update sum with inline function example, update test file
* Fix typo
* Delete line
* Simplify wording
* Fix conflict fix typo
---------
Co-authored-by: Liam Thompson <leemthompo@gmail.com>
Co-authored-by: Liam Thompson <32779855+leemthompo@users.noreply.github.com>
* Add initial structure for ST_CENTROID
* Revert "Revert stab at implementing forStats for doc-values vs source"
This reverts commit cfc4341bf4.
* Refined csv-spect tests with st_centroid
* Spotless disagrees with intellij
* Fixes after reverting fieldmapper code to test GeoPointFieldMapper
* Get GeoPointFieldMapperTests working again after enabling doc-values reading
* Simplify after rebase on main
In particular, field-mappers that do not need to know about fields can have simpler calls.
* Support local physical planning of forStats attributes for spatial aggregations
* Get st_centroid aggregation working on doc-values
We changed it to produce BytesRef, so we don't (yet) need any doc-values types.
* Create both DocValues and SourceValues versions of st_centroid
* Support physical planning of DocValues and SourceValues SpatialCentroid
* Improve test for physical planning of DocValues in SpatialCentroid
* Fixed show functions for st_centroid
* More st_centroid tests with mv_expand
To test single and multi-value centroids
* Fix st_centroid from point literals
The blocks contained BytesRef byte[] with multiple values, and we were ignoring the offsets when decoding, so decoding the first value over and over instead of decoding the subsequent values.
* Teach CsvTests to handle spatial types alternative loading from doc-values
Spatial GEO_POINT and CARTESIAN_POINT load from doc-values in some cases. If the physical planner has planned for this, we need the CsvTests to also take that into account, changing the type of the point field from BytesRefBlock to LongBlock.
* Fixed failing NodeSubclassTests
Required making the new constructor public and enabling Set as a valid parameter in the test framework.
* More complex st_centroid tests and fixed bug with multiple aggs
When there were multiple agregations in the same STATS, we were inadvertently re-ordering them, causing the wrong Blocks to be fed to the wrong aggregator in the coordinator node.
* Update docs/changelog/104218.yaml
* Fix automatically generated changelog file
* Fixed failing test
The nodes can now sometimes be Set, which is also a Collection, but not a List, and therefor never can be a subset of the children.
* More tests covering more combinations including MV_EXPAND and grouping
* Added cartesian st_centroid with grouping test
We could not add MV_EXPAND tests since the cartesian data does not have multi-value columns, but the geo_point tests are sufficient for this since they share the same code.
* Reduce flaky tests by sorting results
* Reduce flaky tests by sorting results
* Added tests for stats on stats to ensure planner coped
* Add unit tests to ensure doc-values in query planning complex cases
* Some minor updates from code review
* Fixes after rebase on main
* Get correct error message on unsupported geo_shape for st_centroid
* Refined point vs shape differences after merging main
* Added basic docs
* Delete docs/changelog/104218.yaml
* Revert "Delete docs/changelog/104218.yaml"
This reverts commit 4bc596a442.
* Fixed broken docs tag link
* Simplify BlockReaderSupport in MapperTestCase from code review
* Moved spatial aggregations into a sub-package
* Added some more code review updates, including nested tests
* Get nested functions working, if only from source values for now
* Code review update
* Code review update
* Added second location column to airports for wider testing
* Use second location in tests, including nulls
Includes a test fix for loading and converting nulls to encoded longs.
* Fixed bug supporting multi spatial aggregations in the local node
The local physical planner only marked a single field for stats loading, but marked all spatial aggregations for stats loading, which load to only one aggregation getting the right data, while the rest would get the wrong data.
* Renamed forStats to fieldExtractPreference for clarity
Now the planner decides whether to load data from doc-values. To remove the confusion of preferDocValues==false in the non-spatial cases, we use an ENUM with the default value of NONE, to make it clear we're leaving the choice up to the field type in all non-spatial cases.
* EsqlSpecIT was failing on very high precision centroids on different computers
This was not reproducible on the development machine, but CI machines were sufficiently different to lead to very tiny precision changes over very large Kahan summations. We fixed this by reducing the need for precision checks in clustered integration tests.
* Delete docs/changelog/104218.yaml
* Revert "Delete docs/changelog/104218.yaml"
This reverts commit 12c6980881.
* Fixed changelog entry
Allow nested expressions to be used both for grouping or inside
aggregate functions inside the stats command.
As such the grammar has been tweaked to allow the stats group to have
optional aliasing.
As part of this fix, preserve the original field declaration (including
spaces) for implicit aliases.
Improve validation for incorrect aggregate function use (as arguments,
grouping or inside evals).
Fix#99828
This adds unit tests cases for all the functions that were missing tests
checking on the correct generation of the Warning headers in case the
execution raised an Exception that lead to a `null` result.
This creates the `MV_FIRST` and `MV_LAST` functions that return the
first and last values from a multivalue field. They are noops from a
single valued field. They are quite similar to `MV_MIN` and `MV_MAX`
except they work on positional data rather than relative size. That
sounds like a large distinction, but in practice our multivalued fields
are often sorted. And when they operate on sorted arrays `MV_MIN` does
*the same* thing as `MV_FIRST`.
But there are some cases where it really does matter - say you are
`SPLIT`ing something - so `MV_FIRST(SPLIT("foo;bar;baz", ";"))` gets you
`foo` like you'd expect. No sorting needed.
Relates to #103879
This commit adds detailed API docs for the ES|QL async apis, as well as moving the narrative to the ES|QL specific REST docs. Additionally, a few top-level lists are reflowed to reflect that there are now multiple APIs.
This updates the use of the exceptions subclassed from
`QlServerException` when the failure reason is user-caused. This ensures
that a 400-class response is returned, instead of a 500-class one.
This optimizes loading fields across many, many indices by resolving the
field loading infrastructure when it's first needed rather than up
front. This speeds things up because, if you are loading from many many
shards, you often don't need to set up the field loading infrastructure
for all shards at all - often you'll just need to set it up for a couple
of the shards.
This fixes `null`'s handling in date math. So far the `null` (of type
`NULL`) has been rejected by the type resolution. This is now allowed
through, leading to a `null` result, inline with the other types.
Fixes#103085.
Improve the docs for is_nan, is_finite, is_infinite functions.
This also adjusts the CamelCase to snake_case conversion, to not
consider the last capital letter (like in `IsNaN`).
This commit adds two new endpoints to support ES|QL async querying.
The endpoints added are:
1. GET /_query/async - for initiating a query
2. GET /_query/async/{id} - for retrieving the result
* Start working on geo_point and point docs for ESQL
* Added to_cartesianpoint and includes
* Sub-headings for easier reading
* Improve sub-headings
* Hide to_long and support for longs in to_geopoint and to_cartesianpoint
This adds a tiny blurb for each operator to the docs with a railroad
diagram of the operator's syntax and a table of the input and output
types. This also fixes the tests to correctly generate the tables for
operators.
When encountering a multi-value, a single-value function (i.e. all
non-`mv_xxx()`) returns a `null`. This behaviour is opaque to the user.
This PR adds the functionality for these functions to emit a `Warning`
header, so the user is informed about the cause for the `null`s.
Within testing, there are some differences between the emulated
CSV-based tests (`TestPhysical*`) and the REST CSV-tests and thus the
exact messages in the warnings: * The REST ones can push operations to
Lucene; when this happens, a query containing a negation, `not
<predicate>`, can be translated to a `must_not` query, that will include
the `not` in the `Source`. But outside of Lucene, the execution would
consider the predicate first, then the negation. So when the predicate
contains a SV function, only this part's `Source` will show up in the
warning. * When pushed to Lucene, a query is wrapped within the
`SingleValueQuery`. This emits now warnings when encountering MVs (and
returning no match). However, this only happens once the query that it
wraps returns something itself. Comparatively, the `TestPhysical*`
filters will issue a warning for every encountered MV (irrespective of
sigle values within the MV matching or not).
To differentiate between the slightly differing values of the warnings,
one can now append the `#[Emulated:` prefix to a warning, followed by
the value of the warning for the emulated checks, then a corresponding
`]`. Example: `warning:Line 1:24: evaluation of [not(salary_change <
1)] failed, treating result as null. Only first 20 failures
recorded.#[Emulated:Line 1:28: evaluation of [salary_change < 1] failed,
treating result as null. Only first 20 failures recorded.]`
Closes#98743.
* [DOCS] Add ES|QL 'getting started' code snippets to CSV tests
* Change dots in columns names into underscores
* Add LIMIT 0 to ENRICH test
* Move code snippets out of docs.csv-spec
* Replace code snippets by includes
* Add missing semicolon
This adds the missing unit tests for the conversion functions.
It also extends the type support by adding the `TEXT` type to those functions that support `KEYWORD` already (which also simplifies the testing, actually). Some functions did have it, some didn't; they now all do.
The change also fixes two defects resulting from better testing coverage: `ToInteger` and `ToUnsignedLong` had some missing necessary exceptions declarations in the decorators for the evaluators.
It also updates `ToInteger`'s `fromDouble()` conversion to use a newly added utility, so that the failed conversions contain the right message (`out of [integer] range`, instead of the confusing `out of [long] range`).
Related: #102488, #102552.
This corrects an earlier mistake in the ES|QL language design. Initially we had thought to have pow return the same type as its inputs, but in practice even for integer inputs this quickly grows out of the representable range, and we returned null much of the time. This also created a lot of edge cases around casting to/from doubles (which the underlying java function uses). The version in this PR follows the java spec, by always casting its inputs to doubles, and returning a double. Doing it this way also allows for a rather significant reduction in lines of code.
I removed many of the tests covering pow specific edge cases. This seems reasonable to me as I expect java.lang.math.pow to be well behaved and most of those edge cases were around type testing which no longer applies. At the same time, this simplification allows us to leverage the new scalar function testing framework, which means better null coverage, better type coverage, and much easier extensibility.
We do consider this a breaking change, but as the feature is still in tech preview and this is a relatively small surface area, we are not too concerned with disruptions.
Resolves#99055
Relates to #100558
---------
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
* Page structure
* More getting started content
* Fix build errors
* Small improvements
* Typo
* Add link to public demo environment
* Review feedback
* Update docs/reference/esql/esql-get-started.asciidoc
Co-authored-by: Andrei Stefan <astefan@users.noreply.github.com>
* Review feedback
---------
Co-authored-by: Andrei Stefan <astefan@users.noreply.github.com>
Small improvements in ESQL docs and addition of examples
---------
Co-authored-by: Alexandros Batsakis <abatsakis@elastic.co>
Co-authored-by: Abdon Pijpelink <abdon.pijpelink@elastic.co>
This adds more tests for some of the `MV_` functions and updates their
docs now that the railroad diagram and table generated by the tests
covers all of the types.
* Break out 'Limitations' into separate page
* Add REST API docs
* Restructure commands, functions, and operators refs
* Add placeholder for getting started guide
* Group 'Syntax', 'Metafields', and 'MV fields' under 'Language'
* Add placeholder for Kibana page
* Add link from landing page
* Apply uniform formatting to ACOS, CASE, and DATE_PARSE function refs
* Reword default LIMIT
* Add support for COUNT(*)
* Move 'Commands' and 'Functions and operators' to individual pages
---------
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
Fixes https://github.com/elastic/elasticsearch/issues/99507
Enhance SHOW FUNCTIONS command to return as _structured_ information as
possible about the function signature, ie. - function name - return type
- param names - param types - param descriptions
For now, as an example, the annotations are used only on `sin()` and
`date_parse()` functions; **if we agree on this approach**, I'll proceed
to - enhance all the currently implemented functions with the needed
information - improve the function tests to verify that every new
implemented function provides meaningful information
---
This feature can be useful for the end user, but the main goal is to
give Kibana an easy way to produce in-line documentation (contextual
messages, autocomplete) for functions
Similar to current implementation, that has a `@Named("paramName")`
annotation for function parameters, this PR introduces two more
annotations `@Param(name, type, description, optional)` and
`@FunctionInfo()` to provide information about single parameters and
functions.
The result of `SHOW FUNCTIONS` query will have the following columns: -
name (keyword): the function name - synopsis (keyword): the full
signature of the funciton, eg. `double sin(n:integer|long|double)` -
argNames (keyword MV): the function argument names - argTypes (keyword
MV): the function argument types - argDescriptions (keyword MD): a
textual description of each function argument - returnType (keyword):
the return type of the function - description (keyword): a textual
description of the function
---
Open questions: - ~~how structured shoud *types* be? Eg. should we have
a strict `@Typed("keyword")`/`@Typed({"keyword", "text"})` or should we
have a more generic type description, eg. `@Typed("numeric")`,
`@Typed("any")`? The first one is more useful for API consumption but
it's hard with our complex type system (type classes, custom types,
unsupported and so on); the second one is less structured, but probably
more useful for documentation, that is the most immediate use case of
this feature.~~ All the types are listed explicitly
- ~~we have alternatives for the synopsis, eg.~~
- ~~`functionName(<paramName>:<paramType>, ...): <returnType>`~
- ~~`<returnType> functionName(<paramName>:<paramType>, ...)`~~
- ~~`<returnType> functionName(<paramType> <paramName>, ...)`~~
Using `<returnType> functionName(<paramName>:<paramType>, ...)` for now. If multiple types are supported, then they will be separated by pipes, eg. `double sin(n:integer|long|double)`.
This creates `Block.Ref`, a reference to a `Block` which may or may not
be part of a `Page`. `Block.Ref` is `Releasable` and closing it is a
noop if the `Block` is part of a `Page`, but if it is "free floating"
then closing the `Block.Ref` will close the block.
It also modified `ExpressionEvaluator` to return a `Block.Ref` instead
of a `Block` - so you tend to work with `ExpressionEvaluator`s like
this:
```
try (Block.Ref ref = eval.eval(page)) {
return ref.block().doStuff();
}
```
This should make it *much* easier to release the memory from `Block`s
built by `ExpressionEvaluator`s.
This change is mostly mechanical, introducing the new signature for
`ExpressionEvaluator`. In a follow up change I'll modify the tests to
make sure we're correctly using it to close pages.
I did think about changing `ExpressionEvaluator` to add a method telling
you if the block that it returns must be closed or not. This would have
been more difficult to work with, and, ultimately, limiting.
Specifically, it is possible for an `ExpressionEvaluator` to *sometimes*
return a free floating block and other times return one that is
contained in a `Page`. Imagine `mv_concat` - it returns the block it
receives if the block doesn't have multivalued fields. Otherwise it
concats things. If that block happens to come directly out of the
`Page`, then `mv_concat` will sometimes produce free floating blocks and
sometimes not.
Bogdan Pintea
Fang Xing
Mark Tozzi
Liam Thompson
Alexander Spies
Nik Everett
Craig Taverner
Luigi Dell'Aquila
Tommaso Teofili
Ioana Tagirta
Nhat Nguyen
Costin Leau
Abdon Pijpelink
Anton Dosov
Ignacio Vera
Chris Hegarty
Marco Liberati
ChrisHegarty
David Turner
Nicolas Gras
Nastasha Solomon
Fabio Busatto
Andrei Stefan
AlexB
gheorghepucea