Commit Graph

11200 Commits

Author SHA1 Message Date
Jake Landis 78c6a7c4bb
security-minimal-setup.asciidoc: replace KIB_PATH_CONF by KBN_PATH_CONF (#104582) (#104631)
Co-authored-by: Fabien Baligand <fbaligand@gmail.com>
2024-01-22 16:43:03 -05:00
Brian Seeders ffcb12456c
Update 8.12.0 release notes for #102183 and fix other release notes (#104551) 2024-01-22 10:31:21 -05:00
Michael Peterson e8370f8c43
Update search-across-clusters API docs to include incremental partial results (#104489) 2024-01-22 08:34:20 -05:00
Abdon Pijpelink 9948f5a995
Update 8.12.0.asciidoc (#104605) (#104613)
Split Notable and Breaking changes, as the notables were appearing to be breaking changes.

(cherry picked from commit 1c1925c16f)

Co-authored-by: David Brimley <dave.brimley@gmail.com>
2024-01-22 05:17:59 -05:00
Albert Zaharovits aeb2b77c3c
Add support for the `simple_query_string` to the Query API Key API (#104132)
This adds support for the simple_query_string query type to the Query API key Information API.
In addition, this also adds support for querying all the API Key metadata fields simultaneously,
rather than requiring each to be specified, such as metadata.x, metadata.y, etc.

Relates: #101691
2024-01-19 14:51:12 +02:00
Mayya Sharipova 669d4ae9b9
Add hybrid search to knn query documentation (#104562)
Relates to PR #98916
Closes elastic/search-docs-team#39
2024-01-18 15:53:48 -05:00
David Turner 1c11249c05
Fix docs about uneven disk usage (#104541)
There's a note in the docs saying we only consider shard count and not
disk usage which is no longer true. This commit fixes the note to
reflect today's implementation.
2024-01-18 16:02:37 +00:00
Anton Dosov 990b603cbd
[Docs] Add ES|QL CSV limitations in Kibana (#104530)
Mentioned that the CSV rows limit is also 10,000 rows. CSV export added here: https://github.com/elastic/kibana/issues/173390
2024-01-18 16:32:57 +01:00
Abdon Pijpelink ef867c56a6
Update esql-security-solution.asciidoc (#104531)
Fixed a typo
2024-01-18 15:48:43 +01:00
Abdon Pijpelink 7b37d4242e
[DOCS] Mention that vector quantization increases disk usage (#104509) 2024-01-18 14:01:07 +01:00
Craig Taverner 9ac356bbcf
Update docs with support for geo_shape and cartesian_shape  (#104480)
* Document to_geoshape and to_cartesianshape

* Note that geo_point and point are now supported Elasticsearch types

* Fix doc links
2024-01-18 13:00:38 +01:00
Costin Leau 607185b280
ESQL: Nested expressions inside stats command (#104387)
Allow nested expressions to be used both for grouping or inside
 aggregate functions inside the stats command.
As such the grammar has been tweaked to allow the stats group to have 
 optional aliasing.
As part of this fix, preserve the original field declaration (including 
 spaces) for implicit aliases.
Improve validation for incorrect aggregate function use (as arguments,
 grouping or inside evals).

Fix #99828
2024-01-17 15:49:20 -08:00
Brian Seeders 58cfa5e05a
Add final 8.12.0 docs 2024-01-17 15:03:22 -05:00
Benjamin Trent e4feaff900
Add support for more than one inner_hit when searching nested vectors (#104006)
This commit adds the ability to gather more than one inner_hit when
searching nested kNN.

# Global kNN example

```
POST test/_search
{
    "_source": false,
    "fields": [
        "name"
    ],
    "knn": {
        "field": "nested.vector",
        "query_vector": [
            -0.5,
            90,
            -10,
            14.8,
            -156
        ],
        "k": 3,
        "num_candidates": 3,
        "inner_hits": {
            "size": 2,
            "fields": [
                "nested.paragraph_id"
            ],
            "_source": false
        }
    }
}
```

Results in

<details>

```
{
    "took": 66,
    "timed_out": false,
    "_shards": {
        "total": 2,
        "successful": 2,
        "skipped": 0,
        "failed": 0
    },
    "hits": {
        "total": {
            "value": 2,
            "relation": "eq"
        },
        "max_score": 0.009090909,
        "hits": [
            {
                "_index": "test",
                "_id": "2",
                "_score": 0.009090909,
                "fields": {
                    "name": [
                        "moose.jpg"
                    ]
                },
                "inner_hits": {
                    "nested": {
                        "hits": {
                            "total": {
                                "value": 2,
                                "relation": "eq"
                            },
                            "max_score": 0.009090909,
                            "hits": [
                                {
                                    "_index": "test",
                                    "_id": "2",
                                    "_nested": {
                                        "field": "nested",
                                        "offset": 0
                                    },
                                    "_score": 0.009090909,
                                    "fields": {
                                        "nested": [
                                            {
                                                "paragraph_id": [
                                                    "0"
                                                ]
                                            }
                                        ]
                                    }
                                },
                                {
                                    "_index": "test",
                                    "_id": "2",
                                    "_nested": {
                                        "field": "nested",
                                        "offset": 1
                                    },
                                    "_score": 0.004968944,
                                    "fields": {
                                        "nested": [
                                            {
                                                "paragraph_id": [
                                                    "2"
                                                ]
                                            }
                                        ]
                                    }
                                }
                            ]
                        }
                    }
                }
            },
            {
                "_index": "test",
                "_id": "3",
                "_score": 0.0021519717,
                "fields": {
                    "name": [
                        "rabbit.jpg"
                    ]
                },
                "inner_hits": {
                    "nested": {
                        "hits": {
                            "total": {
                                "value": 1,
                                "relation": "eq"
                            },
                            "max_score": 0.0021519717,
                            "hits": [
                                {
                                    "_index": "test",
                                    "_id": "3",
                                    "_nested": {
                                        "field": "nested",
                                        "offset": 0
                                    },
                                    "_score": 0.0021519717,
                                    "fields": {
                                        "nested": [
                                            {
                                                "paragraph_id": [
                                                    "0"
                                                ]
                                            }
                                        ]
                                    }
                                }
                            ]
                        }
                    }
                }
            }
        ]
    }
}
```

</details>

# kNN Query example

With a kNN query, this opens an interesting door, which allows for
multiple inner_hit scoring schemes.

## Nearest by max passage only

```
POST test/_search
{
    "size": 3,
    "query": {
        "nested": {
            "path": "nested",
            "score_mode": "max",
            "query": {
                "knn": {
                    "field": "nested.vector",
                    "query_vector": [
                        -0.5,
                        90,
                        -10,
                        14.8,
                        -156
                    ],
                    "num_candidates": 5
                }
            },
            "inner_hits": {
                "size": 2,
                "_source": false,
                "fields": [
                    "nested.paragraph_id"
                ]
            }
        }
    }
}
```

</details>

closes: https://github.com/elastic/elasticsearch/issues/102950
2024-01-17 11:32:46 -05:00
Nik Everett 919d282aa6
ESQL: Add option to drop null fields (#102428)
This adds an option to drop columns that are entirely null from the
results. Is this something we want?
2024-01-17 09:06:19 -05:00
Ignacio Vera 585face0cf
ESQL: Support loading shapes from source into WKB blocks (#104269)
This commit adds support for reading geo_shape and shape fields into wkb blocks so they can use top project the 
data to the result of a ES|QL query.
2024-01-17 14:55:37 +01:00
Liam Thompson 52aefa59eb
[DOCS] Ingest processors docs improvements (#104384)
* [DOCS] Categorize ingest processors on overview page, summarize use cases

* Add overview info, subheading, links

* Apply suggestions from review

Co-authored-by: István Zoltán Szabó <istvan.szabo@elastic.co>

* Insert space

---------

Co-authored-by: István Zoltán Szabó <istvan.szabo@elastic.co>
2024-01-17 11:50:29 +01:00
David Turner a3c2b2d611
Add stable master indicator troubleshooting links (#104395)
Adds links from the stable master health indicator to the relevant
troubleshooting docs, as well as making the "contact support" link a
versioned link directly to the right subsection of the troubleshooting
docs page.
2024-01-16 15:22:33 -05:00
Kostas Krikellas c4c2ce83cb
Downsampling supports date_histogram with tz (#103511)
* Downsampling supports date_histogram with tz

This comes with caveats, for downsampled indexes at intervals more than
15 minutes. For instance,
 - 1-hour downsampling will produce inaccurate
results for 1-hour histograms on timezones shifted by XX:30
 - 1-day downsampling will produce inaccurate daily
histograms for not-UTC timezones as it tracks days at UTC.

Related to #101309

* Update docs/changelog/103511.yaml

* test daylight savings

* update documentation

* Offset time buckets over downsampled data with TZ

* Update docs/changelog/103511.yaml

* check for TSDS

* fixme for transport version

* add interval to index metadata

* add transport version

* bump up transport version

* address feedbcak

* spotless fix
2024-01-16 10:27:33 +02:00
Luigi Dell'Aquila ad28dc9a6c
ESQL: Add TO_UPPER and TO_LOWER functions (#104309) 2024-01-15 11:58:57 +01:00
István Zoltán Szabó f9398ed506
[DOCS] Adds Hugging Face service to the inference API docs (#104190) 2024-01-15 11:21:22 +01:00
Chris Hegarty 5d6b833fe6
ES|QL Minor async query doc parameter clarification (#104327)
This commit adds a minor clarification to an ESQL async query doc
parameter.
2024-01-12 11:50:00 -05:00
Liam Thompson 01751c0298
[Docs] Add connectors links, cleanup connectors API docs (#104262)
* [Docs] Add connectors links

* 🧹 Cleanup abbreviations, add missing tech preview labels

* Unify remaining tech preview, abbreviations, update CLI verbiage

* Unify remaining tech preview, abbreviations
2024-01-12 12:13:34 +01:00
Brian Seeders 10a84cbee9
[main] Add 8.11.4 release notes (#104276) (#104282) 2024-01-11 16:13:37 -05:00
Ignacio Vera f64147f6c1
ESQL: Remove the possibility of CartesianPoint or GeoPoint literal to a long (#104197) 2024-01-11 14:25:48 +01:00
ShourieG 147484b059
[elasticsearch][processors] - Added support for override flag in rename processor (#103565)
* added override flag for rename processer along with factory tests

* added yaml tests for rename processor using the override flag

* updated renameProcessor tests to include override flag as a parameter

* updated rename processor tests to incorporate override flag = true scenario

* updated rename processor asciidoc with override option

* updated rename processor asciidoc with override option

* removed unnecessary supresswarnings tag

* corrected formatting errors

* updated processor tests

* fixed yaml tests

* Prefer early throw style here

* Whitespace

* Move and rewrite this test

It's just a simple test of the primary behavior of the rename
processor, so put it first and simplify it.

* Rename this test

It doesn't actually exercise template snippets

* Tidy up this test

---------

Co-authored-by: Joe Gallo <joegallo@gmail.com>
2024-01-11 16:00:02 +05:30
Johannes Fredén cc9fba36e6
Add ApiKey expiration time to audit log (#103959)
Follow up to PR: https://github.com/elastic/elasticsearch/pull/103453
2024-01-11 04:40:52 -05:00
Marco Liberati 3145e9f9fc
[ES|QL] Annotate all ESQL functions (for SHOW FUNCTIONS) (#103686)
* 🔧 Initial annotations

* ✏️ Add substring doc

* 🏷️ Fix imports

* ✏️ Add annotations for string fns

* ♻️ Fix issues

* :rotatin_light: Fix linting issues

* ✏️ Add more annotations

* :rotatin_light: Fix linting issues

* ♻️ Add missing import

* ✏️ Add more annotations

* 🚨 Fix linting

*  Fix many tests for new annotations

*  Fix more tests

* 🐛 Fix missing flag

*  Annotate new functions

*  fix more tests

*  Fix signature issue

*  Make all csv tests pass

*  Fix remaining tests

*  New assets from annotations

*  Refactor test

*  Fix updated signature
2024-01-11 10:15:30 +01:00
Albert Zaharovits f4aaa20f28
Add support for the `type` parameter to the Query API Key API (#103695)
This adds support for the type parameter to the Query API key API.
The type for an API Key can currently be either rest or cross_cluster.

Relates: #101691
2024-01-11 10:53:50 +02:00
Bogdan Pintea 9f6e5a4ccf
ESQL: Add Warning tests for all remaining functions (#103739)
This adds unit tests cases for all the functions that were missing tests
checking on the correct generation of the Warning headers in case the
execution raised an Exception that lead to a `null` result.
2024-01-10 22:27:13 +01:00
Volodymyr Krasnikov f6f86d11d3
Calc follower vs leader indexing lag based on shard global checkpoints (#104015)
* Calc follower vs leader indexing lag baed on shard global checkpoints

* code simplify

* follow PR comments

* spotless

* Update docs/reference/ccr/apis/follow/get-follow-stats.asciidoc

Co-authored-by: Iraklis Psaroudakis <kingherc@gmail.com>

---------

Co-authored-by: Iraklis Psaroudakis <kingherc@gmail.com>
2024-01-10 11:03:52 -08:00
David Turner 312d4c2fa1
Mention `IndexFormatToo{Old,New}Exception` as corruption (#104204)
If a file header is corrupted then the exception may be reported as a
bad index format version rather than a checksum mismatch. This commit
adjusts the docs to cover this case.
2024-01-10 08:54:07 -05:00
Benjamin Trent 73f537170b
Update nested knn search documentation about inner-hits (#104154)
Adding a link tag for inner hits behavior and kNN search. Additionally
adding a note that if you are using multiple knn clauses, that the inner
hit name should be provided.
2024-01-10 07:46:42 -05:00
Martijn van Groningen 4b8d99252d
Update documentation around index.look_ahead_time setting. (#103975)
Adjusted the default after #103898
2024-01-10 09:48:17 +01:00
Mark Tozzi 282f0f0a00
[ESQL] Remove is_nan, is_finite, and is_infinite (#104091)
A while ago we decided we weren't going to support NaN or infinite values in the language. Removing these functions is part of that effort.
2024-01-09 14:51:32 -05:00
Jake Landis 15258c8688
Minor doc fixes for PKI realm (#103831)
Remove the reference to the transport client and clarify the usage of username_pattern
2024-01-09 09:49:59 -06:00
István Zoltán Szabó e93892c0fd
[DOCS] Expands inference API docs (#104047)
Co-authored-by: David Kyle <david.kyle@elastic.co>
2024-01-09 15:46:46 +01:00
Nik Everett 5ef5dca334
ESQL: `MV_FIRST` and `MV_LAST` (#103928)
This creates the `MV_FIRST` and `MV_LAST` functions that return the
first and last values from a multivalue field. They are noops from a
single valued field. They are quite similar to `MV_MIN` and `MV_MAX`
except they work on positional data rather than relative size. That
sounds like a large distinction, but in practice our multivalued fields
are often sorted. And when they operate on sorted arrays `MV_MIN` does
*the same* thing as `MV_FIRST`.

But there are some cases where it really does matter - say you are
`SPLIT`ing something - so `MV_FIRST(SPLIT("foo;bar;baz", ";"))` gets you
`foo` like you'd expect. No sorting needed.

Relates to #103879
2024-01-09 08:46:34 -05:00
Luigi Dell'Aquila 770fc19b14
ESQL: add date_diff function (#104118)
Same as https://github.com/elastic/elasticsearch/pull/103208

Fixes #101942

We had to revert it after a Checkstyle failure (strange it didn't pop up
in the CI before merging)
2024-01-09 07:03:58 -05:00
ChrisHegarty 8500d33841 Fix doc typos 2024-01-09 10:54:23 +00:00
David Turner 943b2eae70 Revert "Esql/create DATE_DIFF function (#103208)"
This reverts commit ec2e18536d.
2024-01-09 09:50:48 +00:00
Nicolas Gras ec2e18536d
Esql/create DATE_DIFF function (#103208) 2024-01-09 10:37:42 +01:00
Chris Hegarty f14d87be58
Add ES|QL async query api docs (#104054)
This commit adds detailed API docs for the ES|QL async apis, as well as moving the narrative to the ES|QL specific REST docs. Additionally, a few top-level lists are reflowed to reflect that there are now multiple APIs.
2024-01-09 09:17:02 +00:00
Craig Taverner 6111d967a4
Fix missing links in geo_point docs for ESQL (#104073) 2024-01-08 23:09:07 +01:00
Bogdan Pintea a5aa900358
ESQL: Update the use of some user-caused exceptions (#104046)
This updates the use of the exceptions subclassed from
`QlServerException` when the failure reason is user-caused. This ensures
that a 400-class response is returned, instead of a 500-class one.
2024-01-08 21:13:20 +01:00
Mary Gouseti 046cdeae23
Introduce lazy rollover for mapping updates in data streams (#103309)
In this PR we implement the idea to introduce a flag, that a data stream needs to be rolloved over before the next document is indexed.
2024-01-08 15:07:16 +02:00
Abdon Pijpelink c6723a3c1d
[DOCS] More ES|QL backtick examples (#103995) 2024-01-08 12:13:33 +01:00
Abdon Pijpelink 3768fa6148
[DOCS] ES|QL backtick changes in 8.13 (#103958) 2024-01-08 12:13:16 +01:00
Abdon Pijpelink ea4b6fd3ea
[DOCS] Change order on 'tune knn' page (#104036) 2024-01-08 12:04:39 +01:00
Chris Hegarty a6c642d4ac
Enable async in ES|QL REST tests (#103700)
This commit enables async in ES|QL REST tests, and also fixes a couple of issue relating to warnings and completion.
2024-01-06 22:06:04 +00:00