Commit Graph

10835 Commits

Author SHA1 Message Date
Jonathan Buttner 1ca66bde91
[ML] Safely drain deployment request queues before allowing node to shutdown (#98406)
* isSafeToShutdown checks routing table

* Rebalancer changes and tests

* Update docs/changelog/98406.yaml

* Forcing lifecycle tests to avoid over time case

* Changes and remaining tests

* Adding node service changes

* Finishing unit tests

* Adding wait for completion paramater

* Adding stop deployment integration tests

* Cleaning up code

* Fixing stop deployment test

* Fixing string formatter issue and timeout

* Investigating deadlock

* More testing

* More logging

* Prevent model reloading while stopping

* Fixing compile error

* More code clean up

* Adding test for loading model after stopping

* Addressing review feedback

* Fixing a couple shutdown -> shutdownNow tests

* Adding doc changes and refactoring
2023-08-31 15:37:11 -04:00
Joe Gallo 3284903205
Document the redact processor's skip_if_unlicensed option (#99063) 2023-08-31 14:00:12 -04:00
Michael Peterson 649821e992
Support cluster/details for CCS minimize_roundtrips=false (#98457)
This commit tracks progress for each shard search by cluster alias
using a new SearchProgressListener (CCSSingleCoordinatorSearchProgressListener).
Both sync and async CCS searches use this new progress listener when
minimize_roundtrips=false.

Two of the SearchProgressListener method had to be extended to allow tracking
per-cluster took values (TransportSearchAction.SearchTimeProvider) and
whether searches timed out (by passing in QuerySearchResult to the onQueryResult
listener method).

This commit brings parity between minimize_roundtrips=true and false to have
the same _cluster/details sections in CCS search responses.

Note that there are still a few differences between minimize_roundtrips=true and false.
1. The per-cluster took value for minimize_roundtrips=true is accurate, but the
   for 'false' it is only measured at the granualarity of each partial reduce,
   so the per cluster took time is overestimated in basically all cases.
2. For minimize_roundtrips=true, a skip_unavailable=false cluster that disconnects
   during the search or has all searches on all shards fail, will cause the entire
   search to fail. This is (still) not true for minimize_roundtrips=false. The search
   is only failed if the skip_unavailable=false cluster cannot be connected to at the
   start of the search. (This will likely be changed in a follow up ticket that implements
   fail-fast logic for in-progress searches that should fail due to a skip_unavailable=true
   cluster failing.)
3. The shard accounting for minimize_roundtrips=false is always accurate (total shard counts
   are known at the start of the search). For minimize_roundtrips=true, the shard accounting
   is only accurate per cluster unless all clusters have successful (or partially successful)
   searches. For clusters that have failures we do not have shard count info.
2023-08-31 12:56:20 -04:00
Leandro de Lima Camargo e31eed37c5 Fix voting_only:true in node spec docs (#99099) 2023-08-31 17:41:50 +01:00
Stef Nestor 9573f6f1f4
[DOC+] snapshot-restore single index example (#99065)
* [DOC+] snapshot-restore single index example

👋🏼 howdy, team! I'd like to append an example to snapshot-restore a single index. Support usually points users to [this page](https://www.elastic.co/guide/en/elasticsearch/reference/master/restore-snapshot-api.html) but then users attempt the `rename_pattern` example (which makes sense!). I'd like to point them to a more literal "close index > restore on that index" example in the future.

* Fix test failure and reword

---------

Co-authored-by: Abdon Pijpelink <abdon.pijpelink@elastic.co>
2023-08-31 16:23:50 +02:00
Tim baa4e5d6ab
[DOCS] Deduplicate the JVM sizing instructions in Docker documentation (#99079)
- Deduplicates the JVM sizing instructions
- Links the `-m` tag tip section to the JVM sizing instructions
- Replaces the fully typed out container registry to {docker-image}

Co-authored-by: James Rodewig <james.rodewig@elastic.co>
2023-08-31 08:34:53 -04:00
Abdon Pijpelink 792f9c1647
[DOCS] Remote cluster migration guide (#98999)
* [DOCS] Remote cluster migration guide

* Review feedback

* Clarify that any extra local privileges will be suppressed by the cross-cluster API key’s privileges
2023-08-31 10:24:20 +02:00
Nik Everett b73cc0c529
ESQL: Only generate syntax diagrams locally (#99059)
CI will skip building them. Lot's of CI machines don't have font support
so they can't generate these. But all local machine have a GUI so they
can.

Also, super-lazy initialize the font so CI don't bump into it by
accident.

Closes #99018
2023-08-30 14:44:14 -04:00
Kyle Carter 8ea41ddf1d
Fix link to Lucene docs for TrimFilter (#99050) 2023-08-30 16:34:54 +02:00
Benjamin Trent 7b9c367aeb
Auto-normalize dot_product vectors at index & query (#98944)
`dot_product` requires vectors to be unit-length. Previously, we would
check that vectors were unit-length and throw if they were not. 

Instead, we will now auto-normalize vectors as they are indexed.

`cosine` will continue to behave as usual, not normalizing the vectors.

closes: https://github.com/elastic/elasticsearch/issues/98935
2023-08-30 09:50:49 -04:00
Joe Gallo a354be7b69
Fix a couple of docs typos (#99028) 2023-08-30 09:01:34 -04:00
Bogdan Pintea 9beddb18d6
ESQ: Add metadata fields documentation (#98991)
This adds documentation for the supported metadata fields.

---------

Co-authored-by: Abdon Pijpelink <abdon.pijpelink@elastic.co>
2023-08-30 12:51:31 +02:00
Abdon Pijpelink b6c85dbe82
[DOCS] Update ES|QL Kibana instructions (#98940) 2023-08-30 11:18:49 +02:00
Liam Thompson dfbec46c3d
[Docs] Add link to labs from semantic search overview (#98985) 2023-08-30 10:54:24 +02:00
Stef Nestor 0781bafac1
[DOC+][Hot Spotting] Pull detailed Node Tasks (#98879)
Co-authored-by: David Turner <david.turner@elastic.co>
2023-08-29 14:25:10 -04:00
Stef Nestor de380ea2af
[DOC+] Write threadpool also covers ingest pipelines (#99010)
Co-authored-by: James Rodewig <james.rodewig@elastic.co>
2023-08-29 13:51:18 -04:00
Liam Thompson a3c96caa51
[DOCS] Add link to Elasticsearch labs ELSER Python notebook (#98983)
* Add link to Elasticsearch labs ELSER Python notebook

* Fix typos

* Use {es} variable

Co-authored-by: István Zoltán Szabó <istvan.szabo@elastic.co>

---------

Co-authored-by: István Zoltán Szabó <istvan.szabo@elastic.co>
2023-08-29 15:26:00 +02:00
James Rodewig 1301995a46
[DOCS] Remove unneeded `ifeval`s from install docs (#98952)
Problem: The current install docs contain several `ifeval` statements that hide commands and code snippets based on the branch's release state. These statements make the Asciidoc hard to read  and maintain. It also makes doc changes difficult to preview.

Solution: Remove `ifeval` statements that hide commands or code snippets. Leave in any `ifeval` statements used to add warnings.
2023-08-29 07:56:10 -04:00
James Rodewig 73293dca86
[DOCS] Remove unused widgets (#98953)
No-op change to remove a few unused Asciidoc widgets. These aren't referenced or published anywhere in the docs.
2023-08-29 07:53:26 -04:00
dreamquster 2644ccbb8a
Implement the 'left' function in issue #98545 (#98942)
@nik9000  Recheck out the main branch. Refactor the 'left' function to
cut the prefix string in place. But I meet a adversity that left failed
the test case 'testEvaluateInManyThreads'. I find that in multiple
thread situation,  `  EvalOperator.ExpressionEvaluator eval =
evalSupplier.get(); for (int c = 0; c < count; c++) {      
assertThat(toJavaObject(eval.eval(page), 0), testCase.getMatcher()); } `
toJavaObject function return a BytesRef with length=2, content is
[81,89]. However, assertThat function in junit4 receive the BytesRef
parameters that its length is 10. Can you give me some clues? I can't
find which variable is mutual.

Rerun failed test case's command: `gradlew ':x-pack:plugin:esql:test'
--tests
"org.elasticsearch.xpack.esql.expression.function.scalar.string.LeftTests.testEvaluateInManyThreads
{TestCase=Left basic test}" -Dtests.seed=44459C172243712
-Dtests.locale=lv-LV -Dtests.timezone=Asia/Irkutsk -Druntime.java=20`
2023-08-28 13:17:16 -04:00
Nik Everett 4cd7f40712
ESQL: More docs (#98890)
Adds some more typed docs to the ESQL functions.
2023-08-28 11:17:04 -04:00
James Rodewig 29e2e6f141
[DOCS] Simplify single-node Docker instructions (#98886)
Problem:
The current [single-node Docker instructions](https://www.elastic.co/guide/en/elasticsearch/reference/current/docker.html#docker-cli-run-dev-mode) are overly verbose and contain a lot of unneeded info about security internals. 

Solution:
- Restructure the above docs to focus primarily on actionable steps.
- Test the docs across operating systems (Mac, Linux, Windows) to ensure they work.
2023-08-28 08:54:30 -04:00
James Rodewig 31408bddc1
[DOCS] Add $ELASTIC_PASSWORD env var to install docs (#98898)
- Removes duplicated security autoconfiguration output from the docs. This is difficult to keep updated and makes the docs longer.
- Encourages the user to store the `elastic` password as an environment variable. Users don't need to rely on curl's password prompts.
- Removes unused `api-call-widget` files. These aren't published anywhere in the docs currently.
2023-08-28 08:25:13 -04:00
Alexander Spies ca3dc3a882
ESQL: Add `CEIL` function (#98847)
Add the unary scalar function CEIL.

Analogously to FLOOR,  it rounds up its argument.

- Implement CEIL, add it to the function registry and make sure it is serializable.
- Add csv tests, unit tests and docs.
- Add additional csv tests with different data types and some edge cases for both CEIL and FLOOR
- Add unit tests and update docs for FLOOR.
2023-08-28 12:31:56 +02:00
Nik Everett ff01fb680b
ESQL: Standardize font used in railroad diagrams (#98897)
Locks the railroad diagrams to always use the same font, this one named
`roboto mono`. This makes sure that when we render the railroad diagrams
we always size them the same way. Because everyone has a copy of roboto
mono. Because gradle resolves that dependency.
2023-08-26 14:19:47 -04:00
Nik Everett 649ceb74ab
ESQL docs: generate references for functions (#98856)
This generates a "railroad diagram" svg image that can be embedded into
the docs for any function to explain it's syntax. It's basic, but it's
something we can iterate on.

It also generates a table of supported types from the list of types that
we test. It can be included in the docs for reference as well.
2023-08-25 09:07:25 -04:00
Yang Wang f307e6fedd
[Docs] Document IP filtering for RCS 2.0 (#98553)
This PR adds documentation for RCS 2.0 IP filtering.

Relates: #98472
2023-08-24 23:23:40 -04:00
Abdon Pijpelink 1955bd8ad4
[DOCS] New docs for remote clusters using API key authentication (#98330)
* New docs structure for remote clusters

* Fix broken cross-book link errors

* More broken cross-book link errors

* Remove redirects for new pages

* Link to generic remote cluster docs instead

* Drop 'API' from the abbreviated title

* Add 'Establish trust with a remote cluster' section

* Restructure 'Establish trust' section into Prprequisite/local/remote instructions

* Add 'Configure roles and users' section

* Add 'Connect to a remote cluster' section

* Move version compatibility to prerequisites

* Fix test errors

* Incorporate review feedback

* Mention version 8.10 or later in the intro for API keys

* Add license prerequisite
2023-08-24 12:30:03 +02:00
Mary Gouseti b9b818e28e
Allow explain data stream lifecycle to accept a data stream. (#98811)
Currently the `GET target/_lifecycle/explain` API only works for
indices. In this PR we extend this behaviour to allow the target to be a
data stream so we can get the overview lifecycle status for all the
backing indices of a data stream.
2023-08-24 06:29:09 -04:00
Dianna Hohensee a25e176692
Add node "roles" to allocation explain response (#98550)
Report node "roles" in the /_cluster/allocation/explain response.
Nodes with limited sets of roles may affect shard distribution in ways
users did not originally consider, so it is helpful to surface this
information along with node allocation decision explanations.
2023-08-23 08:30:35 -04:00
István Zoltán Szabó 947128e76d
[DOCS] Fixes NOTE display error. (#98783) 2023-08-23 12:18:54 +02:00
Peter Dyson ef7b7ccded
[DOCS] mention closed indices are included by default for snapshot (#98766) 2023-08-23 10:41:35 +02:00
Lee Hinman 815d596daa
Add 'dataset' size to cat indices and cat shards (#98622)
* Add 'dataset' size to cat indices and cat shards

This adds the `dataset` computed size for the `/_cat/indices` and `/_cat/shards` APIs. This new
column is reported by default.

Resolves #95092
2023-08-22 15:36:32 -06:00
Nik Everett 65ea90d3fd
ESQL: LEAST and GREATEST functions (#98630)
Adds `LEAST` and `GREATEST` functions to find the min or max of the
values in many columns.
2023-08-22 14:15:04 -04:00
Bogdan Pintea 372458c9fd
ESQL: date_trunc(): swap order of arguments (#98624)
Swap arguments order so that the range parameter is first and datetime
one second, inline with other languages.
2023-08-22 18:20:05 +02:00
Andrei Dan 01ed7de99f
GA the data stream lifecycle (#98644)
This makes the data stream lifecycle generally available. This will allow
data streams to take advantage of a native simplified and resilient
lifecycle implementation.
2023-08-21 17:28:54 +01:00
Abdon Pijpelink 68aee20d01
[DOCS] Removing 'coming' admonition from 8.9.1 release notes (#98658) 2023-08-21 14:05:42 +02:00
Abdon Pijpelink 2bbc1df9bd
[DOCS] Remove 'coming' notice from 8.8.2 release notes (#98659) 2023-08-21 14:05:15 +02:00
Abdon Pijpelink d04c0f7976
Fix response for IS NULL (#98671)
The response shown for IS NULL here is actually the response for the IS NOT NULL snippet. This PR fixes that.
2023-08-21 13:20:36 +02:00
Abdon Pijpelink 4d7193d76d
[main] Update docs for v8.9.1 release (#98416) (#98655)
* Update docs for v8.9.1 release (#98416)

(cherry picked from commit 7985f8ef0f)

* Resolve merge conflicts

---------

Co-authored-by: Volodymyr Krasnikov <129072588+volodk85@users.noreply.github.com>
2023-08-21 10:53:39 +02:00
Nik Everett 44e61341f2
ESQL: COALESCE function (#98542)
This adds a `COALESCE` function that returns the first non-null value.
2023-08-17 13:51:44 -04:00
Roberto Seldner 79d2879564
Add deprecated note for `balanced` allocator (#98610)
Co-authored-by: James Rodewig <james.rodewig@elastic.co>
2023-08-17 12:50:52 -04:00
Nik Everett a380e8c369
ESQL: LTRIM, RTRIM and fix unicode whitespace (#98590)
Here we add support for the following two ESQL functions:
* LTRIM: remove leading spaces from a string
* RTRIM: remove trailing spaces from a string

We also fix an issue with the handling of unicode white spaces. We
make use of unicode code points to identify unicode whitespace
characters instead of relying on ASCII codes.

Moreover, iterating bytes in a Unicode string needs to consider
that some Unicode characters are encoded using multiple bytes.
2023-08-17 11:30:12 -04:00
Craig Taverner aad16b7d6b
Simple ESQL pow() docs fixes after re-reviewing (#98601) 2023-08-17 17:29:06 +02:00
Kathleen DeRusso 8c12a7b7cd
Query rules docs clarification (#98605)
* Query rules docs clarification

* Update docs/reference/search/search-your-data/search-using-query-rules.asciidoc

* Update docs/reference/search/search-your-data/search-using-query-rules.asciidoc
2023-08-17 11:11:49 -04:00
Carlos Delgado 2b838ae853
Dense vector field types are indexed by default (#98268)
* First version

* Spotless, I liked my version better

* Fix param default values

* Add a supplier for default value to ensure it's calculated correctly

* Can't improve this without breaking tests

* Added checks for not specifying a body in PUT requests

* Fix default provider for enum params

* Added yaml test

* Changed docs and fix TODO

* Removing synonyms changes

* Added separate methods for providing default value as suppliers in enums

* Fixed test

* Add a supplier for default value to ensure it's calculated correctly

* Added checks for not specifying a body in PUT requests

* Remove synonyms changes

* Remove some supplier changes

* Better call enumParam with supplier version

* Fix compiler error on supplier

* Apply validators or requires depending on index version

* Solved BWC tests that involved using validators instead of requiresParameters

* Add tests

* Spotless

* Update docs/changelog/98268.yaml

* Update changelog

* Update docs/changelog/98268.yaml

* PR comments

* PR feedback

* Serialize index only for new index versions

---------

Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
2023-08-17 10:53:14 -04:00
Craig Taverner dfe9bdc45f
Simple grammar fix for MVT docs (#98591) 2023-08-17 16:10:26 +02:00
ChrisHegarty 21dcb75836 Merge upstream 2023-08-17 11:40:42 +01:00
Simon Cooper 9538208fb7
Fix shard-stores doc test (#98581)
The replacement wasn't actually doing anything due to different JSON types
2023-08-17 11:38:31 +01:00
Nick Chow 5de0a9013f
Documentation update that fixes a query rules code example (#98540)
* Change example field in rule query guide

* Change fuzzy to contains to get tests to work

---------

Co-authored-by: Kathleen DeRusso <kathleen.derusso@elastic.co>
2023-08-16 15:14:32 -07:00