Commit Graph

10962 Commits

Author SHA1 Message Date
David Turner 5dff56a00e
Mention network handler logging in docs (#100118)
Mentions the `InboundHandler` (and `OutboundHandler`) as potential
sources of useful log messages when tracking down a network threading
bug.
2023-10-02 08:52:16 +01:00
AlexB 2ccdae6745
Eval REPLACE function (#98909)
Co-authored-by: Alexandros Batsakis <abatsakis@splunk.com>
Co-authored-by: Andrei Stefan <andrei@elastic.co>
2023-09-29 17:41:20 +03:00
Nik Everett e1b1f6f1db
ESQL: Create `Block.Ref` (#100042)
This creates `Block.Ref`, a reference to a `Block` which may or may not
be part of a `Page`. `Block.Ref` is `Releasable` and closing it is a
noop if the `Block` is part of a `Page`, but if it is "free floating"
then closing the `Block.Ref` will close the block.

It also modified `ExpressionEvaluator` to return a `Block.Ref` instead
of a `Block` - so you tend to work with `ExpressionEvaluator`s like
this:

```
try (Block.Ref ref = eval.eval(page)) {
  return ref.block().doStuff();
}
```

This should make it *much* easier to release the memory from `Block`s
built by `ExpressionEvaluator`s.

This change is mostly mechanical, introducing the new signature for
`ExpressionEvaluator`. In a follow up change I'll modify the tests to
make sure we're correctly using it to close pages.

I did think about changing `ExpressionEvaluator` to add a method telling
you if the block that it returns must be closed or not. This would have
been more difficult to work with, and, ultimately, limiting.
Specifically, it is possible for an `ExpressionEvaluator` to *sometimes*
return a free floating block and other times return one that is
contained in a `Page`. Imagine `mv_concat` - it returns the block it
receives if the block doesn't have multivalued fields. Otherwise it
concats things. If that block happens to come directly out of the
`Page`, then `mv_concat` will sometimes produce free floating blocks and
sometimes not.
2023-09-29 09:26:44 -04:00
Kostas Krikellas 98b9e819ee
Represent histogram value count as long (#99912)
* Represent histogram value count as long

Histograms currently use integers to store the count of each value,
which can overflow. Switch to using long integers to avoid this.

TDigestState was updated to use long for centroid value count in #99491

Fixes #99820

* Update docs/changelog/99912.yaml

* spotless fix
2023-09-29 12:30:55 +03:00
István Zoltán Szabó c2072fc4d3
[DOCS] Adds reference documentation for inference API (#99658)
* [DOCS] Creates documentation structure.

* [DOCS] Adds PUT inference API docs and part of GET inference API docs.

* [DOCS] Fixes complaining CI.

* [DOCS] Adds GET and DELETE API docs for inference API.

* [DOCS] Adds POST inference API docs.

* Apply suggestions from code review
2023-09-29 10:12:07 +02:00
Kostas Krikellas c7705aa32a
Improve time-series error and documentation (#100018)
* Improve time-series error and documentation

* spotless fix

* Update docs/changelog/100018.yaml

* Fix changelist

* Change exception type
2023-09-29 10:42:01 +03:00
István Zoltán Szabó 9d01def3dc
[DOCS] Changes semantic search tutorials to use ELSER v2 and sparse_vector field type (#100021)
* [DOCS] Changes semantic search tutorials to use ELSER v2 and sparse_vector field type.

* [DOCS] More edits.
2023-09-29 09:24:36 +02:00
Chris Cressman f99a80f42d
[DOCS] Create URLs for docs migrating from Enterprise Search (#100032)
Several docs are going to migrate from Enterprise Search to
Elasticsearch.

Create the new URLs without yet placing the pages into the Elasticsearch
navigation. This will allow us to handle redirects for Kibana, docs, and
the web without waiting for additional content design decisions.

The new URLs will be:

- https://www.elastic.co/guide/en/elasticsearch/reference/master/ingest-pipeline-search.html
- https://www.elastic.co/guide/en/elasticsearch/reference/master/ingest-pipeline-search-inference.html
- https://www.elastic.co/guide/en/elasticsearch/reference/master/ingest-pipeline-search-inference.html#ingest-pipeline-search-inference-update-mapping
- https://www.elastic.co/guide/en/elasticsearch/reference/master/nlp-example.html
- https://www.elastic.co/guide/en/elasticsearch/reference/master/behavioral-analytics-overview.html
- https://www.elastic.co/guide/en/elasticsearch/reference/master/behavioral-analytics-start.html
- https://www.elastic.co/guide/en/elasticsearch/reference/master/behavioral-analytics-api.html
- https://www.elastic.co/guide/en/elasticsearch/reference/master/behavioral-analytics-event.html
- https://www.elastic.co/guide/en/elasticsearch/reference/master/behavioral-analytics-event-reference.html
- https://www.elastic.co/guide/en/elasticsearch/reference/master/behavioral-analytics-cors.html
- https://www.elastic.co/guide/en/elasticsearch/reference/master/search-application-overview.html
- https://www.elastic.co/guide/en/elasticsearch/reference/master/search-application-api.html
- https://www.elastic.co/guide/en/elasticsearch/reference/master/search-application-client.html
- https://www.elastic.co/guide/en/elasticsearch/reference/master/search-application-security.html
2023-09-28 17:32:37 -04:00
James Rodewig 8f3d374d6d
[DOCS] Add security update to 8.9.2 release notes (#99949)
Adds an update for the [audit logs security announcement](https://discuss.elastic.co/t/elasticsearch-8-9-2-and-7-17-13-security-update/342479) to the 8.9.2 release notes. This was overlooked when the announcement was first made.

I plan to add a similar note to the 7.17.13 release notes.
2023-09-28 16:05:12 -04:00
Max Hniebergall 7c21ce3f1b
Platform specific models (#99584)
* Added platform architecture field to TrainedModelMetadata and users of TrainedModelMetadata

* Added TransportVersions guarding for TrainedModelMetadata

* Prevent platform-specific models from being deployed on the wrong architecture

* Added logic to only verify node architectures for models which are platform specific

* Handle null platform architecture

* Added logging for the detection of heterogeneous platform architectures among ML nodes and refactoring to support this

* Added platform architecture field to TrainedModelConfig

* Stop platform-speficic model when rebalance occurs and the cluster has a heterogeneous architecture among ML nodes

* Added logic to TransportPutTrainedModelAction to return a warning response header when the model is paltform-specific and cannot be depoloyed on the cluster at that time due to heterogenous architectures among ML nodes

* Added MlPlatformArchitecturesUtilTests

* Updated Create Trained Models API docs to describe the new platform_architecture optional field.

* Updated/incremented InferenceIndexConstants

* Added special override to make  models with linux-x86_64 in the model ID to be platform specific
2023-09-28 13:56:45 -04:00
Andrei Dan f202ad02fe
GET _data_stream displays both ILM and DSL information (#99947)
This add support to the `GET _data_stream` API for displaying the value
of the `index.lifecycle.prefer_ilm` setting both at the backing index
level and at the top level (top level meaning, similarly to the existing
`ilm_policy` field, the value in the index template that's backing the
data stream), an `ilm_policy` field for each backing index displaying
the actual ILM policy configured for the index itself, a `managed_by`
field for each backing index indicating who manages this index (the
possible values are: `Index Lifecycle Management`, `Data stream
lifecycle`, and `Unmanaged`).

This also adds a top level field to indicate which system would manage
the next generation index for this data stream based on the current
configuration. This field is called `next_generation_managed_by` and the
same values as the indices level `managed_by` field has are available.

An example output for a data stream that has 2 backing indices managed
by ILM and the write index by DSL:

```
{
	"data_streams": [{
		"name": "datastream-psnyudmbitp",
		"timestamp_field": {
			"name": "@timestamp"
		},
		"indices": [{
			"index_name": ".ds-datastream-psnyudmbitp-2023.09.27-000001",
			"index_uuid": "kyw0WEXvS8-ahchYS10NRQ",
                        "prefer_ilm": true,
			"ilm_policy": "policy-uVBEI",
			"managed_by": "Index Lifecycle Management"
		}, {
			"index_name": ".ds-datastream-psnyudmbitp-2023.09.27-000002",
			"index_uuid": "pDLdc4DERwO54GRzDr4krw",
			"prefer_ilm": true,
			"ilm_policy": "policy-uVBEI",
			"managed_by": "Index Lifecycle Management"
		}, {
			"index_name": ".ds-datastream-psnyudmbitp-2023.09.27-000003",
			"index_uuid": "gYZirLKcS3mlc1c3oHRpYw",
			"prefer_ilm": false,
			"ilm_policy": "policy-uVBEI",
                        "managed_by": "Data stream lifecycle"
		}],
		"generation": 3,
		"status": "YELLOW",
		"template": "indextemplate-obcvkbjqand",
		"lifecycle": {
			"enabled": true,
			"data_retention": "90d"
		},
		"ilm_policy": "policy-uVBEI",
                "next_generation_managed_by": "Data stream lifecycle",
		"prefer_ilm": false,
		"hidden": false,
		"system": false,
		"allow_custom_routing": false,
		"replicated": false
	}]
}
```
2023-09-28 13:48:17 -04:00
Benjamin Trent 92cea2797e
Add nested support for dense_vector fields and knn search (#99763)
* Nested dense_vector support

* Adjust nested support based on new lucene version

* fixing after rebase

* fixing some code

* fixing tests adding transport version

* spotless

* [Automated] Update Lucene snapshot to 9.9.0-snapshot-b3e67403aaf

* Adds new max_inner_product vector similarity function (#99527)

Adds new max_inner_product vector similarity function. This differs from dot_product in the following ways:

Doesn't require vectors to be normalized
Scales the similarity between vectors differently to prevent negative scores

* requiring top level filter to be parent filter

* adding docs & fixing tests

* adding and fixing docs

* adding changlog

* removing unnecessary file changes

* removing unused imports

* fixing test

* maybe fix doc tests

* continue tests in docs

* fixing more tests

* fixing tests

---------

Co-authored-by: Jim Ferenczi <jim.ferenczi@elastic.co>
Co-authored-by: elasticsearchmachine <infra-root+elasticsearchmachine@elastic.co>
2023-09-28 11:38:04 -04:00
Yulia Čech d6526f8d4b
[Index Management] Update docs screenshots with for the new index details page (#99973) 2023-09-28 13:41:53 +02:00
Luca Cavanna 15c87b681c Merge branch 'main' into lucene_snapshot_9_8 2023-09-28 12:19:14 +02:00
Matteo Piergiovanni d9c15c526e
Add counters to _clusters response for all states (#99566)
To help the user know what the possible cluster states are and to 
provide an accurate accounting, we added counters summarising
`running`, `partial` and `failed` clusters to the `_clusters` section.
Changes:
- Now in the response is present the number of `running` clusters.
- We split up `partial` and `successful` (before was summed up in the 
`successful` counter).
- We now have a counter for `failed` clusters.
- Now `total` is always equal to `running` + `skipped` + `failed` + 
`partial` + `successful`.
2023-09-28 09:28:45 +02:00
Nhat Nguyen ae17505557
Introduce authorization for enrich in ESQL (#99646)
This change introduces a new privilege monitor_enrich. Users are 
required to have this privilege in order to use the enrich functionality
in ESQL. Additionally, it eliminates the need to use the enrich_origin
when executing enrich lookups. The enrich_origin will only be used when
resolving enrich policies to prevent warnings when accessing system
indices directly.

Closes #98482
2023-09-27 12:45:39 -07:00
David Turner 8b2a9c646b
Add breaking change docs for #92820 (#99849)
In #92820 we adjusted the indices resolve API to use the
`IndexNameExpressionResolver` to align its behaviour with other similar
APIs, but this was a subtle breaking change in its behaviour when there
were no matching indices. This adds a note in the docs to record this
change in behaviour.
2023-09-27 16:00:22 +01:00
Kostas Krikellas 137bb45662
Support runtime fields in synthetic source (#99796)
* Support runtime fields in synthetic source

* Update docs/changelog/99796.yaml

* Introduce SyntheticSourceProvider

* Address comments

* More fixes

* Fix checkstyle violation

* More unittest updates

* Use SourceProvider in MapperServiceTestCase

* Remove runtime field from unittest

* Update synthetic source doc
2023-09-26 14:29:56 +03:00
István Zoltán Szabó e0cc375b14
[DOCS] Adds text_expansion config to inference processor reference docs. (#99900) 2023-09-26 12:58:19 +02:00
DeDe Morton e3f1c15788
Update ml-delayed-data-detection.asciidoc (#99843) 2023-09-26 11:18:24 +02:00
Bogdan Pintea eaf21483fb
ESQL: Document the existing result-set limitations (#99880)
Document the newly implicit limit of 500 (if no other limit is
provided), as well as the global 10K one.

Related: #99816
2023-09-26 04:12:23 -04:00
David Turner a1caba1521
Add repository backup warning (#99788)
Adds a note about the consequences of trying to back up and restore a
snapshot repository without taking steps to make sure the copy is
consistent.
2023-09-25 17:17:03 +01:00
Nik Everett 5e3ab06151
ESQL: Prevent `CONCAT` from using a ton of memory (#99716)
This prevents `CONCAT` from using an unbounded amount of memory by
hooking it's temporary value into the circuit breaker. To do so, it
makes *all* `ExpressionEvaluator`s `Releasable`. Most of the changes in
this PR just plumb that through to every evaluator. The rest of the
changes correctly release evaluators after their use.

I considered another tactic but didn't like it as much, even though the
number of changes would be smaller - I could have created a fresh,
`Releasable` temporary value for every `Page`. It would be pretty
contained keep the releasable there. But I wanted to share the temporary
state across runs to avoid a bunch of allocations.

Here's a script that used to crash before this PR but is fine after:
```
curl -uelastic:password -XDELETE localhost:9200/test
curl -HContent-Type:application/json -uelastic:password -XPUT localhost:9200/test -d'{
   "mappings": {
      "properties": {
         "short": {
            "type": "keyword"
         }
      }
   }
}'
curl -HContent-Type:application/json -uelastic:password -XPUT localhost:9200/test/_doc/1?refresh -d'{"short": "short"}'

echo -n '{"query": "FROM test ' > /tmp/evil
for i in {0..9}; do
   echo -n '| EVAL short = CONCAT(short' >> /tmp/evil
   for j in {1..9}; do
      echo -n ', short' >> /tmp/evil
   done
   echo -n ')' >> /tmp/evil
done
echo '| EVAL len = LENGTH(short) | KEEP len"}'>> /tmp/evil
curl -HContent-Type:application/json -uelastic:password -XPOST localhost:9200/_query?pretty --data-binary @/tmp/evil
```
2023-09-22 11:27:13 -04:00
James Rodewig ed8ea1f206
[main] [DOCS] Time series indices support non-metric/dimension fields (#99709) (#99811)
Co-authored-by: James Rodewig <james.rodewig@elastic.co>
Co-authored-by: Gilad Gal <gilad.gal@elastic.co>
2023-09-22 09:11:29 -04:00
James Rodewig 4da2d31390
[main] [DOCS] Fix typo in query_cache.asciidoc (#99713) (#99810)
Co-authored-by: Joseph AFARI <71259267+joeafari@users.noreply.github.com>
2023-09-22 08:58:05 -04:00
Luca Cavanna b3e769987d Merge branch 'main' into lucene_snapshot_9_8 2023-09-22 13:11:10 +02:00
William Brafford 571cebc415
Warn about third-party JVM agents in docs (#99718)
* Warn about third-party JVM agents in docs

* Reword, and move warning to its own section

Co-authored-by: James Rodewig <james.rodewig@elastic.co>
2023-09-21 23:49:04 -04:00
Simon Cooper 3f32affbb6
Add component info versions to node info in a pluggable way (#99631)
This adds a `ComponentVersionNumber` service interface for modules to provide version numbers for individual components to be reported inside node info. Initial implementations for `MlConfigVersion` and `TransformConfigVersion` are provided.
2023-09-21 17:08:43 +01:00
David Turner bf34036c8c
Discovery troubleshooting next steps (#99743)
Adds a little more detail on how to react if you see evidence that the
Elasticsearch process is pausing for a long time due to long GCs or VM
pauses.
2023-09-21 13:00:13 +01:00
David Turner 16744b8a94
Remove obsolete docs for G1GC check (#99729)
We removed this bootstrap check in #85361 but didn't remove its docs.
This commit removes the obsolete docs.
2023-09-21 10:42:02 +01:00
James Rodewig 60940c7a9c
[main] [DOCS] Add 8.10.2 release notes (#99722) 2023-09-21 05:06:29 -04:00
Mayya Sharipova ddf17e6be5
Increase the max vector dims to 4096 (#99682) 2023-09-20 15:43:40 -04:00
Benjamin Trent dee85de61c Adds new max_inner_product vector similarity function (#99527)
Adds new max_inner_product vector similarity function. This differs from dot_product in the following ways:

Doesn't require vectors to be normalized
Scales the similarity between vectors differently to prevent negative scores
2023-09-20 20:51:46 +02:00
Jake Landis 19e3036458
Support rotatating the JWT shared secret (#99278)
This commit adds support to reload the JWT shared secret.
Notably this commit also includes support for a rotatable secret which includes
support for a configurable grace period where the elder value (after rotation)
is still accessible. This allows a time bound leniency where both values are
valid at the same time to help mitigate tightly coupled systems rotations.
The rotatable secret currently only supports checking if it is set or it matches
an external secret. However, future updates will accept a function that
can be be used as input to a 3rd party system that can try the current secret
but automatically fall back to the prior secret if that fails during the grace period.
The implementation of rotatable secret uses a StampedLock with optimistic
reads to help ensure minimal performance impact for the reading and expiry
of the secret.
2023-09-20 09:56:36 -05:00
Felix Barnsteiner ebd5ead943
Remove ineffective options of preventing mapping explosions (#99665)
Removes the recommendations to use the object field type and to set index: false.
Both of these options are not effective with avoiding mapping explosions.
2023-09-20 13:59:03 +02:00
Felix Barnsteiner 3a7bdb5838
Make reroute processor GA (#99531) 2023-09-20 13:22:36 +02:00
Bogdan Pintea 34eea49ef5
ESQL: Swap arguments of remaining date_xxx() functions (#99561)
This swaps the argument of `date_extract()`, `date_format()` and
`date_parse()` functions, to align with `date_trunc()`. The field
argument is now always last, even for _format() and _parse(), whose
optional argument will now be provided as the first one.
2023-09-19 20:22:34 +02:00
Jake Landis 369c5f4926
Fix doc for deprecated TLS settings (#98513) 2023-09-19 11:11:54 -05:00
Ignacio Vera 4bc1afddda
Move Aggregator#buildTopLevel() to search worker thread. (#98715)
This commit introduces an AggregatorCollector that contains a finish method which performs aggregation 
post-collection and builds the internal aggregation for this collector. This method is called on the worker 
thread at the end of the collection phase.
2023-09-19 09:46:51 +02:00
James Rodewig 88e862fa64
[main] [DOCS] Add 8.10.1 release notes (#99634) (#99636) 2023-09-18 13:47:47 -04:00
gheorghepucea d58b9ea87d
Added esql ends_with implementation (#99613)
Added an implementation for `ends_with` function in esql.  `ends_with` -
Returns a boolean that indicates whether a keyword string ends with
another string. Also made sure that the docs look alright: 

<img width="1677" alt="Screenshot 2023-09-16 at 18 10 46"
src="https://github.com/elastic/elasticsearch/assets/91881042/eccd81e1-40a2-4a66-a514-cf3e4205f9da">
2023-09-18 11:29:20 -04:00
David Turner 3691312aca
Slightly adjust docs about S3 incompatibilities (#99624)
It's often useful to quote these docs to users encountering problems
with their not-quite-S3-compatible storage system. In practice we don't
need to quote the bits in the middle but we do need the last sentence
about working with the supplier to address incompatibilities. This
commit reorders things so that the most commonly quoted sentences form a
standalone paragraph.
2023-09-18 08:26:03 -04:00
John Verwolf b5ea3560f1
Forward port 8.10.0 docs (#99505) 2023-09-14 07:18:02 -07:00
Simon Cooper 8726f1653f
Add IndexVersion to node info (#99515)
This adds index_version field to node info, so we can read it in bwc tests to determine the index version we are upgrading from
2023-09-14 10:09:51 +01:00
Kostas Krikellas b1da97af17
Document how to reindex a TSDS (#99476)
* Document how to reindex a TSDS

Time-series data streams require updating start and end times in the
destination index template, to avoid errors during copying of older
docs.

* Update docs/changelog/99476.yaml

* Spotless fix.

* Refresh indexes in unittest.

* Fix typo.

* Delete docs/changelog/99476.yaml

* Fix page link name.

* Update docs/reference/data-streams/tsds-reindex.asciidoc

Co-authored-by: Abdon Pijpelink <abdon.pijpelink@elastic.co>

* Update docs/reference/data-streams/tsds-reindex.asciidoc

Co-authored-by: Abdon Pijpelink <abdon.pijpelink@elastic.co>

* Update docs/reference/data-streams/tsds-reindex.asciidoc

Co-authored-by: Abdon Pijpelink <abdon.pijpelink@elastic.co>

* Update docs/reference/data-streams/tsds-reindex.asciidoc

Co-authored-by: Abdon Pijpelink <abdon.pijpelink@elastic.co>

* Update docs/reference/data-streams/tsds-reindex.asciidoc

Co-authored-by: Abdon Pijpelink <abdon.pijpelink@elastic.co>

* Update docs/reference/data-streams/tsds-reindex.asciidoc

Co-authored-by: Abdon Pijpelink <abdon.pijpelink@elastic.co>

* Update docs/reference/data-streams/tsds-reindex.asciidoc

Co-authored-by: Abdon Pijpelink <abdon.pijpelink@elastic.co>

* Update docs/reference/data-streams/tsds-reindex.asciidoc

Co-authored-by: Abdon Pijpelink <abdon.pijpelink@elastic.co>

* Update docs/reference/data-streams/tsds-reindex.asciidoc

Co-authored-by: Abdon Pijpelink <abdon.pijpelink@elastic.co>

* Update docs/reference/data-streams/tsds-reindex.asciidoc

Co-authored-by: Abdon Pijpelink <abdon.pijpelink@elastic.co>

---------

Co-authored-by: Abdon Pijpelink <abdon.pijpelink@elastic.co>
2023-09-13 18:28:03 +03:00
James Rodewig 9da1022461
[DOCS] Fix `welcome-to-elastic` links (#99530)
**Problem:** In https://github.com/elastic/docs/pull/2752, we updated the URL prefix (`welcome-to-elastic`) and name for the "Welcome to Elastic Docs" docs. However, we still have some stray links that use the old `/welcome-to-elastic` URL prefix

**Solution:** Updates several outdated links.
2023-09-13 11:09:54 -04:00
David Pilato 7064bc9e5c
Generated field is `ml.tokens` (#99049)
The generated field name is `ml.tokens` and not `ml-tokens`.
2023-09-13 15:21:27 +02:00
James Rodewig daddca75d1
[DOCS] Fix Usage API snippet test (#99497) 2023-09-12 15:53:55 -04:00
James Rodewig 255c9a7f95
[DOCS] Move x-pack docs to `docs/reference` dir (#99209)
**Problem:**
For historical reasons, source files for the Elasticsearch Guide's security, watcher, and Logstash API docs are housed in the `x-pack/docs` directory. This can confuse new contributors who expect Elasticsearch Guide docs to be located in `docs/reference`. 

**Solution:**
- Move the security, watcher, and Logstash API doc source files to the `docs/reference` directory
- Update doc snippet tests to use security

Rel: https://github.com/elastic/platform-docs-team/issues/208
2023-09-12 14:53:41 -04:00
Nik Everett 0d8a1975a9
ESQL: Fix test for unsigned long (#99441)
We were generating negative values which made the tests confused.
2023-09-12 11:46:09 -04:00
Abdon Pijpelink 54f6e4f51b
[DOCS] Remove 'coming in 8.10' from remote cluster API key auth docs (#99462) 2023-09-12 13:25:56 +02:00
Paweł Krześniak 30f6e51804
Update ES|QL (#99467)
To make it more clear let's use different index names for comma-separated index list
2023-09-12 13:13:38 +02:00
Joe Gallo 404aa77849
Fix a typo in the data_stream _stats API documentation (#99438) 2023-09-11 15:13:50 -04:00
Nik Everett 44c3cde48c ESQL: Fix compile
Two PRs cross in the night. Then nothing compiles.
2023-09-11 14:35:05 -04:00
Nik Everett 936e69ddd5
ESQL: Yet more function tests and docs (#99009)
This adds tests, supported types, and a signature image for `to_string`
and `to_version`. It also fixes the resolution of functions who's names
contain an `_`

Finally, it updates the docs for `ceil` to render the image more nicely.
2023-09-11 14:10:17 -04:00
James Rodewig b3646595f1
[DOCS] Add `docker-verify-signature` anchor to Docker docs (#99431)
**Problem:**

Other Elastic doc sets ([Beats](https://www.elastic.co/guide/en/beats/auditbeat/current/running-on-docker.html), [APM](https://www.elastic.co/guide/en/apm/guide/current/running-on-docker.html)) link to the [Verify the Elasticsearch Docker image signature](https://www.elastic.co/guide/en/elasticsearch/reference/8.9/docker.html#docker-verify-signature) section of the Docker docs. This section was removed as part of https://github.com/elastic/elasticsearch/pull/99371. When we bump to the next version, this will create broken links and break the docs build.

**Solution:**
Re-add the anchor so we don't create broken links or break the docs build.

This commit was added to the backports of https://github.com/elastic/elasticsearch/pull/99371:

- https://github.com/elastic/elasticsearch/pull/99429
- https://github.com/elastic/elasticsearch/pull/99430
2023-09-11 12:10:01 -04:00
James Rodewig 22371de7c9
[DOCS] Streamline Docker Compose docs (#99371)
**Problem**:
The [Docker Compose docs](https://www.elastic.co/guide/en/elasticsearch/reference/current/docker.html#docker-compose-file) are overly long. They currently display the entire length of related `.env` and `docker-compose.yml` files.

**Solution**:
- Rewrite the Docker Compose docs as a procedural with ordered steps.
- Provide download links for the `.env` and `docker-compose.yml` files rather than display them.
- Move info about pulling and verifying Docker images into the [Run Elasticsearch in Docker docs](https://www.elastic.co/guide/en/elasticsearch/reference/current/docker.html#docker-cli-run-dev-mode). These steps don't apply to the Docker Compose docs.

Closes https://github.com/elastic/platform-docs-team/issues/186
Depends on https://github.com/elastic/enterprise-search-pubs/pull/3788
2023-09-11 11:05:08 -04:00
Abdon Pijpelink 91759ce592
[DOCS] Some minor ES|QL docs fixes (#99423) 2023-09-11 16:20:10 +02:00
Benjamin Trent 83b70e37ef
Revert "Auto-normalize dot_product vectors at index & query (#98944)" (#99421)
This reverts commit 7b9c367aeb.
2023-09-11 09:33:17 -04:00
Johannes Mahne 4aa0c698f2
Update ilm-shrink.asciidoc (#99366)
Formatting
2023-09-11 12:56:34 +02:00
Abdon Pijpelink 2ef7da53e3
[DOCS] Set up redirect for old anchor (#99401) 2023-09-11 10:13:24 +02:00
dreamquster 04381664c1
ESQL: Implement 'right' function (#98974)
Add the 'right' function, which extracts a substring beginning from its
right end (opposite function of 'left').
---------

Co-authored-by: Alexander Spies <alexander.spies@elastic.co>
2023-09-08 17:27:59 +02:00
Abdon Pijpelink af76a3a436
[DOCS] Add 'Troubleshooting an unstable cluster' to nav (#99287)
* [DOCS] Add 'Troubleshooting an unstable cluster' to nav

* Adjust docs links in code

* Revert "Adjust docs links in code"

This reverts commit f3846b1d78.

---------

Co-authored-by: David Turner <david.turner@elastic.co>
2023-09-08 13:42:50 +02:00
Martijn van Groningen 3e3ee42589
Add index.look_back_time setting for tsdb data streams (#98518)
This change adds a `index.look_back_time` index setting that sets the `index.time_series.start_time` setting for the first backing index when a data stream is created.

This allows accepting data that is older for initial indexing without changing the `index.look_ahead_time` setting. This setting also controls the `index.time_series.end_time` setting and would affect rollovers as well.

The default for the `index.look_back_time` is `2h`, which means documents with `@timestamp` up to 2 hours after creation of the data stream are allowed to be indexed. This is the same as is without this change, because `index.look_ahead_time` is used to set `index.time_series.start_time` of the first backing index.

Closes #98463
2023-09-08 11:11:43 +02:00
Stef Nestor 61228b888e
[DOC+] ILM Searchable Snapshot migrations require repository "name" to be the same (#99308)
👋 howdy, team! Expanding reference to [internal](https://github.com/elastic/cloud/pull/118105) update, we've just confirmed ILM requires the repository name to be the same among migrating clusters. This is a hard block for Searchable Snapshots which requires un-Searchable-Snapshotting or redoing migration to resolve.
2023-09-08 10:28:30 +02:00
Marius Iversen 4b41b17772
Update documentation for Set Processor (#99191) 2023-09-07 14:47:07 -04:00
Craig Taverner 9f5ff0cfda
Forward port 8.9.2 release notes (#99323) 2023-09-07 20:35:32 +02:00
James Rodewig 87195052d7
[DOCS] Fix syntax errors in Docker docs (#99313)
**Problem:**
- The `elasticsearch-reset-password` commands in the ES Docker install docs are missing the required `-u` flag
- The `ifeval` blocks in the Kibana section of the ES Docker install docs aren't rendering correctly in released docs.


**Solution:**
- Add the `-u` flag to `elasticsearch-reset-password` examples
- Fix the Asciidoc syntax to correctly render the `ifeval` blocks. Example:

Rel: https://github.com/elastic/elasticsearch/pull/99112
2023-09-07 12:19:25 -04:00
Ioana Tagirta 7cd1987e5d
Make _index optional for pinned query docs (#97450)
Currently pinned queries require either the `ids` or `docs` parameter.
`docs` allows pinning documents from specific indices. However for
`docs` the `_index` field is always required:

```
GET test/_search
{
  "query": {
    "pinned": {
      "organic": {
        "query_string": {
          "query": "something"
        }
      },
      "docs": [
        { "_id": "1" }
      ]
    }
  }
}
```

returns an error:

```
{
  "error": {
    "root_cause": [
      {
        "type": "parsing_exception",
        "reason": "[10:22] [pinned] failed to parse field [docs]",
        "line": 10,
        "col": 22
      }
    ],
    "type": "parsing_exception",
    "reason": "[10:22] [pinned] failed to parse field [docs]",
    "line": 10,
    "col": 22,
    "caused_by": {
      "type": "x_content_parse_exception",
      "reason": "[10:22] [pinned] failed to parse field [docs]",
      "caused_by": {
        "type": "illegal_argument_exception",
        "reason": "Required [_index]"
      }
    }
  },
  "status": 400
}
```

The proposal here is to make `_index` optional. I don't think we have a
strong requirement for making `_index` required, when it was initially
introduced in https://github.com/elastic/elasticsearch/pull/74873, we
mostly wanted the ability to pin docs from specific indices.

Making `_index` optional can give more flexibility to use a combination
of pinned documents from specific indices or just document ids. This
change can also help with pinned query rules. Currently pinned query
rules can accept either `ids` or `docs`. If multiple pinned query rules
match and they use a combination of `ids` and `docs`, we cannot build a
pinned query and we would need to return an error. This is because a
pinned query cannot accept both `ids` and `docs`. By making `_index`
optional we would no longer need to return an error when pinned query
rules use a combination of `ids` and `docs`, because we can easily
translate `ids` in `docs`.

The following pinned queries would be equivalent:

```
GET test/_search
{
  "query": {
    "pinned": {
      "organic": {
        "query_string": {
          "query": "something"
        }
      },
      "docs": [
        { "_id": "1" }
      ]
    }
  }
}

GET test/_search
{
  "query": {
    "pinned": {
      "organic": {
        "query_string": {
          "query": "something"
        }
      },
      "ids": [1]
    }
  }
}
```

The scores should be consistent when using a combination of _docs that
might use _index or not - see example

<details>   <summary>Example </summary>

```

PUT test-1/_doc/1 {   "title": "doc 1" }

PUT test-1/_doc/2 {   "title": "doc 2" }

PUT test-2/_doc/1 {   "title": "doc 1" }

PUT test-2/_doc/3 {   "title": "lalala" }

POST test-1,test-2/_search {   "query": {     "pinned": {      
"organic": {         "query_string": {           "query": "lalala"      
}       },       "docs": [         { "_id": "2", "_index": "test-1" },  
{ "_id": "1" }       ]     }   } }

```

response:

```

{   "took": 1,   "timed_out": false,   "_shards": {     "total": 2,    
"successful": 2,     "skipped": 0,     "failed": 0   },   "hits": {    
"total": {       "value": 4,       "relation": "eq"     },    
"max_score": 1.7014124e+38,     "hits": [       {         "_index":
"test-1",         "_id": "2",         "_score": 1.7014124e+38,        
"_source": {           "title": "doc 2"         }       },       {      
"_index": "test-1",         "_id": "1",         "_score": 1.7014122e+38,
// same score as doc with id 1 from test-2         "_source": {         
"title": "doc 1"         }       },       {         "_index": "test-2", 
"_id": "1",         "_score": 1.7014122e+38, // same score as doc with
id 1 from test-1         "_source": {           "title": "doc 1"        
}       },       {         "_index": "test-2",         "_id": "3",      
"_score": 0.8025915, // organic result         "_source": {          
"title": "lalala"         }       }     ]   } }

```

</details>

For query rules, if we have two query rules that both match and use a
combination of `ids` and `pinned`:

```
PUT _query_rules/test-ruleset
{
  "ruleset_id": "test-ruleset",
  "rules": [
    {
      "rule_id": "1",
      "type": "pinned",
      "criteria": [
        {
          "type": "exact",
          "metadata": "query_string",
          "value": "country"
        }
      ],
      "actions": {
        "docs": [
          { "_index": "singers", "_id": "1" }
        ]
      }
    },
    {
      "rule_id": "2",
      "type": "pinned",
      "criteria": [
        {
          "type": "exact",
          "metadata": "query_string",
          "value": "country"
        }
      ],
      "actions": {
        "ids": [
          2
        ]
      }
    }
  ]
}
```

and the following query:

```
POST singers/_search
{
    "query": {
        "rule_query": {
            "organic": {
                "query_string": {
                  "default_field": "name",
                  "query": "country"
                }
            },
            "match_criteria": {
                "query_string": "country"
            },
            "ruleset_id": "test-ruleset"
        }
    }
}
```

then this would get translated into the following pinned query:

```
POST singers/_search
{
    "query": {
        "pinned": {
            "organic": {
                "query_string": {
                  "default_field": "name",
                  "query": "country"
                }
            },
            "docs": [
               { "_index": "singers", "_id": "1" },
               {"_id": 2 }
            ]
        }
    }
}
```

I think we can also simplify the pinned query rule so that it only
receives `docs`:

```
PUT _query_rules/test-ruleset
{
  "ruleset_id": "test-ruleset",
  "rules": [
    {
      "rule_id": "1",
      "type": "pinned",
      "criteria": [
        {
          "type": "exact",
          "metadata": "query_string",
          "value": "country"
        }
      ],
      "actions": {
        "docs": [
          { "_id": "1" },
          { "_id": "2", "_index": "singers" }
        ]
      }
    }
  ]
}
```
2023-09-07 04:39:56 -04:00
Kathleen DeRusso 258d0cb0be
Automatically map floats as dense vector (#98512) 2023-09-06 16:06:29 -04:00
James Rodewig 43abd92b37
[DOCS] Add Docker instructions for Kibana (#99112)
**Problem:**
The [Kibana Docker install docs](https://www.elastic.co/guide/en/kibana/master/docker.html) duplicate the Elasticsearch Docker instructions. This makes the two doc sets harder to maintain. For example, the docs currently use different container names, which makes them incompatible.

**Solution:**
Adds Kibana setup instructions to the Elasticsearch Docker install docs. This will let us eventually merge the two doc sets.

**Issues:**
Rel: https://github.com/elastic/platform-docs-team/issues/182
2023-09-06 07:50:51 -04:00
Abdon Pijpelink 0421c4fe9b
[DOCS] Remote cluster troubleshooting guide (#99128)
* [DOCS] Remote cluster troubleshooting guide

* Fix test failures

* Apply suggestions from code review

Co-authored-by: Yang Wang <ywangd@gmail.com>

* Review feedback

* Group issues under 'common' and 'API key'

* Apply suggestions from code review

Co-authored-by: Yang Wang <ywangd@gmail.com>

---------

Co-authored-by: Yang Wang <ywangd@gmail.com>
2023-09-05 15:10:45 +02:00
Reza Torabi 310af09b6a
Add persian language stemmer (#99106) 2023-09-05 12:13:27 +01:00
Saikat Sarkar c9a2555f09
[Search Relevance] GET search application API should return the alias indices (#99029)
* Add indices to GET search_application endpoint

* Update x-pack/plugin/ent-search/src/main/java/org/elasticsearch/xpack/application/search/SearchApplication.java

Co-authored-by: Carlos Delgado <6339205+carlosdelest@users.noreply.github.com>

* Update x-pack/plugin/ent-search/src/main/java/org/elasticsearch/xpack/application/search/SearchApplication.java

Co-authored-by: Carlos Delgado <6339205+carlosdelest@users.noreply.github.com>

* Update x-pack/plugin/ent-search/src/main/java/org/elasticsearch/xpack/application/search/SearchApplication.java

Co-authored-by: Carlos Delgado <6339205+carlosdelest@users.noreply.github.com>

* Add indices to GET search_application endpoint

---------

Co-authored-by: Carlos Delgado <6339205+carlosdelest@users.noreply.github.com>
2023-09-04 16:43:38 -06:00
István Zoltán Szabó f5dc68abc6
[DOCS] Fine-tunes the reindexing step of the ELSER tutorial. (#99155) 2023-09-04 11:04:58 +02:00
Yang Wang ebe4fe9f15
[Doc] Add links to the new API key based remote cluster page (#99115)
This PR adds links to the new API key based remote cluster page in
multiple places.

Relates: #98330
2023-09-01 06:08:49 -04:00
Abdon Pijpelink 4f1bf97776
[DOCS] Expand the step that enables the remote cluster server (#99084)
* [DOCS] Expand the step that enables the remote cluster server

* Update docs/reference/modules/cluster/remote-clusters-api-key.asciidoc

* Reword

* Reword
2023-09-01 10:35:46 +02:00
Jonathan Buttner 1ca66bde91
[ML] Safely drain deployment request queues before allowing node to shutdown (#98406)
* isSafeToShutdown checks routing table

* Rebalancer changes and tests

* Update docs/changelog/98406.yaml

* Forcing lifecycle tests to avoid over time case

* Changes and remaining tests

* Adding node service changes

* Finishing unit tests

* Adding wait for completion paramater

* Adding stop deployment integration tests

* Cleaning up code

* Fixing stop deployment test

* Fixing string formatter issue and timeout

* Investigating deadlock

* More testing

* More logging

* Prevent model reloading while stopping

* Fixing compile error

* More code clean up

* Adding test for loading model after stopping

* Addressing review feedback

* Fixing a couple shutdown -> shutdownNow tests

* Adding doc changes and refactoring
2023-08-31 15:37:11 -04:00
Joe Gallo 3284903205
Document the redact processor's skip_if_unlicensed option (#99063) 2023-08-31 14:00:12 -04:00
Michael Peterson 649821e992
Support cluster/details for CCS minimize_roundtrips=false (#98457)
This commit tracks progress for each shard search by cluster alias
using a new SearchProgressListener (CCSSingleCoordinatorSearchProgressListener).
Both sync and async CCS searches use this new progress listener when
minimize_roundtrips=false.

Two of the SearchProgressListener method had to be extended to allow tracking
per-cluster took values (TransportSearchAction.SearchTimeProvider) and
whether searches timed out (by passing in QuerySearchResult to the onQueryResult
listener method).

This commit brings parity between minimize_roundtrips=true and false to have
the same _cluster/details sections in CCS search responses.

Note that there are still a few differences between minimize_roundtrips=true and false.
1. The per-cluster took value for minimize_roundtrips=true is accurate, but the
   for 'false' it is only measured at the granualarity of each partial reduce,
   so the per cluster took time is overestimated in basically all cases.
2. For minimize_roundtrips=true, a skip_unavailable=false cluster that disconnects
   during the search or has all searches on all shards fail, will cause the entire
   search to fail. This is (still) not true for minimize_roundtrips=false. The search
   is only failed if the skip_unavailable=false cluster cannot be connected to at the
   start of the search. (This will likely be changed in a follow up ticket that implements
   fail-fast logic for in-progress searches that should fail due to a skip_unavailable=true
   cluster failing.)
3. The shard accounting for minimize_roundtrips=false is always accurate (total shard counts
   are known at the start of the search). For minimize_roundtrips=true, the shard accounting
   is only accurate per cluster unless all clusters have successful (or partially successful)
   searches. For clusters that have failures we do not have shard count info.
2023-08-31 12:56:20 -04:00
Leandro de Lima Camargo e31eed37c5 Fix voting_only:true in node spec docs (#99099) 2023-08-31 17:41:50 +01:00
Stef Nestor 9573f6f1f4
[DOC+] snapshot-restore single index example (#99065)
* [DOC+] snapshot-restore single index example

👋🏼 howdy, team! I'd like to append an example to snapshot-restore a single index. Support usually points users to [this page](https://www.elastic.co/guide/en/elasticsearch/reference/master/restore-snapshot-api.html) but then users attempt the `rename_pattern` example (which makes sense!). I'd like to point them to a more literal "close index > restore on that index" example in the future.

* Fix test failure and reword

---------

Co-authored-by: Abdon Pijpelink <abdon.pijpelink@elastic.co>
2023-08-31 16:23:50 +02:00
Tim baa4e5d6ab
[DOCS] Deduplicate the JVM sizing instructions in Docker documentation (#99079)
- Deduplicates the JVM sizing instructions
- Links the `-m` tag tip section to the JVM sizing instructions
- Replaces the fully typed out container registry to {docker-image}

Co-authored-by: James Rodewig <james.rodewig@elastic.co>
2023-08-31 08:34:53 -04:00
Abdon Pijpelink 792f9c1647
[DOCS] Remote cluster migration guide (#98999)
* [DOCS] Remote cluster migration guide

* Review feedback

* Clarify that any extra local privileges will be suppressed by the cross-cluster API key’s privileges
2023-08-31 10:24:20 +02:00
Nik Everett b73cc0c529
ESQL: Only generate syntax diagrams locally (#99059)
CI will skip building them. Lot's of CI machines don't have font support
so they can't generate these. But all local machine have a GUI so they
can.

Also, super-lazy initialize the font so CI don't bump into it by
accident.

Closes #99018
2023-08-30 14:44:14 -04:00
Kyle Carter 8ea41ddf1d
Fix link to Lucene docs for TrimFilter (#99050) 2023-08-30 16:34:54 +02:00
Benjamin Trent 7b9c367aeb
Auto-normalize dot_product vectors at index & query (#98944)
`dot_product` requires vectors to be unit-length. Previously, we would
check that vectors were unit-length and throw if they were not. 

Instead, we will now auto-normalize vectors as they are indexed.

`cosine` will continue to behave as usual, not normalizing the vectors.

closes: https://github.com/elastic/elasticsearch/issues/98935
2023-08-30 09:50:49 -04:00
Joe Gallo a354be7b69
Fix a couple of docs typos (#99028) 2023-08-30 09:01:34 -04:00
Bogdan Pintea 9beddb18d6
ESQ: Add metadata fields documentation (#98991)
This adds documentation for the supported metadata fields.

---------

Co-authored-by: Abdon Pijpelink <abdon.pijpelink@elastic.co>
2023-08-30 12:51:31 +02:00
Abdon Pijpelink b6c85dbe82
[DOCS] Update ES|QL Kibana instructions (#98940) 2023-08-30 11:18:49 +02:00
Liam Thompson dfbec46c3d
[Docs] Add link to labs from semantic search overview (#98985) 2023-08-30 10:54:24 +02:00
Stef Nestor 0781bafac1
[DOC+][Hot Spotting] Pull detailed Node Tasks (#98879)
Co-authored-by: David Turner <david.turner@elastic.co>
2023-08-29 14:25:10 -04:00
Stef Nestor de380ea2af
[DOC+] Write threadpool also covers ingest pipelines (#99010)
Co-authored-by: James Rodewig <james.rodewig@elastic.co>
2023-08-29 13:51:18 -04:00
Liam Thompson a3c96caa51
[DOCS] Add link to Elasticsearch labs ELSER Python notebook (#98983)
* Add link to Elasticsearch labs ELSER Python notebook

* Fix typos

* Use {es} variable

Co-authored-by: István Zoltán Szabó <istvan.szabo@elastic.co>

---------

Co-authored-by: István Zoltán Szabó <istvan.szabo@elastic.co>
2023-08-29 15:26:00 +02:00
James Rodewig 1301995a46
[DOCS] Remove unneeded `ifeval`s from install docs (#98952)
Problem: The current install docs contain several `ifeval` statements that hide commands and code snippets based on the branch's release state. These statements make the Asciidoc hard to read  and maintain. It also makes doc changes difficult to preview.

Solution: Remove `ifeval` statements that hide commands or code snippets. Leave in any `ifeval` statements used to add warnings.
2023-08-29 07:56:10 -04:00
James Rodewig 73293dca86
[DOCS] Remove unused widgets (#98953)
No-op change to remove a few unused Asciidoc widgets. These aren't referenced or published anywhere in the docs.
2023-08-29 07:53:26 -04:00
dreamquster 2644ccbb8a
Implement the 'left' function in issue #98545 (#98942)
@nik9000  Recheck out the main branch. Refactor the 'left' function to
cut the prefix string in place. But I meet a adversity that left failed
the test case 'testEvaluateInManyThreads'. I find that in multiple
thread situation,  `  EvalOperator.ExpressionEvaluator eval =
evalSupplier.get(); for (int c = 0; c < count; c++) {      
assertThat(toJavaObject(eval.eval(page), 0), testCase.getMatcher()); } `
toJavaObject function return a BytesRef with length=2, content is
[81,89]. However, assertThat function in junit4 receive the BytesRef
parameters that its length is 10. Can you give me some clues? I can't
find which variable is mutual.

Rerun failed test case's command: `gradlew ':x-pack:plugin:esql:test'
--tests
"org.elasticsearch.xpack.esql.expression.function.scalar.string.LeftTests.testEvaluateInManyThreads
{TestCase=Left basic test}" -Dtests.seed=44459C172243712
-Dtests.locale=lv-LV -Dtests.timezone=Asia/Irkutsk -Druntime.java=20`
2023-08-28 13:17:16 -04:00
Nik Everett 4cd7f40712
ESQL: More docs (#98890)
Adds some more typed docs to the ESQL functions.
2023-08-28 11:17:04 -04:00
James Rodewig 29e2e6f141
[DOCS] Simplify single-node Docker instructions (#98886)
Problem:
The current [single-node Docker instructions](https://www.elastic.co/guide/en/elasticsearch/reference/current/docker.html#docker-cli-run-dev-mode) are overly verbose and contain a lot of unneeded info about security internals. 

Solution:
- Restructure the above docs to focus primarily on actionable steps.
- Test the docs across operating systems (Mac, Linux, Windows) to ensure they work.
2023-08-28 08:54:30 -04:00
James Rodewig 31408bddc1
[DOCS] Add $ELASTIC_PASSWORD env var to install docs (#98898)
- Removes duplicated security autoconfiguration output from the docs. This is difficult to keep updated and makes the docs longer.
- Encourages the user to store the `elastic` password as an environment variable. Users don't need to rely on curl's password prompts.
- Removes unused `api-call-widget` files. These aren't published anywhere in the docs currently.
2023-08-28 08:25:13 -04:00