Commit Graph

71895 Commits

Author SHA1 Message Date
James Rodewig 255c9a7f95
[DOCS] Move x-pack docs to `docs/reference` dir (#99209)
**Problem:**
For historical reasons, source files for the Elasticsearch Guide's security, watcher, and Logstash API docs are housed in the `x-pack/docs` directory. This can confuse new contributors who expect Elasticsearch Guide docs to be located in `docs/reference`. 

**Solution:**
- Move the security, watcher, and Logstash API doc source files to the `docs/reference` directory
- Update doc snippet tests to use security

Rel: https://github.com/elastic/platform-docs-team/issues/208
2023-09-12 14:53:41 -04:00
Keith Massey 8685c8afe3
Updating data streams module to use new rest test framework (#99391) 2023-09-12 13:38:14 -05:00
Mark Vieira b2f4534851
Prune changelogs after 8.10.0 release 2023-09-12 10:08:06 -07:00
Mark Vieira ba5b079fa6
Bump versions after 8.10.0 release 2023-09-12 10:07:42 -07:00
David Turner b48640c8bf
Fork computation in TransportGetShutdownStatusAction (#99490)
This action does O(#shards) work so it must not happen on a transport
worker. Also it can take minutes to complete in a huge cluster, so it
really should react to cancellations properly.

Closes #99487
2023-09-12 12:59:17 -04:00
Nik Everett 0d8a1975a9
ESQL: Fix test for unsigned long (#99441)
We were generating negative values which made the tests confused.
2023-09-12 11:46:09 -04:00
William Brafford b5e06da143
Add mappings versions to CompatibilityVersions (#99307)
CompatibilityVersions now holds a map of system index names to their
mappings versions, alongside the transport version. We also add mapping
versions to the "minimum version barrier": if a node has a system index
whose version is below the cluster mappings version for that system
index, it is not allowed to join the cluster.
2023-09-12 11:16:55 -04:00
Benjamin Trent 2072be90b2
Utilize optimized dot_product where possible when calculating vector magnitude (#99448)
Lucene provides an optimized `dot_product` calculation for vectors. We
should use that when calculating a vector's magnitude.
2023-09-12 10:05:29 -04:00
Mark Tozzi e26dca469d
[ESQL] Plumb through ranges and warnings for the casting to double tests (#99452)
Add the warnings and range checking parameters to unary and binary casting to double test generators. I also moved the data type to the value supplier, which the binary case needed. That feels more right - that's what I was intending with TypedData to begin with, but our abstractions are still messy here.
2023-09-12 08:12:20 -04:00
Milton Hultgren 403bcb366a
Update CODEOWNER paths for Stack Monitoring mappings (#99428) 2023-09-12 14:11:30 +02:00
Abdon Pijpelink 54f6e4f51b
[DOCS] Remove 'coming in 8.10' from remote cluster API key auth docs (#99462) 2023-09-12 13:25:56 +02:00
Paweł Krześniak 30f6e51804
Update ES|QL (#99467)
To make it more clear let's use different index names for comma-separated index list
2023-09-12 13:13:38 +02:00
Andrei Stefan 22a2cd0a02
More deterministic tests (#99469) 2023-09-12 14:04:58 +03:00
Yang Wang 16a4e542f0
[Test] More robust order of assertions (#99461)
This PR adjusts the order of assertions to ensure we are done with the
atomic reference variable in the previous assertions before changing it
to null.

Resolves: #99406
2023-09-12 05:24:45 -04:00
Kostas Krikellas ae2aacb66a
Disable FilterByFilterAggregator through ClusterSettings (#99417)
`search.aggs.rewrite_to_filter_by_filter` allows disabling
FilterByFilterAggregator when used in terms and range aggregation. The
same should apply to filter aggregation.

Fixes #99335
2023-09-12 04:54:54 -04:00
Yang Wang 4c8888de9e
Do not report failure after connections are made (#99117)
Today, when the number of attempts is exhausted, ProxyConnectionStrategy
checks the number of connections before returns. It reports connection
failure if the number of connections is zero at the time of checking.
However, this behaviour is incorrect. In rare cases, a connection can be
dropped right after it is initially established and before the number
checking. From the perspective of the `openConnections` method, it
should not care whether or when opened connections are subsequently
closed. As long as connections have been initially established, it
should report success instead of failure. 

This PR adjusts the code to report success in above situation.

Relates: #94998 Resolves: #99113
2023-09-12 04:08:05 -04:00
David Turner 4ee229779b
Clean up delete code in S3BlobContainer (#99447)
Simplifies things using utils from `Iterators` that didn't exist when
the code was first written.
2023-09-12 07:16:39 +01:00
Andrei Stefan f241f2bb6a
ESQL: Make the stats test more deterministic (for multi-node testing) (#99451) 2023-09-12 01:22:27 +03:00
Andrei Stefan 6602b6c726
ESQL: create a Vector when needed for IN (#99382) 2023-09-12 00:22:11 +03:00
Brian Seeders 590f27a5cb
Fix pull-requests.json 2023-09-11 16:25:43 -04:00
Brian Seeders 6f90fd7ecf
[buildkite] Add buildkite pr-bot config for triggering builds with opt-in label (#99446) 2023-09-11 16:18:48 -04:00
Nik Everett a7617db23d
Drop changelog (#99436)
We don't want it.
2023-09-11 15:19:43 -04:00
Joe Gallo 404aa77849
Fix a typo in the data_stream _stats API documentation (#99438) 2023-09-11 15:13:50 -04:00
Nik Everett 44c3cde48c ESQL: Fix compile
Two PRs cross in the night. Then nothing compiles.
2023-09-11 14:35:05 -04:00
Nik Everett 936e69ddd5
ESQL: Yet more function tests and docs (#99009)
This adds tests, supported types, and a signature image for `to_string`
and `to_version`. It also fixes the resolution of functions who's names
contain an `_`

Finally, it updates the docs for `ceil` to render the image more nicely.
2023-09-11 14:10:17 -04:00
Rene Groeschke 72de108b97
Fix check for restricting invoking ecs tests directly (#99435) 2023-09-11 13:25:42 -04:00
Nik Everett f9107e34c9
ESQL: Disable optimizations with bad null handling (#99434)
* ESQL: Disable optimizations with bad null handling

We have optimizations that kick in when aggregating on the following
pairs of field types:
* `long`, `long`
* `keyword`, `long`
* `long`, `keyword`

These optimizations don't have proper support for `null` valued fields
but will grow that after #98749. In the mean time this disables them in
a way that prevents them from bit-rotting.

* Update docs/changelog/99434.yaml
2023-09-11 13:18:23 -04:00
Mark Tozzi 6ab6b237c6
[ESQL] Add the ability to assert warnings in the new test generation logic (#99381)
This extends the test case generation functions to take expected warnings, and demonstrates the use of that functionality by testing expected nulls for log10. We can build on this to get proper null handling and tests for the rest of the math functions.

Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
2023-09-11 13:02:38 -04:00
James Rodewig b3646595f1
[DOCS] Add `docker-verify-signature` anchor to Docker docs (#99431)
**Problem:**

Other Elastic doc sets ([Beats](https://www.elastic.co/guide/en/beats/auditbeat/current/running-on-docker.html), [APM](https://www.elastic.co/guide/en/apm/guide/current/running-on-docker.html)) link to the [Verify the Elasticsearch Docker image signature](https://www.elastic.co/guide/en/elasticsearch/reference/8.9/docker.html#docker-verify-signature) section of the Docker docs. This section was removed as part of https://github.com/elastic/elasticsearch/pull/99371. When we bump to the next version, this will create broken links and break the docs build.

**Solution:**
Re-add the anchor so we don't create broken links or break the docs build.

This commit was added to the backports of https://github.com/elastic/elasticsearch/pull/99371:

- https://github.com/elastic/elasticsearch/pull/99429
- https://github.com/elastic/elasticsearch/pull/99430
2023-09-11 12:10:01 -04:00
eyalkoren c43f83d88c
[CI] Adding continuous testing for ECS dynamic templates (#97901)
* Enforcing formatting
* Add proper reporting and assertions
* Verify ECS multi-field mappings
* Test flattened field mappings
* Add ecs dynamic template build job
* Setup notifications for failing ecs template tests

---------

Co-authored-by: Rene Groeschke <rene@elastic.co>
2023-09-11 17:26:26 +02:00
David Turner 9f12658748
Small simplification to MockUtils (#99398) 2023-09-11 16:16:09 +01:00
James Rodewig 22371de7c9
[DOCS] Streamline Docker Compose docs (#99371)
**Problem**:
The [Docker Compose docs](https://www.elastic.co/guide/en/elasticsearch/reference/current/docker.html#docker-compose-file) are overly long. They currently display the entire length of related `.env` and `docker-compose.yml` files.

**Solution**:
- Rewrite the Docker Compose docs as a procedural with ordered steps.
- Provide download links for the `.env` and `docker-compose.yml` files rather than display them.
- Move info about pulling and verifying Docker images into the [Run Elasticsearch in Docker docs](https://www.elastic.co/guide/en/elasticsearch/reference/current/docker.html#docker-cli-run-dev-mode). These steps don't apply to the Docker Compose docs.

Closes https://github.com/elastic/platform-docs-team/issues/186
Depends on https://github.com/elastic/enterprise-search-pubs/pull/3788
2023-09-11 11:05:08 -04:00
Abdon Pijpelink 91759ce592
[DOCS] Some minor ES|QL docs fixes (#99423) 2023-09-11 16:20:10 +02:00
William Brafford e386a3ffa4
Pass compatibility versions in ClusterCoordinationPlugin (#99396)
In https://github.com/elastic/elasticsearch/pull/99321, we passed
CompatibilityVersions into GatewayMetaState so that a node's initial
cluster state will hold correct initial values for CompatibilityVersions
for a single node, and not have to wait until a node join event to
populate that part of cluster state.

However, a ClusterCoordinationPlugin may provide a different
`PersistedClusterStateService`, which might also need its own access to
CompatibilityVersions. Here, we add a `CompatibilityVersions` argument
to `PersistedClusterStateServiceFactory#newPersistedClusterStateService`
so that an implementation of `PersistedClusterStateServiceFactory` can
pass CompatibilityVersions into whatever mechanism it uses for creating
the initial cluster state.
2023-09-11 10:17:19 -04:00
Artem Prigoda 79aba8c67f
Bump transport version for stateless plugin (#98031) 2023-09-11 15:45:58 +02:00
Benjamin Trent 83b70e37ef
Revert "Auto-normalize dot_product vectors at index & query (#98944)" (#99421)
This reverts commit 7b9c367aeb.
2023-09-11 09:33:17 -04:00
Joe Gallo 79bae69123
Document the origin of the DateProcessor format literals (#99388) 2023-09-11 09:21:16 -04:00
David Turner b53756a34d
Remove `transport_versions` from cluster state API (#99223)
Since #99114 the cluster state API exposes per-node versions such as
their transport versions under `node_versions` so there's no need to
duplicate this information under `transport_versions`. This commit
removes the unnecessary information.
2023-09-11 09:13:51 -04:00
Przemyslaw Gomulka 19378f656e
Mark 'index.hidden' setting as public (#99338)
the setting is used by both external and internal users at the moment.
this commit is marking `index.hidden` setting as public for serverless
2023-09-11 13:05:40 +02:00
Johannes Mahne 4aa0c698f2
Update ilm-shrink.asciidoc (#99366)
Formatting
2023-09-11 12:56:34 +02:00
Krishna Chaitanya Reddy Burri 957989474e
Add "manage" permission for fleet managed threat intel indices (#99231)
* Add "manage" permission needed for creating and maintaining aliases of threat intel transforms
2023-09-11 16:05:20 +05:30
Craig Taverner 3ca48a4752
NamedWritable keys must be strings, not dynamic (#99408)
When developing the GenericNamedWriteable instances we were dynamically generating the key names from the class names, but it is better to use strings so that class name refactoring does not change the keys causing compatibility issues in inter-node communication in mixed clusters.
2023-09-11 11:50:23 +02:00
David Kyle af4b2d98be
Mute MoreExpressionIT::testStringSpecialValueVariable (#99411)
For #99156
2023-09-11 10:47:20 +01:00
David Kyle 871ca07e99
Mute SearchWithRandomIOExceptionsIT::testRandomDirectoryIOExceptions (#99410)
For #99174
2023-09-11 10:39:55 +01:00
Luigi Dell'Aquila c0a2a3442d
Make ESQL RepetitiveEval test deterministic (#99405) 2023-09-11 11:09:11 +02:00
David Turner 356a5401a4
Deeper `ChunkedBroadcastResponse` chunking (#99364)
Today the chunked responses to broadcast APIs (indices stats, indices
segments and field usage stats) emit one chunk per index, but each
index-level chunk could itself scale with the number of shard copies,
segments, or fields in the index. This commit introduces finer-grained
chunking, such that each chunk now has O(1) size.
2023-09-11 04:54:04 -04:00
David Kyle 7e33919105
Mute continuous transform test cases (#99407)
Mutes `HistogramGroupByIT` and `TermsGroupByIT`

For https://github.com/elastic/elasticsearch/issues/97263
2023-09-11 04:47:27 -04:00
Abdon Pijpelink 2ef7da53e3
[DOCS] Set up redirect for old anchor (#99401) 2023-09-11 10:13:24 +02:00
David Turner 9f83169676 AwaitsFix for #99403 2023-09-11 08:58:57 +01:00
David Turner 2dffc5e57f
Replace executor(SAME) with DIRECT (#99399)
Cleans up the places where we are unnecessarily looking up the SAME
executor on a threadpool.
2023-09-11 03:31:05 -04:00