Remove help_url,rename summary->symptom,user_actions->diagnosis
Separate the diagnosis `message` field in `cause` and `action`
Co-authored-by: Mary Gouseti <mgouseti@gmail.com>
* Convert disk watermarks to RelativeByteSizeValues
Similar to the existing watermark setting for the frozen tier.
Pre-requisite for PR 88639 that plans to introduce max headroom
settings for the disk watermarks, similar to the frozen tier max
headroom setting.
* Add changelog
* Revert 20gb to 20GB
* Make formatNoTrailingZerosPercent non static
* ByteSizeValue.MINUS_ONE
* Remove getMinimumTotalSizeForBelowWatermark
* Remove comment
* Fix minor stuff
* Make parsing of RelativeByteSizeValue faster
Mimicks older definitelyNotPercentage function
* Remove Locale from Strings.format
* More MINUS_ONE
This PR adds a new `knn` option to the `_search` API to support ANN search.
It's powered by the same Lucene ANN capabilities as the old `_knn_search`
endpoint. The `knn` option can be combined with other search features like
queries and aggregations.
Addresses #87625
This adds support for the `cardinality` aggregation within a random_sampler.
This usecase is helpful in determining the ratio of unique values compared to the count of total documents within the sampled set.
* [DOCS] Add minimal security steps back to docs
* Update instructions to use reset password tool
* Update setting built-in user passwords with the es reset passwords tool
* Revert "Update setting built-in user passwords with the es reset passwords tool"
This reverts commit 51b72fdfdf.
* Address review feedback and make clearer distinctions between security configurations
With: https://github.com/elastic/ml-cpp/pull/2305 we now support caching pytorch inference responses per node per model.
By default, the cache will be the same size has the model on disk size. This is because our current best estimate for memory used (for deploying) is 2*model_size + constant_overhead.
This is due to the model having to be loaded in memory twice when serializing to the native process.
But, once the model is in memory and accepting requests, its actual memory usage is reduced vs. what we have "reserved" for it within the node.
Consequently, having a cache layer that takes advantage of that unused (but reserved) memory is effectively free. When used in production, especially in search scenarios, caching inference results is critical for decreasing latency.
Currently we have two parameters that control how the source of a document
is stored, `enabled` and `synthetic`, both booleans. However, there are only
three possible combinations of these, with `enabled:false` and `synthetic:true`
being disallowed. To make this easier to reason about, this commit replaces
the `enabled` parameter with a new `mode` parameter, which can take the values
`stored`, `synthetic` and `disabled`. The `mode` parameter cannot be set
in combination with `enabled`, and we will subsequently move towards
deprecating `enabled` entirely.
The build_flavor was previously removed since it is no longer relevant;
only the default distribution now exists. However, the removal of build
flavor included removing it from the version information on the info
response for the root path. This API is supposed to be stable, so
removing that key was a compatibility break. This commit adds the
build_flavor back to that API, hardcoded to `default`. Additionally, a
test is added to ensure the key exists going forward, until it can be
properly deprecated.
closes#88318
This PR adds a new setting to enable tcp keepalive probes for the
connections used by the oidc back-channel communication. It defaults to
true as tcp keepalive is generally useful for ES.
Relates: #87773
* Adding discovery troubleshooting link
* Add tags to pull in discovery troubleshooting content
* Move discovery troubleshooting to separate page and add redirects
Co-authored-by: Adam Locke <adam.locke@elastic.co>
Plumbs through a new parameter for the cardinality aggregation, to allow configuring the execution mode. This can have significant impacts on speed and memory usage. This PR exposes three collection modes and two heuristics that we can tune going forward. All of these are treated as hints and can be silently ignored, e.g. if not applicable to the given field type. I've change the default behavior to optimize for time, which potentially uses more memory. Users can override this for the old behavior if needed.
* Generate release notes for v8.3.0 (#87294)
* Generate release notes for v8.3.0
* [DOCS] Add 8.3 migration file to index
* Fixed version number
* Fix formatting of deprecation in 85326
* Use asciidoc format for deprecations
* Extract static content from migration/index
* This was just an enhancement
* Nope, it was an upgrade
* Added migration/index.asciidoc generation support (#87318)
Including extracting static content from migration/index, so the template would be as light as possible.
The reason for this work is because the gradle task `generateReleaseNotes` was not correctly adding new links and imports to the migrations/index and that caused documentation to fail building for 8.3.0.
* [DOCS] Add ml-cpp PRs to release notes
* Added back incorrectly deleted changlog
* Added missing highlight
* Fixed spelling of StackOverflowError
Co-authored-by: lcawl <lcawley@elastic.co>
* Brought back missing changelog for 87235 (#87370)
* Brought back missing changelog for 87235
* Regenerate release notes
* Regenerate release notes for BC3 (#87449)
* Regenerate release notes for BC3
* Re-applied manual fixes that Lisa Cawley used
* Fixed breaking changes generation
To match the manual edits done by Lisa Cawley
* Fixed failing test
Since the manual edits by Lisa removed the `-SNAPSHOT` from the docs
we remove it from the tests too.
* Remove changelogs for intermediate lucene upgrades
* Update release notes for BC4 (#87635)
* Update release notes for BC4
* Adding missing changelog for geo_grid
* Added missing highlight notes for 84250
* Update docs/reference/release-notes/highlights.asciidoc
Co-authored-by: David Kilfoyle <41695641+kilfoyle@users.noreply.github.com>
* Update docs/reference/release-notes/highlights.asciidoc
Co-authored-by: David Kilfoyle <41695641+kilfoyle@users.noreply.github.com>
* Backported fixes to original yaml
* Control sort order of release highlights
So that small changes to an individual highlight don't completely shuffle the entire document.
We also added links to the PRs from the highlight titles, for convenience.
Otherwise readers need to search the release notes for the changelog entry and click there,
which is a lot more work.
* Fixed failing test after new ordered highlights
Also made test verify re-ordering of highlights
Co-authored-by: David Kilfoyle <41695641+kilfoyle@users.noreply.github.com>
* Update release notes for BC5 (#87808)
* Update release notes for BC6 (#87912)
* Update release notes for BC9 (#88011)
* Regenerate release notes after version bump 8.4.0
And after forward porting 8.3.0 release notes changes.
* Regenerate after some changelog entries removed
* Re-prune the changelogs
* Removed three changelog entries
Co-authored-by: lcawl <lcawley@elastic.co>
Co-authored-by: David Kilfoyle <41695641+kilfoyle@users.noreply.github.com>
The ingest attachment processor is currently available as a plugin. This
commit moves the processor to the default distribution so it is always
available.
When using LVM or software raid on Linux the kernel, or specific
distribution rules, may use higher ergonomic defaults for the
readahead of resulting block device(s). This can adversely affect
search performance due to high page cache thrashing, in search
heavy scenarios when mmap is involved.
Add a clarification section in the docs raising awareness about this
value and preferring the lower default.
This PR adds listings of all the current details that can be returned from the implemented
Health Indicator Services. Response details are unique to each indicator and describe the
state of the system that the indicator is basing its health decisions on.
Co-authored-by: Andrei Dan <andrei.dan@elastic.co>
Added a requirement that index.look_ahead_time index setting
can't be lower than time_series.poll_interval setting.
Additional changes:
* Fixed a mistake in the docs that referenced indices.lifecycle.poll_interval
instead of time_series.poll_interval.
* Moved index.look_ahead_time setting to data stream module.
In some environment, the back-channel connection can be dropped
without sending a TCP RST to ES. When that happens, reusing the same
connection results into timeout error.
This PR adds a new http.connection_pool_ttl setting to control how long
a connection in the OIDC back-channel pool can be idle before it is
closed. This allows ES to more actively close idle connections to avoid
the timeout issue.
The new setting has a 3min default which means idle connections are
closed every 3 min if server response does not specify a shorter keep-alive.
Resolves: #75515
As the number of cores in CPUs is typically a power of 2,
this commit adds a validation that trained model deployments
start with `threads_per_allocation` set to be a power of 2.
When we look for how we distribute the allocations across the
cluster, this prevents situations where we have a lot of wasted
CPU cores.
In addition, we add a max value limit of `32`.
* Revert "Revert "[DOCS] Add TSDS docs (#86905)" (#87702)"
This reverts commit 0c86d7b9b2.
* First fix to tests
* Add data_stream object to index template
* small rewording
* Add enable data stream object in gradle example setup
* Add bullet about data stream must be enabled in template
* [DOCS] Add TSDB docs
* Update docs/build.gradle
Co-authored-by: Adam Locke <adam.locke@elastic.co>
* Address Nik's comments, part 1
* Address Nik's comments, part deux
* Reword write index
* Add feature flags
* Wrap one more section in feature flag
* Small fixes
* set index.routing_path to optional
* Update storage reduction value
* Update create index template code example
Co-authored-by: James Rodewig <40268737+jrodewig@users.noreply.github.com>
Co-authored-by: Adam Locke <adam.locke@elastic.co>
Changes the type of the version parameter in `IngestDocument` from
`Long` to `long` and moves it to the third argument, so all required
values occur before nullable arguments.
The `IngestService` expects a non-null version for a document and will
throw an `NullPointerException` if one is not provided.
Related: #87309
Adds measures of the total size of all mappings and the total number of
fields in the cluster (both before and after deduplication).
Relates #86639
Relates #77466
The current docs mention that Elasticsearch indexes prefixes between 2 and 5 characters in a separate field. 2 and 5 are default values, and the size of the prefixes indexed depend on the configuration settings.
* Move fix common cluster issues to troubleshooting
* Include fix common cluster issues in the troubleshooting doc
* [DOCS] Remove extra include from How-To
Co-authored-by: Deb Adair <debadair@elastic.co>
This change tries to make it easier to find kNN with filtering in the docs:
* Mention filtering support in the kNN API description
* In kNN tutorial, link to the kNN search API page more prominently
This guidance does not apply any longer.
The overhead per shard has been significantly reduced in recent versions
and removed rule of thumb will be too pessimistic in many if not
most cases and might be too optimistic in other specific ones.
=> Replace guidance with rule of thumb per field count on data nodes and
rule of thumb by index count (which is far more relevant nowadays than
shards) for master nodes.
relates #77466
Co-authored-by: David Turner <david.turner@elastic.co>
Co-authored-by: Henning Andersen <33268011+henningandersen@users.noreply.github.com>
This adds troubleshooting documentation for the case when the ShardsAvailabilityHealthIndicatorService
reports that there are not enough nodes in the data tier (user action "increase_node_capacity_for_allocations" or
"increase_tier_capacity_for_allocations_". This covers both the cloud and self-managed environments. For
cloud we first recommend increasing the number of availability zones (because you cannot directly add nodes), and
decreasing index.number_of_replicas if that is not possible. For self-managed, we first recommend adding nodes,
and decreasing index.number_of_replicas if that is not possible.
This removes "data streams" from the docs for the `index`, `delete`,
and `update` actions because data streams only support the `update`
action.
Closes#87231
Add cluster mapping stats for indexed dense_vectors
Currently _cluster/stats mapping section displays all mapping types
along with their count. In 8.0 we introduced indexed dense_vector
types, and we would like to collect more enhanced stats on them:
- number of indexed dense_vector fields
- sum of dims across all indexed dense_vector fields
This allows to differentiate how indexed dense_vector types are
used as opposed to unindexed dense_vector types.
Add `ignore_missing_pipeline` option to `pipeline` processor. This
controls whether the `pipeline` processor should fail with an error if
no pipeline with a name specified in the `name` option exists.
This enhancement is useful to setup a pipeline infrastructure that
lazily adds extension points for overwrites. So that for specific
cluster setups custom pre-processing can be added at a later point in
time.
Relates to #87323
In #85074 we added docs on discovery troubleshooting that really only
talked about troubleshooting master elections. There's also the case
where the master is elected fine but some other node can't join it. This
commit adds troubleshooting docs about that too.
Co-authored-by: Adam Locke <adam.locke@elastic.co>
Including extracting static content from migration/index, so the template would be as light as possible.
The reason for this work is because the gradle task `generateReleaseNotes` was not correctly adding new links and imports to the migrations/index and that caused documentation to fail building for 8.3.0.
Back when we introduced the fields parameter to the search API, it could only fetch values from _source, hence
the corresponding sub-fetch phase fails early whenever _source is disabled. Today though runtime fields can
be retrieved from a separate value fetcher that reads from fielddata, and metadata fields can be retrieved
from stored fields. These two scenarios currently throw an unnecessary error whenever _source is disabled.
This commit removes the check for disabled _source, so that runtime fields and metadata fields can be retrieved even when _source is disabled. Fields that need to be loaded from _source are simply skipped whenever _source is disabled, similar to when a field is not found in _source.
Closes#87072
Add support for
allow_partial_search_results = true/false
command in SQL CLI.
If true, returns partial results if there are shard request timeouts or shard failures.
If false, returns an error with no partial results.
This adds an example of the precision loss for `double`, `float`, and
`half_float` numbers that we can link folks to when explaining what
happened to their numbers. You can link directly to it with something
like:
```
/guide/number.html#floating_point
```
* Adding Getting Help section
Add getting help section in the troubleshooting guide to be pointed by health API when issues are too complicated to be addressed.
This is taken from https://www.elastic.co/guide/en/cloud/current/ec-get-help.html, someone might want to elaborate it a bit more?
* Fix broken partintro, modify headings, and update wording
Co-authored-by: Adam Locke <adam.locke@elastic.co>
The get settings api has accepts the expand_wildcards option. The docs
state the default value is `all`, but it is actually now `open` (which
does not include hidden or closed indices by default). This commit
changes the docs to match the existing behavior.
This adds a troubleshooting doc for indices that mix index filtering allocation
with data tiers routing.
Co-authored-by: Lee Hinman <dakrone@users.noreply.github.com>
This replaces the implementation of the categorize_text aggregation
with the new algorithm that was added in #80867. The new algorithm
works in the same way as the ML C++ code used for categorization jobs
(and now includes the fixes of elastic/ml-cpp#2277).
The docs are updated to reflect the workings of the new implementation.
n larger clusters with complicated datafeed requirements, being able to preview only a specific window of time is important. Previously, datafeed previews would always start at 0 (or from the beginning of the data). This causes issues if the index pattern contains indices on slower hardware, but when the datafeed is actually started, the "start" time is set to more recent data (and thus on faster hardware).
Additionally, when _preview is unbounded (as before), it attempts to only preview indices that are NOT frozen or cold. This is done through a query against the _tier field. Meaning, it only effects newer indices that actually have that field set.
This commit adds support for CPU ranges in the desired nodes API.
This aligns better with environments where administrators/orchestrators
can define lower and upper bounds for the amount of CPUs that the
desired node would get once deployed.
This allows to provide information about the expected CPU and possible
allowed overcommit that the desired node will run on.
This was the previous expected body for the desired nodes API (we still support it):
```
PUT /_internal/desired_nodes/history/1
{
"nodes" : [
{
"settings" : {
"node.name" : "instance-000187",
"node.external_id": "instance-000187",
"node.roles" : ["data_hot", "master"],
"node.attr.data" : "hot",
"node.attr.logical_availability_zone" : "zone-0"
},
"processors" : 8,
"memory" : "58gb",
"storage" : "1700gb",
"node_version" : "8.3.0"
}
]
}
```
Now it's possible to define `processors` or `processors_range` as in:
```
PUT /_internal/desired_nodes/history/1
{
"nodes" : [
{
"settings" : {
"node.name" : "instance-000187",
"node.external_id": "instance-000187",
"node.roles" : ["data_hot", "master"],
"node.attr.data" : "hot",
"node.attr.logical_availability_zone" : "zone-0"
},
"processors_range" : {"min": 8.0, "max": 16.0},
"memory" : "58gb",
"storage" : "1700gb",
"node_version" : "8.3.0"
}
]
}
```
Note that `max` in `processors_range` is optional.
This commit also moves from representing CPUs as integers to
accept floating point numbers.
Note: I disabled the bwc yamlRestTests for versions < 8.3 since we introduced
a few "breaking changes" but since this is an internal API it should be fine.
* Soft-deprecation of point/geo_point formats
Since GeoJSON and WKT are now common formats for all three types:
geo_shape, geo_point and point
We decided to soft-deprecate the other point formats by ordering:
* GeoJSON (object with keys `type` and `coordinates`)
* WKT `POINT(x y)`
* Object with keys `lat` and `lon` (or `x` and `y` for point)
* Array [lon,lat]
* String `"lat,lon"` (or `"x,y"` in point)
* String with geohash (only in `geo_point`)
The geohash is last because it is only in one field type.
The string version is second last because it is the most controversial
being the only version to reverse the coordinate order from all other
formats (for geo_point only, since the coordinates are not reversed
in point).
In addition we replaced many examples in both documentation and tests
to prioritize WKT over the plain string format.
Many remaining examples of array format or object with keys still exist
and could be replaced by, for example, GeoJSON, if we feel the need.
* Incorrect quote position
This PR adds support for a new mapping parameter to the configuration of the object mapper (root as well as individual fields), that makes it possible to store metrics data where it's common to have fields with dots in their names in the following format:
```
{
"metrics.time" : 10,
"metrics.time.min" : 1,
"metrics.time.max" : 500
}
```
Instead of expanding dotted paths the their corresponding object structure, objects can be configured to preserve dots in field names, in which case they can only hold leaf sub-fields and no further objects.
The mapping parameter is called subobjects and controls whether an object can hold other objects (defaults to true) or not. The following example shows how it can be configured in the mappings:
```
{
"mappings" : {
"properties" : {
"metrics" : {
"type" : "object",
"subobjects" : false
}
}
}
}
```
Closes#63530
Support label position in REST vector tiles
There is a need to provide sensibly calculated label positions for polygons and lines in Kibana maps. A very convenient way to satisfy this need is through a runtime field that the rest API can make use of when labels are requested. This has the advantage of providing painless access to the label position as well.
This work adds support for the REST API to provide label positions to MVT queries, both for the HITS layer and the AGGS layer. To enable this feature, set with_labels to true as a query parameter to the vector tile search query.
* [DOCS] Document when security auto configuration is skipped
* Add xpack autoconfiguration parameter and links
* Fix typo and add related link 🔗
* Incorporate review feedback
* Remove keystore section and instead integrate into previous section
When creating and updating transforms, it is possible for clients to provide secondary headers.
When PUT, _preview, _update is called with secondary authorization headers, those are then used or stored with the transform.
closes: https://github.com/elastic/elasticsearch/issues/86731
It was previously required that the _start API caller required the same roles as the create API caller.
This does not make sense as when the transform is actually running (after _start) we rely solely on the roles of the caller who created the transform.
Consequently, this commit does the permission validations and various checks with the roles of user who created the transform, not the one calling _start
"Add" was out of the hyperlink context which I have fixed it.
Earlier line 71 was like : * *Add* <<set-up-lifecycle-policy,*lifecycle policy*>>
After rectifying line 71 is like : * <<set-up-lifecycle-policy,*Add lifecycle policy*>>
(cherry picked from commit 3b8d51c696)
Co-authored-by: Tapomoy Bhowmik <99604828+TapomoyBhowmik@users.noreply.github.com>
* correct way of getting node heap size
in [[shard-count-recommendation]], we explain that the number of shards should be at most 20 shards per GB of heap.
but the command to get relevant heap size should be _cat/nodes?v=true&h=heap.max and not _cat/nodes?v=true&h=heap.current . The latter gives the current memory consumption, which is alway moving. Here we need to consider the max allocated heap size (-Xmx)
* Adds heap.max to valid columns
Co-authored-by: Adam Locke <adam.locke@elastic.co>
Today the add/clear voting config exclusions APIs route a request to the
master node but do not expose the usual `?master_timeout` parameter
allowing to change the timeout for this phase of execution. This commit
adds the missing parameter.
Remove usage of deprecated elasticsearch.rest-test in DocsTestPlugin
we keep some files in src/test in docs projects as moving them would require more changes
in build-docs project outside this repository
* document cloud_id usage
* actually no cloud id used
* [source,console]
* suggested change
* Mark example as NOTCONSOLE
* Add tests
* Add comma
* Fix comma (for real this time)
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
Co-authored-by: Adam Locke <adam.locke@elastic.co>
When starting a trained model deployment the user can tweak performance
by setting the `model_threads` and `inference_threads` parameters.
These parameters are hard to understand and cause confusion.
This commit renames these as well as the fields where their values are
reported in the stats API.
- `model_threads` => `number_of_allocations`
- `inference_threads` => `threads_per_allocation`
Now the terminology is as follows.
A model deployment starts with a requested `number_of_allocations`.
Each allocation means the model gets another thread for executing
parallel inference requests. Thus, more allocations should increase
throughput. In its turn, each allocation is may be using a number
of threads to parallelize each individual inference request.
This is the `threads_per_allocation` setting and increases inference
speed (which might also result in improved throughput).
This attempts to shrink the index by implementing a "synthetic _source" field.
You configure it by in the mapping:
```
{
"mappings": {
"_source": {
"synthetic": true
}
}
}
```
And we just stop storing the `_source` field - kind of. When you go to access
the `_source` we regenerate it on the fly by loading doc values. Doc values
don't preserve the original structure of the source you sent so we have to
make some educated guesses. And we have a rule: the source we generate would
result in the same index if you sent it back to us. That way you can use it
for things like `_reindex`.
Fetching the `_source` from doc values does slow down loading somewhat. See
numbers further down.
## Supported fields
This only works for the following fields:
* `boolean`
* `byte`
* `date`
* `double`
* `float`
* `geo_point` (with precision loss)
* `half_float`
* `integer`
* `ip`
* `keyword`
* `long`
* `scaled_float`
* `short`
* `text` (when there is a `keyword` sub-field that is compatible with this feature)
## Educated guesses
The synthetic source generator makes `_source` fields that are:
* sorted alphabetically
* as "objecty" as possible
* pushes all arrays to the "leaf" fields
* sorts most array values
* removes duplicate text and keyword values
These are mostly artifacts of how doc values are stored.
### sorted alphabetically
```
{
"b": 1,
"c": 2,
"a": 3
}
```
becomes
```
{
"a": 3,
"b": 1,
"c": 2
}
```
### as "objecty" as possible
```
{
"a.b": "foo"
}
```
becomes
```
{
"a": {
"b": "foo"
}
}
```
### pushes all arrays to the "leaf" fields
```
{
"a": [
{
"b": "foo",
"c": "bar"
},
{
"c": "bort"
},
{
"b": "snort"
}
}
```
becomes
```
{
"a" {
"b": ["foo", "snort"],
"c": ["bar", "bort"]
}
}
```
### sorts most array values
```
{
"a": [2, 3, 1]
}
```
becomes
```
{
"a": [1, 2, 3]
}
```
### removes duplicate text and keyword values
```
{
"a": ["bar", "baz", "baz", "baz", "foo", "foo"]
}
```
becomes
```
{
"a": ["bar", "baz", "foo"]
}
```
## `_recovery_source`
Elasticsearch's shard "recovery" process needs `_source` *sometimes*. So does
cross cluster replication. If you disable source or filter it somehow we store
a `_recovery_source` field for as long as the recovery process might need it.
When everything is running smoothly that's generally a few seconds or minutes.
Then the fields is removed on merge. This synthetic source feature continues
to produce `_recovery_source` and relies on it for recovery. It's *possible*
to synthesize `_source` during recovery but we don't do it.
That means that synethic source doesn't speed up writing the index. But in the
future we might be able to turn this on to trade writing less data at index
time for slower recovery and cross cluster replication. That's an area of
future improvement.
## perf numbers
I loaded the entire tsdb data set with this change and the size:
```
standard -> synthetic
store size 31.0 GB -> 7.0 GB (77.5% reduction)
_source 24695.7 MB -> 47.6 MB (99.8% reduction - synthetic is in _recovery_source)
```
A second _forcemerge a few minutes after rally finishes should removes the
remaining 47.6MB of _recovery_source.
With this fetching source for 1,000 documents seems to take about 500ms. I
spot checked a lot of different areas and haven't seen any different hit. I
*expect* this performance impact is based on the number of doc values fields
in the index and how sparse they are.
The health API has a notion of details within each health indicator that is returned. These details can sometimes be
expensive to compute or transfer. This change allows a user to specify whether the details are generated and
returned. By default now all details are generated and returned (previously this was only the case if a component
was specified in the request). This behavior can be changed with the explain query param.
Closes#86215
1. Adds a note that you can restore older snapshots (to recover from a
failed upgrade) even after newer snapshots were taken.
2. Copies the note about incompatible S3 repo implementations to the top
level to avoid misunderstandings.
This commit adds a new `_ml/trained_models/{model_id}/_infer` API. This api works for both native NLP models and supervised models trained via Data Frame analytics.
The format of the API is the same as the old `_ml/trained_models/{model_id}/deployment/_infer`. Taking a `docs` and an `inference_config` parameter.
This PR also deprecates the old experimental `_ml/trained_models/{model_id}/deployment/_infer` API.
The biggest difference is that the response now nests all results under an "inference_results" object.
closes: https://github.com/elastic/elasticsearch/issues/86032
* [doc] Add information for how to find if compressed ordinary object pointers is in use using the REST APIs.
* Update docs/reference/setup/advanced-configuration.asciidoc
Co-authored-by: Nikola Grcevski <6207777+grcevski@users.noreply.github.com>
Co-authored-by: Nikola Grcevski <6207777+grcevski@users.noreply.github.com>
This commit adds tracking for desired nodes cluster membership.
When desired nodes are updated they are matched against the current
cluster members. Additionally when a node joins the cluster the
desired nodes cluster membership is updated.
In #50535 (ES v7.6) the default values for the
`DocumentSubsetBitsetCache` settings were changed. However, the docs
were not updated at that time, and still reflect the old values for
these settings
Adds a parameter `index_names` to the get snapshots API so that users may exclude the potentially very long index name lists when listing out snapshots.
closes#82937
Users should be able to specify specific metrics/keys within a specific bucket key.
An example is `agg["bucket_foo"]._count`.
This change now allows that.
closes: https://github.com/elastic/elasticsearch/issues/76320
Fixes a few scalability issues around join validation:
- compresses the cluster state sent over the wire
- shares the serialized cluster state across multiple nodes
- forks the decompression/deserialization work off the transport thread
Relates #77466Closes#83204
Today you cannot remove archived index settings by applying a setting
update `{"archived.*":null}` because `IndexSettings#same` incorrectly
treats such an update as a no-op. This commit fixes that.
This adds support for partial results to SQL.
The lenient mode is controlled by a new query paramter,
`allow_partial_search_results`, false by default. On shard failures, the
errors are added as Warning headers to the response. Only a first set of
failures are sent to the client, the last header briefs on the number of
remaining suppressed ones.
Ensures that on every page of the docs that mentions
`cluster.initial_master_nodes` also mentions that this setting must be
removed after bootstrapping completes.
This renames the internal concept of a trained model allocation into an assignment.
Now models are assigned to a node and routes created for inference. Not "allocated".
This is an internal rename only. The user facing concepts of trained models and deployments are untouched.
Add initial feature overview documentation pages for User Profile and
Security Domain.
Co-authored-by: Adam Locke <adam.locke@elastic.co>
Co-authored-by: Tim Vernum <tim@adjective.org>
Co-authored-by: Albert Zaharovits <albert.zaharovits@gmail.com>
The default distribution is the only remaining build flavor, and has been for
quite a while now. This commit removes flavor from the internal Build
class. It keeps rest api compat for nodes info for now by hardcoding
`default`.
The no-jdk distributions exist in 7.x and before. They were removed with
8.0. This commit removes the remaining deprecation messages for using
the no-jdk distribution. Note that when talking with an older node, we
drop the bundledJdk attribute. This is ok because it is only possible
for this to not be true when talking with a 7.17 node, during an upgrade,
and the usingBundledJdk is retained, which is the important thing if
debugging a problem.
relates #76896
relates #85758
Today the docs say that new major versions need new snapshot
repositories or else corruption may occur. This isn't true, we support
using the same repository across upgrades.
Since 8.0.0, we only support Java 17 and up. This change cleans up unsupported Java versions from jvm.options and updates documentation to reference supported Java versions, in case it is confusing to users.
When a transform has a `retention_policy` it needs to be able to delete documents in the destination index.
`create_index` does not necessitate that we can delete documents from it. So, even if we create the index, we need to verify that we can delete documents given the `retention_policy` definition.
This is not a crucial bug as the transform will simply fail later. Its nicer to fail sooner.
closes https://github.com/elastic/elasticsearch/issues/85409
* adds a test for mixed cluster requests
* fixes a bad stream version check (above test will fail if this isn't included)
* replaces private FieldCapsFilter interface with Predicate
* renames 'allowedTypes' to 'types' to maintain consistency with external API
* adds javadoc to ResponseRewriter
* removes isRuntimeField from FieldTypeLookup
Relates to #83636
This commit clarifies the force merge behavior for the `forcemerge` action either as a standalone
ILM action, or as part of the searchable snapshot action. When using a searchable snapshot it can be
confusing especially because the force merge can occur in a phase prior to the phase configured in
the ILM policy (since ILM doesn't migrate the index prior to the searchable snapshot mounting).
This commit adds back some notes which were lost when we consolidated
the snapshot/restore documentation into a single location.
The notes in question are that:
1) If a Snapshot repository contains Security's feature state, then
that repository contains security-sensitive information. This may
be obvious to some, but is good to state explicitly.
2) Some files, such as the keystore and TLS keys, are not included in
snapshots and are important to back up via other means.
Today it's no longer true that by default nodes will auto-discover other
nodes on the same host and bootstrap them all into a cluster. This
commit fixes the docs on auto-bootstrapping to recognise this.
* Resolve indices api: add 'system' attribute
* Update docs/changelog/85042.yaml
* Remove magic strings for attribute values
* Update API docs to provide possible resolved index attributes
Today we don't really say anything about the requirements for the data
path in terms of correctness, and we specifically say to avoid NFS for
performance reasons. This isn't wholly accurate: some NFS
implementations work just fine. This commit documents a more balanced
position on local vs remote storage.
This reverts commit 4eaedb265d.
On further investigation of how to improve allocation of trained models,
we concluded that being able to set `inference_threads` in combination with
`model_threads` is fundamental for scalability.
This adds the `xpack.notification.email.account.domain_allowlist` dynamic cluster setting that
allows an administrator to specify a list of domains to which emails are allowed to be sent. The
default value for this setting is `["*"]` which means all domains are allowed. It supports
rudimentary globbing (`*`) in the domain name, so `*.company.com` will work as a valid option.
Resolves https://github.com/elastic/elasticsearch/issues/84739
This commit adds a new optional parameter on the vector tiles API called `grid_agg` with two
possible values, geotile (default) and geohex. This will allow to build the aggs layer using different
grid aggregations, for example we can have a grid aggregation that is built using hexagons.
In #84340 we adjusted the docs to show instructions on how to
retrieve TLS keystores passwords from the elasticsearch keystore
but the security-files-reference.asciidoc was not referenced in
all places where the instructions should be used. This commit
changes that and also removes references for beta versions.
When using a multi-field we need to extract data from the document
using the correct field name. That is the name of the top field.
Here we delegate extraction of the correct name to a method in the
SearchContext that is wrapped by the AggregationContext.
Issue: #82918
This PR integrates support for ANN with filtering added in Lucene 9.1. It adds
a new `filter` section to the `_knn_search` endpoint, which accepts a query (in
the Elasticsearch query DSL). The value can either be a single query or a list
of queries, which matches the syntax we use for defining filter clauses in a
`bool` query.
Closes#81788.
* Added buffer pixels to vector tile spec parsing
Previously this was hard-coded to 5, but now is configurable using the
format z/x/y@extent:buffer, where both extent and buffer are optional
and default to 4096 and 5 pixels respectively.
Co-authored-by: James Rodewig <james.rodewig@elastic.co>
The way this specific doc was structured meant that the path for
DEB/RPM packages were wrong (an extra `config` was added to the
dir path). This commit changes the structure to be relevant to the
ES_CONF_PATH env variable that should work uniformly for all of
our installation means.
Users can encounter 401 HTTP error with NPE as the underlying cause when
using audit logging with `emit_request_body` and requests with filtered
fields. This PR adds this issue to known issues of the 8.1 release
notes.
Relates: #84784
Co-authored-by: Tim Vernum <tim@adjective.org>
Co-authored-by: James Rodewig <james.rodewig@elastic.co>
Fixes an intro sentence for the Docker install instructions.
Co-authored-by: James Rodewig <40268737+jrodewig@users.noreply.github.com>
(cherry picked from commit 472a7d8e91)
Co-authored-by: Alexander Reelsen <alexander@reelsen.net>
Starting a trained model deployment the user may set values for `inference_threads`
of `model_threads`. The first improves latency whereas the latter improves throughput.
It is easier to reason on how a model allocation uses resources if we ensure only
one of those two may be greater than one. In addition, it allows us to distribute
the cores of the ML nodes in the cluster across the model allocations in the future.
This commit adds a validation that prevents both `inference_threads` and `model_threads`
to be greater than one.
Today the note in the docs about S3-compatible repositories notes that
the repo must behave correctly, but it's also important that it has the
same performance profile. This commit extends the docs to include this
info.
Throughput is measured as the number of inference requests
processed per minute. The node level stats peak_throughput_per_minute,
throughput_last_minute and average_inference_time_ms_last_minute are
added with a deployment level stat peak_throughput_per_minute which
is the summed throughput of all nodes.
* add data tier configuration information for Cloud
* Move configuration docs and add headings
* update ILM tutorial image
* add save changes step
Co-authored-by: James Rodewig <40268737+jrodewig@users.noreply.github.com>
The use of `apt-key` is deprecated and will no longer be available after
Debian 11 and Ubuntu 22.04. This updates the installation instructions
for Debian-based distributions.
Closes#84644
Andrei Dan
Iraklis Psaroudakis
Julie Tibshirani
Elasticsearch addict
eltomello
Benjamin Trent
Abdon Pijpelink
Seth Michael Larson
Adam Locke
Alan Woodward
Ignacio Vera
Andrey
Nhat Nguyen
István Zoltán Szabó
Sean Letendre
Yannick Welsch
Ryan Ernst
debadair
Yang Wang
Leaf-Lin
David Kilfoyle
Mark Tozzi
Henning Andersen
David Turner
Jingguo Yao
Luca Cavanna
apeltop
Lisa Cawley
Rory Hunter
Craig Taverner
Keith Massey
Alexander Zaitsev
Dimitrios Liappis
Dimitris Athanasiou
Robson Sutton
Luigi Dell'Aquila
James Baiera
Martijn van Groningen
Yang Wang
Mayya Sharipova
Stuart Tettemer
Jake Landis
Nik Everett
Armin Braun
Przemysław Witek
Leaf-Lin
Justin Cranford
Jean-Fabrice Bobo
Pooya Salehi
David Roberts
Joe Gallo
wallrik
Francisco Fernández Castaño
Umut Uz
Przemyslaw Gomulka
Mark Tozzi
Sohail Mirza
Tim Vernum
vincetrumental
Rene Groeschke
Luca Belluccini
Antonio Bonuccelli
Stef Nestor
Tanguy Leroux
Nicole Albee
Salvatore Campagna
Rune Antonsen
James Garside
Gabi Davar
Bogdan Pintea
Artem Prigoda
Nikola Grcevski
Rick Boyd
Lee Hinman
James Rodewig
Gordon Brown
William Brafford
Karen Metts
Ignacio Vera
Ioannis Kakavas
Alex Chan
Ievgen Degtiarenko
David Kyle
Dan Roscigno
Abele Mălan
er0k