Refactors calculation of node memory load so that there is a
framework for supporting various different ML tasks. This results
in simpler code and it is a step towards making it easier to
add memory tracking for future tasks.
previously removed in #38514, deprecated in 7.x and defaulted to true.
With rest api compatibility and when value is true it is ignored and warning is emitted
when value is false, an exception is thrown
relates #51816
the adjust_pure_negative value used to be always present in to_xcontent
response, but it was changed in #49543 to only serialise this field when
the value is false
relates #51816
Instead relying on the countDownLatch, the entry assertion is now
perfomred in an assertBusy block so that non-atomic file update will not
cause failures.
SimpleFS is deprecated and will be removed in Lucene 9. This commit
deprecates SimpleFS in 7.x and uses NIOFS for SimpleFS in Elasticsearch
7.15 or later as it offers superior or equivalent performance to
SimpleFS.
Closes#74036. Since some orchestration platforms forbid periods in
environment variable names, allow Docker users to pass settings to ES
using an alternative name scheme. For example:
bootstrap.memory_lock
...becomes:
ES_BOOTSTRAP_MEMORY__LOCK
The setting name is uppercased, prefixed, all underscores are converted
to double underscores, and all periods are converted to underscores.
This PR adds the following constraints to dimension fields:
It must be an indexed field and must has doc values
It cannot be multi-valued
The number of dimension fields in the index mapping must not be more than 16. This should be configurable through an index property (index.mapping.dimension_fields.limit)
keyword fields cannot be more than 1024 bytes long
keyword fields must not use a normalizer
Based on the code added in PR #74450
Relates to #74660
This commit changes the default indices options of the new terms enum API
to be consistent with _search.
This change also fixes the shards statistics in the response when ccs is involved.
Closes#75155
Adjust GeoIpDownloaderIT test suit to wait for managed databases files
to be removed after each test.
After each test geoip downloader is disabled, which should eventually
remove the managed geoip database files. This happens in the background.
However a new test starts that assumes that the builtin databases are used
then that test can fail, because expected assertions will fail. The changes
in this commit should address this.
Closes#74358
indices upgrade api (/_upgrade or /{index}/_upgrade) was removed and _reindex is suggested to be used instead.
There is no easy way to translate _upgrade request to _reindex requests. The dummy Upgrade action will return an exception to a user with a message indicating that _reindex should be used.
upgrade api removal #64732
relates #51816
`field_masking_span` is the only span query that does not begin with
`span_`. This commit deprecates the existing name and adds a new
name `span_field_masking` to better fit with the other queries.
In 8.0, with security on by default, we store the HTTP
layer CA PrivateKeyEntry in the http.ssl keystore (along
with the node certificate) so that it is available in our
Enrollment API transport actions.
When loading a keystore, the current behavior is that the
X509ExtendedKeyManager will iterate through the PrivateKeyEntry
objects and will return the first key/certificate that satisfies
the requirements of the client and the server configuration,
and lacks any additional logic/filters.
We need the KeyManager to deterministically pick the node
certificate/key in all cases as this is the intended entry to be
used for TLS on the HTTP layer.
This change introduces filtering when creating the in-memory
keystore the KeyManager is loaded with, so that it will not
include PrivateKeyEntry objects when:
- there are more than 1 PrivateKeyEntry objects in the keystore
- The leaf certificate associated with the PrivateKeyEntry is a
CA certificate
Related: #75097
Co-authored-by: Ioannis Kakavas <ioannis@elastic.co>
This PR adds support for using the `slice` option in point-in-time searches. By
default, the slice query splits documents based on their Lucene ID. This
strategy is more efficient than the one used for scrolls, which is based on the
`_id` field and must iterate through the whole terms dictionary. When slicing a
search, the same point-in-time ID must be used across slices to guarantee the
partitions don't overlap or miss documents.
Closes#65740.
Using getStringRep here rather than (implicitly) toString so that the
interval is serialized exactly as it was provided, rather than
converting to some other human readable form.
* Migrate to data tiers fix for phase with deactivated migrate action
This fixes the migration to also take into account explicitly deactivated
migrate actions. A phase with both an allocate action (which we update
to not contain routing rules anymore) and a deactivated migrate action
will not configure any allocation routing rules.
This changes the migrate service to also enable the migrate action.
* Remove the deactivated migrate action
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
SearchModuleTests checks through a list of queries that it expects to be
registered, and separately checks a through a list of queries that have
not been deprecated. This second check filters out non-deprecated
queries by seeing if they have deprecated names; but this will miss queries
that have been renamed (for example, we plan to rename field_masking_span
to span_field_masking, and this query will be perfectly valid but have a
deprecated name). This commit changes the filter to check that there are
any undeprecated names instead.
Avoid ConcurrentModificationException by using a thread safe collection
in the test, as we write to the ILM history store in an async way.
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
From version 7.15 the ML native processes will have
a command line option to confirm that a valid Elastic
license key is installed when they are started.
This change adds the necessary argument for the autodetect,
normalize and data_frame_analyzer processes.
BERT models have a limit on the length of the input sequence (the number of tokens
in the tokenized string rather than the input string) and throw an error if that length
is exceeded. This PR adds a check on the tokenized string and throws a bad request
exception with a clear error message if the input is too large.
The max length defaults to 512 but is configurable via the max_sequence_length field
in the task config.
This change adjusts the supported version added in #74115
following the backport to 7.15.0 that was done in #75002.
It is possible to remove far more code from master following
the backport, however, this would make the ML change to use
the functionality harder as then that would need substantial
changes during its backport. It will be easier to make the
ML change that uses the functionality while master still has
the concept of supported/not supported for this functionality,
then do a final followup PR that removes the unnecessary code
from master.
Follow up to #75022 removing now outdated BwC code from `8.x` + some additional related dead BwC removal.
In `7.x` we can't drop the index-level transport action I think because of technical
BwC with the transport client so this is just an 8.x PR.
Temporarily muting BWC tests to allow #75002 to be merged.
Once this is merged to 7.x the BWC tests on master will be
reenabled in #75067.
Relates #74115
This is a followup to #74976.
The changes of #74976 reverted many of the changes of #71656
because #74415 made them redundant. #74415 did this by making
killed jobs as closing so that the standard "job closed immediately
after open" functionality was used instead of reissuing the kill
immediately after opening. However, it turns out that this
"job closed immediately after open" functionality is not
perfect for the case of a job that is killed while it is opening.
It causes AutodetectCommunicator.close() to be called instead
of AutodetectCommunicator.killProcess(). Both do a lot of the
same things, but AutodetectCommunicator.close() finalizes
the job, and this can cause problems if the job is being killed
as part of a feature reset.
This change reinstates some of the functionality of #71656
but in a different place that hopefully won't reintroduce the
problems that led to #74415.
We can detect that a kill has happened early on during an
open or close operation by checking if the task's allocation
ID has been removed from the map after ProcessContext.setDying()
returns true. If ProcessContext.setDying() returns true this
means the job has not been previously closed, so it must have
been killed. Then we can call AutodetectCommunicator.killProcess()
instead of AutodetectCommunicator.close() during the cleanup
that happens when we detect that a recently started process is
no longer wanted.
Relates #75069
StoreKeyConfig#x509Certificates was introduced in #72207 in order
to retrieve all certificates from the keystore used for HTTP TLS
and then filter out to get the CA certificate. Since we explicitly
add the CA key/cert as a PrivateKeyEntry on startup, we can
iterate on the PrivateKeyEntry objects instead.
We disable all these tests in FIPS mode in this QA project so
Testing conventions task fails with
Testing conventions [Test classes are not included in any enabled task ():
org.elasticsearch.xpack.ilm.CCRIndexLifecycleIT] are not honored
This change disables Testing Conventions check in FIPS mode for
the specific QA project
this PR removes tests which are not meant to be fixed (ml/, vectors/) to a separate "not to be fixed list" so that we can see which compatible changes are meant to be implemented.
relates #51816
James Rodewig
Dimitris Athanasiou
Ignacio Vera
Przemyslaw Gomulka
Przemysław Witek
Tim Vernum
Yang Wang
Nhat Nguyen
Lisa Cawley
Jim Ferenczi
Julie Tibshirani
Dan Hermann
Rory Hunter
Stuart Tettemer
Francisco Fernández Castaño
Christos Soulios
Martijn van Groningen
David Turner
André Pessanha
Lyudmila Fokina
Julie Tibshirani
Joe Gallo
Andrei Dan
Alan Woodward
István Zoltán Szabó
David Roberts
David Kyle
Felix Barnsteiner
Armin Braun
Ioannis Kakavas