This change makes the GeoIp persistent task executor/downloader multi-project aware.
- the database downloader persistent task will be at the project level, meaning there will be a downloader instance per project
- persistent task id is prefixed with project id, namely `<project-id>/geoip-downloader` for cluster in MP mode
Due to the way how stored fields get flushed when index sorting is active, it is possible that we encounter significant page cache faults when memory is scarce. In order to mitigate some of the slowness around this, we're planning to no longer mmap the fdt temp file. Initially behind a feature flag, to check for unforeseen side effects.
Typically using always mmap directory is better compared to noifs directory given there is a sufficient memory available to the OS for filesystem caching. However when that isn't the case, then indexing performance can vary a lot (often very slow). This is more true for files tmp files that stored fields create during flushing. These files exist for only a brief moment to sort stored fields in the order of the configured index sorting and are then removed. If these tmp files are mmapped there is risk to trash file system cache.
This change only avoids using mmap for the fdt tmp file. This the file that actually contains the data and can large compared to other files that get flushed. The fdm (metadata) and fdi (stored field index) remain being mmapped.
With this change we will create first the tmp file and the posting list and once the file is deleted we will
merge the vectors on the vec file. Therefore we only have two copies of the vector at the same time.
TransportStartDatafeedAction previously tried to validate remote index cluster
names in datafeed jobs, before checking if the local cluster had
remote_cluster_client role. Because this role enables retrieval of the remote
cluster names, the validation step would always fail with a no-such-cluster
exception. This was confusing. This change moves the remote_cluster_client check
ahead of cluster name validation, and adds a test.
Closes ES-11841
Closes#121149
There is an issue where for Flattened fields with synthetic source, if there is a key with a scalar value, and a duplicate key with an object value, one of the values will be left out of the produced synthetic source. This fixes the issue by replacing the object with paths to each of its keys. These paths consist of the concatenation of all keys going down to a given scalar, joined by a period. For example, they are of the form foo.bar.baz. This applies recursively, so that every value within the object, no matter how nested, will be accessible through a full specified path.
* Making progress on different request parameters
* Working tests
* Adding custom service validator for rerank
* Fixing embedding bug
* Adding transport version check
* Fixing tests
* Fixing license header
* Fixing writeTo
* Moving file and removing commented code
* Fixing test
* Fixing tests
* Refactoring and tests
* Fixing test
This test has been muted for a long time and only seemed to fail on v7
compatability tests. We're unmuting this to see if it's still relevant.
Closes#112189
* Field infos calculation method inside Engine
* buildSeqNoStats as static public method
So it can be overriden in stateless if/as needed.
Relates ES-11457
* Update kibana system user permissions for cases analytics index.
* [CI] Auto commit changes from spotless
* Add permissions for aliases.
Create constants for indexes and aliases.
Update tests.
* [CI] Auto commit changes from spotless
---------
Co-authored-by: elasticsearchmachine <infra-root+elasticsearchmachine@elastic.co>
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
* add index privileges to support reroute processor
* [CI] Auto commit changes from spotless
* add more indices
* update
* fix test
* remove unnecessary indices
---------
Co-authored-by: elasticsearchmachine <infra-root+elasticsearchmachine@elastic.co>
This PR addresses ES-12071.
We want to collect metrics for the time that is spent waiting for the next chunk of a bulk request. This can help with diagnosing high bulk latency in case the latency is attributable to external factors such as network connection.
Co-authored-by: Francisco Fernández Castaño <francisco.fernandez.castano@gmail.com>
When migrating RepositoriesMetadata from cluster custom to project
custom (#125398), we needed temporary BWC handling for clusters running
on a version that is before this change but after the initial MP change.
Such a cluster can only exist in the serverless environment which has
progressed way past any applicable versions. Therefore we no longer need
the BWC handling and this PR removes it.
Relates: #125398
This PR addresses a bug where aborted merges are blocked if there's
insufficient disk space.
Previously, the merge disk space estimation did not consider if the
operation has been aborted when/while it was enqueued for execution.
Consequently, aborted merges, for e.g. when closing a shard, were
blocked if their disk space estimation was exceeding the available disk
space threshold. In this case, the shard close operation would itself
block.
This fix estimates a disk space budget of `0` for aborted merges, and it
periodically checks if any enqueued merge tasks have been aborted (more
generally, it checks if the budget estimate for any merge tasks has
changed, and reorders the queue if so). This way aborted merges are
prioritized and are never blocked.
Closes https://github.com/elastic/elasticsearch/issues/129335
In the last two months a lot of tests were converted to use the newer rest test framework. Some tests start 1 node, other start 3 nodes, others even more, the framework runs tests in parallel but it doesn't know how many nodes its tests needs meaning that running 3 tests in parallel, for example, can be very different when they are single node clusters or 3 node clusters etc. During this execution we saw the 3x more CPU load than what we would want to have ideally.
Currently there is no good solution for this because if dial down the concurrency we will use the nodes inefficiently, but if we keep the concurrency to where it is we risk longer start up times. Considering that the starting time of elasticsearch is not related to this test, we choose to increase the timeout to reduce the noise.
Luigi Dell'Aquila
Rene Groeschke
Sam Xiao
Martijn van Groningen
elasticsearchmachine
Ignacio Vera
Chris Hegarty
Simon Chase
Nik Everett
Parker Timmins
Lorenzo Dematté
Jonathan Buttner
Niels Bauman
Carlos Delgado
Ievgen Degtiarenko
Iraklis Psaroudakis
Luke Whiting
Ioana Tagirta
Moritz Mack
Jan Kuipers
Antonio
Julia Bardi
Dimitris Rempapis
Nick Tindall
Ankita Kumar
Yang Wang
Brendan Cully
Iván Cea Fontenla
Mikhail Berezovskiy
Albert Zaharovits
Mary Gouseti
Aurélien FOUCRET