With the removal of some ref counting that one ensureOpen became
redunndant. Also, no need to keep a reference to the channel now that it
doesn't hold any state.
The current slicing algorithm for concurrent search only creates multiple slices iff it can add at least 50k documents in each slice. This means that we need to have at least 100K documents in order to trigger concurrent search.
In tests, we can create several segments, but normally those segments contains few documents, therefore they will never trigger concurrent search.
This PR adds a cluster setting that can control the minimum number documents we require to create multiple slices. This setting is only registered on tests, therefore it is not exposed to the users. During test we can set this number to a
very low value to increase the likelihood that the test will exercise the concurrent code-path.
This change affects ESIntegTestCase as well as EsSingleNodeTestCase.
---------
Co-authored-by: Luca Cavanna <javanna@apache.org>
in serverless packages often are named with co.elastic prefix
validate module was preventing to use a convention where a dominating root package
is a name of the module.
This commit makes validateModule task to allow co.elastic module name
Build flavor was added back in #97770, but the main endpoint response
is still hardcoded to "default". This commit updates the response to
return the flavor from the build info.
Xpack core contains dozens of templates. These files are loaded via a
shared template utils method. While they should ideally be moved to
their appropriate modules, that would be large undertaking. This commit
moves these resources to a single module for xpack core (which may be
overriden by serverless).
We shouldn't complete the publication listener under the `mutex`, and
it's a little tricky to see which exceptions are or are not impossible
from the way the code is written today.
This commit solves all that by throwing the exception out to a handler
outside the mutex, and splitting the enclosing `try...catch` block into
several smaller blocks.
It also fixes a bug where we could fail to complete the publish listener
if the publication failed to start properly. With this change, we now
clearly complete the publication listener on all failure paths.
Closes#97798
Saw this one in recent profiling sessions -> remove redundant multiple
iterations of the results collection and cleanup slightly inefficient
assertion usage.
In many cases we access the same file in rapid succession. We can cache
the most recently accessed region in the file to avoid the costly lookup
from the CHM. We only do this for a time window that is inside the
throttle time for frequency counting in the LFU logic to avoid the
overhead of randomly jumping to a synchronized method on the fast-path.
In #87246 we describe some reasons why it's a good idea to limit the doc
count of a shard, and we started to do so in #94065, so this commit
adjusts the sizing guidance docs to match.
No need to use ref-counting here to keep the shared bytes alive after
they got closed. SharedBytes live for as long as the node lives, so if
this ever becomes an issues we should fix it by fixing the shutdown
order of services and not deal with the significant overhead of
ref-counting on every read.
To support differentiation of serverless product types it's now
possible to disable subsets of ML functionality.
The ML usage action was not updated to reflect this, so still
attempted to call actions to find out about usage of disabled
sub-features. Because that action is designed to never fail,
it actually returned reasonable output in these cases. However,
it generated a lot of log spam with warnings about calling
non-existent actions.
This PR adjusts the logic of the ML usage action to take into
account which sub-features might be disabled, and avoid calling
actions which won't exist in this case.
With this commit we add the required infrastructure to detect whether
indices need to be migrated and define migrations to update mappings or
dynamic index settings.
Security system indices should rely on the implicit
`index.refresh_interval` setting, which is tuned by "usecase", rather
than have it hard-coded by the Security plugin.
Tweaks our snyk integration to support different snyk orgs which is
required to have support for elasticsearch serverless. Also replaces
hardcoded remote url property by a dynamic version that resolves that
property from the git origin url.
The Build class keeps information about the running build of
Elasticsearch. There was previously a build flavor, which has since been
narrowed to just return the "default" flavor. This commit adds flavor
back to Build so that a plugged in Build can include a change to flavor.
With the expansion of the data stream lifecycle configuration that is
coming (for example, downsampling). It will be helpful to reduce the
number of constructors and mainly use the builder to initialise the
DataStreamLifecycle object.
The desired balance API response is chunked but it includes the
`ClusterInfo` in a single chunk, and only splits up the routing table
into per-index chunks. This commit makes the chunking more fine-grained.
If a shard has no desired assignment then we cannot send the desired
balance over the wire or render it as JSON because the
`ShardAssignmentView` is `null`. This commit replaces the spurious
`null` with an empty object, and cleans up the tests a little to remove
some unnecessary mocking.
* Avoid creating capturing lambda for hot-path region looking
* Made assertion code not run if asserts are off
* and used proper atomic array instead of manually messing with an array of atomics in assertions
* Extracted cold path for init cache chunk
If a shard has no desired assignment then we cannot send the desired
balance over the wire or render it as JSON because the
`ShardAssignmentView` is `null`. This commit replaces the spurious
`null` with an empty object, and cleans up the tests a little to remove
some unnecessary mocking.
We have allowed hybrid search since 8.4. This means the internal changes for sub_searches must be
able to write a compound query as early as 8.4, but currently we only do that back to 8.8. This change
fixes that issue.
Closes ##97144
Prior to this PR, when a channel was accepted after Elasticsearch
received a shutdown signal, `AbstractHttpServerTransport` would directly
close the connection. This PR makes it throw an exception instead, which
results in the connection being closed by Netty. This keeps the flow of
control more consistent and readable, as well as logging additional
details such as the host which made the request.
This PR doesn't include any test changes because the behavior doesn't
really change in a way that's easy to test.
Changed the downsample threadpool size to one eight of the allocated processors. Downsampling is a cpu intensive operation (search + heavy aggregation that rolls up documents) and could overwhelm a node. On top of this the downsample operation also delegates to force merge api, which has a threadpool that is also bounded to one eight of the allocated processors.
Relates to #97141
The `JoinValidationService` doesn't need to send the cluster state if
we're not currently the master - instead it can just send a join ping.
This means it doesn't need to do any work adjusting the blocks.
Closes#97313
Armin Braun
Marantidis Kiriakos
Ignacio Vera
Przemyslaw Gomulka
Ryan Ernst
David Turner
Abdon Pijpelink
Luigi Dell'Aquila
David Roberts
Iraklis Psaroudakis
Albert Zaharovits
Daniel Mitterdorfer
Rene Groeschke
Slobodan Adamović
Ievgen Degtiarenko
Saarika Bhasi
Tim Vernum
Mary Gouseti
Nicolas Chaulet
Ioana Tagirta
Jonathan Buttner
István Zoltán Szabó
Jack Conradson
Athena Brown
Martijn van Groningen
Mark Vieira