This splits out the registry and the service, which makes testing easier and removes much of the delegation from the old `APMMeter` to `Instruments` (now renamed `APMMeterRegistry`).
APMMeterService takes care of the lifecycle and APMMeterRegistry holds the instruments.
This PR wires the new Meter interface into S3BlobStore. The new meter
field remains unused in this PR. Actual metric collection will be
addressed in follow-ups.
Relates: ES-6801
Today blobstore stats are collected against each HTTP operation, e.g.
Get, List. This is not granular enough because the same HTTP operration
can be performed for different purposes, e.g. cluster state, indices or
translog. This PR adds a new Purpose enum to provide further breakdown
for the same HTTP operation.
Relates: ES-6800
Adds @SuppressWarnings("this-escape") to all necessary places to that
Elasticsearch can compile with -Werror on JDK21
No investigation has been done to determine whether any of the cases
are a potential source of errors - we have simply suppressed all
existing occurrences.
Resolves: #99845
* Update Guava dependency version
* Update gradle verification metadata
* fix :repository-gcs:thirdPartyAudit
* fix :discovery-gce:thirdPartyAudit
* Use single sha256 hash in allowed list
Drying this up further and adding the same short-cut for single node
tests. Dealing with most of the spots that I could grab via automatic
refactorings.
Currently we have a number of input streams that specifically override
the skip() method disabling the ability to skip bytes. In each case the
skip implementation works as we have properly implemented the
read(byte[]) methods used to discard bytes. However, we appear to have
disabled it as it would be possible to retry from the end of a skip if
there is a failure in the middle. At this time, that optimization is not
really necessary, however, we sporadically used skip so it would be nice
for the IS to support the method. This commit enables the super.skip()
and adds a comment about future optimizations.
Today the blob store register supports recording only a `long`,
represented as an 8-byte blob. We need to store a little more data in
the register, so this commit generalises things to work with a
`BytesReference` directly.
Instead of using raw TCP sockets and parsing HTTP requests manually which is tends to be quite error-prone, we can create an HTTP proxy server using ApacheHttpServer. In contrast to com.sun.net an Apache HTTP server it can act as an HTTP proxy.
This change also allows the server to support persistent keep-alive connections. It should prevent creating excessive threads to handle new HTTP connection which matters when we run heavy tests in GcsProxyIntegrationTests.
It's a more straighforward alternative to #93891.
Fixes#93824Fixes#93811Fixes#93816
Further work towards the S3 compare-and-exchange implementation showed
that we would like this API to permit async operations. This commit
moves to an async API.
Also, this change made it fairly awkward to use an exception to deliver
to the caller the indication that the current value could not be read,
so this commit adjusts things to use `OptionalLong` throughout as
suggested in the discussion on #93955.
Only for testing purposes through the `FsRepository` for now and rather simple,
but should get the job done and technically be correct for a compliant NFS implementation.
Co-authored-by: David Turner <david.turner@elastic.co>
Currently, if the user set proxy setting for the GCS repository, they are only used for interactions with the GCS API, but not the other network interactions that the the module performs (refreshing OAuth tokens, fetching metadata) outside of the SDK.
Fix that by making sure that all network interactions go via proxy if it's configured.
Fixes#91952
Some optimisations that I found when reusing searchable snapshot code elsewhere:
* Add an efficient input stream -> byte buffer path that avoids allocations + copies for heap buffers, this is non-trivial in its effects IMO
* Also at least avoid allocations and use existing thread-local buffer when doing input stream -> direct bb
* move `readFully` to lower level streams class to enable this
* Use same thread local direct byte buffer for frozen and caching index input instead of constantly allocating new heap buffers and writing those to disk inefficiently
Use local-independent `Strings.format` method instead of `String.format(Locale.ROOT, ...)`.
Inline `ESTestCase.forbidden` calls with `Strings.format` for the consistency sake.
Add `Strings.format` alias in `common.Strings`
Upgrade to the latest version of the SDK that doesn't have known CVEs
and builds w/o complaining. Since 2.2.0 the automatic retry behaviour
has changed and the old behaviour can still be used as
LegacyStorageRetryStrategy. The default retry strategy would cause some
test failures, and therefore we'd need to explicitly set a retry
strategy.
Relates #92474
This commit adds a new test framework for configuring and orchestrating
test clusters for both Java and YAML REST testing. This will eventually
replace the existing "test-clusters" Gradle plugin and the build-time
cluster orchestration.
This comes out of a user heap dump investigation. In some snapshot
corner cases we ran into about 100M of duplicate 0b instances.
-> even though it's a little heavy handed, lets make it so the common
constants that we already have are used whenever possible.
As discussed, we can be up to twice as fast without increasing CPU use
much on high latency blob stores so increasing the pool size to 10 here
to better utilize larger data nodes.
Today if the GCS credentials file setting is invalid we report some kind
of JSON parsing error but it's not clear what JSON is being parsed so
the error is hard to track down. This commit adds the problematic
setting name to the exception message.
Removing the custom dependency checksum functionality in favor of Gradle build-in dependency verification support.
- Use sha256 in favor of sha1 as sha1 is not considered safe these days.
Closes https://github.com/elastic/elasticsearch/issues/69736
This is a result of structural search/replace in intellij. This only affects log methods with a signature
logger.info(ParametrizedMessage)
logger.info(ParametrizedMessage, Throwable)
relates https://github.com/elastic/elasticsearch/issues/86549
This is a result of structural search/replace in intellij. This only affects log methods with a signature
logger.info(Supplier<?>, Throwable) where level could be info/debug etc and supplier argument is in a form of
()-> new ParameterizedMessage
relates #86549
This is a result of structural search/replace in intellij. This only affects log methods with a signature
logger.info(Supplier<?>) where level could be info/debug etc and supplier argument is in a form of
()-> new ParameterizedMessage
This commit also introduced a Strings utility class to avoid passing Locale.ROOT to every
String.format(Locale.ROOT, pattern, args)
relates #86549
Most of the Jackson uses, eg in x-content and azure, have already been
upgraded. This commit upgrades the rest of the uses. Note that it does
not yet upgrade the aws sdk, this should also be done on its own.
Most classes under elasticsearch-core had been moved to the o.e.core
package. However, a couple io related classes remained in an "internal"
package. This commit moves Streams and IOUtils to the core package, as
they are no more "internal" than the rest of the classes in core.
This removes the overhead of zipping modules that are referenced in
test cluster configurations
We now Support declaring task providers as module input in testclusters
I noticed a few spots where we rely on the not-guaranteed
behavior of `Collectors.toList()` producing a mutable List.
I fixed those that Idea could quickly find and replaced
many other spots that were using the collector with `toList()`
which always creates a faster+immutable list (to make sure
we were not relying on mutable lists in more spots).
Relates to #84760
This change isolates the Jackson implementation of x-content parsers and generators to a separate classloader. The code is loaded dynamically upon accessing any x-content functionality.
The x-content implementation is embedded inside the x-content jar, as a hidden set of resource files. These are loaded through a special classloader created to initialize the XContentProvider through service loader. One caveat to this approach is that IDEs will no longer trigger building the x-content implementation when it changes. However, running any test from the command line, or running a full Build in IntelliJ will trigger the directory to be built.
Co-authored-by: ChrisHegarty <christopher.hegarty@elastic.co>
We disallow including guava on the compile classpaths, but currently
this only forbids the main guava artifact. Yet there are other artifacts
(eg failureaccess) provided by guava. This commit changes the dependency
check to use the guava group instead of artifact name.
Follow-up from #77144 (comment) with converting id/_id to always be strings instead of integers. This makes the type value in the Elasticsearch specification be only string instead of string | number.
this change was generated using following command on ubuntu
find . -type f -name "*.yml" -print0 | xargs -0 sed -i -r 's/([^a-zA-Z0-9_\.]id|[^a-zA-Z0-9_]_id):(\s*)([0-9]+)/\1:\2"\3"/g'
This method's name is trappy: it is easy to misinterpret it as returning
an instance from the elected master, but in fact it uses any
master-eligible node. If you want an instance from the elected master,
you have to use `getCurrentMasterNodeInstance()` instead.
This commit renames the method to clarify that it might not get an
instance from the elected master, and adds docs with cross-refs to help
developers choose the right method.
After we fixed getting application credentials in a GCE environment in #82974, we can actually
get credentials set automatically when creating a new GCS client
Fixes#83131
Stuart Tettemer
Yang Wang
Tim Vernum
Francisco Fernández Castaño
Volodymyr Krasnikov
Armin Braun
Pooya Salehi
Tim Brooks
Mark Vieira
David Turner
Artem Prigoda
Joe Gallo
Jake Landis
Rene Groeschke
Iraklis Psaroudakis
Keith Massey
Ievgen Degtiarenko
Chris Hegarty
Przemyslaw Gomulka
Ryan Ernst