Commit Graph

130 Commits

Author SHA1 Message Date
Armin Braun b7eafce32c
Make some practically static methods static (#97565)
Another round of automated fixes to this, marking things that can be
made static as static. Saves some JIT cycles but also turns some lambdas
from capturing to non-capturing and makes the "utilityness" of some
classes visible.
2023-10-06 23:37:07 +02:00
Nik Everett 0e219307f2
ESQL: Track blocks (#100025)
This tracks blocks from topn and a few other places. We're going to try
and track blocks all the places.
2023-09-28 13:17:08 -04:00
Ryan Ernst a9e6da7336
Don't use test name in jarhell test (#98950)
This commit fixes a jarhell test to create an unnamed temp dir, instead
of the existing creation which uses the test method name. The reason
this causes problems is when running with many iterations, the test
method name is artificially adjusted to include seed information, using
special characters that are potentially invalid path characters.

closes #98949
2023-08-28 14:02:42 -07:00
Armin Braun 63e64ae61b
Cleanup Stream usage in various spots (#97306)
Lots of spots where we did weird things around streams like redundant stream creation, redundant collecting
before adding all the collected elements to another collection or so, redundant streams for joining strings
and using less efficient `Collectors.toList` and in a few cases also incorrectly relying on the result being mutable.
2023-07-03 14:24:57 +02:00
David Turner eca1e4f831
Avoid using assertOnce things as map keys (#96565)
It's legitimate to wrap the delegate twice, with two different
assertOnce calls, which would yield different objects if and only if
assertions are enabled. So we'd better not ever use these things as map
keys etc.
2023-06-06 08:31:33 +01:00
Armin Braun 1ae63ac315
Optimize KeyedLock and related concurrency primitives (#96372)
This class was quite hot in recent benchmarks of shared-cached based
searches and we can make instantiating the releasable locks a little cheaper.
Also, those same benchmarks showed a lot of visible time spent on
dealing with ref counts. I removed one layer of indirection in atomic
use from both the release-once and the abstract ref count which
should save a little in CPU caches as well.
2023-05-30 21:55:31 +02:00
David Turner 7eaa868295
Reduce indirection in AbstractRefCounted (#96403)
Today `AbstractRefCounted` holds an `AtomicInteger` which holds the
actual ref count, which is an extra heap object and means that
acquiring/releasing refs always goes through that extra pointer lookup.
We use this utility extensively, on some pretty hot paths, so with this
commit we move to using a primitive `refCount` field with atomic
operations via a `VarHandle`.
2023-05-29 10:54:52 +01:00
Ryan Ernst 9c370b8d47
Allow qualified exports to extended modules (#95243)
In #94884 the ability to add qualified exports and opens from jars
upstream of server was added. Some Elasticsearch components need to
qualify their exports to another component. This commit tweaks the
loading of the exports services so that each loaded plugin/component
has their qualified exports handled automatically.
2023-05-02 14:01:13 -07:00
Ievgen Degtiarenko bc6b5c9672
Remove MemoizedSupplier that is no longer used. (#95219)
This commit removes MemoizedSupplier that is no longer used across the codebase
2023-04-18 09:11:49 +02:00
Ryan Ernst c619be4b5e
Move preallocate module to libs (#94884)
The preallocate module needs access to java.io internals. However, in
order to open java.io to a specific module, rather than the unnamed
module as was previously done, the said module must be in the boot
layer.

This commit moves the preallocate module to libs. It adds it to the main
lib dir, though it does not add it as a compile dependency of server.
2023-04-10 13:05:43 -07:00
Rory Hunter fe1083f6c5
Upgrade spotless plugin to 6.17.0 (#94994)
Fixes #82794. Upgrade the spotless plugin, which addresses the issue
around formatting `instanceof` expressions. Formatting of statements
including lambdas seems to have improved too.
2023-04-04 10:03:32 +01:00
David Turner aef7dc0826
Improve toString() on Releasables utils (#94626)
Today both `Releasables#wrap` implementations return a lambda which does
not render well in logs or exception messages. This commit improves
that. It also adds tests for these methods and moves the test suite to
the correct module.
2023-03-27 08:34:11 +01:00
David Turner 6da721edda
Assert TaskTransportChannel completed once (#94598) 2023-03-21 13:56:45 -04:00
Armin Braun f2760c6e18
Nicer buffer handling (#93491)
Some optimisations that I found when reusing searchable snapshot code elsewhere:
* Add an efficient input stream -> byte buffer path that avoids allocations + copies for heap buffers, this is non-trivial in its effects IMO
  * Also at least avoid allocations and use existing thread-local buffer when doing input stream -> direct bb
  * move `readFully` to lower level streams class to enable this
* Use same thread local direct byte buffer for frozen and caching index input instead of constantly allocating new heap buffers and writing those to disk inefficiently
2023-02-06 10:55:56 +01:00
David Turner 1a32789985
Ensure one-shot wrappers release their delegates (#92928)
We recently adjusted `RunOnce`, `Releasables#releaseOnce` and
`ActionListener#notifyOnce` so that once they have fired they drop the
now-unnecessary reference to the delegate. This commit introduces tests
to verify that this reference does genuinely become unreachable (i.e.
available for garbage collection) as expected.

It also fixes a bug in `ActionListener#notifyOnce` which caused us to
unexpectedly retain a reference to the delegate 🤦

Relates #92452 Relates #92507 Relates #92537
2023-01-16 14:38:22 -05:00
Artem Prigoda 2bc7398754
Use `Strings.format` instead of `String.format(Locale.ROOT, ...)` in tests (#92106)
Use local-independent `Strings.format` method instead of `String.format(Locale.ROOT, ...)`. 
Inline `ESTestCase.forbidden` calls with `Strings.format` for the consistency sake.
Add `Strings.format` alias in `common.Strings`
2023-01-03 19:28:27 +01:00
David Turner eb8cb109a4
Misc improvements to AbstractRefCounted (#92616)
Adds a null check and a `toString()` implementation which passes through
to the wrapped runnable. Also renames `RefCountedTests` to
`AbstractRefCountedTests` since they're really all about testing this
specific implementation.
2023-01-03 11:42:04 +00:00
David Turner 823b5dad9e
Release delegate ref in Releasables#releaseOnce (#92537)
Like #92452 and #92507 but for `Releasables#releaseOnce`: there's no
need to keep hold of the wrapped releasable after closing it, and in
some cases this might hold on to excessive heap. With this commit we
drop the reference to the delegate when it's complete.
2022-12-23 06:22:25 -05:00
David Turner f8636c6313
Fix ref-counting in DisruptableMockTransport (#92245)
Today `DisruptableMockTransport` leaks refs to transport messages in
various ways if the transport is rebooted. This commit adds the missing
ref-count handling.

Closes #91837
2022-12-12 10:26:26 -05:00
Rene Groeschke fe93b8161b
Provide default jarHell configuration dependencies (#90810)
With the new StableBuildPlugin we do not add default dependencies to the plugin classpath.
That exposed an issue with the JarHellPlugin not really taking care of configuring the
jarHell configuration.
The JarHell Plugin only worked so far as the plugin build plugin has added a transitive dependency
to elasticsearch-core and therefore kind of only worked by accident-.

This adds a default configuration of the jarHell configuration that can be overridden manually.
Furthermore we clear some inter plugin dependencies explicit and add proper functional
test coverage
2022-10-12 14:15:30 +02:00
Rene Groeschke 43a0377735
Update forbiddenapis to 3.4 (#90624)
Fix breaking changes to source validation after change in default jdk rule set
2022-10-06 16:52:06 +02:00
Armin Braun 099fb84cfc
Remove CompletableContext and replace it with StepListener (#90415)
These two mostly do the same, making `CompletableContext` and
the compatibility layer between the context and `ActionListener`
redundant.
Removing it here to in part to set up a clean solution for #77999.
2022-09-28 11:12:22 +02:00
Tanguy Leroux f2154e8687
Fix unnecessary string concatenations (#90405) 2022-09-27 16:14:38 +02:00
Artem Prigoda 8a159e9759
Make Tuple a record (#88280)
Tuple is used extensively across the ES codebase and can be effectively represent as a Java record.
2022-07-28 09:14:12 +02:00
Chris Hegarty f3cff66877
Fix EmbeddedImplClassLoaderTests on Windows (#88813) 2022-07-27 00:08:37 +01:00
Chris Hegarty 1ce64290a1
Add package cache to EmbeddedImplClassLoader (#88537)
The initial implementation of the embedded class loader took a brute
force approach to supporting multi-release JARs - iterating over all
possible release versions when searching for classes and resources. This
change improves upon that approach by deriving and caching package and
version specific maps, so class and resource loading can go directly to
the class and resource bytes, respectively, rather than searching.

It's hard to get empirical numbers to quanify just how much this change
improves the performance of classes loaded by this loader, and there is
typically only a couple of hundred classes loaded, but the initial cli
seems observably much quicker, while the server startup has improved
just a bit (at least on my machine).
2022-07-25 15:31:58 +01:00
Chris Hegarty 810736519f
Always close directory streams (#88560) 2022-07-15 08:49:16 +01:00
Ignacio Vera ff6604f1ea
Use a faster but less accurate log algorithm for computing Geotile Y coordinate (#87515)
This commit introduces a new algorithm to ESSloppyMath to compute logarithm (base e)
2022-06-29 12:25:22 +02:00
Armin Braun 64798df2f3
Cleanup indirection in batch close utils (#87685)
Some of these show up in profiling of heavy transport load.
It's not a big thing but it's always nice to save a few cycles
on transport threads.
2022-06-22 11:54:36 +02:00
Chris Hegarty ac9e1013bb
Re-enable several tests since JDK-8287097 has been fixed (#87803) 2022-06-17 16:46:33 +01:00
Ryan Ernst 779871e73c
Remove impossible checked exception from jarhell (#87541)
The jarhell check declares a URISyntaxException. However, this should
not be possible as the paths and URLs come from the jdk conversion. This
commit makes a URISyntaxException when converting form URL to URI an
assertion error, similar to MalformedURLException when creating a URL.
2022-06-09 06:22:13 -07:00
Armin Braun f2987b417f
Lower overhead of RestApiVersion use in x-content parsing (#87356)
Noticed loads of duplicate lambdas on the heap and code related to these predicates
etc. pop up during benchmarking things that are hot on x-content parsing.
These changes way simplify the rest-api code (though it could be made even simpler I think)
and remove it from profiling for the most part.

Co-authored-by: Joe Gallo <joe.gallo@elastic.co>
2022-06-08 12:22:17 +02:00
Przemyslaw Gomulka 416a1b352c
Catch an exception due to incorrect pattern in Strings.format (#87132)
Strings.format method, which is used heavily in logging with
Supplier should handle exceptions when a format is incorrect.
This will prevent a hard to catch mistakes to blow up in server.
Those mistakes are especially hard to detect in logging when a
code to create a message might be only executed when logger is debug
or trace. Which is not always the case in CI.

relates #87077 (comment)

relates #86549
2022-05-30 09:05:40 +02:00
Przemyslaw Gomulka 24fa003f5c
Replace supplier of ParameterizedMessage with java.util.Supplier<String> (#86971)
This is a result of structural search/replace in intellij. This only affects log methods with a signature
logger.info(Supplier<?>) where level could be info/debug etc and supplier argument is in a form of
()-> new ParameterizedMessage

This commit also introduced a Strings utility class to avoid passing Locale.ROOT to every
String.format(Locale.ROOT, pattern, args)
relates #86549
2022-05-23 08:51:07 +02:00
Chris Hegarty 0d5db357df
Skip on 19 (#87000) 2022-05-20 20:49:27 +01:00
Chris Hegarty 3071c6a055
Modularize Elasticsearch (#81066)
This PR represents the initial phase of Modularizing Elasticsearch (with
Java Modules).

This initial phase modularizes the core of the Elasticsearch server
with Java Modules, which is then used to load and configure extension
components atop the server. Only a subset of extension components are
modularized at this stage (other components come in a later phase).
Components are loaded dynamically at runtime with custom class loaders
(same as is currently done). Components with a module-info.class are
defined to a module layer.

This architecture is somewhat akin to the Modular JDK, where
applications run on the classpath. In the analogy, the Elasticsearch
server modules are the platform (thus are always resolved and present),
while components without a module-info.class are non-modular code
running atop the Elasticsearch server modules. The extension components
cannot access types from non-exported packages of the server modules, in
the same way that classpath applications cannot access types from
non-exported packages of modules from the JDK. Broadly, the core
Elasticseach java modules simply "wrap" the existing packages and export
them. There are opportunites to export less, which is best done in more
narrowly focused follow-up PRs.

The Elasticsearch distribution startup scripts are updated to put jars
on the module path (the class path is empty), so the distribution will
run the core of the server as java modules. A number of key components
have been retrofitted with module-info.java's too, and the remaining
components can follow later. Unit and functional tests run as
non-modular (since they commonly require package-private access), while
higher-level integration tests, that run the distribution, run as
modular.

Co-authored-by: Chris Hegarty <christopher.hegarty@elastic.co>
Co-authored-by: Ryan Ernst <ryan@iernst.net>
Co-authored-by: Rene Groeschke <rene@elastic.co>
2022-05-20 13:11:42 +01:00
Chris Hegarty 74c03c21dc
Expand jar hell to include modules (#86622)
Expands jar hell to include modules from the system class loader.
2022-05-13 07:54:12 +01:00
Chris Hegarty ddc354f62f
Make embedded class loader module aware (#86355)
This change adds support to embedded class loader to load the provider
and implmentation dependencies as modules - within their own module
layer - when the caller itself is a named module. Currently, this code
is not yet triggered during deployment, since the caller is always an
unnamed module, but the caller will be moularized in a subsequent
change.
2022-05-10 09:20:15 +01:00
Chris Hegarty e30a069839
Fix Windows EmbeddedImplClassLoaderTests (#86413)
Fix Windows EmbeddedImplClassLoaderTests. Remove use of openStream in favor of getResourceAsStream.

Co-authored-by: Ryan Ernst <ryan@iernst.net>
2022-05-06 09:53:05 +01:00
Alan Woodward 4d076eee20
Upgrade to Lucene 9.2 snapshot efa5d6f4d43 (#86227)
Notable changes include:

count implementations for MultiRangeQuery and IndexSortedNumericDocValuesRangeQuery, which may speed up certain aggregations
more efficient decoding of docids in BKD reader
2022-05-05 15:48:13 +01:00
Armin Braun 1c497ac516
Fix using FilterOutputStream without overriding bulk write (#86304)
Fix using the filter output stream without overriding the bulk write.
The usage in `directFieldAsBase64` seems like a serious performance bug
since the stream is used to write a potentially larger response here.

I also removed the `BlobOutputStream` that used to contain the same
fix now added to the no-close stream after realizing the class is pointless
to begin with to cut down on our usage of `FilterOutputStream` where the bulk
write fix is needed.
2022-05-02 21:25:14 +02:00
Chris Hegarty a6177121ea
Make embedded class loader MRJAR aware (#86316)
This change updates the embedded class loader to be MRJAR aware.
2022-05-01 12:23:46 +01:00
Chris Hegarty 06b14e57ec
Move embedded loader and provider locator to core (#86081) 2022-04-25 08:23:45 +01:00
Chris Hegarty ce6d2e7f31
remove xmlbeans jarhell exception (#86100) 2022-04-22 19:52:07 +01:00
Ryan Ernst b2c9028384
Move io utils to core package (#85954)
Most classes under elasticsearch-core had been moved to the o.e.core
package. However, a couple io related classes remained in an "internal"
package. This commit moves Streams and IOUtils to the core package, as
they are no more "internal" than the rest of the classes in core.
2022-04-19 21:26:28 -07:00
Ryan Ernst d9e6fc8161
Move netutil to netty module (#85953)
This utils class was previously shared across transport implementations,
but is only used by netty now.
2022-04-19 14:53:56 -07:00
Ryan Ernst e307f3222f
Check only major version for java compatibility (#85886)
This commit adjusts the version check for plugins to only check the Java
major version, instead of the entire thing.

fixes #85880
2022-04-14 11:02:57 -07:00
Ryan Ernst 0832232b99
Replace JavaVersion with Runtime.Version (#85361)
Since Java 9, the JDK has provided a means of parsing Java versions and
getting the current Java version. That class obviates the need for the
JavaVersion class of Elasticsearch. This commit removes the JavaVersion
class in favor of Runtime.Version.

Note that most of the changes here simply removed logic around
versioning because this change is intended only for the master branch,
where Java 17 is required.
2022-03-28 12:46:59 -07:00
Chris Hegarty 50528d5d79
Add missing explicit no-args ctors (#84763) 2022-03-09 11:08:48 +00:00
Mayya Sharipova 26c3dd6857
Upgrade to lucene-9.1.0-snapshot-1336263051c (#83667)
Lucene issues that resulted in elasticsearch changes:

LUCENE-9820 Separate logic for reading the BKD index from logic to intersecting it.
LUCENE-10377: Replace 'sortPos' with 'enableSkipping' in SortField.getComparator()
LUCENE-10301: make the test-framework a proper module by moving all test
classes to org.apache.lucene.tests
LUCENE-10300: rewrite how resources are read in ukrainian morfologik analyzer:
LUCENE-10054 Make HnswGraph hierarchical
2022-02-22 09:53:20 +01:00