* Fix ExceptionSerializationTests to use getCodeSource instead of getResource.
Using getResource makes this sensitive to unrelated classpath entries,
such as the entitlement bridge library, that get prepended to the classpath.
* FIx logging tests to use org.elasticsearch.index instead of root logger.
Using the root logger makes this sensitive to unrelated
logging, such as from the entitlement library.
* Fix entitlement error message by stashing the module name in ModuleEntitlements.
Taking the actual module name from the class doesn't work in tests,
where those classes are loaded from the classpath and so their module
info is misleading.
* Ignore server locations whose representative class isn't loaded
* Partial initial implementation
* System properties: testOnlyClasspath and enableForTests
* Trivially allow some packages
* DEBUG: use TreeMap in TestScopeResolver for readability
* Special case bouncycastle for security plugin
* Add CONFIG to TestPathLookup
* Add the classpath to the source path list for every plugin
* Add @WithoutEntitlements to tests that run ES nodes
* Set es.entitlement.enableForTests for all libs
* Use @WithoutEntitlements on ingest plugin tests
* Substitute ALL-UNNAMED for module name in non-modular plugins
* Add missing entitlements found by unit tests
* Comment in TestScopeResolver
* Properly compute bridge jar location for patch-module
* Call out nonServerLibs
* Don't build two TestPathLookups
* More comments for meta-tests
* Remove redundant dependencies for bridgeJarConfig.
These are alread set in ElasticsearchJavaBasePlugin.
* Add bridge+agent dependencies only if those exist.
For serverless, those project dependencies don't exist, and we'll need
to add the dependencies differently, using Maven coordinates.
* [CI] Auto commit changes from spotless
* Pass testOnlyPath in environment instead of command line.
It's typically a very very long string, which made Windows angry.
* [CI] Auto commit changes from spotless
* Split testOnlyPathString at File.pathSeparator
* Use doFirst to delay setting testOnlyPath env var
* Trivially allow jimfs (??)
* Don't enforce entitlements on internalClusterTest for now
* Replace forbidden APIs
* Match testOnlyClasspath using URI instead of String.
We already get the "needle" in the form of a URI, so this skips
a step, and has the benefit of also working on Windows.
* [CI] Auto commit changes from spotless
* More forbidden APIs
* Disable configuration cache for LegacyYamlRestTestPluginFuncTest
* Strip carriage-return characters in expected output for ReleaseNotesGeneratorTest.
The template generator also strips these, so we need to do so to make this pass
on Windows.
Note that we use replace("\r", "") where the template generator uses
replace("\\r", ""). The latter didn't work for me when I tried it on Windows,
for reasons I'm not aware of.
* Move configureEntitlements to ElasticsearchTestBasePlugin as-is
* Use matching instead of if
* Remove requireNonNull
* Remove default configuration
* Set inputs instead of dependencies
* Use test.systemProperty
* Respond to PR comments
* Disable entitlement enforcement for ScopedSettingsTests.
This test works by altering the logging on the root logger.
With entitlements enabled, that will cause additional log statements to appear,
which interferes with the test.
* Address PR comments
* Moritz's configureJavaBaseModuleOptions
* Allow for entitlements not yet enforced in serverless
* fix entitlementBridge config after rename
* drop empty file collections
* Remove workaround in LegacyYamlRestTestPluginFuncTest
---------
Co-authored-by: elasticsearchmachine <infra-root+elasticsearchmachine@elastic.co>
Co-authored-by: Lorenzo Dematté <lorenzo.dematte@elastic.co>
Co-authored-by: Moritz Mack <mmack@apache.org>
optimize OptimizedScalarQuantizer#scalarQuantize when destination can optimize
OptimizedScalarQuantizer#scalarQuantize when destination can be an integer array
This commit updates the low-level bounds checks in JDKVectorLibrary and add benchmark, so that we can more easily bench the low-level operations.
Note: I added the mr-jar gradle plugin to the benchmarks so that we can compile with preview features in Java 21, namely MemorySegment.
When the entitelement agent fails to load the underlying exception can
be cryptic. In some casees it may be that the path the agent jar is bad.
This commit expands the exception message to show the agent path
that we tried to load.
Java class transformers swallow exceptions, so any instrumentation
failures, for example due to a java version mismatch, will silently
proceed with startup, which then will cryptically fail the entitlement
self test. This commit logs exceptions that occur during
instrumentation, as well as plumb through the fact that any occured so
that bootstrap can fail rather than allow startup to proceed.
Keep better track of shard contexts using RefCounted, so they can be released more aggressively during operator processing. For example, during TopN, we can potentially release some contexts if they don't pass the limit filter.
This is done in preparation of TopN fetch optimization, which will delay the fetching of additional columns to the data node coordinator, instead of doing it in each individual worker, thereby reducing IO. Since the node coordinator would need to maintain the shard contexts for a potentially longer duration, it is important we try to release what we can eariler.
An even more advanced optimization is to delay fetching to the main cluster coordinator, but that would be more involved, since we need to first figure out how to transport the shard contexts between nodes.
Summary of main changes:
DocVector now maintains a RefCounted instance per shard.
Things which can build or release DocVectors (e.g., LuceneSourceOperator, TopNOperator), can also hold RefCounted instances, so they can pass them to DocVector and also ensure contexts aren't released if they can still be potentially used later.
Driver's main loop iteration (runSingleLoopIteration), now closes its operators even between different operator processing. This is extra aggressive, and was mostly done to improve testability.
Added a couple of tests to TopNOperator and a new integration test EsqlTopNShardManagementIT, which uses the pausable plugin framework to check that TopNOperator releases things as early as possible..
Some features are unavailable in serverless and are thus not worth the
investment to make fully project-aware. This new annotation can be used
to clearly mark blocks of code that are intentionally not made properly
project-aware, in case we need to revisit them in the future.
Using a static `diff` or epsilon just doesn't work for this test as the
scores can be very large, but relatively close.
Maybe there is a simpler way, but my mind wasn't wanting to "math" very
much.
For example, the seed that this previously failed on had scores like
`1.726524E9` and `1.7265239E9`, which, given their size, are really
close together (within 128). But a static epsilon wouldn't capture that.
closes: https://github.com/elastic/elasticsearch/issues/128485
Follow-up to #126492 to apply the json parsing optimization to strings
containing unicode characters and some backslash-escaped characters.
Supporting backslash-escaped strings is tricky as it requires modifying the
string. There are two types of modification: some just remove the backslash
(e.g. \", \\), and some replace the whole escape sequence with a new
character (e.g. \n, \r, \u00e5). In this implementation, the optimization
only supports the first case--removing the backslash. This is done by
making a copy of the data, skipping the backslash. It should still be more
optimized than full String decoding, but it won't be as fast as
non-backslashed strings where we can directly reference the input bytes.
Relates to #129072.
* Support multiple plugin source paths
* Refactor: remove unncessary PathLookup method.
It's only called in one place, and there's no need to override it for testing.
Removing it just makes things simpler.
* Refactor: local var for pathLookup
* Fix bugs in test build info parsing
* Fix representative_class in test
* Move BridgeUtilTests.
Tests in org.elasticsearch.entitlement.bridge are going to be uniquely hard to
test once we patch the bridge into java.base, due to Java's prohibition on
split packages.
Let's just move this guy to another package.
* Upcast (?!) Java23EntitlementChecker to EntitlementChecker
* Empty TestPathLookup
* Create PolicyManager during bootstrap, allowing us to share initialization
* Use empty component path list instead of null
* Downcast to the class of the check method.
In our unit test, we have a mock checker that doesn't extend
EntitlementChecker, so downcasting to that would require us to needlessly
rework the unit test.
* Fix javadoc typos
This ensures we package an aggregation zip with all artifacts we want to publish to maven central as part of a release.
Running zipAggregation will produce a zip file in the build/nmcp/zip folder. The content of this zip is meant to match the maven artifacts we have currently declared as dra maven artifacts.
When parsing documents, we receive the document as UTF-8 encoded data which
we then parse and convert the fields to java-native UTF-16 encoded Strings.
We then convert these strings back to UTF-8 for storage in lucene.
This patch skips the redundant conversion, instead passing lucene a
direct reference to the received UTF-8 bytes when possible.
This PR is a precursor to #126492.
It does three things:
1. Move org.elasticsearch.common.text.Text from :server to
org.elasticsearch.xcontent.Text in :libs:x-content.
2. Refactor the Text class to use a new EncodedBytes record instead of
the elasticsearch BytesReference.
3. Add the XContentString interface, with the Text class implementing
that interface.
These changes were originally implemented in #127666 and #128316,
however they were reverted in #128484 due to problems caused by the
mutable nature of java ByteBuffers. This is resolved by instead using a
new immutable EncodedBytes record.
Instead of waiting for the next run of the `ClusterStateObserver` (which
might be arbitrarily far in the future, but bound by the timeout if one
is set), we notify the listener immediately that the task has been
cancelled. While doing so, we ensure we invoke the listener only once.
Fixes#117971
docs-build / docs-preview (push) Waiting to runDetails
Validate Gradle Wrapper / Validation (push) Waiting to runDetails
Entitlement instrumentation works by reflectively calling back into the
entitlements lib to grab the checker. It must be fully in place before
any classes are instrumented. This commit fixes a bug that was
introduced by refactoring which caused the checker to not be set until
after all classes were instrumented. In some situations this could lead
the checker to being null when it is grab (and statically cached) by the
entitlement bridge.
* Rename and encapsulate InitializeArgs
* Move ElasticsearchEntitlementChecker out of api package.
It's an implementation detail that doesn't need to be exposed to the rest of
the system.
* Stub TestPathLookup (not yet implemented)
* Revert "Fix the Text class package change in example plugins (#128316)"
This reverts commit cc486480e3.
* Revert "Update Text class to use native java ByteBuffer (#127666)"
This reverts commit db0c3c7a28.
Co-authored-by: Lorenzo Dematté <lorenzo.dematte@elastic.co>
* Use package to suppress warning for entitlement self-test
* [CI] Auto commit changes from spotless
---------
Co-authored-by: elasticsearchmachine <infra-root+elasticsearchmachine@elastic.co>
Entitlements do a stack walk to find the calling class. When method
refences are used in a lambda, the frame ends up hidden in the stack
walk. In the case of using a method reference with
AccessController.doPrivileged, the call looks like it is the jdk itself,
so the call is trivially allowed. This commit adds hidden frames to the
stack walk so that the lambda frame created for the method reference is
included. Several internal packages are then necessary to filter out of
the stack.
This PR is a precursor to #126492.
It does three things:
- Move org.elasticsearch.common.text.Text from :server to
org.elasticsearch.xcontent.Text in :libs:x-content.
- Refactor the Text class to use a java-native ByteBuffer instead
of the elasticsearch BytesReference.
- Add the XContentString interface, with the Text class implementing
that interface.
While backporting entitlement initialization refactorings, I realized there is a mismatch in getVersionSpecificCheckerClass signature, and also that this function in the backports is used in more places (DynamicInstrumentation), making it "strange" to have this in EntitlementInitialization. This PR extracts the function to a separate static class (package-private) and makes the signature uniform with backports.
This will need to be backported manually to the 8.x branches, and will make the backported version of DynamicInstrumentation cleaner.
Moves FilesEntitlements validation to a separate class. This is the final PR to make EntitlementsInitialization a simpler "orchestrator" of the various steps in the initialization phase.
Our path comparison for file access is string based, due to the fact that we need to support Paths created for different file systems/platforms.
However, Windows files and paths are (sort of) case insensitive.
This PR fixes the problem by abstracting String comparison operations and making them case sensitive or not based on the host OS.
Patrick Doyle
Ignacio Vera
Chris Hegarty
Ryan Ernst
Gal Lalouche
Niels Bauman
Ankit Sethi
Benjamin Trent
Simon Cooper
Jordan Powers
Rene Groeschke
Mike Pellegrini
Lorenzo Dematté
Craig Taverner
John Wagster