* Delegated authorization using Microsoft Graph (SDK)
---------
Co-authored-by: elasticsearchmachine <infra-root+elasticsearchmachine@elastic.co>
Co-authored-by: Johannes Freden Jansson <johannes.freden@elastic.co>
Co-authored-by: Johannes Fredén <109296772+jfreden@users.noreply.github.com>
This adds some testing tools for verifying vector recall and latency
directly without having to spin up an entire ES node and running a rally
track.
Its pretty barebones and takes inspiration from lucene-util, but I
wanted access to our own formats and tooling to make our lives easier.
Here is an example config file. This will build the initial index, run
queries at num_candidates: 50, then again at num_candidates 100 (without
reindexing, and re-using the cached nearest neighbors).
```
[{
"doc_vectors" : "path",
"query_vectors" : "path",
"num_docs" : 10000,
"num_queries" : 10,
"index_type" : "hnsw",
"num_candidates" : 50,
"k" : 10,
"hnsw_m" : 16,
"hnsw_ef_construction" : 200,
"index_threads" : 4,
"reindex" : true,
"force_merge" : false,
"vector_space" : "maximum_inner_product",
"dimensions" : 768
},
{
"doc_vectors" : "path",
"query_vectors" : "path",
"num_docs" : 10000,
"num_queries" : 10,
"index_type" : "hnsw",
"num_candidates" : 100,
"k" : 10,
"hnsw_m" : 16,
"hnsw_ef_construction" : 200,
"vector_space" : "maximum_inner_product",
"dimensions" : 768
}
]
```
To execute:
```
./gradlew :qa:vector:checkVec --args="/Path/to/knn_tester_config.json"
```
Calling `./gradlew :qa:vector:checkVecHelp` gives some guidance on how
to use it, additionally providing a way to run it via java directly
(useful to bypass gradlew guff).
- Adds docker image based on chainguard base fips image
- x86 only for now as the base image is x86 only
- the image does not provide any elasticsearch.yml configuration. for testing purposes you can follow the elasticsearch fips guide available at https://github.com/elastic/FIPSGuide/tree/main/elasticsearch
The image is shipped with:
- org.bouncycastle:bc-fips:1.0.2.5 and org.bouncycastle:bctls-fips:1.0.19 in Elasticsearch libs folder
- config/jvm.options.d/fips.options for fips specific JVM options
- fips_java.security file
- fips_java.policy
Out of scope:
- Add packaging test coverage (part of later PR as we want to provide that image for testing early and packaging tests require more general restructuring for support fips scenarios)
This commit adds compatibility tests that target ES revisions that align with specific Lucene versions. In this case, we are intending to upgrade from Lucene 10.0 to 10.1. Since no on-prem Elasticsearch release exists with 10.0, we need another method to ensure compatibility with Lucene 10.0 indicies.
The work here is a bit hacky since all our compatibility testing infrastructure is centered around versions and we're now effectively doing compatibility tests between two different revisions of Elasticsearch that both report the same version. Ideally this specific testing would be replaced by unit tests, rather that reusing our full cluster restart tests for this purpose.
We'll also want to bump the commit referenced in the CI pipelines here to align with the last commit using Lucene 10.0.
The STABLE_CONFIGURATION_CACHE feature flag enables the following:
- tasks using a shared build service without declaring the requirement via the Task.usesService method will emit a deprecation warning.
- when the configuration cache is not enabled but the feature flag is present, deprecations for the following configuration cache requirements are also enabled:
- Registering build listeners
- Using task extensions and conventions at execution time
Rephrase the authorization check in `S3HttpFixture` in terms of a
predicate provided by the caller so that there's no need for a separate
subclass that handles session tokens, and so that it can support
auto-generated credentials more naturally.
Also adapts `Ec2ImdsHttpFixture` to dynamically generate credentials
this way.
Also extracts the STS fixture in `S3HttpFixtureWithSTS` into a separate
service, similarly to #117324, and adapts this new fixture to
dynamically generate credentials too.
Relates ES-9984
The S3 and IMDS services are separate things in practice, we shouldn't
be conflating them as we do today. This commit introduces a new
independent test fixture just for the IMDS endpoint and migrates the
relevant tests to use it.
Relates ES-9984
The distribution tools are meant to be CLIs. This commit moves the
entitlements jar projects to the libs dir, under a single
libs/entitlement root directory to keep the related jars together.
The libs projects are configured to all begin with `elasticsearch-`.
While this is desireable for the artifacts to contain this consistent
prefix, it means the project names don't match up with their
directories. Additionally, it creates complexities for subproject naming
that must be manually adjusted.
This commit adjusts the project names for those under libs to be their
directory names. The resulting artifacts for these libs are kept the
same, all beginning with `elasticsearch-`.
* Replace cloud-ess docker image with wolfi-ess
We just replaced the existing implementation of cloud-ess with what was wolfi-ess which is a wolfi based ess image.
The cloud image itself will be removed in a future commit it was not used anywhere
* Switch to test cloud docker image instead of default docker in packaging pr tests.
This adds way more coverage than the default docker image which is also barely touched
* Entitlements for System.exit
* Respond to Simon's comments
* Rename trampoline -> bridge
* Require exactly one bridge jar
* Use Type helpers to generate descriptor strings
* Various cleanup from PR comments
* Remove null "receiver" for static methods
* Use List<Type> instead of voidDescriptor
* Clarifying comment
* Whoops, getMethod
* SuppressForbidden System.exit
* Spotless
* Use embedded provider plugin to keep ASM off classpath
* Oops... forgot the punchline
* Move ASM license to impl
* Use ProviderLocator and simplify bridgeJar logic
* Avoid eager resolution of configurations during task configuration
* Remove compile-time dependency agent->bridge
---------
Co-authored-by: Mark Vieira <portugee@gmail.com>
* Initial hello-world entitlements agent
* Respond to Ryan's comments
* License header
* Fix forbidden APIs setup
* Rename EntitlementAgent
* Automated refactor missed one
* Automated rename really let me down here
* Very serious test name
* README files for the new modules
* Use "tasks.named('jar')"
Co-authored-by: Rene Groeschke <rene@breskeby.com>
* Use 'tasks.named('test')'
Co-authored-by: Rene Groeschke <rene@breskeby.com>
* More deferral of gradle tasks
Co-authored-by: Rene Groeschke <rene@breskeby.com>
* Even more deferral
Co-authored-by: Rene Groeschke <rene@breskeby.com>
* FIx gradle syntax for javaagent arg
---------
Co-authored-by: Rene Groeschke <rene@breskeby.com>
* Update Gradle gradleEnterprise plugin (develocity now)
* Fix imports in build scan script
* Fix build scan api usage
* Dependency cleanup and API fix
* Fix API update for BuildResult in Build scans
* Fix buildkite buildscan annotations based on gradle failures
ImmutableCollections uses a seed, set early during JVM startup, which
affects the iteration order of collections. Although we do not want to
rely on the iteration order of Map and Set collections, bugs do
sometimes occur. In order to reproduce those bugs to fix them, it is
important the test seed for Elasticsearch matches the seed used in
ImmutableCollections.
Unfortunately ImmutableCollections is internal to the JDK, and the seed
used is private and final. This commit works around these limitations by
creating a patched version of ImmutableCollections which allows access
to the seed member. ESTestCase is then able to reflectively set the seed
at runtime based on the Elasticsearch seed.
Note that this only affects tests. ImmutableCollections remains is
unchanged for production code.
relates #94946
Fix incompatibility with 8.8 and our internal api usages
- Update ospackage to a version that contains a fix we provided
- Tweak build logic to avoid deprecation warnings
- Use newer permission api
- Use custom shadowplugin
- Rework ElasticsearchDistribution dependencies resolution
- Update Gradle wrapper to 8.8
This ports our krb5kdc test fixture to test container and reworks hdfs handling to also be based on test containers.
The yaml rest tests that are using hdfs required introducing variable substitution in yamlresttestparser handling.
This commit makes zstd compression available to Elasticsearch. The
library is pulled in through maven in jar files for each platform, then
bundled in a new platform directory under lib. Access to the zstd
compression/decompression is through NativeAccess.
This introduces a new getHistoricalFeatures() method on ESRestTestCase
which returns a map of historical feature version mappings loaded from
FeatureSpecification implementations from any plugins/modules in use
by the current test suite. The mappings are generated by a new Gradle
task at build time, and then injected into the test runtime as a
System property.
This commit upgrades to OpenSAML v4.3.0
Versions of OpenSAML ≥ 4.1 have a hard dependency on the non-FIPS release of BouncyCastle.
This would prevent ES from being able to run in a JVM where BC-FIPS is configured as the security provider.
Closes: #71983
Co-authored-by: Tim Vernum tim@adjective.org
Using gradle toolchain support in gradle requires refactoring how the composite build is composed.
We added three toolchain resolver
1. Resolver for resolving defined bundled version from oracle as openjdk
2. Resolve all available jdks from Adoption
3. Resolve archived Oracle jdk distributions.
We should be able to remove the JdkDownloadPlugin altogether without having that in place, but we'll do that in a separate effort.
Fixes#95094
We need to verify, for each release, that our stable plugin APIs
are not breaking.
This commit adds some Gradle support for basic backwards compatibility
testing. On the Gradle side, we add a new qa project to test the
current commit against downloads of released versions, and against
fresh builds of snapshot versions.
As for the actual comparison, we break up the output of javap (the
decompiler) by line and create maps of classes to public class,
field, and method declarations within those class files. We then
check that the signature map from the new jar is not missing any
elements present in the old jar. This method has known limitations,
which are documented in the JarApiComparisonTask class.
Co-authored-by: Mark Vieira <portugee@gmail.com>
This commit adds a new test framework for configuring and orchestrating
test clusters for both Java and YAML REST testing. This will eventually
replace the existing "test-clusters" Gradle plugin and the build-time
cluster orchestration.
This project has a problem with availability of Docker images after
release. Disabling individual tasks is tricky because it uses test
fixtures, so instead just skip the project entirely until we can work
out a way forward.
Richard Dennehy
Benjamin Trent
Rene Groeschke
Brian Seeders
Mark Vieira
David Turner
Mariusz Józala
Ryan Ernst
Patrick Doyle
Joe Gallo
Albert Zaharovits
William Brafford
Rory Hunter