Commit Graph

1416 Commits

Author SHA1 Message Date
Patrick Doyle 89f701f4c4
Bootstrap entitlements for testing (#129268)
* Fix ExceptionSerializationTests to use getCodeSource instead of getResource.

Using getResource makes this sensitive to unrelated classpath entries,
such as the entitlement bridge library, that get prepended to the classpath.

* FIx logging tests to use org.elasticsearch.index instead of root logger.

Using the root logger makes this sensitive to unrelated
logging, such as from the entitlement library.

* Fix entitlement error message by stashing the module name in ModuleEntitlements.

Taking the actual module name from the class doesn't work in tests,
where those classes are loaded from the classpath and so their module
info is misleading.

* Ignore server locations whose representative class isn't loaded

* Partial initial implementation

* System properties: testOnlyClasspath and enableForTests

* Trivially allow some packages

* DEBUG: use TreeMap in TestScopeResolver for readability

* Special case bouncycastle for security plugin

* Add CONFIG to TestPathLookup

* Add the classpath to the source path list for every plugin

* Add @WithoutEntitlements to tests that run ES nodes

* Set es.entitlement.enableForTests for all libs

* Use @WithoutEntitlements on ingest plugin tests

* Substitute ALL-UNNAMED for module name in non-modular plugins

* Add missing entitlements found by unit tests

* Comment in TestScopeResolver

* Properly compute bridge jar location for patch-module

* Call out nonServerLibs

* Don't build two TestPathLookups

* More comments for meta-tests

* Remove redundant dependencies for bridgeJarConfig.

These are alread set in ElasticsearchJavaBasePlugin.

* Add bridge+agent dependencies only if those exist.

For serverless, those project dependencies don't exist, and we'll need
to add the dependencies differently, using Maven coordinates.

* [CI] Auto commit changes from spotless

* Pass testOnlyPath in environment instead of command line.

It's typically a very very long string, which made Windows angry.

* [CI] Auto commit changes from spotless

* Split testOnlyPathString at File.pathSeparator

* Use doFirst to delay setting testOnlyPath env var

* Trivially allow jimfs (??)

* Don't enforce entitlements on internalClusterTest for now

* Replace forbidden APIs

* Match testOnlyClasspath using URI instead of String.

We already get the "needle" in the form of a URI, so this skips
a step, and has the benefit of also working on Windows.

* [CI] Auto commit changes from spotless

* More forbidden APIs

* Disable configuration cache for LegacyYamlRestTestPluginFuncTest

* Strip carriage-return characters in expected output for ReleaseNotesGeneratorTest.

The template generator also strips these, so we need to do so to make this pass
on Windows.

Note that we use replace("\r", "") where the template generator uses
replace("\\r", ""). The latter didn't work for me when I tried it on Windows,
for reasons I'm not aware of.

* Move configureEntitlements to ElasticsearchTestBasePlugin as-is

* Use matching instead of if

* Remove requireNonNull

* Remove default configuration

* Set inputs instead of dependencies

* Use test.systemProperty

* Respond to PR comments

* Disable entitlement enforcement for ScopedSettingsTests.

This test works by altering the logging on the root logger.
With entitlements enabled, that will cause additional log statements to appear,
which interferes with the test.

* Address PR comments

* Moritz's configureJavaBaseModuleOptions

* Allow for entitlements not yet enforced in serverless

* fix entitlementBridge config after rename

* drop empty file collections

* Remove workaround in LegacyYamlRestTestPluginFuncTest

---------

Co-authored-by: elasticsearchmachine <infra-root+elasticsearchmachine@elastic.co>
Co-authored-by: Lorenzo Dematté <lorenzo.dematte@elastic.co>
Co-authored-by: Moritz Mack <mmack@apache.org>
2025-07-02 13:44:38 -04:00
Ryan Ernst 40841edfdf
Handle newlines in release notes test (#130134)
Newlines are system dependent. The release notes generator uses groovy's
template engine, which produces system dependent newlines. This commit
adjusts the test to account for newlines on both windows and nix
systems.
2025-06-30 08:37:02 -07:00
Jordan Powers 40a7d02269
Pull match_only_text fixes into main (#130049)
This brings in the fixes from #130020, with minor fixes to address review
nits from that PR.

Co-authored-by: Martijn van Groningen <martijn.v.groningen@gmail.com>
2025-06-27 04:31:33 +10:00
elasticsearchmachine f70ff89456 Bump to version 9.2.0 2025-06-26 15:32:23 +00:00
Benjamin Trent 4d8f9fa505
Add OptimizedScalarQuantizer to forbidden API checks (#129873)
we have various ES only optimizations within OptimizedScalarQuantizer,
we should ensure callers within ES are using the optimized code.
2025-06-25 00:20:36 +10:00
Rene Groeschke 648c5ad15f
[Build] Make DependenciesInfoPlugin configuration cache compatible (#129791)
We can now run projects check tasks and :distribution:generateDependenciesReport with 
configuration cache enabled (--configuration-cache)
2025-06-23 15:45:54 +02:00
Chris Hegarty 1255a64832
Upgrade to Lucene 10.2.2 (#129546)
This commit upgrades to Upgrade to Lucene 10.2.2.

With the release of 10.2.2, we no longer need to workaround the Lucene bug mentioned in 128671.
2025-06-22 13:37:22 +01:00
Luke Whiting 996c99bd04
Add new option to allow full delegation of IntelliJ tests to Gradle (#126772)
* Add new option to allow full delegation of tests to gradle in IntelliJ plus docs

* Update CONTRIBUTING.md

Co-authored-by: Rene Groeschke <rene@breskeby.com>

* Update CONTRIBUTING.md

Fix typo in settings file name in docs

---------

Co-authored-by: Rene Groeschke <rene@breskeby.com>
2025-06-18 11:20:36 +01:00
Yang Wang c8233588a1
Add project-id to SnapshotDeletionsInProgress and RepositoryCleanupInProgress (#129462)
This PR adds project-id to both SnapshotDeletionsInProgress and
RepositoryCleanupInProgress so that they become project aware. Note that
making service code to configure and use the project-id accordingly will
be worked on separately.

Resolves: ES-11380 Relates: #125470
2025-06-17 14:38:54 +10:00
Moritz Mack 9e5cac34a4
Expand bcUpgradeTask to run more test suites. (#128983)
Relates to ES-11904

#128984 contains the changes to the PR buildkite pipeline to test this change while the buildkite changes are not merged yet.
2025-06-13 12:58:49 +02:00
Rene Groeschke b70f108ff7
[Test] Ignore PublishPluginFuncTest on windows for now (#129348)
Related to https://github.com/elastic/elasticsearch/issues/129100
2025-06-13 09:22:58 +02:00
Richard Dennehy 63da93d4c3
Delegated authorization using Microsoft Graph (SDK) (#128396)
* Delegated authorization using Microsoft Graph (SDK)
---------

Co-authored-by: elasticsearchmachine <infra-root+elasticsearchmachine@elastic.co>
Co-authored-by: Johannes Freden Jansson <johannes.freden@elastic.co>
Co-authored-by: Johannes Fredén <109296772+jfreden@users.noreply.github.com>
2025-06-12 11:03:32 +02:00
Rene Groeschke 916cd052bd Revert "[Gradle] Spotless plugin update (#115750)"
This reverts commit 6370d600b0.
2025-06-11 22:27:53 +02:00
Rene Groeschke 6370d600b0
[Gradle] Spotless plugin update (#115750)
- provides better configuration cache support
- requires some rework due to changed defaults
2025-06-12 04:22:13 +10:00
Rene Groeschke 71de00c3ce
Update Gradle wrapper to 8.14.2 (#129179) 2025-06-11 16:40:05 +02:00
Tanguy Leroux fd57c6e132
Add "Searchable Snapshots" to changelog validation schema (#129180)
We created a new ":Distributed Indexing/Searchable Snapshots" label recently on Github, so I think it makes sense to also have a "Searchable Snapshots" label in the changelog. It also makes sense since there is automatic changelog generation based on the pull request label.
2025-06-11 14:21:48 +02:00
Richard Dennehy 94e951341f
improve support for bytecode patching signed jars (#128613)
* improve support for bytecode patching signed jars

* Update docs/changelog/128613.yaml
---------

Co-authored-by: elasticsearchmachine <infra-root+elasticsearchmachine@elastic.co>
Co-authored-by: Johannes Freden Jansson <johannes.freden@elastic.co>
2025-06-11 11:15:21 +02:00
Rene Groeschke 1e13409049
Fix DRA dependenciesInfo task dependency resolution (#129209) 2025-06-10 20:06:24 +02:00
Rene Groeschke 342083100b
[Build] Add support for publishing to maven central (#128659)
This ensures we package an aggregation zip with all artifacts we want to publish to maven central as part of a release.
Running zipAggregation will produce a zip file in the build/nmcp/zip folder. The content of this zip is meant to match the maven artifacts we have currently declared as dra maven artifacts.
2025-06-06 17:35:44 +02:00
Brian Seeders 4762111b44
[release-notes] Fix tests and make it easy to regenerate output after changes (#128781) 2025-06-02 16:35:18 -04:00
Benjamin Trent c28e01ecf4
Adding forbidden API check for buggy range query behavior (#128671) (#128765)
This adds the buggy functions to forbidden API check. Will remove once
Lucene is updated.

Related #128671
2025-06-03 02:26:40 +10:00
Brian Seeders 40b486a6b9
[release-notes] Move 'coming' applies_to to the correct spot 2025-06-02 10:36:10 -04:00
Rene Groeschke 657f5ffe53
Update Gradle wrapper to 8.14.1 (#128438)
Update Gradle wrapper to 8.14.1
2025-05-30 12:23:06 +02:00
Brian Seeders c2ad34b97f
[release-notes] Update automation to use new markdown format (#124161) 2025-05-28 14:53:02 -04:00
Chris Hegarty 2134a1e6e1
Remove pesky restricted method warnings in tests. (#128212)
Since our unit and IT tests run as unnamed modules we should add `--enable-native-access=ALL-UNNAMED` to suppress the pesky warning that arise from calls to `System::load`.
2025-05-27 09:23:09 +01:00
Brian Seeders a13a46a2da
[tests] StandaloneRestIntegTestTask should set a minimum maxParallelForks of 1 (#126346) 2025-05-23 10:42:34 -04:00
Rene Groeschke 38c90ca8d4
Restructure docker files for docker distributions (#127960)
Restructures docker files for docker distributions

- Put Dockerfiles in specific distro specific folders keeping "Dockerfile" naming convention
- Allows better ide support
- Allows easier renovate integration
- Explicitly set base image in dockerfile
- simplify renovate configuration
- Cleanup DockerBase file to not contain ess fips base image information

This lives now in the Dockerfile content directly

* Workaround docker test issue

* Fix labels for fips image
2025-05-19 19:47:34 +02:00
Rene Groeschke 9b1815c84a
[Build] Update Develocity Plugin to 4.0.1 (#128053)
After updating Develocity to 2015.1.3 we can also update the according plugin
2025-05-13 21:37:44 -07:00
Ryan Ernst 60ad8ba744
Remove custom SecurityManager (#127778)
Since SecurityManager is no longer used, the custom subclass of
SecurityManager, SecureSM, is no longer needed.
2025-05-06 16:16:46 -07:00
Rene Groeschke a2e580fb60
Update Gradle wrapper to 8.14 (#126519)
* Fix PatternSetFactory incompatibility
* Update ospackage plugin
* Remove ambigious method definitions
* Cleanup verification metadata
* Some cleanup on unused methods and attributes
2025-05-06 13:00:15 +02:00
Nik Everett 15c461d842
ESQL: Disable format checks on generated imports (#127648)
This builds the infrastructure to disable spotless and some checkstyle
rules on generated imports. This works around the most frustrating part
of ESQL's string template generated files - the imports. It allows
unused and out of order imports. This can let us remove a lot of
cumbersome, tricky, and fairly useless `$if$` blocks from the templates.
2025-05-05 18:02:56 +02:00
Rene Groeschke d546fc8217
Adjust docker fips entrypoint and cmd (#127630)
Also extract docker fips configuration in explicit docker file
2025-05-02 14:43:55 +02:00
Chris Hegarty 898dd47c93
Upgrade to Lucene 10.2.1 (#127343)
This commit upgrades to Lucene 10.2.1 (from a previous 10.2.1-SNAPSHOT).

Given that we're on a snapshot, there are a few things to note:

 * No index version update is necessary, we already have IndexVersions.UPGRADE_TO_LUCENE_10_2_1, and there are no format changes in the final non-snapshot release.
2025-05-01 14:24:42 +01:00
Rene Groeschke aa309515f8
Rename docker fips image to cloud-ess-fips (#127561) 2025-04-30 15:56:24 +02:00
David Turner b028c0af56
Upgrade `repository-s3` to AWS SDK v2 (#126843)
Closes #120993
2025-04-24 21:21:03 +10:00
Julio 76f6006a42
Update Elasticsearch main with snapshot version of Lucene (#127125) 2025-04-22 00:25:08 +02:00
Brian Seeders 3d78c9428c
[bwc] Add bugfix3 project (#126880) 2025-04-16 16:27:03 -04:00
David Turner aa40147142
Add integ tests for `ftp://` URL repository (#126757)
We document support for snapshot repositories using `ftp://` URLs but it
seems this functionality has not worked for many years because of
security-manager restrictions, although nobody noticed because it was
not covered by any tests. The migration to the Entitlements framework
means that this functionality now works again, so this commit adds tests
to make sure we do not break it again in future.
2025-04-15 12:57:00 +01:00
Lorenzo Dematté 2697a3a872
Patcher for AWS SDKv2 locale-dependent formatting (#126326)
AWS SDK v2 has a bug (aws/aws-sdk-java-v2#5968) where PathResolver uses locale-dependent formatting.

This PR adds a patcher to the discovery-ec2 build process to replace calls to String.format(<format>, <args>) with String.format(Locale.ROOT, <format>, <args>).

Relates to ES-11279
2025-04-15 12:49:56 +02:00
Ignacio Vera ffdfcec334
Upgrade to Lucene 10.2.0 (#126594)
This commit upgrade Elasticsearch to lucene 10.2.0
2025-04-14 13:50:52 +02:00
Lorenzo Dematté e4af657c12
Patcher improvements (HDFS) (#126449)
Patchers transform specific classes in some "broken" dependencies to ensure they behave correctly (fixing a bug, disabling some undesired or dangerous behaviour, updating calls to deprecated or removed method overloads).

If we upgrade one of the dependencies we patch, we have a concerns that the patchers may not work against the classes in the new version.
This PR addresses this concern by introducing a check on the SHA256 digest of the class, to ensure we are operating on the same bytes the patcher was designed for; if the digest changes that means the class has been changed (e.g. for a dependency update). If that happens, we break the build process with a specific error, so we can double check that the patchers still work against the new classes.

Extracted from #126326

Relates to ES-11279
2025-04-11 17:20:45 +02:00
Rene Groeschke ca195736ca
Address review feedback on es default docker image (#126330)
This addresses feedback we got for our default image at https://github.com/docker-library/official-images/pull/18692
This also introduces separate docker source files to make maintaining those easier.

We cannot take over all suggested changes as we require certain settings to have our packaging tests pass as expected.
2025-04-10 17:14:22 +02:00
Ryan Ernst 3bac50e818
Use logs dir as working directory (#124966)
In the unexpected case that Elasticsearch dies due to a segfault or
other similar native issue, a core dump is useful in diagnosing the
problem. Yet core dumps are written to the working directory, which is
read-only for most installations of Elasticsearch. This commit changes
the working directory to the logs dir which should always be writeable.
2025-04-09 07:07:11 -07:00
Rene Groeschke 6e4cb8142b
Add initial esql test benchmark tests (#126224)
* Add initial esql test benchmark tests
* Fix build-benchmark pipeline
* Update gredle profiler
2025-04-08 13:28:34 +02:00
Rene Groeschke 6cfb2e9d4a
Resolve fips specific metricbeat and filebeat libs for ES fips image (#126312)
Also updated DRA handling to reflect resolving fips specific artifacts
2025-04-05 06:47:30 +11:00
Rene Groeschke d39db02a74
Use official ubi9-minimal base image for es docker (#126280)
Based on feedback we got here: Based on feedback we got
https://github.com/docker-library/official-images/pull/18692\#issuecomment-2749045229
we can not rely on non official images here
2025-04-04 22:41:16 +11:00
elastic-renovate-prod[bot] 7b46621c19
Pin docker.elastic.co/wolfi/chainguard-base-fips Docker tag to ebfc3f1 (#126065)
This PR contains the following updates:

| Package | Update | Change | |---|---|---| |
docker.elastic.co/wolfi/chainguard-base-fips | pinDigest |  -> `ebfc3f1`
|

---

### Configuration

📅 **Schedule**: Branch creation - "after 1pm on tuesday" (UTC),
Automerge - At any time (no schedule defined).

🚦 **Automerge**: Disabled by config. Please merge this manually once
you are satisfied.

♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the
rebase/retry checkbox.

🔕 **Ignore**: Close this PR and you won't be reminded about this update
again.

---

 - [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check this box

---

This PR has been generated by [Renovate
Bot](https://redirect.github.com/renovatebot/renovate).
<!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiIzOS4xMDcuMCIsInVwZGF0ZWRJblZlciI6IjM5LjEwNy4wIiwidGFyZ2V0QnJhbmNoIjoibWFpbiIsImxhYmVscyI6WyI6RGVsaXZlcnkvUGFja2FnaW5nIiwiPm5vbi1pc3N1ZSIsIlRlYW06RGVsaXZlcnkiLCJhdXRvLW1lcmdlLXdpdGhvdXQtYXBwcm92YWwiXX0=-->
2025-04-02 06:22:31 +11:00
elastic-renovate-prod[bot] de5377df45
Update docker.elastic.co/wolfi/chainguard-base:latest Docker digest to 29150cd (#126066)
This PR contains the following updates:

| Package | Update | Change | |---|---|---| |
docker.elastic.co/wolfi/chainguard-base | digest | `c4e10ec` ->
`29150cd` |

---

### Configuration

📅 **Schedule**: Branch creation - "after 1pm on tuesday" (UTC),
Automerge - At any time (no schedule defined).

🚦 **Automerge**: Disabled by config. Please merge this manually once
you are satisfied.

♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the
rebase/retry checkbox.

🔕 **Ignore**: Close this PR and you won't be reminded about this update
again.

---

 - [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check this box

---

This PR has been generated by [Renovate
Bot](https://redirect.github.com/renovatebot/renovate).
<!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiIzOS4xMDcuMCIsInVwZGF0ZWRJblZlciI6IjM5LjEwNy4wIiwidGFyZ2V0QnJhbmNoIjoibWFpbiIsImxhYmVscyI6WyI6RGVsaXZlcnkvUGFja2FnaW5nIiwiPm5vbi1pc3N1ZSIsIlRlYW06RGVsaXZlcnkiLCJhdXRvLW1lcmdlLXdpdGhvdXQtYXBwcm92YWwiXX0=-->
2025-04-02 06:19:48 +11:00
Joe Gallo 5c9cdeeb87
Bump junit and hamcrest (#126055) 2025-04-01 14:05:58 -04:00
Mark Vieira 66ba3c2a53
Convert CCR module tests to new testing framework (#125894) 2025-03-31 14:16:19 -07:00