Removing the custom dependency checksum functionality in favor of Gradle build-in dependency verification support.
- Use sha256 in favor of sha1 as sha1 is not considered safe these days.
Closes https://github.com/elastic/elasticsearch/issues/69736
When we feature freeze Elasticsearch, we need to create stub
documentation for the next version. This turns out to be as simple as
running the usual `generateReleaseNotes` task without any inputs.
We use `between(x, y)` calls with `switch` statements in tests to
randomize test behaviour. However, some usages define `case` statements
that can never execute, because the `case` value is outside the range
defined by the `between` call.
Write a rule that inspects the switches and the cases, and fails on the
broken cases. This rule checks `between`, `randomIntBetween` and
`randomInt`.
We only rely on the checkstyle version in the buildLibs.toml gradle version catalogue with this change.
Also added some hints for gradle best practices.
This is an aftermath of #88283
Adds a new ci Jenkins job configuration for running snyk dependency monitoring on a daily basis. We setup a service account in snyk and resolve the api token for publishing in vault.
Related to #87620
This changes the LoggedExec task to be configuration cache compatible. We changed the implementation
to use `ExecOperations` instead of extending `Exec` task. As double checked with the Gradle team this task
is not planned to be made configuration cache compatible out of the box anytime soon.
This is part of the effort on https://github.com/elastic/elasticsearch/issues/57918
This introduces the ability to simply configure a java tool chain for elasticsearch java projects to be used.
If an environment variable `JAVA_TOOLCHAIN_HOME `is declared, this JDK will be used as toolChain in elasticsearch.java projects. If JAVA_RUNTIME_HOME is configured, it takes precedence over JAVA_TOOLCHAIN_HOME
for configuring test cluster runtimes.
This should make testing our build with java `ea` versions easier and allows detangling the used compiler jdk from the gradle java runtime.
We will use target reference to distinguish between different versions or branches of our elasticsearch project
to be able to trace vulnerable dependencies down to the version.
snyk lifecycle property allows filtering the project overview by `production` or `development`. When version
is ending with SNAPSHOT we configure the lifecycle as development. Otherwise its production.
Related to #87620
This changes AbstractGradleFuncTest to run with configuration cache enabled by default. That gives us better test coverage on how our build logic works with configuration cache enabled and makes it explicit which parts are not yet working.
We fixed some minor configuration cache related issues in our code.
* Generate release notes for v8.3.0 (#87294)
* Generate release notes for v8.3.0
* [DOCS] Add 8.3 migration file to index
* Fixed version number
* Fix formatting of deprecation in 85326
* Use asciidoc format for deprecations
* Extract static content from migration/index
* This was just an enhancement
* Nope, it was an upgrade
* Added migration/index.asciidoc generation support (#87318)
Including extracting static content from migration/index, so the template would be as light as possible.
The reason for this work is because the gradle task `generateReleaseNotes` was not correctly adding new links and imports to the migrations/index and that caused documentation to fail building for 8.3.0.
* [DOCS] Add ml-cpp PRs to release notes
* Added back incorrectly deleted changlog
* Added missing highlight
* Fixed spelling of StackOverflowError
Co-authored-by: lcawl <lcawley@elastic.co>
* Brought back missing changelog for 87235 (#87370)
* Brought back missing changelog for 87235
* Regenerate release notes
* Regenerate release notes for BC3 (#87449)
* Regenerate release notes for BC3
* Re-applied manual fixes that Lisa Cawley used
* Fixed breaking changes generation
To match the manual edits done by Lisa Cawley
* Fixed failing test
Since the manual edits by Lisa removed the `-SNAPSHOT` from the docs
we remove it from the tests too.
* Remove changelogs for intermediate lucene upgrades
* Update release notes for BC4 (#87635)
* Update release notes for BC4
* Adding missing changelog for geo_grid
* Added missing highlight notes for 84250
* Update docs/reference/release-notes/highlights.asciidoc
Co-authored-by: David Kilfoyle <41695641+kilfoyle@users.noreply.github.com>
* Update docs/reference/release-notes/highlights.asciidoc
Co-authored-by: David Kilfoyle <41695641+kilfoyle@users.noreply.github.com>
* Backported fixes to original yaml
* Control sort order of release highlights
So that small changes to an individual highlight don't completely shuffle the entire document.
We also added links to the PRs from the highlight titles, for convenience.
Otherwise readers need to search the release notes for the changelog entry and click there,
which is a lot more work.
* Fixed failing test after new ordered highlights
Also made test verify re-ordering of highlights
Co-authored-by: David Kilfoyle <41695641+kilfoyle@users.noreply.github.com>
* Update release notes for BC5 (#87808)
* Update release notes for BC6 (#87912)
* Update release notes for BC9 (#88011)
* Regenerate release notes after version bump 8.4.0
And after forward porting 8.3.0 release notes changes.
* Regenerate after some changelog entries removed
* Re-prune the changelogs
* Removed three changelog entries
Co-authored-by: lcawl <lcawley@elastic.co>
Co-authored-by: David Kilfoyle <41695641+kilfoyle@users.noreply.github.com>
Test methods tend to have descriptive names for the cases that they are
testing. Sometimes they require javadoc, for sure, but not often enough
to add a little "WEAK WARNING" marker on every single test method in my
IDE.
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
This adds the generation and upload logic of Gradle dependency graphs to snyk
We directly implemented a rest api based snyk plugin as:
the existing snyk gradle plugin delegates to the snyk command line tool the command line tool
uses custom gradle logic by injecting a init file that is
a) using deprecated build logic which we definitely want to avoid
b) uses gradle api we avoid like eager task creation.
Shipping this as a internal gradle plugin gives us the most flexibility as we only want to monitor
production code for now we apply this plugin as part of the elasticsearch.build plugin,
that usage has been for now the de-facto indicator if a project is considered a "production" project
that ends up in our distribution or public maven repositories. This isnt yet ideal and we will revisit
the distinction between production and non production code / projects in a separate effort.
As part of this effort we added the elasticsearch.build plugin to more projects that actually end up
in the distribution. To unblock us on this we for now disabled a few check tasks that started failing by applying elasticsearch.build.
Addresses #87620
- Make our build tool tests more readable
- One step closer to remove outdated test fixtures like GradleUnitTestCase and friends
Didn't port some basic setter getter tests like PluginExtensionTests as it does not add much value and we have integration test coverage for this stuff too
The ingest attachment processor is currently available as a plugin. This
commit moves the processor to the default distribution so it is always
available.
This is intended to help us getting closer to #57918 by implicitly
testing our build logic configuration-cache support. Plugin and Task
tests can be marked as configuration cache compatible now and we will
always run then with configuration cache enabled.
By default, gradle will fail the build if configuration cache problems
have been detected during build execution. That should be in general
better then adding explicit tests for testing configuration cache
compatibility per Test class
The `:Data Management/Ingest` label was recently renamed to `:Data
Management/Ingest Node`. This commit changes the build tools changelog
schema to know about the rename.
This PR uses Lucene-9.3 snapshot in Elasticsearch 8.4. Noticeable changes in this Lucene snapshot:
- Merge-on-refresh (disabled)
- No more pathological merging
- SortedSetDocValues#count for value_count aggs
This PR reworks the testing conventions precommit plugin. This plugin now:
- is compatible with yaml, java rest tests and internalClusterTest (aka different sourceSets per test type)
- enforces test base class and simple naming conventions (as it did before)
- adds one check task per test sourceSet
- uses the worker api to improve task execution parallelism and encapsulation
- is gradle configuration cache compatible
This also ports the TestingConventions integration testing to Spock and removes the build-tools-internal/test kit folder that is not required anymore. We also add some common logic for testing java related gradle plugins.
We will apply further cleanup on other tests within our test suite in a dedicated follow up cleanup
RHEL 9 suffers the same problems as Ubuntu 22.04 that were fixed by
https://github.com/elastic/ml-cpp/pull/2272.
The glibc version on RHEL 9 is 2.34 (versus 2.35 on Ubuntu 22.04),
so we need to adjust the skip logic.
ParameterizedMessage will not be part of the new ES logging API and therefore should not be used.
java.util.Supplier and String.format should be used instead.
this commit adds ParameterizedMessage to forbidden api
relates #86549
This fixes references to project that makes the plugin incompatible with Gradle
configuration cache. We also remove custom xpackProject utility:
using xpackProject in certain situations can break configure configuration cache compatibility as it uses a mutual project object under the hood that is discouraged to use in some use cases (e.g. at execution time)
It always breaks compatibility with --configure-on-demand
using xpackProject uses the project of the :x-pack project. referencing other project objects from other subproject should avoided where possible to decouple (sub project configurations). There's a good explanation of why we want to decouple our project configurations as much as possible here: https://docs.gradle.org/current/userguide/multi_project_configuration_and_execution.html#sec:decoupled_projects
it adds little value over default out of the box gradle api (just use project(':x-pack:someProject') instead of xpackProject('someProject') Also in some occasions its even shorter. e.g. when this is used as xpackProject('someProject').path instead of just passing :x-pack:someProject
I'll try to put a bit more context in the PR description in the future to make the motivation behind these kind of changes more clear upfront
Related to #57918
Introducing a stable logging API under libs/logging.
This change covers the most common use cases for logging: fetching a logger with LogManager, emitting a log messages with Logger and Level.
It is influenced by log4j2-api, but do not include Marker and LogBuilder methods.
Also methods using org.apache.logging.log4j.util.Supplier are replaced with java.util.Supplier
The basic implementation is present in server and injected statically in LogConfigurator
relates #84478
Yoann Rodière
Mark Vieira
Rene Groeschke
Ignacio Vera
Rory Hunter
Chris Hegarty
Nhat Nguyen
Craig Taverner
Nik Everett
Ryan Ernst
Tim Vernum
Lee Hinman
Amey Kulkarni
Armin Braun
Mary Gouseti
David Roberts
Przemyslaw Gomulka