This commit updates CI to test against Java 24 instead of Java 23 which
is EOL.
Due to Spotbugs not having released version 4.9.4 yet, we can't run
Spotbugs on Java 24. Instead, we are choosing to run Spotbugs, and the
rest of the compile and validate build step, on Java 17 for now.
Once 4.9.4 has released, we will switch to using Java 24 for this.
Exclude spotbugs from the run-tests gradle action. Spotbugs is already
being run once in the build by "compile and validate", there is no
reason to run it again as part of executing tests.
Reviewers: Chia-Ping Tsai <chia7712@gmail.com>
There are few README not added because I am not sure if they need to be
mentioned in root README.
```
./test-common/test-common-internal-api/src/main/java/org/apache/kafka/common/test/api/README.md
./storage/src/test/java/org/apache/kafka/tiered/storage/README.md
./.github/workflows/README.md
./raft/README.md
./committer-tools/README.md
```
Reviewers: Ken Huang <s7133700@gmail.com>, TengYao Chi
<kitingiao@gmail.com>, PoAn Yang <payang@apache.org>, Jhen-Yung Hsu
<jhenyunghsu@gmail.com>, Chia-Ping Tsai <chia7712@gmail.com>
The Docker image version should use `latest` instead of `3.7.0`,
allowing users to always pull the most up-to-date image from Docker Hub.
Reviewers: Sanskar Jhajharia <sjhajharia@confluent.io>, TengYao Chi
<kitingiao@gmail.com>
Split the JUnit tests into "new", "flaky", and the remainder.
On PR builds, "new" tests are anything that do not exist on trunk. They are run with zero tolerance for flakiness.
On trunk builds, "new" tests are anything added in the last 7 days. They are run with some tolerance for flakiness.
Another change included here is that we will not update the test catalog if any test job fails on a trunk build. We have had difficulty determining if all the tests had or not (due to timeout or failures in upstream Gradle tasks). By requiring green ":test" jobs, we can be sure that the resulting catalog will be valid.
---
The purpose of this change is to discourage contributors from adding flaky tests, but give some leeway for trunk so we have successful builds.
The "quarantinedTest" Gradle target has been consolidated into the regular "test" target. There are now some
runtime properties to control what tests are run.
* kafka.test.catalog.file: path to test catalog
* kafka.test.run.new: include new tests. this selection depends on the age of the loaded test catalog
* kafka.test.run.flaky: include tests marked as `@Flaky` (replaces the `excludeTags 'flaky'` directive)
* kafka.test.verbose: include additional logging from new JUnit classes (enabled by default if re-running GitHub workflow with debug logging enabled)
* maxTestRetries: how many retries to allow via Develocity retry plugin (default 0)
* maxTestRetryFailures: how many failures to allow before stopping retries (default 0)
Thanks to Jun Rao for inspiring the idea.
Reviewers: Chia-Ping Tsai <chia7712@gmail.com>, Ismael Juma <ismael@juma.me.uk>, Jun Rao <junrao@gmail.com>
1. Fix ambiguous statement where it could be interpreted that scalac is used for the clients module
2. Remove "in KRaft mode" because that's the only mode
3. Put IntelliJ first in IDE section and only mention the native gradle integration
4. Restore instructions for installing all projects to the local Maven repo, this was accidentally removed as part of the Scala 2.12 support removal.
This pull request replaces Log4j with Log4j2 across the entire project, including dependencies, configurations, and code. The notable changes are listed below:
1. Introduce Log4j2 Instead of Log4j
2. Change Configuration File Format from Properties to YAML
3. Adds warnings to notify users if they are still using Log4j properties, encouraging them to transition to Log4j2 configurations
Co-authored-by: Lee Dongjin <dongjin@apache.org>
Reviewers: Luke Chen <showuon@gmail.com>, Mickael Maison <mickael.maison@gmail.com>, Chia-Ping Tsai <chia7712@gmail.com>
Introduce new quarantinedTest that excludes tests tagged with "flaky". Also introduce two new build parameters "maxQuarantineTestRetries" and "maxQuarantineTestRetryFailures".
Reviewers: Chia-Ping Tsai <chia7712@gmail.com>
We have a detail usage guide for running Kafka in docker. However, it might be easy for people to miss it if they only scan through the README.
This PR add a basic example command for quick start and link directing users to the detailed usage guide
Reviewers: Luke Chen <showuon@gmail.com>
Implement some of the metrics from KIP-938: Add more metrics for
measuring KRaft performance.
Add these metrics to QuorumControllerMetrics:
kafka.controller:type=KafkaController,name=TimedOutBrokerHeartbeatCount
kafka.controller:type=KafkaController,name=EventQueueOperationsStartedCount
kafka.controller:type=KafkaController,name=EventQueueOperationsTimedOutCount
kafka.controller:type=KafkaController,name=NewActiveControllersCount
Create LoaderMetrics with these new metrics:
kafka.server:type=MetadataLoader,name=CurrentMetadataVersion
kafka.server:type=MetadataLoader,name=HandleLoadSnapshotCount
Create SnapshotEmitterMetrics with these new metrics:
kafka.server:type=SnapshotEmitter,name=LatestSnapshotGeneratedBytes
kafka.server:type=SnapshotEmitter,name=LatestSnapshotGeneratedAgeMs
Reviewers: Ron Dagostino <rndgstn@gmail.com>
The caching store layers were passing down writes into lower store layers upon eviction, but not setting the context to the evicted records' context. Instead, the context was from whatever unrelated record was being processed at the time.
Reviewers: Matthias J. Sax <mjsax@apache.org>
The current README instruction for local publishing boils the ocean by building and installing every jar in the project with both 2.12 and 2.13. While that is some times what people want to do, they are also often trying to just build a specific jar.
Reviewers: Bill Bejeck <bbejeck@apache.org>
* MINOR: Provide valid examples in README page.
- `testMetadataUpdateWaitTime` method is removed from MetadataTest class.
- Removed the travis CI documentation.
Reviewers: Luke Chen <showuon@gmail.com>
Introduce `maxScalacThreads` and set the default to the lowest of `8`
and the number of processors available to the JVM. The number `8` was
picked empirically, the sweet spot is between 6 and 10.
On my desktop, `./gradlew clean core:compileScala core:compileTestScala`
improved from around 60s to 51s ( with this change.
While at it, we improve the build output to include more useful
information at the start: build id, max parallel forks, max scala
threads and max test retries.
Reviewers: Manikumar Reddy <manikumar.reddy@gmail.com>, Sean Li
Java 17 is at release candidate stage and it will be a LTS release once
it's out (previous LTS release was Java 11).
Details:
* Replace Java 16 with Java 17 in Jenkins and Readme.
* Replace `--illegal-access=permit` (which was removed from Java 17)
with `--add-opens` for the packages we require internal access to.
Filed KAFKA-13275 for updating the tests not to require `--add-opens`
(where possible).
* Update `release.py` to use JDK8. and JDK 17 (instead of JDK 8 and JDK 15).
* Removed all but one Streams test from `testsToExclude`. The
Connect test exclusion list remains the same.
* Add notable change to upgrade.html
* Upgrade to Gradle 7.2 as it's required for proper Java 17 support.
* Upgrade mockito to 3.12.4 for better Java 17 support.
* Adjusted `KafkaRaftClientTest` and `QuorumStateTest` not to require
private access to `jdk.internal.util.random`.
Reviewers: Manikumar Reddy <manikumar.reddy@gmail.com>, Chia-Ping Tsai <chia7712@gmail.com>
Update the readme to note the deprecation. We will also mention the deprecation
in the downloads page when the release is done.
Reviewers: Konstantine Karantasis <konstantine@confluent.io>, David Jacot <djacot@confluent.io>, José Armando García Sancio <jsancio@users.noreply.github.com>
Gradle 7.1 improves Java incremental compilation:
https://docs.gradle.org/7.1.1/release-notes.html
We previously kept the JDK 15 build because some
tests didn't work with JDK 16. Since then, a number
of PRs were submitted to fix this so it's best
to remove the JDK 15 build before we create the
3.0 release branch.
Finally bump `test-retry` gradle plugin version too.
Reviewers: Chia-Ping Tsai <chia7712@gmail.com>, Luke Chen <showuon@gmail.com>
Evanston Zhou
Stig Døssing
S.Y. Wang
Ken Huang
mingdaoy
David Arthur
Shahbaz Aamir
TengYao Chi
Swikar Patel
Ismael Juma
Yung
Dejan Stojadinović
kevin-wu24
Ted Yan
TaiJuWu
Chung, Ming-Yen
Omnia Ibrahim
gongxuanzhang
Cheng-Kai, Zhang
Colin Patrick McCabe
Said Boudjelda
Chia-Ping Tsai
Lucas Brutschy
José Armando García Sancio
John Roesler
Kamal Chandraprakash
Dmitriy Fishman
Manikumar Reddy