In [KAFKA-19359](https://issues.apache.org/jira/browse/KAFKA-19359), the
commons-beanutils transitive dependency was force bumped in the project
to avoid related CVEs. The commons-validator already has a new release,
which solves this problem:
https://github.com/apache/commons-validator/tags
The workaround could be deleted as part of the version bump.
Reviewers: Chia-Ping Tsai <chia7712@gmail.com>
**This upgrade includes:**
- Dependency configurations are now realized only when necessary, which
helps improve configuration performance and memory usage.
- The configuration cache improves build time by caching the result of
the configuration phase and reusing it for subsequent builds. This
feature can significantly improve build performance.
reference: [Gradle 8.14.3 Release
Notes](https://docs.gradle.org/8.14.3/release-notes.html#build-authoring-improvements)
Reviewers: Chia-Ping Tsai <chia7712@gmail.com>
Dependencies such as api-ldap-client-api and mina-core used by the
current version of apacheds have several critical CVEs: CVE-2018-1337,
CVE-2024-52046 and CVE-2019-0231.
Reviewers: Luke Chen <showuon@gmail.com>, Chia-Ping Tsai <chia7712@gmail.com>
Use Java 24 for the spotbugs checks, now that Spotbugs works on Java
24.
Added some more warning exclusions for warnings that are new to 4.9.4.
Reviewers: Chia-Ping Tsai <chia7712@gmail.com>
This patch updates the code and the dependency with the latest namespace
and version.
Signed-off-by: Federico Valeri <fedevaleri@gmail.com>
Reviewers: Mickael Maison <mickael.maison@gmail.com>
Upgraded RocksDB from 9.7.3 to 10.1.3, deprecate two configuration in
`RocksDBGenericOptionsToDbOptionsColumnFamilyOptionsAdapter.java`
- random_access_max_buffer_size (removed since v9.11.1
541761eaaa)
- rate_limiter (deprecated since v7.6.0
25cc564ff7)
Add one configuration:
- daily_offpeak_time_utc (introduced since v9.11.1
9b1d0c02e9)
Reviewers: Bruno Cadonna <cadonna@apache.org>
Adds a test dependency on
[mock-oauth2-server](https://github.com/navikt/mock-oauth2-server/) for
integration tests for OAuth layer. Also includes fixes for some
regressions that were caught by the integration tests.
Reviewers: Manikumar Reddy <manikumar@confluent.io>, Lianet Magrans
<lmagrans@confluent.io>
Bump the commons-beanutils for CVE-2025-48734. Since `commons-validator`
hasn't had new release with newer `commons-beanutils` versions, we manually bump it in kafka.
Reviewers: Mickael Maison <mickael.maison@gmail.com>
Update opentelemetry-proto from 1.0.0-alpha to 1.3.2-alpha.
OpenTelemetry-Proto versions from v1.0.0 up to and including v1.3.2
introduce no breaking changes.
[release
note](https://github.com/open-telemetry/opentelemetry-proto/releases)
For example, starting with v1.4.0, protobuf-java was updated to version
4.28.3. To mitigate the risk of protobuf compatibility issues, upgrading
to v1.3.2 first allows the existing protobuf version to remain unchanged
for now.
Reviewers: poorv Mittal <apoorvmittal10@gmail.com>, TengYao Chi
<kitingiao@gmail.com>, Chia-Ping Tsai <chia7712@gmail.com>
jira: https://issues.apache.org/jira/browse/KAFKA-19382
Upgrade junit from 5.10.2 to
[5.13.1](https://github.com/junit-team/junit5/releases).
A new behavior was introduced to junit 5.12
(89a46dfa10),
disallowing `ClusterTestExtensions` to generate empty invocation
contexts. However, `ClusterTestExtensions` is invoked by junit extension
so it could result in empty contexts for some tests.
```
> Configure project :
Starting build with version 4.1.0-SNAPSHOT (commit id c4a769bc) using
Gradle 8.14.1, Java 17 and Scala 2.13.16
Build properties: ignoreFailures=false, maxParallelForks=10,
maxScalacThreads=8, maxTestRetries=0
> Task :core:test kafka.api.ConsumerBounceTest.initializationError
failed, log available in
/Users/lansg/Project/OpenSource/kafka/kafka-fork/kafka/core/build/reports/testOutput/kafka.api.ConsumerBounceTest.initializationError.test.stdout
Gradle Test Run :core:test > Gradle Test Executor 5 > ConsumerBounceTest
> testCloseDuringRebalance(String) > initializationError FAILED
org.junit.platform.commons.PreconditionViolationException: Provider
[ClusterTestExtensions] did not provide any invocation contexts, but was
expected to do so. You may override
mayReturnZeroTestTemplateInvocationContexts() to allow this. at
java.base@17.0.13/java.util.ArrayList.forEach(ArrayList.java:1511) at
java.base@17.0.13/java.util.ArrayList.forEach(ArrayList.java:1511)
kafka.api.ConsumerBounceTest.initializationError failed, log available
in
/Users/lansg/Project/OpenSource/kafka/kafka-fork/kafka/core/build/reports/testOutput/kafka.api.ConsumerBounceTest.initializationError.test.stdout
```
Reviewers: Chia-Ping Tsai <chia7712@gmail.com>, TengYao Chi
<frankvicky@apache.org>, Ken Huang <s7133700@gmail.com>
These dependencies have been updated across both files:
caffeine: From 3.1.8 to 3.2.0 javassist: From 3.29.2-GA to
3.30.2-GA Jetty-related: All Jetty components have been updated
from 12.0.15 to 12.0.22, including: jetty-alpn-client
jetty-client jetty-ee10-servlet jetty-ee10-servlets
jetty-http jetty-io jetty-security
jetty-server jetty-session jetty-util jose4j:
From 0.9.4 to 0.9.6 Jersey-related: All Jersey components have been
updated from 3.1.9 to 3.1.10, including: jersey-client
jersey-common jersey-container-servlet
jersey-container-servlet-core jersey-hk2 jersey-server
classgraph: From 4.8.173 to 4.8.179 jline: From 3.25.1 to 3.30.4
pcollections: From 4.0.1 to 4.0.2 re2j: From 1.7 to 1.8
snappy-java: From 1.1.10.5 to 1.1.10.7
New Dependency (LICENSE-binary only)
A new dependency, jspecify-1.0.0, has been added to LICENSE-binary.
gradle/dependencies.gradle Specific Updates
These updates are only reflected in the gradle/dependencies.gradle file:
bcpkix: From 1.78.1 to 1.80 bndlib: From 7.0.0 to 7.1.0 jacoco:
From 0.8.10 to 0.8.13 hamcrest: From 2.2 to 3.0 jqwik: From
1.8.3 to 1.9.2
Reviewers: Ken Huang <s7133700@gmail.com>, Chia-Ping Tsai
<chia7712@gmail.com>
This patch upgrades to Gradle 8.14.1. Note that we are switching to
using the recommended "-bin" package.
Signed-off-by: Federico Valeri <fedevaleri@gmail.com>
Reviewers: Luke Chen <showuon@gmail.com>
Gradle 8.14 starts to support Java 24, so we should update the Gradle
version accordingly.
Reviewers: TengYao Chi <kitingiao@gmail.com>, PoAn Yang
<payang@apache.org>, Chia-Ping Tsai <chia7712@gmail.com>
* Add `com.dynatrace.hash4j:hash4j:0.22.0` to dependencies.
* Add `computeTopicHash` to `org.apache.kafka.coordinator.group.Utils`.
* If topic name is non-existent, return 0.
* If topic name is existent, use streaming XXH3 to compute topic hash
with magic byte, topic id, topic name, number of partitions, partition
id and sorted racks.
* Add `computeGroupHash` to `org.apache.kafka.coordinator.group.Utils`.
* If topic map is empty, return 0.
* If topic map is not empty, use streaming XXH3 to compute group
metadata hash with sorted topic hashes by topic names.
* Add related unit test.
Reviewers: Ismael Juma <ismael@juma.me.uk>, Chia-Ping Tsai <chia7712@gmail.com>, Sean Quah <squah@confluent.io>, David Jacot <djacot@confluent.io>
---------
Signed-off-by: PoAn Yang <payang@apache.org>
- update reflection-config.json and resource-config.json to include log4j2 and jackson
- remove unused jackson scala library
- fix the incorrect path of log4j2.yaml
- adopt workaround (--standalone) to make this PR work and it will be fixed by KAFKA-18737)
Reviewers: TengYao Chi <kitingiao@gmail.com>, Chia-Ping Tsai <chia7712@gmail.com>
- Fixed the RemoteIndexCacheTest that fails with caffeine > 3.1.1
Reviewers: Luke Chen <showuon@gmail.com>, Kamal Chandraprakash <kamal.chandraprakash@gmail.com>
This patch removes dropwizard metrics in the dependency list as it is not used any more. It was introduced in 4f5b4c868e because it was required by Zookeeper. Zookeeper is no longer there so we can remove it too.
Reviewers: Ismael Juma <ismael@juma.me.uk>
This PR upgrades RocksDB from 7.9.2 to 9.7.3 and addresses the following compatibility issues introduced by the RocksDB upgrade:
- Removal of AccessHint: The AccessHint class was completely removed in RocksDB 9.7.3. This required removing all import statements, variable declarations, method parameters, method return types, and static method calls related to AccessHint in RocksDBGenericOptionsToDbOptionsColumnFamilyOptionsAdapter.java RocksDBGenericOptionsToDbOptionsColumnFamilyOptionsAdapterTest.java Unused methods are removed in RocksDBGenericOptionsToDbOptionsColumnFamilyOptionsAdapter.java
- Removal of NO_FILE_CLOSES: The NO_FILE_CLOSES metric was also removed in RocksDB 9.7.3. The calculation for numberOfOpenFiles in RocksDBMetricsRecorder.java has been adjusted to now track the total number of file opens since the last reset. The previous calculation, which subtracted NO_FILE_CLOSES from NO_FILE_OPENS, is no longer possible. The reason RocksDB removed NO_FILE_CLOSES seems to be that it did not properly work: https://github.com/search?q=repo%3Afacebook%2Frocksdb+NO_FILE_CLOSES&type=issues
- Removal of methods related to compressed block cache configuration in BlockBasedTableConfig
- Change of the signature of org.rocksdb.Options.setLogger()
Reviewers: Anna Sophie Blee-Goldman <ableegoldman@apache.org>, Matthias J. Sax <matthias@confluent.io>, Bruno Cadonna <cadonna@apache.org>
Remove Apache ZooKeeper from the Apache Kafka build. Also remove commons IO, commons CLI, and netty, which were dependencies we took only because of ZooKeeper.
In order to keep the size of this PR manageable, I did not remove all classes which formerly interfaced with ZK. I just removed the ZK types. Fortunately, Kafka generally wrapped ZK data structures rather than using them directly.
Some classes were pretty entangled with ZK, so it was easier just to stub them out. For ZkNodeChangeNotificationListener.scala, PartitionStateMachine.scala, ReplicaStateMachine.scala, KafkaZkClient.scala, and ZookeeperClient.scala, I replaced all the functions with "throw new UnsupportedOperationException". Since the tests for these classes have been removed, as well as the ZK-based broker code, this should be OK as an incremental step.
Reviewers: Chia-Ping Tsai <chia7712@gmail.com>
This pull request replaces Log4j with Log4j2 across the entire project, including dependencies, configurations, and code. The notable changes are listed below:
1. Introduce Log4j2 Instead of Log4j
2. Change Configuration File Format from Properties to YAML
3. Adds warnings to notify users if they are still using Log4j properties, encouraging them to transition to Log4j2 configurations
Co-authored-by: Lee Dongjin <dongjin@apache.org>
Reviewers: Luke Chen <showuon@gmail.com>, Mickael Maison <mickael.maison@gmail.com>, Chia-Ping Tsai <chia7712@gmail.com>
This commit implements the changes for KIP-1032. This updates Kafka to Jakarta specs, JavaEE 10 and Jetty 12. The changes here primarily effect Kafka Connect and MM2.
Todo/Notes:
1) I bumped the connect modules to JDK 17 but I also had to bump a couple other things that had a dependency on conect. The tools project depends on connect so that had to be bumped, and streams depends on tools so that needed to be bumped. This means we may need to separate some things if we don't want to enforce JDK 17 on streams.
2) There is an issue with a test in DedicatedMirrorIntegrationTest that I had to change for now that involves escaping characters and not quite sure what to do about it yet. The cause is the Servlet 6 spec changing what is allowed in the path. See: Jetty 12: 400: Ambiguous URI path encoding for path <%=FOO%>~1 (encoded: %3C%25%3DFOO%25%3E%7E1) jetty/jetty.project#11890
3) I had to configure the idle timeout in Jetty requests to match our request timeout so tests didn't fail. This was needed to fix the ConnectWorkerIntegrationTest#testPollTimeoutExpiry() test
Testing is being done by just using the existing tests for Connect and MM2 which should be sufficient.
Reviewers: Greg Harris <greg.harris@aiven.io>, David Arthur <mumrah@gmail.com>, Chia-Ping Tsai <chia7712@gmail.com>
This commit adds AK 3.9 to the system tests on trunk.
Follow-up of #17797
Reviewers: Chia-Ping Tsai <chia7712@gmail.com>, Bruno Cadonna <cadonna@apache.org>
This patch is the first of a series of patches to introduce support for server side regular expression. It introduces the re2j dependency.
Co-authored-by: Lianet Magrans <lmagrans@confluent.io>
Reviewers: Lianet Magrans <lmagrans@confluent.io>
Dejan Stojadinović
Patrik Nagy
Mickael Maison
Chang-Chi Hsu
Michael Morris
Sanskar Jhajharia
Gantigmaa Selenge
Stig Døssing
Federico Valeri
Dmitry Werner
Kuan-Po Tseng
Kirk True
Luke Chen
Chih-Yuan Chien
Nick Guo
YuChia Ma
S.Y. Wang
Gaurav Narula
Ken Huang
PoAn Yang
David Jacot
Ismael Juma
TaiJuWu
Swikar Patel
Colin Patrick McCabe
TengYao Chi
Christopher L. Shannon
mingdaoy
Joao Pedro Fonseca Dantas
Chia-Chuan Yu
Josep Prat
David Jacot
kartik-3513