CVE-2020-36518 vulnerability affects jackson-databind (see GHSA-57j2-w4cx-62h2).
Upgrading to jackson-databind version 2.12.6.1 addresses this CVE.
Reviewers: Luke Chen <showuon@gmail.com>, Bruno Cadonna <cadonna@apache.org>
RocksDB 6.27.3 does not run on arm64 M1 Macs which would prevent people on this platform to run Kafka Streams. Thus, this PR upgrades RocksDB to 6.29.4.1 which contains the following fix to allow to run RocksDB on arm64 M1 Macs:
facebook/rocksdb#7720
The source compatibility report between 6.27.3 and 6.29.4.1 (attached to the ticket) reports a couple of incompatibilities. However, the incompatibilities do not seem to affect Kafka Streams' backwards compatibility.
The changes to class RocksDB only apply when inheriting from RocksDB. RocksDB is not exposed to users in Streams.
The changes to class WriteBatch and class WriteBatchInterface also only apply with inheritance. Both classes are not exposed to users in Streams.
-The change to enum SanityLevel seem also not to apply to Streams since SanityLevel is only used in ConfigOptions which is only used to load options from files and properties objects. Loading options from files or properties is not exposed to users in Streams.
Reviewers: Bill Bejeck <bbejeck@apache.org>, Matthias J. Sax <mjsax@apache.org>, A. Sophie Blee-Goldman <ableegoldman@apache.org>
Updates Gradle to its newer version 7.3.3. This version includes the
following relevant features:
- Support for Java 17
- Support for Scala 3
For a further description of the release notes see:
https://docs.gradle.org/7.3.3/release-notes.html
I did the update as per the description in Gradle's release notes:
```
./gradlew wrapper --gradle-version=7.3.3
```
This means `gradlew` script is updated to the newest version.
Verified that `gradlewAll jar` and `gradlew releaseTarGz` still succeed.
Reviewers: Ismael Juma <ismael@juma.me.uk>
RocksDB v6.27.3 has been released and it is the first release to support s390x. RocksDB is currently the only dependency in gradle/dependencies.gradle without s390x support.
RocksDB v6.27.3 has added some new options that require an update to streams/src/main/java/org/apache/kafka/streams/state/internals/RocksDBGenericOptionsToDbOptionsColumnFamilyOptionsAdapter.java but no other changes are needed to upgrade.
I have run the unit/integration tests locally on s390x and also the :streams tests on x86_64 and they pass.
Reviewers: Luke Chen <showuon@gmail.com>, Bruno Cadonna <cadonna@apache.org>
grgit 4.1.0 caused unsupported version error during gradle builds.
The reason was that grgit 4.1.0 uses always the latest JGit version
internally. Unfortunately, the latest JGit version was compiled with
a Java version later than Java 8 which caused the unsupported version
error during gradle builds for Java 8.
grgit 4.1.1 fixed this issue by upper bounding the version of JGrit
to a version that is still compiled with Java 8. Consequently, we can
remove the hotfix we merged in commit d1e0d2b474
and instead bump the grgit version from 4.1.0 to 4.1.1.
Reviewer: John Roesler <vvcephei@apache.org>
Implements the major part of the IQv2 framework as proposed in KIP-796.
Reviewers: Patrick Stuedi <pstuedi@apache.org>, Vicky Papavasileiou <vpapavasileiou@confluent.io>, Bruno Cadonnna <cadonna@apache.org>
A new version of JGit that is used by grgit that is used by gradle
causes the following error:
org/eclipse/jgit/storage/file/FileRepositoryBuilder has been compiled
by a more recent version of the Java Runtime (class file version 55.0),
this version of the Java Runtime only recognizes class file versions
up to 52.0
The reason is that version 6.0.0.202111291000-r of JGrit was compiled
with a newer Java version than Java 8, probably Java 11.
Explicitly setting the version of JGrit in gradle to 5.12.0.202106070339-r fixes
the issue.
Reviewers: David Jacot <djacot@confluent.io>, Ismael Juma <ismael@juma.me.uk>, Alexander Stohr, David Arthur <mumrah@gmail.com>
This task is to provide a concrete implementation of the interfaces defined in KIP-255 to allow Kafka to connect to an OAuth/OIDC identity provider for authentication and token retrieval. While KIP-255 provides an unsecured JWT example for development, this will fill in the gap and provide a production-grade implementation.
The OAuth/OIDC work will allow out-of-the-box configuration by any Apache Kafka users to connect to an external identity provider service (e.g. Okta, Auth0, Azure, etc.). The code will implement the standard OAuth client credentials grant type.
The proposed change is largely composed of a pair of AuthenticateCallbackHandler implementations: one to login on the client and one to validate on the broker.
See the following for more detail:
KIP-768
KAFKA-13202
Reviewers: Yi Ding <dingyi.zj@gmail.com>, Ismael Juma <ismael@juma.me.uk>, Jun Rao <junrao@gmail.com>
Java 17 is at release candidate stage and it will be a LTS release once
it's out (previous LTS release was Java 11).
Details:
* Replace Java 16 with Java 17 in Jenkins and Readme.
* Replace `--illegal-access=permit` (which was removed from Java 17)
with `--add-opens` for the packages we require internal access to.
Filed KAFKA-13275 for updating the tests not to require `--add-opens`
(where possible).
* Update `release.py` to use JDK8. and JDK 17 (instead of JDK 8 and JDK 15).
* Removed all but one Streams test from `testsToExclude`. The
Connect test exclusion list remains the same.
* Add notable change to upgrade.html
* Upgrade to Gradle 7.2 as it's required for proper Java 17 support.
* Upgrade mockito to 3.12.4 for better Java 17 support.
* Adjusted `KafkaRaftClientTest` and `QuorumStateTest` not to require
private access to `jdk.internal.util.random`.
Reviewers: Manikumar Reddy <manikumar.reddy@gmail.com>, Chia-Ping Tsai <chia7712@gmail.com>
Updates the scala fmt to the latest stable version.
Applies all the style fixes (all source code changes are done by scala
fmt).
Removes setting about dangling parentheses as `true` is already the
default.
Reviewer: John Roesler <john@confluent.io>
Also adjusted the acceptable recovery lag to stabilize Streams tests.
Reviewers: Justine Olshan <jolshan@confluent.io>, Matthias J. Sax <mjsax@apache.org>, John Roesler <vvcephei@apache.org>
* Improve the test prior to reimplementing KafkaFutureImpl using CompletableFuture.
* KAFKA-9687: Reimplement KafkaFutureImpl using a CompleteableFuture
* KIP-707: Add KafkaFuture.toCompletionStage
Reviewers: Chia-Ping Tsai <chia7712@gmail.com>, David Jacot <djacot@confluent.io>, Konstantine Karantasis <k.karantasis@gmail.com>
Gradle 7.1 improves Java incremental compilation:
https://docs.gradle.org/7.1.1/release-notes.html
We previously kept the JDK 15 build because some
tests didn't work with JDK 16. Since then, a number
of PRs were submitted to fix this so it's best
to remove the JDK 15 build before we create the
3.0 release branch.
Finally bump `test-retry` gradle plugin version too.
Reviewers: Chia-Ping Tsai <chia7712@gmail.com>, Luke Chen <showuon@gmail.com>
Update the ZooKeeper version to v3.6.3. This requires adding dropwizard
as a new dependency.
Also, add Kafka v2.8.0 to the ducktape system test image.
Reviewers: Luke Chen <showuon@gmail.com>, Colin P. McCabe <cmccabe@apache.org>, Ismael Juma <ismael@juma.me.uk>
This PR aims to upgrade `zstd-jni` from `1.4.9-1` to `1.5.0-2`.
This change will incorporate a number of bug fixes and performance improvements made in `1.5.0` of `zstd`:
- https://github.com/facebook/zstd/releases/tag/v1.5.0
- https://github.com/luben/zstd-jni/releases/tag/v1.5.0-1
- https://github.com/luben/zstd-jni/releases/tag/v1.5.0-2
The most recent `1.5.0` release offers +25%-140% (compression) and +15% (decompression) performance
improvements under certain conditions. Those conditions are unlikely to apply to Kafka with the default
configuration, however.
Since this is a dependency change, this should pass all the existing CIs.
Reviewers: Lee Dongjin <dongjin@apache.org>, Ismael Juma <ismael@juma.me.uk>
It's quite verbose to include all configs for every partition loaded/created.
Also make sure to redact sensitive and unknown config values.
Unit test included.
Reviewers: David Jacot <djacot@confluent.io>, Kowshik Prakasam <kprakasam@confluent.io>, Luke Chen <showuon@gmail.com>
2.10.x is no longer supported, so we should move to 2.12 for the 3.0
release.
ScalaObjectMapper has been deprecated and it looks like we don't
actually need it, so remove its usage.
Reviewers: David Jacot <djacot@confluent.io>
CVE-2021-26291, which makes Man-In-The-Middle-Attack possible, was fixed in maven 3.8.1.
Reviewers: Luke Chen <showuon@gmail.com>, Manikumar Reddy <manikumar.reddy@gmail.com>
This includes TASTy Reader support for Scala 3.0.0. This makes it easier
for Kafka libraries to be used in Scala 3.0 projects
Release notes: https://github.com/scala/scala/releases/tag/v2.13.6
Reviewers: Ismael Juma <ismael@juma.me.uk>
This PR upgrades RocksDB to 6.19.3. After the upgrade the Gradle build exited with code 134 due to SIGABRT signals ("Pure virtual function called!") coming from the C++ part of RocksDB. This error was caused by RocksDB state stores not properly closed in Streams' code. This PR adds the missing closings and updates the RocksDB option adapter.
Reviewers: Anna Sophie Blee-Goldman <ableegoldman@apache.org>, Guozhang Wang <wangguoz@gmail.com>
The version of the Eclipse Jersey library brought as dependences,
2.31, has a known vulnerability, CVE-2021-28168 (https://github.com/advisories/GHSA-c43q-5hpj-4crv).
This replaces it with 2.34, which is fully compatible with
2.31, except for bugs and vulnerabilities.
Reviewers: Manikumar Reddy <manikumar.reddy@gmail.com>
The Gradle RAT plugin properly declares inputs and outputs and is also
cachable. This also relieves the Kafka developers from maintaining the build
integration with RAT.
The generated RAT report is identical to the one generated previously. The only
difference is the RAT report name: the RAT plugin sets the HTML report name to
`index.html` (still under `build/rat`).
Verified that the rat task fails if unlicensed files are present (and not excluded). Also
`./gradlew rat` succeeds when there is no .git folder.
1. replace org.junit.Assert by org.junit.jupiter.api.Assertions
2. replace org.junit by org.junit.jupiter.api
3. replace Before by BeforeEach
4. replace After by AfterEach
5. remove ExternalResource from all scala modules
6. add explicit AfterClass/BeforeClass to stop/start EmbeddedKafkaCluster
Noted that this PR does not migrate stream module to junit 5 so it does not introduce callback of junit 5 to deal with beforeAll/afterAll. The next PR of migrating stream module can replace explicit beforeAll/afterAll by junit 5 extension. Or we can keep the beforeAll/afterAll if it make code more readable.
Reviewers: John Roesler <vvcephei@apache.org>
Edwin
Bruno Cadonna
Mickael Maison
Ismael Juma
Luke Chen
Josep Prat
Jonathan Albrecht
David Jacot
Bruno Cadonna
John Roesler
Kirk True
Lee Dongjin
Justine Olshan
Kamal Chandraprakash
Ron Dagostino
Tom Bentley
CHUN-HAO TANG
David Christle
Dejan Stojadinović
Shay Elkin
A. Sophie Blee-Goldman
Jim Hurne
Chia-Ping Tsai