Since the new features are not used, you may right. However, I believe the benefits are three-fold.
- There exists ZSTD side bug-fixes at ZSTD 1.4.9.
- There exists ZSTD JNI side memory optimization improvements at ZSTD JNI 1.4.8-5 ~ 1.4.8-7. (This includes some incompatible changes and recovery. So, 1.4.9 is more human-readable stable version number.).
- I hope this will reduce the chance of future potential version conflict issues across Apache projects. It's important when some downstream project starts to use new feature.
- Apache Spark 3.2.0 will use ZSTD 1.4.9. (ba7e525a11)
- Apache Parquet 1.12.0 will use ZSTD 1.4.9 (66ac28ce23)
- Apache Avro 1.10.3 will use ZSTD 1.4.9 (806667cb2b)
Reviewers: Lee Dongjin <dongjin@apache.org>, Ismael Juma <ismael@juma.me.uk>, Chia-Ping Tsai <chia7712@gmail.com>
socket.connection.setup.timeout.max.ms should be 30 seconds.
The current value of 127 seconds is longer than the default API timeout
for AdminClient, and longer than the default request timeouts for the
producer and consumer. We should bring these configs into line with
each other.
Reviewers: Jason Gustafson <jason@confluent.io>
This patch adds logic to delete old snapshots. There are three cases we handle:
1. Remove old snapshots after a follower completes fetching a snapshot and truncates the log to the latest snapshot
2. Remove old snapshots after a new snapshot is created.
3. Remove old snapshots during recovery after the node is restarted.
Reviewers: Cao Manh Dat<caomanhdat317@gmail.com>, José Armando García Sancio <jsancio@users.noreply.github.com>, Jason Gustafson <jason@confluent.io>
For KIP-698, we need a way to validate internal topics before we create them. This PR adds a validation method to the InternalTopicManager for that purpose.
Reviewers: Rohan Desai <rohan@confluent.io>, Guozhang Wang <wangguoz@gmail.com>
The method StreamsBuilder#addGlobalStore was simplified via KIP-233 in 1.1.0 release. This PR removes the old and deprecated overload.
Reviewers: A. Sophie Blee-Goldman <sophie@confluent.io>, Guozhang Wang <guozhang@confluent.io>
The KIP-500 early access release will not support creating a partition with a manual
partition assignment that includes a broker that is not currently online. This patch disables
system tests for Raft-based metadata quorums where the test depends on this functionality
to pass.
Reviewers: Colin P. McCabe <cmccabe@apache.org>
- Use consistent options for `javadoc` and `aggregatedJavadoc`
- `aggregatedJavadoc` depends on `compileJava`
- `connect-api` inherits `options.links`
- `streams` and `streams-test-utils` javadoc exclusions should be more
specific to avoid unexpected behavior in `aggregatedJavadoc` when the
javadoc for multiple modules is generated together
Reviewers: Chia-Ping Tsai <chia7712@gmail.com>
The KIP-500 server requires users to run `kafka-storage.sh` to format log directories before the server will start. If the directory is not formatted, the error message complains about a missing `meta.properties` file. It is useful for the message to refer users to `kafka-storage.sh` directly since formatting is a new requirement.
This patch also reduces the log level of a very spammy log message in `BrokerLifecycleManager`.
Reviewers: Colin P. McCabe <cmccabe@apache.org>
This patch implements additional handling logic for `RemoveTopic` records:
- Update `MetadataPartitions` to ensure addition of deleted partitions to `localRemoved` set
- Ensure topic configs are removed from `ConfigRepository`
- Propagate deleted partitions to `GroupCoordinator` so that corresponding offset commits can be removed
This patch also changes the controller topic id generation logic to use `Uuid.randomUuid` rather than `Random`.
Reviewers: Ismael Juma <ismael@juma.me.uk>, Chia-Ping Tsai <chia7712@gmail.com>
Removed broker number checks for invalid replication factor when doing the forwarding, in order to reduce false alarms for clients.
Reviewers: Jason Gustafson <jason@confluent.io>
This patch ensures that the constant max batch size defined in `KafkaRaftClient` is propagated to the constructed log configuration in `KafkaMetadataLog`. We also ensure that the fetch max size is set consistently with appropriate testing.
Reviewers: Chia-Ping Tsai <chia7712@gmail.com>, David Arthur <mumrah@gmail.com>
One gauge is missing in ReplicaManager#removeMetrics
Reviewers: Chia-Ping Tsai <chia7712@gmail.com>, Anna Sophie Blee-Goldman < ableegoldman@apache.org>
This patch enables delete topic support for the new KIP-500 controller. Also fixes the following:
- Fix a bug where feature level records were not correctly replayed.
- Fix a bug in TimelineHashMap#remove where the wrong type was being returned.
Reviewers: Jason Gustafson <jason@confluent.io>, Justine Olshan <jolshan@confluent.io>, Ron Dagostino <rdagostino@confluent.io>, Chia-Ping Tsai <chia7712@gmail.com>, Jun Rao <junrao@gmail.com>
Co-authored-by: Jason Gustafson <jason@confluent.io>
Gradle 7.0 is required for Java 16 compatibility and it removes a number of
deprecated APIs. Fix most issues preventing the upgrade to Gradle 7.0.
The remaining ones are more complicated and should be handled
in a separate PR. Details of the changes:
* Release tarball no longer includes includes test, sources, javadoc and test sources jars (these
are still published to the Maven Central repository).
* Replace `compile` with `api` or `implementation` - note that `implementation`
dependencies appear with `runtime` scope in the pom file so this is a (positive)
change in behavior
* Add missing dependencies that were uncovered by the usage of `implementation`
* Replace `testCompile` with `testImplementation`
* Replace `runtime` with `runtimeOnly` and `testRuntime` with `testRuntimeOnly`
* Replace `configurations.runtime` with `configurations.runtimeClasspath`
* Replace `configurations.testRuntime` with `configurations.testRuntimeClasspath` (except for
the usage in the `streams` project as that causes a cyclic dependency error)
* Use `java-library` plugin instead of `java`
* Use `maven-publish` plugin instead of deprecated `maven` plugin - this changes the
commands used to publish and to install locally, but task aliases for `install` and
`uploadArchives` were added for backwards compatibility
* Removed `-x signArchives` line from the readme since it was wrong (it was a
no-op before and it fails now, however)
* Replaces `artifacts` block with an approach that works with the `maven-publish` plugin
* Don't publish `jmh-benchmark` module - the shadow jar is pretty large and not
particularly useful (before this PR, we would publish the non shadow jars)
* Replace `version` with `archiveVersion`, `baseName` with `archiveBaseName` and
`classifier` with `archiveClassifier`
* Update Gradle and plugins to the latest stable version (7.0 is not stable yet)
* Use `plugin` DSL to configure plugins
* Updated notable changes for 3.0
Reviewers: Chia-Ping Tsai <chia7712@gmail.com>, Randall Hauch <rhauch@gmail.com>
KAFKA-12393: Document multi-tenancy considerations
Addressed review feedback by @dajac and @rajinisivaram
Ported from apache/kafka-site#334
Reviewers: Bill Bejeck <bbejeck@apache.org>
1. rename INVALID_HIGHWATERMARK to INVALID_HIGH_WATERMARK
2. replace FetchResponse.AbortedTransaction by FetchResponseData.AbortedTransaction
3. remove redundant constructors from FetchResponse.PartitionData
4. rename recordSet to records
5. add helpers "recordsOrFail" and "recordsSize" to FetchResponse to process record casting
Reviewers: Ismael Juma <ismael@juma.me.uk>
This patch refactors the code, which constructs the `LeaderAndIsrResponse` in `ReplicaManager#becomeLeaderOrFollower`, to improve the readability and to remove unnecessary operations.
Reviewers: Chia-Ping Tsai <chia7712@gmail.com>
Cast SMT transformation for bytes -> string.
Without this fix, the conversion becomes ByteBuffer.toString(), which always gives this useless result:
"java.nio.HeapByteBuffer[pos=0 lim=4 cap=4]"
With this change, the byte array is converted into a base64 string of the byte buffer content.
Reviewers: Mickael Maison <mickael.maison@gmail.com>, Randall Hauch <rhauch@gmail.com>, Konstantine Karantasis <k.karantasis@gmail.com>
KIP-405 introduces tiered storage feature in Kafka. With this feature, Kafka cluster is configured with two tiers of storage - local and remote. The local tier is the same as the current Kafka that uses the local disks on the Kafka brokers to store the log segments. The new remote tier uses systems, such as HDFS or S3 or other cloud storages to store the completed log segments. Consumers fetch the records stored in remote storage through the brokers with the existing protocol.
We introduced a few SPIs for plugging in log/index store and remote log metadata store.
This involves two parts
1. Storing the actual data in remote storage like HDFS, S3, or other cloud storages.
2. Storing the metadata about where the remote segments are stored. The default implementation uses an internal Kafka topic.
You can read KIP for more details at https://cwiki.apache.org/confluence/display/KAFKA/KIP-405%3A+Kafka+Tiered+Storage
Reviewers: Jun Rao <junrao@gmail.com>
This security vulnerability was found in netty-codec-http, but [caused by netty itself](c735357bf2) and [fixed in 4.1.59.Final](https://github.com/netty/netty/security/advisories/GHSA-5mcr-gq6c-3hq2). So, upgrade the netty version from 4.1.51.Final to 4.1.59.Final.
Author: Lee Dongjin <dongjin@apache.org>
Reviewers: Manikumar Reddy <manikumar.reddy@gmail.com>
Closes#10235 from dongjinleekr/feature/KAFKA-12389
Always grab a new thread.id and verify that a thread has fully shut down to DEAD before removing from the `threads` list and making that id available again
Reviewers: Walker Carlson <wcarlson@confluent.io>, Bruno Cadonna <cadonna@confluent.io>
Log start offset retention is the cheapest retention to evaluate and does not require access to maxTimestamp fields for segments, nor segment sizes. In addition, it may unblock other types of retention such as time based retention. Without this change retention is not idempotent. It's possible for one deleteOldSegments call to delete segments due to log start offset retention, and a follow up call to delete due to time based retention, even if the time has not changed.
Reviewers: Jun Rao <junrao@gmail.com>
To implement the explicit user initialization of Kafka Streams as
described in KIP-698, we first need to extract the code for the
setup of the changelog topics from the Streams partition assignor
so that it can also be called outside of a rebalance.
Reviewers: Anna Sophie Blee-Goldman <ableegoldman@apache.org>, Guozhang Wang <guozhang@confluent.io>
We now accept topicIds in the `DeleteTopic` request. If the client principal does not have `Describe` permission, then we return `TOPIC_AUTHORIZATION_FAILED`. This is justified because the topicId is not considered sensitive. However, in this case, we should not return the name of the topic in the response since we do consider it sensitive.
Reviewers: David Jacot <djacot@confluent.io>, dengziming <dengziming1993@gmail.com>, Justine Olshan <jolshan@confluent.io>, Chia-Ping Tsai <chia7712@gmail.com>
Bruno Cadonna
Luke Chen
bmaidics
Dongjoon Hyun
Colin Patrick McCabe
dengziming
Jorge Esteban Quilcate Otoya
Matthias J. Sax
Rohit Deshpande
Lee Dongjin
Ron Dagostino
Ismael Juma
Tom Bentley
Jason Gustafson
Chia-Ping Tsai
Boyang Chen
Walker Carlson
Manikumar Reddy
A. Sophie Blee-Goldman
José Armando García Sancio
Michael G. Noll
David Jacot
Sven Erik Knop
David Arthur
Satish Duggana
Lucas Bradstreet