From c465cf6b4befc3c29aa8c56804d8eebfb3fb74cf Mon Sep 17 00:00:00 2001 From: Lianet Magrans <98415067+lianetm@users.noreply.github.com> Date: Thu, 13 Feb 2025 11:51:56 -0500 Subject: [PATCH] KAFKA-17298: Update upgrade notes for 4.0 KIP-848 (#18756) Reviewers: David Jacot --- .../clients/consumer/ConsumerConfig.java | 6 +- docs/ops.html | 75 +++++++++++++++++++ docs/upgrade.html | 17 ++++- 3 files changed, 94 insertions(+), 4 deletions(-) diff --git a/clients/src/main/java/org/apache/kafka/clients/consumer/ConsumerConfig.java b/clients/src/main/java/org/apache/kafka/clients/consumer/ConsumerConfig.java index b0b39274075..d9cee3ec179 100644 --- a/clients/src/main/java/org/apache/kafka/clients/consumer/ConsumerConfig.java +++ b/clients/src/main/java/org/apache/kafka/clients/consumer/ConsumerConfig.java @@ -121,9 +121,9 @@ public class ConsumerConfig extends AbstractConfig { */ public static final String GROUP_REMOTE_ASSIGNOR_CONFIG = "group.remote.assignor"; public static final String DEFAULT_GROUP_REMOTE_ASSIGNOR = null; - public static final String GROUP_REMOTE_ASSIGNOR_DOC = "The server-side assignor to use. If no assignor is specified, " + - "the group coordinator will pick one. This configuration is applied only if group.protocol is " + - "set to \"consumer\"."; + public static final String GROUP_REMOTE_ASSIGNOR_DOC = "The name of the server-side assignor to use. " + + "If not specified, the group coordinator will pick the first assignor defined in the broker config group.consumer.assignors." + + "This configuration is applied only if group.protocol is set to \"consumer\"."; /** * bootstrap.servers diff --git a/docs/ops.html b/docs/ops.html index 4827b4e0075..22060e67c75 100644 --- a/docs/ops.html +++ b/docs/ops.html @@ -4056,6 +4056,81 @@ $ bin/kafka-topics.sh --create --topic tieredTopic --bootstrap-server localhost:

For more information, please check Kafka Tiered Storage GA Release Notes.

+

6.10 Consumer Rebalance Protocol

+ +

Overview

+ +

Starting from Apache Kafka 4.0, the Next Generation of the Consumer Rebalance Protocol (KIP-848) + is Generally Available (GA). It improves the scalability of consumer groups while simplifying consumers. It also decreases rebalance times, thanks to + its fully incremental design, which no longer relies on a global synchronization barrier.

+ +

Consumer Groups using the new protocol are now referred to as Consumer groups, while groups using the old protocol are referred to as Classic groups. Note + that Classic groups can still be used to form consumer groups using the old protocol.

+ +

Server

+ +

The new consumer protocol is automatically enabled on the server since Apache Kafka 4.0. Enabling and disabling the protocol is controlled by the + group.version feature flag.

+ +

The consumer heartbeat interval and the session timeout are controlled by the server now with the following configs:

+ + +

The assignment strategy is also controlled by the server. The group.consumer.assignors configuration can be used to specify the list of available + assignors for Consumer groups. By default, the uniform assignor and the range assignor are configured. The first assignor + in the list is used by default unless the Consumer selects a different one. It is also possible to implement custom assignment strategies on the server side + by implementing the org.apache.kafka.coordinator.group.api.assignor.ConsumerGroupPartitionAssignor interface and specifying the full class name in the configuration.

+ +

Consumer

+ +

Since Apache Kafka 4.0, the Consumer supports the new consumer rebalance protocol. However, the protocol is not enabled by default. The group.protocol + configuration must be set to consumer to enable it. When enabled, the new consumer protocol is used alongside an improved threading model.

+ +

The group.remote.assignor configuration is introduced as an optional configuration to overwrite the default assignment strategy configured + on the server side.

+ +

The subscribe(SubscriptionPattern) and subscribe(SubscriptionPattern, ConsumerRebalanceListener) methods have been added to + subscribe to a regular expression with the new consumer rebalance protocol. With these methods, the regular expression uses the RE2J format and is now evaluated + on the server side.

+ +

New metrics have been added to the Consumer when using the new rebalance protocol, mainly providing visibility over the improved threading model. + See New Consumer Metrics.

+ +

When the new rebalance protocol is enabled, the following configurations and APIs are no longer usable:

+ + +

Upgrade & Downgrade

+ +
Offline
+ +

Consumer groups are automatically converted from Classic to Consumer and vice versa when they are empty. Hence, it is possible to change the protocol + used by the group by shutting down all the consumers and bringing them back up with the group.protocol=consumer configuration. The downside is that it requires taking + the consumer group down.

+ +
Online
+ +

Consumer groups can be upgraded without downtime by rolling out the consumer with the group.protocol=consumer configuration. When the first consumer using + the new consumer rebalance protocol joins the group, the group is converted from Classic to Consumer, and the classic rebalance protocol is + interoperated to work with the new consumer rebalance protocol. This is only possible when the classic group uses an assignor that does not embed custom metadata.

+ +

Consumer groups can be downgraded using the opposite process. In this case, the group is converted from Consumer to Classic when the last + consumer using the new consumer rebalance protocol leaves the group.

+ +

Limitations

+ +

While the new consumer rebalance protocol works for most use cases, it is still important to be aware of the following limitations:

+ +
diff --git a/docs/upgrade.html b/docs/upgrade.html index 3c42da28214..8e4b3f02cee 100644 --- a/docs/upgrade.html +++ b/docs/upgrade.html @@ -45,6 +45,21 @@ latest Kafka releases. +
  • + Apache Kafka 4.0 ships with a brand-new group coordinator implementation (See here. + Functionally speaking, it implements all the same APIs. However, the group coordinator runs in its own dedicated threads. + The number of threads can be defined with the group.coordinator.threads configuration. It defaults to 1 and + must certainly be adjusted for production deployments. +
    • +
  • + The Next Generation of the Consumer Rebalance Protocol (KIP-848) + is now Generally Available (GA) in Apache Kafka 4.0. The protocol is automatically enabled on the server when the upgrade to 4.0 is finalized. + Note that once the new protocol is used by consumer groups, care must be taken to downgrade to a previous version. Downgrading + to version 3.4.1 or above is safe. Downgrading to an earlier version requires to ensure that all the records in the + __consumer_offsets are removed because those versions cannot handle them. This can be achieved by deleting all the + consumer groups using the new protocol, disabling the new protocol and letting compaction remove the tombstones. + Check here for details. +
  • A number of deprecated classes, methods, configurations and tools have been removed.