mirror of https://github.com/apache/kafka.git
425 lines
36 KiB
HTML
425 lines
36 KiB
HTML
<!--
|
|
Licensed to the Apache Software Foundation (ASF) under one or more
|
|
contributor license agreements. See the NOTICE file distributed with
|
|
this work for additional information regarding copyright ownership.
|
|
The ASF licenses this file to You under the Apache License, Version 2.0
|
|
(the "License"); you may not use this file except in compliance with
|
|
the License. You may obtain a copy of the License at
|
|
|
|
http://www.apache.org/licenses/LICENSE-2.0
|
|
|
|
Unless required by applicable law or agreed to in writing, software
|
|
distributed under the License is distributed on an "AS IS" BASIS,
|
|
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
|
See the License for the specific language governing permissions and
|
|
limitations under the License.
|
|
-->
|
|
|
|
<script><!--#include virtual="js/templateData.js" --></script>
|
|
|
|
<script id="upgrade-template" type="text/x-handlebars-template">
|
|
|
|
<h4><a id="upgrade_4_0_0" href="#upgrade_4_0_0">Upgrading to 4.0.0</a></h4>
|
|
|
|
<h5><a id="upgrade_clients_4_0_0" href="#upgrade_clients_4_0_0">Upgrading Clients to 4.0.0</a></h5>
|
|
|
|
<p><b>For a rolling upgrade:</b></p>
|
|
|
|
<ol>
|
|
<li>Upgrade the clients one at a time: shut down the client, update the code, and restart it.</li>
|
|
<li>Clients (including Streams and Connect) must be on version 2.1 or higher before upgrading to 4.0. Many deprecated APIs were removed in Kafka 4.0.
|
|
For more information about the compatibility, please refer to the <a href="/{{version}}/documentation/compatibility.html">compatibility matrix</a>
|
|
or <a href="https://cwiki.apache.org/confluence/x/y4kgF">KIP-1124</a>.</li>
|
|
</ol>
|
|
|
|
<h5><a id="upgrade_servers_4_0_0" href="#upgrade_servers_4_0_0">Upgrading Servers to 4.0.0 from any version 3.3.x through 3.9.x</a></h5>
|
|
|
|
<p>Note: Apache Kafka 4.0 only supports KRaft mode - ZooKeeper mode has been removed. As such, <b>broker upgrades to 4.0.0 (and higher) require KRaft mode and
|
|
the software and metadata versions must be at least 3.3.x</b> (the first version when KRaft mode was deemed production ready). For clusters in KRaft mode
|
|
with versions older than 3.3.x, we recommend upgrading to 3.9.x before upgrading to 4.0.x. Clusters in ZooKeeper mode
|
|
have to be <a href="/{{version}}/documentation.html#kraft_zk_migration">migrated to KRaft mode</a> before they can be upgraded to 4.0.x.
|
|
</p>
|
|
|
|
<p><b>For a rolling upgrade:</b></p>
|
|
|
|
<ol>
|
|
<li>Upgrade the brokers one at a time: shut down the broker, update the code, and restart it. Once you have done so, the
|
|
brokers will be running the latest version and you can verify that the cluster's behavior and performance meets expectations.
|
|
</li>
|
|
<li>Once the cluster's behavior and performance has been verified, finalize the upgrade by running
|
|
<code>
|
|
bin/kafka-features.sh --bootstrap-server localhost:9092 upgrade --release-version 4.0
|
|
</code>
|
|
</li>
|
|
<li>Note that cluster metadata downgrade is not supported in this version since it has metadata changes.
|
|
Every <a href="https://github.com/apache/kafka/blob/trunk/server-common/src/main/java/org/apache/kafka/server/common/MetadataVersion.java">MetadataVersion</a>
|
|
has a boolean parameter that indicates if there are metadata changes (i.e. <code>IBP_4_0_IV1(23, "4.0", "IV1", true)</code> means this version has metadata changes).
|
|
Given your current and target versions, a downgrade is only possible if there are no metadata changes in the versions between.</li>
|
|
</ol>
|
|
|
|
<h5><a id="upgrade_servers_400_notable" href="#upgrade_servers_400_notable">Notable changes in 4.0.0</a></h5>
|
|
<ul>
|
|
<li>
|
|
Old protocol API versions have been removed. Users should ensure brokers are version 2.1 or higher before upgrading Java clients
|
|
(including Connect and Kafka Streams which use the clients internally) to 4.0.
|
|
Similarly, users should ensure their Java clients (including Connect and Kafka Streams) version is 2.1 or higher before upgrading brokers to 4.0.
|
|
Finally, care also needs to be taken when it comes to kafka clients that are not part of Apache Kafka, please see
|
|
<a href="https://cwiki.apache.org/confluence/display/KAFKA/KIP-896%3A+Remove+old+client+protocol+API+versions+in+Kafka+4.0">KIP-896</a> for the details.
|
|
</li>
|
|
<li>
|
|
Apache Kafka 4.0 only supports KRaft mode - ZooKeeper mode has been removed. About version upgrade,
|
|
check <a href="/{{version}}/documentation.html#upgrade_4_0_0">Upgrading to 4.0.0 from any version 3.3.x through 3.9.x</a> for more info.
|
|
</li>
|
|
<li>
|
|
Apache Kafka 4.0 ships with a brand-new group coordinator implementation (See <a href="https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=217387038#KIP848:TheNextGenerationoftheConsumerRebalanceProtocol-GroupCoordinator">here</a>).
|
|
Functionally speaking, it implements all the same APIs. There are reasonable defaults, but the behavior of the new
|
|
group coordinator can be tuned by setting the configurations with prefix <code>group.coordinator</code>.
|
|
</li>
|
|
<li>
|
|
The Next Generation of the Consumer Rebalance Protocol (<a href="https://cwiki.apache.org/confluence/display/KAFKA/KIP-848%3A+The+Next+Generation+of+the+Consumer+Rebalance+Protocol">KIP-848</a>)
|
|
is now Generally Available (GA) in Apache Kafka 4.0. The protocol is automatically enabled on the server when the upgrade to 4.0 is finalized.
|
|
Note that once the new protocol is used by consumer groups, the cluster can only downgrade to version 3.4.1 or newer.
|
|
Check <a href="/{{version}}/documentation.html#consumer_rebalance_protocol">here</a> for details.
|
|
</li>
|
|
<li>
|
|
Transactions Server Side Defense (<a href="https://cwiki.apache.org/confluence/display/KAFKA/KIP-890%3A+Transactions+Server-Side+Defense">KIP-890</a>)
|
|
brings a strengthened transactional protocol to Apache Kafka 4.0. The new and improved transactional protocol is enabled when the upgrade to 4.0 is finalized.
|
|
When using 4.0 producer clients, the producer epoch is bumped on every transaction to ensure every transaction includes the intended messages and duplicates are not
|
|
written as part of the next transaction. Downgrading the protocol is safe. For more information check <a href="/{{version}}/documentation.html#transaction_protocol">here</a>
|
|
</li>
|
|
<li>
|
|
Eligible Leader Replicas (<a href="https://cwiki.apache.org/confluence/display/KAFKA/KIP-966%3A+Eligible+Leader+Replicas">KIP-966 Part 1</a>)
|
|
enhances the replication protocol for the Apache Kafka 4.0. Now the KRaft controller keeps track of the data partition replicas that are
|
|
not included in ISR but are safe to be elected as leader without data loss. Such replicas are stored in the partition metadata as
|
|
the <code>Eligible Leader Replicas</code>(ELR).
|
|
For more information check <a href="/{{version}}/documentation.html#eligible_leader_replicas">here</a>
|
|
</li>
|
|
<li>
|
|
Since Apache Kafka 4.0.0, we have added a system property ("org.apache.kafka.sasl.oauthbearer.allowed.urls") to
|
|
set the allowed URLs as SASL OAUTHBEARER token or jwks endpoints. By default, the value is an empty list.
|
|
Users should explicitly set the allowed list if necessary.
|
|
</li>
|
|
<li>A number of deprecated classes, methods, configurations and tools have been removed.
|
|
<ul>
|
|
<li><a id="upgrade_400_notable_common" href="#upgrade_400_notable_common"><b>Common</b></a>
|
|
<ul>
|
|
<li>The <code>metrics.jmx.blacklist</code> and <code>metrics.jmx.whitelist</code> configurations were removed from the <code>org.apache.kafka.common.metrics.JmxReporter</code>
|
|
Please use <code>metrics.jmx.exclude</code> and <code>metrics.jmx.include</code> respectively instead.
|
|
</li>
|
|
<li>The <code>auto.include.jmx.reporter</code> configuration was removed. The <code>metric.reporters</code> configuration
|
|
is now set to <code>org.apache.kafka.common.metrics.JmxReporter</code> by default.
|
|
</li>
|
|
<li>The constructor <code>org.apache.kafka.common.metrics.JmxReporter</code> with string argument was removed.
|
|
See <a href="https://cwiki.apache.org/confluence/display/KAFKA/KIP-606%3A+Add+Metadata+Context+to+MetricsReporter">KIP-606</a> for details.
|
|
</li>
|
|
<li>The <code>bufferpool-wait-time-total</code>, <code>io-waittime-total</code>, and <code>iotime-total</code> metrics were removed.
|
|
Please use <code>bufferpool-wait-time-ns-total</code>, <code>io-wait-time-ns-total</code>, and <code>io-time-ns-total</code> metrics as replacements, respectively.
|
|
</li>
|
|
<li>The <code>kafka.common.requests.DescribeLogDirsResponse.LogDirInfo</code> class was removed. Please use the <code>kafka.clients.admin.DescribeLogDirsResult.descriptions()</code> class
|
|
and <code>kafka.clients.admin.DescribeLogDirsResult.allDescriptions()</code>instead.
|
|
</li>
|
|
<li>The <code>kafka.common.requests.DescribeLogDirsResponse.ReplicaInfo</code> class was removed. Please use the <code>kafka.clients.admin.DescribeLogDirsResult.descriptions()</code> class
|
|
and <code>kafka.clients.admin.DescribeLogDirsResult.allDescriptions()</code>instead.
|
|
</li>
|
|
<li>The <code>org.apache.kafka.common.security.oauthbearer.secured.OAuthBearerLoginCallbackHandler</code> class was removed.
|
|
Please use the <code>org.apache.kafka.common.security.oauthbearer.OAuthBearerLoginCallbackHandler</code> class instead.
|
|
</li>
|
|
<li>The <code>org.apache.kafka.common.security.oauthbearer.secured.OAuthBearerValidatorCallbackHandler</code> class was removed.
|
|
Please use the <code>org.apache.kafka.common.security.oauthbearer.OAuthBearerValidatorCallbackHandler</code> class instead.
|
|
</li>
|
|
<li>The <code>org.apache.kafka.common.errors.NotLeaderForPartitionException</code> class was removed. The <code>org.apache.kafka.common.errors.NotLeaderOrFollowerException</code> is returned
|
|
if a request could not be processed because the broker is not the leader or follower for a topic partition.
|
|
</li>
|
|
<li>The <code>org.apache.kafka.clients.producer.internals.DefaultPartitioner</code> and <code>org.apache.kafka.clients.producer.UniformStickyPartitioner</code> class was removed.
|
|
</li>
|
|
<li>The <code>log.message.format.version</code> and <code>message.format.version</code> configs were removed.
|
|
</li>
|
|
<li>The function <code>onNewBatch</code> in <code>org.apache.kafka.clients.producer.Partitioner</code> class was removed.
|
|
</li>
|
|
<li>The default properties files for KRaft mode are no longer stored in the separate <code>config/kraft</code> directory since Zookeeper has been removed. These files have been consolidated with other configuration files.
|
|
Now all configuration files are in <code>config</code> directory.
|
|
</li>
|
|
<li>The valid format for <code>--bootstrap-server</code> only supports comma-separated value, such as <code>host1:port1,host2:port2,...</code>.
|
|
Providing other formats, like space-separated bootstrap servers (e.g., <code>host1:port1 host2:port2 host3:port3</code>), will result in an exception, even though this was allowed in Apache Kafka versions prior to 4.0.
|
|
</li>
|
|
</ul>
|
|
</li>
|
|
<li><a id="upgrade_400_notable_broker" href="#upgrade_400_notable_broker"><b>Broker</b></a>
|
|
<ul>
|
|
<li>The <code>delegation.token.master.key</code> configuration was removed.
|
|
Please use <code>delegation.token.secret.key</code> instead.
|
|
</li>
|
|
<li>
|
|
The <code>offsets.commit.required.acks</code> configuration was removed.
|
|
See <a href="https://cwiki.apache.org/confluence/x/9YobEg">KIP-1041</a> for details.
|
|
</li>
|
|
<li>The <code>log.message.timestamp.difference.max.ms</code> configuration was removed.
|
|
Please use <code>log.message.timestamp.before.max.ms</code> and <code>log.message.timestamp.after.max.ms</code> instead.
|
|
See <a href="https://cwiki.apache.org/confluence/display/KAFKA/KIP-937%3A+Improve+Message+Timestamp+Validation">KIP-937</a> for details.
|
|
</li>
|
|
<li>
|
|
The <code>remote.log.manager.copier.thread.pool.size</code> configuration default value was changed to 10 from -1.
|
|
Values of -1 are no longer valid. A minimum of 1 or higher is valid.
|
|
See <a href="https://cwiki.apache.org/confluence/display/KAFKA/KIP-1030%3A+Change+constraints+and+default+values+for+various+configurations">KIP-1030</a>
|
|
</li>
|
|
<li>
|
|
The <code>remote.log.manager.expiration.thread.pool.size</code> configuration default value was changed to 10 from -1.
|
|
Values of -1 are no longer valid. A minimum of 1 or higher is valid.
|
|
See <a href="https://cwiki.apache.org/confluence/display/KAFKA/KIP-1030%3A+Change+constraints+and+default+values+for+various+configurations">KIP-1030</a>
|
|
</li>
|
|
<li>
|
|
The <code>remote.log.manager.thread.pool.size</code> configuration default value was changed to 2 from 10.
|
|
See <a href="https://cwiki.apache.org/confluence/display/KAFKA/KIP-1030%3A+Change+constraints+and+default+values+for+various+configurations">KIP-1030</a>
|
|
</li>
|
|
<li>
|
|
The minimum <code>segment.bytes/log.segment.bytes</code> has changed from 14 bytes to 1MB.
|
|
See <a href="https://cwiki.apache.org/confluence/display/KAFKA/KIP-1030%3A+Change+constraints+and+default+values+for+various+configurations">KIP-1030</a>
|
|
</li>
|
|
</ul>
|
|
</li>
|
|
<li><a id="upgrade_400_notable_MirrorMaker" href="#upgrade_400_notable_MirrorMaker"><b>MirrorMaker</b></a>
|
|
<ul>
|
|
<li>The original MirrorMaker (MM1) and related classes were removed. Please use the Connect-based
|
|
MirrorMaker (MM2), as described in the <a href="/{{version}}/documentation/#georeplication">Geo-Replication section.</a>.
|
|
</li>
|
|
<li>The <code>use.incremental.alter.configs</code> configuration was removed from <code>MirrorSourceConnector</code>.
|
|
The modified behavior is identical to the previous <code>required</code> configuration, therefore users should ensure that brokers in the target cluster are at least running 2.3.0.
|
|
</li>
|
|
<li>The <code>add.source.alias.to.metrics</code> configuration was removed from <code>MirrorSourceConnector</code>.
|
|
The source cluster alias is now always added to the metrics.
|
|
</li>
|
|
<li>The <code>config.properties.blacklist</code> was removed from the <code>org.apache.kafka.connect.mirror.MirrorSourceConfig</code>
|
|
Please use <code>config.properties.exclude</code> instead.
|
|
</li>
|
|
<li>The <code>topics.blacklist</code> was removed from the <code>org.apache.kafka.connect.mirror.MirrorSourceConfig</code>
|
|
Please use <code>topics.exclude</code> instead.
|
|
</li>
|
|
<li>The <code>groups.blacklist</code> was removed from the <code>org.apache.kafka.connect.mirror.MirrorSourceConfig</code>
|
|
Please use <code>groups.exclude</code> instead.
|
|
</li>
|
|
</ul>
|
|
</li>
|
|
<li><a id="upgrade_400_notable_tools" href="#upgrade_400_notable_tools"><b>Tools</b></a>
|
|
<ul>
|
|
<li>The <code>kafka.common.MessageReader</code> class was removed. Please use the
|
|
<a href="/{{version}}/javadoc/org/apache/kafka/tools/api/RecordReader.html"><code>org.apache.kafka.tools.api.RecordReader</code></a>
|
|
interface to build custom readers for the <code>kafka-console-producer</code> tool.
|
|
</li>
|
|
<li>The <code>kafka.tools.DefaultMessageFormatter</code> class was removed. Please use the <code>org.apache.kafka.tools.consumer.DefaultMessageFormatter</code> class instead.
|
|
</li>
|
|
<li>The <code>kafka.tools.LoggingMessageFormatter</code> class was removed. Please use the <code>org.apache.kafka.tools.consumer.LoggingMessageFormatter</code> class instead.
|
|
</li>
|
|
<li>The <code>kafka.tools.NoOpMessageFormatter</code> class was removed. Please use the <code>org.apache.kafka.tools.consumer.NoOpMessageFormatter</code> class instead.
|
|
</li>
|
|
<li>The <code>--whitelist</code> option was removed from the <code>kafka-console-consumer</code> command line tool.
|
|
Please use <code>--include</code> instead.
|
|
</li>
|
|
<li>Redirections from the old tools packages have been removed:
|
|
<code>kafka.admin.FeatureCommand</code>,
|
|
<code>kafka.tools.ClusterTool</code>,
|
|
<code>kafka.tools.EndToEndLatency</code>,
|
|
<code>kafka.tools.StateChangeLogMerger</code>,
|
|
<code>kafka.tools.StreamsResetter</code>,
|
|
<code>kafka.tools.JmxTool</code>.
|
|
</li>
|
|
<li>The <code>--authorizer</code>, <code>--authorizer-properties</code>, and <code>--zk-tls-config-file</code> options were removed from the <code>kafka-acls</code> command line tool.
|
|
Please use <code>--bootstrap-server</code> or <code>--bootstrap-controller</code> instead.
|
|
</li>
|
|
<li>
|
|
The <code>kafka.serializer.Decoder</code> trait was removed, please use the
|
|
<a href="/{{version}}/javadoc/org/apache/kafka/tools/api/Decoder.html"><code>org.apache.kafka.tools.api.Decoder</code></a>
|
|
interface to build custom decoders for the <code>kafka-dump-log</code> tool.
|
|
</li>
|
|
<li>The <code>kafka.coordinator.group.OffsetsMessageFormatter</code> class was removed. Please use the <code>org.apache.kafka.tools.consumer.OffsetsMessageFormatter</code> class instead.
|
|
</li>
|
|
<li>The <code>kafka.coordinator.group.GroupMetadataMessageFormatter</code> class was removed. Please use the <code>org.apache.kafka.tools.consumer.GroupMetadataMessageFormatter</code> class instead.
|
|
</li>
|
|
<li>The <code>kafka.coordinator.transaction.TransactionLogMessageFormatter</code> class was removed. Please use the <code>org.apache.kafka.tools.consumer.TransactionLogMessageFormatter</code> class instead.
|
|
</li>
|
|
<li>The <code>--topic-white-list</code> option was removed from the <code>kafka-replica-verification</code> command line tool.
|
|
Please use <code>--topics-include</code> instead.
|
|
</li>
|
|
<li>The <code>--broker-list</code> option was removed from the <code>kafka-verifiable-consumer</code> command line tool.
|
|
Please use <code>--bootstrap-server</code> instead.
|
|
</li>
|
|
<li>kafka-configs.sh now uses incrementalAlterConfigs API to alter broker configurations instead of the deprecated alterConfigs API,
|
|
and it will fall directly if the broker doesn't support incrementalAlterConfigs API, which means the broker version is prior to 2.3.x.
|
|
See <a href="https://cwiki.apache.org/confluence/display/KAFKA/KIP-1011%3A+Use+incrementalAlterConfigs+when+updating+broker+configs+by+kafka-configs.sh">KIP-1011</a> for more details.
|
|
</li>
|
|
<li>The <code>kafka.admin.ZkSecurityMigrator</code> tool was removed.
|
|
</li>
|
|
</ul>
|
|
</li>
|
|
<li><a id="upgrade_400_notable_connect" href="#upgrade_400_notable_connect"><b>Connect</b></a>
|
|
<ul>
|
|
<li>The <code>whitelist</code> and <code>blacklist</code> configurations were removed from the <code>org.apache.kafka.connect.transforms.ReplaceField</code> transformation.
|
|
Please use <code>include</code> and <code>exclude</code> respectively instead.
|
|
</li>
|
|
<li>The <code>onPartitionsRevoked(Collection<TopicPartition>)</code> and <code>onPartitionsAssigned(Collection<TopicPartition>)</code> methods
|
|
were removed from <code>SinkTask</code>.
|
|
</li>
|
|
<li>The <code>commitRecord(SourceRecord)</code> method was removed from <code>SourceTask</code>.
|
|
</li>
|
|
</ul>
|
|
</li>
|
|
<li><a id="upgrade_400_notable_consumer" href="#upgrade_400_notable_consumer"><b>Consumer</b></a>
|
|
<ul>
|
|
<li>The <code>poll(long)</code> method was removed from the consumer. Please use <code>poll(Duration)</code> instead. Note that there is
|
|
a difference in behavior between the two methods. The <code>poll(Duration)</code> method does not block beyond the timeout awaiting
|
|
partition assignment, whereas the earlier <code>poll(long)</code> method used to wait beyond the timeout.
|
|
</li>
|
|
<li>The <code>committed(TopicPartition)</code> and <code>committed(TopicPartition, Duration)</code> methods were removed from the consumer.
|
|
Please use <code>committed(Set<TopicPartition>)</code> and <code>committed(Set<TopicPartition>, Duration)</code> instead.
|
|
</li>
|
|
<li>
|
|
The <code>setException(KafkaException)</code> method was removed from the <code>org.apache.kafka.clients.consumer.MockConsumer</code>.
|
|
Please use <code>setPollException(KafkaException)</code> instead.
|
|
</li>
|
|
</ul>
|
|
</li>
|
|
<li><a id="upgrade_400_notable_producer" href="#upgrade_400_notable_producer"><b>Producer</b></a>
|
|
<ul>
|
|
<li>The <code>enable.idempotence</code> configuration will no longer automatically fall back when the <code>max.in.flight.requests.per.connection</code> value exceeds 5.
|
|
</li>
|
|
<li>The deprecated <code>sendOffsetsToTransaction(Map<TopicPartition, OffsetAndMetadata>, String)</code> method has been removed from the Producer API.
|
|
</li>
|
|
<li>The default <code>linger.ms</code> changed from 0 to 5 in Apache Kafka 4.0 as the efficiency gains from larger batches typically result in
|
|
similar or lower producer latency despite the increased linger.
|
|
</li>
|
|
</ul>
|
|
</li>
|
|
<li><a id="upgrade_400_notable_admin_client" href="#upgrade_400_notable_admin_client"><b>Admin client</b></a>
|
|
<ul>
|
|
<li>
|
|
The <code>alterConfigs</code> method was removed from the <code>org.apache.kafka.clients.admin.Admin</code>.
|
|
Please use <code>incrementalAlterConfigs</code> instead.
|
|
</li>
|
|
<li>The <code>org.apache.kafka.common.ConsumerGroupState</code> enumeration and related methods have been deprecated. Please use <code>GroupState</code> instead
|
|
which applies to all types of group.
|
|
</li>
|
|
<li>The <code>Admin.describeConsumerGroups</code> method used to return a <code>ConsumerGroupDescription</code> in state
|
|
<code>DEAD</code> if the group ID was not found. In Apache Kafka 4.0, the <code>GroupIdNotFoundException</code>
|
|
is thrown instead as part of the support for new types of group.
|
|
</li>
|
|
<li>The <code>org.apache.kafka.clients.admin.DeleteTopicsResult.values()</code> method was removed.
|
|
Please use <code>org.apache.kafka.clients.admin.DeleteTopicsResult.topicNameValues()</code> instead.
|
|
<li>The <code>org.apache.kafka.clients.admin.TopicListing.TopicListing(String, boolean)</code> method was removed.
|
|
Please use <code>org.apache.kafka.clients.admin.TopicListing.TopicListing(String, Uuid, boolean)</code> instead.
|
|
</li>
|
|
<li>The <code>org.apache.kafka.clients.admin.ListConsumerGroupOffsetsOptions.topicPartitions(List<TopicPartition>)</code> method was removed.
|
|
Please use <code>org.apache.kafka.clients.admin.Admin.listConsumerGroupOffsets(Map<String, ListConsumerGroupOffsetsSpec>, ListConsumerGroupOffsetsOptions)</code> instead.
|
|
</li>
|
|
<li>
|
|
The deprecated <code>dryRun</code> methods were removed from the <code>org.apache.kafka.clients.admin.UpdateFeaturesOptions</code>.
|
|
Please use <code>validateOnly</code> instead.
|
|
</li>
|
|
<li>The constructor <code>org.apache.kafka.clients.admin.FeatureUpdate</code> with short and boolean arguments was removed.
|
|
Please use the constructor that accepts short and the specified UpgradeType enum instead.
|
|
</li>
|
|
<li>
|
|
The <code>allowDowngrade</code> method was removed from the <code>org.apache.kafka.clients.admin.FeatureUpdate</code>.
|
|
</li>
|
|
<li>The <code>org.apache.kafka.clients.admin.DescribeTopicsResult.DescribeTopicsResult(Map<String, KafkaFuture<TopicDescription>>)</code> method was removed.
|
|
Please use <code>org.apache.kafka.clients.admin.DescribeTopicsResult.DescribeTopicsResult(Map<Uuid, KafkaFuture<TopicDescription>>, Map<String, KafkaFuture<TopicDescription>>)</code> instead.
|
|
</li>
|
|
<li>
|
|
The <code>values()</code> method was removed from the <code>org.apache.kafka.clients.admin.DescribeTopicsResult</code>.
|
|
Please use <code>topicNameValues()</code> instead.
|
|
</li>
|
|
<li>
|
|
The <code>all()</code> method was removed from the <code>org.apache.kafka.clients.admin.DescribeTopicsResult</code>.
|
|
Please use <code>allTopicNames()</code> instead.
|
|
</li>
|
|
</ul>
|
|
</li>
|
|
<li><a id="upgrade_400_notable_kafka_streams" href="#upgrade_400_notable_kafka_streams"><b>Kafka Streams</b></a>
|
|
<ul>
|
|
<li>
|
|
All public API, deprecated in Apache Kafka 3.6 or an earlier release, have been removed, with the exception of <code>JoinWindows.of()</code> and <code>JoinWindows#grace()</code>.
|
|
See <a href="https://issues.apache.org/jira/browse/KAFKA-17531">KAFKA-17531</a> for details.
|
|
</li>
|
|
<li>
|
|
The most important changes are highlighted in the <a href="/{{version}}/documentation/streams/upgrade-guide.html#streams_api_changes_400">Kafka Streams upgrade guide</a>.
|
|
</li>
|
|
<li>
|
|
For a full list of changes, see <a href="https://issues.apache.org/jira/browse/KAFKA-12822">KAFKA-12822</a>.
|
|
</li>
|
|
</ul>
|
|
</li>
|
|
</ul>
|
|
</li>
|
|
<li>Other changes:
|
|
<ul>
|
|
<li>
|
|
The minimum Java version required by clients and Kafka Streams applications has been increased from Java 8 to Java 11
|
|
while brokers, connect and tools now require Java 17.
|
|
See <a href="https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=181308223">KIP-750</a> and
|
|
<a href="https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=284789510">KIP-1013</a> for more details.
|
|
</li>
|
|
<li>
|
|
Java 23 support has been added in Apache Kafka 4.0
|
|
</li>
|
|
<li>
|
|
Scala 2.12 support has been removed in Apache Kafka 4.0
|
|
See <a href="https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=181308218">KIP-751</a> for more details
|
|
</li>
|
|
<li>
|
|
Logging framework has been migrated from Log4j to Log4j2.
|
|
Users can use the log4j-transform-cli tool to automatically convert their existing Log4j configuration files to Log4j2 format.
|
|
See <a href="https://logging.staged.apache.org/log4j/transform/cli.html#log4j-transform-cli">log4j-transform-cli</a> for more details.
|
|
Log4j2 provides limited compatibility for Log4j configurations.
|
|
See <a href="https://logging.apache.org/log4j/2.x/migrate-from-log4j1.html#ConfigurationCompatibility">Use Log4j 1 to Log4j 2 bridge</a> for more information,
|
|
</li>
|
|
<li>
|
|
KafkaLog4jAppender has been removed, users should migrate to the log4j2 appender
|
|
See <a href="https://logging.apache.org/log4j/2.x/manual/appenders.html#KafkaAppender">KafkaAppender</a> for more details
|
|
</li>
|
|
<li>
|
|
The <code>--delete-config</code> option in the <code>kafka-topics</code> command line tool has been deprecated.
|
|
</li>
|
|
<li>
|
|
For implementors of RemoteLogMetadataManager (RLMM), a new API `nextSegmentWithTxnIndex` is
|
|
introduced in RLMM to allow the implementation to return the next segment metadata with a transaction
|
|
index. This API is used when the consumers are enabled with isolation level as READ_COMMITTED.
|
|
See <a href="https://cwiki.apache.org/confluence/display/KAFKA/KIP-1058:+Txn+consumer+exerts+pressure+on+remote+storage+when+collecting+aborted+transactions">KIP-1058</a> for more details.
|
|
</li>
|
|
<li>
|
|
The criteria for identifying internal topics in ReplicationPolicy and DefaultReplicationPolicy have
|
|
been updated to enable the replication of topics that appear to be internal but aren't truly internal to Kafka and Mirror Maker 2.
|
|
See <a href="https://cwiki.apache.org/confluence/display/KAFKA/KIP-1074%3A+Allow+the+replication+of+user+internal+topics">KIP-1074</a> for more details.
|
|
</li>
|
|
<li>
|
|
KIP-714 is now enabled for Kafka Streams via <a href="https://cwiki.apache.org/confluence/display/KAFKA/KIP-1076%3A++Metrics+for+client+applications+KIP-714+extension">KIP-1076</a>.
|
|
This allows to not only collect the metric of the internally used clients of a Kafka Streams appliction via a broker-side plugin,
|
|
but also to collect the <a href="/{{version}}/documentation/#kafka_streams_monitoring">metrics</a> of the Kafka Streams runtime itself.
|
|
</li>
|
|
<li>
|
|
The default value of 'num.recovery.threads.per.data.dir' has been changed from 1 to 2. The impact of this is faster
|
|
recovery post unclean shutdown at the expense of extra IO cycles.
|
|
See <a href="https://cwiki.apache.org/confluence/display/KAFKA/KIP-1030%3A+Change+constraints+and+default+values+for+various+configurations">KIP-1030</a>
|
|
</li>
|
|
<li>
|
|
The default value of 'message.timestamp.after.max.ms' has been changed from Long.Max to 1 hour. The impact of this messages with a
|
|
timestamp of more than 1 hour in the future will be rejected when message.timestamp.type=CreateTime is set.
|
|
See <a href="https://cwiki.apache.org/confluence/display/KAFKA/KIP-1030%3A+Change+constraints+and+default+values+for+various+configurations">KIP-1030</a>
|
|
</li>
|
|
<li>Introduced in KIP-890, the <code>TransactionAbortableException</code> enhances error handling within transactional operations by clearly indicating scenarios where transactions should be aborted due to errors. It is important for applications to properly manage both <code>TimeoutException</code> and <code>TransactionAbortableException</code> when working with transaction producers.</li>
|
|
<ul>
|
|
<li><b>TimeoutException:</b> This exception indicates that a transactional operation has timed out. Given the risk of message duplication that can arise from retrying operations after a timeout (potentially violating exactly-once semantics), applications should treat timeouts as reasons to abort the ongoing transaction.</li>
|
|
<li><b>TransactionAbortableException:</b> Specifically introduced to signal errors that should lead to transaction abortion, ensuring this exception is properly handled is critical for maintaining the integrity of transactional processing.</li>
|
|
<li>To ensure seamless operation and compatibility with future Kafka versions, developers are encouraged to update their error-handling logic to treat both exceptions as triggers for aborting transactions. This approach is pivotal for preserving exactly-once semantics.</li>
|
|
<li> See <a href="https://cwiki.apache.org/confluence/display/KAFKA/KIP-890%3A+Transactions+Server-Side+Defense">KIP-890</a> and
|
|
<a href="https://cwiki.apache.org/confluence/display/KAFKA/KIP-1050%3A+Consistent+error+handling+for+Transactions">KIP-1050</a> for more details </li>
|
|
</ul>
|
|
</ul>
|
|
</li>
|
|
</ul>
|
|
|
|
<h4><a id="upgrade_3_9_0" href="#upgrade_3_9_0">Upgrading to 3.9.0 and older versions</a></h4>
|
|
|
|
<p>See <a href="/39/documentation/#upgrade">Upgrading From Previous Versions</a> in the 3.9 documentation.</p>
|
|
|
|
</script>
|
|
|
|
<div class="p-upgrade"></div>
|