mirror of https://github.com/apache/kafka.git
MINOR: added upgrade and API changes to docs
Author: Matthias J. Sax <matthias@confluent.io> Reviewers: Ismael Juma, Will Marshall, Damian Guy, Guozhang Wang, Michael G. Noll Closes #2461 from mjsax/addStreamsUpdateSecton
This commit is contained in:
Matthias J. Sax
Guozhang Wang
parent
ea70a9bcfc
commit
ab4ebb4bb9
|
|
@ -35,6 +35,9 @@
|
|||
<li><a href="#streams_dsl">High-level streams DSL</a>
|
||||
</ul>
|
||||
</li>
|
||||
<li>
|
||||
<a href="#streams_upgrade">Upgrade guide and API changes</a>
|
||||
</li>
|
||||
</ol>
|
||||
|
||||
<h2><a id="streams_overview" href="#streams_overview">Overview</a></h2>
|
||||
|
|
@ -466,7 +469,88 @@
|
|||
in <code>StreamsConfig</code> before running it. A complete list of
|
||||
Kafka Streams configs can be found <a href="#streamsconfigs"><b>here</b></a>.
|
||||
</p>
|
||||
|
||||
<h2><a id="streams_upgrade" href="#upgrade">Upgrade guide and API changes</a></h2>
|
||||
|
||||
<h3><a id="streams_upgrade_1020" href="#upgrade_1020">Upgrading a Kafka Streams Application</a></h3>
|
||||
|
||||
<h4>Upgrading from 0.10.1.x to 0.10.2.0</h4>
|
||||
<p>
|
||||
See <a href="../#upgrade_1020_streams">Upgrade Section</a> for details.
|
||||
</p>
|
||||
|
||||
<h3><a id="streams_api_changes" href="#api_changes">Streams API changes in 0.10.2.0</a></h3>
|
||||
<li> New methods in <code>KafkaStreams</code>:
|
||||
<ul>
|
||||
<li> set a listener to react on application state change via <code>#setStateListener(StateListener listener)</code> </li>
|
||||
<li> retrieve the current application state via <code>#state()</code> </li>
|
||||
<li> retrieve the global metrics registry via <code>#metrics()</code> </li>
|
||||
<li> apply a timeout when closing an application via <code>#close(long timeout, TimeUnit timeUnit)</code> </li>
|
||||
<li> specify a custom indent when retrieving Kafka Streams information via <code>#toString(String indent)</code> </li>
|
||||
</ul>
|
||||
</li>
|
||||
<li> Parameter updates in <code>StreamsConfig</code>:
|
||||
<ul>
|
||||
<li> parameter <code>zookeeper.connect</code> was deprecated </li>
|
||||
<ul>
|
||||
<li> a Kafka Streams application does no longer interact with Zookeeper for topic management but uses the new broker admin protocol
|
||||
(cf. <a href="https://cwiki.apache.org/confluence/display/KAFKA/KIP-4+-+Command+line+and+centralized+administrative+operations#KIP-4-Commandlineandcentralizedadministrativeoperations-TopicAdminSchema.1">KIP-4, Section "Topic Admin Schema"</a>) </li>
|
||||
<li> thus, parameter "zookeeper.connect" is ignored in 0.10.2 and should be removed from <code>StreamsConfig</code> </li>
|
||||
</ul>
|
||||
<li> added many new parameters for metrics, security, and client configurations </li>
|
||||
</ul>
|
||||
</li>
|
||||
<li> Changes in <code>StreamsMetrics</code> interface:
|
||||
<ul>
|
||||
<li> removed methods: <code>#addLatencySensor()</code> </li>
|
||||
<li> added methods: <code>#addLatencyAndThroughputSensor()</code>, <code>#addThroughputSensor()</code>, <code>#recordThroughput()</code>,
|
||||
<code>#addSensor()</code>, <code>#removeSensor()</code> </li>
|
||||
</ul>
|
||||
</li>
|
||||
<li> New methods in <code>TopologyBuilder</code>:
|
||||
<ul>
|
||||
<li> added overloads for <code>#addSource()</code> that allow to define a <code>auto.offset.reset</code> policy per source node </li>
|
||||
<li> added methods <code>#addGlobalStore()</code> to add global <code>StateStore</code>s </li>
|
||||
</ul>
|
||||
</li>
|
||||
<li> New methods in <code>KStreamBuilder</code>:
|
||||
<ul>
|
||||
<li> added overloads for <code>#stream()</code> and <code>#table()</code> that allow to define a <code>auto.offset.reset</code> policy per input stream/table </li>
|
||||
<li> <code>#table()</code> always requires store name </li>
|
||||
<li> added method <code>#globalKTable()</code> to create a <code>GlobalKTable</code> </li>
|
||||
</ul>
|
||||
</li>
|
||||
<li> New joins for <code>KStream</code>:
|
||||
<ul>
|
||||
<li> added overloads for <code>#join()</code> to join with <code>KTable</code> </li>
|
||||
<li> added overloads for <code>#join()</code> and <code>leftJoin()</code> to join with <code>GlobalKTable</code> </li>
|
||||
</ul>
|
||||
</li>
|
||||
<li> Aligned <code>null</code>-key handling for <code>KTable</code> joins:
|
||||
<ul>
|
||||
<li> like all other KTable operations, <code>KTable-KTable</code> joins do not throw an exception on <code>null</code> key records anymore, but drop those records silently </li>
|
||||
</ul>
|
||||
</li>
|
||||
<li> New window type <em>Session Windows</em>:
|
||||
<ul>
|
||||
<li> added class <code>SessionWindows</code> to specify session windows </li>
|
||||
<li> added overloads for <code>KGroupedStream</code> methods <code>#count()</code>, <code>#reduce()</code>, and <code>#aggregate()</code>
|
||||
to allow session window aggregations </li>
|
||||
</ul>
|
||||
</li>
|
||||
<li> Changes to <code>TimestampExtractor</code>:
|
||||
<ul>
|
||||
<li> method <code>#extract()</code> has a second parameter now </li>
|
||||
<li> new default timestamp extractor class <code>FailOnInvalidTimestamp</code>
|
||||
(it gives the same behavior as old (and removed) default extractor <code>ConsumerRecordTimestampExtractor</code>) </li>
|
||||
<li> new alternative timestamp extractor classes <code>LogAndSkipOnInvalidTimestamp</code> and <code>UsePreviousTimeOnInvalidTimestamps</code> </li>
|
||||
</ul>
|
||||
</li>
|
||||
<li> Relaxed type constraints of many DSL interfaces, classes, and methods (cf. <a href="https://cwiki.apache.org/confluence/display/KAFKA/KIP-100+-+Relax+Type+constraints+in+Kafka+Streams+API">KIP-100</a>). </li>
|
||||
</ul>
|
||||
</pre>
|
||||
|
||||
|
||||
</script>
|
||||
|
||||
<!--#include virtual="../includes/_header.htm" -->
|
||||
|
|
|
|||
|
|
@ -47,14 +47,16 @@ Kafka cluster before upgrading your clients. Version 0.10.2 brokers support 0.8.
|
|||
|
||||
<p><b>Note:</b> Bumping the protocol version and restarting can be done any time after the brokers were upgraded. It does not have to be immediately after.
|
||||
|
||||
<ol>
|
||||
<li>Upgrading a Kafka Streams Applications:
|
||||
<ul>
|
||||
<li>You need to recompile your code. Just swapping the jar file will not work and will break your appliation.</li>
|
||||
<li>If you use a custom timestamp extractor, you will need to update this code, because the <code>TimestampExtractor</code> interface was changed.</li>
|
||||
</ul>
|
||||
</li>
|
||||
</ol>
|
||||
<h5><a id="upgrade_1020_streams" href="#upgrade_1020_streams">Upgrading a Kafka Streams Application</a></h5>
|
||||
<ul>
|
||||
<li> Upgrading your Streams application from 0.10.1 to 0.10.2 does not require a broker upgrade.
|
||||
A Kafka Streams 0.10.2 application can connect to 0.10.2 and 0.10.1 brokers (it is not possible to connect to 0.10.0 brokers though). </li>
|
||||
<li> You need to recompile your code. Just swapping the Kafka Streams library jar file will not work and will break your application. </li>
|
||||
<li> <code>KStreamBuilder#table()</code> always requires a store name. </li>
|
||||
<li> <code>KTable#through()</code> always requires a store name. </li>
|
||||
<li> If you use a custom (i.e., user implemented) timestamp extractor, you will need to update this code, because the <code>TimestampExtractor</code> interface was changed. </li>
|
||||
<li> If you register custom metrics, you will need to update this code, because the <code>StreamsMetric</code> interface was changed. </li>
|
||||
</ul>
|
||||
|
||||
<h5><a id="upgrade_1020_notable" href="#upgrade_1020_notable">Notable changes in 0.10.2.0</a></h5>
|
||||
<ul>
|
||||
|
|
@ -67,6 +69,8 @@ Kafka cluster before upgrading your clients. Version 0.10.2 brokers support 0.8.
|
|||
A new close API with timeout has been added to <code>KafkaConsumer</code> to control the maximum wait time.</li>
|
||||
<li>Multiple regular expressions separated by commas can be passed to MirrorMaker with the new Java consumer via the --whitelist option. This
|
||||
makes the behaviour consistent with MirrorMaker when used the old Scala consumer.</li>
|
||||
<li>Upgrading your Streams application from 0.10.1 to 0.10.2 does not require a broker upgrade.
|
||||
A Kafka Streams 0.10.2 application can connect to 0.10.2 and 0.10.1 brokers (it is not possible to connect to 0.10.0 brokers though).</li>
|
||||
<li>The Zookeeper dependency was removed from the Streams API. The Streams API now uses the Kafka protocol to manage internal topics instead of
|
||||
modifying Zookeeper directly. This eliminates the need for privileges to access Zookeeper directly and "StreamsConfig.ZOOKEEPER_CONFIG"
|
||||
should not be set in the Streams app any more. If the Kafka cluster is secured, Streams apps must have the required security privileges to create new topics.</li>
|
||||
|
|
|
|||
Loading…
Reference in New Issue