root
/
kafka
mirror of https://github.com/apache/kafka.git
1
0
Fork 0
Code Issues Actions 2 Packages Projects Releases Wiki Activity

KAFKA-15022: [10/N] docs for rack aware assignor (#14181) 

Reviewers: Matthias J. Sax <matthias@confluent.io>
This commit is contained in:
Hao Li 2023-08-11 11:40:33 -07:00 committed by GitHub
parent 1fd58e30cf
commit 4268e502ec
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
6 changed files with 84 additions and 4 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

7
docs/streams/architecture.html
View File

@ -167,6 +167,13 @@
rack of the active tasks fails. See <code>rack.aware.assignment.tags</code>
in the <a href="/{{version}}/documentation/streams/developer-guide/config-streams.html#rack-aware-assignment-tags"><b>Kafka Streams Developer Guide</b></a> section.
</p>
<p>
There is also a client config <code>client.rack</code> which can set the rack for a Kafka consumer. If brokers also have their rack set via <code>broker.rack</code>, then rack aware task
assignment can be enabled via <code>rack.aware.assignment.strategy</code> (cf. <a href="/{{version}}/documentation/streams/developer-guide/config-streams.html#rack-aware-assignment-strategy"><b>Kafka Streams Developer Guide</b></a>) to compute a task assignment which can reduce cross rack traffic by trying to assign tasks to clients with the same rack.
Note that <code>client.rack</code> can also be used to distribute standby tasks to different racks from the active ones, which has a similar functionality as <code>rack.aware.assignment.tags</code>.
Currently, <code>rack.aware.assignment.tag</code> takes precedence in distributing standby tasks which means if both configs present, <code>rack.aware.assignment.tag</code> will be used for distributing
standby tasks on different racks from the active ones because it can configure more tag keys.
</p>
<div class="pagination">
<a href="/{{version}}/documentation/streams/core-concepts" class="pagination__btn pagination__btn__prev">Previous</a>

60
docs/streams/developer-guide/config-streams.html
View File

@ -84,7 +84,10 @@ settings.put(... , ...);</code></pre>
<li><a class="reference internal" href="#partition-grouper" id="id12">partition.grouper</a></li>
<li><a class="reference internal" href="#probing-rebalance-interval-ms" id="id30">probing.rebalance.interval.ms</a></li>
<li><a class="reference internal" href="#processing-guarantee" id="id25">processing.guarantee</a></li>
<li><a class="reference internal" href="#rack-aware-assignment-non-overlap-cost" id="id37">rack.aware.assignment.non_overlap_cost</a></li>
<li><a class="reference internal" href="#rack-aware-assignment-strategy" id="id35">rack.aware.assignment.strategy</a></li>
<li><a class="reference internal" href="#rack-aware-assignment-tags" id="id34">rack.aware.assignment.tags</a></li>
<li><a class="reference internal" href="#rack-aware-assignment-traffic-cost" id="id36">rack.aware.assignment.traffic_cost</a></li>
<li><a class="reference internal" href="#replication-factor" id="id13">replication.factor</a></li>
<li><a class="reference internal" href="#rocksdb-config-setter" id="id20">rocksdb.config.setter</a></li>
<li><a class="reference internal" href="#state-dir" id="id14">state.dir</a></li>
@ -685,6 +688,45 @@ streamsConfiguration.put(StreamsConfig.DEFAULT_TIMESTAMP_EXTRACTOR_CLASS_CONFIG,
<p>This is discussed in more detail in <a class="reference internal" href="datatypes.html#streams-developer-guide-serdes"><span class="std std-ref">Data types and serialization</span></a>.</p>
</div></blockquote>
</div>
<div class="section" id="rack-aware-assignment-non-overlap-cost">
<h4><a class="toc-backref" href="#id37">rack.aware.assignment.non_overlap_cost</a><a class="headerlink" href="#rack-aware-assignment-non-overlap-cost" title="Permalink to this headline"></a></h4>
<blockquote>
<div>
<p>
This configuration sets the cost of moving a task from the original assignment computed either by <code class="docutils literal"><span class="pre">StickyTaskAssignor</span></code> or
<code class="docutils literal"><span class="pre">HighAvailabilityTaskAssignor</span></code>. Together with <code class="docutils literal"><span class="pre">rack.aware.assignment.traffic_cost</span></code>,
they control whether the optimizer favors minimizing cross rack traffic or minimizing the movement of tasks in the existing assignment. If this config is set to a larger value than <code class="docutils literal"><span class="pre">rack.aware.assignment.traffic_cost</span></code>,
the optimizer will try to maintain the existing assignment computed by the task assignor. Note that the optimizer takes the ratio of these two configs into consideration of favoring maintaining existing assignment or minimizing traffic cost. For example, setting
<code class="docutils literal"><span class="pre">rack.aware.assignment.non_overlap_cost</span></code> to 10 and <code class="docutils literal"><span class="pre">rack.aware.assignment.traffic_cost</span></code> to 1 is more likely to maintain existing assignment than setting
<code class="docutils literal"><span class="pre">rack.aware.assignment.non_overlap_cost</span></code> to 100 and <code class="docutils literal"><span class="pre">rack.aware.assignment.traffic_cost</span></code> to 50.
</p>
<p>
The default value is null which means default <code class="docutils literal"><span class="pre">non_overlap_cost</span></code> in different assignors will be used. In <code class="docutils literal"><span class="pre">StickyTaskAssignor</span></code>, it has a default value of 10 and <code class="docutils literal"><span class="pre">rack.aware.assignment.traffic_cost</span></code> has
a default value of 1, which means maintaining stickiness is preferred in <code class="docutils literal"><span class="pre">StickyTaskAssignor</span></code>. In <code class="docutils literal"><span class="pre">HighAvailabilityTaskAssignor</span></code>, it has a default value of 1 and <code class="docutils literal"><span class="pre">rack.aware.assignment.traffic_cost</span></code> has
a default value of 10, which means minimizing cross rack traffic is preferred in <code class="docutils literal"><span class="pre">HighAvailabilityTaskAssignor</span></code>.
</p>
</div>
</blockquote>
</div>
<div class="section" id="rack-aware-assignment-strategy">
<h4><a class="toc-backref" href="#id35">rack.aware.assignment.strategy</a><a class="headerlink" href="#rack-aware-assignment-strategy" title="Permalink to this headline"></a></h4>
<blockquote>
<div>
<p>
This configuration sets the strategy Kafka Streams uses for rack aware task assignment so that cross traffic from broker to client can be reduced. This config will only take effect when <code class="docutils literal"><span class="pre">broker.rack</span></code>
is set on the brokers and <code class="docutils literal"><span class="pre">client.rack</span></code> is set on Kafka Streams side. There are two settings for this config:
</p>
<ul class="simple">
<li><code class="docutils literal"><span class="pre">none</span></code>. This is the default value which means rack aware task assignment will be disabled.</li>
<li><code class="docutils literal"><span class="pre">min_traffic</span></code>. This settings means that the rack aware task assigner will compute an assignment which tries to minimize cross rack traffic.</li>
</ul>
<p>
This config can be used together with <a class="reference internal" href="#rack-aware-assignment-non-overlap-cost">rack.aware.assignment.non_overlap_cost</a> and
<a class="reference internal" href="#rack-aware-assignment-traffic-cost">rack.aware.assignment.traffic_cost</a> to balance reducing cross rack traffic and maintaining the existing assignment.
</p>
</div>
</blockquote>
</div>
<div class="section" id="rack-aware-assignment-tags">
<h4><a class="toc-backref" href="#id34">rack.aware.assignment.tags</a><a class="headerlink" href="#rack-aware-assignment-tags" title="Permalink to this headline"></a>
</h4>
@ -718,6 +760,24 @@ rack.aware.assignment.tags: zone,cluster | rack.aware.assignment.tags: zone,cl
</p>
</div>
</blockquote>
</div>
<div class="section" id="rack-aware-assignment-traffic-cost">
<h4><a class="toc-backref" href="#id36">rack.aware.assignment.traffic_cost</a><a class="headerlink" href="#rack-aware-assignment-traffic-cost" title="Permalink to this headline"></a></h4>
<blockquote>
<div>
<p>
This configuration sets the cost of cross rack traffic. Together with <code class="docutils literal"><span class="pre">rack.aware.assignment.non_overlap_cost</span></code>,
they control whether the optimizer favors minimizing cross rack traffic or minimizing the movement of tasks in the existing assignment. If this config is set to a larger value than <code class="docutils literal"><span class="pre">rack.aware.assignment.non_overlap_cost</span></code>,
the optimizer will try to compute an assignment which minimize the cross rack traffic. Note that the optimizer takes the ratio of these two configs into consideration of favoring maintaining existing assignment or minimizing traffic cost. For example, setting
<code class="docutils literal"><span class="pre">rack.aware.assignment.traffic_cost</span></code> to 10 and <code class="docutils literal"><span class="pre">rack.aware.assignment.non_overlap_cost</span></code> to 1 is more likely to minimize cross rack traffic than setting
<code class="docutils literal"><span class="pre">rack.aware.assignment.traffic_cost</span></code> to 100 and <code class="docutils literal"><span class="pre">rack.aware.assignment.non_overlap_cost</span></code> to 50.
</p>
<p>
The default value is null which means default traffic cost in different assignors will be used. In <code class="docutils literal"><span class="pre">StickyTaskAssignor</span></code>, it has a default value of 1 and <code class="docutils literal"><span class="pre">rack.aware.assignment.non_overlap_cost</span></code>
has a default value of 10. In <code class="docutils literal"><span class="pre">HighAvailabilityTaskAssignor</span></code>, it has a default value of 10 and <code class="docutils literal"><span class="pre">rack.aware.assignment.non_overlap_cost</span></code> has a default value of 1.
</p>
</div>
</blockquote>
</div>
<div class="section" id="max-task-idle-ms">
<span id="streams-developer-guide-max-task-idle-ms"></span><h4><a class="toc-backref" href="#id28">max.task.idle.ms</a><a class="headerlink" href="#max-task-idle-ms" title="Permalink to this headline"></a></h4>

7
docs/streams/upgrade-guide.html
View File

@ -133,6 +133,13 @@
More details about the new config <code>StreamsConfig#TOPOLOGY_OPTIMIZATION</code> can be found in <a href="https://cwiki.apache.org/confluence/display/KAFKA/KIP-295%3A+Add+Streams+Configuration+Allowing+for+Optional+Topology+Optimization">KIP-295</a>.
</p>
<h3><a id="streams_api_changes_360" href="#streams_api_changes_360">Streams API changes in 3.6.0</a></h3>
<p>
Rack aware task assignment was introduced in <a href="https://cwiki.apache.org/confluence/display/KAFKA/KIP-925%3A+Rack+aware+task+assignment+in+Kafka+Streams">KIP-925</a>.
Rack aware task assignment can be enabled for <code>StickyTaskAssignor</code> or <code>HighAvailabilityTaskAssignor</code> to compute task assignments which can minimize cross rack traffic under certain conditions.
For more information, including how it can be enabled and further configured, see the <a href="/{{version}}/documentation/streams/developer-guide/config-streams.html#rack-aware-assignment-strategy"><b>Kafka Streams Developer Guide</b></a>.
</p>
<h3><a id="streams_api_changes_350" href="#streams_api_changes_350">Streams API changes in 3.5.0</a></h3>
<p>
A new state store type, versioned key-value stores, was introduced in

6
docs/upgrade.html
View File

@ -37,6 +37,12 @@
For more detailed information please refer to the Compatibility, Deprecation, and Migration Plan section in
<a href="https://cwiki.apache.org/confluence/display/KAFKA/KIP-902%3A+Upgrade+Zookeeper+to+3.8.1">KIP-902</a>.
</li>
<li>
Kafka Streams has introduced a new task assignor, <code>RackAwareTaskAssignor</code>, for computing task assignments which can minimize
cross rack traffic under certain conditions. It works with existing <code>StickyTaskAssignor</code> and <code>HighAvailabilityTaskAssignor</code>.
See <a href="https://cwiki.apache.org/confluence/display/KAFKA/KIP-925%3A+Rack+aware+task+assignment+in+Kafka+Streams">KIP-925</a>
and <a href="/{{version}}/documentation/streams/developer-guide/config-streams.html#rack-aware-assignment-strategy"><b>Kafka Streams Developer Guide</b></a> for more details.
</li>
</ul>
<h4><a id="upgrade_3_5_0" href="#upgrade_3_5_0">Upgrading to 3.5.0 from any version 0.8.x through 3.4.x</a></h4>

4
streams/src/main/java/org/apache/kafka/streams/StreamsConfig.java
View File

@ -756,8 +756,8 @@ public class StreamsConfig extends AbstractConfig {
public static final String DEFAULT_CLIENT_SUPPLIER_CONFIG = "default.client.supplier";
public static final String DEFAULT_CLIENT_SUPPLIER_DOC = "Client supplier class that implements the <code>org.apache.kafka.streams.KafkaClientSupplier</code> interface.";
public static final String RACK_AWARE_ASSIGNMENT_STRATEGY_NONE = "NONE";
public static final String RACK_AWARE_ASSIGNMENT_STRATEGY_MIN_TRAFFIC = "MIN_TRAFFIC";
public static final String RACK_AWARE_ASSIGNMENT_STRATEGY_NONE = "none";
public static final String RACK_AWARE_ASSIGNMENT_STRATEGY_MIN_TRAFFIC = "min_traffic";
/** {@code } rack.aware.assignment.strategy */
@SuppressWarnings("WeakerAccess")

4
streams/src/test/java/org/apache/kafka/streams/StreamsConfigTest.java
View File

@ -1383,13 +1383,13 @@ public class StreamsConfigTest {
@Test
public void shouldReturnDefaultRackAwareAssignmentConfig() {
final String strategy = streamsConfig.getString(StreamsConfig.RACK_AWARE_ASSIGNMENT_STRATEGY_CONFIG);
assertEquals("NONE", strategy);
assertEquals("none", strategy);
}
@Test
public void shouldtSetMinTrafficRackAwareAssignmentConfig() {
props.put(StreamsConfig.RACK_AWARE_ASSIGNMENT_STRATEGY_CONFIG, StreamsConfig.RACK_AWARE_ASSIGNMENT_STRATEGY_MIN_TRAFFIC);
assertEquals("MIN_TRAFFIC", new StreamsConfig(props).getString(StreamsConfig.RACK_AWARE_ASSIGNMENT_STRATEGY_CONFIG));
assertEquals("min_traffic", new StreamsConfig(props).getString(StreamsConfig.RACK_AWARE_ASSIGNMENT_STRATEGY_CONFIG));
}
@Test