diff --git a/docs/ops.html b/docs/ops.html index d2b3e96a121..4b2058cafef 100644 --- a/docs/ops.html +++ b/docs/ops.html @@ -351,16 +351,17 @@ Topic:foo PartitionCount:1 ReplicationFactor:3 Configs: Topic: foo Partition: 0 Leader: 5 Replicas: 5,6,7 Isr: 5,6,7

Limiting Bandwidth Usage during Data Migration

- Kafka lets you apply a throttle to replication traffic, setting an upper bound on the bandwidth used to move replicas from machine to machine. This is useful when rebalancing a cluster, bootstrapping a new broker or adding or removing brokers, as it limits the impact these data-intensive operations will have on users. + Kafka lets you apply a throttle to replication traffic, setting an upper bound on the bandwidth used to move replicas from machine to machine and from disk to disk. This is useful when rebalancing a cluster, adding or removing brokers or adding or removing disks, as it limits the impact these data-intensive operations will have on users.

There are two interfaces that can be used to engage a throttle. The simplest, and safest, is to apply a throttle when invoking the kafka-reassign-partitions.sh, but kafka-configs.sh can also be used to view and alter the throttle values directly.

- So for example, if you were to execute a rebalance, with the below command, it would move partitions at no more than 50MB/s. - 
$ bin/kafka-reassign-partitions.sh --bootstrap-server localhost:9092 --execute --reassignment-json-file bigger-cluster.json --throttle 50000000
+ So for example, if you were to execute a rebalance, with the below command, it would move partitions at no more than 50MB/s between brokers, and at no more than 100MB/s between disks on a broker. + 
$ bin/kafka-reassign-partitions.sh --bootstrap-server localhost:9092 --execute --reassignment-json-file bigger-cluster.json --throttle 50000000 --replica-alter-log-dirs-throttle 100000000
When you execute this script you will see the throttle engage: - 
The inter-broker throttle limit was set to 50000000 B/s
+  
The inter-broker throttle limit was set to 50000000 B/s
+The replica-alter-dir throttle limit was set to 100000000 B/s
 Successfully started partition reassignment for foo1-0

-  
Should you wish to alter the throttle, during a rebalance, say to increase the throughput so it completes quicker, you can do this by re-running the execute command with the --additional option passing the same reassignment-json-file:

+  
Should you wish to alter the throttle, during a rebalance, say to increase the inter-broker throughput so it completes quicker, you can do this by re-running the execute command with the --additional option passing the same reassignment-json-file:

   
$ bin/kafka-reassign-partitions.sh --bootstrap-server localhost:9092 --additional --execute --reassignment-json-file bigger-cluster.json --throttle 700000000
 The inter-broker throttle limit was set to 700000000 B/s

 
@@ -378,12 +379,13 @@ Reassignment of partition [my-topic,0] is completed
 Clearing broker-level throttles on brokers 1,2,3
 Clearing topic-level throttles on topic my-topic
-

The administrator can also validate the assigned configs using the kafka-configs.sh. There are two pairs of throttle - configuration used to manage the throttling process. First pair refers to the throttle value itself. This is configured, at a broker +

The administrator can also validate the assigned configs using the kafka-configs.sh. There are two sets of throttle + configuration used to manage the throttling process. First set refers to the throttle value itself. This is configured, at a broker level, using the dynamic properties: 

leader.replication.throttled.rate
-follower.replication.throttled.rate
+follower.replication.throttled.rate +replica.alter.log.dirs.io.max.bytes.per.second

Then there is the configuration pair of enumerated sets of throttled replicas:

@@ -392,16 +394,16 @@ follower.replication.throttled.replicas

Which are configured per topic.

-

All four config values are automatically assigned by kafka-reassign-partitions.sh (discussed below).

+

All five config values are automatically assigned by kafka-reassign-partitions.sh (discussed below).

To view the throttle limit configuration:

$ bin/kafka-configs.sh --describe --bootstrap-server localhost:9092 --entity-type brokers
-Configs for brokers '2' are leader.replication.throttled.rate=700000000,follower.replication.throttled.rate=700000000
-Configs for brokers '1' are leader.replication.throttled.rate=700000000,follower.replication.throttled.rate=700000000
+Configs for brokers '2' are leader.replication.throttled.rate=700000000,follower.replication.throttled.rate=700000000,replica.alter.log.dirs.io.max.bytes.per.second=1000000000 +Configs for brokers '1' are leader.replication.throttled.rate=700000000,follower.replication.throttled.rate=700000000,replica.alter.log.dirs.io.max.bytes.per.second=1000000000 -

This shows the throttle applied to both leader and follower side of the replication protocol. By default both sides - are assigned the same throttled throughput value.

+

This shows the throttle applied to both leader and follower side of the replication protocol (by default both sides + are assigned the same throttled throughput value), as well as the disk throttle.

To view the list of throttled replicas:

@@ -441,8 +443,7 @@ Configs for topic 'my-topic' are leader.replication.throttled.replicas=1:102,0:1 
kafka.server:type=FetcherLagMetrics,name=ConsumerLag,clientId=([-.\w]+),topic=([-.\w]+),partition=([0-9]+)

The lag should constantly decrease during replication. If the metric does not decrease the administrator should - increase the - throttle throughput as described above.

+ increase the throttle throughput as described above.

Setting quotas