adee6f0cc1
Implement request handling for the updated versions of the KRaft RPCs (Fetch, FetchSnapshot, Vote, BeginQuorumEpoch and EndQuorumEpoch). This doesn't add support for KRaft replicas to send the new version of the KRaft RPCs. That will be implemented in KAFKA-16529. All of the RPCs responses were extended to include the leader's endpoint for the listener of the channel used in the request. EpochState was extended to include the leader's endpoint information but only the FollowerState and LeaderState know the leader id and its endpoint(s). For the Fetch request, the replica directory id was added. The leader now tracks the follower's log end offset using both the replica id and replica directory id. For the FetchSnapshot request, the replica directory id was added. This is not used by the KRaft leader and it is there for consistency with Fetch and for help debugging. For the Vote request, the replica key for both the voter (destination) and the candidate (source) were added. The voter key is checked for consistency. The candidate key is persisted when the vote is granted. For the BeginQuorumEpoch request, all of the leader's endpoints are included. This is needed so that the voters can return the leader's endpoint for all of the supported listeners. For the EndQuorumEpoch request, all of the leader's endpoints are included. This is needed so that the voters can return the leader's endpoint for all of the supported listeners. The successor list has been extended to include the directory id. Receiving voters can use the entire replica key when searching their position in the successor list. Updated the existing test in KafkaRaftClientTest and KafkaRaftClientSnapshotTest to execute using both the old version and new version of the RPCs. Reviewers: Luke Chen <showuon@gmail.com>, Colin P. McCabe <cmccabe@apache.org> |
||
|---|---|---|
| .. | ||
| bin | ||
| config | ||
| src | ||
| .gitignore | ||
| README.md | ||
README.md
KRaft (Kafka Raft)
KRaft (Kafka Raft) is a protocol based on the Raft Consensus Protocol tailored for Apache Kafka.
This is used by Apache Kafka in the KRaft (Kafka Raft Metadata) mode. We also have a standalone test server which can be used for performance testing. We describe the details to set this up below.
Run Single Quorum
bin/test-kraft-server-start.sh --config config/kraft.properties
Run Multi Node Quorum
Create 3 separate KRaft quorum properties as the following:
cat << EOF >> config/kraft-quorum-1.properties
node.id=1
listeners=PLAINTEXT://localhost:9092
controller.listener.names=PLAINTEXT
controller.quorum.voters=1@localhost:9092,2@localhost:9093,3@localhost:9094
log.dirs=/tmp/kraft-logs-1
EOF
cat << EOF >> config/kraft-quorum-2.properties
node.id=2
listeners=PLAINTEXT://localhost:9093
controller.listener.names=PLAINTEXT
controller.quorum.voters=1@localhost:9092,2@localhost:9093,3@localhost:9094
log.dirs=/tmp/kraft-logs-2
EOF
cat << EOF >> config/kraft-quorum-3.properties
node.id=3
listeners=PLAINTEXT://localhost:9094
controller.listener.names=PLAINTEXT
controller.quorum.voters=1@localhost:9092,2@localhost:9093,3@localhost:9094
log.dirs=/tmp/kraft-logs-3
EOF
Open up 3 separate terminals, and run individual commands:
bin/test-kraft-server-start.sh --config config/kraft-quorum-1.properties
bin/test-kraft-server-start.sh --config config/kraft-quorum-2.properties
bin/test-kraft-server-start.sh --config config/kraft-quorum-3.properties
Once a leader is elected, it will begin writing to an internal
__raft_performance_test topic with a steady workload of random data.
You can control the workload using the --throughput and --record-size
arguments passed to test-kraft-server-start.sh.