History

Jason Gustafson 7ac06065f1 KAFKA-12161; Support raft observers with optional id (#9871 ) We would like to be able to use `KafkaRaftClient` for tooling/debugging use cases. For this, we need the localId to be optional so that the client can be used more like a consumer. This is already supported in the `Fetch` protocol by setting `replicaId=-1`, which the Raft implementation checks for. We just need to alter `QuorumState` so that the `localId` is optional. The main benefit of doing this is that it saves tools the need to generate an arbitrary id (which might cause conflicts given limited Int32 space) and it lets the leader avoid any local state for these observers (such as `ReplicaState` inside `LeaderState`). Reviewers: Ismael Juma <ismael@juma.me.uk>, Boyang Chen <boyang@confluent.io>		2021-01-15 14:10:17 -08:00
..
bin	KAFKA-10601; Add support for append linger to Raft implementation (#9418 )	2020-10-27 12:10:13 -07:00
config	KAFKA-10492; Core Kafka Raft Implementation (KIP-595) (#9130 )	2020-09-22 11:32:44 -07:00
src	KAFKA-12161; Support raft observers with optional id (#9871 )	2021-01-15 14:10:17 -08:00
README.md	KAFKA-10601; Add support for append linger to Raft implementation (#9418 )	2020-10-27 12:10:13 -07:00

README.md

Kafka Raft

Kafka Raft is a sub module of Apache Kafka which features a tailored version of Raft Consensus Protocol.

Eventually this module will be integrated into the Kafka server. For now, we have a standalone test server which can be used for performance testing. Below we describe the details to set this up.

Run Single Quorum

bin/test-raft-server-start.sh config/raft.properties

Run Multi Node Quorum

Create 3 separate raft quorum properties as the following (note that the zookeeper.connect config is required, but unused):

cat << EOF >> config/raft-quorum-1.properties

broker.id=1
listeners=PLAINTEXT://localhost:9092
quorum.voters=1@localhost:9092,2@localhost:9093,3@localhost:9094
log.dirs=/tmp/raft-logs-1

zookeeper.connect=localhost:2181
EOF

cat << EOF >> config/raft-quorum-2.properties

broker.id=2
listeners=PLAINTEXT://localhost:9093
quorum.voters=1@localhost:9092,2@localhost:9093,3@localhost:9094
log.dirs=/tmp/raft-logs-2

zookeeper.connect=localhost:2181
EOF

cat << EOF >> config/raft-quorum-3.properties

broker.id=3
listeners=PLAINTEXT://localhost:9094
quorum.voters=1@localhost:9092,2@localhost:9093,3@localhost:9094
log.dirs=/tmp/raft-logs-3

zookeeper.connect=localhost:2181
EOF

Open up 3 separate terminals, and run individual commands:

bin/test-raft-server-start.sh --config config/raft-quorum-1.properties
bin/test-raft-server-start.sh --config config/raft-quorum-2.properties
bin/test-raft-server-start.sh --config config/raft-quorum-3.properties

Once a leader is elected, it will begin writing to an internal __cluster_metadata topic with a steady workload of random data. You can control the workload using the --throughput and --record-size arguments passed to test-raft-server-start.sh.