Kafka Multi-Broker Cluster

Kafka Multi-Broker Cluster

Kafka Multi-Broker Cluster means running more than one Kafka broker in the same cluster so that topic partitions can be distributed, replicated, and kept available when one broker is stopped. In this tutorial, we shall set up a three-broker Kafka cluster on a single local machine for learning and testing.

This tutorial keeps the ZooKeeper-based commands used by older Kafka installations. If your Kafka version uses KRaft mode or your command-line tools no longer support the --zookeeper option, use the --bootstrap-server alternatives shown in the compatibility notes below. For modern reference, see the Apache Kafka Quickstart and the Confluent multi-broker cluster tutorial.

Kafka three-broker local cluster layout

We shall use three broker instances, three listener ports, and three separate log directories. The default broker uses config/server.properties, and two additional brokers use new configuration files.

Broker instance	Configuration file	Listener port	Log directory
Broker 0	config/server.properties	9092	Default Kafka log directory
Broker 1	config/server-1.properties	9093	/tmp/kafka-logs-1
Broker 2	config/server-2.properties	9094	/tmp/kafka-logs-2

For a local tutorial, all three brokers may run on the same computer. In a production Kafka cluster, brokers usually run on separate servers or containers so that a machine-level failure does not stop multiple brokers at once.

Prepare Kafka broker configuration files

We need to create config files, one for each broker instance.

As we have planned a three-node Kafka cluster, and there is already a default server.properties file, we shall create two more config files.

To create config files for each broker, follow these steps.

1. Navigate to the Kafka root directory.

2. Open a Terminal from this location.

3. Execute the following copy command for Ubuntu or any Linux based machine.

$ cp config/server.properties config/server-1.properties
$ cp config/server.properties config/server-2.properties

Note : All of the commands mentioned should be run from Kafka root directory.

For Windows, use this copy command

$ copy config/server.properties config/server-1.properties
$ copy config/server.properties config/server-2.properties

4. Once you copy the config files, copy paste the following content to the config files respectively.

config/server-1.properties

broker.id=1
listeners=PLAINTEXT://:9093
log.dir=/tmp/kafka-logs-1
zookeeper.connect=localhost:2181

config/server-2.properties

broker.id=2
listeners=PLAINTEXT://:9094
log.dir=/tmp/kafka-logs-2
zookeeper.connect=localhost:2181

broker.id uniquely identifies a Kafka broker instance in the cluster. Do not reuse the same broker id for two brokers that share the same cluster.

By default Kafka broker starts at port 9092. For other two broker instances, we have provided the ports 9093 and 9094.

The log.dir value is provided because all Kafka broker instances are running on the same local machine. Separate log directories prevent the brokers from writing their data files into the same location.

Once the config files are ready, we need bring our Zookeeper and Broker instances up.

Start ZooKeeper for the Kafka broker cluster

Run the following command

$ bin/zookeeper-server-start.sh config/zookeeper.properties

$ bin/zookeeper-server-start.sh config/zookeeper.properties
[2017-09-30 12:21:18,920] INFO Reading configuration from: config/zookeeper.properties (org.apache.zookeeper.server.quorum.QuorumPeerConfig)
[2017-09-30 12:21:18,960] INFO autopurge.snapRetainCount set to 3 (org.apache.zookeeper.server.DatadirCleanupManager)
[2017-09-30 12:21:18,960] INFO autopurge.purgeInterval set to 0 (org.apache.zookeeper.server.DatadirCleanupManager)
[2017-09-30 12:21:18,960] INFO Purge task is not scheduled. (org.apache.zookeeper.server.DatadirCleanupManager)

Keep this terminal window open. ZooKeeper must be running before the ZooKeeper-based Kafka broker instances are started.

Start three Kafka broker instances on separate ports

Once the Zookeeper is up and running, Open three different terminal windows and run the following commands in each of them, to start Kafka Broker Instances.

$ bin/kafka-server-start.sh config/server.properties
$ bin/kafka-server-start.sh config/server-1.properties
$ bin/kafka-server-start.sh config/server-2.properties

Each broker should remain active in its own terminal. If one terminal exits immediately, check the last few log lines for a duplicate broker id, an occupied port, or a log directory that is already locked by another broker.

Create a Kafka topic with replication factor 3

Create a topic with replication factor of three, so that we can demonstrate the replication of topic partition across the three nodes.

$ bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 3 --partitions 1 --topic my-topic

$ bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 3 --partitions 1 --topic my-topic
Created topic "my-topic".

The replication factor is set to 3 because the local Kafka cluster has three brokers. A replication factor greater than the number of available brokers will fail because Kafka cannot place more replicas than the broker count for the same partition.

For newer Kafka command-line tools that use broker bootstrap addresses instead of ZooKeeper, the equivalent command is:

$ bin/kafka-topics.sh --create --bootstrap-server localhost:9092 --replication-factor 3 --partitions 1 --topic my-topic

Describe the Kafka topic leader, replicas, and ISR

There are three broker instances running and to know which Kafka broker is doing what with the Kafka topic that we created in the earlier step, run the following command.

$ bin/kafka-topics.sh --describe --zookeeper localhost:2181 --topic my-topic

$ bin/kafka-topics.sh --describe --zookeeper localhost:2181 --topic my-topic
Topic:my-topic    PartitionCount:1 ReplicationFactor:3 Configs:
    Topic: my-topic    Partition: 0 Leader: 1 Replicas: 1,2,0 Isr: 1,2,0

For newer Kafka command-line tools, use --bootstrap-server to describe the topic:

$ bin/kafka-topics.sh --describe --bootstrap-server localhost:9092 --topic my-topic

Leader: 1 is the broker instance responsible for all reads and writes for the given partition. Reads and Writes are done by Kafka Consumers and Kafka Producers respectively.

Replicas: 1,2,0 means broker instances 1, 2 and 0 have copies of this partition. One replica is the leader, and the remaining replicas follow the leader.

ISR: 1,2,0 means broker instances 1, 2 and 0 are in-sync replicas. ISR is important because it shows which replicas are currently caught up enough to be considered healthy for this partition.

Now Kafka Produces may send messages to the Kafka topic, my-topic and Kafka Consumers may subscribe to the Kafka Topic.

Test Kafka multi-broker fault tolerance by stopping the leader

We know the leader (broker instance 1) for the Kafka Topic, my-topic. Lets kill it and see what zookeeper does when the leader goes down.

Find the id of broker-1 instance.

kafkauser@tutorialkart:/home/kafkauser/kafka/kafka_2.11-0.11.0.0$ ps -aef|grep server-1.properties
tutorialkart 5058 4749 1 14:14 pts/16 00:01:23 /usr/lib/jvm/default-java/jre/bin/java -Xmx1G -Xms1G

The first number is the id we need. Kill the broker instance using this id.

$ kill -9 5058

Once the instance is killed, describe the topic to check what happened to the Kafka topic, my-topic

arjun@arjun-VPCEH26EN:/media/arjun/0AB650F1B650DF2F/SOFTs_/kafka/kafka_2.11-0.11.0.0$ bin/kafka-topics.sh --describe --zookeeper localhost:2181 --topic my-topic Topic:my-topic    PartitionCount:1 ReplicationFactor:3 Configs:
    Topic: my-topic    Partition: 0 Leader: 2 Replicas: 1,2,0 Isr: 2,0

When the broker instance 1 went down intentionally or unintentionally, zookeeper has elected another node as leader. In this case it elected broker instance 2 as leader.

Notice that the replica list is still 1,2,0, but the ISR changed to 2,0. Broker 1 is no longer in sync because it is down. When broker 1 is started again and catches up with the leader, it can return to the ISR.

Produce and consume messages after leader failover

After the leader changes, test the Kafka multi-broker cluster by producing and consuming a few messages. This confirms that clients can still use the topic while another in-sync broker becomes the leader.

$ bin/kafka-console-producer.sh --broker-list localhost:9092,localhost:9093,localhost:9094 --topic my-topic

$ bin/kafka-console-consumer.sh --bootstrap-server localhost:9092,localhost:9093,localhost:9094 --topic my-topic --from-beginning

In newer Kafka clients, --bootstrap-server is preferred for producer and consumer commands. You may provide more than one broker address so the client can discover the cluster even if one broker is not available.

Kafka multi-broker cluster troubleshooting notes

If the cluster does not start or the topic is not created, check these common causes before changing the topic command.

• Port already in use: each broker must use a different listener port such as 9092, 9093, and 9094.
• Duplicate broker id: every broker configuration must have a unique broker.id.
• Shared log directory: when brokers run on the same local machine, use separate log.dir values.
• Replication factor error: the topic replication factor cannot be greater than the number of active brokers.
• ZooKeeper option not recognized: use the --bootstrap-server examples if your Kafka tools no longer accept --zookeeper.

KRaft note for modern Kafka multi-broker clusters

Kafka has moved toward KRaft-based metadata management in modern installations, which means ZooKeeper is not used for new cluster metadata in those setups. The idea of a multi-broker cluster remains the same: topics are split into partitions, partitions have leaders, and replicas are kept on other brokers. The main difference is how cluster metadata and controller election are managed.

If you are creating a fresh Kafka cluster with a current distribution, compare this ZooKeeper-based tutorial with the official KRaft documentation before using it in a new environment. The commands in this page are useful for understanding broker ids, listeners, replication factor, leader election, replicas, and ISR.

Kafka multi-broker cluster QA checklist

• Confirm that the article clearly says this setup runs three Kafka brokers on one local machine for learning.
• Check that broker ids 0, 1, and 2 are unique and mapped to separate listener ports.
• Check that the topic uses replication factor 3 only after all three brokers are running.
• Confirm that the explanation of Leader, Replicas, and ISR matches the sample topic description output.
• Verify that the ZooKeeper-based commands are not presented as the only option for newer Kafka distributions.

Kafka multi-broker cluster FAQ

Can I run a Kafka multi-broker cluster on one machine?

Yes. You can run multiple Kafka broker instances on one machine for local learning if each broker uses a unique broker id, listener port, and log directory. This is not the same as production fault tolerance because one machine failure can stop all local brokers.

Why does a Kafka topic with replication factor 3 need three brokers?

A replication factor of 3 means Kafka must place three copies of each partition on brokers in the cluster. For a single partition, Kafka needs three active brokers to place those three replicas properly.

What does ISR mean in a Kafka topic description?

ISR means in-sync replicas. These are the replicas that are currently caught up with the partition leader. If a broker stops, it is removed from ISR until it comes back and catches up.

Should I use ZooKeeper or KRaft for a new Kafka multi-broker cluster?

For older Kafka learning environments, ZooKeeper-based examples may still be useful. For a new Kafka setup, check your Kafka distribution documentation because modern Kafka deployments commonly use KRaft mode and --bootstrap-server based tooling.

What happens when the leader broker goes down in this tutorial?

Kafka elects another in-sync replica as the leader for the partition. In the sample output, broker 2 becomes the leader and broker 1 is removed from ISR while it is unavailable.

Kafka multi-broker cluster summary

In this Apache Kafka Tutorial, we have learnt to build a Kafka Multi-Broker Cluster on a local machine, create a replicated Kafka topic, describe its leader and replicas, and observe how leader failover works when one broker goes down.