Kafka Producer – Example using Java

Learn how a Kafka producer works and how to write a simple Kafka Producer example in Java. This tutorial explains the producer role in Apache Kafka, the main producer configuration properties, synchronous and asynchronous send styles, and a complete Java producer program that publishes records to a Kafka topic.

What is a Kafka Producer in Apache Kafka?

A Kafka producer is an application that writes data to one or more Kafka topics. The data may come from a web application, log collector, IoT device, payment system, analytics job, or any service that has events to publish. The producer uses the Kafka Producer API to package each event as a record and send it to a Kafka broker.

Each record sent by a Kafka producer usually contains a topic name, an optional key, a value, optional headers, and a timestamp. Kafka stores the record in a partition of the target topic. If a key is provided, Kafka uses it to decide the partition, so records with the same key can be routed consistently to the same partition.

Following is a picture demonstrating the working of Producer in Apache Kafka.

How a Kafka Producer Sends Records to a Topic

When a Java producer sends a record, the Kafka client serializes the key and value, decides the topic partition, batches records when possible, and sends the batch to the appropriate Kafka broker. The broker appends the record to the partition log and returns metadata such as topic, partition, and offset after the record is acknowledged.

• Topic: The logical name to which the producer writes records.
• Key: Optional value used for partition selection and ordering for the same key within a partition.
• Value: The actual message payload sent by the producer.
• Serializer: Converts Java objects such as Integer or String into bytes before sending them to Kafka.
• Acknowledgment: Defines when the broker response is considered successful by the producer.

Kafka Producer Example in Java: Tutorial Plan

In this Apache Kafka Tutorial, we shall learn Producer in Apache Kafka with a Java Example program. Following is a step by step process to write a simple Producer Example in Apache Kafka.

• Create a Java project for the Kafka producer program.
• Add Kafka client libraries to the project.
• Configure producer properties such as bootstrap server, client id, and serializers.
• Create a KafkaProducer instance.
• Send records synchronously or asynchronously.
• Run Kafka locally and start the Java producer.

Kafka Producer Java Project Setup

Create Java Project for Kafka Producer Example

Create a new Java Project called KafkaExamples, in your favorite IDE. In this example, we shall use Eclipse. But the process should remain same for most of the other IDEs.

Add Kafka Producer Client Jars to Build Path

Add following jars to the Java Project Build Path.Note : The jars are available in the lib folder of Apache Kafka download from [https://kafka.apache.org/downloads].

• kafka_2.11-0.11.0.0.jar
• kafka-clients-0.11.0.0.jar
• scala-library-2.12.3.jar
• slf4j-api-1.7.25.jar
• slf4j-log4j12-1.7.25.jar
• log4j-1.2.17.jar

For new Java projects, it is usually easier to use a build tool such as Maven or Gradle instead of manually adding jar files. The producer code in this tutorial uses the Kafka Java client API, so the main dependency is kafka-clients. Choose a version that matches the Kafka version used in your environment.

<dependency>
    <groupId>org.apache.kafka</groupId>
    <artifactId>kafka-clients</artifactId>
    <version>3.7.0</version>
</dependency>

The rest of this page keeps the original jar-based setup and Java classes so that the example remains easy to follow in a basic IDE project.

Kafka Producer Properties for Java Client

New SampleProducer Thread

Create a new class for a sample Producer, SampleProducer.java, that extends Thread. So that Producer could be launched as a new thread from a machine on demand.

public class SampleProducer extends Thread {
    . . .
}

Properties of Kafka Producer

Provide the information like Kafka Server URL, Kafka Server Port, Producer’s ID (Client ID), Serializers for Key and Value.

Properties properties = new Properties();
properties.put("bootstrap.servers", KAFKA_SERVER_URL + ":" + KAFKA_SERVER_PORT);
properties.put("client.id", "DemoProducer");
properties.put("key.serializer", "org.apache.kafka.common.serialization.IntegerSerializer");
properties.put("value.serializer", "org.apache.kafka.common.serialization.StringSerializer");

Note : Make sure that the Server URL and PORT are in compliance with the values in //config/server.properties.

The important Kafka producer properties in this example are explained below.

Producer property	Purpose in this Java Kafka producer example
bootstrap.servers	Comma-separated list of Kafka broker addresses used by the producer to connect to the Kafka cluster.
client.id	Logical name for the producer client. It helps identify the producer in logs and metrics.
key.serializer	Serializer class used to convert the record key into bytes.
value.serializer	Serializer class used to convert the record value into bytes.

In production applications, you may also configure properties such as acks, retries, delivery.timeout.ms, batch.size, linger.ms, and enable.idempotence, depending on the required reliability and throughput.

Create KafkaProducer with the Producer Properties

With the properties that have been mentioned above, create a new KafkaProducer.

KafkaProducer producer = new KafkaProducer<>(properties);

Synchronous and Asynchronous Send in Kafka Producer

Synchronous or Asynchronous

You may send the events from Producer to the Kafka Server synchronously or asynchronously.

Send Messages Synchronously from Java Kafka Producer

You may send messages synchronously (i.e., a new message is sent only after completing the previous message/transaction) as shown below :

try {
    producer.send(new ProducerRecord<>(topic,
            messageNo,
            messageStr)).get();
} catch (InterruptedException | ExecutionException e) {
    e.printStackTrace();
    // handle the exception
}

When messages are sent synchronously, they are prone to interruption or stoppage of their transmission to the Kafka Server. InterruptedException and ExecutionException thrown by the send function have to be handled.

Synchronous send is simple to understand because the call waits for the broker response. It is useful for learning, debugging, or cases where the application must confirm the result before continuing. However, it can reduce throughput because each send waits for completion before the next send is processed.

Send Messages Asynchronously from Java Kafka Producer

You may send messages asynchronously as shown below :

producer.send(new ProducerRecord<>(topic,
        messageNo,
        messageStr), new DemoCallBack(startTime, messageNo, messageStr));

When a message is sent asynchronously, you need to provide a CallBack class that implements onCompletion() method which is called when a message is sent successfully and acknowledged by Kafka Server. We have provided a DemoCallBack class here for the call back purpose.

Asynchronous send is the common choice for higher-throughput producer applications. The producer does not block for every record. Instead, it continues sending records and handles the result in the callback when Kafka responds.

Start Kafka Locally Before Running Java Producer

Start Zookeeper and Kafka Cluster

Navigate to the root of Kafka directory and run each of the following commands in separate terminals to start Zookeeper and Kafka Cluster.

$ bin/zookeeper-server-start.sh config/zookeeper.properties
$ bin/kafka-server-start.sh config/server.properties

The commands above match older Kafka distributions that use ZooKeeper. In newer Kafka setups, you may use Kafka in KRaft mode, where ZooKeeper is not required. Follow the startup commands for the Kafka version you downloaded. Before running the Java producer, make sure a broker is listening on localhost:9092 and the target topic exists.

You can create the sample topic from the Kafka installation directory using a command similar to the following.

bin/kafka-topics.sh --create \
  --topic testTopic \
  --bootstrap-server localhost:9092 \
  --partitions 1 \
  --replication-factor 1

To verify the records produced by the Java program, you can start a console consumer for the same topic.

bin/kafka-console-consumer.sh \
  --topic testTopic \
  --from-beginning \
  --bootstrap-server localhost:9092

Run the SampleProducer Thread

Start the SampleProducer thread

SampleProducer producerThread = new SampleProducer(TOPIC, isAsync);
producerThread.start();

Complete Java Kafka Producer Example

Complete Java Producer Example in Apache Kafka

SampleProducer.java

import org.apache.kafka.clients.producer.Callback;
import org.apache.kafka.clients.producer.KafkaProducer;
import org.apache.kafka.clients.producer.ProducerRecord;
import org.apache.kafka.clients.producer.RecordMetadata;

import java.util.Properties;
import java.util.concurrent.ExecutionException;

/**
 * Producer Example in Apache Kafka
 * @author www.tutorialkart.com
 */
public class SampleProducer extends Thread {
	private final KafkaProducer<Integer, String> producer;
	private final String topic;
	private final Boolean isAsync;

	public static final String KAFKA_SERVER_URL = "localhost";
	public static final int KAFKA_SERVER_PORT = 9092;
	public static final String CLIENT_ID = "SampleProducer";

	public SampleProducer(String topic, Boolean isAsync) {
		Properties properties = new Properties();
		properties.put("bootstrap.servers", KAFKA_SERVER_URL + ":" + KAFKA_SERVER_PORT);
		properties.put("client.id", CLIENT_ID);
		properties.put("key.serializer", "org.apache.kafka.common.serialization.IntegerSerializer");
		properties.put("value.serializer", "org.apache.kafka.common.serialization.StringSerializer");
		producer = new KafkaProducer<>(properties);
		this.topic = topic;
		this.isAsync = isAsync;
	}

	public void run() {
		int messageNo = 1;
		while (true) {
			String messageStr = "Message_" + messageNo;
			long startTime = System.currentTimeMillis();
			if (isAsync) { // Send asynchronously
				producer.send(new ProducerRecord<>(topic,
						messageNo,
						messageStr), new DemoCallBack(startTime, messageNo, messageStr));
			} else { // Send synchronously
				try {
					producer.send(new ProducerRecord<>(topic,
							messageNo,
							messageStr)).get();
					System.out.println("Sent message: (" + messageNo + ", " + messageStr + ")");
				} catch (InterruptedException | ExecutionException e) {
					e.printStackTrace();
					// handle the exception
				}
			}
			++messageNo;
		}
	}
}

class DemoCallBack implements Callback {

	private final long startTime;
	private final int key;
	private final String message;

	public DemoCallBack(long startTime, int key, String message) {
		this.startTime = startTime;
		this.key = key;
		this.message = message;
	}

	/**
	 * onCompletion method will be called when the record sent to the Kafka Server has been acknowledged.
	 * 
	 * @param metadata  The metadata contains the partition and offset of the record. Null if an error occurred.
	 * @param exception The exception thrown during processing of this record. Null if no error occurred.
	 */
	public void onCompletion(RecordMetadata metadata, Exception exception) {
		long elapsedTime = System.currentTimeMillis() - startTime;
		if (metadata != null) {
			System.out.println(
					"message(" + key + ", " + message + ") sent to partition(" + metadata.partition() +
					"), " +
					"offset(" + metadata.offset() + ") in " + elapsedTime + " ms");
		} else {
			exception.printStackTrace();
		}
	}
}

KafkaProducerDemo.java

public class KafkaProducerDemo {
	public static final String TOPIC = "testTopic";
	
    public static void main(String[] args) {
        boolean isAsync = false;
        SampleProducer producerThread = new SampleProducer(TOPIC, isAsync);
        // start the producer
        producerThread.start();

    }
}

Run KafkaProducerDemo.java.

Sent message: (1, Message_1)
Sent message: (2, Message_2)
Sent message: (3, Message_3)
Sent message: (4, Message_4)
Sent message: (5, Message_5)
Sent message: (6, Message_6)
Sent message: (7, Message_7)
Sent message: (8, Message_8)
Sent message: (9, Message_9)
Sent message: (10, Message_10)
Sent message: (11, Message_11)
Sent message: (12, Message_12)

Messages are sent synchronously. You may change the value of isAsync to true to send messages Asynchronously to Kafka Cluster.

Kafka Producer Example Output and Verification

In synchronous mode, the program prints a line after each record is sent successfully. In asynchronous mode, the callback prints the topic partition and offset returned by Kafka. The offset is useful because it confirms where the record was appended in the partition log.

If the console consumer is running for testTopic, it should receive values such as the following.

Message_1
Message_2
Message_3
Message_4
Message_5

Common Kafka Producer Errors in Java Example

If the Java Kafka producer does not send records, check these points first.

• Broker is not running: Make sure Kafka is started and reachable at localhost:9092.
• Topic is missing: Create testTopic before running the producer, or enable topic auto-creation only in development environments.
• Wrong serializer: The key is an Integer and the value is a String in this example, so the configured serializers must match those Java types.
• Port mismatch: Check server.properties and confirm that the broker listener matches the producer bootstrap.servers value.
• Program runs forever: The sample producer uses while (true) for demonstration. In a real application, add a stopping condition and close the producer gracefully.

Kafka Producer Java Example Best Practices

The sample program is intentionally simple. For a real Java application, consider the following producer practices.

• Close the producer using producer.close() when the application shuts down.
• Use callbacks for asynchronous sends so that failed records are logged or handled.
• Configure retries and delivery timeout according to the application requirement.
• Use meaningful keys when records for the same entity should go to the same partition.
• Avoid creating a new KafkaProducer for every record; reuse a producer instance where appropriate.
• Do not ignore serialization errors, connection errors, or callback exceptions in production code.

Kafka Producer in Java FAQ

Can Kafka be used with Java?

Yes. Kafka provides official Java client libraries for producer, consumer, admin, and streams use cases. Java is commonly used with Kafka because the Kafka client API is mature and works directly with the Kafka protocol.

How do I start a Kafka producer?

Start a Kafka broker, create a topic, configure producer properties such as bootstrap.servers and serializers, create a KafkaProducer object, and call send() with a ProducerRecord.

How does a producer work in Kafka?

A Kafka producer serializes records, chooses the target partition, batches records when possible, sends them to the broker that owns the partition, and receives acknowledgement metadata after Kafka accepts the record.

Should I send Kafka records synchronously or asynchronously?

Use synchronous send when you need simple control flow or are learning the API. Use asynchronous send with callbacks for most high-throughput producer applications because it avoids waiting for each record before sending the next one.

Is Kafka better with Java or Python?

Kafka can be used with both Java and Python. Java is the natural choice when you want to use the official Kafka client directly and build JVM-based services. Python is useful for scripts, data tasks, and applications that already use a Python stack. The better choice depends on the application, deployment environment, and team skill set.

Editorial QA Checklist for Kafka Producer Java Tutorial

• Does the article clearly explain what a Kafka producer is before showing Java code?
• Are bootstrap.servers, client.id, and serializer properties explained in the Kafka producer context?
• Does the tutorial distinguish synchronous send from asynchronous send with callback?
• Are the local Kafka startup and topic creation steps clear for a beginner?
• Does the article warn that the sample loop runs continuously and should be controlled in real applications?

Conclusion :

In this Apache Kafka Tutorial – Kafka Producer Example, we have learnt about Kafka Producer, and presented a step by step guide to realize a Kafka Producer Application using Java.