Kafka Producer with Java Example

Learn about Kafka Producer and a Producer Example in Apache Kafka with step by step guide to realize a producer using Java.

What is a Producer in Apache Kafka ?

A producer is an application that is source of data stream. It generates tokens or messages and publish it to one or more topics in the Kafka cluster. The Producer API from Kafka helps to pack the message or token and deliver it to Kafka Server.

Following is a picture demonstrating the working of Producer in Apache Kafka.

Producer in Apache Kafka - Producer Example in Apache Kafka - Apache Kafka Tutorial - www.tutorialkart.com

Producer Application in Apache Kafka

Producer Example in Apache Kafka

In this Apache Kafka Tutorial, we shall learn Producer in Apache Kafka with a Java Example program. Following is a step by step process to write a simple Producer Example in Apache Kafka.

  1. Create Java Project

    Create a new Java Project called KafkaExamples, in your favorite IDE. In this example, we shall use Eclipse. But the process should remain same for most of the other IDEs.

  2. Add Jars to Build Path.

    Add following jars to the Java Project Build Path.
    Note : The jars are available in the lib folder of Apache Kafka download from https://kafka.apache.org/downloads.

    • kafka_2.11-0.11.0.0.jar
    • kafka-clients-0.11.0.0.jar
    • scala-library-2.12.3.jar
    • slf4j-api-1.7.25.jar
    • slf4j-log4j12-1.7.25.jar
    • log4j-1.2.17.jar
  3. New SampleProducer Thread

    Create a new class for a sample Producer, SampleProducer.java, that extends Thread. So that Producer could be launched as a new thread from a machine on demand.

  4. Properties of Kafka Producer

    Provide the information like Kafka Server URL, Kafka Server Port, Producer’s ID (Client ID), Serializers for Key and Value.

    Note : Make sure that the Server URL and PORT are in compliance with the values in //config/server.properties.

  5. Create Kafka Producer with the Properties

    With the properties that have been mentioned above, create a new KafkaProducer.

  6. Synchronous or Asynchronous

    You may send the events from Producer to the Kafka Server synchronously or asynchronously.

  7. Send Messages Synchronously

    You may send messages synchronously (i.e., a new message is sent only after completing the previous message/transaction) as shown below :

    When messages are sent synchronously, they are prone to interruption or stoppage of their transmission to the Kafka Server. InterruptedException and ExecutionException thrown by the send function have to be handled.

  8. Send Messages Asynchronously

    You may send messages asynchronously as shown below :

    When a message is sent asynchronously, you need to provide a CallBack class that implements onCompletion() method which is called when a message is sent successfully and acknowledged by Kafka Server. We have provided a DemoCallBack class here for the call back purpose.

  9. Start Zookeeper and Kafka Cluster

    Navigate to the root of Kafka directory and run each of the following commands in separate terminals to start Zookeeper and Kafka Cluster.

    $ bin/zookeeper-server-start.sh config/zookeeper.properties
    $ bin/kafka-server-start.sh config/server.properties
  10. Start the SampleProducer thread

Complete Java Producer Example in Apache Kafka

Run KafkaProducerDemo.java.
Messages are sent synchronously. You may change the value of isAsync to true to send messages Asynchronously to Kafka Cluster.

Conclusion :

In this Apache Kafka Tutorial, we have learnt about Kafka Producer, and presented a step by step guide to realize a Kafka Producer Application using Java.