Kafka Consumer with Example Java Application

Apache Kafka Tutorial – Learn about Apache Kafka Consumer with Example Java Application working as a Kafka consumer. Step by step guide to realize a Kafka Consumer is provided for understanding.

What is a Kafka Consumer ?

A Consumer is an application that reads data from Kafka Topics. It subscribes to one or more topics in the Kafka cluster and feeds on tokens or messages from the Kafka Topics.

The connectivity of Consumer to Kafka Cluster is known using Heartbeat. Heartbeat is setup at Consumer to let Zookeeper or Broker Coordinator know if the Consumer is still connected to the Cluster. Absence of heartbeat means the Consumer is no longer connected to the Cluster, in which case the Broker Coordinator has to re-balance the load. Heartbeat is an overhead to the cluster. The interval at which the heartbeat at Consumer should happen is configurable by keeping the data throughput and overhead in consideration.

Also, consumers could be grouped and the consumers in the Consumer Group could share the partitions of the Topics they subscribed to. If there are N partitions in a Topic, N consumers in the Consumer Group, and the group has subscribed to a Topic, each consumer would read data from a partition of the topic. This is just a heads up that Consumers could be in groups. We shall go into details of Consumer Group in out next tutorial.

The Consumer API from Kafka helps to connect to Kafka cluster and consume the data streams.

Following is a picture demonstrating the working of Consumer in Apache Kafka.

Consumer in Apache Kafka - Consumer Example in Apache Kafka - Apache Kafka Tutorial - www.tutorialkart.com

Consumer in Apache Kafka

Kafka Consumer with Example Java Application

Following is a step by step process to write a simple Consumer Example in Apache Kafka.

  1. Create Java Project

    Create a new Java Project called KafkaExamples, in your favorite IDE. In this example, we shall use Eclipse. But the process should remain same for most of the other IDEs.

  2. Add Jars to Build Path.

    Add following jars to the Java Project Build Path.
    Note : The jars are available in the lib folder of Apache Kafka download from https://kafka.apache.org/downloads.

    • kafka_2.11-
    • kafka-clients-
    • scala-library-2.12.3.jar
    • slf4j-api-1.7.25.jar
    • slf4j-log4j12-1.7.25.jar
    • log4j-1.2.17.jar
  3. New SampleConsumer Thread

    Create a new class for a sample Consumer, SampleConsumer.java, that extends Thread. So that Consumer could be launched as a new thread from a machine on demand.

  4. Properties of Kafka Consumer

    Provide the information like Kafka Server URL, Kafka Server Port, Consumer’s ID (Client ID), Serializers for Key and Value.

    Note : Make sure that the Server URL and PORT are in compliance with the values in //config/server.properties.

  5. Create Kafka Consumer with the Properties

    With the properties that have been mentioned above, create a new KafkaConsumer.

  6. Subscribe Consumer to a Topic

    Consumer has to subscribe to a Topic, from which it can receive records.

  7. Fetch Records for the Topic

    Fetch Records for the Topic that the Consumer has been subscribed to, using poll(long interval).
    interval is the time period over which, the records are aggregated

  8. Consume the records

    You may consumer the records as per your need or use case. Here, in this tutorial, we shall print those messages to console output.

  9. Start Zookeeper and Kafka Cluster

    Navigate to the root of Kafka directory and run each of the following commands in separate terminals to start Zookeeper and Kafka Cluster.

    $ bin/zookeeper-server-start.sh config/zookeeper.properties
    $ bin/kafka-server-start.sh config/server.properties
  10. Start the Kafka Producer

    Well! There has to be a Producer of records for the Consumer to feed on. Start the Kafka Producer by following Kafka Producer with Java Example.
    Also note that, if you are changing the Topic name, make sure you use the same topic name for the Kafka Producer Example and Kafka Consumer Example Java Applications.

  11. Start the SampleConsumer thread

Example Java Application that works as Kafka Consumer

Run KafkaConsumerDemo.java.


Conclusion :

In this Apache Kafka Tutorial – Kafka Consumer with Example Java Application, we have learnt about Kafka Consumer, and presented a step by step guide to realize a Kafka Consumer Application using Java.