Kafka Interview Questions

Apache Kafka interview questions commonly cover brokers, topics, partitions, replication, producer acknowledgements, consumer groups, offsets, delivery semantics, stream processing, and operational troubleshooting. The questions below move from fundamentals to scenario-based topics so that both freshers and experienced developers can review the concepts expected in Kafka interviews.

Kafka Interview Questions and Answers

1. What is Apache Kafka?

Apache Kafka is a distributed event-streaming platform. Applications use it to publish, store, read, and process streams of records. Kafka is commonly used for event-driven systems, log aggregation, data pipelines, stream processing, and communication between services.

2. What are the basic components of a Kafka system?

• Cluster: A group of Kafka brokers working together.
• Broker: A Kafka server that stores partitions and handles client requests.
• Topic: A named stream or category of records.
• Partition: An ordered shard of a topic.
• Producer: An application that publishes records.
• Consumer: An application that reads records.
• Consumer group: A set of consumers that divide partitions among themselves.
• Kafka Connect: A framework for moving data between Kafka and external systems.
• Kafka Streams: A Java library for processing records stored in Kafka.

3. What is a Kafka topic?

A Kafka topic is a named stream of records. Producers write records to a topic, and consumers read records from it. A topic is divided into one or more partitions to support parallel processing and distribution across brokers.

4. What is a Kafka partition?

A partition is an append-only, ordered log within a topic. Each record in a partition receives an offset. Kafka guarantees record order within a single partition, but it does not provide one global order across every partition in a topic.

5. Why are partitions important in Kafka?

Partitions allow Kafka to distribute data across brokers and process records in parallel. They are the main unit of storage, replication, and consumer-group assignment. Increasing the number of partitions can increase parallelism, although it also adds operational overhead and can affect key-based ordering when records are redistributed.

6. What is an offset in Kafka?

An offset is the sequential position of a record within a partition. Consumers track offsets to know which records they have processed. The same offset value can exist in different partitions, so a record position is identified by topic, partition, and offset together.

7. What is a Kafka broker?

A broker is a Kafka server. It stores topic partitions, accepts writes from producers, serves reads to consumers, replicates partition data, and participates in cluster coordination. A Kafka cluster normally contains multiple brokers for scalability and fault tolerance.

8. Where is Kafka cluster metadata stored?

Modern Kafka clusters can use Kafka’s built-in KRaft metadata quorum, where dedicated or combined controller nodes manage cluster metadata using the Raft consensus protocol. Older Kafka deployments used Apache ZooKeeper for metadata and controller coordination. An interview answer should mention both approaches and clarify that ZooKeeper applies to legacy deployments.

9. What is a partition leader and a follower replica?

Each replicated partition has one leader and one or more follower replicas. Producers and consumers normally communicate with the leader. Followers copy records from the leader. If the leader fails, an eligible in-sync replica can be elected as the new leader.

10. What is the replication factor in Kafka?

The replication factor is the number of copies maintained for each partition. A replication factor of three means one leader replica and two follower replicas are stored across brokers, subject to the cluster’s broker count and replica-placement rules.

11. What is an in-sync replica?

An in-sync replica, or ISR, is a replica that is sufficiently caught up with the partition leader according to Kafka’s replication rules. Kafka normally elects a new leader from the ISR set to reduce the risk of losing acknowledged records.

12. What are the main Kafka APIs?

• Producer API: Publishes records to Kafka topics.
• Consumer API: Reads records from Kafka topics.
• Admin API: Manages and inspects topics, configurations, consumer groups, and other cluster resources.
• Kafka Streams API: Builds stream-processing applications over Kafka topics.
• Kafka Connect API: Develops and runs source and sink connectors for external systems.

13. How does a Kafka producer choose a partition?

A producer can specify a partition directly, provide a key that is processed by a partitioner, or send a record without a key and allow the producer’s partitioning strategy to select a partition. Records with the same key are generally routed to the same partition while the partition count remains unchanged.

14. What do producer acknowledgements mean in Kafka?

• acks=0: The producer does not wait for a broker acknowledgement.
• acks=1: The leader acknowledges after writing the record locally.
• acks=all: The leader waits for all required in-sync replicas to acknowledge, according to the topic and broker configuration.

For stronger durability, interviews often expect discussion of acks=all, an appropriate min.insync.replicas value, retries, and idempotent producer behavior.

15. What is an idempotent Kafka producer?

An idempotent producer prevents duplicate records caused by producer retries within a producer session and partition. Kafka uses producer identifiers and sequence numbers so that a broker can detect duplicate writes. Idempotence is an important building block for reliable delivery and Kafka transactions.

16. What is a Kafka consumer group?

A consumer group is a collection of consumers that cooperate to read a topic. Within one group, each partition is assigned to only one consumer at a time. Different consumer groups can read the same topic independently and maintain separate offsets.

17. What happens when there are more consumers than partitions?

Within a single consumer group, consumers beyond the number of assigned partitions remain idle because one partition cannot be processed by two consumers in the same group at the same time. To use more consumers concurrently, the topic generally needs more partitions.

18. What is a consumer-group rebalance?

A rebalance redistributes partition assignments among consumers in a group. It can occur when a consumer joins or leaves, subscriptions change, partitions are added, or a consumer is considered failed. Rebalances can pause processing temporarily, so applications should use suitable timeout settings, cooperative assignment where appropriate, and efficient record processing.

19. What is the difference between automatic and manual offset commits?

With automatic commits, the consumer periodically commits offsets according to its configuration. With manual commits, the application decides when offsets should be stored. Manual control can better align the committed position with completed processing, but incorrect commit timing can cause duplicates or skipped records.

20. What do at-most-once, at-least-once, and exactly-once mean in Kafka?

• At-most-once: Records may be lost, but they are not processed more than once.
• At-least-once: Records are retried and may be processed more than once.
• Exactly-once: Each input record has one effective result within the boundaries supported by the processing design, such as Kafka transactions and compatible read-process-write workflows.

Exactly-once behavior is not achieved by a single setting for every external system. The complete workflow, including sinks and side effects, must support the required guarantees.

21. How does Kafka retain records after consumers read them?

Kafka does not normally delete a record simply because a consumer has read it. Records remain available according to the topic’s retention policy. Time-based and size-based retention remove old log segments, while log-compacted topics retain the latest record for each key, subject to compaction behavior and delete markers.

22. What is Kafka log compaction?

Log compaction is a cleanup policy that preserves the latest value for each record key rather than retaining every historical update indefinitely. It is useful for changelog topics, state restoration, and streams that represent the current value of an entity.

23. What is the difference between Kafka and a traditional message queue?

Kafka stores records in partitioned logs and allows multiple consumer groups to read the same records independently. Traditional queues often focus on delivering each message to one competing consumer and removing or acknowledging it as completed. The exact comparison depends on the queue product because modern messaging systems can support overlapping features.

24. Is Kafka better than a REST API?

Kafka and REST solve different communication needs. REST is suitable for synchronous request-response operations where the caller needs an immediate result. Kafka is suitable for asynchronous event distribution, buffering, replay, and decoupling producers from consumers. Many systems use both: REST for commands or queries and Kafka for resulting events.

25. What is Kafka Connect?

Kafka Connect is a framework for transferring data between Kafka and external systems. Source connectors import data into Kafka, while sink connectors export records from Kafka. Connect supports distributed execution, offset management, configuration, converters, transformations, and task scaling.

26. What is Kafka Streams?

Kafka Streams is a Java client library for building stream-processing applications. It supports stateless operations such as filtering and mapping, and stateful operations such as aggregation, joins, and windowing. Kafka topics serve as its input, output, and internal state-changelog streams.

27. What is the difference between a KStream, KTable, and GlobalKTable?

• KStream: Treats each record as an independent event in a stream.
• KTable: Treats records as updates to the latest value for each key and partitions the table across application instances.
• GlobalKTable: Replicates the complete table to every application instance, which can simplify joins when the reference data is reasonably sized.

28. How is record ordering maintained in Kafka?

Kafka preserves the append order of records within each partition. To keep related records ordered, producers normally use a stable key so those records are sent to the same partition. Ordering across multiple partitions requires additional application-level coordination and can reduce parallelism.

29. What is consumer lag?

Consumer lag is the difference between the latest available offset in a partition and the consumer group’s processed or committed offset. Increasing lag can indicate slow processing, insufficient consumer capacity, downstream latency, repeated failures, or partition imbalance.

30. How would you troubleshoot increasing Kafka consumer lag?

1. Confirm whether lag affects every partition or only specific partitions.
2. Check consumer errors, processing time, retries, pauses, and downstream dependencies.
3. Verify that consumers poll frequently enough and that batch sizes are manageable.
4. Compare the number of active consumers with the partition count.
5. Check for rebalances, failed instances, network latency, and broker throttling.
6. Review whether one key or partition receives disproportionate traffic.
7. Scale consumers only when additional partitions are available for assignment.

31. How would you prevent duplicate processing in a Kafka application?

Duplicate processing is usually handled through idempotent application logic. Common approaches include using a unique event identifier, storing processed identifiers, using an idempotent database operation such as an upsert, committing offsets only after successful processing, or using Kafka transactions for supported Kafka-to-Kafka workflows.

32. How would you handle a poison message in Kafka?

A poison message repeatedly fails processing because of invalid data or an unsupported condition. A typical design limits retries, records the error context, and sends the failed record to a retry topic or dead-letter topic. The application should preserve the original key, payload, topic, partition, offset, and failure details where appropriate for diagnosis.

33. How do you choose the number of partitions for a Kafka topic?

The partition count should account for expected throughput, consumer parallelism, ordering requirements, broker capacity, retention volume, and future growth. More partitions provide more parallelism, but they also increase metadata, open files, replication work, recovery time, and operational complexity. Partition count should be tested with realistic workloads rather than selected from a fixed rule.

34. What happens if the Kafka partition count is increased?

New partitions become available for future records, which can increase consumer parallelism. Existing records are not automatically redistributed. Key-to-partition mapping can change because the partition count is part of common partitioning calculations, so records for a key may start going to a different partition after the increase.

35. How can Kafka support fault tolerance?

Kafka supports fault tolerance through partition replication, leader election, in-sync replica tracking, distributed brokers, durable logs, producer acknowledgements, consumer offset management, and replicated metadata controllers. The final resilience level depends on replication settings, broker placement, minimum in-sync replicas, client configuration, and operational practices.

Scenario-Based Kafka Interview Questions

A consumer processes a record and crashes before committing its offset. What happens?

After restart or reassignment, the consumer can read the record again because the committed offset still points before it. This is an at-least-once scenario. The processing operation should therefore be idempotent, or the application should use a transaction design that coordinates the result with offset advancement.

A consumer commits an offset before completing the database update. What is the risk?

If the consumer fails after committing but before completing the database update, Kafka may resume from a later offset and the unfinished record may not be processed again. This creates an at-most-once failure window and can cause data loss from the application’s point of view.

One Kafka partition receives much more traffic than the others. What should you investigate?

Inspect the record-key distribution and producer partitioning logic. A small number of frequently used keys can create a hot partition. Possible responses include choosing a better-distributed key, redesigning the event model, splitting a hot entity’s workload when ordering permits, or using a custom partitioning strategy.

How would you replay Kafka records without affecting the current consumer?

Create a separate consumer group so it maintains independent offsets, then start it from the required position using offset-reset or seek operations. Confirm that downstream processing is safe to repeat before replaying production data.

How would you design Kafka processing when strict order is required per customer?

Use the customer identifier as the record key so all events for that customer are routed to the same partition. Process each partition sequentially for the order-sensitive operation. This preserves per-customer order while allowing customers assigned to different partitions to be processed in parallel.

Kafka Interview Preparation Checklist

• Explain topics, partitions, offsets, brokers, leaders, replicas, and consumer groups without mixing their responsibilities.
• State clearly that Kafka ordering is guaranteed within a partition, not across an entire multi-partition topic.
• Compare KRaft-based metadata management with ZooKeeper-based legacy clusters.
• Describe how acknowledgement, replication, and minimum in-sync replica settings interact.
• Explain when duplicate processing can occur and how idempotent handling reduces its impact.
• Connect consumer lag to partition count, processing rate, downstream latency, and rebalances.
• Use scenario answers that mention failure timing, committed offsets, retries, and recovery behavior.
• Avoid claiming that exactly-once delivery automatically covers every external database or API.

Frequently Asked Questions About Kafka Interviews

What Kafka topics should freshers study first?

Freshers should start with topics, partitions, brokers, producers, consumers, consumer groups, offsets, replication, retention, and basic command-line administration. They should also be able to describe one simple producer-to-consumer data flow.

What Kafka questions are common for experienced developers?

Experienced candidates are commonly asked about partition strategy, delivery semantics, consumer lag, rebalances, idempotence, transactions, schema evolution, retry and dead-letter designs, capacity planning, security, monitoring, and failure recovery.

Are scenario-based Kafka interview questions important?

Yes. Scenario questions test whether a candidate can apply Kafka concepts to duplicate processing, failed consumers, hot partitions, replay, data loss risks, and slow downstream systems. A strong answer explains both the immediate behavior and the design trade-offs.

Is Kafka difficult to learn for an interview?

Kafka fundamentals are manageable when learned through a producer, topic, partition, consumer-group, and offset workflow. Advanced areas such as transactions, stream processing, security, performance tuning, and cluster operations require practical experience and deeper study.

Should a Kafka interview answer still mention ZooKeeper?

Yes, when discussing older Kafka deployments or migrations. For current architecture discussions, explain KRaft first and identify ZooKeeper as the metadata and coordination system used by legacy Kafka clusters.