𝗔𝗽𝗮𝗰𝗵𝗲 𝗞𝗮𝗳𝗸𝗮 20-20 Interview Questions with Answers

1. What is Apache Kafka, and how does it work in a distributed system?

• Answer: Kafka is a distributed messaging system designed for high-throughput, low-latency event streaming. It works on the publish-subscribe model where producers publish messages to topics, and consumers subscribe to those topics to read messages. Kafka distributes data across brokers (nodes in the cluster) and uses partitions for scalability.

2. How does Kafka handle fault tolerance?

• Answer: Kafka achieves fault tolerance by replicating data across multiple brokers. Each partition can be replicated to other brokers, and if a broker fails, Kafka elects a new leader for the partition from the replicas. This ensures that no data is lost during broker failures.

3. What is a Kafka topic, and how are they partitioned?

• Answer: A Kafka topic is a category or feed name to which records are published. Topics are divided into partitions, which allow Kafka to distribute data and scale horizontally. Each partition is ordered, and messages are appended to it. Partitions also allow parallel consumption, improving throughput.

4. Explain the difference between Kafka’s replication factor and partitioning.

• Answer: Partitioning refers to dividing a topic into multiple partitions to enable parallel processing. Each partition is handled by a single broker, but the replication factor defines how many copies of each partition will be maintained across the Kafka cluster. A replication factor of 3 means each partition has two replicas in addition to the leader.

5. How do you ensure ordering of messages in Kafka?

• Answer: Kafka guarantees message order within a partition. To ensure strict ordering, messages with the same key should be sent to the same partition. If ordering across partitions is not critical, Kafka will balance load across partitions. Otherwise, you must carefully design partitioning logic.

6. What happens when a Kafka producer sends a message to a partition where the leader is unavailable?

• Answer: When a producer tries to send a message and the partition leader is unavailable, the producer will experience a LeaderNotAvailableException. Kafka will automatically elect a new leader from the replicas. After leader election, the producer retries the message send.

7. How do Kafka consumers handle failures or downtime?

• Answer: Kafka consumers track their read offsets using consumer groups. If a consumer in a group fails, Kafka will automatically redistribute the partitions assigned to that consumer among the remaining consumers in the group, ensuring no messages are missed.

8. Explain how Kafka achieves high throughput and low latency.

• Answer: Kafka achieves high throughput by writing data to disk in a sequential manner, avoiding random access bottlenecks. Data is stored in batches, reducing network overhead, and consumers read data in large chunks. Kafka uses zero-copy technology for fast disk-to-network data transfer, and data replication ensures durability without compromising latency.

9. What is the role of ZooKeeper in Kafka?

• Answer: ZooKeeper manages the metadata for Kafka, such as broker information, partition leader election, and maintaining the state of the Kafka cluster. It acts as a coordination service to manage and monitor Kafka brokers. However, Kafka’s newer versions are transitioning away from ZooKeeper with KRaft mode.

10. What is the role of an offset in Kafka, and how can you manage it?

• Answer: An offset is a unique identifier assigned to each message in a partition. Consumers keep track of offsets to know which messages they have processed. Kafka provides two ways to manage offsets:
  • Automatic offset management: Managed by Kafka, which commits offsets automatically.
  • Manual offset management: Consumers manage offsets themselves, providing greater control over message processing.

11. Describe the process of rebalancing in Kafka consumer groups.

• Answer: Rebalancing occurs when the number of consumers in a consumer group changes or partitions are reassigned. During rebalancing, partitions are redistributed among consumers to ensure each partition has one consumer. However, rebalancing can cause brief downtime, so it should be optimized by minimizing consumer changes.

12. How can you ensure exactly-once delivery in Kafka?

• Answer: Kafka supports exactly-once delivery semantics using a combination of idempotent producers and transactional messages. The idempotent producer ensures that duplicate messages aren’t written, and transactions allow atomic writes to multiple partitions. Consumers can use Kafka’s read-committed mode to ensure they only read committed transactions.

13. Explain how Kafka handles backpressure in a high-load scenario.

• Answer: Kafka handles backpressure through the consumer lag mechanism. If consumers cannot keep up with the producers, lag builds up. Kafka doesn’t apply backpressure at the broker level but relies on consumers to control their rate of consumption. If consumers fall too far behind, you can scale out the number of consumers in the consumer group.

14. How would you implement message deduplication in Kafka?

• Answer: Kafka provides idempotent producers that automatically deduplicate messages based on a unique sequence number. If a duplicate message is received, Kafka ignores it. Alternatively, consumers can handle deduplication by tracking processed message offsets or maintaining an external store of processed message IDs.

15. Describe a scenario where you would use Kafka Streams.

• Answer: Kafka Streams is ideal for real-time stream processing. For example, in a fraud detection system, Kafka Streams can process events from payment systems in real-time, joining streams of user activity with historical transaction data to flag suspicious transactions.

16. What is the difference between Kafka Streams and Kafka Connect?

• Answer:
• Kafka Streams is a library for processing data in real-time within Kafka topics. It supports stateful operations and joins between streams.
• Kafka Connect is a framework for moving large amounts of data into and out of Kafka using connectors. It is used to integrate Kafka with external systems like databases, NoSQL stores, or cloud storage.

17. What are Kafka’s log retention policies, and how do they work?

• Answer: Kafka’s log retention policies determine how long messages are stored. You can configure logs based on time (log.retention.hours) or size (log.retention.bytes). Once a log segment exceeds the retention time or size, it is marked for deletion, freeing up storage while keeping Kafka performance optimal.

18. Explain how Kafka handles large messages.

• Answer: Kafka is optimized for relatively small messages, but large messages can be handled by:
  • Increasing message size limits: Configuring message.max.bytes for producers, brokers, and consumers.
  • Compression: Using Kafka’s built-in compression algorithms (e.g., Snappy, GZIP).
  • Splitting large messages: Breaking large messages into smaller parts at the producer level.

19. How would you monitor a Kafka cluster in a production environment?

• Answer: Kafka can be monitored using metrics exposed via JMX, which can be integrated with monitoring tools like Prometheus, Grafana, and Confluent Control Center. Key metrics include:
  • Broker metrics: Disk usage, throughput, request latencies.
  • Consumer lag: How far consumers are behind in processing messages.
  • Producer metrics: Batch sizes, retries, message send rates.
  • ZooKeeper: Leader election times and connection health.

20. What is the significance of Kafka’s ISR (In-Sync Replicas), and how does it impact data durability?

• Answer: The ISR (In-Sync Replicas) are replicas that are fully caught up with the leader for a partition. Kafka ensures data durability by acknowledging writes only after all ISR replicas have received the data. If a replica falls behind (out of sync), it is removed from the ISR. The leader will only acknowledge the message once all remaining replicas in the ISR have successfully written the message.