Spring Kafka: @KafkaListener, Consumer Rebalancing, Offset Commits
- Last updated
- Prerequisites:
- Kafka topics, partitions, consumer groups
- Spring Boot basics
- spring
- kafka
- kafka-listener
- consumer-group
- rebalance
- offset
Read at your depth
The practical view
Spring Kafka consumes messages with @KafkaListener(topics = "orders", groupId = "order-service"). Configure with KafkaListenerContainerFactory (concurrency = consumers per listener). Offsets are committed automatically by default (auto.offset.reset, enable.auto.commit) — for at-least-once semantics. For exactly-once-ish processing, use manual ack: AckMode.MANUAL and acknowledge(). Handle rebalances with ConsumerRebalanceListener (onPartitionsRevoked/Assigned) if you need state cleanup or final offset commit. Errors: @RetryableTopic for retries with dead-letter topics, ErrorHandler / SeekToCurrentErrorHandler for failure handling.
The same idea in other frameworks
N/A — server-side Spring/Kafka concern.N/A — Spring concern.N/A — Spring concern.N/A — Spring concern.Legacy vs modern
Auto-commit at-least-once vs manual ack with error handling
Default auto-commit gives at-least-once but can lose the offset on crashes and cannot react to failures; manual ack with retry gives control.
@KafkaListener(topics = "orders", groupId = "svc")
public void on(Order o) { // ❌ OLD: auto-commit, no failure handling
process(o);
}@RetryableTopic // ✅ NEW: retries + dead-letter
@KafkaListener(topics = "orders", groupId = "svc")
public void on(Order o, Acknowledgment ack) {
process(o);
ack.acknowledge(); // ✅ NEW: commit only after success
}Interview gotchas
Context
This is the critical Kafka reliability question — interviewers want the rebalance window and its effect on at-least-once vs at-most-once, which determines whether messages duplicate or are lost.
The mechanical answer
During a rebalance, partitions are reassigned from one consumer to another. A consumer that processed records but had not yet committed their offsets loses those uncommitted positions: when the new owner of the partition starts, it resumes from the last committed offset, so the records between the last commit and the rebalance are PROCESSED AGAIN — this is the at-least-once duplication window. If instead you committed offsets BEFORE processing (at-most-once), a rebalance skips records that were committed but not processed — message loss. There is no way to get exactly-once without extra machinery (idempotent processing or the transactional exactly-once API). This is why the ack/commit timing is the central design decision: with at-least-once, you must make processing idempotent to tolerate replays.
Trap
The naive answer is 'rebalance just redistributes partitions'. It also reopens the delivery window — uncommitted records replay. Another trap: assuming auto-commit means 'commit after each message' — it commits on a timer (auto.commit.interval.ms), so a crash can replay far more than one record. Candidates who tie offset timing to the delivery guarantees show real Kafka understanding.
Context
This tests the delivery semantics model — interviewers want the commit-timing trade-offs and which configuration produces which guarantee, plus how exactly-once is actually achieved.
The mechanical answer
At-least-once: commit AFTER processing — if a crash happens between processing and commit, the record replays (duplicates possible). At-most-once: commit BEFORE processing — a crash after commit skips the record (loss possible). Exactly-once: each record is processed exactly once despite failures — Kafka achieves this with the transactional producer/consumer (the read-process-write cycle is atomic across the input offset and output writes via a transaction), or with idempotent downstream writes keyed by record id. In Spring Kafka: AUTO ack (default) with processing before the timer commit is effectively at-least-once; MANUAL ack lets you control the commit point; AckMode.RECORD commits after each record (at-least-once, more precise). Exactly-once additionally requires idempotence in your processing or the transactional EOS API, since a redelivery can still re-run a non-idempotent side effect.
Trap
The naive answer is 'exactly-once is automatic in Kafka'. It requires the transactional exactly-once semantics AND idempotent sinks. Another trap: confusing ack-mode with enable.auto.commit — MANUAL ack disables auto-commit and hands the commit decision to you; forgetting to acknowledge means offsets never advance and records replay indefinitely. Candidates who map ack modes to the delivery guarantees show real depth.