Exactly-Once in Kafka: Idempotence, Transactions, and Streams

A payment processor publishes a debit event. Kafka accepts it, but the acknowledgment disappears before reaching the producer. The producer now faces an uncomfortable decision: retry and risk recording the debit twice, or stop and risk losing it entirely.

This uncertainty is where exactly-once semantics earns its value. The challenge is preserving a correct result when an application cannot tell whether its previous action succeeded.

Kafka addresses different parts of that problem through producer idempotence, transactions, and Kafka Streams. Understanding their boundaries matters more than memorizing configuration flags. A pipeline can use all three and still charge a customer twice if the final database update sits outside the protected boundary.

1. Delivery semantics begin with failure behavior

A missing acknowledgment does not prove that a write failed. The request might never have reached the broker, or the broker might have persisted it before the connection failed. Producer retries address this ambiguity, but processing introduces another failure window: the interval between applying a result and recording progress.

Consider a consumer that updates a balance and then commits its offset. If it crashes between those operations, its replacement reads the event again. Reverse the order, and a crash can leave the offset committed without the balance update. Kafka’s delivery semantics documentation explains why coordinating output with consumer position is essential.

These are behavioral guarantees across a defined boundary. Setting acks=1 does not, by itself, mean at-most-once. Likewise, enabling producer idempotence does not make arbitrary consumer code exactly-once.

2. Idempotent producers remove duplicates caused by retries

Enable idempotence explicitly when it is a requirement:

enable.idempotence=true
acks=all
max.in.flight.requests.per.connection=5

Kafka requires positive retries and at most five in-flight requests per connection for idempotence. Leave retries at its supported default and use delivery.timeout.ms to bound delivery attempts. Explicitly enabling idempotence also prevents conflicting settings from silently disabling it. See the producer configuration reference.

The mechanism uses a producer ID, producer epoch, and sequence information tracked per partition. A retried batch retains its identity, allowing the broker to recognize a previously accepted batch. This is protocol-level duplicate detection; Kafka does not compare business payloads to decide whether two payments mean the same thing. The original design is documented in KIP-98.

That last row is the boundary engineers often miss. Calling send() twice with the same payment ID is two application sends. Producer idempotence does not collapse them into one. Nor does a newly created nontransactional producer automatically remember every business event its predecessor published. The KafkaProducer API documentation explicitly distinguishes internal retries from application-level resends.

Idempotence works independently on each partition the producer writes to. Atomic coordination across those partitions requires transactions.

3. Transactions connect output with consumed offsets

Suppose a processor consumes an order and emits both a ledger event and an audit event. Three things must agree: the ledger output, the audit output, and the input offset indicating that the order was handled.

A Kafka transaction can commit these together. Configure a transactional.id, initialize the producer, begin the transaction, publish outputs, and submit consumed offsets through that same producer. Kafka’s transaction coordinator maintains transaction state; reinitializing a stable transactional identity also prevents a stale producer instance from continuing to write. KIP-98 describes this coordination and fencing model.

The following Java excerpt shows the transaction body. It assumes configured string serializers/deserializers, enable.auto.commit=false, isolation.level=read_committed, and a producer with a transactional ID unique to its logical worker.

// Initialize once, before processing batches.
producer.initTransactions();

// One batch; surrounding lifecycle/error handling is omitted.
var records = consumer.poll(Duration.ofMillis(100));
if (!records.isEmpty()) {
    producer.beginTransaction();

    Map<TopicPartition, OffsetAndMetadata> offsets = new HashMap<>();

    for (var record : records) {
        // Illustrative fan-out; no external side effects.
        producer.send(new ProducerRecord<String, String>(
            "ledger-events", record.key(), record.value()));
        producer.send(new ProducerRecord<String, String>(
            "audit-events", record.key(), record.value()));

        offsets.put(
            new TopicPartition(record.topic(), record.partition()),
            new OffsetAndMetadata(record.offset() + 1)
        );
    }

    producer.sendOffsetsToTransaction(offsets, consumer.groupMetadata());
    producer.commitTransaction();
}

The offset is the next record to consume, hence offset() + 1. After an abort, continuing requires resetting the consumer position so the failed input is replayed; aborting alone does not rewind it. Fatal producer errors require closing the producer. An uncertain commit outcome also needs API-specific handling rather than a blanket abort-and-resend loop. Consult the transaction methods and error contracts before implementing recovery.

Downstream consumers must use read_committed. They receive committed transactional records and ordinary nontransactional records, while aborted records are excluded. An open transaction can hold back later records in the same partition through the last stable offset. This makes transaction duration relevant to downstream latency. Consumer configuration reference.

Atomic commit does not mean a consumer receives every record from a transaction in one poll(), or that separate consumers observe all participating partitions simultaneously.

4. Kafka Streams coordinates state as well as output

A counting application has another obligation: maintaining its accumulated state. Protecting output and offsets is insufficient if a replay increments a local counter twice.

Kafka Streams provides exactly-once processing for its managed state, Kafka output, and input offsets. Its recovery mechanism coordinates state with Kafka-backed changelogs so processing can resume consistently after failure. The guarantee concerns the committed result; processor code may execute again. See Kafka Streams core concepts.

application.id=order-counts
bootstrap.servers=localhost:9092
processing.guarantee=exactly_once_v2
commit.interval.ms=100

The exactly_once_v2 mode requires compatible clients and brokers; its broker requirement is Kafka 2.5 or newer. Current configuration documentation uses a 100 ms default commit interval under exactly-once processing. Streams configuration guide.

Keep external actions separate in your reasoning. A processor that calls inventoryDb.decrement() can repeat that call after a crash. A lookup against a mutable service can also return a different answer during replay. If reproducibility matters, capture the lookup result or reference-data version as durable input.

For database effects, a useful design is to record a unique event ID and apply its business mutation in the same database transaction. A replay encounters the existing ID and skips the mutation. Recording the ID separately creates another failure window.

Connectors require the same scrutiny. Confluent’s JDBC sink documents at-least-once delivery with support for idempotent upserts. That does not make arbitrary balance increments exactly-once. JDBC sink documentation.

5. Choose the guarantee around the business effect

For ledger and audit events emitted together into Kafka, transactions provide a useful atomic boundary. A database ledger still needs its own transaction rules, business-event uniqueness, and reconciliation. Kafka cannot determine whether two separately submitted records represent the same real-world payment.

For warehouse ingestion, keyed upserts may make at-least-once delivery sufficient. This depends on the update semantics: setting a row to a particular version is different from adding an amount each time a message arrives. Older replays must also be prevented from overwriting newer state when order matters.

For counts and sums, Streams exactly-once protects against duplication caused by processing recovery. It does not remove duplicate business events already present at different input offsets. Those require an explicit deduplication rule and retention policy.

For disposable telemetry, some loss may be acceptable. Make that a deliberate product decision. A dashboard used for informal trends and a stream used for customer billing have different correctness requirements, even if both contain “metrics.”

6. Measure the cost of committing

Transactions add coordination and transaction markers. Larger batches spread that work across more records, while shorter transactions make committed output available sooner. The useful tuning question is how much visibility delay the application can tolerate.

As an illustrative steady-state estimate, if a worker processes (R) records per second and commits every (T) seconds:

[ \text{records per transaction} \approx R \times T ]

At 10,000 records per second, a 100 ms interval groups roughly 1,000 records; a one-second interval groups roughly 10,000. These are planning estimates, not Kafka benchmark results. Processing stalls, partition distribution, and commit time affect the actual batches.

Under uniformly arriving traffic and regular commits, waiting for the next commit adds roughly (T/2) on average. It is not a latency bound: replication delays, backpressure, and unresolved transactions can add more time.

Benchmark realistic message sizes, partition counts, and failure conditions. Measure committed throughput, downstream p99 latency, abort rates, and recovery duration. Also provision replication appropriately across application and internal topics; the Streams production configuration guidance recommends replication factor three with a minimum in-sync replica count of two.

7. Make the guarantee explainable under failure

The strongest review question is concrete: if the process dies immediately after this line, what happens when it restarts?

Apply that question at every boundary: after publishing output, before committing progress, after changing a database row, and during a consumer rebalance. A test should verify the resulting business state, including what committed consumers observe, rather than merely checking that the application restarts.

Producer idempotence protects retries. Kafka transactions coordinate Kafka records and offsets. Streams extends that coordination to managed processing state. External effects need their own recovery design.

Exactly-once becomes useful when those responsibilities are explicit. The architecture should let you explain why repeated execution still produces one valid committed effect—and where that explanation stops.

If you are building a ledger, an event-driven platform, or an AI agent system that must recover without repeating consequential actions, get in touch through Heunify. Those failure boundaries are where the most valuable architecture discussions begin.