Delivery Guarantees in Practice

Introduction

The ticket said "customer charged twice," and the customer was right. We pulled the logs, and there it was, clean as anything: the same PaymentAuthorized event, processed twice, 4 seconds apart, 2 calls to the payment processor, 2 captures, 1 very annoyed cardholder. What made it memorable wasn't the double charge. It was that 3 weeks earlier we had migrated that exact consumer onto Kafka's exactly-once semantics specifically so this could not happen. We had turned on the feature named after the guarantee we wanted, watched the config roll out, and closed the ticket that asked for it. And here was the thing it was supposed to prevent, sitting in the logs with our name on it.

Nobody had lied to us. Kafka's exactly-once semantics do exactly what they say — inside Kafka. The problem was that the capture wasn't inside Kafka. It was an HTTPS call to a payment processor 2 networks away, and the moment our consumer made that call, it had stepped outside the boundary the feature protects, into territory where the only guarantee anyone can honestly offer is at least once. The transaction wrapped our reads and our writes to other topics. It could not wrap someone else's money.

This is the trap the whole chapter is about, and it catches experienced people precisely because they did read the docs. "Exactly-once" is the most oversold phrase in event-driven systems, not because it's a lie, but because it's a true statement about a smaller thing than you think. The broker can make a within-cluster read-process-write loop atomic. It cannot reach across the network to the effect you actually care about — the charge, the email, the shipment, the webhook — and it never claimed it could. Stripe, whose entire business is not charging people twice, does not pretend otherwise: it delivers webhook events at least once and tells you, in the documentation, to dedupe them yourself by event id. The company with the most to lose from duplicates ships at-least-once and a warning label. That should tell you where the guarantee actually lives.

Here's the ground we'll walk together:

• Why there are only really 2 honest delivery guarantees — lose-maybe and duplicate-maybe — and why the third one everybody wants, exactly-once delivery, is not a hard engineering problem but a physically impossible one
• Where a duplicate is actually born: a specific, small, nameable window in every consumer, which once you can see it you cannot un-see
• Why the fix cannot live in the broker no matter how much you pay it — the end-to-end argument, the one distributed-systems principle this chapter is built on
• How to construct effectively-once out of parts you already have: at-least-once delivery plus idempotency, applied at the effect, with the inbox pattern to make it atomic
• The 2 numbers that quietly decide whether your deduplication is real or theater — how wide your key is, and how long you remember it

By the end, "just turn on exactly-once" should sound to you like "just turn on being-correct" — a thing nobody sells, because it isn't a switch.

The Problem, Precisely

Exactly-once delivery isn't hard. It's impossible — and the sooner you believe that, the sooner you build the thing that actually works.

Start with the spectrum, because there are fewer options than the marketing implies. When a consumer receives a message and does something with it, exactly one of 3 things can be promised about how many times that happens.

At-most-once: the message is delivered 0 or 1 times. You send it, you don't retry, and if it's lost in flight — network blip, consumer crash, full buffer — it's simply gone, and no one comes looking. This is fire-and-forget, and the Internet-Scale Product Systems atlas has a name for its failure: lost events. It's cheap, it's simple, and it's correct for exactly one kind of workload: the one where losing some is genuinely fine. A metrics sample. A "user is typing" indicator. Lose one and the next overwrites the loss.

At-least-once: the message is delivered 1 or more times. The sender retries until it gets an acknowledgment, so nothing is lost — but if the acknowledgment is what got lost, the sender retries anyway, and now the message arrives twice. This is what every real broker gives you across a real network, and it's the honest default. Duplicates, not gaps.

Exactly-once: the message is delivered 1 time, no more, no less. This is what everyone wants, what the config screen offers, and what does not exist at the delivery layer. Not "is expensive." Not "requires Kafka." Does not exist.

Here's why, in 4 sentences. A consumer finishes processing and must tell the broker "done" so it stops redelivering — and that acknowledgment travels over the same unreliable network as everything else, so it can be lost. When the broker doesn't hear an ack, it faces a question it can't answer: did the consumer finish and the ack vanish, or did it die before finishing? It has exactly 2 options, and they are the whole spectrum — redeliver (risking a duplicate, if the consumer had finished) or don't (risking a loss, if it hadn't). There's no third option, because the broker can't see the consumer's soul. Everyone recognizes this as the two-generals problem in a work uniform, then goes and enables exactly-once in production anyway, because the config screen offers it and the class was a while ago.

Put a number on the duplicate side, because it's the side you'll live on. Say a downstream dependency slows down and your consumers start timing out and retrying. 100 in-flight messages, each retried an average of 10 times before things recover, is 1,000 deliveries — a 10× amplification, and every one of those extra 900 is a message your handler has, in some sense, already seen. If your handler's idea of "seen it" is "did the effect," you didn't have a slow afternoon. You had 900 extra charges, emails, or shipments, and a very long evening.

So the real question is not "how do I get exactly-once delivery." You don't; nobody does. The question is: given that the network hands me at-least-once and a pile of duplicates, how do I make my processing come out as if each message arrived once? That property has a name — effectively-once, or exactly-once processing — and unlike its delivery-layer cousin, it is entirely achievable. You just have to build it, in the one place it can be built.

Naive Solutions, and What They Cost

Before the construction, the 3 things teams reach for first. They appear in this order because each is the next thing you try after the last one fails, and each is reasonable enough to ship.

Turn on the broker's exactly-once feature. The config-screen solution, and the most common by a wide margin, because the feature is right there and named after your exact problem. You enable Kafka's EOS, set the isolation level, restart the consumers, and close the ticket. It even works — for the part of the pipeline that lives inside Kafka. Reads from the input topic, writes to output topics, and the offset commit all become one atomic unit: they all happen or none do. What it does not wrap is the call your handler makes to anything that isn't Kafka — the database in another service, the payment API, the email provider. The transaction commits inside the cluster; the charge already happened outside it. You've bought a real guarantee for the hop you weren't worried about and none for the one you were. The cost is the worst kind: not a failure, but a false sense of safety that holds right up until the redelivery storm, at which point you discover the boundary experimentally, in production, from a support ticket.

Add a dedup check at the top of the handler. The correction, once you've learned the broker won't save you. Keep a set of message ids you've already processed; on each message, check the set, skip if present, otherwise process and add it. This is the right idea — dedup at the consumer — implemented in the 2 wrong places. First, the check and the effect aren't atomic: you check "not seen," then the process crashes after the effect but before you record "seen," and the redelivery sails through a check that never got updated. That's a check-then-act race, and it fires exactly when you're already having a bad day. Second, the set can't be infinite, so it has a time limit — and if a duplicate arrives after that limit, the check passes and the effect runs again. Both failures share a signature: the dedup guards the message, and the thing that hurt you was the effect.

Wrap the handler in a database transaction. The sophisticated-looking one. Put the effect and the "mark processed" write in a single transaction so they commit together — genuinely good, and half the real answer. But teams then put an external call inside that transaction: capture funds via the payment API, then commit the row. A network call doesn't roll back because your database transaction did. If the API succeeds and the commit fails, you rolled back the record and kept the charge — and every retry adds another, each dutifully un-recorded. The transaction is real; it just doesn't reach other people's systems, which is the same wall the broker hit, one layer up.

Notice that all 3 are the same mistake at different addresses: each tries to make the plumbing — broker, cache, database — responsible for a guarantee that only means anything at the effect. The effect is where the customer feels it. The effect is the only place the guarantee can be enforced. Everything else is moving the responsibility around and hoping it lands somewhere cheaper.

Failure Modes Worth Naming

The field guide before the framework. Each of these is a delivery-guarantee misunderstanding coming back as an incident, and each hides the same way — invisible until a failure or a retry exercises the path.

The row that surprises people is ambiguous acknowledgment, because the duplicate is born from a success. The consumer processed the message correctly and completely; only the "I'm done" going back failed. From the broker's side that's indistinguishable from a crash, so it redelivers, and your healthy consumer processes a perfectly good message twice. You can't engineer this away at the transport — you can only make the second processing harmless, which is the entire construction, and it's next.

The Construction: At-Least-Once Plus Idempotency

Everything so far has been clearing the ground. Here is the thing you actually build, and it rests on one principle worth stating plainly.

The end-to-end argument. A correctness property that ultimately depends on what happens at the endpoints can only be completely and correctly enforced at those endpoints — not by the communication system in between. It comes from a 1984 paper by Saltzer, Reed, and Clark, and it is the single most useful idea in this chapter. The network can help — retries reduce how often you hit the edge cases — but it cannot guarantee the property, because it doesn't know what "done" means to your application. Only the consumer, at the moment it performs the effect, knows that. So that is where deduplication has to live. Not because the broker is inadequate, but because the broker is the wrong layer, structurally, permanently, no matter how good it gets.

Which turns the impossible-sounding goal into a simple recipe. Effectively-once processing = at-least-once delivery + an idempotent effect. Let the broker deliver the message one-or-more times — the honest guarantee, the one you actually have — and make the effect such that doing it twice is the same as doing it once. Then duplicates stop being a correctness problem and become a performance footnote: a redelivered message costs you a wasted cycle and changes nothing the customer sees. You have not defeated duplicates. You have made them boring, which is the only victory available.

Idempotency is the b2c atlas's subject, so here we only need the 3 shapes it takes at a consumer, cheapest first. Natural idempotency: structure the effect so repetition is inherently safe — a unique constraint that rejects the second insert, an upsert that overwrites with the same value, a "set status = shipped" that's already shipped. When you can get it, it's free and needs no bookkeeping; the database is your dedup store and it was going to run the write anyway. A dedup key: when the effect isn't naturally safe, give each operation an identity — the event id works, that's what Stripe's evt_... is for — and record the ones you've completed, skipping any you've seen. A distributed lock: the heavyweight, for when two workers might process the same key at the same instant and even the database race matters; you serialize on the key and pay in latency and a new dependency. Reach down this list only as far as the effect forces you.

But where you record "I did this" is the part teams get wrong, and it's the failure the naive dedup-check walked into. If recording the key and performing the effect are two separate steps, you haven't closed the duplicate window — you've moved it between the two steps. The fix is to make them one step, and it has a name that mirrors a pattern you already know.

The inbox pattern. The Internet-Scale Product Systems atlas taught the outbox: to publish an event reliably, write it to an outbox table in the same transaction as your state change, so the write and the intent-to-publish commit atomically. The inbox is the same trick pointed the other way, at the consumer. Record the processed event id and perform the effect in one transaction. If the effect is a database write, this is exact: insert the event id into a processed_events table with a unique constraint, do the business write, commit both together. A redelivered event tries to insert an id that's already there, the unique constraint rejects it, the whole transaction rolls back, and the business write never happens a second time. The dedup and the effect share a fate — both commit or neither does — so there is no window between them for a crash to exploit.

The honest limit, and the chapter would be lying without it. Some effects aren't in a database and can't be rolled back: an email sent, a push delivered, money moved through a rail that doesn't take backs. The inbox needs the effect and the record to share a transaction, and you can't enlist an email provider in your Postgres commit. For these, the b2c atlas's partial idempotency is the grown-up answer: get the record and the effect as close to atomic as you can, make the effect idempotent at the provider (many now accept your idempotency key — Stripe does), or decide the occasional duplicate email is cheaper than the machinery to prevent it. Naming which effects you cannot make idempotent isn't a gap in the design; it's the design being honest about physics.

Tradeoffs Worth Arguing About

2 decisions here don't have a universal answer, and pretending otherwise is how you end up with the wrong one.

Broker transactions versus consumer idempotency. These are two ways to reach effectively-once, and the argument between them is real. Broker transactions — Kafka's EOS — are genuinely excellent at what they cover: if your pipeline is Kafka-to-Kafka, a stream job reading a topic and writing derived topics with nothing external in between, EOS makes the whole read-process-write loop atomic and you write almost no dedup code. That is a large, real gift, and refusing it out of end-to-end purism is its own mistake. The cost is that it stops at the cluster edge, it adds a transaction coordinator and a throughput tax, and it lulls people into thinking it covers effects it doesn't. Consumer idempotency — the inbox, the dedup key — works everywhere, including across the network boundary EOS can't cross, and it makes the guarantee visible in code where you can reason about it. Its cost is that you own a dedup store, its scope, and its window, forever.

So the line is not "idempotency good, transactions bad." It's about where your effects are. If they live inside Kafka, lean on EOS and skip the hand-rolled dedup; you'd be rebuilding what the broker already gives you. If the effect that matters is an external charge, an email, a row in another service's database, then EOS is answering a question you didn't ask, and consumer idempotency is the only thing that reaches the effect. Most systems are the second kind, which is why this atlas talks about idempotency far more than transactions — but the first kind is real, and a stream-processing pipeline that ignores EOS is leaving a free guarantee on the table.

How wide the key, and how long the memory. Idempotency has 2 dials the b2c atlas named, and they're where correct-looking dedup quietly fails. Scope is what counts as "the same" operation — too wide (dedup on user id) and you drop legitimately distinct events, too narrow (dedup on a timestamp that varies per delivery) and duplicates slip through as "different." Window is how long you remember a key. And the window has a hard floor most people never compute: it must be longer than the longest possible gap between a message and its duplicate. That gap isn't set by your traffic; it's set by your worst redelivery — a consumer that was down for 2 hours, a dependency that queued messages during an outage and released them at once. A 10-minute dedup TTL against a 40-minute outage is not dedup; it's dedup-shaped decoration that passes every test and fails every incident.

What the Companies Actually Built

Stripe is the cleanest illustration in the chapter, because a payments company cannot be casual about charging twice, and look at what they chose. Stripe delivers webhook events at least once, and says so plainly: your endpoint might receive the same event more than once — a retry after a slow response, a network hiccup, a redelivery after your server processed the event but failed to return a 2xx in time. Their documented instruction is not "enable a setting." It's: log the event ids you've processed, and don't process a logged id again. Every event carries an evt_... id built for exactly this. That is the dedup-key construction, prescribed by the company with the most to lose, as the consumer's job. Stripe could have chased exactly-once delivery to your endpoint; instead they gave you a stable id and told you where the guarantee lives. The lesson isn't "copy Stripe" — it's that the firm whose entire product is not-double-charging judged exactly-once delivery the wrong problem and shipped at-least-once plus an id. If they don't fight physics, neither should you.

And now the example that complicates the chapter's own advice, because a chapter that only praises consumer idempotency is hiding the other half. Kafka's exactly-once semantics — EOS, shipped in 0.11 — are real, and do something consumer-side dedup can't do as cleanly: they make a read-process-write loop atomic across output partitions and the offset commit, via an idempotent producer, transactions, and sendOffsetsToTransaction. For a stream job that reads a topic, computes, and writes derived topics, this is the right tool, and hand-rolling an inbox there fights a guarantee the platform hands you for free. The sharp edge — the one that put a double charge in my logs — is written in the docs if you read past the feature name: EOS applies to data within Kafka. The moment your handler calls a database in another service or a payment API, you're outside the transaction, and end-to-end exactly-once now needs that system to be transactional or your write to be idempotent. Kafka isn't overselling; the feature name just arrives 3 sentences before the boundary, and most people stop reading at the name. (Confirm your Kafka version's exact EOS guarantees before relying on this; the boundary has been stable since 0.11 but the configuration around it has moved.)

The two examples are the whole chapter in miniature. Stripe had effects outside any broker — real money on real cards — so the guarantee had to be built at the consumer, and they built it with an id. Kafka's own pipelines keep the effect inside the cluster, so the guarantee can be built in the broker, and they built it with a transaction. Same target, effectively-once. Opposite correct answers, chosen entirely by where the effect lives.

Technologies Worth Knowing

Read each of these as a position on "where does the guarantee stop," which is the only question that matters here.

Kafka's idempotent producer and transactions are two features people blur into one. The idempotent producer fixes a narrow case: a producer retrying after a lost ack would write the record twice, so Kafka numbers each producer's writes and the broker drops the duplicate — at-least-once sending made effectively-once on the write into Kafka. Transactions build on it to make multi-partition writes plus the offset commit atomic. The sharp edge, because it's the costliest to forget: both live and die at the cluster boundary — superb for Kafka-to-Kafka, silent about everything else.

AWS SQS FIFO is the most honest exactly-once in the business, precisely because it tells you the expiration date. A FIFO queue deduplicates messages with the same deduplication id — but only within a 5-minute window. Send a duplicate inside 5 minutes and SQS drops it; send it at minute 6 and SQS treats it as new. That's not a limitation to sneer at; it's the truth about all dedup, printed on the label. Every dedup has a window; SQS just makes you see it. Its other edge is the one every queue has: you get each message once only if you delete it inside the visibility timeout, so a slow consumer that doesn't extend the timeout hands the message to someone else — at-least-once returning through the back door.

Idempotency at the provider is the newest and most quietly important. Stripe, and increasingly other APIs, accept an idempotency key on the request itself: send the same key twice and the provider does the operation once and returns the first result. This is the end-to-end argument implemented by the endpoint you were worried about — the payment API making itself safe to call twice — which is exactly where it belongs. When an effect offers this, use it; it turns a non-idempotent external call into a naturally idempotent one, and it's the closest thing to a free lunch this chapter contains.

The Principal Engineer's Perspective

Exercises

None of these has a clean answer, and that's deliberate. Paste any into an AI and you'll get a confident, tidy reply in 4 seconds — one that assumes your effects are transactional, your window is infinite, and your redelivery gap is small. The value is in the arguing, on your actual system.

Exercise 1 — Find the window. Pick one consumer you own that does something the outside world can see. Draw its 4 steps: receive, effect, record-that-you-did-it, commit. Now put a crash marker between each pair of steps and say, for each, what a restart does. Exactly one of those gaps produces a duplicate effect. Name it, then say what it would take to close it — and whether you actually have.

Exercise 2 — Rebuild the double charge in your domain. Our worked example was a payment captured, then a crash before the offset commit, then a redelivered event and a second capture. Rebuild it with your own effect: name the visible action, the point where you do it before recording it, the redelivery trigger, and the point where the mapping breaks — an effect you genuinely cannot make idempotent. That breakdown is the interesting part; a system where everything maps cleanly is one where you haven't found the email-already-sent case yet.

Exercise 3 — Price your window. Find the dedup window for one idempotent consumer. Then find, from real incidents, your longest redelivery gap — the worst outage or backlog that consumer has survived. If the window is shorter than the gap, you've found a live bug; if it's longer, compute what the storage costs and decide out loud whether you'd defend that number in a cost review. Either way you now know a thing you were guessing at.

Exercise 4 — Argue EOS versus idempotency for one real pipeline. Take a pipeline you run and make the strongest case for leaning on broker transactions, then the strongest case for consumer idempotency — using where your effects actually live, not a diagram. "It depends" is allowed only if you then say what it depends on and commit to an answer for this pipeline.

Exercise 5 — Hunt the effect-before-record. Grep your consumers for the shape where an external call (http, charge, send, notify) happens before the line that records the message as processed or commits the offset. Each hit is an armed duplicate, waiting for a crash to land in the window. You're not looking for a bug that's firing; you're looking for the one that will fire the next time a pod dies mid-handler.

Connections

← Brokers, Logs, and Queues (Chapter 3). Chapter 3 set up exactly where the duplicate is born. The queue's ack-then-delete and the log's offset commit are both a consumer saying "done," and both can be crashed-through or lost — the redelivery-on-timeout named there is the at-least-once this chapter builds on. Chapter 3 chose the carrier; this chapter handles the duplicate the carrier's acknowledgment model makes inevitable.

← The Anatomy of an Event (Chapter 2). Intent decides how much a duplicate costs. Replay a fact twice — OrderPlaced — and a well-built consumer shrugs. Execute a command twice — CaptureFunds — and money moves twice. The fact/command distinction from Chapter 2 is also a map of which streams need the inbox pattern most: the commands, where the effect is the point.

← From Request/Response to Events (Chapter 1). Chapter 1's "fire and forget" was at-most-once adopted by accident — the quiet drop of the one event that mattered. This chapter names the whole spectrum that reflex sits on, and gives you the other two points to choose from on purpose.

← Idempotency and the async foundations (the Internet-Scale Product Systems atlas, ch 2 and 7). That atlas taught idempotency, dedup keys, natural idempotency, scope, and window as request-handling tools, and defined at-least-once delivery and the consumer offset. This chapter assembles them into the end-to-end construction and adds the inbox pattern — the outbox's consumer-side twin — as the atomic form.

→ Ordering, Keys, and Partitions (Chapter 5). Dedup and ordering interact: a partition rebalance is a prime birthplace of redelivery, and a dedup key that ignores partition semantics can behave differently after a repartition. Chapter 5 makes partition-key selection its subject; hold the thought that your idempotency key and your partition key are related decisions.

→ Replays, Backfills, and Reprocessing (Chapter 11). Everything here scales up to the replay: re-reading a topic from the beginning is at-least-once delivery on purpose, at volume, and it will re-fire every non-idempotent effect it touches unless you built the construction from this chapter. Replay safety is this chapter's discipline applied to a million messages at once.

→ Poison Pills, Dead Letters, and Error Flow (Chapter 12). At-least-once plus a message that can never be processed is its own problem — infinite redelivery of a duplicate you can't make succeed. Chapter 12 is where that message goes to be quarantined instead of retried forever.

Appendix A: Reference Implementations

The snippets below are illustrative, not production drop-ins. They exist to make one difference concrete: an effect recorded after it happens versus an effect recorded with it.

A.1 — The bug: effect before commit

// A consumer under at-least-once delivery. This double-charges. Find the window.
while (true) {
  msg = consumer.poll()
  paymentApi.capture(msg.chargeId, msg.amount)   // (1) external effect — happens NOW, on someone else's system
  consumer.commitOffset(msg.offset)              // (2) "done" — but if we crash between (1) and (2)...
}
// On restart, the broker redelivers msg (we never committed), and capture() runs again.
// No transaction can help here: capture() already left our process and moved real money.
// The window between (1) and (2) is the entire bug — small, invisible in tests, catastrophic in a rebalance.

The load-bearing line is the ordering of capture() and commitOffset(): the effect is externally visible and irreversible, and it happens before the system records that it happened. Swapping the order doesn't fix it — commit first and a crash before capture() loses the charge instead. There is no safe ordering of two independent steps; that's the point. The fix isn't reordering, it's making them one step, which needs a shared transaction the external API can't join — so A.2 moves the effect somewhere it can.

A.2 — The inbox pattern: dedup and effect in one transaction

// The effect is a local DB write, so dedup and effect can share a transaction.
msg = consumer.poll()
tx = db.begin()
try {
  tx.execute("INSERT INTO processed_events(event_id) VALUES (?)", msg.eventId)  // UNIQUE(event_id)
  tx.execute("UPDATE account SET balance = balance - ? WHERE id = ?", msg.amount, msg.acctId)
  tx.commit()                         // both writes commit together, or neither does
} catch (UniqueViolation) {
  tx.rollback()                       // we've seen this event_id — the effect does NOT run again
}
consumer.commitOffset(msg.offset)     // safe to lose: a redelivery just re-hits the UNIQUE violation

The load-bearing line is the UNIQUE(event_id) constraint doing double duty: it's the dedup store and it shares the transaction with the business write. A redelivered event fails the insert, rolls back the whole transaction, and the balance is untouched — the duplicate became a no-op. Note that the offset commit is now allowed to be unreliable: if it's lost and the message is redelivered, the unique violation catches it. We moved the guarantee off the fragile offset commit and onto the atomic database transaction, which is the only place it can be enforced. This only works because the effect is in the same database as the dedup record; when it isn't, you're in partial-idempotency territory (A.3's boundary).

A.3 — Kafka read-process-write, and exactly where it stops

// EOS makes THIS atomic: the reads, the writes to Kafka, and the offset commit.
producer.initTransactions()
records = consumer.poll()
producer.beginTransaction()
for (r in records) {
  producer.send(outputTopic, transform(r))       // covered — a write into Kafka
  // paymentApi.capture(r)                         // NOT covered — this leaves the cluster; EOS ends here
}
producer.sendOffsetsToTransaction(offsets, groupMetadata)  // offsets commit inside the transaction
producer.commitTransaction()                       // all-or-nothing, within Kafka only

The load-bearing line is the commented-out paymentApi.capture(r): everything above it that touches Kafka is inside the transaction and genuinely exactly-once; the moment you uncomment a call that leaves the cluster, that call is outside the transaction and back to at-least-once, and you need A.2's discipline (or a provider idempotency key) for it. This is the boundary the chapter's opening incident crossed without noticing. EOS is not overselling — it is precisely scoped, and the scope is "within Kafka." Read the feature name and the boundary as one sentence, and it never surprises you.

Next: Chapter 5 — Ordering, Keys, and Partitions: why global ordering is a myth you pay for, per-entity ordering is the guarantee you actually need, and the partition key you already have is the one that decides both — plus the hot partition that surprises everyone who did the rest right.