Zenorator
Atlas of Internet-Scale Product Systems — Chapter 12

Saga Pattern (Distributed Transactions)

A saga is a sequence of local transactions stitched together by events. Order placed → Payment charges the card → Inventory reserves the item →…

26 min read4 figuresSee the concept map ↓
Chapter 12 · Concept map

The shape of the whole thing

5 stages, in the order the chapter argues them. Read it top to bottom, or jump straight to the part you came for.

FoundationsPatternsTrade-offsCase studiesOperating it

Introduction

A saga is a sequence of local transactions stitched together by events. Order placed → Payment charges the card → Inventory reserves the item → Notification emails the customer. Each step commits on its own. When a later step fails, compensating actions walk back the earlier ones: refund the card, release the reservation, send the apology. This is how Uber, DoorDash, and Amazon move money and goods across services that share no database and trust no central coordinator to hold the whole thing together.

Start with what a saga is not: a database transaction. You cannot BEGIN TRANSACTION, call three separate microservices, and COMMIT. The database has never heard of those services. The services have never heard of each other's databases. What looks like one atomic operation — "place an order" — is a distributed workflow that can die halfway through, leave state scattered across three systems, and demand deliberate cleanup. The saga is the pattern that drags that cleanup into the open, where you can name it, test it, and recover from it.

Odds are you've already built one. You just didn't call it a saga. Somewhere in your codebase is a method that calls Payment, then Inventory, then Notification, wrapped in try-catch blocks of varying conviction, with a comment that reads // TODO: handle compensation. That comment was added in the second sprint, and it will outlive the service. You're operating a saga. The pattern doesn't hand you new work — it hands you vocabulary and structure for the work you're already doing, badly, in a catch block.

Two flavors dominate. Orchestration puts a single service in charge: it calls the shots step by step and owns every failure decision. Choreography puts no one in charge: each service reacts to the previous service's event, and the workflow exists only as an emergent property of who's listening to whom. Both work. Both fail — just differently, and the difference is the decision. Choosing between them is one of the more consequential calls you'll make in a microservices system, not because one is correct, but because the failure modes you're signing up for compound quietly over years.

Two properties from earlier chapters are load-bearing here, so internalize them before you go further. From Chapter 2: every step has to be idempotent, because events get replayed and commands get retried, and "charge this payment" will arrive more than once — that's not an edge case, it's Tuesday. From Chapter 8: the system is eventually consistent, which is the polite way of saying there's always a window where the customer has been charged but the inventory isn't reserved yet — a state you didn't design as a resting state and will nonetheless find your system resting in. Design for that window. It's there whether you acknowledge it or not; acknowledging it is just cheaper.

The Problem: Why You Can't Just Lock Everything

The textbook answer to distributed transactions is two-phase commit (2PC). Ask every participant to "prepare" — acquire its locks, declare it's ready — and once they've all declared, tell them all to "commit" at once. Databases do this internally and it works beautifully, right up until preparation doesn't go to plan, which at scale is roughly always.

Picture it. Order Service sends "prepare" to Payment and Inventory. Payment says "ready" and grabs a lock on the customer's credit limit. Inventory… times out. Now you're stuck, and stuck in the worst possible way: Payment is holding that lock, and Inventory either prepared and crashed or never heard you at all — and you cannot tell which. That ambiguity is the entire problem. Until Inventory answers or a timeout fires and you abort, Payment sits there locked. Call it 30 seconds. At a few thousand orders a minute, 30-second locks on payment accounts pile up considerably faster than they drain, and your throughput problem quietly becomes a credit-limit-contention problem.

The lock duration is worse than the first glance suggests. A distributed transaction holds its locks across every participant for the entire transaction, and the transaction lasts about as long as the sum of its parts: 100ms for Payment, 150ms for Inventory, 200ms for Notification. So you lock Payment for 450ms to do 100ms of actual Payment work. The locks scale with traffic, contention scales with the locks, and you've built a system that gets more fragile precisely as it gets more popular — which is exactly the wrong way for a system to respond to success.

The deepest problem isn't the locks, though. It's the coordinator. 2PC assumes a transaction coordinator that everyone trusts and that never, ever fails. When the coordinator dies mid-commit, the participants are left holding prepared-but-not-committed state with nobody to tell them how the story ends. This is the in-doubt transaction problem, and resolving it tends to involve manual intervention, reading transaction logs you've never had reason to read before, and a 2 a.m. phone call you'll be describing to people for years. It's the database equivalent of a room full of people holding hands, refusing to move, waiting for the one person who knows the plan to come back from lunch.

Sagas walk around all of it. Each service commits its own local transaction and publishes an event. No global lock, no coordinator holding everyone hostage. If a step fails, compensating actions undo what came before. The consistency model drops from atomic — all or nothing, all at once — to eventual — everything completes, or everything reverses, given a moment. That's a real trade, not a free upgrade: you're swapping "momentarily impossible to observe an inconsistent state" for "briefly easy to, if you go looking." For most B2C systems, a consistency window of 100ms to 2s is invisible to users and is, not coincidentally, the only model that survives contact with scale.

The Naive Solutions

Two naive versions show up constantly, usually in codebases that would be a little offended to hear the word "saga."

Synchronous chaining, no compensation. The order API calls Payment over HTTP and waits, calls Inventory over HTTP and waits, calls Notification over HTTP and waits, then returns success. (Full code: Appendix A.1.)

This is the first saga every team builds, and it looks completely reasonable on the whiteboard — which is the problem, because the whiteboard doesn't have a network on it. Here's the failure: Inventory returns a 500. Payment has already charged the customer. The exception propagates up the stack. Now answer the question the architecture forgot to ask — who issues the refund?

In this design, the answer is "whoever caught the exception," which means your OrderService quietly grows refund logic, then inventory-rollback logic, then notification-cancellation logic, all bolted into its catch blocks beside the actual order logic. Three months later it grows a second layer of emergency logic to handle the refund itself failing. This is the well-documented process by which a tidy 300-line service becomes a 2,000-line service nobody will touch, because the compensation code is load-bearing, undocumented, and buried exactly where the tests don't look.

And that's the obvious failure. The subtle one is worse: InventoryService.reserve() doesn't return a clean 500 — it times out. Did it reserve the item or not? You genuinely do not know. Refund, and you might be canceling an order that actually shipped. Don't refund, and you've charged someone for nothing. A timeout isn't an error; it's the absence of an answer, and in financial flows the absence of an answer is the support ticket that lands Friday at five and doesn't get resolved without someone running queries against production by hand.

Events, still no compensation. The slightly more sophisticated version emits events between steps: Order → PaymentProcessedInventoryReserved. It feels more grown-up. Then Inventory fails, the event chain just… stops, the order gets marked failed in some table, and the payment is still charged — same hole, nicer plumbing.

Worse, the notification service was subscribed to PaymentProcessed and has already done its job. The customer now holds a cheerful "your order is confirmed!" email for an order that will never ship. When they call to ask where it is, a human on your team issues the refund and writes the apology by hand. Compensation, everyone agrees, will be added next sprint. It shows up in planning as "saga cleanup," gets bumped one quarter, then another, and finally gets prioritized the week it earns a proper name — "the payment-inventory incident" — which is a much more expensive way to schedule the same work.

Failure Modes That Actually Surprise People

Strip away the state machines for a second and stand where the customer stands. You tap Place order. The spinner turns. The page comes back with Something went wrong — please try again — and your banking app, one swipe away, has already buzzed with a charge for the exact amount. Now you're making a decision no shopper should have to make: tap Place order again and risk paying twice, or wait and risk that the first attempt never really happened. That specific dread — reloading the order history to find nothing there, screenshotting the bank notification just in case — is what a saga failure feels like from the outside.

Done right, you never feel any of it. Either the charge quietly reverses within seconds, or the page tells you plainly that your card was refunded and nothing shipped: no confirmation email for a phantom order, no second guess, no Sunday-night support ticket. The entire distance between those two experiences is whether someone actually built the compensation paths in this section — or left them as a // TODO and shipped anyway. The failure modes below are the ways that gap gets manufactured.

Partial completion is the failure everyone designs for: Payment succeeds, Inventory fails, refund runs, done, gold star. Almost everyone stops there. The scenario that actually bites is one step deeper: Payment succeeds, Inventory fails, the refund is initiated — and the refund times out. Did it go through? Unknown, again. Your compensation just failed, and there's no compensation for a failed compensation. The system is now genuinely wedged: customer charged, inventory not reserved, refund in a state best described as "maybe."

The only thing that gets you out is a saga state machine — something that records which steps completed, which compensations completed, and which are stuck in between. A timed-out refund lands in compensation_pending, and a background job keeps retrying it. This is safe only because the compensation is idempotent: the refund carries the saga ID as its idempotency key, so retrying one the provider already processed comes back "success" instead of refunding a second time. That state machine is the entire difference between running a saga and running a sequence of HTTP calls with hope as your error-handling strategy. Hope does not appear in the on-call runbook.

Out-of-order compensation is rarer and meaner. In a choreography saga, every event for a given saga has to be partitioned by saga ID in Kafka, so that all of order abc123's events land on one partition in the order they were published. Skip that, and you eventually earn the bug report from hell: a CancelOrder event reaches Inventory before the OrderCreated it was meant to cancel, because the two rode different partitions with different consumer lag. Inventory receives "release the reservation," shrugs because there's nothing to release, then receives "reserve the inventory" and dutifully reserves it. The reservation is now locked, permanently, against an order that was canceled before it existed. The fix is one line of Kafka config. Discovering you needed that line is several days and a whiteboard.

Idempotency in compensations is non-negotiable, and I mean that literally — compensations are retried, by design, because the state machine refuses to give up until they land. If Payment.Refund isn't idempotent, you will refund some customers twice. At enough volume this isn't a possibility, it's a schedule: it happens on a Friday, goes unnoticed over the weekend, and surfaces Monday when the payout reconciliation report flags twelve customers who got their money back twice. Finance notices these things. Finance always notices these things.

Pattern 1

Orchestration-Based Saga

An orchestrator is a single service — or, better, a workflow component — that knows the whole sequence and drives it from the top. It issues each command, waits for the result, and owns every failure decision in one place. Concretely: it creates the order (a local DB write plus an Outbox entry), commands Payment to charge, and waits for PaymentProcessed or PaymentFailed; on success it commands Inventory to reserve and waits for InventoryReserved or InventoryFailed; on success it commands Notification and marks the order COMPLETED. If payment fails, it marks the order FAILED and stops — nothing was charged, nothing to undo. If inventory fails after payment already succeeded, it commands Payment to refund and marks the order FAILED_WITH_REFUND_PENDING. (Full code: Appendix A.2.)

Figure · Orchestration-Based Saga.
Surveyor’s note · figure not yet drawn(Inline figure — render here, beside this subsection.) Center: an Order Service box labeled "Orchestrator." Three outbound solid arrows to Payment Service, Inventory Service, and Notification Service, each labeled with its command (ChargePayment, ReserveInventory, SendNotification). Three return arrows labeled with the events that come back (PaymentProcessed/Failed, InventoryReserved/Failed, NotificationSent). Below, a dashed arrow labeled RefundPayment flowing from the orchestrator back to Payment Service, annotated "fires when InventoryFailed arrives."
the orchestrator is the brain — every command and every compensation flows through it, and no participant service needs to know any other participant exists. Shared visual grammar with the other saga figures: happy-path commands and events as solid arrows, compensation as a dashed arrow running backward.

The upside is clarity, and clarity is underrated. The entire workflow lives in one file. Something failed at 3 a.m.? There's one service to read. A new engineer asks "what happens if inventory fails after payment succeeds?" and you point at a function instead of convening a working group.

The downside is just as plain: that one service is a single point of failure and a throughput bottleneck wearing a nice suit. Every order flows through it, and its capacity for concurrent in-flight sagas becomes your ceiling. Teams that park saga state in one orders table meet this the hard way — index bloat, row contention, and eventually a meeting about sharding the saga state, which is precisely the distributed-database headache they adopted microservices to avoid. The complexity doesn't leave. It just waits.

Pattern 2

Choreography-Based Saga

Choreography fires the conductor. There's no central service; each participant listens for the event the previous step emitted and reacts. The Order Service emits OrderCreated. Payment is listening for that, charges the card, and emits PaymentProcessed or PaymentFailed. Inventory is listening for PaymentProcessed, reserves the item, and emits InventoryReserved or InventoryFailed. Notification is listening for InventoryReserved, sends the email, and emits NotificationSent. The failure path is just more subscriptions: on InventoryFailed, Payment emits RefundInitiated and the Order Service marks the order failed. The saga isn't written down anywhere — it emerges from who's subscribed to what, which is either elegant or terrifying depending on the day.

Figure · Choreography-Based Saga.
Surveyor’s note · figure not yet drawn(Inline figure — render here, beside this subsection.) A left-to-right event chain of four service boxes. Happy path (solid arrows): OrderCreated → PaymentProcessed → InventoryReserved → NotificationSent, each arrow handing off from one service to the next. Failure path (dashed arrows): InventoryFailed branching backward to Payment Service (which emits RefundInitiated) and to Order Service (which marks the order failed). Above every arrow, a small tag: "Kafka — key = saga ID."
no central coordinator; services are coupled only to the events they consume; the failure paths force services to subscribe to events outside their own happy path; and partitioning by saga ID is structural, not a tuning knob. Shared visual grammar with the other saga figures: happy path solid, compensation/failure path dashed and running backward.

The upside is real: no single point of failure, no bottleneck, and each service scales on its own. Adding a step means subscribing a new service to an existing event — no orchestrator to crack open and edit. When different teams own different participants, that loose coupling is worth a lot; nobody has to file a ticket against the orchestrator team to ship.

The downside is that there is no single place that knows what the saga is. Debugging a failure means reconstructing the story from events scattered across services, correlated by saga ID, across four codebases you now have open in four windows. The logic isn't in one file; it's smeared across every participant, and only the events remember the plot.

Which is why nearly every team that starts with choreography eventually builds a dashboard that stitches the events back together to show each saga's state. Congratulations: you've built an orchestrator. You just built it after the outage instead of before, and you call it "observability" so it doesn't count against the architecture. The tension between decoupled execution and centralized visibility never actually resolves — it just relocates to wherever you're least prepared for it.

Pattern 3

Compensating Actions

Every step needs a defined way to undo it, and the mapping should be explicit and written down before anyone writes code:

Step Compensation
Order.Create Order.Cancel
Payment.Charge Payment.Refund
Inventory.Reserve Inventory.Release
Notification.Send (no reversal possible)
Figure · Compensation runs the saga backward.
Surveyor’s note · figure not yet drawn(Inline figure — render here, beside this subsection.) Top row, left to right (solid arrows): the forward steps Order.Create → Payment.Charge → Inventory.Reserve → Notification.Send, each with a small "✓ committed" tag as it succeeds. Mark a failure point at Inventory.Reserve (red X). Bottom row, right to left (dashed arrows): compensations firing in reverse from the failure — Payment.Refund ← Order.Cancel — each aligned directly under the forward step it undoes, with steps after the failure greyed out (never ran, nothing to undo). Far right: Notification.Send capped with a red "no reversal" stub and a callout, "the apology email is a new action, not an undo."
compensation is not a rollback the infrastructure performs for you — it's a set of forward actions you write yourself, run in reverse order, and only for the steps that actually committed. Shared visual grammar with the other saga figures: forward path solid, compensation path dashed and running backward.

That last table row is the one to sit with. Some actions have no undo. Once an email is sent, it's sent; there is no unsend. The "compensation" for a notification is a second email asking the customer to disregard the first, which the customer will disregard. So design your compensations before you design your saga, not after — and if a step has no meaningful compensation, that's the system telling you it should happen before the saga commits to anything expensive, not in the middle where you can't take it back.

Compensations run in reverse order of the steps they undo — last thing done, first thing reversed — and like everything else here they must be idempotent and carry the saga ID, so a downstream service handed the same compensation twice can recognize it and decline to act again. None of this is new infrastructure. It's the idempotency machinery from Chapter 2, applied with discipline to every step and every compensation in the saga. The saga doesn't invent reliability primitives; it just insists you actually use the ones you were already supposed to have.

Tradeoffs

Orchestration vs. choreography is really a question of where you want the complexity to live, because it does not get to leave. Orchestration concentrates it in one service: visible, testable, debuggable from a single place — and coupled, because that service has to know the whole workflow, so every workflow change is a change to it. Choreography smears the complexity across participants, each knowing only its own step and its neighbors — decoupled, and nearly impossible to see whole, because no participant has the full picture and neither, at 3 a.m., do you. Pick your poison: a bottleneck you can read, or a system you can't.

Orchestration wins when the process has real branching and multiple failure paths, when the team is small enough that everyone wants the workflow in one place, or when you're on Temporal and most of the state-machine misery is handled for you. Choreography wins when separate teams own separate participants and you're optimizing to minimize cross-team coordination, or when the steps are simple enough that a shared orchestrator would be more friction than the lost visibility costs you. Notice that both of choreography's winning conditions are organizational, not technical — that's a hint about what you're really trading.

Synchronous vs. asynchronous is less of a debate than it looks, once you've been on the wrong end of it. Synchronous orchestration — plain HTTP between steps — is the fastest to build and fails in the least recoverable ways. One slow service blocks the orchestrator; one service down when you call it fails the saga on the spot, with no retry and no graceful recovery, just a stack trace and a charged customer.

Asynchronous orchestration — commands and events over Kafka, published via the Outbox pattern from Chapter 11 — is more work to get right and dramatically harder to knock over. Services can be down for a bit; the orchestrator just keeps redelivering the command until they come back. Failures don't cascade, they queue. For any B2C system leaning on external dependencies — payment providers, third-party inventory, notification vendors, all of which will have a bad afternoon — build the saga asynchronously. The synchronous version works fine until the day it strands a saga somewhere only a manual database edit can reach it, and that day is never a quiet one.

Company Examples

Uber: The Saga Sequence Is a Business Decision

Uber's ride booking is an orchestrated saga, but the interesting part isn't the orchestration — it's the order of the steps, which is a business decision wearing engineering clothes. Roughly: TripRequested, then DriverMatched (the matching algorithm finds a driver and the driver accepts), then PaymentAuthorized (the card is pre-authorized, not yet charged), then DriverAssigned (the driver is committed), then NotificationSent (rider and driver get the confirmation).

Look at where payment authorization sits: step 3, before the driver is assigned, not after. That's deliberate. Assigning a driver is expensive — matching, negotiating with the driver's app, committing a real human in a real car. Uber would rather not spend any of that only to find the rider's card is declined. So they authorize payment first; if it fails, the abort is cheap — drop the authorization, tell the rider, done — and crucially no driver was ever committed, so there's nothing expensive to compensate.

That's the whole principle, and it's the most portable idea in this chapter: order your saga steps by the cost of their compensation. Cheap-to-undo things go early. Expensive-to-undo things go late, after the cheap steps have confirmed the saga is even worth continuing. When you design one, ask of each step: "if this fails, what does cleanup cost?" — and push the expensive cleanups as late as the logic allows, so fewer of them are ever in flight when something breaks.

Interactive · Order steps by compensation cost.
Surveyor’s note · figure not yet drawn(Inline figure — render here, beside this concept.) A horizontal saga of four or five step tiles the reader can drag to reorder — e.g., Match Driver (expensive to undo), Authorize Payment (cheap), Reserve Inventory (medium), Send Notification (impossible to undo). Each tile shows two dials: its probability of failing and its compensation cost. Above the row, three live readouts recompute as the reader reorders: expected total compensation cost, the share of failures that require an expensive undo, and a verdict badge — "cost-ordered ✓" when cheap-to-undo steps precede expensive ones, "fragile ✗" when an expensive step sits early. Pre-load it in a deliberately bad order (expensive step first) and let the reader discover that dragging the expensive step later makes the expected-cost number fall.
sequencing isn't cosmetic — putting the expensive-to-undo step last is what makes the common failure cheap, and the reader should feel the number drop as they move it. Shared visual grammar: the forward saga reads left-to-right as in the other figures; compensation is the cost you pay walking back.

One more thing about Uber worth copying: the orchestrator isn't a hand-rolled state machine. It runs on a workflow engine in the Temporal lineage, so retries, timeouts, and compensation sequencing are the platform's problem, not application code's. This matters more than it sounds. The number of edge cases in a saga state machine that handles retries and timeouts and idempotency and concurrent events correctly is large enough that building it yourself is a multi-quarter project you will get subtly wrong. Use the engine. The people who wrote it have already made the mistakes for you.

DoorDash: The External Participant Problem

DoorDash's saga has a participant nobody else's does: a restaurant, staffed by humans, who accept or reject the order on human time — 10 to 30 seconds, sometimes a great deal longer. The sequence runs OrderCreated, RestaurantNotified (it hits the restaurant's tablet), RestaurantConfirmed/RestaurantRejected (a person taps a button), DriverAssigned (a Dasher is matched), PaymentProcessed (now the customer is charged), NotificationSent.

Payment is deliberately near the end. If the restaurant rejects at step 3, there's no charge to refund — compensation is just "mark it failed, tell the customer," about as cheap as cleanup gets. Same principle as Uber, different kitchen.

The failure that caused real operational pain is later: no driver can be found at step 4, after the restaurant has confirmed and started cooking. Now compensation has to reach back into the physical world — tell the restaurant to stop, void the pre-authorization, apologize to the customer. And the restaurant-facing half of that is a push notification to a tablet that might be muted, on bad WiFi, in a loud kitchen, watched by someone with eight other things on fire. DoorDash's saga tracks that compensation with a retry policy that polls for an acknowledgment for up to ten minutes. For those ten minutes, a kitchen is cooking food for an order that no longer exists.

There's no clean fix, and that's the lesson, not a failure of imagination. There's only a saga that models the uncertainty honestly and degrades as gracefully as a flaky tablet allows. The moment your saga includes people rather than services, failures stop being deterministic and turn probabilistic — "acknowledged" and "sent but not acknowledged" become genuinely different states you have to track separately, with an explicit policy for how long you wait before you give up and page a human. Most distributed-systems problems are about computers being unreliable. This one's about a tablet behind a counter, which is harder.

Technologies

Temporal

If you're building orchestrated sagas at any serious scale, look hard at Temporal before you write a state machine by hand. Temporal runs saga execution as durable workflows: if your orchestrator process falls over mid-saga, Temporal replays the workflow from the last committed step and continues as if nothing happened. Your compensation logic is just ordinary application code. Timeouts and retries are first-class, not something you bolt on afterward.

The mental-model shift is the real product. Instead of maintaining a state machine, a saga-state table, a background retry job, and a dashboard to hunt down stuck sagas, you write a function that awaits paymentService.charge(amount) and let Temporal handle the rest — payment service down, it retries; orchestrator crashes, it resumes from the checkpoint. All the infrastructure you'd otherwise build and babysit becomes someone else's maintained infrastructure.

It isn't free, and pretending otherwise is how teams get surprised. Temporal is itself a distributed system with its own failure modes and its own operational learning curve. For a shop running many saga types with gnarly compensation logic, that cost is obviously worth paying. For one saga with three steps, it may be more machine than the job needs — and "we adopted Temporal for a three-step workflow" is its own genre of postmortem.

Axon Framework (Java)

Axon is an opinionated Java framework that marries event sourcing (Chapter 9) to saga management. Saga state is persisted as events, compensation handlers are annotated methods, and routing, persistence, and replay are the framework's job. It pays off most in greenfield Java systems that need both event sourcing and sagas, where the tight integration genuinely removes boilerplate. In a polyglot shop, Temporal's language-agnostic SDK is usually the more practical call — Axon's gravity is real, and it's all Java.

Principal Engineer Perspective

The Principal Engineer’s View

When Does a Saga Actually Make Sense?

The diagnostic is one question: if step N fails after steps 1 through N−1 succeeded, is there anything to clean up? If yes, you need compensating actions. If that cleanup spans more than one service, you need a saga. Everything else is detail.

Where the threshold actually sits surprises people. It's not about volume — it's about cross-service state. A saga is overkill for a workflow where "it failed" just means "nothing happened" and there's no residue to sweep up, and it's overkill for anything one database transaction can handle inside one service. But the moment a user-facing action writes to two services in sequence, and the first write is visible to someone before the second one lands, you have a saga candidate, scale or no scale. In B2C, that's nearly every order, booking, and payment. You probably crossed the threshold years ago and have been paying for it in catch blocks ever since.

A second diagnostic, which doubles as a design smell: is there a compensating action for every step? If you sketch the saga and step 4 has no meaningful undo, the system is telling you something — either step 4 belongs later in the sequence, after the risky stuff has settled, or your saga boundary is drawn in the wrong place. Listen to it. A step you can't compensate is a step you should commit to last, or keep out of the saga entirely.

The Partition Key You'll Forget

The single most common choreography bug, by a wide margin: saga events not partitioned by saga ID in Kafka. The assumption underneath it is innocent — surely events for the same order arrive in order — and it's wrong in the specific way that's expensive. Kafka guarantees ordering within a partition and promises exactly nothing across partitions. Without explicit partitioning by saga ID, two events for the same order can land on different partitions and get consumed out of order, and your compensation runs before the thing it was supposed to compensate.

The fix costs one line: set the Kafka message key to the saga ID when you publish. Now every event for order abc123 shares a key, routes to one partition, and arrives in publication order. That's the whole fix. It's non-negotiable for choreography, and the only reason it's worth its own section is that the alternative way to learn it — from a production incident where a cancellation ran before its order existed — costs considerably more than one line, and nobody gives you the days back.

Testing Sagas

Unit tests cover the happy path. Sagas don't break on the happy path; they break on the failure paths, which is exactly the code your tests are least likely to exercise. The only strategy that actually validates a saga is chaos injection: for each step, force a failure and confirm the compensation runs; then force the compensation to fail and confirm the system lands in a retryable state instead of a wedged one; then fail two steps at once and confirm the state machine survives the intersection.

That demands a test harness that can pause a saga at any step and inject failure on command. Build it early, while the saga is still simple, because you will never have more time for it than you do right now. Without it, the saga works right up until it doesn't — and "until it doesn't" has an uncanny correlation with peak traffic, the one window where step failure rates rise and their nastier interactions multiply at the same time.

One metric to add on day one: saga completion rate broken out by terminal state — COMPLETED, FAILED_WITH_COMPENSATION, COMPENSATION_PENDING, COMPENSATION_MANUAL_REQUIRED. A rising count in COMPENSATION_PENDING is your early warning that retry infrastructure or a downstream dependency is quietly degrading. By the time the trouble reaches a user-facing metric, you're not getting a warning anymore — you're getting a backlog of inconsistent states to reconcile by hand.

Exercise: Compensation Failure Recovery

Scenario: A customer places an order. Payment succeeds (PaymentProcessed emitted). Inventory reservation fails (InventoryFailed emitted). The orchestrator initiates compensation and emits RefundPayment. The payment service attempts the refund and times out — no response within 30 seconds.

  1. What is the observable state of the system? What does the customer see?
  2. Should the orchestrator retry the refund? How many times, and with what backoff?
  3. What saga state does the orchestrator record while the refund is pending?
  4. If the refund timed out but the provider actually processed it, what happens when you retry?
  5. How does an operator tell this saga apart from one that's still safely inside its automatic-retry budget?

Hints: The saga needs a COMPENSATION_PENDING state distinct from FAILED. The orchestrator retries with exponential backoff up to a configured ceiling; past that ceiling, the saga flips to COMPENSATION_MANUAL_REQUIRED and fires an alert. Because the refund uses the saga ID as its idempotency key, retrying one the provider already processed returns success rather than refunding twice. The operator dashboard lists every saga in COMPENSATION_MANUAL_REQUIRED with its saga ID, the step where compensation is stuck, and the timestamp of the last retry — enough to open a case with the payment provider without guessing.

Connections

← Chapter 2 (Idempotency). Every saga step and every compensating action has to be idempotent, because sagas retry and duplicates are guaranteed, not hypothetical. The idempotency key is usually the saga ID combined with the step name.

← Chapter 8 (Eventual Consistency). Sagas are eventually consistent by construction. There's always a window — sometimes seconds long — where some steps are done and others aren't. Anything that reads from an in-progress saga has to tolerate that window, because it cannot be wished away.

← Chapter 11 (Outbox Pattern). Saga events get published reliably through the Outbox pattern. Without it, a crash in the gap between the local DB commit and the Kafka publish strands the saga in an ambiguous state with no automatic way back — the dual-write problem, wearing a saga costume.

→ Chapter 13 (Inventory Reservation). The inventory-reservation step is involved enough to earn its own chapter. Optimistic reservation, reservation timeouts, and the double-reserve problem all live there.

The one thing to carry out of this chapter: sagas don't make the complexity of distributed transactions go away. They make it explicit, nameable, and recoverable, which is the most you can actually ask. Two-phase commit hides the failure modes behind a coordinator you're trusting to handle them — and you discover, at the worst possible moment and never a better one, that the trust was misplaced. A saga puts every failure mode on the table on day one. Compensation is just another code path: you can read it, test it, monitor it, retry it. That isn't less work than 2PC. It's the same work, done with your eyes open.

Appendix A: Reference Implementations

The chapter keeps the code out of the narrative on purpose. Here it is, collected and annotated — read the prose for the idea, come here for the literal shape. Both listings are illustrative rather than production-hardened: they exist to make the structure concrete, not to be pasted into your order service and shipped before lunch.

A.1 — Synchronous Chaining Without Compensation

Python27 lines
1def place_order(order_data: dict) -> dict:
2 order = Order.create(order_data)
3 
4 try:
5 payment = payment_service.charge(
6 order_id=order.id,
7 amount=order.total,
8 )
9 except PaymentException as e:
10 order.mark_failed(reason=str(e))
11 raise
12 
13 try:
14 inventory = inventory_service.reserve(
15 order_id=order.id,
16 items=order.items,
17 )
18 except InventoryException as e:
19 # Payment already charged — who refunds?
20 # This catch block is where the real work begins.
21 payment_service.refund(payment_id=payment.id) # What if this fails?
22 order.mark_failed(reason=str(e))
23 raise
24 
25 notification_service.send_confirmation(order_id=order.id)
26 order.mark_complete()
27 return order.to_dict()

This is the naive saga in full: a refund buried in a catch block, with a # What if this fails? comment doing the load-bearing work of an entire state machine. It's worth typing out precisely because it looks so reasonable — the bug isn't in any line you can point to, it's in the quiet assumption that the lines after payment_service.charge are guaranteed to run. They aren't. A process that dies between the charge and the reserve leaves a charged customer and no record that anything is owed back.

A.2 — Orchestration-Based Saga (Temporal)

Python46 lines
1from temporalio import workflow, activity
2from datetime import timedelta
3 
4@workflow.defn
5class OrderSagaWorkflow:
6 @workflow.run
7 async def run(self, order_data: dict) -> dict:
8 order_id = order_data["order_id"]
9 
10 # Step 1: Charge payment
11 payment_result = await workflow.execute_activity(
12 charge_payment,
13 args=[order_id, order_data["amount"]],
14 start_to_close_timeout=timedelta(seconds=30),
15 retry_policy=RetryPolicy(maximum_attempts=3),
16 )
17 
18 if not payment_result.success:
19 await workflow.execute_activity(
20 cancel_order, args=[order_id]
21 )
22 return {"status": "FAILED", "reason": "payment_failed"}
23 
24 # Step 2: Reserve inventory
25 inventory_result = await workflow.execute_activity(
26 reserve_inventory,
27 args=[order_id, order_data["items"]],
28 start_to_close_timeout=timedelta(seconds=30),
29 )
30 
31 if not inventory_result.success:
32 # Compensate: refund the payment
33 await workflow.execute_activity(
34 refund_payment,
35 args=[order_id, payment_result.payment_id],
36 # Temporal retries this until it succeeds or hits a limit
37 retry_policy=RetryPolicy(maximum_attempts=10),
38 )
39 return {"status": "FAILED_WITH_REFUND", "reason": "inventory_failed"}
40 
41 # Step 3: Send notification (no compensation possible)
42 await workflow.execute_activity(
43 send_confirmation, args=[order_id]
44 )
45 
46 return {"status": "COMPLETED", "order_id": order_id}

The same workflow with the failure paths made explicit and the bookkeeping handed to the platform. Note maximum_attempts=10 on the refund versus 3 on the original charge: the compensation is the step you most need to not give up on, because abandoning it is exactly what strands a customer mid-refund. Temporal replays this function from its last completed step if the process dies, which is the entire reason the compensation can be ordinary code instead of a hand-rolled state machine with its own database table and retry loop.

Next: Chapter 13 — Inventory Reservation: the reserve step from this chapter pulled apart on its own terms — optimistic reservation, expiring holds, and the double-reserve race that turns one in-stock item into two confirmed orders.