Stream Processing

A stream is an unbounded, ordered sequence of events. That's the textbook definition, and it's the kind of sentence that teaches you nothing, so set it aside. Here's the version that matters: open the Uber app, tap to request a ride, and you haven't made a request — you've started a workflow. Your tap emits an event. Matching logic reads it and emits another. Payment runs. A driver gets pinged. Your screen updates. Not one of those steps happens in the same function, the same service, or necessarily the same datacenter as the others. They happen in sequence, across time, stitched together by the stream running between them. The "looking for a driver" spinner you get back isn't the work — it's a receipt confirming the work has been queued. The work is happening downstream, in a pipeline the app can't see.

That shift — from request to workflow — is subtler than it looks, and it catches experienced people because they're experienced. Request-response is intuitive — it mirrors conversation. Client asks, server answers; you've been having that conversation since your first CRUD app. Streams mirror nothing you already know. They're a conveyor belt: an event rides through a sequence of stations, each one transforming it a little, each running at its own speed, nobody waiting on anybody. The senior engineer's problem isn't that streams are hard to understand. It's that a decade of request-response instinct is the wrong instinct here, and instinct is the last thing you think to question. Netflix turns "user watched X" into a recommendation this way. DoorDash moves "order placed" through "restaurant confirmed," "driver assigned," "delivered." Stripe takes one payment event and fans it to fraud detection, analytics, and billing at once — three independent consumers, none of them waiting on the others.

Here's the ground we'll walk together:

• Why a multi-step workflow stuffed into one synchronous request buckles at scale — and the three things every team builds first instead of streams, each of which works in the demo and fails in production in its own instructive way
• The difference between a stream and a queue, which sounds like pedantry until it's the root cause on an incident timeline: a queue forgets a message the moment it's read; a stream remembers, and that single property is most of the game
• The failure modes that don't exist until you're at scale and then never quite leave — lost events that vanish without a trace, duplicates that bill the customer twice, out-of-order processing that charges the card before it checks inventory, late data that silently corrupts your aggregations (the out-of-order one ambushes the engineers who already understand the other two)
• The core patterns that replace the synchronous chain — partitioned Kafka topics, stateless vs. stateful processing, exactly-once semantics — and the three trade-offs that actually decide between them: throughput vs. latency, ordering vs. parallelism, and what state really costs once you scale it
• What Netflix, Uber, and DoorDash actually built, and what Kafka, Flink, and Spark Streaming are each good at — and where each one quietly lets you down, because the lesson is never "reach for the impressive tool," it's "set your latency SLA from the business, not from what sounds fast in a design review"

A multi-step workflow stuffed into one synchronous request is a state machine with no place to keep its state.

Here's the canonical version, stated without abstraction. A user places an order. Three things now have to happen: charge the card, reserve inventory, send a confirmation. Who coordinates that? In the simplest possible system, the API does, synchronously — call payment, then inventory, then notification, then return success. This works flawlessly in staging, where every service answers in 50 milliseconds and the whole chain closes in 200. Production is less accommodating. Payment averages 500ms and spikes to 4 seconds at p99. The email vendor has slow nights. Inventory sits behind a service that makes its own downstream calls. Your client timeout is 30 seconds — comfortable, right up until you have ten thousand orders in flight at once, each pinning a thread, each waiting on a chain of services that may or may not answer this decade.

The latency problem is the obvious one. The durability problem is the one that bites you first in production.

If the API crashes after the charge succeeds but before the confirmation goes out, the money moved and the notification didn't. If inventory fails mid-chain, you've charged a card for something you can't ship. If the notification service is just slow enough to blow the client timeout, the client sees an error and retries — and now you've charged the card twice, unless you built idempotency into the payment step (which you did, because you read Chapter 2). That's the shape of the thing: a multi-step workflow crammed into a synchronous request is a state machine with no durable state, held together by optimism and good timing. The instant any single step misbehaves, the state of the whole workflow becomes a mystery you get to reconstruct from logs.

This is exactly what Uber ran into — not as a thought experiment but as a scaling reality. At a million rides per hour, each ride is five or six discrete operations: request, match, payment, driver notification, rider notification, location updates. None of them tolerate each other's latency. The matching engine can't wait for payment. The notification service can't block matching. Each step needs to run independently, at its own pace, triggered by the completion of the previous one rather than synchronized to it. The only structure that supports that is a stream. The ride request becomes an event. That event fans out. Each service subscribes, reacts, and publishes its own event when it's done. The workflow proceeds asynchronously, with no single thread holding it all together.

This is the structural argument for streams, and it surprises people because it isn't about performance: it's about durability. Streams make the state of a multi-step workflow durable by default, because the events themselves are durable. If a consumer crashes mid-processing, the event it was chewing on doesn't vanish — it's still in the stream, waiting to be picked up again. You can reason about where a workflow is at any moment by looking at which events have been produced and consumed. That is a different universe from "check the API logs and hope."

I have watched three different teams arrive at streams, and every one of them built the same three things first, in the same order, each convinced their situation was the exception. All three work in the demo. None of them work at scale. They're worth walking through, because each one fails in a way that teaches the next.

Synchronous request chains are the first attempt: the API calls Payment, which calls Inventory, which calls Notification, which returns up the chain to the client. The appeal is honesty — the logic lives in one place, the failure modes are visible, a stack trace tells you everything. The cost is that latency is additive. If each downstream hop takes a second, the total is four, and that's the median. The p99 stacks worse. At a million events an hour, four seconds apiece, you need roughly 1,100 threads just to keep up — and that's the day you learn that thread-pool exhaustion, not the slow service that caused it, is the thing that actually shows up in the incident channel.

Cascading failure is the second tax. If the notification service has a slow hour — not down, just slow — the slowness propagates backward through the chain. Inventory queues behind it. Payment queues behind Inventory. The API response drags. Client timeouts climb. Retries stack. One sluggish third-party email vendor has now taken down the entire checkout flow. This is precisely the scenario Chapter 4's circuit breaker exists to contain, and it's also the reason that bolting a breaker onto every hop in a synchronous chain turns your architecture diagram into something that would make an electrician nervous.

Fire-and-forget with background jobs is the second attempt, and it's a real improvement: the API processes the order, enqueues a job, and returns to the client immediately. The job handles payment, inventory, and notification on its own time. This is closer to the right shape, and it fails in more interesting ways. The classic: the job executor crashes after payment succeeds but before inventory is reserved. The money's gone, inventory is untouched, and the order is sitting in a half-finished state nobody modeled — because the framework made half-finished feel impossible. Most job systems are also ephemeral: the queue lives in memory, or in a database that was never designed for this workload, with no native way to track where a four-step workflow is when step two fails and steps three and four need to compensate for it. So you write a state machine by hand, inside the job, which is a state machine with nowhere durable to keep its state. It's a fun way to learn why event sourcing exists.

Cron-based batch processing is the third attempt, and it doesn't deserve much ceremony: every five minutes, process all pending orders. The latency is five minutes. The ordering guarantee is whatever ORDER BY created_at gives you, which turns out not to be much under concurrent writes. And at enough volume the batch takes longer than five minutes, the next run starts before the last one finishes, and now two jobs are processing overlapping orders — idempotency, again, arriving uninvited. The throughput ceiling is whatever you can clear in five minutes, which approaches "one million events per hour" at roughly the speed a wet Tuesday approaches summer.

The three first attempts, side by side:

Adopt proper stream processing and the naive failure modes go away. A new set comes along for the ride. These aren't theoretical; they're what fills the postmortem template.

Lost events are the simplest failure and the hardest to debug, because by definition they leave no trace. A consumer reads an event, starts processing, crashes before writing the result, and restarts with its offset already advanced past that event. The event is gone. Payment never ran. Nobody knows. The observable symptom is an order stuck in "processing" that never moves — which, from the outside, looks exactly like a slow order, which is why it goes unnoticed for so long. You catch these with consumer-lag monitoring plus order-state timeouts, but only if you built both, which most teams do in the canonical order: streams first, observability eighteen months later, during the third incident they couldn't explain.

Duplicate processing is the mirror image: a consumer reads an event, processes it, writes the result, then crashes before committing its offset. On restart, the event gets reprocessed. If the processing isn't idempotent — if charging a card is a side effect that can't be repeated safely — the customer pays twice. Chapter 2 covers idempotency in depth, and it is not optional here. If you haven't read it, stop here and go read it; the rest of this chapter assumes it.

Out-of-order processing is the one that surprises the engineers who already understand the other two. In a multi-partition Kafka topic, events on different partitions are processed by different consumers in parallel. If "reserve inventory" and "charge card" land on different partitions, both can run at the same time — and if the "charge card" consumer happens to be a hair faster, the card is charged before inventory is confirmed available. You've now billed someone for something you can't fulfill, and nothing in the code looks wrong. The fix is to partition on the entity that needs ordering — order_id here — so every event for a single order lands on the same partition and is processed in sequence by the same consumer. The cost is that parallelism is now capped at the number of distinct order IDs, which is almost always fine, and is far better understood on a whiteboard than discovered in production data.

Watermarking issues show up in systems that do time-windowed aggregations. The textbook case: Uber computes surge pricing from the number of ride requests in the last five minutes. A driver's phone loses signal, buffers five minutes of GPS events, and reconnects — dumping events into the stream with timestamps from five minutes ago. Does the surge calculation reprocess the last five minutes? Ignore the stragglers? Most stream processors handle this with watermarks — a marker that says "I've seen everything up to timestamp T; anything older than T that shows up now is late." What counts as "too late" is a number you choose, and choosing it wrong produces the kind of anomaly that is genuinely difficult to reason about at 2 a.m. with a dashboard that disagrees with itself.

The four that fill the postmortem template:

The fundamental shape of a Kafka pipeline: a producer writes an event to a topic, and multiple independent consumers read from that topic at their own pace. The producer neither knows nor cares how many consumers exist or how far behind any of them are. When an order is placed, the "order placed" event goes to Kafka once. Payment, inventory, and notification each consume it independently. Each does its work. None waits on the others.

The mechanism that makes this work is the consumer offset: each consumer holds a pointer to its current position in the stream. Payment might be at offset 1,000. Notification, slower this afternoon because its vendor is having a day, is at 940. Both are valid — neither is wrong, neither blocks the other. That decoupling is the whole point. In a synchronous chain, a slow notification service slows checkout. In a Kafka pipeline, a slow notification service is the notification service's problem, sealed inside its own consumer group, invisible to everyone upstream.

Here is the thing that trips up nearly everyone new to Kafka: the ordering guarantee is partition-local, not global. All events within one partition are ordered; events across partitions are not. This has real design consequences. Need every event for a given user processed in order? Partition by user_id. Need it for a given order? Partition by order_id. Need global ordering — every order's events interleaved and processed in strict arrival sequence across the whole topic? Then you're on a single partition, which means a single consumer, which means no horizontal scalability at all. Global ordering and horizontal scalability are, in Kafka, mutually exclusive. You don't get to want both. You pick the scope of ordering you actually need and partition for it.

Stateless processing handles each event in isolation. The consumer reads an event, does some work — enrichment, validation, transformation — and emits output, with no memory of anything that came before. Enriching an order event with the customer's name from a user service is stateless: one event, one lookup, one enriched output. It scales horizontally with zero coordination between consumer instances, which is a genuinely pleasant property and worth appreciating before we ruin it.

Stateful processing means the output depends on events that came before. A customer's lifetime value needs every prior purchase. Spotting an emerging fraud pattern needs the last twenty transactions. Counting how many times a user hit a specific error in the last hour needs a per-user counter that survives across events. That state has to live somewhere. In practice, processors like Flink keep it in a local embedded store — usually RocksDB — that lives right next to the consumer process. Local state is fast, no network round-trip, but it creates a recovery problem: if the process dies, the state dies with it. Managed processors handle this by periodically checkpointing state to durable storage, so a restarting consumer can restore from the last checkpoint instead of replaying the entire stream from the dawn of time. How often you checkpoint is a direct trade between recovery speed and storage cost.

The confusion point engineers hit here is sharp: stateful processing with a local store is not the same animal as stateful processing with a shared external database. Both have the word "state" in them; their failure modes live in different universes. Shared external state lets every consumer instance read and write the same counters — which sounds appealing right up until you're debugging a race condition in a Redis cluster at 3 a.m. Local state with checkpointing keeps each partition's state with the consumer that owns that partition, which sidesteps the concurrency problem entirely — but only works if your partitioning puts all related events on the same partition. Partition by user_id if the state is per-user. Get the partitioning wrong and your local state is quietly computing aggregates over a random subset of each user's events, which is incorrect in a way the output will never confess to.

Exactly-once is the thing everyone wants and almost nobody actually has. The phrase gets thrown around loosely in stream-processor marketing, and the looseness is exactly where people get hurt. What exactly-once means in practice is: no duplicates in the output, assuming the output sink supports idempotent writes. It does not mean processing happens once. It means the observable effect of processing happens once, even when the processing itself happens twice.

The mechanism is idempotent writes plus careful offset management. An event is read at offset N. The consumer processes it — say, records a payment — using a unique key derived from the event (payment_id = "payment_12345"), so that a second attempt at the same write is a no-op. Only after the write succeeds does the consumer commit offset N+1 back to Kafka. If it crashes between the write and the commit, it restarts, reprocesses offset N, attempts the same idempotent write, finds it already done, and advances. The customer sees one charge. The processing happened twice; the effect happened once. (Full code: Appendix A.1.)

The insight that changes how you think about this: exactly-once is a property of the system, not the processor. Flink and Kafka together can give you transactional guarantees between the stream and a Kafka sink, because both speak the same two-phase commit protocol. But the moment your output goes to an HTTP API, a database that won't do idempotent inserts, or any sink that can't be rolled back, exactly-once becomes your problem, not the framework's. And most real pipelines write to exactly those sinks. Which means most real exactly-once guarantees are built on idempotent application logic, not framework magic. Build the idempotency first. The framework can help you; it cannot do it for you.

Throughput vs. latency is the dial you turn most often. Batching inside the consumer — accumulate a thousand events, then process the batch — dramatically lifts throughput by amortizing per-batch overhead. It also adds latency equal to however long the batch takes to fill. Netflix's recommendation pipeline runs in micro-batches, refreshing recommendation lists every two minutes, and nobody on Earth notices that the suggestion for what to watch next is 90 seconds stale. Uber's dispatch pipeline processes location events as they land, sub-second, because matching a rider to a driver depends on where drivers are now, not two minutes ago. The rule of thumb: if staleness changes the user's decision in the moment, process in real time; if staleness only dents the quality of a background computation, batch it and pocket the throughput.

Ordering vs. parallelism is the trade that ambushes engineers arriving from queue-based systems. One partition buys strict ordering and zero parallelism. More partitions buy parallelism at the cost of global ordering. For most consumer systems the resolution is to partition by the entity that needs ordering — user_id for user activity, order_id for order processing — and accept no ordering across users or orders. In a system doing a million orders an hour, the fact that order 123 and order 456 have no defined order relative to each other is irrelevant. The fact that every event for order 123 is processed in sequence is essential. Partition for the second; let go of the first.

State cost is the trade that stays invisible until you scale a stateful pipeline from a hundred thousand events a day to ten million. Stateless processing scales linearly — double the consumers, double the throughput. Stateful processing doesn't, because state has to be sharded: each partition's state lives with its consumer, so scaling out means redistributing state across more consumers, and redistribution means migration. Kafka Streams and Flink handle this through rebalancing, but rebalancing isn't free. A rebalance on a stateful pipeline doing real-time fraud signals introduces processing latency at the worst conceivable moment — while you're adding capacity because load is already high. Set the partition count generously high at topic creation; changing it later is one of the more unpleasant afternoons you can spend with a Kafka cluster.

The three dials, at a glance:

Netflix: Real-Time Recommendations

Every time a member finishes an episode, Netflix produces a "user watched X" event. It flows into a stream — Kafka, here — and multiple consumers subscribe: the recommendation engine, the analytics pipeline, the billing system, the content-licensing ledger. None of them know about each other. If billing goes down for an hour, the recommendation engine keeps running; when billing comes back, it replays from its last committed offset and catches up. The Kafka topic is the single point of coordination, and everything else is decoupled from everything else.

The recommendation pipeline is stateful. Deciding what to suggest next means knowing what a user has already watched — genre leanings, recent history, how long they hung on before abandoning a show. That state lives in the consumer's local store, checkpointed periodically to S3. The finished recommendation list for each user is materialized into Redis, so opening the app is a cache lookup, not a computation.

The latency Netflix accepts here is real and deliberate: two minutes. They run the pipeline as micro-batches, refreshing recommendations for batches of a thousand users at a time. Why tolerate two minutes? Money. Real-time personalized recommendations for 200 million concurrent users would mean a pipeline processing two million events a second at sub-second latency, with state for two hundred million users sitting in memory. The infrastructure bill for that pipeline dwarfs the revenue difference between "your recommendation updated in 100ms" and "your recommendation updated in two minutes." Netflix did the arithmetic and chose the batch. The lesson isn't that streaming is expensive — it's that latency SLAs should be set by business requirements, not by what feels technically impressive in an architecture review.

Uber: Dispatch and the Watermark Problem

Uber's dispatch system processes three streams at once: ride requests, driver location updates, and matching decisions. Ride requests are low-volume but time-critical — a request that sits unmatched for thirty seconds is a failed experience. Location updates are the firehose: every active driver pings every four seconds, which at a million active drivers is 250,000 events a second. The matching logic consumes both streams and produces a third — driver assignments.

The wall Uber hit was out-of-order location updates. A driver's phone drops signal, buffers location events, reconnects, and publishes them — and now the stream holds location events timestamped sixty seconds in the past, interleaved with current events from everyone else. If the matching engine trusts stale location data, it matches riders to where a driver was a minute ago, dispatch quality degrades, and ETAs go fictional. The fix is watermarking: the matching engine only honors events timestamped within the last sixty seconds, and discards the rest. That sixty-second window wasn't a guess — it was calibrated to the 99th-percentile reconnection time for mobile devices, so the overwhelming majority of legitimate late events fall inside it, and the tail that gets dropped is genuine staleness rather than ordinary network jitter.

This is the kind of operational detail that never makes the conference talk but entirely determines whether the system behaves at scale. Watermarks are a trade: too tight and you discard real data; too loose and stale data poisons your aggregations. There's no clever default. The right answer is empirical — measure your actual late-arrival distribution in production before you tune the watermark, not after the surge pricing goes haywire.

DoorDash: The Order State Machine

DoorDash's order pipeline is a stateful stream that walks each order through a five-stage lifecycle: placed → restaurant confirmed → driver assigned → picked up → delivered. Every transition is an event. The state machine for each order lives in the consumer's local store, keyed by order_id.

The operational reality is that every stage of this pipeline fails constantly. Restaurants decline orders. Drivers cancel mid-route. Customers change their minds while the food is on the grill. Each failure demands a compensating action — refund, reassign, notify — and those actions have to be triggered by the stream, not by a cron job politely checking order state every five minutes. The stream is what makes the system feel alive: the moment a "restaurant declined" event lands, the consumer that owns that order's partition sees it, updates local state, and emits a "reassign or refund" event, usually inside a second.

The quiet win here is observability. Because every state transition is an event in the stream, DoorDash can replay the log for any order and reconstruct exactly what happened, in what sequence. When a customer calls asking why their order was cancelled, support replays that order's event history and reads the precise chain of events. Compare that to debugging a workflow that lived inside a synchronous API: by the time the ticket arrives, the context is scattered across five services' logs — if it was logged at all — correlated by a trace ID that someone, hopefully, remembered to thread through the right headers. The event stream is the audit trail, and it costs nothing extra to produce, because it's the native output of the system rather than an afterthought bolted on for compliance.

Apache Kafka is the de facto standard for distributed event streaming, and it earns the title. Topics, partitions, consumer groups, durable retention — it handles the broker problem well and has a client library in every language that matters. Its weaknesses are operational: clusters are non-trivial to run, ZooKeeper (or KRaft, in newer versions) adds coordination complexity, and consumer rebalancing under load can hand you latency spikes at precisely your highest-traffic moment. Confluent Cloud abstracts most of this away, at a price that starts to make sense past a certain scale.

Apache Flink is the right tool for stateful stream processing with exactly-once semantics. It does checkpointing, state recovery, and genuinely hard operators like windowed joins correctly — the kind of thing that looks simple in a design doc and is brutal to implement yourself. The operational cost is real: Flink clusters are fiddly to tune and debug, and the API has a learning curve that is not a gentle slope. Reach for it when you need what it provides; don't deploy it as a general-purpose stream processor when Kafka Streams would do.

Kafka Streams and Flink both give you stateful processing; the difference is the operational model. Kafka Streams is a library that runs inside your application process — no separate cluster, no extra infrastructure — and suits moderate-throughput, moderately-complex pipelines. Flink is a separate processing cluster built for high-throughput, complex pipelines with joins, windows, and exactly-once guarantees. Start with Kafka Streams; graduate to Flink when the complexity actually warrants it, not when it sounds impressive.

Spark Streaming uses micro-batching under the hood: it accumulates events for a configurable interval (typically 500ms to a few seconds) and runs each batch as a mini Spark job. That makes it easy to write — familiar DataFrame API, SQL support — and easy to slot into the Spark ecosystem (Spark SQL, Delta Lake, MLlib). The cost is latency: sub-500ms response times are structurally off the table. If your pipeline can live above that floor, Spark Streaming's operational familiarity is a genuine advantage, especially for teams already running Spark for batch analytics who would rather not adopt a second processing engine to learn, staff, and page on.

Google Pub/Sub and AWS SQS/Kinesis trade control for operational simplicity — the right call for teams that need reliable delivery but have no appetite for running Kafka. The gotcha lurks in SQS: it's a queue, not a stream. Messages are deleted on consumption — no replay, no consumer groups in the Kafka sense. Kinesis and Pub/Sub sit closer to streams and support replay within their retention window. At high scale the managed cost-per-million-messages starts to sting; at moderate scale the operational savings more than pay for themselves.

Exercise 1: The out-of-order payment.

Your order pipeline has two event types on the same Kafka topic: InventoryReserved (from the Inventory service) and PaymentCharged (from the Payment service). Both use the default partitioning — a hash of the service name — meaning InventoryReserved and PaymentCharged events for the same order_id routinely land on different partitions, handled by different consumer instances. Your fulfillment consumer sees this interleaving for order_456: it processes PaymentCharged first, then InventoryReserved. The card is charged, then inventory turns out to be unavailable, and a refund kicks off. The customer is unhappy. So are you.

Design a partitioning strategy that prevents this. What key do you partition on, what does that require of the producers, and what happens to your throughput?

Hint: Partition both event types by order_id. Producing to a specific partition means the producers must set the partition key explicitly. Throughput is bounded by the number of unique order IDs in the window — high enough that it isn't a real constraint. The trade is that you lose ordering across orders, which you never needed.

Exercise 2: Consumer lag at 10,000 messages.

Your payment consumer is 10,000 messages behind. You have three options: (a) add more consumer instances, (b) optimize the processing logic, (c) skip messages older than one hour. Walk through when each is the right call.

Option (a) is right when the logic is sound but there simply aren't enough consumers for the volume — lag is a capacity problem, not an efficiency one. Add consumers up to the partition-count ceiling.

Option (b) is right when profiling shows each message taking longer than it should — a slow external call, an unindexed query, needless work per event. Scaling horizontally just buys you more copies of the same inefficiency.

Option (c) is right when the lag is genuinely stale data whose processing would do more harm than good — notifications for orders that finished an hour ago, recommendations built on obsolete signals. Not all lag is worth recovering.

The failure mode to avoid: choosing (a) when the real problem is (b). A hundred inefficient consumers are a hundred copies of the bottleneck — and now you've also blown through your database connection pool.

← Chapter 5 (Retries). Stream consumer retries are subject to the same thundering-herd dynamics as HTTP retries. A consumer that fails an event and retries immediately can, at volume, kick off a retry storm that amplifies load on everything downstream. Backoff and jitter matter here for exactly the reasons they mattered there.

← Chapter 2 (Idempotency). Stream processing requires idempotent sinks. This is not optional. If your output isn't idempotent, your exactly-once guarantee is marketing copy. If you haven't read Chapter 2, exactly-once stream processing is the single best argument for why that chapter exists.

→ Chapter 8 (Event Sourcing). Event sourcing takes the stream one step further: instead of using events to drive downstream workflows, it makes the event log the authoritative source of truth, deriving current state from event history. The stream stops being plumbing and becomes the database.

→ Chapter 9 (Eventual Consistency). Streams produce eventual consistency by construction: when "user updated email" propagates to three downstream services, each updates its own read model on its own schedule. The models are eventually consistent with the event that triggered them. Chapter 9 is about reasoning honestly about the size of that window and what it costs you.

→ Chapter 12 (Saga Pattern). Sagas use streams as the coordination mechanism for distributed transactions. Each step publishes an event on success or failure; compensating actions fire by consuming those events. The stream is what makes a saga observable and recoverable. Strip it out and a multi-step distributed workflow is a state machine with nowhere to keep its state — the exact fire-and-forget problem from the naive solutions, reached from the opposite direction.

The intuition to carry out of this chapter: the response is not the work. In a request-response system you can fool yourself into believing the 200 OK means something finished — and at small scale, it did. A stream forces the honest version. The tap emits an event; the event is durable; the work happens later, downstream, in stages that fail and recover on their own clocks. Everything hard about stream processing — ordering, state, exactly-once, lag — is the price of making that honesty operational. And everything that makes streams worth the price comes from the same place: the workflow stops living in a thread that can crash and starts living in a log that can't. Find the log, and you can always answer the one question that matters at 3 a.m. — where is this workflow, right now?

The chapter keeps the code out of the narrative on purpose. Here it is — the full version of what the body describes in plain language. Read the prose for the idea; come here when you want to type it out. It's illustrative rather than production-hardened: the metrics, serialization, connection management, and error-handling ceremony the real thing demands are left out so the one idea that matters stays in view.

A.1 — Exactly-once processing loop (idempotent write + offset management)

1import kafka
2import database
3 
4consumer = kafka.KafkaConsumer(
5    "orders",
6    group_id="payment-processor",
7    enable_auto_commit=False,  # manual offset management is required
8    bootstrap_servers=["kafka:9092"],
9)
10 
11for message in consumer:
12    order_id = message.value["order_id"]
13    amount = message.value["amount"]
14 
15    # Idempotent write: INSERT ... ON CONFLICT DO NOTHING
16    # The conflict key is (order_id, "payment"), unique per order.
17    # If this consumer crashes after the insert but before the commit,
18    # the re-processed message hits the conflict clause and is a no-op.
19    database.execute(
20        """
21        INSERT INTO payments (order_id, amount, status, created_at)
22        VALUES (%s, %s, 'charged', NOW())
23        ON CONFLICT (order_id) DO NOTHING
24        """,
25        (order_id, amount),
26    )
27 
28    # Commit offset only after the write succeeds.
29    # Crash between write and commit → reprocess → idempotent no-op → commit.
30    # Crash before write → reprocess → write → commit. No loss, no duplicate.
31    consumer.commit()

The enable_auto_commit=False is load-bearing. Auto-commit advances the offset on a schedule, independent of whether processing succeeded — it is the mechanism by which "at-most-once" behavior hides inside frameworks that advertise at-least-once. With auto-commit off, the offset only advances when you explicitly commit after a successful write. The pair — idempotent sink plus manual offset commit — is what produces exactly-once observable behavior.

Next: Chapter 8 — Event Sourcing: when the event log is the database, not a side effect of it.