From Request/Response to Events
The shape of the whole thing
Five stages, from the guarantees a synchronous call gives you for free to the map of every 3 AM page the rest of this atlas is organized around. Read it top to bottom, or jump to the part you came for.
Request/response hands you three guarantees for free; price what each one becomes the instant a call turns into an event.
Events remove temporal coupling and quietly reintroduce logical coupling — "decoupled" is a claim about time, not about dependency.
Three ambushes waiting in a flow you believed was asynchronous, one of which is still synchronous and doesn't know it.
Every sync-to-async conversion owes three questions honest answers before the queue goes in — the conversion, not the broker, is what you're designing.
When this earns its complexity, what to check before converting, and where each 3 AM page you're signing up for gets its own chapter.
Introduction
The queue was for emails. That was the whole pitch, and it was a good one. Checkout was slow because sending the confirmation happened inline, the SMTP provider had a bad afternoon roughly once a month, and nobody wanted a payment to fail because a mail server three vendors away was having feelings. So we put a queue in front of the mailer, checkout published and returned, latency dropped, everyone moved on. Reasonable. I'd approve it again.
Two years later that same queue carried order-placed, order-paid, order-cancelled, refund-issued, and nine other things, because once the pipe existed it was the obvious place to hang the next feature. Analytics, the warehouse integration, and the customer order timeline all read from it. And the afternoon we needed to answer a regulator — what was the exact state of order 4815162342 at 14:07 on the third — the closest thing we had to a source of truth was a topic we'd provisioned to send emails. Nobody had decided that; nobody would have. It accreted, one reasonable pull request at a time, until business-critical truth lived in infrastructure we'd sized for newsletters.
Peripheral async infrastructure accretes load-bearing state until it is a system of record no one chose. Find the side channel in your own architecture that quietly became authoritative.
That is the decision this atlas is about, and the uncomfortable part is that good, senior teams make it exactly the way we did: not in a design review, but as a gradient nobody was standing at the top of. You do not usually adopt event-driven architecture. You back into it. And it matters, because the moment a synchronous call becomes an event, three things request/response gave you for free stop being free — quietly, with no error and no alert. They don't disappear; they become your application's problem. The bill is real; it just arrives later, itemized, usually at 3 AM.
Here's the ground we'll walk together:
- The three guarantees request/response hands you silently, and what each becomes the instant a call turns into an event
- Why "decoupling" is an honest word for the benefit and a dishonest one for the cost — and what a shared event schema quietly re-couples
- The three failure modes that ambush the flow you thought was asynchronous, one of which is still synchronous and doesn't know it
- The three questions every sync-to-async conversion has to answer on purpose, because skipping one doesn't remove it — it defers the invoice
- How to read the rest of this atlas: where each 3 AM page you're signing up for gets its own chapter
None of this argues against events — only against adopting them by accident. By the end, a proposal to "just put it on a queue" should tell you exactly which guarantees you're spending, and whether you're getting anything worth the price.
The Problem, Precisely
The trap is that request/response is so good at hiding its guarantees that you only learn their names when you lose them.
Synchronous request/response is not the boring baseline everyone treats it as. It's a bundle of three strong guarantees delivered so smoothly that most engineers have never had to articulate them — which is exactly why giving them up feels weightless right up until it doesn't.
The first is the synchronous answer: when a call returns you know the outcome, and you don't proceed until you have it — a great deal of ordinary correctness comes free from line two not running until line one actually happened. The second is total ordering: two calls into one process happen in the order you issued them, and everyone agrees. You've never thought about this because you've never not had it. The third is read-your-writes: write a row, read it back on the next line, it's there. So obvious it sounds silly to name — which is how much you'll miss it.
Now turn one synchronous call into an event — publish, return, handle it downstream later — and price each guarantee.
The synchronous answer becomes a receipt. The 200 OK no longer means "the work happened"; it means "the work is scheduled." (That's the receipt-not-the-work reframe from the Internet-Scale Product Systems atlas, where async workflows were introduced — here we're pricing the receipt, not re-deriving it.) The gap between "accepted" and "done" was zero and is now a window: milliseconds when the consumer is healthy, minutes when it's behind, unbounded when it's wedged. You didn't remove the work. You removed your knowledge of when it finishes.
Total ordering becomes per-partition ordering, at best. A stream keeps order only within one partition; shard across P partitions for throughput and the number of orderings the system can legally produce is no longer one. Two events for the same order can share a partition and stay ordered, or land on different ones and race — and the code that assumed "cancelled always arrives after placed" now holds a belief the infrastructure never made. There's no exception for "these arrived out of order." There's just a cancellation processed against an order that, as far as this consumer knew, didn't exist yet.
Read-your-writes becomes eventual consistency — bounded drift that converges, which the Internet-Scale atlas taught as something systems mostly accrete rather than choose. Publish an event, then read a store some consumer will update shortly, and you have a read-your-writes violation waiting for a user fast enough to click before the projection catches up. The window is usually small. "Usually small" isn't a guarantee; it's a description of the days you weren't paged.
- a synchronous answer
- total ordering
- read-your-writes
- a receipt (accepted ≠ done)
- per-partition ordering (a race across keys)
- eventual consistency (converges later, for a "later" you don't control)
Here's what catches experienced people: none of these become errors. Tests pass — one thing at a time on a quiet machine where the consumer is fast and the partition is one — and staging is green for the same reason. The guarantees you gave up were invisible when present, so their absence is too, until production supplies the concurrency, the second partition, and the user who clicks faster than your projection. You didn't make a mistake you could see; you spent something you couldn't.
What "Decoupling" Actually Bills You
Every talk sells events with one word: decoupling. It's true, and it's the most expensive true thing in the field, because it names the benefit precisely and the cost not at all.
What events remove is temporal coupling — the requirement that caller and callee both be alive and responsive inside the same request's lifetime. Under request/response, if the mailer is down, checkout is down; they're welded together in time whether the org chart says so or not. Put a durable stream between them and the weld is cut: checkout publishes whether or not the mailer is up. That's genuine and valuable — availability composes better, a slow consumer stops being the producer's outage. It's the real prize, and why the queue-for-emails was a good idea.
But coupling isn't one thing, and cutting the temporal kind leaves the other standing. Logical coupling — sometimes called semantic coupling — is dependence on shared meaning: producer and every consumer must agree on what an event is called, what it carries, and what that signifies. Request/response has this too, but the contract is small, versioned at the endpoint, and enforced synchronously — send the wrong shape, get a 400 immediately. An event schema is a contract with everyone who has ever subscribed, enforced nowhere in particular, and — because old events live in the log forever — one you can't fully renegotiate later. You didn't decouple the services; you traded a small, loud, temporal coupling for a large, quiet, permanent logical one, then told standup the system was "decoupled now."
"Decoupled" is a claim about time, not about dependency. Two services joined by an event schema are as tightly bound in meaning as they ever were; they've just stopped needing to be awake at the same moment.
This is why "we added a queue, so they're independent" is the most common wrong model in the whole subject, and why it survives review. The diagram genuinely shows a box in the middle and the synchronous arrow genuinely disappeared. The word decoupling is doing motivated work: it describes what got better and lets you skip what got worse. A staff engineer who can say "we removed temporal coupling and took on logical coupling and eventual consistency in exchange" is thinking clearly. One who says "it's decoupled now" stopped reading halfway through the trade.
Failure Modes Worth Naming
The field guide before the framework — the three that ambush the flow you believed was asynchronous. Trigger, the symptom as on-call sees it, why it hides, what it costs.
| Failure mode | Trigger | Symptom | Cost |
|---|---|---|---|
| The hidden synchronous dependency | You made the call asynchronous, but the caller still can't proceed without the consumer's result — a poll or a "check status until ready" loop, and the flow is synchronous again in an event costume | p99 on the "async" endpoint tracks the consumer's health perfectly, which should be impossible for something you decoupled | The worst of both worlds: you pay a stream's operational surface and kept the coupling you adopted it to escape |
| Silent reordering across partitions | Two events for one entity take different partitions and process out of emission order — the default the moment you shard for throughput, not an exotic edge | A small, stubborn rate of impossible states — a cancellation applied to an order that isn't placed yet — "it's only 0.2% of records" | Corrupt state no single log line explains, and a debugging session that opens with "this is impossible" |
| The dropped event that mattered | Publish without confirming the broker accepted it, or ack before the effect is durable — the one event you can't afford to lose vanishes in the redelivery gap | Nothing at all — that's the whole problem — no error, no retry, no dead letter, just a support ticket three days later about an order that was paid and never shipped | The failures you can't count, which are the ones you can't fix |
None of the three is a Kafka bug or a RabbitMQ bug; they're consequences of the guarantee swap arriving as incidents. The broker is behaving exactly as designed. The design was just more expensive than the pitch admitted.
The Three Questions
Everything above collapses into a discipline. When you convert a synchronous interaction into an event — and you should, sometimes; this isn't a temperance lecture — you owe three questions honest answers before the queue goes in, because the queue doesn't answer them, it hides them.
Who needs this answer, and when? The synchronous caller got the outcome immediately. Who needed that, and do they still? If a human is staring at a spinner waiting on the result, "eventually" is a UX regression you're about to ship. If nothing downstream needs the answer inside the request — the email, the analytics event, the cache warm — async is close to free and you should take it. The question isn't "can this be async"; almost anything can. It's "who gets a synchronous answer today, and what breaks for them when it becomes a receipt?"
What happens if it arrives twice, or out of order? Durable streams redeliver — at-least-once delivery, from the Internet-Scale atlas, makes duplicates a guarantee to absorb, not a bug to fix. So the consumer's effect must be idempotent, or you dedup, or one duplicate is genuinely harmless; and if two events for one entity can race, either the key pins them to a partition or the logic tolerates reorder. You don't get to answer "that won't happen." It will; the only question is whether you decided what happens, or let production decide.
Who is the source of truth now? This is the one the emails queue flunked. When state changes flow as events, something is authoritative — the database, the log, a projection — and everything else is a copy allowed to be stale. If you can't name the system of record in one sentence, you don't have eventual consistency; you have several disagreeing stores and a future incident where three dashboards show three numbers. "The topic we set up for emails" is not a decision; it's a confession.
Idempotency, at-least-once delivery, eventual consistency, the system-of-record boundary — those answers reach for nodes taught in the atlases beneath this one. This chapter's contribution isn't any of them; it's the insistence that converting sync to async is a decision that must answer all three on purpose. The conversion, not the broker, is the thing you're designing.
What the Companies Actually Built
Uber didn't move dispatch onto event streams for decoupling. As fleet and trip volume grew, the dispatch path — match rider to driver, hold the driver, confirm, update everyone — stopped fitting in a synchronous request that fanned out to a wall of RPCs, because that design pinned the correctness of a trip to every one of those services being up and fast at the same instant, and at that fleet size something always wasn't. They moved trip state onto durable event streams so the flow could survive a slow participant and be reconstructed from the log. The lesson worth stealing isn't "use streams." It's that they had all three answers: the consumers tolerated a receipt, events were keyed by trip so per-entity order held, and the log was made the authoritative history on purpose. They took the bill because the synchronous version had become the more expensive option — the only good reason to.
Now the example that complicates the advice: our emails queue from the opening. Same technology as Uber's streams, same publish-and-move-on ergonomics — and a disaster, because not one of the three questions was asked. Question one drifted as features quietly began needing those events in-request. Question two was never asked; nobody knew if the consumers were idempotent, and some weren't. Question three was answered by accident, in the worst way, when a topic sized for newsletters became the ledger. The difference between Uber's dispatch and our mailer wasn't tooling and wasn't talent. One team made the decision; the other backed into it.
"emit an event so we can debug"
"move the slow part off the request"
"checkout shouldn't wait on email"
"wait — is that our order history?"
How to Read This Atlas
The north-star question for everything that follows is one you can ask of any event-driven design: what breaks when events are the source of truth, and would I still choose this after the third 3 AM incident? The atlas is organized so each thing that breaks gets its own chapter. (Links are wired as each chapter is written; until then, entries stay plain text.)
Volume I — Why Events is the decision, not the plumbing. This chapter is the decision; Chapter 2 (The Anatomy of an Event) shows how fact-versus-command-versus-notification sets coupling and blast radius before you write a line; Chapter 3 (Brokers, Logs, and Queues) separates queue semantics from log semantics, which share a vocabulary and almost nothing else.
Volume II — The Hard Guarantees is the three properties everyone assumes the broker provides and none get for free: delivery (Chapter 4 — exactly-once is constructed, not a setting), ordering (Chapter 5 — per-entity order is the one you can afford), schema evolution (Chapter 6 — you can't deprecate the past).
Volume III — Event-First Architecture is what follows once the log is the source of truth: event sourcing at system scale (Chapter 7), consumer-owned read models (Chapter 8), workflows and sagas (Chapter 9), and stateful stream processing (Chapter 10) — where your pipeline becomes a database you now operate.
Volume IV — Living With It is the operational reality, which is to say the pages: replays and backfills (Chapter 11), poison pills and dead letters (Chapter 12), observing systems that go blind at the first async hop (Chapter 13), testing the conversation rather than the service (Chapter 14), and multi-region, where every Volume II guarantee fractures again (Chapter 15).
Volume V — Judgment is what a principal gets asked to weigh in on: the anti-patterns (Chapter 16), migrating a CRUD system without a big-bang rewrite (Chapter 17), and — the chapter that should make you trust the rest — when not to use events at all (Chapter 18). If this atlas does its job, you'll reach for that last one as often as any pattern in the book.
What breaks when events are the source of truth — and would I still choose this after the third 3 AM incident?
(Today only this chapter is live — its row is marked "You are here." The rest link as each is designed and released.)
The Principal Engineer's Perspective
When this earns its complexity — and when it doesn't. Events earn their bill when you genuinely need one of four things: temporal decoupling (a producer that must survive a slow or down consumer), fan-out (many independent consumers of the same fact), replay or audit (a durable history you reconstruct from), or buffering (absorbing a spike the downstream can't take synchronously). If a proposal can't point at one of those four, it's paying the full cost of a distributed system for the aesthetic of arrows on a diagram. And if it needs exactly one of them for exactly one hop, the answer is often a single well-placed queue, not an event-driven architecture — which is what Chapter 18 is about.
The observability that has to exist first. Don't convert a synchronous flow to events until you can see consumer lag and a correlation ID rides through every hop. The synchronous world gave you debuggability free: one request, one stack trace, one answer. The event world takes it away and sells it back as instrumentation you build — lag as the heartbeat of every consumer, correlation IDs (from the Internet-Scale atlas) as the only way to reconstruct "where is my order?" across three topics and five consumers. If you can't answer that today for the synchronous flow, you won't once it's asynchronous. Chapter 13 is the whole flashlight.
The business decision in an engineering costume. "Should this be an event?" looks technical and is mostly organizational. Making the log the source of truth relocates authority: whoever owns that stream now owns a contract with every consumer, present and future, and the power to break all of them with a schema change. The accidental version redraws team boundaries without anyone noticing, until an incident reveals which team is actually on the hook for the ledger. Decide who owns the source of truth before it's load-bearing, because deciding after is called a postmortem.
Cost awareness. The sticker price is a broker. The real price is the standing operational surface — consumer groups, partitions, lag alerts, dead-letter handling, replay tooling, schema governance — plus the cognitive tax on every engineer who now reasons about time, ordering, and consistency to make a one-line change. That tax is paid forever, by everyone. It's worth it at Uber's dispatch scale and absurd for a three-service app that added Kafka to feel modern.
Questions to take back to your team.
- For the flow we're about to make asynchronous — who gets a synchronous answer today, and what specifically breaks for them when it becomes a receipt? If it's "a user staring at a spinner," we have a UX decision, not just an architecture one.
- Name our system of record for this state in one sentence, out loud. If it takes a paragraph or a whiteboard, we don't have one yet.
- Are the consumers of this event idempotent? Not "should they be" — are they, today, and how do we know? Who tested a duplicate?
- Can we see consumer lag right now, and does a correlation ID survive every hop? If not, we're not ready to convert — we're ready to build the observability that lets us.
- Which of the four reasons — decoupling, fan-out, replay, buffering — is this for? If we can't pick one, why isn't it a synchronous call with a retry?
An event is not a way to send a message, it's a decision to make time, ordering, and consistency your application's problem instead of the runtime's. Sometimes right, often worth it — but always a decision, and the teams that suffer most are the ones who never noticed they made it. Ask the three questions out loud, or production will ask them for you, later, with interest.
Exercises
These don't have clean answers, which is the point. Paste any one into an AI and you'll get a confident, well-structured reply in four seconds that skips the only thing that matters — your consumers, your source of truth, your blast radius. The value is in the arguing, not the answer.
Exercise 1 — Audit a real conversion. Take one synchronous call in a system you own and walk it through the three questions as if you were about to make it an event. Who gets the synchronous answer today? What happens on duplicate or reorder? Who becomes the source of truth? The interesting output isn't "convert it" or "don't" — it's discovering which question you can't currently answer, because that gap is exactly the incident you'd be signing up for.
Exercise 2 — Rebuild the ledger, in your domain. Our worked example was an email queue that became an order ledger. Rebuild it in your own system: name your side channel (the pipe you added for something peripheral), its original innocent purpose, the business-critical state it has quietly accreted, and — the part that teaches — the point where the mapping breaks, where your architecture would have caught the drift ours didn't. If it maps cleanly with no break, you may already be living in the failure and calling it normal.
Exercise 3 — Argue the other side. A colleague proposes replacing a working synchronous call — A calls B, gets an answer, moves on — with an event, citing "decoupling." Make the strongest case for keeping it synchronous, then the strongest case against. Name what temporal decoupling would actually buy here, and what logical coupling and eventual consistency it would cost. "It depends" is allowed only if you say precisely what it depends on.
Exercise 4 — Find the hidden synchronous dependency. Sketch a flow you call asynchronous, then find where a caller still can't proceed without a consumer's result — the poll, the status check, the callback wait. Is the async hop buying anything, or are you paying a broker to keep the coupling you meant to remove? If it's buying something, name it. If not, that's a queue you could delete on Monday.
Connections
← The async volume (Internet-Scale Product Systems atlas) and the system-of-record boundary (Data Platforms atlas). This chapter stands on those and does not re-teach them. Event streams, at-least-once delivery and its duplicates, out-of-order processing, lost events under fire-and-forget, eventual consistency, idempotency, the system-of-record boundary — all assumed here. What's new is the framing: those were mechanisms; this chapter prices the decision to adopt them. If a recap felt thin, that's deliberate — the depth lives there.
→ The Anatomy of an Event (Chapter 2). The three questions decide whether to convert; Chapter 2 decides what shape the event takes when you do — fact versus command versus notification, a distinction that governs coupling and blast radius as much as anything here. It's the lens the accidental-adoption failures were missing.
→ Delivery, Ordering, and the Source of Truth (Chapters 4, 5, 7). Each of the three questions becomes a chapter: "arrives twice" is constructed exactly-once processing (Chapter 4), "out of order" is per-entity ordering via partition keys (Chapter 5), and "who is the source of truth" is event sourcing at system scale (Chapter 7), where the log becomes the ledger on purpose.
→ Poison Pills and When Not to Use Events (Chapters 12, 18). The dropped-event failure gets its routing answer — the dead-letter queue — in Chapter 12. And the whole argument for asking the three questions honestly reaches its conclusion in Chapter 18, which is willing to answer "don't."
Next: Chapter 2 — The Anatomy of an Event: why naming a message "OrderEvent" defers the most important decision you'll make about it, and how fact, command, and notification each set your coupling before you write a line.