Event Sourcing

Most databases are lying to you.

Not maliciously — they're doing exactly what you told them to. When a user changes their email, you run UPDATE users SET email = '[email protected]' WHERE id = 42, and the database faithfully forgets what was there before. You asked it to remember the present, so it threw away the past. That isn't a bug. That's the feature you paid for.

It's the right feature for most problems. It's a catastrophe for a few.

Picture that one row a week later, when something's wrong with it. Was the email changed by the user at 10:05, then changed again by a support agent at 10:07 who had no idea the first change had landed? Was it overwritten by a batch job chewing through the wrong account? Did someone ship a bug that rewrote a whole column? The row can't tell you. It wasn't there for the journey — it only kept the destination.

Event sourcing flips the model. Instead of storing current state, you store history: a sequence of immutable facts about what happened. EmailChanged(from='[email protected]', to='[email protected]', by='user', at=2024-03-01T10:05:03Z). The current email — what it is right now — isn't a row you read. It's something you derive by replaying those facts in order. It isn't gone. It's computed.

This is how Git works, and the comparison is worth holding onto, because you already trust it with your life's work. Git doesn't store the current state of your repository; it stores commits, and the current state is what you get when you apply them all in order. Rewind to any commit and you have a working snapshot. Diff any two points. Blame any line. The log is the truth, and the working tree is just the most recent thing you can fold the log down to.

Here's the part that trips up experienced engineers, and it's worth being honest about why: everything in your career has trained you to treat current state as the truth and history as a nice-to-have you bolt on later. Your ORM, your REST resources, your users table, every CRUD tutorial you've ever read — all of them put the present in the center and shove the past into an audit table nobody reads. Event sourcing inverts the thing you've been standing on the whole time. That's not a small mental adjustment, and pretending it is the fastest way to build one badly.

Netflix, Stripe, and Uber all run event-sourced systems at the core of their most critical workflows — payments, view history, ride state. Not because it's fashionable; it isn't, particularly, and it's a genuine operational handful. They do it because their domains leave them no choice. Charges can't be retroactively rewritten. Watch history can't be lost. A ride has to be reconstructable if a service dies in the middle of it. For those problems, the event log isn't one option among several. It's the only data model that makes the guarantee without crossing its fingers.

Here's the ground we'll walk together:

• The event store itself — an append-only log whose only verb is append, and why "current state" stops being a row you read and becomes something you compute by replaying history
• Why CQRS (Command Query Responsibility Segregation) turns up the instant event sourcing does — separating the write path that appends facts from the read path that serves them
• Projections — how one immutable log feeds many differently-shaped read models, and why adding the fifth one doesn't take a meeting with the teams who own the other four
• Snapshotting — the optimization that stops replay from getting slower every year you stay in business
• Where the whole thing comes apart — lost events, stale projections, schema drift, and the replay bug that fixes your data and corrupts everyone else's — which is where we'll linger longer than on the happy path, because the happy path is thoroughly documented and these are the ones that surprise you at 2 AM
• How Stripe, Netflix, and Uber actually run this — three different domains, three different reasons, all of them load-bearing — and the questions a principal engineer answers before adopting any of it, starting with whether to adopt it at all

Mutable state is fine until the day you need to know how a record got the way it is. That day, it isn't fine at all.

A customer emails support: their subscription shows as cancelled, but they renewed two weeks ago and have the credit-card charge to prove it. Your support engineer queries the database. subscription_status = 'cancelled'. When did it change? Unknown — one row, one status, no timestamp on the transition. Who changed it? No idea. There's an audit_log table, but it was bolted on six months after the subscriptions table and only covers changes since. Was there a migration? There was, actually — a data migration last month with a bug that touched about 0.3% of subscriptions. Finding all of them is an afternoon of forensics you hadn't scheduled, on behalf of a customer who is, reasonably, furious.

That's the debugging problem with mutable state in one sentence: by the time you discover the wrong answer, the evidence of how you got there has already been overwritten by the answer.

The corruption problem is sharper. Think about what actually happens on a single "change email" request: update the users table, invalidate the cache entry, update the search index, write a row to the audit log. Four writes, four systems, and any of them can fail independently, in any combination. Users table succeeds but the cache invalidation doesn't — now people see stale data until the TTL saves you. The search index write fails silently — and search index writes do fail silently, in practice — and now a user simply cannot be found by their new email, which they will report as "your search is broken." The audit log write fails and compliance is your problem now. One logical change, four ways to end up in a state nobody designed.

This isn't hypothetical, and you don't need a famous outage to have seen it. Anyone who has run a payments system has watched a charge show one status in the service that wrote it and a different status in the service reading from a replica that hadn't caught up — the two of them disagreeing about whether a customer just paid you. Stripe's charges are event-sourced precisely because a status column invites that disagreement and an append-only log doesn't. Mutable state with synchronous replication is a consistency problem you re-solve on every single write. Immutable events with async projection is a consistency problem you solve once, in the projection layer, sitting down, where you can actually think.

The audit requirement is what finally turns all of this from an annoyance into an engineering emergency. "Show me every change made to this customer account in the last six months" is a hard requirement in financial services, healthcare, and a growing list of regulated industries — and if your answer is "join five tables and cross your fingers," the regulation is unmoved. You needed the audit trail before the auditor asked for it. Event sourcing hands it to you for nothing, because the audit trail is the source of truth, not a shadow table you maintain in parallel and pray stays in sync.

When you realize you need history, the natural instinct is to add history on top of the system you already have. I've watched several teams arrive at event sourcing, and almost all of them tried the same two or three things first — each one reasonable, each one a trap with a timer on it.

Write-once versioning looks elegant on a whiteboard: don't update the row, insert a new one. users_v1, users_v2. Current state is whichever row has the highest version. This works beautifully until you meet the user who has changed their email forty-seven times — mobile apps with autocomplete bugs are remarkably creative about this — and your "get current state" query is now joining across forty-seven rows per user on a path you hit on every API request. Version tables grow without bound, indexes bloat, and "reconstruct current state" becomes a phrase that makes the query planner visibly nervous. You haven't solved the problem. You've deferred it and made every read slower in the meantime.

The audit table is the next move: keep the main table for current state, write a parallel row to audit_log on every change. Now you have two sources of truth, which is exactly one more than the number that can ever agree under load. The main table is what the application reads; the log is the paper trail. This works until the main table update succeeds and the audit_log write fails, at which point you've produced a mutation with no audit record — which is, of course, precisely the mutation compliance will ask about. You can wrap both in a transaction to make them atomic, but now every update holds a lock across two tables, which is a decision you will revisit, loudly, at scale.

And there's a deeper rot: the audit table is secondary. The application doesn't derive current state from it — it reads from users and treats the log as a shadow. The day the two diverge, because of a migration or a bug or a transaction that rolled back after the audit row already committed, you have no canonical way to decide which one is right. "What is the current email for user 42?" now has two answers and no tiebreaker, which is somehow worse than having none.

Kafka as an afterthought is the most common variant in modern systems, and it's the one that fools people who've shipped a lot of software — precisely because it has events in it, and events feel like event sourcing. You do the database write, then publish an EmailChanged event to Kafka. Downstream consumers light up, you get an event history, everyone nods. But the database is still the source of truth and Kafka is a side effect hanging off the end of it. You have events. You do not have event sourcing. You have a database that also sends a newsletter. And the moment the database write succeeds but the Kafka publish fails, your log has a silent gap, every consumer downstream is now permanently behind, and there is no way for any of them to detect the gap from where they sit. Event sourcing requires that the log be the source of truth — not a transcript of one that may or may not be complete.

Event sourcing's failure modes are not the ones you've spent your career learning to dodge, and they're worse in a few specific, instructive ways.

Lost events happen when the log is genuinely secondary — when some code path writes to the database and forgets to write to the log. This produces invisible corruption, which is the worst kind. Projections look perfect for current users, because their database writes succeeded; replay produces wrong state, because replay only knows what's in the log. The system reports itself healthy. You don't find out the log is incomplete until you replay — in a test environment months later, or during an incident, which is the universe's preferred time to mention it.

Stale projections are the expected, designed-for failure of eventual consistency applied to read models. When your ProjectorService falls behind — a deployment, a slow downstream database, a consumer-group rebalance — the read models age while events pile up behind them. Users are querying a read model that's an hour stale. That's a manageable, even mundane situation if you designed for it and put the lag on a dashboard. It's a 9 PM support escalation if you didn't. The log is authoritative; projections trail it. The only real question is whether you know which one your users are actually reading.

Replay bugs are the canonical event-sourcing horror story, and the first time you live through one it takes a few years off you. You discover your projection has been calculating order totals wrong for three weeks — a botched discount, an off-by-one in rounding. A million orders affected. The fix is clean: correct the projection code, replay from the beginning. Twelve hours later, replay completes. Beautiful.

Then you notice the corrected projection produced different results for a class of orders that touched a currency conversion whose behavior changed in month three of your log — and those orders now show totals that never existed in any version of your UI, and customers are holding receipts with different numbers on them. This is the part nobody warns you about, because experienced engineers treat replay like rebuilding a search index: safe, idempotent, push the button. It is not that. Replay re-runs today's code over yesterday's facts. It's a superpower, and it's also the only tool that lets you apply a bug to your entire history at once, at scale, with no undo button. (More on this in the exercise at the end of the chapter.)

Event versioning is the slow one — the leak under the sink. Your events have schemas. Schemas change. An event logged two years ago has fields that no longer exist and is missing fields that are now required, and your replay logic expects today's shape. So either you write migration logic for every schema version you've ever shipped — work nobody budgets at the time, because it has no demo — or you meet the mismatch when you replay those old events for the first time, during an incident, at 2 AM, while someone important watches over your shoulder. The standard answer is upcasting: transform old event shapes into the current one at read time. It works. It's also exactly the kind of unglamorous plumbing that gets deprioritized until the day it's a production problem with a name.

The event store is the center of the whole architecture, and it's almost aggressively simple: an append-only log where every event, once written, is immutable. No updates. No deletes. Only appends.

A single stream is just the biography of one thing. For order 4291, that's three records you only ever add to and never edit: OrderCreated, then PaymentAuthorized, then OrderShipped, each carrying its own timestamp and payload. (Full code: Appendix A.1.)

Current state is not stored anywhere. You reconstruct it by reading every event in the stream and folding them together in order. The order doesn't exist as a row in a table — it exists as the list of things that happened to it, and "the order" is whatever you get when you replay that list to the end.

That buys you three things mutable state can't touch. First, immutability: once written, an event can't be changed, so the history is locked — which is exactly why Stripe's charges are legally defensible, since you can't go back and alter what happened, not even to fix a typo you'd really like to fix. Second, completeness: every transition is recorded, and there are no gaps unless there's a bug in your write path, which you monitor for on purpose rather than hope about. Third, temporal queries: "what was the state of order 4291 at 10:01:00?" is a question with an answer. Replay to that timestamp and stop. Without a log, that question doesn't even parse.

CQRS — Command Query Responsibility Segregation — is the pattern that makes event-sourced systems actually fast in production. The name sounds like it was assembled by a committee that billed by the syllable, but the idea fits in one sentence: writes go to the log, reads come from denormalized views built from the log.

The write side is simple. A command arrives — PlaceOrder, RefundCharge, ChangeEmail. The handler validates it, produces one or more events, appends them to the store. No joins, no normalization, no coordinating four tables. You're appending to a log. This is fast, and it stays fast.

The read side is a separate world. A ProjectorService consumes events and maintains denormalized read models — tables or documents shaped for the exact queries you run. Orders get an order_view row with everything an API response needs in one place. Users get a flat user_search_view document built for full-text search. Nothing is reconstructed at read time; you just fetch the pre-built view.

The trade-off is explicit and non-negotiable: read models are derived, so they lag the log. Milliseconds in steady state, seconds to minutes after a deployment or a cold restart. This is the eventual consistency from the last chapter, now wearing a specific architecture. How much lag you can stomach depends on the read model — and you decide that per read model, not once for the whole system, for reasons we'll get to.

The same event stream can feed many projections at once, and this is one of the properties that makes event sourcing worth its operational tax. You don't have to decide up front which view of the data matters. You build all of them from the same canonical history, and they don't have to agree on anything except the events.

From a single order stream you might run, in parallel: an OrderView for the API (a flat row with status, total, line items, address); an OrderAnalytics projection for the warehouse (one row per event, for time-series queries); a UserOrderHistory for the customer UI (recent orders per customer); and a FraudSignals projection for the risk engine (features extracted from OrderCreated). Four projections, one source.

Adding a fifth consumer means writing projection logic and replaying history. It does not mean going back to three application teams and begging them to start logging a field they were never asked to log. The events are already there; the new view is your problem and nobody else's.

Projections must be idempotent, and this is not optional. At-least-once delivery guarantees you'll process some events more than once — at startup, after a crash, after a rebalance. A projection that double-counts on a duplicate is simply a projection with a bug that hasn't reported itself yet. Use the event's ID as a dedup key, either by checking before insert or by leaning on upsert semantics keyed on it.

An order with three events reconstructs instantly. An account with 20,000 events — one per transaction over five years — does not, and you are paying that replay on every single read. Snapshotting is the obvious optimization: every so often, freeze the computed state, and on the next read load the freeze and replay only the events that landed after it. (Full code: Appendix A.2.)

The snapshot is not the source of truth. The log is. If a snapshot is corrupt or wrong, you regenerate it by replaying from the beginning — expensive, but correct. That's what makes snapshotting safe in a way mutable state never is: the worst case for a bad snapshot is a slow read, not a wrong answer. You can't permanently lose the right state, because the right state was never in the snapshot — it was always in the log, waiting to be recomputed.

Snapshot often enough that reconstruction stays inside your read-latency budget; once every 100–1,000 events is common. Where you store snapshots — the event store, a separate table, blob storage — is a genuinely minor decision that, in my experience, absorbs a wildly disproportionate share of the design review. Pick something cheap to read and cheap to overwrite, version the snapshot format from day one, and move on. Schema migrations on snapshots are a gift you give your future self; skipping them is a problem you mail to that same person, postage due.

Write complexity versus read complexity. Event sourcing makes writes almost trivially simple — append to a log — and moves all the complexity to the read side: maintain projections, handle staleness, reason about lag. Traditional databases do the opposite: writes normalize across tables, enforce referential integrity, coordinate transactions; reads are a SELECT against a known schema. Which complexity you'd rather own depends on your read/write ratio and, honestly, on which side your team is less bad at operating. Most teams find projection logic more debuggable than write-time normalization bugs — but "most" is doing real work in that sentence, and you should check whether it applies to you before betting the architecture on it.

Real-time consistency versus eventual consistency. Synchronous projection — write the event, block until the read model is updated, then return — gives you strong consistency at the price of write latency, because your write path is now exactly as slow as your slowest projection. Async projection — write, return immediately, let the projector catch up — gives you fast writes and eventually consistent reads. The lag is usually fine; the word "usually" is the entire reason you measure it and hang an SLO on it. Billing tolerates zero lag. Recommendations tolerate minutes. "Eventual" is not an answer to "how eventual?" — that question wants a number. Measure it, alert on it, and write it into the design doc, where it's a decision, instead of leaving it in your head, where it's a hope.

Log size and retention. The event log is forever, by design. Forever, it turns out, has a monthly invoice — and for a system doing millions of events a day, that invoice eventually gets someone's attention. Your options: retain everything in cold storage (cheap, complete); prune old events after snapshotting (smaller log, but history is gone past the prune horizon); or apply a per-stream retention policy (nuanced, and operationally annoying forever). This isn't really a technical decision — it's a product and compliance one about how much history you actually need. Stripe keeps charge events forever for legal reasons; your anonymous analytics events probably don't need to outlive the decade. Decide early, because migrating a retention policy after five years of growth is the kind of project that gets scoped, estimated, deprioritized, re-scoped, and then quietly never happens.

Stripe stores every charge as a sequence of events. A charge is not a row with a status column — it's a stream: Charge.Created, Charge.Authorized, Charge.Captured, Charge.Refunded (maybe). The Charge object you get back from the API is a projection of that stream. The reason is half compliance — payment records are legally required to be immutable and auditable — and half pure operational survival: a payment flow involves a bank, a card network, Stripe, and a merchant, and any of them can emit events out of order or late. A log absorbs out-of-order events without flinching. A status column does not. You can set a column to captured and then receive an authorization reversal from the bank that was supposed to arrive before the capture — and now you have a correctness problem and a customer-service problem at the same time. Stripe doesn't have that problem, because they never had a column to corrupt.

Netflix uses event sourcing for watch history, and here the driver is signal, not compliance. Recommendations train on complete viewing behavior — not just "what did you watch," but "what did you watch halfway through at 11 PM on a Tuesday, then abandon, then come back to three days later." That texture is only recoverable if you kept the full event history; a mutable last_watched column flattens it into nothing. Those viewing events feed several projections at once: the continue-watching list (low latency, short window), the recommendation pipeline (high latency, full history), content analytics (aggregates), and per-market legal-hold retention (full history, specific users). One log, four consumers, four completely different latency budgets, built and operated by teams that mostly don't have to talk to each other — and adding a fifth doesn't require a meeting with the other four.

Uber models ride state as an event stream from the moment a request is created to the moment a receipt is issued — every location ping, every status transition, every driver acceptance and cancellation. The ride is the sequence of facts. If a service crashes mid-ride, the state is reconstructed from the log and handed back to the recovering service. If there's a fare dispute, the full history is the evidence, not a screenshot and a guess. The GPS trace is an event stream; the payment authorization is an event; the driver acceptance is an event. Which also means the post-ride analytics pipeline, the driver-payout system, and the receipt service all read the same history independently — and a bug in one projection doesn't quietly poison the other two.

EventStoreDB is the dedicated event store, built specifically for this pattern by Greg Young, who did more than anyone to popularize event sourcing in the DDD community. It treats stream-per-aggregate, projections, snapshots, and catch-up subscriptions as first-class concepts rather than things you assemble yourself. Think Git, but for your domain events, with a streaming API. If you're committed to event sourcing and you'd rather not hand-roll a storage primitive, this is the first thing to evaluate.

Kafka as an event log is the pattern most teams reach for first — because they already run Kafka, and the most attractive technology in any decision is the one already in production. It works, with caveats. Kafka is optimized for high-throughput fan-out to many consumers; EventStoreDB is optimized for per-aggregate stream access and temporal queries. Reading the full history of one specific order in Kafka means reading a partition from the beginning and filtering, which is slow and doesn't scale per aggregate. For projections fed from a firehose of all events, Kafka is exactly right. For per-aggregate event sourcing with targeted replay, you want per-stream indexing — which is to say, not Kafka, however much you'd like the answer to be the thing you already operate.

Axon Framework (Java) and Marten (PostgreSQL + .NET) are framework-level implementations. Axon wires up the command/event/projection machinery with adapters for Kafka and EventStoreDB. Marten stores event streams in PostgreSQL's JSONB columns — a deeply pragmatic choice, because your events live next to the rest of your data and you inherit Postgres's operational maturity for free, which for a lot of teams is the difference between shipping event sourcing and merely reading about it. Worth understanding even if you're on neither stack, because the patterns they encode are the ones you'll otherwise rebuild by hand, probably less cleanly.

Your projection calculates order totals by summing line-item prices. It worked correctly for eleven months. Then you shipped a discount feature, and your projection code — written before discounts existed — doesn't account for them. Every order since launch shows an incorrect total. 2.3 million orders.

You fix the projection. You replay from the beginning of the log. Twelve hours later, replay completes, and your order_view table now has correct totals. Crisis over.

Now ask the uncomfortable question: what happened to the systems that consumed order_total from your read model over those eleven months?

Your analytics warehouse ingested the wrong totals and built two quarters of reports on them. Your tax service called your API for each order total and filed records with incorrect amounts — which is the kind of sentence that ends up in front of a lawyer. Your fraud model trained on corrupted features; it learned that a "normal" order looks like one with the wrong discount applied, and it has been making decisions on that basis.

The replay fixed the projection. It did not fix the systems that trusted the projection.

That's the real lesson of replay. Event sourcing makes recovery from a projection bug possible — you replay with corrected logic and the source of truth was intact the whole time. It does not make recovery simple, because your read models are not islands. They sit upstream of other systems, and those systems already ingested the wrong answers and already acted on them. Replay is the first line of the recovery plan, not the whole thing. The rest is finding every downstream consumer of the bad data, working out what each one did with it, and deciding whether and how to walk it back. Write that playbook before you need it. You will not enjoy writing it during the incident.

← Chapter 7 (Stream Processing). Event sourcing and stream processing are the same model wearing different hats. The log is a stream; projections are stream consumers. Everything in Chapter 7 about consumer lag, partition assignment, and at-least-once delivery applies directly to how your ProjectorService reads the log, and the failure modes rhyme: a projector falling behind is just a stream consumer with growing lag and a business owner who hasn't noticed yet.

← Chapter 8 (Eventual Consistency). Event sourcing makes eventual consistency architectural — you chose it on purpose this time, rather than backing into it via an infrastructure ticket. The log is strongly consistent; projections are eventually consistent. The vocabulary from Chapter 8 — consistency windows, replication lag, read-after-write — transfers wholesale to the gap between the log and its derived read models, and so does the discipline: measure the lag, put an SLO on it, know which paths can tolerate staleness and which can't.

→ Chapter 12 (Sagas). Sagas use events to coordinate multi-step distributed transactions — each step emits an event that triggers the next. That's event sourcing pointed at process coordination instead of single-aggregate state, and the tooling and failure modes overlap heavily. If event sourcing clicked here, sagas will read like the same ideas operating at a wider blast radius.

The intuition to carry out of this chapter: event sourcing stores what happened, not what is. Current state is an opinion you compute from history; history is the fact. That one inversion makes audit trails free, time travel possible, and replay a genuine recovery tool — and in exchange it makes read models second-class citizens, perpetually catching up to a truth they didn't author. The complexity concentrates in the projection layer, which is actually the good news: the log is simple to reason about, events are immutable and testable in isolation, and a projection is a deterministic function of a well-defined input. When something goes wrong, you replay and watch it happen. With a mutable database, the evidence was overwritten before you knew you'd want it.

The chapter keeps the representations out of the narrative on purpose. Here they are, collected and annotated — read the prose for the idea, come here for the literal shape. Both are illustrative rather than production-hardened: they're here to make the structure concrete, not to be copied into a payment system.

A.1 — An Event-Store Stream (the shape of an append-only log)

1{id: 1, stream: "order-4291", type: "OrderCreated",       ts: "10:00:00", data: {customer: 99, total: 142.50}}
2{id: 2, stream: "order-4291", type: "PaymentAuthorized",  ts: "10:00:03", data: {payment_id: "ch_abc", amount: 142.50}}
3{id: 3, stream: "order-4291", type: "OrderShipped",       ts: "10:02:14", data: {tracking: "1Z999AA10123456784"}}

Three immutable records for one order, in append order. There is no current-state row anywhere — "the order" is what you get by folding these three together left to right. In production the gaps hide here: the id has to be monotonic per stream (your gap detector depends on it), data shapes drift over time (the event-versioning problem), and "append only" is a property your write path has to actually enforce, not just intend.

A.2 — Snapshot Plus Incremental Replay (when full replay gets expensive)

1Snapshot at event 19,847: {account_state: {balance: 142.50, tier: "gold", ...}}
2Events 19,848–20,001: [TransactionPosted, RewardPointsAdded, ...]
3Reconstruction: load snapshot + replay 154 events  (fast)

Instead of folding 20,001 events on every read, you load the most recent snapshot and replay only the 154 events after it. The snapshot is a cache of the fold, not the source of truth — if it's wrong, you throw it away and replay from zero, which is slow but always correct. Version that account_state blob from the first commit; the day its shape changes, an unversioned snapshot becomes the thing that silently reconstructs the wrong state.

Next: Chapter 10 — The Outbox Pattern: making event publishing atomic with your database writes, so the "Kafka as afterthought" anti-pattern from this chapter becomes the "Kafka as guaranteed delivery" pattern you actually want.