Ordering, Keys, and Partitions
The shape of the whole thing
6 stages, from the promise the broker actually keeps to the judgment that decides where you build the one you need. Read it top to bottom, or jump to the part you came for.
"The broker preserves order" is true — but only within a partition, and a partition is a throughput ceiling you can compute before writing a line of code.
3 familiar ways to choose a partition key, in the order teams reach for them, and the reorder each one quietly ships.
Key by the entity whose order matters and get order where it counts plus parallelism everywhere else — and the one corner with no cheap answer.
The hot partition no number of consumers can drain, and the repartition that quietly reorders history while the throughput graph climbs.
Ordering-vs-parallelism resolved by key size, skew traded for order via salting, partition count as a near-permanent bet — and 2 companies whose correct decisions manufactured their own hot partitions.
Kafka, Kinesis, and Pulsar as 3 answers to "what's your unit of ordering and how hard is it to change," then when this decision earns real attention and the questions to take back to your team.
Introduction
The customer topped up their wallet, waited for the confirmation, and then got declined at checkout for insufficient funds — on money that had cleared 8 seconds earlier. We pulled the trace and both events were right there, both green: WalletCredited for the top-up, WalletDebited for the purchase. The debit had just been processed first. Same account, same second, the two events that had to happen in order happening in the wrong one. The consumer handling the debit read a balance that predated the credit, saw not enough money, and declined a customer who'd done everything right.
What made it memorable was the dashboard: every panel healthy. Throughput normal, lag low, no errors, both events acknowledged in the order they arrived — on their own partitions. Because nobody had decided this on purpose: to balance load, the producer keyed wallet events by a random id, so one account's credit and debit scattered onto 2 partitions and 2 consumers raced. The team believed what everyone believes — that the broker keeps things in the order you sent them. It does. It keeps them in order within a partition, a different and much smaller promise than the one they thought they had.
That gap between the promise people hear and the one they get is this whole chapter. "The broker preserves ordering" is the most quietly misleading sentence in event-driven systems — not false, just true about something smaller than you assumed, and the smallness only surfaces at the scale where you can no longer fix it cheaply. DoorDash keys its order pipeline by order id so a single order's states can't overtake each other; Uber keys trip events by trip id for the same reason. Both run millions a day in parallel, and neither has, or wants, ordering across the whole stream — because that costs one consumer's throughput forever, and they figured out long ago they never needed it.
Here's the ground we'll walk together:
- Why "the broker preserves order" is a true statement about one partition, and why one partition is a throughput ceiling you can predict from arithmetic before you write a line of code
- The one ordering guarantee you actually need — per-entity — and why you already hold the key to it: it's the aggregate id you were going to put in the event anyway
- Why the partition key is an ordering decision wearing a load-balancing costume, and how choosing it for even fills is precisely how you ship before you charge
- The hot partition that lags for hours while every other partition sits idle, why adding consumers does exactly nothing for it, and the one lever that actually moves it
- Why partition count is close to permanent — what changing it does to every key at once, and the reordering it quietly opens while you think you're just turning up throughput
- The corner with no cheap answer — order across 2 entities at the same time — and why naming that out loud is the senior move, not a failure
By the end, "does the broker keep things in order?" should feel like the wrong question; the right one is "in order with respect to what?" — and you'll already know the answer sits in your event's key.
The Problem, Precisely
Total order is available. It ships at a fixed price — one partition, one consumer — and almost nobody who bought it needed what they paid for.
A partition is 2 things at once, and that coincidence is the entire chapter. It is the unit of ordering: events on one partition reach a consumer in the order they were written, period. And it is the unit of parallelism: within a consumer group, only one consumer reads a given partition at a time — which is how the order is preserved, because 2 consumers on one partition would race, exactly like our wallet did. So partition count is simultaneously your ordering boundary and the ceiling on how many consumers can ever work at once. You cannot raise one without spending the other.
Follow that to its conclusion, because it's uncomfortable. Total order across a stream — every event ordered against every other — means every event on one partition. One partition means one consumer, which caps your entire pipeline at a single machine's throughput no matter how many you own. Put a number on it: if one consumer handles 5,000 events a second and you need 50,000, global order is off the table before the design review starts. You need at least 10 partitions to move that load, and the instant you have 2, "the order I produced in" stops being something the topic gives you across the whole thing. You didn't lose ordering to a bug; you traded it for the throughput you also required, and the trade was forced.
Ordering and parallelism are the same resource spent 2 ways. Every event you force into a shared order is an event 2 consumers can't work on at once. Global order isn't expensive because the feature is hard; it's expensive because you're voluntarily serializing a system you built to run in parallel.
So the real question was never "how do I get ordering." You get ordering for free, within a partition. The question is ordering of what, with respect to what? — and the answer, for almost every stream that has ever mattered, is: per entity. One account's events in order relative to each other; one order's states in sequence; one trip's updates in the order they happened. Nobody at checkout cares whether your debit is ordered against my credit. They care, intensely, that their own 2 events don't swap.
Naive Solutions, and What They Cost
3 things teams reach for before they reach for the right one. They arrive in this order because each is what you try after the last one embarrasses you, and each is reasonable enough to ship on a Tuesday.
Trust the broker to order everything. The zero-config assumption, most common by far because it costs nothing and appears to work. You don't think about keys, and on your dev topic — one partition — events come out in exactly the order they went in. Total order, for free, confirmed by every test. Then traffic climbs, someone bumps the topic to 12 partitions to keep up, and the guarantee you never knew you leaned on evaporates without a single error. Events for the same entity just start, occasionally, coming out wrong — and "occasionally" means "under load," which means "in production, not in the test that added the partitions." The cost isn't a failure; it's a guarantee that went quietly false at the exact moment scale removed your ability to notice.
Key for balance. The correction, once you've learned keys exist: pick a key that spreads load evenly, something high-cardinality — a random id, a UUID, round-robin. It balances beautifully. It also scatters one entity's events across every partition, which is how the wallet put its debit ahead of its credit — the failure the async-foundations material calls out-of-order processing, here with a single findable cause: the key. A partition key chosen for even fills is an ordering bug with excellent intentions; it optimizes the one property (balance) that was never in danger and sacrifices the one (per-entity order) that quietly holds your state together. The tell is that it passes every load test, because load tests send uniform traffic and never make the same entity act twice in a row the way a real user does.
Add partitions when throughput is capped. The scaling reflex, most defensible of the 3: you're consumer-bound, partitions are the ceiling, so raise the ceiling. Sound reasoning that walks into a trap we'll open shortly — changing the partition count re-runs the hash that places each key, so most keys move, and an entity's in-flight events split across the old partition and the new one with no order between them. You scaled throughput and reordered history in one config change, and the second effect doesn't show up on the graph you were watching.
All 3 are one mistake at 3 addresses: treating the partition key as a plumbing detail — a balance knob, a throughput dial — when it's the single most important piece of domain modeling in the event's envelope. The key decides what is ordered against what: a design decision you make once, early, and mostly can't take back, not an infrastructure concern you tune later.
The Construction: Per-Key Ordering
Here's the thing you actually build, and it's less a technique than a change of question.
Start with what you're giving up, so it's a decision and not a loss. Global ordering — total order across the stream — is real, and on a single-partition topic you have it. It's also a throughput ceiling of one consumer, and the mistake is never choosing it; it's inheriting it from a dev topic and shattering it by accident when you scale. If you genuinely need every event ordered against every other, keep one partition and accept the ceiling with open eyes. You almost certainly don't.
What you need instead lives in the key. A partition key is the field the producer pulls from each event and hashes to pick a partition: partition = hash(key) mod partition_count. That one line does 2 jobs, and holding both in your head is the whole game. It decides where the event goes — same key, same partition, every time. And because a partition is an ordering boundary, it decides what the event is ordered against — everything with the same key, and nothing with a different one.
P2 and P5 keep their own internal order — there is no ordering relationship between them
order-42 and order-77 have no reason to be ordered, so the system doesn't pay to order them.Now make the key the id of the entity whose order you care about — its aggregate id. The async-foundations atlas calls this entity-keyed partitioning, and the mechanism is one sentence: key by the entity, and every event for it lands on one partition in produced order. That's per-key ordering, the guarantee you were after the entire time. Key wallet events by account id and that account's credit always precedes its debit, on one partition, read by one consumer, in order — while 10,000 other accounts stream through the other partitions in parallel. You didn't choose between order and throughput; you got per-entity order and cross-entity parallelism from one correct choice of key.
A stream of events about many independent entities, where order matters within an entity but not across them. Find the entity in your domain whose events must never overtake each other, and make its id the key.
You don't need global order. You need per-entity order, and you already have the key: it's the id you were going to stamp on the event anyway. The partition key isn't a new thing to invent; it's an old thing to take seriously. Choosing it is domain modeling, and you do it once.
And the honest limit, because a chapter that only sells the happy path hides the corner where senior engineers actually get stuck. Per-key ordering orders events within one key. The moment you need order across 2 entities — an order and its separate payment, the 2 accounts in a transfer — there is no cheap answer, and pretending otherwise is how you get the bug reconciliation finds 3 weeks later. You have 2 honest moves. Co-locate the entities under one shared key so they land on one ordered partition — which works, but couples them forever and concentrates their combined traffic onto one partition (hold that thought; it's the hot partition, coming next). Or accept that the 2 streams interleave and make the consumer tolerate it — versions, reconciliation, explicit out-of-order handling. Both are legitimate. What isn't is needing cross-entity order and assuming the broker will supply it. It won't, and it never said it would.
Failure Modes Worth Naming
You keyed by the entity; you did the domain modeling. And 2 failures still wait — not because you got it wrong, but because you got it right, and the right thing has edges. Here's the field guide; the 2 that surprise people get the full treatment after.
| Failure mode | Trigger | Symptom on-call sees | Cost |
|---|---|---|---|
| Cross-partition reordering | Key chosen for balance, or no key — one entity's events land on multiple partitions | State corrupted by events applied out of sequence (ship before pay, debit before credit) | Silent bad state; found by reconciliation, not by an alarm |
| The hot partition | One key takes a large share of traffic (a whale tenant, a hot merchant, a synthetic test account) | One partition lags hours behind; the rest sit idle; aggregate lag looks fine | The dominant entity — often your most important one — is the most delayed |
| Repartition reordering | Partition count changed, or a shard split/merged, or a rebalance mid-flight | A burst of out-of-order processing for keyed events, during and just after the change | Reordered writes for the duration of the transition; a second incident inside the first |
| Rebalance at the seam | A consumer group rebalance reassigns a partition mid-batch | Brief redelivery and reordering as ownership moves | Duplicates and local reordering at the handoff |
The one that surprises people who did everything else right is the hot partition, because it's immune to the reflex that fixes everything else: adding capacity. Key by entity and load spreads in proportion to how your keys are distributed — fine, until one key isn't like the others: a tenant that's 60% of your traffic, a merchant on Black Friday, a load-test account someone forgot to kill. That key hashes to a single partition, now doing 60% of the work while its peers idle, its lag climbing into the hours. And you cannot add your way out. One partition is one consumer's job — the promise that keeps it ordered — so more consumers just help the idle partitions and touch the hot one not at all. You provisioned parallelism and the bottleneck declined to use it.
Uniform traffic: every partition carries about the same load, lag stays low across the board.
The second surprise is repartition reordering, the trap the third naive solution walked toward. Partition assignment is hash(key) mod partition_count; change the count and the same key hashes to a different partition — for most keys at once. Grow a topic from 4 partitions to 6 and a key that lived on partition 1 now belongs on partition 3, with in-flight events on both: old ones draining from 1, new ones landing on 3, no ordering between them. The per-key order you carefully built breaks for the duration of the transition, precisely when you weren't looking — because you were watching the throughput graph go up.
Partition 1 still draining K1, K2, K3 (in order) while partition 3 already receives K4, K5 — no ordering relationship between the two sides. Order broken.
Tradeoffs Worth Arguing About
3 decisions here don't have a universal answer, and every one of them is where a real system gets keyed well or badly.
Ordering versus parallelism — resolved by the size of your key. The chapter's spine trade doesn't disappear once you've chosen per-key ordering; it moves into how big the entity is. Key by a coarse entity — tenant, region — and you get strong ordering but few distinct keys, which means little parallelism and a fat target for skew. Key by a fine entity — one order, one account — and you get maximum parallelism and minimal hot-partition risk, but you've declared that you don't order across those entities, so any invariant spanning 2 of them is your consumer's problem. The resolution isn't a rule, it's a question: what is the smallest entity that carries the invariant you must protect? Key by that, and no smaller.
Skew versus ordering — the salt. When one key runs hot, the standard fix is to split it: append a suffix so tenant-7 becomes tenant-7:0 through tenant-7:7, spreading that entity across 8 partitions instead of pinning one. The tall bar comes down. It also forfeits exactly what keying bought you: tenant-7's events are now scattered across 8 partitions with no order between them. So salting is only free when the hot entity's internal order doesn't matter, or when you'll reconcile downstream. That's the real decision behind "we have a hot partition": not whether to spread the load, but which you value more for this whale — throughput or ordering. Say which out loud, because the salt makes the choice for you the moment you type it.
Partition count — over-provision now, or reshard later. Because growing the count live reorders keyed events, the count wants to be decided once, up front, for a throughput ceiling 2 years out — deliberate over-provisioning. Its cost is real but bounded: more file handles, more replication overhead, more metadata, longer rebalances. The cost of under-provisioning is the reshard — a drain-and-switch you'll schedule at an awkward hour and hold your breath through. Kinesis makes you do it explicitly and calls it resharding; Kafka lets you add partitions with a one-liner that looks harmless and reorders your keyed topic. The senior move is to size for the load you'll have in 2 years, not today, and to treat "just add partitions" like a schema migration — a real operation with a rollback plan, not a config tweak.
What the Companies Actually Built
DoorDash keys its order pipeline by order id, and the choice does one valuable thing: a single order's state machine — created, confirmed, prepared, picked up, delivered — is processed in that sequence, on one partition, no matter how busy the platform is, while different orders stream through in parallel. The lesson worth stealing isn't "key by order id"; it's the reasoning — they found the entity whose event order was load-bearing (one order's lifecycle) and keyed by that, declining to order anything larger. Key by customer instead and you'd serialize a customer's unrelated orders for nothing and hand yourself a hot partition for every power user.
Uber keys trip events by trip id for the same reason — requested → matched → started → completed must hold — and it's the other half of their design that complicates the advice. Dispatch streams key by geography, because matching riders to drivers needs a city region's events together. Geography is sensible right up until one region isn't like the others: a downtown on a Friday night is a hot partition by construction — the celebrity/whale problem the async-foundations atlas named, wearing a map instead of a follower count. Uber's good decision manufactures the exact skew this chapter warns about, and they live with it through the tradeoffs above. A keying choice can be correct and be the direct cause of your worst hot partition — not a contradiction, the same decision seen from 2 sides.
Pulsar's Key_Shared subscription genuinely complicates the "parallelism is capped at partition count" line this chapter has been leaning on, so let's complicate it honestly. Pulsar delivers per-key ordering while allowing more consumers than partitions: it hashes each key to a consumer and keeps that key's events on that consumer in order, decoupling parallelism from partition count. Kafka's model doesn't do this natively, and if the one-consumer-per-partition ceiling is your pain, it's worth knowing the ceiling is a property of a design, not a law of physics. What Key_Shared does not repeal is the hot partition: a single hot key still serializes to a single consumer, because that's what preserves its order. So it buys finer parallelism for your well-distributed keys and changes nothing for your whale — the honest shape of most "this broker solves ordering" claims. It moves the ceiling; it doesn't remove the entity that was going to hit it.
Technologies Worth Knowing
Read each of these as an answer to one question: what is your unit of ordering, and how hard is it to change?
Kafka makes the partition the unit. A keyed record is hashed to a partition and ordered there; a keyless record goes to the sticky partitioner, which batches to one partition then rotates — not per-record round-robin, so "no key" doesn't mean "evenly spread one message at a time." The sharp edge is the one the naive solution hit: you can increase partition count but never decrease it, and increasing it rehomes keyed records, so the command that reads as "scale up" breaks keyed order while dressed as a config change. Treat kafka-topics --alter --partitions with the respect you'd give a migration.
Kinesis calls partitions shards and refuses to let you pretend resharding is free. Ordering is per-shard; scaling is an explicit split or merge with its own API, rate limits, and new sequence numbers on the far side. That's more friction than Kafka's one-liner, and the friction is a feature — it makes you look at the reshard instead of triggering it by accident. You will schedule resharding as a real task, which is the right ceremony for an operation that rewrites your key-to-shard map.
Pulsar hands you the ordering-versus-parallelism trade as a per-subscription setting — flexibility or foot-gun depending on who's holding it. Failover gives one consumer per partition and strict order; Key_Shared gives per-key order with consumers beyond the partition count; Shared gives maximum parallelism and no ordering at all. Powerful — you pick your point on the trade per consumer — and dangerous, because "no ordering" is one dropdown from "strict ordering" and whoever configures the subscription may not know which invariant the stream carries. The edge is organizational: the tool is only as safe as the reviewer who catches a Shared subscription on a stream that needed Key_Shared.
The Principal Engineer's Perspective
When per-entity ordering earns its keep — and when to pay for none. The first question isn't "how do we order this," it's "do these events even need ordering?" If they commute — applying them in any order lands the same place, like independent metrics, presence pings, or unrelated notifications — ordering is a cost with no benefit; key for balance and enjoy the even load. If they mutate shared state in sequence — a ledger, a state machine, anything where event B assumes event A happened — key by the entity. The failure at the top isn't picking the wrong key; it's applying one policy everywhere: the team that painstakingly orders its fire-and-forget metrics and the team that round-robins its ledger are making the same mistake in opposite directions — not asking whether the stream's events commute.
Who pays, and when. The partition key is chosen once, usually early, often by whoever wrote the producer first — and because changing it means resharding, that day-one choice becomes a day-400 constraint you can't easily revisit. A junior engineer typing key = uuid() to make the load graph look nice is, without knowing it, deciding an ordering property finance will care about later. The partition key is a business decision in an engineering costume; it deserves a design review, not a default. Inherit a system and "what do we key by, and who decided?" is one of the first questions to ask — "a UUID, nobody remembers" is a finding, not an answer.
The observability that has to exist first. You cannot manage any of this from aggregate dashboards, because the hot partition hides behind healthy averages. 3 views: per-partition lag, not the group's mean — the hot partition is invisible in the average and glaring per partition; key distribution, so you see a whale forming before it pins a partition; and rebalance frequency, because a group that rebalances constantly keeps reintroducing the redelivery-and-reordering seam. Consumer lag is the heartbeat here, but only per partition. Aggregate lag is a heartbeat averaged across a healthy patient and one in cardiac arrest.
Failure and recovery. The hot partition is detected in per-partition lag and mitigated by salting the whale (accepting its lost internal order) or accepting the lag as the price of that entity's ordering — a real choice, made explicitly. The reshard reorder is caught by an out-of-order metric on your consumers and prevented by draining the old assignment before trusting the new one, never by a live count bump under load. Both recoveries depend on having decided in advance whether the stream values throughput or order more — the decision this whole chapter pushes you to make on purpose.
Testing it means testing the distribution you'll actually get. Uniform test traffic is a lie your load generator tells you; production keys are skewed, and the properties that matter — order under a hot key, order across a partition change, behavior during a rebalance — only show up under skew and during operations. Test with a deliberately lopsided key distribution, and test the reshard path itself: the reorder-during-transition isn't an edge case, it's the main case of a partition change, just infrequent. Infrequent is what production is for.
Questions to take back to your team:
- For each stream we run — do its events commute, or must they be ordered? If we've never asked per stream, we are both paying for order we don't need and missing order we do.
- What are we keying each stream by, who chose it, and could we change it without a reshard? "A UUID, nobody remembers, and no" is a latent incident, not a shrug.
- What is our most skewed key's share of traffic, and what's the lag on its partition versus the average? If we only watch aggregate lag, our most important entity may be our most delayed and we'd never see it.
- If we doubled partitions tomorrow, what would reorder — and do we have a drain-then-switch runbook, or just a one-line command and optimism?
- Where do we need ordering across 2 entities, and have we admitted out loud that there's no cheap answer — co-locate and couple, or interleave and reconcile — or are we quietly hoping the broker handles it?
The broker never promised you order — it promised order within a partition, a smaller and more useful thing than the total order you assumed. A partition is the unit of ordering and of parallelism at once, so global order costs one consumer's throughput forever, and you almost never need it. What you need is per-entity order, and you already hold the key: make the partition key the id of the entity whose events must not overtake each other, and let everything else run in parallel. Respect the 2 edges of that choice — the hot partition you can't out-consume because one partition is one consumer's job, and the reshard that rehomes your keys and reorders history while you think you're adding throughput. And when you need order across 2 entities, say it out loud: there's no cheap answer, only co-locate-and-couple or interleave-and-reconcile. The partition key is the most important decision in the envelope — domain modeling, not plumbing — and you make it once.
Exercises
None of these has a clean answer, which is the point. Paste any of them into an AI and you'll get a confident reply in 4 seconds that assumes your keys are uniform, your partition count is free to change, and every stream needs the same treatment. The value is in arguing it on your actual system.
Exercise 1 — Find your ordering unit. Pick one stream you own. Name the entity whose events must never overtake each other, then go read what you actually key by. If they differ, you have either a live reordering bug or a lucky single-partition topic that becomes a bug the day someone scales it — decide which, and how you'd know.
Exercise 2 — Rebuild the reorder in your domain. Our worked example was a credit and a debit for one account, keyed for balance, reordered into a wrongful decline. Rebuild it with your own entity: name the 2 events that would corrupt state if they swapped, the key that keeps them ordered, and the key that would become hot. Then find where the mapping breaks — an invariant that spans 2 entities at once, where there's no cheap key — and say what you'd do about it. That breakdown is the interesting part.
Exercise 3 — Price your hot partition. Find your most skewed key and its share of traffic. Estimate the lag its partition would carry at 2× current load while the others stay near zero. Then decide, out loud and in advance: salt it and lose its internal ordering, or accept the lag and protect the order. Can't decide without more information? Name the information — that's the real output.
Exercise 4 — Plan the reshard you're avoiding. Take a topic you've wanted more partitions on and haven't grown. Write the actual steps to double its partition count without reordering keyed events — drain, switch, verify. If you can't write them cleanly, you've discovered why the topic is still under-partitioned, and what you'd need to build to fix it safely.
Exercise 5 — Argue Key_Shared versus one-consumer-per-partition. For one real stream, make the strongest case for decoupling parallelism from partition count (Pulsar-style), then the strongest case against. Where does it genuinely relieve a ceiling, and where does it just let you avoid a hot key a little longer? Commit to an answer for this stream.
Connections
← Delivery Guarantees in Practice (Chapter 4). Chapter 4 promised your idempotency key and your partition key are related decisions, and here's the payment: a consumer group rebalance is both a redelivery birthplace (Chapter 4's duplicate) and a reordering birthplace (this chapter's seam), so the same event delivered twice can also arrive out of order, and a dedup scheme that ignores partition semantics can quietly start missing duplicates after a reshard rehomes its keys. The 2 keys are a pair; design them together.
← Brokers, Logs, and Queues (Chapter 3). Chapter 3 set up the consumer models — competing consumers, consumer groups, independent readers — and this chapter puts ordering onto that model: order lives at the partition, and the consumer group decides who reads it and, at every rebalance, hands it off. Chapter 3's replayability is also where a reshard bites hardest — replaying re-keys history through whatever partition map is current.
← The Anatomy of an Event (Chapter 2). Chapter 2 said the event envelope's key is the field you'll wish you'd standardized on day one. This is where that's cashed in: the key isn't metadata, it's the ordering decision, and naming it deliberately in the envelope is what lets you key by the aggregate id instead of by whatever was convenient.
← The async foundations (the Internet-Scale Product Systems atlas, ch 6–7). That atlas defined entity-keyed partitioning, the ordering-versus-parallelism tension, out-of-order processing, the consumer offset, and the hot key. This chapter assumes all of them and deepens the one that matters most operationally — under a strict ordering requirement, the hot key becomes a hot partition you cannot out-consume, a sharper constraint than the hot key alone.
→ Event Schema Evolution (Chapter 6). A partition key you can't change without resharding is a cousin of the problem Chapter 6 owns: an event you can't un-publish. Both are commitments the log makes permanent, and both reward choosing conservatively up front — the key and the schema are the 2 parts of the envelope far easier to get right than to change.
→ Replays, Backfills, and Reprocessing (Chapter 11). A replay is at-least-once delivery plus re-keying at volume: replay a topic through a partition count that's changed since the events were written, and you replay them into a different ordering than they originally had. Chapter 11 is where per-key ordering meets the reprocessing that assumes it.
→ Poison Pills, Dead Letters, and Error Flow (Chapter 12). Per-partition ordering has a dark side: one message that can't be processed blocks every message behind it on the same partition, because letting the others past would violate the order you asked for. Chapter 12 is where that head-of-line block gets its own machinery — the ordering guarantee and the poison pill are the same coin.
Appendix A: Reference Implementations
The snippets below are illustrative, not production drop-ins. They make one difference concrete: a key chosen for balance versus a key chosen for ordering — and what the standard hot-partition fix quietly gives up.
A.1 — The bug: keying for balance scatters an entity
// Wallet events, keyed for "even load." This reorders one account's events.
for (event in walletEvents) {
producer.send(
topic = "wallet-events",
key = UUID.randomUUID().toString(), // (!) a fresh key per event → round-robin-ish spread
value = event
)
}
// hash(random) mod N lands each event on an arbitrary partition, so an account's
// WalletCredited and WalletDebited can end up on different partitions, read by
// different consumers, applied in either order. Balance is perfect; ordering is gone.The load-bearing line is key = UUID.randomUUID(): a unique key per event maximizes spread and destroys per-entity order, because the 2 events that had to stay together were handed different partitions. What's seductive is that the partitions fill evenly and every uniform load test passes. The failure needs the same account to act twice in quick succession — which synthetic traffic rarely does and real users constantly do.
A.2 — The fix: key by the aggregate id
// Same stream, keyed by the entity whose order matters.
for (event in walletEvents) {
producer.send(
topic = "wallet-events",
key = event.accountId, // all of one account's events → one partition, in order
value = event
)
}
// hash(accountId) mod N is stable for a given account, so every event for that
// account lands on the same partition and is consumed in produced order.
// Different accounts still spread across partitions → full cross-entity parallelism.The load-bearing change is key = event.accountId: the account is the entity whose event order matters, so its id is the key. This is the whole construction in one line — per-key ordering where you need it, parallelism everywhere else. It also sets up A.3's problem: if one account is a whale, its events all pile onto one partition, which becomes the hot one.
A.3 — The salt: spreading a hot key, and exactly what it costs
// Mitigate a hot account by splitting it into sub-keys. This spreads load AND drops the account's ordering.
fun keyFor(event): String {
if (event.accountId in HOT_ACCOUNTS) {
val bucket = Math.floorMod(event.eventId.hashCode(), 8) // 8 sub-partitions for the whale
return event.accountId + ":" + bucket // e.g. "acct-7:3"
}
return event.accountId // normal accounts: unchanged, still ordered
}
// The whale's events now spread across 8 partitions → the hot partition cools.
// But "acct-7"'s events are no longer ordered against each other. Only safe if the
// whale's internal order doesn't matter, or a downstream step reconciles it.The load-bearing line is the sub-key accountId + ":" + bucket: it trades the hot account's ordering for its throughput, on purpose. The comment is the real content — salting isn't a free performance win, it's a decision to stop ordering a specific entity, correct only when that entity's internal order is dispensable or you rebuild it downstream. Everything not in HOT_ACCOUNTS keeps its per-key order untouched, which is why you salt the whale and nothing else.
Next: Chapter 6 — Event Schema Evolution: why an event, once published, is immortal — and why you can't deprecate the past.