Circuit Breakers

Rate limiting protects you from too much traffic. Circuit breakers protect you from traffic that was never going to work — requests aimed at a dependency that has already fallen over and just hasn't finished telling you yet. The mental image is a fuse: when the thing downstream draws current it can't survive, the fuse pops and stops the flow before the whole house burns. Cheap, ugly, and the reason your kitchen still exists.

Most engineers get the difficulty backwards. Rate limiting clicks immediately — too many requests, turn some away, a little queuing theory, done by lunch. Circuit breakers take longer to land, and not because they're complicated. The state machine fits on a napkin. They take longer because the failure they prevent doesn't look like a failure until you've stood in the middle of one. You're not shedding load from a healthy system; you're refusing to send load to a broken one. The system is already failing by the time a breaker earns its keep — all it gets to decide is whether the failure stays in one room or takes the building.

To see why that matters, watch what a slow dependency actually does to you, because slow is worse than dead. When a downstream service breaks, the merciful version is that it fails fast — connection refused, instant error, you move on. The version you actually get is that it fails slowly. Your Payment Service stops answering, but your 10-second timeout means every request waits the full ten seconds before giving up. A hundred concurrent users, a hundred threads, each parked for ten seconds doing nothing but waiting to be disappointed. Most web apps run thread pools of 200 to 500. You just spent half of yours holding the line for requests that were dead on arrival. New requests queue behind them, the queue fills, and your entire API goes unresponsive — not because you have a bug, but because something you depend on does, and your threads are blocked waiting for it with a patience that would be admirable if it weren't fatal.

That's the cascade in miniature. One service's outage becomes two, two becomes three, and the arithmetic compounds in the wrong direction. The original failure is rarely the thing that takes you down. The waiting is what takes you down — your healthy services lining up to die behind a sick one.

If you've lived anywhere with an unreliable grid, you know this failure in your bones; it just runs at a different scale. On 31 July 2012, an overloaded line in northern India tripped and shoved its load onto its neighbors, which overloaded and tripped in turn, and the fault walked across the interconnections until three regional grids had collapsed and more than 620 million people — about half the country — were in the dark. The largest blackout in history wasn't a shortage of power. It was load with nowhere safe to go and nothing willing to isolate the fault before it spread. The grid's whole defense against that is the device in your wall, scaled up: breakers that trip locally and fast, spending one feeder to spare the rest. Your services need the same thing for the same reason — except in software the cascade doesn't take hours, it takes about as long as one bad on-call ping.

A circuit breaker cuts the line. It watches failures, and once they cross a threshold it stops sending requests at all — the next request is rejected locally, in microseconds, without ever touching the network. No ten-second wait. No blocked thread. No cascade. The user still gets an error, but a fast error, and your service stays standing to serve everyone whose request didn't depend on the broken thing.

The three states are the whole fuse, and the analogy holds the entire way down:

• Closed — current flows. Requests go through normally. This is the state you live in.
• Open — current is cut. Requests are rejected immediately and locally. Nobody touches the broken thing.
• Half-Open — testing the wire. A trickle of requests is allowed through to find out whether the downstream recovered, before you commit to trusting it again.

Here's the ground we'll cover:

• Detecting failure — harder than "count the errors," and the spot where most homegrown breakers are quietly wrong
• The three states — closed, open, half-open — and why the whole fuse lives in the transitions between them
• The half-open testing problem — how to find out whether a dependency recovered without being the load that knocks it back down
• How cascades develop and how a breaker contains them — spending one feeder to spare the rest
• Why Netflix bet its home page on breakers, and what Hystrix, Envoy, and Spring Cloud teach — including the awkward fact that Netflix deprecated the library that made the pattern famous
• The questions a principal engineer has to answer before any of it is worth building

One thing to hold onto before we start: this is a pattern that looks obvious in retrospect and stays invisible until the night you need it. The people who built the first circuit breakers weren't writing a paper. They were watching their systems fall over in a way timeouts couldn't fix, at an hour when nobody writes papers, and they needed something that worked before morning. The pattern is elegant because the problem was urgent, and urgent problems have a way of stripping out everything that isn't load-bearing.

The broken dependency isn't what takes you down — it's everything healthy still waiting on it.

When a dependency goes slow, the first thing anyone reaches for is a timeout, and they're right to. Without one you wait forever, which is strictly worse — a thread blocked for ten seconds at least comes back; a thread blocked with no timeout is gone until someone restarts the process. So: add timeouts. Necessary. Just nowhere near sufficient, and the way they come up short is the whole reason this chapter exists.

Make it concrete. Your Payment Service has a query that normally returns in 20 milliseconds and is now taking 30 seconds, because a database somewhere is having the kind of day databases have. Your API calls it with a 10-second client timeout. A customer clicks "buy." Ten seconds pass — ten real seconds, the kind a human notices and resents. The request times out, your code catches the exception, the customer sees an error. So they do what every human who has ever used software does: they click "buy" again. Another ten seconds. Another error. You've now spent twenty seconds of wall-clock time and two threads establishing a fact you could have known after the first millisecond — that the Payment Service is down.

That's one customer. Run a hundred of them at once and you've got a hundred threads each parked for ten seconds on a service that is reliably, dependably broken, plus all the retries those hundred customers are firing because the button "didn't work." The timeout did its job perfectly. It bounded the wait. It just bounded it at ten seconds, a hundred times over, which is its own kind of catastrophe.

The deeper problem is that a timeout treats every request as if it might be the one that works. It has no memory. Request 1 times out after ten seconds; request 2 has no idea request 1 ever happened and waits its own ten seconds; request 500 is still waiting, full of hope, on a service that's been dead for four minutes. Statelessness is a virtue right up until the moment you need the system to learn something, and "the thing we're calling is broken" is exactly the kind of thing worth learning once and remembering.

Why retries make it worse

The instinct after timeouts is retries with exponential backoff, and again it's half right. For transient failure — a dropped packet, a node bouncing, a one-second blip — retry with backoff is exactly correct and you should do it. For sustained failure it's gasoline. A five-attempt backoff that waits 1, 2, 4, 8, then 16 seconds spends 31 seconds per request discovering the service is still down. (Full code: Appendix A.1.) If the dependency is out for five minutes, every request that arrives in those five minutes burns 31 seconds and a thread to learn what the last one already knew.

And there's a subtler trap underneath. Every client backing off on the same schedule wakes up on the same schedule. They all failed at roughly the same moment, so they all retry at roughly the same moment — a synchronized wave slamming into a service at the exact instant it's trying to stand back up. "Add jitter," you say, and you're right, you should, we'll get there. But spreading the wave out is not the same as not throwing it. Jittered or not, the requests still arrive, the threads still block, the connection pools still fill. Jitter fixes the shape of the stampede; it does nothing about the fact that you're stampeding a service you already have every reason to believe is broken.

The cascade

The single broken service is not the thing that should scare you. What should scare you is everything standing behind it.

Walk Netflix's home page backward. The Recommendation Service calls the User Service. The User Service calls the User Database. One afternoon the User Database hits replication lag and goes slow — not down, just slow, which is worse. The User Service starts timing out against it, and retrying, and piling up requests; its thread pool fills. Now the Recommendation Service, calling a User Service that can no longer answer, starts timing out too — and then stops even getting timeouts, because the User Service has no threads left to refuse it with, so the failures curdle into connection-refused. By the time the User Database sorts itself out, three services are down and the home page is blank for 200 million people.

Read that chain again and notice where the damage lands. The thing that broke was a database three hops down. The thing the customer saw was a blank home page three hops up. Everything in between was healthy — it just had the misfortune of waiting on something that wasn't, and waiting is contagious. A circuit breaker anywhere along that chain — Recommendation refusing to keep calling User, or User refusing to keep calling the database — stops the contagion at the sick component. Upstream stays up and serves a degraded page. Downstream gets left alone long enough to actually recover. The failure stays in one room.

Everyone arrives at the circuit breaker by elimination, after trying the things that feel like they should work first. Here's the usual order.

"Make the timeout longer." If the service is slow, give it more time, right? Wrong in both directions at once. A 30-second timeout doesn't make a broken service any less broken — it means you find out in 30 seconds instead of 10, having tied up the thread three times as long to learn the same thing. You haven't bought patience; you've bought a wider, deeper pool of blocked threads and a worse cascade. The service isn't slow because you were rude enough to give up early. It's broken. Waiting longer is just losing more slowly.

"Add more threads." If threads are the bottleneck, buy more threads. This is a fix in the same sense that bailing faster fixes a boat with a hole in it. You extend the runway; you don't change the destination. And the runway is shorter than you think — somewhere up the thread count the runtime itself becomes the bottleneck (the JVM, the interpreter, the event loop — pick your poison), and you've traded a dependency problem for a garbage-collection problem and paid operational complexity for the privilege.

"Back off exponentially, with jitter." Now we're onto something genuinely useful, and you should do this regardless of anything else in the chapter. (Full code: Appendix A.2.) Jitter breaks up the synchronized retry wave so a recovering service doesn't get rear-ended by every client at once. But notice exactly what it does: it improves how you fail. It does not stop you from failing. The requests still go out, the threads still block, the pools still fill — just on a more considerate schedule. Backoff with jitter is good manners. It is not a cure.

What every one of these misses. None of them contains the one piece of logic that actually solves the problem: once you've established that a service is down, stop calling it. Not "call it less often." Stop. Wait a defined interval. Then test the water carefully before you wade back in. That's the circuit breaker, and it's precisely the move the naive fixes refuse to make — they all keep calling the broken thing, just with different timing.

So why do experienced engineers walk right past it? Not because it's hard to understand — it's because it's stateful, and the entire timeout-and-retry mental model is stateless. Each request decides for itself how long to wait and whether to retry; it neither knows nor cares what happened to the request before it. A circuit breaker breaks that independence on purpose. The circuit is open or closed for everyone at once, which means requests have to share state, which means in a single process it's a trivial shared variable and across a fleet of fifty app servers it's a distributed-state problem with all the coordination headaches that phrase drags along. That last part is the real reason teams stall here. They can see the pattern fine; they just don't want to own coordinated state across fifty machines, so they stick with per-request timeouts and tell themselves it's fine. It is fine, right up until the first real cascade — which tends to schedule itself for 2 a.m. on a Friday, because that is when the universe prefers to teach this particular lesson.

The field guide — the specific ways this goes wrong in production, most of which look like something else the first time you see them.

The oscillating system. Your database goes slow. The thread pool fills, requests queue, you wait. Then the database recovers — and every request that piled up during the slow patch executes at once, a synthetic spike you spent the whole degradation window accumulating and saved for the worst possible moment. The just-recovered database takes that spike to the face and slows down again. Pool fills again. Recovers again. Spikes again. This is the failure mode that gets written up as "intermittent issues" and "transient instability," which are the words you reach for when you haven't yet noticed the system is oscillating on a fixed period like a metronome with a grudge. It is not intermittent. It is deterministic — a clean limit cycle between "degraded" and "crashed" — and it keeps cycling until something breaks for good or someone gets lucky restarting things in the right order. A circuit breaker is what breaks the loop: it sheds the queued load instead of hoarding it to detonate on recovery.

The thundering herd on recovery. You added jitter, so you think you're safe from synchronized retries. Not quite. Jitter spreads retries within one client's backoff window. It does nothing about a thousand clients that all hit their first failure at the same instant — they march through their backoff schedules in lockstep and arrive at the door together no matter how much jitter each one sprinkled on. A service that came back up clean gets knocked flat again by its own welcome-back party. Engineers describe this one as "it recovered and then immediately died again for no reason," and the reason is always the same: a thousand clients were sitting in the waiting room watching the door. A breaker fixes it directly — when it's open and the timer expires, it goes to half-open and lets one probe through, not the thousand. The crowd stays outside until the probe says it's safe.

Connection pool exhaustion. HTTP clients keep connection pools, typically 100 to 500. When a downstream goes slow, connections sit open holding their breath, waiting for responses that take forever. New outbound requests block waiting for a free connection. The pool empties not because you're sending too many requests but because the ones in flight won't finish. The tell is distinctive and easy to misread: outbound request rate looks perfectly normal on the dashboard while latency climbs and climbs. You're not watching a traffic spike — you're watching a queue back up in slow motion. And when the downstream finally recovers, the whole backed-up queue drains in one breath, which is yet another synthetic spike. I've watched a team toast a dependency's recovery and then watch their own dashboards go red sixty seconds later, which is a uniquely demoralizing way to learn this, because it feels like the system is mocking you. It is.

The partial failure that hides. A total outage is the easy case — error rate slams to 100%, the breaker trips, everyone agrees what happened. The mean case is partial: 30% of requests failing, 70% returning a cheerful 200 OK. A naive alert set to fire at a 50% error rate sits there quietly while that 30% steadily chews through your thread pool, because slow-and-sometimes-failing is below the threshold but very much not fine. This is exactly why the sliding-window breaker we're about to build measures the fraction of recent requests that failed instead of waiting for a binary cliff: a service failing 30% of the time isn't healthy, it's a service coming apart, and the only question is whether you notice before or after it finishes the job.

Start with the textbook version, because it's correct enough to ship and wrong enough to teach you something. The breaker wraps every call to the dependency and keeps a little state machine: count failures while closed, and when they cross a threshold, flip to open and reject instantly; after a cooldown, flip to half-open and let a probe or two through; if they succeed, close again and resume normal service; if they fail, back to open and reset the clock. (Full code: Appendix A.3.)

The parameters look like trivia and are actually the entire behavior:

• Failure threshold — how many failures trip it. Five is a sane default. Set it to two and a single unlucky GC pause on the downstream trips you into open over nothing; set it to twenty and you've signed up to cascade for a good while before the breaker deigns to act.
• Success threshold — how many half-open probes must succeed before you trust the thing again. Require at least two. A flapping service can absolutely return one clean response and then fall over on the next, and a one-success rule will happily slam the circuit shut directly into that.
• Timeout (the cooldown) — how long to stay open before probing. Sixty seconds is a reasonable start; Netflix runs many services at thirty, because they've measured their restart times and would rather probe a little eagerly than leave a recovered service walled off, twiddling its thumbs, while users eat errors.

Now the part that makes this version "correct enough to ship and wrong enough to teach you something." Look at how success is handled: a success resets the failure count to zero. Sounds reasonable. It isn't. Picture a service failing 80% of the time in the pattern fail, fail, fail, fail, succeed, fail, fail, fail, fail, succeed. Every fifth call lands, and every fifth call wipes the counter back to zero, so the count never reaches five and the circuit never opens. The service is plainly, measurably broken — four of every five requests are failing — and the breaker sits there reset to zero, doing nothing, technically functioning exactly as written. This is the counter-reset problem, and it's not some rare edge: intermittent failure is the common failure. The fix is the next pattern.

The counter-reset problem comes from asking the wrong question. "How many failures in a row?" is brittle, because one stray success resets it. The right question is "what fraction of recent requests failed?" — and that one doesn't care whether the failures came in a tidy streak or salted through a stream of successes.

So instead of a counter, keep a window of the last N results — success or failure — and trip when the failure fraction crosses a threshold. (Full code: Appendix A.4.) The fail-fail-fail-succeed pattern that defeated the counter now reads as a 75–80% error rate over the window, sails past a 50% threshold, and trips the circuit exactly as it should. The occasional success no longer launders the failures away, because nothing gets reset — results just age out of the window as new ones arrive.

This is what Hystrix and Resilience4j do out of the box, and for good reason. The window is usually count-based rather than time-based — easier to reason about, and it self-scales with traffic instead of with the clock. The threshold is a percentage, so a steady drizzle of failures trips it just as surely as a downpour; you're no longer hostage to whether the failures happen to line up consecutively.

One subtlety bites low-traffic circuits specifically: you need a minimum sample before a percentage means anything. Two requests, one failed, is not a "50% error rate" — it's two requests and a coin flip. Hystrix won't even evaluate the threshold until it has seen 20 requests in the window, which is the right instinct, but think about what it implies for a quiet endpoint. A circuit that sees five requests a minute needs four minutes to reach a verdict — four minutes during which a genuinely dead dependency keeps getting called, slowly, the whole time. If you're protecting a critical but low-traffic path, set the minimum sample to your traffic, not to the library's default, or you'll have built a smoke detector that insists on smelling twenty separate fires before it's willing to commit to an opinion.

Error rate has a blind spot, and it's a wide one: a service can return a flawless 200 OK on every single request and still be killing you, as long as each of those 200s takes five seconds to arrive. Zero errors. Circuit stays closed. Thread pool fills anyway with requests that are technically succeeding, slowly, all the way to the bottom. An error-rate breaker watches this happen and sees nothing wrong, because by its own definition nothing is.

The fix is to treat slowness as a kind of failure and add a second trigger on latency — trip when, say, p99 crosses a threshold, regardless of what the error rate is doing. (Full code: Appendix A.5.)

This is Stripe's model for payment processing, and the reasoning is pure product. When p99 latency on a payment-processor call crosses 500ms they start shedding via backoff; cross one second and the circuit opens outright. Why so twitchy? Because for a payment, slow is failed. A customer staring at a spinner during checkout abandons the cart, and a "successful" charge that lands after they've already given up and closed the tab isn't a success — it's a support ticket and a refund wearing a success's clothes. A fast, honest rejection ("try again in a moment") beats a slow yes every time, because the customer is still there to read it.

It costs more to build, and that's a real line on the ledger: per-dependency latency histograms, live percentile math, and a considered choice of window size, none of it free. But on any path where latency is as load-bearing as availability — checkout, search, anything a human is actively waiting on — it's the correct model. An error-only breaker guarding a payment path is a breaker watching for half the ways payments fail and politely ignoring the other half.

Fast fail vs. graceful degradation

When the circuit is open, you still have to hand the caller something. Two schools.

Fast fail returns an error, immediately and honestly. Simple, predictable, and it shows up in your logs and dashboards in bright red where you can see it — which matters more than it sounds, because the alternative can hide problems in plain sight. Good for any operation with no sensible fallback.

Graceful degradation returns a substitute — a stale cache entry, a default, a stripped-down response. Nicer for the user when it works, more code to write, and a brand-new failure surface of its own: now you can serve wrong instead of serving nothing, which is occasionally a worse outcome wearing a friendlier face.

Netflix fast-fails the home page's recommendation call on purpose. Missing recommendations are survivable — the page renders, just with a generic trending row instead of a personalized one, and most users never consciously notice. But serving stale recommendations from three days ago could be actively worse: the titles might be gone, the user's taste might have moved, and now you're confidently showing them the wrong thing instead of admitting you've got nothing. The lesson generalizes past Netflix. "Graceful degradation" sounds strictly superior until you remember it requires a fallback that's actually meaningful, actually safe to serve, and actually handled correctly by every caller. Fast fail is honest about not knowing. Degraded responses are a bet that your fallback is better than the truth, and that bet doesn't always pay.

Threshold sensitivity

Set the trip threshold low — open at a 3% error rate — and you catch real problems fast, along with every transient hiccup that was never going to matter. Your circuit will flap open on an ordinary Tuesday because a downstream had one slow GC pause, and flapping — rapid open/close/open cycling — is its own failure mode: it means the breaker is making decisions on noise. Track state changes as a metric. If a circuit flips more than once an hour during normal operation, the threshold is too jumpy and you're protecting against the weather.

Set it high — open at 30% — and you're robust to noise but slow on the real thing. By the time nearly a third of requests are failing, you've already been in a bad way for a while and a lot of threads have already died waiting.

There's no universal right number, which is exactly why the companies pick different ones and can all be correct. Stripe runs a tight 10% error-rate window, because their failures cost money the instant they happen. Hystrix ships a deliberately loose 50% default, because at Netflix's volume even 50% can be a sliver of total traffic, and tripping a breaker you didn't need to trip is its own outage — one you chose — affecting millions of people. Same pattern, opposite tuning, both defensible, because they're pricing different kinds of pain.

Recovery time

The cooldown — how long the breaker stays open before probing — is the same tension pointed at recovery. Short (ten seconds) and you recover fast but probe a half-healed service aggressively, and every failed probe shoves the clock back to the start. Long (five minutes) and you give a struggling service real room to breathe, at the cost of showing users errors for five solid minutes on a dependency that may well have been fine after thirty seconds.

The honest answer is usually shorter than your nerves want it to be. If a service reliably needs more than a couple of minutes to recover, you don't have a cooldown-tuning problem — you have a deployment or infrastructure problem that no breaker setting can paper over. Set the cooldown to your dependency's actual, measured restart time, not to how nervous you feel about probing too soon. Your anxiety is not a metric.

Netflix: the case for breakers, then the case against the library

Netflix built Hystrix in 2012, open-sourced it in 2013, and put it into maintenance mode in 2018 — a little arc worth sitting with, because it isn't a story of the pattern being wrong. The pattern was dead right. They just outgrew the shape they first built it in.

The original problem was the home page, which depended on a dozen-odd microservices — recommendations, user data, billing, continue-watching, and a long tail of others. At Netflix's scale the odds that all of them were healthy at any given instant rounded to zero; something was always a little degraded somewhere. Without breakers, the home page was only as reliable as its flakiest dependency on its worst day — which is to say, not reliable. With a breaker on each call, a sick dependency got cut loose and the page rendered without it: a static trending list where the personalized row would have been. Not as good. But present, and present beats blank every time. Users don't notice missing personalization. They absolutely notice a white screen.

The insight Netflix made a point of publishing is the one that flips the whole pattern around in your head: the breaker doesn't only protect the caller. It protects the callee — the broken service itself. When you stop hammering a struggling dependency, you stop adding load to something already underwater, and you give it the one thing it needs to recover, which is a moment when fewer people are yelling at it. The breaker is, weirdly, a kindness to the service it's refusing to call. Cutting someone off is sometimes how you help them.

So why deprecate Hystrix? Because Netflix moved to an Envoy-based service mesh, and circuit breaking slid down into the proxy layer, out of application code entirely. Same logic, cleaner home: no per-language library to keep updated across hundreds of services, no forty teams maintaining forty slightly-divergent threshold configs, no arguments about whose breaker behaves how. When the pattern is infrastructure instead of a library, everybody gets it for free and nobody gets to implement it subtly wrong. That's not a knock on Hystrix — it's what success looks like. The idea got important enough to deserve a layer of its own.

Stripe: latency as a first-class signal

Stripe's breakers reflect Stripe's product, the way everyone's do. Netflix can shrug off an occasionally slow home page. Stripe cannot shrug off a slow payment, because a slow payment and a failed payment are nearly the same event to the only person who matters — the customer at checkout, watching a spinner, deciding whether your merchant still deserves the sale.

So Stripe tracks latency as a primary trip signal, not an afterthought to error rate. Cross 500ms p99 on a processor call and they shed via backoff; cross one second and the circuit opens. The caller gets a fast 503 with a Retry-After header instead of a thirty-second hang — which buys two things. The customer's checkout can say "give that another try" right now instead of freezing, and Stripe's own systems don't pile up threads waiting on a processor that's visibly having a bad day. Fast rejection keeps both sides healthy.

The second benefit is the one teams forget to value until they're mid-incident: debuggability. Because the breaker watches latency, its state changes are a leading indicator — p99 creeps up, the backoff rate ticks up, the circuit opens, and all of this shows up in metrics before the first human-facing alert fires. You get a timeline, in order, of a dependency going bad. Compare that to debugging a raw cascade, where you arrive after the fact and reconstruct the sequence backward from logs and vibes. Breaker state turns "what happened?" into "watch it happen," and at 3 a.m. that is worth more than almost any other instrumentation you own.

Netflix Hystrix (Java, deprecated). The original, and still worth understanding because it named the concepts everyone else borrowed. Hystrix did two things: circuit breaking, and thread-pool isolation — giving each dependency its own bounded pool so a slow service could only ever exhaust its threads, not your whole application's. (Full code: Appendix A.6.) Honestly the isolation was the bigger idea, and it gets its own chapter later under its own name: bulkheads. Hystrix has been in maintenance since 2018 — use Resilience4j for new Java work. The API rhymes, it's actively maintained, and its metrics integration is the part you'll be glad for later.

Envoy (sidecar / service mesh). The current default for polyglot fleets, for a good reason: the breaker lives in the proxy, so application code doesn't change, in any language. (Full config: Appendix A.7.) Envoy handles detection, rejection, and recovery probing transparently underneath you. The catch is the flip side of that transparency — Envoy only sees HTTP, so it trips on status codes, not on meaning. If your service returns 200 OK with a body that says {"error": "everything is on fire"} — an antipattern, yes, and also a thing that exists in basically every production system on Earth — Envoy sees a 200 and a healthy dependency. Proxy-layer breaking is only as honest as your status codes are.

Resilience4j / Spring Cloud Circuit Breaker (Java). Annotation-based and genuinely pleasant to bolt onto an existing Spring app — tag a method, name a fallback, done. (Full code: Appendix A.8.) The gotcha is the one from the naive-solutions section, now wearing a production badge: by default the circuit state is per JVM. Fifty app servers means fifty independent breakers, each forming its own private opinion about whether the dependency is healthy. A circuit tripped on three of them is still closed on the other forty-seven, which means your "broken" dependency is cheerfully receiving 94% of normal traffic while you congratulate yourself on having circuit breakers. To get one shared verdict you need a shared backing store — Redis, usually — which gives your breaker a dependency of its own, which is exactly as comfortable as it sounds. This is the distributed-state tax the naive-solutions section warned about, arriving now with interest.

The point of these isn't the answer — it's the argument you have with yourself getting there. Paste either one into an AI and you'll get a clean, confident response in seconds that skips the only thing that matters: your dependencies, your traffic shape, your blast radius. Sit with the friction before you go looking for someone to agree with you.

Exercise 1: The flapping breaker. Your breaker trips at five failures. Payment Service serves this pattern over ninety seconds: three errors, two successes (counter resets to zero), three errors, two successes (resets again), three errors, two successes. The circuit never opens. The service is failing 60% of requests, but the dribble of successes keeps the counter from ever reaching five, so your breaker does precisely nothing while users eat intermittent errors.

Redesign the detection with a sliding window. Specify it concretely: what holds the window, how you compute the error rate, what threshold you set — and the part people skip, what minimum sample you require before the rate is allowed to mean anything. Hint: a deque(maxlen=N) gives you a bounded window for free, and the error rate is just window.count(False) / len(window). With ~60 requests in 60 seconds and a 50% threshold, this trips inside the first minute, and the counter-reset trick that defeated the naive version is gone by construction — there's no counter left to reset.

Exercise 2: The slow success. Payment Service is returning 200 OK on every request, but p99 latency has walked from 100ms to five seconds. Your error-rate breaker stays serenely closed, because by its lights nothing is failing — it's just all arriving very, very late.

Design the detection and the response. How do you instrument latency, what threshold opens the circuit, and what do you hand back to clients when it's open? Hint: keep a rolling window of response times and watch a high percentile (p95 computes cheaper and reacts sooner; p99 is stricter). When p99 stays over ~1000ms for a sustained stretch — say ten consecutive readings, so one slow GC pause doesn't trip you — open the circuit and return a 503 with Retry-After: 30, so clients come back on a schedule instead of immediately. This is the gap a pure error-rate breaker leaves wide open, and it's the one that quietly empties your thread pool.

← Chapter 1 (Rate Limiting). Rate limiting protects a healthy system from too much good traffic; circuit breakers protect a system from traffic aimed at a broken dependency. Ingress versus egress — one caps what comes in, the other cuts off what's going nowhere. They're not alternatives; a serious system runs both, on opposite ends of the request.

← Chapter 2 (Idempotency). A breaker that trips mid-flight can leave an operation half-done, and the natural fix — retry once the circuit closes — is only safe if the handler is idempotent. No double charges, no double bookings. Breakers and idempotency are a matched set; either one without the other leaves a gap exactly where it hurts.

→ Chapter 5 (Retries). Breakers and retries are partners, not rivals: retry with backoff while the circuit is closed or half-open, reject instantly while it's open. Chapter 5 is where we make retries behave — backoff and jitter that don't rebuild the thundering herd on recovery, which is the very stampede the half-open state exists to prevent.

→ Chapter 13 (Bulkheads). Thread-pool isolation — the other half of what Hystrix actually did. A breaker stops calls once a dependency is failing; a bulkhead caps how many calls can be in flight at once, so one slow dependency can't drain a pool everything else needs before the breaker even has the signal to trip. Different moments on the failure timeline, strongest deployed together.

Circuit breakers are one of the few patterns where the failure they prevent is the entire reason to learn them. Read cold, with no cascade in your memory, the whole thing can look like ceremony — a lot of state machine for an error you could have returned anyway. Watch one cascade take down a service that had nothing wrong with it except its choice of friends, and you will never again ship a synchronous call into a critical path without asking where the fuse goes.

The chapter keeps the code out of the narrative on purpose. Here it is, collected, runnable, and annotated. Each entry is the full version of something described in plain language in the body — read the prose for the idea, come here when you want to type it out. All of it is illustrative rather than production-hardened: the in-process versions skip the distributed-state problem, and none of them handle every edge you'll meet in the real thing.

A.1 — Naive Retry with Exponential Backoff (the half-right reflex)

1for attempt in range(5):
2    try:
3        result = call_payment_service()
4        return result
5    except Timeout:
6        wait(2**attempt)  # 1s, 2s, 4s, 8s, 16s

Correct for transient blips, destructive for sustained outages: 31 seconds of blocked thread per request to re-learn that the dependency is still down. Worse, every client's backoff expires in lockstep, so the retries arrive as a synchronized wave right when the service is trying to stand back up.

A.2 — Exponential Backoff with Jitter (better manners, not a cure)

1wait(2**attempt + random.uniform(0, 2**attempt))

The jitter smears the synchronized retry wave across time so a recovering service isn't rear-ended by every client at once. Do it regardless — but it only fixes the shape of the stampede. The requests still go out and the threads still block; this is an improvement to how you fail, not a reason to stop calling a broken service.

A.3 — The Three-State Circuit Breaker

1class CircuitBreaker:
2    def __init__(self, failure_threshold=5, success_threshold=2, timeout=60):
3        self.failure_count = 0
4        self.success_count = 0
5        self.failure_threshold = failure_threshold
6        self.success_threshold = success_threshold
7        self.timeout = timeout
8        self.last_failure_time = None
9        self.state = "CLOSED"
10 
11    def call(self, func):
12        if self.state == "OPEN":
13            if time.time() - self.last_failure_time > self.timeout:
14                self.state = "HALF_OPEN"
15                self.success_count = 0
16            else:
17                raise CircuitOpenError("Circuit is open — failing fast")
18 
19        try:
20            result = func()
21            self._on_success()
22            return result
23        except Exception:
24            self._on_failure()
25            raise
26 
27    def _on_success(self):
28        self.failure_count = 0
29        if self.state == "HALF_OPEN":
30            self.success_count += 1
31            if self.success_count >= self.success_threshold:
32                self.state = "CLOSED"
33 
34    def _on_failure(self):
35        self.failure_count += 1
36        self.last_failure_time = time.time()
37        if self.failure_count >= self.failure_threshold:
38            self.state = "OPEN"

Correct enough to ship, wrong enough to teach you something. The bug is in _on_success: a single success resets failure_count to zero, so an intermittently failing service (fail-fail-fail-fail-succeed, repeating) never trips the circuit even while failing 80% of the time. That counter-reset problem is what the sliding window fixes.

A.4 — Sliding Window Circuit Breaker

1from collections import deque
2 
3window = deque(maxlen=100)  # True = success, False = failure
4 
5def error_rate():
6    if len(window) < 10:  # Need minimum sample before we trust the rate
7        return 0.0
8    return window.count(False) / len(window)
9 
10def record_result(success: bool):
11    window.append(success)
12    if error_rate() > 0.50:
13        open_circuit()

Trips on the fraction of recent requests that failed, not on a consecutive streak — so intermittent failure can't launder itself away with the occasional success. This is the Hystrix/Resilience4j default. Mind the minimum-sample guard: on a low-traffic circuit, raise it to match your real volume or the rate is just noise.

A.5 — Latency-Based Circuit Breaking

1import statistics
2 
3latency_window = deque(maxlen=100)  # Response times in ms
4 
5def p99_latency():
6    if len(latency_window) < 20:
7        return 0
8    sorted_window = sorted(latency_window)
9    return sorted_window[int(len(sorted_window) * 0.99)]
10 
11def record_latency(ms: float):
12    latency_window.append(ms)
13    if p99_latency() > 1000:  # p99 > 1 second
14        open_circuit()

The second trigger an error-rate breaker is missing: a service returning 200 OK on everything while taking five seconds a call shows zero errors and still drains your pool. Watching p99 catches the slow-success failure — the one that loses customers on a checkout path.

A.6 — Hystrix Configuration (Java, deprecated)

1HystrixCommandProperties.Setter()
2    .withExecutionTimeoutInMilliseconds(500)
3    .withCircuitBreakerErrorThresholdPercentage(50)
4    .withCircuitBreakerRequestVolumeThreshold(20)

The original. The RequestVolumeThreshold is the minimum-sample guard from A.4 by another name — 20 requests before the percentage is allowed to mean anything. In maintenance since 2018; reach for Resilience4j on new Java work.

A.7 — Envoy Outlier Detection (YAML)

1outlier_detection:
2  consecutive_5xx: 5
3  interval: 30s
4  base_ejection_time: 30s
5  max_ejection_percent: 50

The same breaker, moved into the proxy so application code doesn't change in any language. The limitation lives in consecutive_5xx: Envoy trips on HTTP status, not meaning, so a service that returns 200 OK with an error body sails right past it. Proxy-layer breaking is only as honest as your status codes.

A.8 — Resilience4j / Spring Cloud Annotation (Java)

1@CircuitBreaker(name = "payment-service", fallbackMethod = "paymentFallback")
2public PaymentResult processPayment(Order order) { ... }

Pleasant to add, with one trap that doesn't show up until you scale out: circuit state is per-JVM by default. Fifty servers hold fifty independent opinions, so a dependency that's "broken" on three of them still gets 94% of normal traffic. A shared verdict needs a shared store (Redis, usually) — and now the breaker has a dependency of its own.

Next: Chapter 5 — Retries: trying again without turning one failure into a thousand.