The LLM as a System ComponentA Dependency, Not an Oracle
The shape of the whole thing
Five stages, from one suspicious function signature to the questions a principal asks before an incident asks them. Read it top to bottom, or jump straight to the part you came for.
One line of code — and the five properties its signature hides.
A vendor, not a library — four of the five resolve to patterns you already own.
The fifth property has no inherited pattern — failure wearing the uniform of success.
Four production scenarios — each with a trigger, a hiding place, and a move.
Judgment — when the reframe earns its weight, and what to ask your team.
Introduction
The demo was perfect. It always is — that's what demos are for.
Someone typed a hard question into the support assistant, the tokens streamed back, and the answer was fluent, specific, and correct. The room did what rooms do: nodded, relaxed, started talking about launch dates. The hard part looked finished. We had wired a model to a prompt and it had spoken, and the speaking was the product. I'd have signed off on it too.
Then it shipped. Three weeks later it told a customer — warmly, in complete sentences, with the calm authority of a system that has never once been unsure — about a thirty-day refund policy the company had never offered.
Nothing threw. The request returned 200 OK. The latency chart was flat, the error rate was flat, every dashboard was the untroubled green of a system with nothing to report. We found out from the customer, three days later, in a channel that wasn't supposed to page anyone and did.
A component that fails by producing a confident, well-formed, wrong answer instead of an error. Find the call in your own system whose failure would never trip an alert, because it returns success.
This is not a hypothetical. In early 2024 a Canadian tribunal held Air Canada responsible after its website assistant invented a bereavement-fare refund policy and a grieving passenger relied on it.
The airline argued that the chatbot was a separate legal entity responsible for its own actions. The tribunal was unmoved. The assistant was confident, fluent, and wrong, and the company owned every word of it.
There is a great deal of noise around this technology right now — half of it selling a miracle, the other half warning about the end of the world. This atlas is neither. It's the calm, operator's version: what the thing actually is once it's behind an API and in front of a paying customer.
And what it is, is mundane. The model is not a function you call. It is a network dependency you don't operate — one that fails in a way none of your others can: by lying instead of breaking.
You already know how to work with flaky remote services. They time out, cost money, and throttle you, and you have timeouts, retries, circuit breakers, and fallbacks for exactly that. Most of what this atlas teaches is pointing that toolkit at a dependency that happens to speak English. The one genuinely new problem is the lying — the answer that comes back confident and wrong with a 200 on it — and that problem is why the other fourteen chapters exist.
Here's the ground we'll walk together:
- The five properties that turn "call the model" into a systems problem — slow, expensive, rate-limited, nondeterministic, and persuasive when wrong
- Why four of those five are flaky-dependency problems you've already solved, and which pattern each one reaches for
- The fifth property, which has no analog in anything you've shipped: a failure that arrives wearing the costume of success
- The production scenarios this creates, and the move that handles each
- How to read the rest of the atlas — where each property gets worked out in full
None of this is an argument against building with LLMs. It's an argument against believing the demo. By the end, "we hooked up the model and it works" should land on you the way "we made the network call and got a 200" does — true, necessary, and nowhere near done.
Five Properties Hiding in One Function Call
From your program's point of view, an LLM is a function with a suspicious signature: text goes in, text comes out. It looks like String → String — pure, total, instant, the kind of call you make without a second thought.
That signature is a lie of omission. What you're actually calling is a synchronous request to a remote service you have no control over, and it has five properties the signature hides. None is exotic. All are load-bearing.
- Slow. Not slow like a bad query — slow measured in seconds, dominated by how much text it decides to generate, and you don't know the length before you ask. A healthy internal RPC answers in single-digit milliseconds. Budget three orders of magnitude more, and design as if a human is watching a spinner, because one usually is.
- Expensive, in a way that scales the wrong direction. You pay per token, in and out, on every call. Ordinary infrastructure cost scales with load you can cap; this scales with use — the exact engagement your product team is paid to grow. The better the feature does, the more it costs. Success is what runs up the bill.
- Rate-limited by someone who isn't you. The capacity is the provider's, metered against a quota you were granted. Cross it and you get a
429— the polite version of not in my house. You can't add a node to fix it; your spike is not their emergency. - Nondeterministic. The same input can return a different output on Tuesday than it did on Monday, by default. Temperature zero narrows the spread but doesn't honestly close it. Every instinct built on "same input, same output" — caching, testing, debugging by re-running — now needs a caveat it never needed before.
- Persuasive when it's wrong. This is the one to sit with. The model returns an answer that is fluent, well-structured, appropriately hedged, shaped exactly like a correct one — and false. It doesn't throw. It doesn't hand you a low-confidence score to threshold on. It gives you a plausible lie with a
200and moves on, and so does your program.
The first four are severity dials on failures you recognize. The fifth is a different kind of thing, and it's the one you'll spend the most effort learning to catch.
| Property | What it forces | The pattern you already own |
|---|---|---|
| Slow | A timeout and a perceived-latency plan | circuit breaker · streaming |
| Expensive | A per-request cost ceiling | caching · budgets |
| Rate-limited | Backoff and a retry budget | exponential backoff · throttling |
| Nondeterministic | Don't assert on exact output | schema / contract validation |
| Persuasive when wrong | … | no inherited pattern — filling this cell is the rest of the atlas |
Now the arithmetic that reframes the job.
In the demo, the model call was the system: one function, one prompt, one string. In production it's one line in the middle of a function that also sets a timeout, retries on a 429 with backoff, falls back when the provider is down, validates what came back, samples the exchange for evals, meters the cost, and logs the whole thing for the day someone asks what happened.
The model call is maybe ten percent of the code, and a good deal less than ten percent of the work. The other ninety percent is the scaffolding.
And the cruel part — the part that catches good teams — is that the ten percent is the part that demos. You can show the fluent answer on day one. The ninety percent has nothing to show until the day it's missing.
A Call, Not a Function
The move that does the most work here is almost embarrassingly cheap: stop importing a library and start integrating a vendor.
A function in your own process gives you guarantees so total you've never had to name them. It returns. It returns fast. It returns the same thing for the same arguments. It costs nothing you can measure, and it can't be throttled, because nobody on the other side is deciding whether to serve you.
The signature String → String promises all of that silently — which is why the model's text-in-text-out signature feels so familiar and lulls so well. It reads like str.upper(). It behaves like a call to a startup's beta API on a bad day.
- deterministic
- free
- instant
- in-process
- always returns
- RPC to a service you don't operate
- priced per token
- rate-limited by someone else's quota
- seconds-slow · nondeterministic
- returns confident falsehoods under a 200
The model's function signature is the most expensive lie in the stack. It says "library"; it means "someone else's server." Every surprise in an LLM system is the gap between those two words, arriving late.
Read the five properties through that lens and four of them stop being new problems. They're a checklist you've half-finished before:
- Slow is a dependency with a fat latency tail. You know the answer: a timeout, so a hung call can't hold a thread hostage, and a circuit breaker, so a sick dependency fails fast instead of dragging every caller down with it.
- Rate-limited is a
429, and a429has one correct response everywhere it appears: back off, exponentially, with jitter, under a retry budget that caps how much of your traffic is allowed to be retries. Google's SRE practice does exactly this; the model call inherits the technique wholesale. - Expensive is a cost you cap with a per-request budget and drive down with caching — the same instinct as any metered dependency, on an invoice that stings more.
- Nondeterministic is the one your test suite hates. An exact-match assertion passes the day you write it and starts flaking weeks later with no change of yours, because the same prompt won't reliably return the same string. So you stop testing for the golden string and validate the shape instead — the structure and invariants a correct answer must have, whatever words it used.
One more thing rides along free from the same recognition. A service you don't operate will have an outage on its schedule, not yours — so you owe it graceful degradation: a canned response, a cached answer, a smaller local model, an honest "try again in a minute." Anything but a spinner that never resolves.
Notice what happened. Four of the five properties — slow, rate-limited, expensive, nondeterministic — resolved to disciplines with names, most of which you argued about in a design review last quarter. The model didn't need a new one for any of them. It only needed you to see, against the grain of that friendly signature, that you're talking to a service and not a library.
The reason that's hard to see is that nothing about the developer experience points you there. You install an SDK, not a service mesh. The call sits inline in your code, not behind an obvious network boundary. And in local development the provider is fast, up, and well under quota — so the call behaves exactly like the pure function it's impersonating, right until production supplies the concurrency, the spike, the outage, and the quota ceiling at once.
That gap is invisible while you're forming your intuitions and expensive once you're not looking. It isn't a beginner's mistake. It's a senior engineer's mistake, made because the resilience instincts are strong enough to assume they've already been applied.
The One Axis With No Analog
Four properties down, each ending at a pattern you already own. The fifth ends nowhere, and that's the point.
Every failure you've engineered around in your career has announced itself. It's so deep in your training you've never clocked it as an assumption. A service that's down returns an error. A malformed query throws. A full disk, a dereferenced nil, a violated contract — they all signal.
Your entire reliability toolkit is, underneath, a toolkit for noticing those signals: timeouts fire on slowness, breakers trip on error rates, validators reject bad shapes, alerts watch for anomalies. Every one assumes failure is, in principle, observable the moment it happens.
A confidently wrong answer breaks that assumption at the root:
- It's well-formed, so no parser complains.
- It's plausible, so no human skims it and flinches.
- It returns
200, so no status code betrays it. - It arrives at the same latency as a correct answer, so no timing anomaly appears.
By every signal your system knows how to read, it's a success. The failure is real and the light is green, and there's no threshold you can set on "green" that catches "wrong."
A normal dependency fails by erroring, which your system is built to notice. This one fails by lying fluently, which your system is built to trust. The whole difficulty of LLM engineering lives in that inversion: the failure mode wears the uniform of the success case.
That inversion is why the question underneath this whole atlas is a strange one to have to ask about software: how do I build something reliable out of a component that is confidently wrong — and how do I know when it is?
The second half is the hard half. Reliability engineering has always been able to assume it would know, because failure signals. Here you have to manufacture the knowing — invent the assertion, ground the answer, sample and grade the outputs, capture the trace — because the component won't volunteer it.
It helps to know this property isn't new; the foundation model only made it fluent. Zillow ran machine-learning models to price homes for its iBuying business, and around 2021 those models mispriced at scale — quietly, confidently, with no exception thrown anywhere — and the company wound the business down at heavy cost. No stack trace was involved in losing that money. The Machine Learning Systems atlas calls this silent wrongness and treats it as the defining hazard of shipping any model.
The language model inherits it and sharpens it. A classical model outputs a number you can bound and monitor. A language model outputs fluent prose whose confidence is a stylistic default, not a calibrated signal — it doesn't sound unsure when it's guessing, because it never sounds unsure.
A lawyer found this out in 2023, when a general-purpose assistant handed him case citations that looked exactly like real ones — proper names, plausible reporters, clean formatting — and were entirely invented. The fabricated brief reached a federal court, and the sanctions were not fluent at all.
The model wasn't malfunctioning. Fluency with no attached truth-signal is what it does. Engineering around that is what you do.
The Scenarios You'll Actually Meet
Here's the field guide — the failures the "we just call the model" mental model actually produces, each with its trigger, how it shows up, and the move that handles it. None is a bug in the model. Each is the gap between what the model actually is and the pure function you let yourself picture.
- You'll see
- Your own service's latency and thread pool degrading in lockstep with a dependency you half-forgot was remote — a "quick AI feature" somehow implicated in a checkout outage.
- Why it hides
- In dev the call is fast and up, so the missing timeout never fires and the missing fallback is never reached. The absence of resilience is invisible until the dependency misbehaves, which it never does on your laptop.
- The move
- Treat it like any remote call. A timeout on every request — non-negotiable — and a circuit breaker so one sick provider can't exhaust the threads that serve everything else. One unbounded model call can hold a request thread for the provider's worst latency; enough at once is a thread-pool outage in a feature that has nothing to do with AI.
- You'll see
- Under load, your provider bill and your throttle rate climbing together while your success rate sits flat — you're paying more to be rejected faster.
- Why it hides
- Below the quota the retry genuinely helps, so it looks correct in every low-traffic test. It only inverts once you're over the line — which is exactly when traffic is highest and attention is lowest.
- The move
- The retry-storm fix the Internet-Scale atlas already teaches — backoff with jitter, a retry budget, and a
429treated as a signal to obey, not a coin to reflip.
- You'll see
- Every AI-touching surface failing at once, in a pattern that maps with eerie precision onto someone else's status page, over which you have zero leverage and zero ETA.
- Why it hides
- Provider uptime is genuinely high, so months of green lull the fallback off the roadmap. The outage you didn't design for is rare, correlated, and total when it lands.
- The move
- Graceful degradation, decided before you need it — a canned path, a cached answer, a smaller model you can run yourself. It's the difference between a degraded feature and a dead one.
- You'll see
- Nothing. That's the whole problem — a wrong value moving downstream at full confidence until a human notices, days later, through a channel that isn't your monitoring.
- Why it hides
- It's engineered to hide. It's the fifth property arriving as an incident:
200 OKover a false payload, feeding a database write, a tool call, or a number shown to a user. - The move
- Validate at the boundary, in proportion to blast radius. A human reading the output can be the check; a program consuming it cannot, so it earns a schema, a bounded repair, an eval set, a trace. This is the one the other three don't cover, and the one the rest of the atlas is about.
One discipline sits under all four, and it's the cheapest insurance in LLM engineering: log the whole exchange — the exact input you assembled and the exact output that came back — not just the output. A wrong answer you can't reproduce is an incident you can't close, and nondeterminism means you can't reproduce it by re-running. Most teams keep the completion and throw the input away, then wonder why every failure is a whodunit. The full flight recorder is Chapter 10; the habit starts now.
How to Read This Atlas
The five properties are the map, and the rest of the atlas walks them one at a time. Find the row that names the thing currently keeping you up — that's the chapter to read first. (Chapter links wire up as each one publishes.)
Read them in order or jump straight to your fire; either way you'll come out able to look at any "let's just call the model" proposal and see the ninety percent it left out — and know which chapter each missing piece lives in.
The Principal Engineer's View — Operating a Confidently-Wrong Component
When the reframe earns its weight, and when it's overkill. Every model call earns a timeout — that one isn't negotiable, because an unbounded remote call is a latent outage at any scale. Everything past that is proportionate to blast radius:
- A call whose output a human reads and sanity-checks before anything happens — a draft, a suggestion, a summary — earns light validation and can absorb the occasional confident miss, because the human is the check.
- A call whose output feeds a program with no human in the loop earns the full apparatus: schema validation, bounded repair, an eval set, a trace.
This is the error tolerance axis from the Machine Learning Systems atlas, unchanged: how much you spend catching wrongness is a function of what a wrong answer costs. The costliest mistake in the field is spending a frontier model's validation budget on a low-stakes call while trusting a high-stakes one because it happened to look fine.
The observability that has to exist first. Don't ship a model call you can't replay. The synchronous world gave you a stack trace for free; this world gives you nothing unless you capture it — the exact context in, the exact completion out, the model version, the parameters, the latency, the cost, joined to whatever downstream signal says it went well or badly. Miss the input and your incident review opens with "we can't reproduce it," because you can't. Chapter 10 is the whole flashlight; the point now is that it has to exist before you need it.
The business decision wearing an engineering costume. "Should this feature use an LLM?" looks like architecture and is mostly cost and liability. The invoice scales with adoption, so a runaway hit is a runaway bill. The outputs are your legal responsibility — Air Canada settled that at its own expense. And the capability sits on a provider's roadmap, not yours. Building on a model means taking a permanent operating cost, a standing correctness risk, and a dependency you can't fully control — often worth it, never free, never purely technical.
Cost awareness. The sticker price is per-token and looks small in the demo. The real price is the ninety percent: the validation layer, the eval harness someone maintains, the trace store, the fallback path, the on-call rotation that now includes "the provider is degraded," and the tax on every engineer who has to reason about nondeterminism to change one line. It's worth paying when the model does something genuinely hard that users value. It's absurd when a regex or a lookup table would have done the job without a single one of these properties — which, and say this part in the design review, is more often than the excitement admits.
Questions to take back to your team:
- What's the timeout on this call, and what happens when it fires? "We don't have one" is a latent outage, not a feature.
- If the model returns a confident, well-formed, wrong answer, how do we find out — a dashboard, or a customer? If it's a customer, we're blind on the one failure mode that defines this technology.
- What does a wrong answer cost here? The number sets how much validation the call earns — say it out loud before arguing about the prompt.
- When the provider has an outage — not if — what does the user see? If the honest answer is "a spinner forever," the feature isn't finished.
- Could a rule, a lookup, or a small classical model do most of this without any of the five properties? If so, why are we signing up for all five?
An LLM is not a function you call, it's an unreliable network dependency you don't operate — slow, expensive, rate-limited, nondeterministic, and, uniquely, able to fail by lying fluently instead of erroring. Four of those you already know how to engineer around; reach for the shelf you stocked for every other flaky service. The fifth is the whole reason this atlas exists, and the whole reason the demo fooled you: it showed you the ten percent that speaks and none of the ninety percent that checks. Treat the model as a dependency to engineer around, not an oracle to trust, and the magic collapses into something better — a system you can actually operate.
Exercises
These reward argument over recall. Paste any into a model and you'll get a confident four-second answer that skips the only things that decide the outcome — your call sites, your blast radius, your provider, your users.
Take one model call in a system you own or can picture precisely. For each property, write the specific thing it forces: the timeout, and what happens when it fires; the per-request cost ceiling, and who set it; the backoff and retry budget on a 429; what "nondeterministic" means for how you test this call; and — the one that matters — how a confident wrong answer would be caught. The four you can fill in fast aren't the exercise. The fifth is, and how long you stare at it.
Our example was an assistant inventing a refund policy. Rebuild it in your own system: name the component, the specific confident-wrong output it could produce, the exact reason no exception would fire, and how you'd currently find out it happened. If you can name the alert that would catch it, you've found a hard failure, not a silent one — keep looking for the call where you can't. That's the one this atlas is for.
A colleague proposes an LLM for a feature. Make the strongest case against — the rule, the lookup, the small classical model that gets most of the way with none of the five properties — then the strongest case for, naming exactly what the model does that the cheaper thing can't. "It's more flexible" counts only if you say flexible at what, worth what, at what cost per token and per wrong answer.
In a system you know, find a model call (or any remote call) with no timeout, no fallback, or no validation on an output that feeds a program. Trace what happens to it when the dependency is slow, throttled, down, and confidently wrong — four scenarios, four honest answers. The uncomfortable ones are the exercise.
Connections
This chapter stands on that one and doesn't re-teach it. Timeouts, circuit breakers, exponential backoff, retry budgets, rate limiting and the 429, backpressure, graceful degradation — all assumed here, because the model call is a new and unusually well-dressed place to apply them. Three of the five properties (slow, rate-limited, expensive) reach straight into that atlas; a fourth (nondeterminism) borrows ordinary contract-validation discipline; only the fifth finds no pattern there at all.
The fifth property predates LLMs. That atlas named silent wrongness as the defining hazard of shipping any model, and error tolerance as the axis that decides how much you spend catching it. This chapter inherits both and adds the foundation-model twist: fluency with no attached truth-signal. Zillow's mispricing is that atlas's evidence, borrowed here because losing money to a confident model with no exception thrown is older than the chatbot.
If the model is a fixed, read-only dependency, the only state you control is what you put in front of it — so the next chapter reframes the prompt as an assembled context window, and the window, not the prompt, as the program you're actually writing.
The north-star question — how do I know when it's wrong — becomes two chapters. Chapter 6 is how you invent the assertion the model won't give you; Chapter 10 is the flight recorder that lets you replay the failure you couldn't reproduce.
This chapter named the confident-wrongness property; Chapter 14 is its full taxonomy — confident fabrication, sycophancy, and model-bump drift — each with a tell, a root cause, and a mitigation that traces back to a discipline from an earlier chapter.
Next: Chapter 2 — Context Engineering: why the context window, not the prompt, is the program you're actually writing, and why the most important token in the buffer — the user's own question — has the weakest claim on the space.