The Accelerator as a Resource

Introduction

The meeting was about buying more GPUs. A team had been three weeks in a quota queue, blocked on cards they'd been promised and hadn't received, and they'd come to ask for budget to go around the queue — reserve a block from a second cloud, pay the premium, unblock the roadmap. Reasonable ask. Everyone in the room had signed off on the shape of it before I got there. I asked the only question I had, which was how busy the cards we already owned were, and someone pulled up the fleet dashboard, and it was beautiful. A wall of green. Average GPU utilization across the cluster: ninety-four percent. Case closed, apparently — we were clearly out of headroom, clearly needed more.

Except I'd been burned by that exact number before, so I asked the second question, which is what "ninety-four percent" was measuring. It took a week and a borrowed script to answer, because nobody had wired up the metric that would answer it. When we finally had it, the number that mattered — the fraction of the hardware's actual throughput going to the computation anyone was waiting on — was in the teens. The cluster was ninety-four percent busy and something like fifteen percent useful. We were not out of GPUs. We were out of GPUs the way a library with every book checked out and unread is out of books. The team waiting three weeks for quota was waiting behind cards that were, in every sense that paid the bill, idle.

That gap — between what the accelerator cost and what it actually did — is the whole subject of this atlas, and this chapter is about why the gap is invisible by default and expensive by design. The instinct it defeats is a good one, honed on every server you've ever run: a machine that's up, reachable, and showing load is a machine earning its keep. Cheap to believe, because for a CPU box it's true. The idle web server in the corner costs a rounding error; nobody audits it. Carry that instinct to a rack of accelerators and it quietly inverts from thrifty to ruinous, because the one resource you've started treating like a server is the first one in decades expensive enough that idleness is the dominant cost, not a rounding error.

Here is the inversion the chapter asks you to make. You were trained — by a career of capacity planning, by every autoscaler you've configured — to keep machines up, and to treat compute as an elastic pool you draw from when load demands and release when it doesn't. The accelerator breaks both halves of that. It is not elastic: you can't assume the cloud has a thousand of them the moment you want them (that's Chapter 10). And keeping it up is worth nothing; keeping it full of useful work is worth everything, because you rent the hours whether or not the work shows up. Meta trained Llama 3 on 16,384 of these cards wired into a single machine and spent a staggering amount of its engineering on one goal — keeping that gang fed — because a card in that cluster doing nothing was the most expensive nothing in the building. Meanwhile a whole industry of specialist clouds — CoreWeave, Lambda, and their kind — exists to rent you the identical silicon by the hour, having already repriced the GPU as exactly what it is: a metered resource, not a server you own and forget.

Here's the ground we'll walk together:

• Why uptime is the wrong metric for an accelerator and occupancy is the right one — and why that one asymmetry, idle-costs-cents versus idle-costs-dollars-a-minute, is the seed the entire atlas grows from.
• Why "utilization" is not one number but three that routinely disagree by huge margins — and why the most confident number on your GPU dashboard is the one that means the least.
• The three shapes of demand — training, online inference, batch — that each want a different resource strategy, and why most waste is one shape running on another's infrastructure.
• Where the money actually leaks on its way from the invoice to useful work: memory, the host boundary, sharing, scheduling, the interconnect, procurement — a map of the chapters ahead.
• The failure modes that keep a seven-figure cluster busy and useless at the same time, and why smart teams ship every one of them.
• The whole atlas laid over that path, so you know which chapter to reach for when a given stretch of it starts hemorrhaging money.

The Problem, Actually

You are billed for the hour, not the work. Everything about the accelerator conspires to sell you hours with no work in them.

Start with the asymmetry, because everything else is a consequence of it. A CPU server that is powered on and idle costs you something on the order of pennies an hour, and you have spent your whole career correctly ignoring that cost — the idle box is cheaper than the meeting it would take to reclaim it. A top-end training GPU is a different kind of object. Idle, doing absolutely nothing, it costs several dollars an hour for a single card and tens of dollars an hour for a full eight-card node — more, per hour, than the salary of the engineer who left it running. The card doesn't care whether it's computing. The meter runs identically. You are not paying for work; you are paying for access to the capacity to do work, by the second, and whether any work happens inside that second is a separate question the invoice never asks.

The ML-systems atlas has a name for the true cost of putting a model into production — the Cost Ledger, the standing army of pipelines, on-call, and infrastructure you fund whether or not the model earns its keep. The accelerator is that army's most expensive soldier by an order of magnitude, and it has a peculiar property the rest of the ledger doesn't: it bills at full rate while asleep. A pipeline that isn't running costs little. A GPU that isn't computing costs exactly what a GPU that is. So the accelerator line item is not really a compute cost at all. It's a reservation cost, and the discipline it demands is the discipline of anyone who has ever paid for a resource by the hour and had to keep it full — a hotel with empty rooms, a factory line running half-staffed, a leased truck driving empty. The GPU is the first compute resource in decades scarce and expensive enough to deserve that kind of thinking, and almost no one applies it, because it doesn't look like a hotel. It looks like a server, and servers you keep up.

Sit with the arithmetic, because it's the atlas in one line. Take a modest cluster — call it a hundred top-end cards, a number a single mid-size team can burn through. Run it at the fifteen percent useful-work rate from the meeting above, which is not a strawman; it is roughly what a fleet does when no one is measuring the right thing. You are then paying for a hundred cards and getting the useful output of fifteen. The other eighty-five are not broken. They are not even, by the dashboard's account, idle. They are busy — running something, warm to the touch, lighting up every activity graph you own — and producing nothing anyone is waiting for. That is the villain of this book: not the crashed cluster, which at least pages someone, but the humming one, the seven-figure line item that reads one hundred percent utilized and does fifteen percent of a job, while three aisles over a team waits three weeks for cards it's told don't exist.

The Three Numbers Called "Utilization"

The meeting went wrong on a single word. "Utilization" sounds like one thing, the way "temperature" sounds like one thing, and so a dashboard that says "94% utilization" feels like a fact about how hard your money is working. It is three different facts wearing one label, and they disagree with each other by margins large enough to reverse a purchasing decision.

Here's the line to carry out of this section: the most confident number on your GPU dashboard is the one that means the least. Device activity is the easiest to collect, so it's everywhere; it's the most reassuring, so it's what gets shown to the people who sign purchase orders; and it's nearly decoupled from useful work, so it's the one you should trust last. If you have felt the itch of recognition here — that your own green dashboards might be measuring liveness and calling it efficiency — that itch is the correct response. It is why the ML-systems atlas's recurring villain rhymes with this one. There, silent wrongness is a green board over an answer that's quietly wrong. Here it's a green board over a card that's quietly doing nothing. Same disease, different organ: the instrument everyone trusts is pointed at the wrong thing, and the wrongness has no exception to throw.

Three Shapes of Demand

The second thing the meeting got wrong, underneath the metric, was treating "GPU workload" as one kind of thing that wants one kind of infrastructure. It isn't, and it doesn't. Accelerated work comes in three shapes, and they have almost nothing in common except the silicon they run on.

Name these shapes and a category of waste becomes obvious that was invisible before, because most GPU waste isn't idleness — it's mismatch. Provision always-warm, latency-tuned inference infrastructure and then run a batch job on it, and you've bought expensive readiness for work that didn't need any and could have run on interruptible capacity at a fraction of the cost. Run a latency-critical service on a cluster tuned for training throughput and you'll eat tail-latency incidents from scheduling that was designed to maximize gang progress, not protect the ninety-ninth percentile. The cards are busy in both cases. The dashboard is green. And the money is going to the difference between the shape of the demand and the shape of the infrastructure it landed on. Nearly every chapter after this one is, at bottom, about matching one of these shapes to the mechanism that serves it without waste.

Where the Utilization Leaks

If you follow a dollar from the invoice to a unit of useful work, it leaks the whole way down, and the chapters of this atlas are organized around the leaks in the order you meet them.

It leaks at memory first, before the card computes anything, because what you can load bounds what you can run: weights, activations, the KV cache that grows with every token, the optimizer state that multiplies during training — miss the budget and the job doesn't run slowly, it doesn't run at all (Chapter 3). It leaks at the host boundary, the slow bus between the CPU that feeds data and the GPU that consumes it, where a card can sit starved and "busy" waiting on an input pipeline that can't keep up — most "slow GPU" incidents are a fast GPU waiting on a slow everything-else (Chapter 2). It leaks through sharing, because one workload rarely fills a modern card and an unshared card is usually a stranded one, but every mechanism for slicing it up trades isolation for utilization differently (Chapter 5). It leaks through scheduling, where the difference between a cluster at forty percent and eighty percent is often no new hardware at all, just a scheduler that can pack a gang, preempt safely, and backfill the gaps (Chapters 6 and 7). It leaks across the interconnect the moment you go past one card, where how your accelerators are wired decides which jobs are even possible and the network, not the chip, becomes the bottleneck (Chapter 8). And it leaks at procurement, where a card you bought for a peak that never came, or a reservation you dropped and now can't get back, is a depreciating asset you're paying for and not using (Chapters 9 and 10).

Failure Modes Worth Naming

Every one of these keeps a cluster expensive and useless at the same time, and every one ships in production not because anyone is careless but because each was a locally reasonable decision. On-call, notice, does not get paged for any of them. That's the point — none of these throw an error.

Utilization theater is the meta-failure the others hide behind, so it gets top billing — it's the reason the fifteen-percent cluster looks like a ninety-four-percent one, and the reason a purchasing decision can be made exactly backwards with everyone in the room acting in good faith. Notice the what you see column: every row is either "nothing" or a second-order symptom (a bill, an incident) that arrives far from the cause. That column is the discipline of this atlas in one glance — the failures that cost the most are the ones with no first-order signal.

The one I'll never fully live down is the idle held card, because it's so stupid and so expensive at once. A single developer opened an interactive session on an eight-GPU node to debug something, got pulled onto an incident, and forgot. The node sat pinned for the better part of a week — allocated at one hundred percent, active at zero, eight of the scarcest cards in the company held hostage by a Jupyter tab. Nobody noticed because nobody was measuring the gap between held and used, and the person who'd have noticed was the person who'd walked away. The card cost the org more that week than the laptop it was opened from. The fix isn't a lecture about closing notebooks; it's making the holding cost something — idle reclamation, time limits, showback that lands the bill on the team that pinned it — because a resource that's free to hoard will be hoarded, every time, by people doing nothing wrong.

What the Companies Actually Built

Meta, and the discipline of keeping the gang fed

The clearest picture of training-as-a-gang is Meta's Llama 3 pretraining, because they published the operational underside most teams hide. The 405-billion-parameter model trained on a cluster of 16,384 H100 GPUs — one job, wired together, running in near-lockstep. At that scale the enemy isn't slowness, it's interruption: over one 54-day snapshot of the run, Meta logged 466 job interruptions, 419 of them unexpected — an unplanned failure roughly once every three hours, with faulty GPUs and their memory accounting for nearly half. A synchronous gang runs only as fast as its unluckiest card, and stops entirely when that card dies, so a failure every three hours is an existential threat to ever finishing.

What they built to survive it is the whole lesson: automated detection of the dying card, fast checkpointing so a failure costs minutes not days, and enough operational machinery that despite a hardware failure every three hours they held effective training time — their name for the fraction of wall-clock spent actually making progress — above ninety percent. Sit with that pairing, because it's this chapter's utilization layers made concrete: ninety percent effective training time — the gang was making progress ninety percent of the wall-clock — and, by Meta's own reporting, a Model FLOPs Utilization around forty percent, meaning that of the arithmetic the hardware could theoretically do, forty percent went to the model. Both numbers are honest; they measure different things; and the naive reader who hears "utilization" collapses them into one and is wrong about the cluster either way.

Google's PaLM, and the metric that refuses to flatter you

The reason we can even say "forty percent" honestly is that Google's PaLM team, training a 540-billion-parameter model back in 2022, needed a number that couldn't be gamed and introduced one. Model FLOPs Utilization measures observed throughput against the hardware's theoretical peak — implementation-independent, impossible to inflate by running an inefficient kernel, and therefore deeply unflattering. PaLM reported an MFU of 46.2 percent for a run its authors were, justifiably, proud of. Read that number the right way: one of the best-resourced, most carefully engineered training runs of its era, executed by people who do this at the frontier, converted a bit under half of its hardware's theoretical capacity into useful work — and that was the good result, the one worth publishing.

The detail I love is that PaLM ran on TPUs, not GPUs at all. MFU doesn't care; it's a ratio of useful work to theoretical peak, and it means the same thing on any accelerator, which is exactly why this atlas treats the accelerator as a resource rather than a product. The vendor changes, the units change, the embarrassing gap between activity and useful work does not. When you find yourself defending a device-activity number, remember that the people who built PaLM had every reason to report the highest figure they honestly could, and the highest honest figure was forty-six percent. If your dashboard says ninety-four, you are not doing twice as well as PaLM. You are measuring a different, easier thing.

The neoclouds, and the accelerator repriced as a meter

The market has already internalized everything this chapter argues, and you can watch it in how the specialist GPU clouds — CoreWeave, Lambda, and the wave behind them — sell. They rent you the identical silicon the hyperscalers rent, substantially cheaper, and they do it by the hour with the pricing model of a resource, not a server. The tell is the billing unit: on the standard eight-GPU node there's often no single-card option — you rent all eight and you pay for all eight, whether your job fills two of them or eight. That's not a trick; it's honesty about what the resource is. The card doesn't get cheaper because you only used part of it, so the price doesn't either.

Which is where this example complicates the chapter's own advice, and I want to be honest about it rather than tidy. Renting from a neocloud does not fix your utilization problem. It converts it from a capital problem into an operating-expense problem and hands the idle-card risk to a vendor who has priced that risk into the rate and made it their margin. If you rent eight cards and usefully use one, you have not saved money by renting; you've just moved the fifteen-percent cluster onto someone else's balance sheet and rented it back at a markup. The neocloud is the correct choice for spiky and uncertain demand precisely because someone has to eat the idle time and it shouldn't be you when your demand is unpredictable — but the buy-versus-rent decision is a utilization bet, not a price comparison, and this whole book is really about the same one bet made in five different places. Chapters 9 and 10 make it explicitly.

How to Read This Atlas

If the gap between the invoice and the useful work is the subject, then this atlas is a walk down the pipe from the first figure — one leak per chapter — and the fastest way to use the book is to find the stretch of pipe that's bleeding money in your shop and turn to its chapter.

Two of those entries are less techniques than promises this chapter is making. Chapter 4 is the fifteen-percent number from the opening meeting, measured properly and defended against the theater — it's the keystone, and the chapters that follow are what you do once you can see the truth. Chapter 20 is the meeting itself, done right: the principal-level habit of walking the utilization layers before signing the purchase order, because most requests for more GPUs are requests to paper over the idle ones you already own. Everything between is a leak in the pipe. (As later chapters are written, each row here links to its chapter; for now the map is the promise.)

The Principal Engineer's Perspective

Exercises

None of these has a clean answer, which is the point. An AI will hand you a confident, four-second response that assumes the fleet you wish you had; the value is in the argument you have with it about the fleet you actually run.

Connections

This is the first chapter of the atlas, so its links run forward and sideways rather than back.

Next: Chapter 2 — What a GPU Actually Is (to a Systems Engineer): why the expensive silicon spends so much of its life waiting on a slow bus, and why most "slow GPU" incidents are a fast GPU starved by a slow everything-else.