What a GPU Actually Is (to a Systems Engineer)
The shape of the whole thing
Four stages, from the machine itself to what a principal makes a team prove before the purchase order. Read it top to bottom, or jump straight to the part you came for.
A GPU is a throughput engine on the far side of a slow bus — not a fast CPU.
Before you spend a dollar: are you compute-bound or memory-bound (the roofline), and how far can you drop precision?
Two ways a hundred-percent dashboard hides an idle card — input starvation and launch overhead — and how each looks on-call.
What the companies prove, which instrument actually answers the question, and what to make a team show before the PO.
Introduction
The ticket said the training job was GPU-bound and asked for a faster card. It was a reasonable request, filed by a competent engineer, and it had the one piece of evidence everyone trusts: a screenshot of nvidia-smi with GPU-Util pinned near a hundred percent. The job took eleven hours. The team wanted the next card up, which cost roughly twice as much per hour, on the theory that twice the card would mean something like half the time. I would have signed it too, a few years and one embarrassment earlier.
Instead we spent an afternoon with a profiler, which is a thing almost nobody does before spending money on hardware, and watched where the time actually went. The card was doing real work for about a third of each step. The other two-thirds it sat there, powered, warm, billed in full, waiting for the CPU to hand it the next batch of images — decode the JPEGs, resize them, augment them, copy them across the bus. The expensive silicon was starved by the cheap silicon that fed it. We didn't buy a faster GPU. We added CPU workers and a prefetch queue so the data was ready when the card wanted it, and the same "GPU-bound" job got about sixty percent faster on the same hardware, for the price of an afternoon.
That gap — between a card that is busy and a card that is fed — is what this chapter is about, and it comes from a single mental error that even strong engineers carry into their first serious GPU work. The error is treating a GPU as a fast CPU. It is not one. It is a different kind of machine with a different theory of speed, and it lives on the far side of a slow bus. Almost every performance surprise you will have with an accelerator comes from forgetting one of those two facts.
Here is the inversion the chapter asks you to make. A CPU is built to do one thing as fast as possible: it is a latency machine, and you reason about it by asking how quickly a single operation finishes. A GPU is built to do enormous numbers of things at once, no one of them in a hurry: it is a throughput machine, and it is only fast when you feed it wide, parallel, arithmetic-heavy work. Ask a GPU to do one thing quickly and it will disappoint you. Ask it to do ten thousand independent things at once and it will astonish you. And either way, it can only work on what has crossed the narrow pipe from the host — so the GPU is often the fastest part of your system and, at the same time, the part that is waiting. Meta serves recommendation models that are bounded not by how fast their GPUs can multiply but by how fast memory can feed them; NVIDIA ships an entire library, DALI, whose reason to exist is that the CPU pipeline in front of the GPU keeps starving it. Both are this chapter, in production, at companies that know exactly what they are doing.
Here's the ground we'll walk together:
- Why a GPU is a throughput machine and not a fast CPU — and why the same instinct that makes you a good CPU engineer will mislead you about every accelerator you touch.
- Where the hidden bottleneck actually lives: the slow bus between the host that feeds data and the device that consumes it, and why "GPU-bound" is far rarer than the dashboard makes it look.
- The one question that predicts whether a faster card will help at all — are you compute-bound or memory-bound — answered without a profiler, on the back of an envelope.
- Precision as a resource lever you didn't know you were holding: how spending numerical range buys you throughput and memory, and what it quietly costs.
- The field guide to a card that is twenty percent busy on a hundred-percent dashboard: the two failure modes that starve accelerators, how each looks on-call, and the fix.
- What to make a team prove before you approve the invoice for a bigger GPU — because the honest answer is usually that the one they have is waiting, not working.
The Problem, Actually
The GPU is the fastest part of your system. Most of the time, it is the part that is waiting.
Start with the two facts, because everything downstream is a consequence of them, and because they are the two facts the nvidia-smi screenshot hides.
The first fact is architectural. A modern CPU has a handful of cores, each one a marvel of getting a single instruction stream to the finish line quickly — deep pipelines, branch prediction, big caches, all the machinery of low latency. A GPU spends its transistor budget the opposite way: thousands of simple lanes, each one individually unimpressive, ganged together to run the same operation across a huge batch of data at once. It hides the latency of any single operation not by finishing it quickly but by having thousands of others in flight to work on while it waits. This is why a GPU is spectacular at a large matrix multiply — tens of thousands of independent multiply-adds, exactly the wide parallel arithmetic it was built for — and why it is mediocre-to-terrible at a branchy, serial, one-thing-depends-on-the-last workload, which is precisely what a CPU is for. Hand the GPU the wrong shape of work and you are paying for ten thousand lanes to watch one of them do something.
The second fact is physical. The GPU is not your computer; it is a separate device attached to your computer, and everything it works on has to get to it. The data lives in host memory, prepared by the CPU, and it crosses to the card over a bus — commonly PCIe, sometimes a faster dedicated link. That bus moves data at tens of gigabytes per second. The memory soldered onto the card itself, the HBM the cores read from, moves data at a few terabytes per second. That is roughly two orders of magnitude. The card can consume data about a hundred times faster than the bus can deliver it, which means that any workload where the card has to keep going back across the bus for more — or where the CPU can't prepare the next batch in time — leaves the cores idling on a starvation diet while the meter runs at full price.
Sit with the arithmetic, because it is the whole chapter in one calculation. Suppose a training step does some fixed amount of compute on a batch, and suppose the batch has to be decoded and copied before the step can run. If the compute takes 40 milliseconds and the data preparation takes 80, then the card works for 40 and waits for 80, and your GPU-Util — which, as the previous chapter established, only asks "was a kernel resident," not "was the kernel worth running" — will read something busy-looking and completely beside the point. You are getting one-third of the card you are paying for, and the fix has nothing to do with the card. A faster GPU makes the 40 into 20 and changes your step time from 120 to 100 milliseconds, a sixth, for double the money. A fed GPU overlaps the 80 with the 40 and gets you most of the way to 80 for the price of a queue. One of those is an architecture decision and the other is a purchase order, and teams reach for the purchase order because it's the one the dashboard seems to be pointing at.
The most expensive stage in a pipeline sitting idle because a cheaper upstream stage can't feed it fast enough. The bottleneck is almost never where the cost is. Find the cheap stage starving your expensive one before you buy a bigger expensive one.
What the Machine Actually Is
The useful mental model of a GPU, for someone who will size instances and debug slow jobs but never write a kernel, has three parts: what the cores are, how work and data reach them, and where memory sits relative to all that. Hold the whole thing in your head and the failure modes stop being mysterious.
The cores are a throughput engine. Picture the card as an enormous room of identical clerks, each slow and none clever, all made to do the same step of the same procedure in lockstep. The design bet is that your work comes in huge, uniform batches — the same operation applied across millions of elements — so that having thousands of clerks matters more than having any single fast one. When the bet pays off, nothing else in your data center comes close. When it doesn't — when the work is a tangle of branches and dependencies where each step needs the answer to the last — most of the clerks stand idle, and you have bought a stadium to seat a chess match. This is the concept to carry: a GPU trades away the latency of any single operation to win at the throughput of many. It is the same throughput-versus-latency trade the internet-scale-systems atlas draws for request handling, made of silicon: the GPU is the most committed throughput machine you will ever provision, and it is exactly as bad at latency-shaped work as that commitment implies.
A CPU is a sprinter and a GPU is a freight train. The train is embarrassing over ten meters and unbeatable over a thousand tons. Choosing the train for a footrace is not a tuning problem; it's the wrong vehicle.
Work and data reach the cores across a boundary — and that boundary is the bottleneck you'll actually hit. Two different things have to cross from host to device, and both can starve the card. The obvious one is data: the batch has to be copied over the bus, and if your input pipeline on the CPU — reading files, decoding, resizing, augmenting — can't produce batches as fast as the card consumes them, the card waits. The less obvious one is control: every unit of work the GPU does is a "kernel" the CPU launches, and each launch has overhead. If your program dispatches thousands of tiny kernels from a chatty Python loop, the CPU-side cost of launching them can dominate the actual computing, and again the card waits — this time not for data but for instructions. Both are the same disease with different symptoms: the accelerator is downstream of a slower feeder, and it is only ever as fast as whatever hands it the next thing to do.
Memory sits in a steep hierarchy, and the level that matters for speed is the one closest to the cores. Briefly, because the next chapter owns it: a little extremely fast memory beside the lanes (registers and on-chip scratch), a large pool of fast memory on the card (HBM), and then, across the bus, the host's ordinary RAM — speed and size trading off at every step, the same latency-versus-memory tension the b2c caching chapter draws, cast in hardware. You need one consequence of it here. Even when all your data is already on the card, feeding the cores from HBM has finite bandwidth, and plenty of workloads are limited by that — how fast memory supplies operands — not by how fast the cores compute. That's not a capacity problem (does it fit?), which is Chapter 3's; it's a bandwidth problem (can we feed the cores fast enough?), and it's the hinge of the next section.
The Two Questions That Size a Workload
Once you accept that the card is a throughput engine fed through a straw, two questions decide almost everything about how a given workload will behave and whether spending money will help. Neither requires a kernel. The first is diagnostic; the second is a lever.
Compute-bound or memory-bound?
Every piece of work moves some number of bytes and does some number of arithmetic operations on them. The ratio — operations per byte moved — is the arithmetic intensity of the work, and it is the single most useful number you can estimate about a GPU workload, because it tells you which resource you'll run out of first. Do a lot of math on each byte you fetch, and the cores are the bottleneck: you are compute-bound, and a card with more arithmetic throughput will genuinely run you faster. Do very little math on each byte — stream a huge amount of data through and touch each value once or twice — and memory bandwidth is the bottleneck: you are memory-bound, the cores spend their time waiting for operands, and a card with more FLOPS will run you at nearly the same speed for more money.
This is the roofline model, named by David Patterson's group at Berkeley in 2009, and it is the most useful thing that fits on a napkin in this entire atlas. Draw achievable throughput on the vertical axis and arithmetic intensity on the horizontal. For low-intensity work the ceiling rises with intensity along a slope set by your memory bandwidth — you are under the sloped "roof," bandwidth-limited. Past a ridge point the line goes flat at the card's peak compute — you are under the flat roof, compute-limited. Where your workload sits on that line, not how big the flat roof is, decides whether a faster card helps.
The reason this matters commercially is that vendors sell FLOPS. The headline number on a new accelerator is its peak arithmetic throughput, each generation's is bigger, and so "faster card" and "more FLOPS" have quietly become synonyms. But a great deal of real work — most of inference, essentially all of recommendation — is memory-bound, sitting out on the sloped part of the roofline where FLOPS were never the constraint. Buy the higher ceiling for one of those and you've paid for altitude you never reach. (Recommendation is the sharpest case; the Companies section returns to it.)
Buying more of the resource you weren't short of. Upgrading the dimension that isn't your binding constraint feels like progress and changes nothing. Before you buy, name the constraint; then check that the thing you're buying is the same thing.
And here the roofline says something bigger than which card to buy. Sometimes "memory-bound, the faster GPU won't help" really means "this workload is the wrong shape for this machine" — embedding-heavy ranking looks like it should love a GPU until you measure it and find a memory-and-networking problem in an accelerator costume, the hardware version of Chapter 1's workload mismatch. The useful move is then to stop feeding expensive silicon cheap-shaped work.
Precision as a lever
The second question is a knob you can turn, and most systems engineers don't realize it's in their hands because it's usually framed as a modeling decision. Numbers on a GPU can be stored and computed at different precisions: the old default is 32-bit floating point, four bytes a value; then 16-bit formats (fp16 and bf16), two bytes; then 8-bit formats (fp8 and int8), one byte. Dropping precision does two things at once that a systems engineer should care about deeply. It shrinks every value, so you move half or a quarter as many bytes — which, if you were memory-bound, is a direct win on the exact resource you were short of. And on hardware with dedicated low-precision units, it multiplies the arithmetic throughput — the same tensor cores do far more low-precision operations per second than high-precision ones. Half the bytes and several times the math: precision is the rare lever that pushes on both walls of the roofline at once.
What you pay for it is numerical range. Fewer bits means fewer distinct values and a narrower window between the largest and smallest number you can represent, so low precision overflows and underflows more easily and long accumulations lose digits — a real trade-off with two live sides, not a free upgrade. bf16 exists precisely because it keeps fp32's range (the same exponent) while spending precision within that range (fewer mantissa bits), which turned out to be the right thing to give up for training stability. Push on to fp8 and you buy more speed and memory for more numerical risk, which frameworks manage with loss scaling and keeping a few sensitive operations higher.
| format | bytes / value | bandwidth pressure | throughput | dynamic range |
|---|---|---|---|---|
| fp32 | 4 | 1× | 1× | |
| bf16 · fp16 | 2 | 0.5× | several× | |
| fp8 · int8 | 1 | 0.25× | more |
Where this chapter draws its line: the hardware consequences of precision — bandwidth, throughput, footprint — are yours, and the economics of running everything in the lowest safe precision return in the cost chapters. Whether a given model tolerates fp8 without losing accuracy is a model-side question that belongs to the ML- and LLM-systems atlases. You own the lever; they own how far a particular model lets you pull it.
Why Your Expensive Card Is Twenty Percent Busy
Now the field guide. Everything above turns into two failure modes that account for a startling share of wasted accelerator spend, and they share the property that made Chapter 1's villain so expensive: they don't throw errors, and the one number everyone watches reads healthy while they happen. A starved card and a saturated card look nearly identical on nvidia-smi, because that number, as we established, only reports that a kernel was resident — not whether the card was fed, not whether the kernel was worth launching. This is the same silent-wrongness pattern the ML-systems atlas names: the instrument everyone trusts is pointed at the wrong thing, and there's no exception to catch.
| Failure mode | Trigger | What on-call sees | Why it hides | The fix |
|---|---|---|---|---|
| Input starvation | CPU input pipeline can't produce batches as fast as the card consumes them | Step time much longer than compute time; GPU-Util moderate or sawtoothing; CPU cores pegged | The card is busy between waits, so activity looks non-zero; the idle gaps are short and averaged away | Overlap prep with compute (prefetch, more loader workers); move decode/augment onto the GPU (DALI); pin and stream memory |
| Launch-overhead-bound | Thousands of tiny kernels; chatty per-op dispatch from the host | Low throughput at "high" utilization; one CPU thread hot; tiny gaps between many small kernels | Each kernel really does run on the card, so it registers as busy; the overhead is in the launching, off-card | Fuse ops into bigger kernels; increase batch size; capture the launch sequence (CUDA graphs) to amortize dispatch |
Input starvation is the one from the opening ticket, and it is the most common expensive mistake I see teams make with their first serious training or high-throughput inference workload. The card is fast; the JPEG decoder is not; the augmentation is running single-threaded on a CPU that's also doing five other things; and so the accelerator you're paying data-center prices for spends its life as the world's most expensive waiting room. It hides because the waiting is interleaved with real work — a burst of compute, a gap, a burst, a gap — so the averaged activity number never drops to something alarming, and nobody looks closer because nobody was paged. The fix is almost always cheap relative to the card: give the input pipeline enough parallelism and buffering to stay ahead of the GPU, and, when the CPU genuinely can't keep up, move the decoding and augmentation onto the GPU itself so the work rides the fast side of the bus.
Launch-overhead-bound is subtler and tends to bite the second kind of workload — many small operations rather than a few big ones. Each individual kernel is so small that the fixed cost of the CPU telling the GPU to run it rivals or exceeds the cost of running it, so you have a card that's technically busy and a program that's actually bottlenecked on its own dispatch loop. It's especially common in eager-mode Python, where every operation is a separate round trip to the device. The fixes all amount to the same idea: do more per launch. Fuse many small operations into fewer big ones, make the batches bigger so each kernel is meatier, or capture a whole repeated sequence of launches once and replay it as a unit so you pay the dispatch cost a single time.
The thing both failure modes should teach you is that nvidia-smi cannot tell you which one you're in, or whether you're in either — it wasn't built to. To see a fed card versus a starved one, or a saturated card versus a launch-bound one, you need a profiler that shows the timeline: what ran on the device, what ran on the host, and, crucially, the gaps. A couple of hours reading a real timeline routinely saves a purchase order — the same discipline of measuring useful work honestly that the utilization chapter is built on, and where you stop guessing which room you're standing in.
What the Companies Actually Built
Meta's recommenders, and the machine that was memory all along
The most instructive public example of "measure the bound before you buy the FLOPS" is recommendation, because it's the workload that most looks like it should be compute-hungry and most isn't. Meta's DLRM and the family of ranking models like it are dominated by embedding lookups: for every item to be scored, fetch a set of learned vectors from tables that can run to hundreds of gigabytes, then do a comparatively trivial amount of arithmetic to combine them. The bytes-to-math ratio is lopsided toward bytes, which puts these models firmly on the memory-bound slope of the roofline, and it's why the published engineering around them is a story about memory — how to hold tables too big for one device, how to shard them across many, how to keep the bandwidth up — rather than a story about arithmetic throughput. A team that walked in assuming "it's on GPUs, so buy the GPU with the most FLOPS" would have optimized the wrong axis and been genuinely surprised when the pricier card ran their ranker at the same speed. The lesson worth stealing isn't about recommendation; it's the reflex to ask what is this workload actually bounded by before treating a bigger number on a spec sheet as an answer.
NVIDIA's DALI, and an admission built into a product
You can read an entire chapter of this book in the mere existence of NVIDIA's DALI library, because a hardware vendor doesn't build and maintain a data-loading library unless the data loading is a serious enough problem to be costing it customer performance on its own chips. DALI exists because the CPU-side input pipeline — decode, resize, augment, copy — routinely can't keep a modern GPU fed, so the card the customer paid a fortune for underperforms not because it's slow but because it's starved. What DALI does is move that pipeline onto the GPU and overlap it with training, so the preparation of the next batch happens on the fast side of the bus, in parallel with the compute of the current one. The framework-native tools solve the same problem from the other direction: the tf.data and PyTorch DataLoader prefetch-and-parallelize machinery exists to overlap host-side preparation with device compute. That the entire ecosystem converged on "hide the input pipeline behind the compute" is the strongest possible evidence that input starvation is not an edge case — it's the default failure of a naive setup, and the tooling is downstream of how often it bites.
Which complicates the tidy fix: DALI moves the bottleneck, it doesn't delete it — decoding on the GPU spends the very cycles and bandwidth you were protecting, so how much to offload is itself a roofline question. The pleasant part of carrying the model in your head is that the lens that diagnoses the disease also sizes the cure.
Technologies Worth Knowing
Two tools, each named for its sharp edge rather than its feature list, because the point of a systems chapter is knowing which instrument answers which question.
A GPU-aware input pipeline (NVIDIA DALI, and the framework-native tf.data / DataLoader prefetching as the same class). The job is to keep the card fed by overlapping data preparation with compute and, where the CPU can't keep up, running the preparation on the GPU. The sharp edge, again: it relocates the bottleneck onto the resource you were defending, so it's a lever to size with the roofline, not a switch to flip. Reach for it the moment a profiler shows gaps between compute bursts — which is to say, the moment you've confirmed starvation rather than assumed a slow card.
A profiler that shows the timeline (NVIDIA Nsight Systems and Nsight Compute as the standard). This is the instrument that distinguishes fed from starved and saturated from launch-bound — the thing nvidia-smi fundamentally cannot do, because a single utilization percentage has thrown away the timeline where the answer lives. Nsight Systems shows you the host and device timelines side by side, so the idle gaps and the launch overhead become visible; Nsight Compute goes down to a single kernel's behavior when you need to know whether that kernel is compute- or memory-bound. The sharp edge is cost of use: profilers have overhead, a learning curve, and a way of telling you inconvenient truths about numbers you'd been reporting proudly. That last part is a feature.
The Principal Engineer's Perspective
When this mental model earns its keep — and when it doesn't. The throughput-machine-across-a-slow-bus model pays for itself the instant "buy a faster card" is on the table, which for a GPU is often, because the cards are expensive enough that a wrong purchase is a real number. It matters most for training and high-throughput inference, where the difference between a fed and a starved card is a third of your spend, and for anyone choosing instance shapes, where picking the card with the most FLOPS for a memory-bound serve is a common and costly reflex. It matters least for a small, latency-bound, comfortably-provisioned inference service that already meets its SLO — there, the profiling afternoon is a cost with no prize, and the honest move is to leave it alone. The failure isn't ignoring the model; it's applying it as ceremony to a workload too small to reward the attention.
The observability that has to exist before you can reason at all. You inherited, from a career of CPU work, metrics that roughly meant something; here the headline metric actively misleads, so the tooling has to be deliberate. Before a team can honestly answer "will a bigger card help," three things need to be reachable: a profiler in the workflow, so that "GPU-bound" is a measurement and not a screenshot; input-pipeline metrics — batch wait time, loader throughput, queue depth — so starvation shows up as a number instead of a mystery; and at least a back-of-envelope arithmetic intensity for the main workloads, so compute-bound versus memory-bound is a known fact about each job rather than a debate. If the only instrument you have is a utilization percentage, you are not measuring the card. You are measuring whether it's turned on.
The business decision wearing an engineering costume. "We need a faster GPU" is, most of the time, "our pipeline can't feed the GPU we have" in a better suit — and the whole value of this chapter at principal level is the reflex to check which before signing. The expensive mistakes I've watched weren't teams buying too little; they were teams buying the wrong axis — more FLOPS for a memory-bound ranker, a bigger card for a job bottlenecked on JPEG decode — spending capital to lift a ceiling they were never touching. The technical work is to make the bound legible before the purchase, because a bound named on a slide is a bound someone can argue with, and a bound buried under a green dashboard is a bound that gets solved with money.
The trade-off you actually control. Precision is the lever with the best ratio of upside to effort on this whole list, and the one most often left unpulled because it's filed under "modeling." Running memory-bound work in the lowest precision it tolerates buys bandwidth and capacity for the price of numerical care, and the systems engineer's job is to make sure the option is on the table and the range risk is understood — not to make the accuracy call, which is the model owner's, but to insist the conversation happen before the alternative (a bigger card) is funded.
Questions to take back to your team:
- For our biggest GPU workload, is it compute-bound or memory-bound — and if the answer is "nobody's measured," how are we justifying the instance type we're paying for?
- When we last called a job "GPU-bound," did a profiler say so, or did a utilization number? Would we bet the cost of the next card up on the difference?
- What is our input pipeline's throughput relative to the card's appetite, and would we even see it if the card were starving?
- Which of our workloads are running in higher precision than they need, and whose decision is it to change that — and have they been asked?
- The next time someone requests a faster GPU, what's the one measurement we'll require with the request?
The intuition to carry out of this chapter: a GPU is a throughput machine on the far side of a slow bus, so it is fast only when fed wide parallel work and only as fast as whatever feeds it — which means most "slow GPU" incidents are a fast GPU waiting on a slow everything-else, and most "we need a faster GPU" requests are really "we can't feed the one we have." Before you buy FLOPS, prove you're compute-bound and fed; the card is rarely the thing that was slow.
Exercises
None of these has a clean answer, which is the point. An AI will hand you a confident four-second reply that assumes the profile you never took; the value is in the argument you have with it about the workload you actually run.
Take a real "GPU-bound" job in your shop — one someone has proposed throwing a bigger card at — and, before touching the budget, estimate where its time goes: compute, data transfer, input preparation, launch overhead. You won't have exact numbers without a profiler; the exercise is discovering which of the four you can't even estimate, because that's the one you're not measuring and probably the one that's winning. Then say what a faster card would and wouldn't fix.
Pick a workload and make the honest case that you should downgrade the accelerator — that the job is memory-bound or starved, that the FLOPS you're paying for go untouched, and that the money is better spent on CPU, bandwidth, or a data pipeline. Then make the opposing case for the same workload at ten times the scale. Where does the argument flip, and what number flips it?
Name your expensive stage, the cheap upstream stage that could starve it, and the metric that would reveal the idle time. Now find where the analogy breaks: is there a stage in your system that genuinely is the expensive bottleneck, where the money really is in the right place? What makes that one different — and how would you have known without assuming?
For one workload, estimate its arithmetic intensity — even crudely, as "lots of math per byte" or "barely any" — and place it on the sloped-then-flat roofline. Predict what a card with double the FLOPS and the same bandwidth does for it, then what a card with double the bandwidth and the same FLOPS does. If your prediction for the two is the same, you haven't found the bound yet.
Connections
This chapter is the first leak on the pipe the opening chapter drew. Chapter 1 established the accelerator as a resource you pay for whether or not it does useful work (the accelerator as a resource); here that resource is idle at the microscopic scale — a fed card versus a starved one, one step at a time. It also cashes in Chapter 1's warning about the three layers of utilization: a starved or launch-bound card is exactly the case where device activity reads healthy over near-zero useful work, so this chapter is what utilization theater looks like under a microscope, and why you can't diagnose it from the number everyone reports.
The memory hierarchy previewed here opens fully in Chapter 3, where memory stops being a bandwidth story and becomes a capacity wall — what fits in VRAM bounds what runs at all. The profiler introduced here as a way to see useful work is the tooling premise of Chapter 4, which measures that useful work honestly (MFU/goodput) and defends the number against theater. The host-device boundary generalizes, past one card, into the interconnect: the same "downstream of a slower feeder" logic that governs one GPU and its bus governs many GPUs and the network between them (Chapters 6 and 8), where topology-aware placement decides which cards can feed each other fast. And precision's economics — running everything you can in the lowest safe precision — return in the cost chapters (Chapters 9 and 10).
The throughput-versus-latency architecture is the internet-scale-systems atlas's throughput-versus-latency trade cast in hardware — a GPU is the most committed throughput machine you'll provision, and its weakness at latency-shaped work is that commitment's price. The on-card memory hierarchy is the same latency-versus-memory trade that atlas draws for caching, in a steeper form. And the starved card that throws no error is a hardware cousin of the ML-systems atlas's silent wrongness: the failure that costs the most is the one with no first-order signal, whether it's a wrong answer on a green dashboard or an idle card on a busy one. The model-side questions this chapter deliberately doesn't answer — whether a given model tolerates fp8, how to batch it — belong to the ML- and LLM-systems atlases, which build on the hardware facts established here.
Next: Chapter 3 — Memory Is the Wall: why what fits in VRAM, not how fast the cores are, is the first question that decides whether your job runs at all.