Memory Is the Wall

Introduction

The model was 13GB. The card had 24GB. An engineer I trust did the subtraction everyone does, saw 11GB of headroom, and shipped it. It served fine in the demo, fine in the load test, fine for the first two days of real traffic. Then a customer pasted a long document into the prompt — a few thousand tokens where the tests had sent a few hundred — and the card threw CUDA out of memory and took the service down with it. The weights everyone had sized around never moved. What moved was everything the parameter count doesn't tell you about: the memory a model needs while it runs, which grows with things a demo never stresses, and which had quietly been eating the 11GB cushion the whole time. On launch day, at real context lengths, under real concurrency, there was no cushion left.

That subtraction — model size from card size, ship the difference — is the most natural mistake in this atlas, and it comes from importing a CPU instinct into a place it doesn't hold. On a CPU you rarely ask whether something fits: memory is abundant and swappable, and when you run low the operating system pages to disk and you get slow rather than dead — so the question you're trained to ask is how fast. On a GPU that instinct betrays you. Device memory is fixed, unswappable, and small relative to the appetites of what you put in it, and when you run out you don't get slow, you get a hard stop and a stack trace. The first question on an accelerator is not how fast but does it fit, and the whole design bends around the answer.

Here is the inversion the chapter asks you to make. You have spent your career treating memory as capacity you top up and compute as the thing you optimize. On a GPU it's the reverse: compute is usually abundant relative to what you can feed it, and memory is the binding constraint — the wall your design keeps hitting. What fits in VRAM decides whether a job runs at all, before FLOPS decide how fast. And the cruel part is that the number you'd size by — the parameter count — is the smallest and best-behaved occupant of the memory a model needs. The ones that actually blow the wall are the ones that grow: the activations of a forward and backward pass, the key-value cache of a running generation, the optimizer state of a training step. vLLM, the most-deployed open-source LLM server, is built around managing one of those; the ZeRO work behind DeepSpeed and PyTorch's FSDP exists to survive another. Both are this chapter, in production, at the far end of the same wall you'll hit on your first serious model.

Here's the ground we'll walk together:

• Why "does it fit" comes before "how fast" on an accelerator — and why the CPU instinct that made you good at capacity planning quietly misleads you here.
• What is actually in your VRAM: the 4 occupants of device memory, only 1 of which is the weights you sized by, and the arithmetic to predict each before it pages you.
• Why the key-value cache is the modern memory villain — a bill that grows with every token the model reads and every request you serve at once.
• Why training the same model needs roughly 8× the memory of serving it, and why the run dies at hour 30 instead of step 1.
• Why a card reporting gigabytes free will still refuse your allocation — and the allocator behavior that turns free memory into memory you're not allowed to touch.
• The 4 levers that move the wall — shrink, recompute, shard, page — and which one to reach for before the purchase order for a bigger card.

The Problem, Precisely

Compute decides how fast the job runs. Memory decides whether it runs at all — and the second question outranks the first.

Start with the budget, because a GPU's memory is a fixed budget and almost every failure in this chapter is an overdraft. A card has some on-board memory — call it the ceiling — and everything your workload needs at the peak instant of a step has to fit under it simultaneously. Not on average. At the peak. Memory isn't a rate you can smooth out with patience the way you smooth out compute by waiting longer; it's a high-water mark, and if it crosses the ceiling for even one step, the job dies. The discipline of this chapter is predicting the high-water mark before the card discovers it for you at three in the morning.

Now the arithmetic, because it's the chapter in one calculation. Take a 7B-parameter model — a size a lot of teams run on a single high-end card. Its weights, 2 bytes each in half precision, are 14GB: the number the parameter count predicts, and if weights were the whole story you'd serve it on a 16GB card and call it done. But to serve it you also need the key-value cache for every request in flight, which can run to a couple of gigabytes per long conversation and climbs with every token. And to train it you need gradients, optimizer state, and a master copy of the weights, pushing the real footprint to something like 112GB — 8× what the weights suggested, more than a single 80GB card can hold. Same 7B parameters; 3 wildly different memory bills depending on what you ask the card to do. The parameter count told you almost nothing about any of them.

This is why memory, not compute, is where the rest of this atlas's design decisions come from. Memory is why you shard a model across several cards (Chapter 8), because it no longer fits on one; why you quantize (economics in Chapter 10), because halving the bytes per value is the cheapest way to move the wall; why sharing a card between tenants is hard to do safely (Chapter 5), because you can slice compute more easily than you can stop one tenant's memory from crushing another's. Every one is downstream of the fact we're establishing here: on a GPU, memory is the wall, and the wall is closer than the parameter count makes it look.

What Is Actually in Your VRAM

The useful model of GPU memory, for someone who will size instances and debug OOMs but never write an allocator, is a budget with 4 occupants. Learn what each one is and how it grows, and OOMs stop being surprises and become arithmetic you did or didn't do.

The weights are the floor — fixed, predictable, and the smallest thing you should worry about. A model's parameters occupy memory equal to their count times the bytes per value: 4 bytes each in full precision, 2 in half, 1 in the 8-bit formats. 14GB for our 7B-parameter model in half precision, and it doesn't change while the model runs — the same block sits there whether you send 1 request or 1,000. This is the occupant everyone budgets for, because it's the number on the model card, and the one that almost never causes your problem. Sizing a card by its weights is like sizing a restaurant by the floor space of the kitchen: necessary, nowhere near sufficient.

The activations are the working memory of a pass, and they scale with how much you push through at once. Every layer produces intermediate results the next consumes, and the training backward pass needs many of them kept around to compute gradients. Their size scales with batch size times sequence length — the volume of data in flight — so the same model that fits at batch 1 can OOM at batch 32, not because the weights grew but because you asked it to hold 32× as much work in progress. In inference the activations are transient and modest; in training they can be the single largest occupant of the card, which is why one of our levers exists to attack them specifically.

The key-value cache is the occupant that grows with every token, and it is the one that ended the story in the introduction. When a model generates text, it attends back over everything it has already seen, and to avoid recomputing the past at every step it keeps a key and a value vector for each prior token, in each layer. That cache did not exist when the request arrived and grows monotonically as the request runs. The arithmetic is worth carrying: for a model with 32 layers and a hidden size of 4,000, each token costs roughly 0.5MB of cache in half precision. A 4,000-token conversation is therefore about 2GB — and 100 served at once is 200GB, more than 10× the 14GB of weights. The weights were never the problem. The cache that grows with context length and multiplies with concurrency was — and it's invisible until real traffic sends long inputs.

The optimizer state is the training tax, and it's why training the same model costs several times what serving it does. To take a gradient step, the card holds not just the weights but a gradient for every parameter and — for the optimizers everyone actually uses — a running average and variance for each too, kept in full precision for stability, plus a full-precision master copy of the weights under mixed precision. Add it up and it's roughly 16 bytes per parameter, against the 2 you'd guess from half-precision weights. Our 7B-parameter model, 14GB to serve, is about 112GB to train — before a single activation. This is what surprises teams migrating from serving to fine-tuning: nothing about the model changed, but the memory bill went up 8×, and the 80GB card that served 3 copies can't train even 1.

Hold those four occupants in your head — weights fixed, activations scaling with batch, KV cache growing with tokens and concurrency, optimizer state multiplying the whole thing during training — and you can predict every OOM in this chapter before it happens. That is the entire point: to stop guessing about memory, learn what's in the budget and do the subtraction yourself.

The Levers Against the Wall

Once memory is the binding constraint, the interesting question stops being "how much memory does the card have" and becomes "which lever makes the workload fit." There are four, they attack different occupants, and each costs something real — the honest move is knowing which one your situation wants before you reach for a bigger card, the fifth lever and usually the worst-value one.

Shrink the values. Lower precision is the first lever because it hits the largest fixed occupant directly: store weights and activations in 2 bytes instead of 4, or 1 instead of 2, and you halve or quarter their footprint on the exact resource you were short of. This is the same precision lever Chapter 2 handed you as a throughput trick; here it's a capacity trick, and often the difference between fitting and not — a model that overflows a card in full precision routinely fits with room to spare in half. What you pay is numerical range, which is why how far you can drop precision without hurting the model belongs to the ML- and LLM-systems atlases: you own the hardware fact that fewer bytes means more fits; they own how few bytes a given model tolerates.

Recompute instead of storing. The activations are big because you keep them; activation checkpointing (sometimes called gradient checkpointing) bets you can throw most of them away and recompute them during the backward pass when you need them. A step that stored every layer's activations now keeps a handful of checkpoints and regenerates the rest, dropping activation memory by a large factor for roughly a third more compute per step. The trade stated plainly: you buy memory with time. It's worth it when memory is the wall and compute isn't — which, this being a chapter about the wall, is often — and a bad trade when you were already compute-bound with memory to spare.

Shard across cards. When a single card can't hold the model even at its cheapest — a 70B-parameter model is 140GB in half precision, past any single card — you split the model, or its optimizer state, across several cards that hold the pieces collectively. Here memory stops being a single-card story and becomes an interconnect story: the pieces have to talk, and the network between the cards decides whether sharding is feasible or merely possible-and-agonizing (Chapter 8). The fact to carry now is that memory is why you shard — the parallelism strategy is the model atlases' subject; the reason it became necessary is this chapter's.

Page the cache. The KV cache is not just big, it's badly behaved — it grows unpredictably per request, so allocating one contiguous block per request wastes enormous space to internal fragmentation. Paging the cache, the way an operating system pages virtual memory, lets a request's cache live in scattered fixed-size blocks instead of one reserved slab, which reclaims the waste and raises how many requests a card can serve at once. This is vLLM's central idea and it's important enough to get its own section, but as a lever it belongs here: when the KV cache is your wall, better memory management buys you capacity without buying memory.

And the trade underneath all four, the one you'll actually argue about in a planning meeting: batch size. A bigger batch is more throughput — more useful work per unit of fixed overhead — but activations and KV cache both grow with it, so batch size trades memory for throughput most directly. "Reduce the batch until it fits" is the reflex fix for an OOM, and it works, and it isn't free: every notch down to survive the wall gives up throughput you were paying for. The card runs; it just runs at a fraction of the work it could do, and unless someone is measuring useful work — which, per the next chapter, almost nobody is — the throughput you left behind is invisible. Shrinking the batch to stop a crash is the most common way a team quietly pays for a card and uses half of it.

Why Your Card OOMs

Now the field guide. The occupants and levers turn into a handful of failure modes behind most wasted memory and most memory outages, and they share a nasty property: two of the three don't look like memory problems until you know where to look, and one lies to the dashboard you'd check.

KV-cache OOM is the failure from the introduction and the one most likely to bite a team shipping its first serious inference service. It hides better than any memory bug has a right to, because everything looks healthy until it isn't: the weights fit, the demo flies, the load test passes — because the load test, written by someone reasonable, sent short prompts at modest concurrency, exactly the regime where the cache stays small. Then production sends a 4,000-token document to each of 80 simultaneous users, the cache balloons past the headroom, and the card that ran for days falls over in a way that correlates with nothing in your deploys and everything in your traffic. The fix is to stop treating the cache as free: bound the context and concurrency a card will accept, page the cache so it packs tightly, and quantize the weights to give it more room to grow into.

OOM with free memory is the one that makes people question their sanity, because it violates the CPU intuition so directly. You catch the CUDA out of memory, check nvidia-smi, and it cheerfully reports 8GB free — on a request for 2GB. The reason is fragmentation: the caching allocator holds memory in blocks it carved up over the run, and after enough allocations and frees of different sizes — exactly what ragged sequence lengths produce — the free space is real but scattered into holes, none individually big enough for the contiguous block you asked for. The card has the memory; it can't give it to you in one piece. The durable lesson: on a GPU the allocator is a component you operate, not a library detail you ignore, and "free memory" is a claim worth distrusting.

Optimizer-state blowup is less a hiding failure than a counterintuitive one, and it catches the team that reasons from weights. The model served all quarter on a single card; someone goes to fine-tune it and it OOMs before the first step finishes, on the same hardware, and the instinct is that something broke. Nothing broke — training simply needs the 8× budget we did the arithmetic for, and the 80GB card that held a 14GB model for serving cannot hold a 112GB training footprint. The fix is the shard-and-recompute toolkit: split the optimizer state across cards, checkpoint the activations, drop the batch — and only when all of that is exhausted, buy more silicon.

The thread through all three is the one this whole chapter is pulling: the number you'd reach for — the model's size, the card's free memory — is the number that misleads you, and the honest signal lives one level down, in the budget you have to compute and the allocator you have to interrogate. Which is the same discipline the utilization chapter is built on, and the reason the next chapter exists.

What the Companies Actually Built

vLLM, and treating the cache like an operating system

The clearest evidence that the KV cache is the memory problem in modern inference is that the most widely adopted open-source LLM server is, at heart, a memory manager for it. The vLLM team found that the naive approach — reserve one contiguous chunk of cache per request, sized for the longest output it might produce — wasted most of the card's cache memory to over-reservation and to holes between requests too small to reuse. Their answer, PagedAttention (Kwon et al., SOSP 2023), borrowed the oldest trick in operating systems: page the cache into fixed-size blocks, let a request's cache live in however many scattered blocks it needs, and keep a table mapping the logical sequence to the physical blocks. Concurrency per card rose sharply — not by adding memory but by stopping the waste of the memory already there. The lesson worth stealing is not "use vLLM," though you probably should; it's that when a resource is the binding constraint, how you manage it is often a bigger lever than how much you buy — and the operating-systems playbook you already know transfers onto the accelerator more often than it looks.

The complication worth stating, because this chapter distrusts tidy fixes: paging the cache makes it pack tightly, not smaller. A well-paged cache still grows with every token and multiplies with concurrency — vLLM raised the ceiling on how many requests fit under the wall, but the wall is still there, and a long enough context at high enough concurrency still hits it. Good management buys you a factor; it doesn't repeal the arithmetic.

ZeRO, FSDP, and admitting the optimizer is the problem

On the training side, the equivalent admission is the ZeRO work out of Microsoft (Rajbhandari et al., 2020) and its PyTorch descendant, FSDP. The premise is the arithmetic we did: for a large model the weights are a minority of the training footprint, and the majority — gradients, optimizer moments, the master copy — is redundantly replicated on every card in standard data parallelism, so an 8-card cluster holds 8 identical copies of the expensive part. ZeRO stops replicating it: shard the optimizer state, then gradients, then parameters across the cards, so each holds only its slice and cluster memory scales with card count instead of being capped by the smallest card. That's what lets teams train models whose footprint dwarfs any single card — not a bigger card, but a refusal to store the same 100GB 8 times. The engineering that made large-model training feasible is, underneath, a memory-deduplication trick — which tells you exactly where the wall was.

The boundary this chapter holds: how you shard — the data, tensor, and pipeline parallelism strategies, the way the model is actually split — is the ML- and LLM-systems atlases' territory, and the interconnect that makes sharded training survive the constant chatter between cards is Chapter 8. What belongs here is the reason any of it exists: optimizer state is the dominant training occupant, replicating it is the waste, and memory is why the whole apparatus of sharded training had to be invented.

Technologies Worth Knowing

Two tools, each named for its sharp edge rather than its feature list, because the point is knowing which instrument addresses which occupant of the budget.

A paged KV-cache server (vLLM and PagedAttention, now echoed by TensorRT-LLM and others). The job is to serve more concurrent requests per card by managing the KV cache as paged blocks instead of contiguous per-request reservations, reclaiming the space naive serving wastes. Reach for it the moment KV-cache memory — long context, high concurrency — is your constraint, which for LLM inference is most of the time. The sharp edge, restated: it manages the cache, it doesn't shrink it, so it raises the ceiling without repealing the growth, and a workload with pathological context lengths still finds the wall behind it. A memory-management win, not a memory-creation one — sizing still matters.

The framework's caching allocator (PyTorch's CUDA caching allocator and its configuration). Most engineers don't know they're using it until it OOMs them with free memory on the card. It sits between your code and the driver's raw allocation calls, caching freed blocks to avoid synchronizing with the device on every allocation — a throughput optimization that is also the source of fragmentation, because those cached blocks are what fragment. The sharp edge is twofold: it hides real usage behind a split between memory reserved (what the allocator grabbed) and allocated (what your tensors use), so the profiler numbers need reading with care; and its defaults are tuned for general throughput, not your pattern, which is why one configuration change — expandable segments, or bounding how it splits blocks — can fix an OOM that looked like a hard capacity limit. Reading reserved-versus-allocated and configuring this thing is the difference between "the card is full" and "the card is fragmented" — different problems with different fixes.

The Principal Engineer's Perspective

Exercises

None of these has a clean answer, which is the point. An AI will hand you a confident four-second reply that sizes by the parameter count and never mentions the occupant that's actually going to page you; the value is in the argument about the workload you actually run.

Connections

Next: Chapter 4 — The Utilization Problem: why the card that fit the model and reads 100% busy is still doing far less useful work than you're paying for, and how to measure the difference honestly.