Context EngineeringYou Program the Window, Not the Model

Introduction

Here is the shift this chapter runs on, and it reorganizes how you build for the rest of the atlas: you are not writing a prompt. You are assembling a window, and the window — not the prompt — is the program.

Chapter 1 left the model as a fixed, read-only dependency. You can't change its weights. You can't teach it anything between calls. You get it exactly as the provider ships it.

So the only state your program actually controls is what you put into the context window on each call. That is the whole mutable surface of the system — the one lever with your hand on it.

"Prompt engineering" is the name we gave this back when the window held one thing: the text you typed. It's a bad name now. It makes the job sound like word choice, when the job is deciding what goes into a scarce, priced buffer, in what order, out of far more material than will ever fit.

The reason experienced engineers walk straight into this is that the word "prompt" points at the wrong object. You picture the string you wrote. In production, that string is the smallest and last-arriving component of a window that also holds retrieved documents, tool outputs, and a history that grows every turn — most of it assembled by code you never thought of as "the prompt."

GitHub has described publicly how its own Copilot works: it doesn't send the model the line you're on, it sends an assembled context built from your open files and the code around your cursor. The prompt you wrote is a tenant in that window, not the landlord — and the bugs live in the parts you didn't type.

Here's the ground we'll walk together:

• Why the context window, not the prompt, is the thing you actually program — and why "prompt engineering" is a name that hides ninety percent of the job
• The anatomy of an assembled window: the five components that compete for it, who owns each, and the distinct way each one fails
• The token budget you're really managing — why relevance-per-token, not raw recall, is the number to drive, and why a bigger window doesn't save you
• Why position is an engineering decision: the model reads unevenly, so where you put a fact decides whether it gets used
• The scenarios you'll actually meet in production, and the move that handles each — including why prompts have to be versioned like the code they already are

None of this is exotic. It's inventory management under a hard cap — a discipline you already own. You've just never had to apply it to something that arrived looking like a text box.

The Window Is the Program

Start with the move that does the most work, because everything else is a consequence of it: stop thinking about the prompt as text you write, and start thinking about the window as state you assemble.

The model is frozen. The window is the only thing that changes between one call and the next. So the window is your program's entire mutable state — and the code that builds it, not the sentence you typed, is where the behavior actually lives.

Teams arrive at that truth by failing past two designs first. Both are reasonable. Both are instructive.

The god-prompt. Everything hardcoded into one enormous system message — every rule, every example, every edge case the team ever hit — shipped identical on every request.

It's appealing for real reasons: one file, in version control, utterly predictable, and it demos beautifully because your demo queries look like the examples you baked in.

It breaks because it can't adapt to the request. The same three thousand tokens of instructions ride along whether the user is asking about refunds or resetting a password, so the one rule that governs this request is drowned among ninety-nine that don't. It works right up until a real query wanders off the examples — which production supplies on day one.

The f-string. The natural next step: build the prompt at runtime by concatenating whatever you've got — system text, retrieved docs, history, the user turn — into one big string, and send it.

It's the smallest possible diff from the demo and takes an afternoon. That's the whole appeal.

It breaks because a concatenation has no budget, no ranking, and no order beyond the accident of what your code appended first. When the pieces get big, the string overflows the cap and the tail drops silently. When retrieval returns junk, the junk goes in at exactly the same standing as the signal. It behaves perfectly on your laptop, where every piece is small and every test document is relevant — and falls apart the first time a real conversation runs long.

Both designs share one disease: nothing in the system is responsible for deciding what belongs in the window for this request. Name that responsibility, give it an owner, and you've built a context assembler. The rest of the chapter is what that assembler has to decide.

Anatomy of an Assembled Window

An assembled window has five components. Each has an owner, a share of the budget, and its own way of failing — and naming them turns "the prompt is bad" into a diagnosis you can act on.

• System instructions — the standing rules and the persona, the small stable core the team owns. Modest budget. They fail by being ignored: buried mid-window or truncated off the top when something upstream grows, still technically present and functionally gone. This is the refund-price bug.
• Retrieved facts — the grounding pulled per request: the documents, records, and snippets that make the answer about this situation instead of the average of the training data. This is RAG's output (Chapter 3), owned by the retrieval pipeline. It's usually the largest and most volatile slice of the window, and it fails by returning noise — plausibly-related chunks that crowd out the few that matter.
• Tool results — the outputs of functions or APIs the model asked to call: an account lookup, a search, a calculation (the single-call mechanics are Chapter 5). Owned by the tool layer. They fail by being stale, or by being enormous — a two-thousand-row query dumped in raw is a component that ate the budget and taught the model almost nothing per token.
• Conversation history — everything said so far, the working memory of the interaction, owned by the session or memory layer (Chapter 4). It's the component that grows without bound, one turn at a time, and it fails two ways: by evicting the turn that mattered, or by quietly filling the whole window until nothing new fits.
• The user turn — the actual request, the reason the call exists. The user owns it, it's usually tiny, and it fails in the cruelest way available: it arrives last, so it's first against the wall when the window overflows. The most important token in the buffer routinely has the weakest claim on the space.

Now watch one request get assembled, because the order of operations is the whole point.

The user asks a question. The assembler pulls their account tier and recent tickets, asks the retriever for relevant policy, appends the history, prepends the system rules, and then — the load-bearing step — ranks, orders, and truncates all of it to fit the budget before a single token reaches the model.

By the time the model runs, the quality of its answer is mostly already decided: by what made the cut, and where it landed. The model is the last and least interesting thing that happens to the request.

The Budget Is the Whole Game

The window is a fixed buffer, measured in tokens, and every component competes for it. The budget is fixed. The demands are not, and most of them grow.

Put real numbers on it, because adjectives hide the problem. Say your working budget is 10,000 tokens. (Illustrative — the real ceilings move every quarter and belong in a dated reference, not in your head.)

• System instructions: 800.
• Twenty retrieved policy chunks at ~250 tokens each: 5,000.
• One tool result: 600.
• Ten turns of history at ~300 tokens each: 3,000.
• The user's actual question: 40.

That's 9,440 of 10,000 spent before the model writes a word of the answer — and not one line of it looks wasteful. Every component was reasonable on its own.

Now something has to give, and the only question that matters is whether you decide what gives, or the runtime decides for you.

Left alone, the default is brutal and dumb: the text overflows the cap and the end gets cut. The end is usually the user's question, or the room the answer needed. The failure isn't "we ran out of space" — everything runs out of space eventually. The failure is that running out of space silently deleted the most important thing in the buffer, because it happened to be last, and nobody chose that.

Here's the target that replaces "fit more in": relevance-per-token. Not how much relevant material you can cram in — the density of signal per token you spend.

The instinct carried over from search is recall: fetch everything that might help, you can always ignore the extra. That instinct is wrong here, and expensively so.

In a search results page, a bad candidate gets ranked to the bottom and no human ever sees it. In a context window, a bad chunk gets read — the model consumes every token you send and weighs it. So over-retrieval doesn't just waste budget. It actively degrades the answer, because the three chunks that hold it are now diluted among seventeen that merely mention the topic.

Which is why the reflex fix — get a bigger window — misreads the problem.

A bigger window raises the weight limit on the suitcase. It does not decide what to pack. Give a team more room with no budget discipline and they arrive with more things they didn't need and still forget the charger.

The window has been getting bigger for years, and "just put everything in the context" has been wrong the whole time, for a reason that survives every increase: the buffer is also metered. Every token you add costs money and latency (Chapter 8 prices this properly). An irrelevant chunk bills you twice — once to send it, once in the worse answer it produced. More room to overspend is not a fix for overspending.

Where You Put It Matters

Even once you've chosen what goes in the window, you still have to decide where — and where turns out to be load-bearing, not cosmetic.

Models read unevenly. The finding named lost in the middle (Liu et al., 2023) is blunt about it: a model uses information at the beginning and the end of a long window far more reliably than information buried in the middle, and the gap is wide enough to change answers.

Put a critical instruction or the key retrieved fact in the middle of a long context, and the model often behaves as if it isn't there.

So placement is a control you own. Put the load-bearing things where the model actually looks — the edges — and stop trusting that "it's in the prompt" means "it will be used."

The model reads a long window the way you read a long contract: the opening carefully, the end because that's where the signature is, and the middle not at all. That is the entire mechanism behind the price the assistant quoted. The rule was in the prompt. It just wasn't anywhere the model was looking.

Then there's what happens at the very edge of the budget. When the assembled window is over the cap, something gets evicted — and you choose whether that's a policy or an accident.

The accidental policy is "drop whatever's last," and it's the worst one available, because last is usually the user's question.

The deliberate version is graceful degradation, which you already apply to any overloaded system: when you're over budget, shed the lowest-value context first — the oldest history turn, the twelfth retrieved chunk, the tool output you can summarize — and never the request itself.

You are priority-queuing tokens. It's traffic tiering under load, pointed at a buffer you forgot was a queue. The window was a resource-allocation problem the whole time; the only question is whether you're doing the allocating, or letting a blind truncation do it for you.

Prompts Are Code

There's one more decision, and it's the one that separates a system you can operate from a pile of f-strings you can only pray at.

The god-prompt and the f-string both smear instructions and data into one undifferentiated string. Pull them apart.

Instructions are code. They get templates, version control, and a diff on every change. Data — retrieved documents, tool output, the user's text — is a parameter you inject into named slots in that template.

The immediate payoff is maintainability. You can review a template and diff it across releases. An f-string assembled at runtime has no version, no history, and nothing to test — which is exactly why quality regresses through it without a trace.

A prompt you can't diff is a production dependency with no change history. You have one. You just called it a string so you wouldn't have to test it.

Separating instruction from data also hands you a trade-off worth naming.

A fixed prefix is cacheable, predictable, and testable — and a stable prefix is cheap to reuse at the hardware layer (the KV-cache economics belong to the GPU atlas; Chapter 8 spends them). A freshly assembled per-request window is more relevant, but a moving target: harder to version, harder to test, more expensive to serve.

Most systems want both — a stable, versioned instruction core with a dynamic data body. The fixed part earns the caching; the moving part earns the relevance.

And keeping the seam clean buys you something you'll be grateful for in Chapter 9. The moment you stop distinguishing "instructions I wrote" from "text a document or a user supplied," you've removed the only boundary between your system and prompt injection — untrusted text sitting in the same buffer as your trusted commands, read by the same eager reader. You keep the seam here for maintainability. You get the security boundary for free.

The Scenarios You'll Actually Meet

Here's the field guide — the failures the "it's just a prompt" model actually produces, each with its trigger, how it shows up, and the move that handles it. Every one is quiet, which is the theme of this atlas: nothing throws.

The health signal that catches most of them before a customer does is one almost nobody watches — window occupancy, how full the window runs per request, as a distribution. When p95 occupancy creeps toward the cap, you're one long conversation away from silent truncation on live traffic. It's the "disk at ninety percent" alarm for a disk you didn't know you had.

One discipline sits under all four, and it's the cheapest insurance in context engineering: log the fully assembled window — the exact tokens that went to the model — not the template you think went, and not just the completion.

The single most common context bug is that the template says X and the window didn't, and those two are indistinguishable after the fact unless you kept the real thing. The tools that make this boring — prompt registries, eval harnesses — are worth buying, and worthless bought before the discipline they enforce. The flight recorder is Chapter 10; the habit starts now.

What the Companies Actually Assemble

GitHub Copilot is context engineering wearing a code-completion costume.

GitHub has described, in its own engineering writing, how each suggestion's context gets built — and it's emphatically not "the line you're on." The assembler pulls from the current file, from snippets in neighboring open tabs chosen by similarity to what you're working on, and from the code on both sides of the cursor, then ranks and trims all of it to a fixed token budget before the model runs.

The lesson worth stealing is where the intelligence lives. When Copilot feels uncanny, it's very often the retrieval and ranking that's uncanny, not the generation. The effort went into deciding what to put in the window, because that's the step that sets the ceiling on how good the completion can be. The model is a commodity they rent; the assembler is the product they built.

The support assistant shows the same discipline producing its own bug.

Reuse our house example from Chapter 1. Its assembler injects the customer's account tier and last three tickets ahead of the free-text question — a good instinct, because grounding the answer in who's actually asking is what turns a generic reply into a useful one.

But grounding and budget are in tension. When a release grew the retrieved-policy section, the extra tokens pushed the user's question down and, on long sessions, into the truncation zone. The assistant started answering a question no one asked, fluently.

The instinct wasn't wrong. It was unbudgeted. Adding useful context evicted more useful context — and nobody owned the total.

And sometimes the right answer is less assembly, not more.

For a narrow, high-stakes task with a stable shape — a compliance check, a fixed extraction — a static, heavily-tested prompt often beats a clever per-request window. Precisely because you can version it, eval it, and cache it, and because there's no assembly step to silently regress.

The assembler is a real system, and every real system has a floor below which its own machinery costs more than it returns. Reach for assembly when the right answer depends on something you can't know until the request arrives. Below that line, a constant string isn't primitive — it's operable, and operable is a feature.

The Principal Engineer's View — Owning the Window

Exercises

These reward argument over recall. Paste any into a model and you'll get a tidy four-second answer that skips your budget, your owners, and your truncation policy — the only things that decide the outcome.

Connections

Next: Chapter 3 — RAG: Building It Is the Easy Part: how the "retrieved facts" slot actually gets filled, why a retrieval system inherits the failure modes of both search and generation, and why most "hallucination" is a retrieval miss wearing a fancier name.