A practitioner's guide to context engineering: what it is, why it matters, and how to do it without drowning your model.

Everyone who has built with these models has watched the same thing happen. The agent is sharp at turn three. By turn thirty it repeats itself, ignores an instruction you gave it four steps ago, and gets confidently wrong about something it had right earlier. The reflex is to blame the model, or to reach for a bigger one. Most of the time the model is fine. The problem is what you put in front of it.

That problem finally has a name that sticks. Anthropic calls it context engineering and frames it as the natural progression of prompt engineering. Prompt engineering was about writing the right instruction. Context engineering is about managing everything the model sees at the moment it runs: the system prompt, the tool definitions, the retrieved documents, the message history, the tool outputs, the note it wrote itself six steps ago. Their one-line version is the one worth taping to your monitor. Find the smallest set of high-signal tokens that maximize the odds of the outcome you want. Everything below serves that sentence.

Here is why it matters, and why it does not go away when the next model ships with a window twice the size.

Context is finite, even when the window is enormous

The models will happily accept a million tokens. Accepting is not the same as using. Anthropic's framing is that a model works from an "attention budget," and every token you add spends some of it. The reason is baked into the architecture. In a transformer, every token can attend to every other token, so the number of relationships the model has to hold grows with the square of the length. Stretch the context far enough and attention spreads thin. Anthropic is blunt that this shows up in every model, some degrading more gently than others, and that context has to be treated as a resource with diminishing returns rather than a bucket you fill to the brim.

The cleanest evidence is Chroma's "Context Rot" study from July 2025. They ran 18 current models, including GPT-4.1, Claude 4, Gemini 2.5 and Qwen3, and held task difficulty constant while varying only how much text they buried the task in. Every single model got less reliable as the input grew, on jobs as trivial as repeating a list of words back. One result should change how you build. On a long-term-memory benchmark, the same question answered from a focused context of roughly 300 tokens beat the same question answered from the full conversation of roughly 113,000 tokens, across every model family. The answer sat in the context both times. Adding 112,000 tokens of true but irrelevant material was enough to wreck it.

There is an older finding underneath this one, and it still carries weight. The 2023 "Lost in the Middle" paper by Nelson Liu and colleagues showed that models use the beginning and the end of a long context far better than the middle. Put the decisive fact halfway down a long document and the model is most likely to skim past it. The curve is U-shaped, and it has held up as windows have grown. OpenAI's own GPT-4.1 guide works around it directly, and so does Google's Gemini documentation, which I will get to.

So the marketing gives you a million tokens. The research gives you a smaller usable window with a soft spot in the middle. The gap between those two numbers is where agents quietly fail. A bigger context window is a bigger inbox, not a better memory.

Waiting for that gap to close on its own is the comfortable move. Anthropic's own take is that context windows of every size will stay subject to pollution and relevance decay for the foreseeable future, at least when you want the strongest performance. Which means the work does not disappear as the models improve. The work is curation, and it is on you.

The practices (mostly the discipline to put less in):

None of what follows is exotic. It is the discipline to include less, and to be deliberate about the little you keep.

Start smaller than feels safe. The instinct is to load everything the model might conceivably need, just in case. That instinct is the disease. Begin with the leanest prompt that could work, run it on the best model you have, and add instructions only where you watch it fail. Anthropic's guidance on the system prompt is to write at the "right altitude," which is the zone between two failure modes everyone recognizes. Too low and you hardcode brittle if-this-then-that logic that snaps the moment reality varies. Too high and you hand the model vague inspirational guidance and assume it shares context it does not have. Aim for specific enough to steer behavior, loose enough to let the model use its judgment. Minimal does not mean short. It means nothing in there that is not earning its place.

Curate the tools as carefully as the prompt. Every tool you attach is context the model has to read and reason about before it does anything. The common failure, in Anthropic's words and in my experience watching teams build, is the bloated toolset that overlaps with itself and creates ambiguous decisions about which tool to reach for. Their test is a good one. If a human engineer cannot say for certain which tool applies in a given situation, the agent has no chance. Give it a minimal set with clear, non-overlapping jobs. On the mechanics, OpenAI's GPT-4.1 guide is specific. Pass tools through the API's dedicated tools field rather than pasting descriptions into the prompt by hand, because the model was trained on the former and does measurably worse with the latter. Same principle on examples. Anthropic still recommends few-shot prompting, but a few clean canonical examples beat a laundry list of edge cases every time. Curate the pictures. Do not stuff the album.

Pull context in, do not preload it. The default pattern for a couple of years was to retrieve everything relevant up front and dump it into the window before the model starts. Anthropic describes the shift toward a "just in time" approach instead, where the agent holds lightweight references like file paths, queries and links, and loads the actual content only when it needs it. Their own Claude Code does this. It reads a large database by writing a targeted query and using commands like head and tail to look at slices, never loading the whole thing. This mirrors how people actually work. You do not memorize the filing cabinet. You remember where it is and open the drawer when you need it. The honest tradeoff is speed, since fetching at runtime is slower than having it ready, so the strongest setups are often hybrid. Claude Code drops a CLAUDE.md into context up front for the stable, always-relevant stuff, and fetches the rest as it goes. Preload what is small and certain. Fetch the rest on demand.

Mind where things sit. Placement is a free lever most people ignore. Because of the lost-in-the-middle effect, the top and bottom of a long context are prime real estate and the middle is where information goes to be forgotten. OpenAI's GPT-4.1 guide says it plainly. If you have long context in the prompt, put your instructions at both the beginning and the end, which they found beats putting them in either spot alone. Google's Gemini documentation gives the same shape of advice from the other direction, telling you to place your actual query after the documents rather than before. If you can only say the important thing once, say it up top. If you can say it twice, bookend the context with it.

For long jobs, give the model a door out of its own window. Some work does not fit in any window. A codebase migration, a multi-hour research task, an agent that runs for a day. Anthropic lays out three techniques, and the right one depends on the job. The first is compaction. When the conversation approaches the limit, summarize it and start a fresh window from the summary. In Claude Code this means keeping the architectural decisions, the open bugs and the current implementation details while throwing away the redundant tool outputs, then carrying on with the compressed version plus the handful of files most recently touched. The second is structured note-taking, which is just letting the agent write to a file outside the window and read it back later. Their example is the agent playing Pokémon on a livestream, keeping a running tally across thousands of steps, drawing its own maps, remembering what worked in a fight, then picking up after a context reset by reading its own notes. Anthropic now ships a memory tool on its developer platform for exactly this. The third is sub-agents. Instead of one agent holding everything, a lead agent delegates focused work to specialists with clean windows. A sub-agent might burn tens of thousands of tokens exploring, then hand back a distilled summary of a thousand or two. The detail stays isolated where it was generated. The lead only sees the conclusion.

Compaction suits work that needs a long back-and-forth to stay coherent. Note-taking suits iterative work with clear milestones. Sub-agents suit research and analysis where parallel digging pays off. All three exist for one reason. To keep the working context small while the total work stays large.

Sources

Anthropic, Applied AI team (Prithvi Rajasekaran, Ethan Dixon, Carly Ryan, Jeremy Hadfield, et al.), "Effective context engineering for AI agents," Sep 29, 2025. https://www.anthropic.com/engineering/effective-context-engineering-for-ai-agents

Anthropic, "Building effective agents," Dec 2024. https://www.anthropic.com/engineering/building-effective-agents

Anthropic, "How we built our multi-agent research system." https://www.anthropic.com/engineering/multi-agent-research-system

Anthropic, "Context management on the Claude Developer Platform" (memory and context editing tools). https://www.anthropic.com/news/context-management

Kelly Hong, Anton Troynikov, Jeff Huber (Chroma), "Context Rot: How Increasing Input Tokens Impacts LLM Performance," Jul 14, 2025. https://research.trychroma.com/context-rot

Nelson F. Liu et al., "Lost in the Middle: How Language Models Use Long Contexts," 2023, arXiv:2307.03172. https://arxiv.org/abs/2307.03172

OpenAI, "GPT-4.1 Prompting Guide," OpenAI Cookbook. https://cookbook.openai.com/examples/gpt4-1_prompting_guide

Google, "Long context," Gemini API documentation. https://ai.google.dev/gemini-api/docs/long-context

‍