Context Engineering for AI Agents: Managing the Context Window in Production

An agent running through a long task is fine for the first twenty turns. Then, with no warning and no error, it starts making mistakes it wasn't making before. It contradicts a decision it made five tool calls ago. It retries an approach you already ruled out together. It forgets a constraint you stated at the start of the session. Nothing crashed. The token counter, if your framework shows you one, still reads comfortably under the limit.

That gap, between "the context still fits" and "the model can still use it," is context rot, and it's the reason context engineering for AI agents has become the skill that decides whether a long-running agent actually works. Prompt engineering asks what to put in one message. Context engineering asks what the model should be looking at during turn thirty of a session it started twenty minutes ago, and what should have already been cleared out of the agent context window.

Context rot: why "it still fits" isn't the same as "the model can use it"

Context rot is a measured property of how these models behave as the context window fills, not a bug specific to one framework. Output quality degrades as more tokens accumulate, even while the total stays under whatever ceiling the model card advertises. The model still has room to read more. It's the reasoning over what's already there that gets less reliable.

This matters because it looks nothing like the failure most teams build monitoring for. Running out of context throws an error you can catch: a 400, a truncation warning, a hard stop. Context rot throws nothing. The agent keeps answering, keeps calling tools, keeps producing text that parses fine. It just gets quietly worse at using what you gave it. A team watching only for hard failures misses this every time, because the dashboard says the request succeeded.

Context engineering for AI agents, not prompt engineering

Prompt engineering treated the problem as a single artifact: find the phrasing, the examples, the system message that gets the best answer out of one call. The debate over prompt engineering vs context engineering isn't much of a debate once an agent runs for an hour instead of one exchange. The prompt is still there, but it stopped being the lever that matters most.

An agent making forty tool calls over an hour isn't one prompt. It's a session, and the job is managing everything the model sees across that session: which tool results stay, which get summarized, which get dropped, what earlier decisions need to survive to turn sixty, and what should never have entered the window in the first place.

Anthropic has published its own engineering guidance on this shift, and the practical upshot holds regardless of which model you're running against. You are not optimizing a prompt anymore. You are managing an information ecosystem that changes on every turn, and the tools for that job look more like memory management than copywriting.

The pairwise attention intuition

The mechanism usually cited for context rot is the transformer's attention itself. Every token attends to every other token in the window, so the number of pairwise relationships the model is juggling doesn't grow with the token count. It grows with the square of it. At 10,000 tokens, that's on the order of 100 million pairwise token relationships. At 100,000 tokens, on the order of 10 billion.

Take that as intuition, not a precise cost model. Nobody has published a formula that maps pairwise relationship count directly to a measured drop in output quality, and the real behavior of attention, which tokens actually get weight and how much gets diluted by irrelevant ones, is more complicated than the raw count suggests. What the numbers do explain is the shape of the problem. Doubling the context doesn't double the burden of relating everything in it to everything else. It roughly quadruples it. A long, cluttered window isn't just more information. It's a much larger number of things competing for the same attention budget, most of which are irrelevant to the next token the model needs to produce.

Four techniques come up whenever people talk about context window management for a long-running agent: compaction, structured note-taking, tool result clearing, and sub-agent architectures. They solve overlapping problems, and most production agents end up using at least two of them together.

Compaction: summarize, reseed, and treat detail as recoverable

Compaction, sometimes shorthanded as context compaction, is the most direct fix for a window that's getting crowded. When the conversation approaches some threshold, you stop appending to it. Instead, you summarize everything so far, discard the raw messages, and start a fresh window seeded with only that summary. The agent keeps working, but now it's working from a page of notes instead of the full transcript.

The trigger is usually a fraction of the model's stated context limit rather than the limit itself, because you want headroom for the summarization call and the next few turns before you're back at the ceiling. Somewhere around three-quarters full is a reasonable starting point to tune from.

interface Message {
  role: "user" | "assistant";
  content: string;
}

interface ConversationState {
  messages: Message[];
  tokenCount: number;
}

const MAX_CONTEXT_TOKENS = 200_000;
const COMPACTION_THRESHOLD = 0.75;

async function maybeCompact(state: ConversationState): Promise<ConversationState> {
  if (state.tokenCount < MAX_CONTEXT_TOKENS * COMPACTION_THRESHOLD) {
    return state;
  }

  const summary = await summarizeConversation(state.messages);
  const seed: Message = {
    role: "user",
    content: `Summary of the conversation so far. Continue from this point.\n\n${summary}`,
  };

  return { messages: [seed], tokenCount: countTokens(seed.content) };
}

async function summarizeConversation(messages: Message[]): Promise<string> {
  const response = await client.messages.create({
    model: "claude-opus-5",
    max_tokens: 2000,
    messages: [
      ...messages,
      {
        role: "user",
        content:
          "Summarize this conversation so it can seed a fresh context window. " +
          "Keep every open task, every decision made, every file path touched, " +
          "and every constraint the user gave. Drop tool output you already acted on.",
      },
    ],
  });

  const block = response.content[0];
  return block.type === "text" ? block.text : "";
}

That last instruction in the summarization prompt, drop tool output you already acted on, is doing real work. Compaction feels riskier than it is, because summarizing a 40-page file read down to two sentences looks like real data loss. It is, until you remember where that file still lives: on disk, or wherever the agent read it from in the first place. Compaction is reversible whenever the discarded information still exists in the environment. If the agent later needs that file again, it reads it again. What you're really compacting is the transcript of the agent having read it once, not the file itself. The cases where compaction actually costs you something are the ones where the source is gone: a piece of reasoning the agent worked out on its own, or a decision made from context that no longer exists anywhere else.

Structured note-taking: give the agent a memory it has to open

Compaction summarizes what happened. Note-taking is the agent choosing, in the moment, what's worth keeping before the summarizer ever runs. You give it a tool to write to an external file, ideally one it's instructed to open at the start of every task or immediately after a compaction event. The notes survive independently of whatever happens to the conversation history.

import { promises as fs } from "node:fs";

const NOTES_PATH = "./.agent/notes.md";

const takeNoteTool = {
  name: "take_note",
  description:
    "Append a durable note to the agent's scratchpad. Use this for decisions, " +
    "open questions, and facts you will need later, especially after this " +
    "conversation gets compacted or a sub-agent returns. Re-read the scratchpad " +
    "at the start of every task.",
  input_schema: {
    type: "object",
    properties: {
      note: { type: "string", description: "The note to record." },
      section: {
        type: "string",
        description: "Heading to file it under, e.g. 'decisions' or 'open questions'.",
      },
    },
    required: ["note", "section"],
  },
};

async function handleTakeNote(input: { note: string; section: string }): Promise<string> {
  const existing = await fs.readFile(NOTES_PATH, "utf-8").catch(() => "");
  const heading = `## ${input.section}`;

  const updated = existing.includes(heading)
    ? existing.replace(heading, `${heading}\n- ${input.note}`)
    : `${existing}\n\n${heading}\n- ${input.note}`;

  await fs.writeFile(NOTES_PATH, `${updated.trim()}\n`);
  return `Noted under "${input.section}".`;
}

The tool is the easy half. The system prompt instruction to actually open the file back up is the half that decides whether this does anything. A note-taking tool nobody re-reads is just a slower way to lose the same information, since the agent still has to hold the fact in its head until it happens to write it down, and then it never looks at the file again. Wire the re-read into the workflow itself: after every compaction, after every sub-agent return, at the start of every new task, load the notes file back into context before doing anything else.

Clearing tool results before they become dead weight

Every tool call returns something, and a lot of what comes back is large and short-lived in its usefulness. A search that returns twenty results when the agent only needed the top three. A file read where one function mattered. A database query with four hundred rows the agent already summarized into a paragraph three turns ago. None of that needs to sit in the window for the rest of the session.

The pattern is to let the agent use a tool result immediately, then replace it in the transcript with a short marker: the tool was called, here's what it returned in one line, the full output is gone. If your framework doesn't do this automatically, you can implement it yourself by walking the message history before each turn and truncating tool results older than some number of turns, keeping only the ones from the most recent exchange. This is a smaller, more surgical version of compaction. Instead of summarizing the whole conversation, you're summarizing the parts of it that were always going to be disposable.

Sub-agent delegation and context folding

Compaction and note-taking manage the window you have. Sub-agent delegation avoids filling it in the first place. Instead of a single agent doing a large subtask inline and accumulating every intermediate tool call and every failed attempt into its own transcript, it spins up a sub-agent to handle that subtask in a separate context window. The sub-agent does its exploring, its retries, its scratch work, none of which the parent needs to see. When it's done, it returns a short result, and the parent's context grows by that result alone.

This pattern has a name: context folding. The agent branches to handle a subtask, then folds the branch when done, collapsing everything that happened inside it down to a short summary of the outcome. The parent's context window never has to hold the folded steps at all, because they never entered it.

The trade-off is worth stating plainly before you reach for this everywhere. Every sub-agent call is an extra round trip to the model, with its own latency and its own cost. And the handoff back to the parent is a compression step, exactly like a compaction summary, which means it can lose nuance the parent would have wanted. A sub-agent that spent thirty tool calls ruling out four approaches and settling on a fifth might report only "used approach five, it worked." If the parent later needs to know why the other four failed, that reasoning is gone unless the sub-agent wrote it down somewhere durable, which is the note-taking pattern again, one level down.

What to measure when you do context engineering for AI agents in production

None of the techniques above are free, and none of them are obviously correct at whatever threshold you pick on the first try. Three things are worth instrumenting before you trust any of this in production.

Where in a session quality actually drops. Run your agent against tasks with a known correct answer and grade the output at intervals, turn ten, turn twenty, turn thirty, rather than only at the end. If the grade holds steady until some turn count and then falls off, that number is your compaction threshold, not a number pulled from the model's advertised context limit. Different tasks fill the window at different rates, so this is worth measuring per task type rather than once for the whole system.

How many turns pass before compaction fires. If compaction is triggering every three or four turns, either your threshold is too aggressive or your tool results are bloating the window faster than they need to, which points back at tool result clearing rather than more frequent compaction. If it never fires on tasks that clearly should have triggered it, check that your token counting matches what the model actually charges you for. A mismatch there is a common, quiet bug.

What a compaction costs against what it saves. The summarization call itself burns tokens and adds latency, and it's easy to lose sight of that cost because it happens in the background. Log the token count of the conversation right before compaction and right after. If a compaction is only buying back a small fraction of the window, the threshold is too low and you're paying for summarization calls more often than the window actually needs them.

The trade-offs nobody puts in the docs

Every technique in this piece trades something away. Compaction trades detail for headroom, and the summary is only as good as the prompt that produced it. A poorly instructed summarizer will happily drop the one constraint that mattered. Sub-agent delegation trades extra model calls and a lossy handoff for a parent window that stays clean, and that trade gets worse the more sub-agents you nest, since each layer adds its own compression step. Note-taking trades nothing structurally, but it only pays off if the re-read is actually wired into the workflow, and it's the easiest of the four to build and then quietly never use.

None of this is theoretical scale. Gartner recorded a 1,445% rise in inquiries about multi-agent systems between the first quarter of 2024 and the second quarter of 2025, which is a lot of teams reaching for sub-agent delegation, compaction, or both, often for the first time, on systems that are about to run a lot longer than a single prompt ever did.

If you're deciding where to start, start with measurement, not the four techniques. Grade a real session at intervals before you build any of this, find the turn where quality actually drops, and only then pick between summarizing the window, delegating the subtask, or clearing tool results you already used. Building compaction against a threshold you guessed is how you end up with an agent that summarizes conversations that were never in trouble.