Context Rot and the Context Budget
A larger context window is not a license to fill it. As tokens pile up, the model starts to miss relevant information — context rot — so accuracy tends to fall as the input grows longer. Instead of writing ever-longer prompts, teams now treat the tokens that enter each call as a budget to be managed. This is context engineering, and it has become the standard: keep the system prompt under 2,000 tokens and allocate the remaining budget across tool definitions and retrieved results.
The Four Techniques: write, select, compress, isolate
Four techniques keep you inside the context budget. Write pushes state out to a scratchpad or memory so the window stays clear. Select retrieves only what each turn actually needs. Compress summarizes long history to reclaim tokens. Isolate splits context into subagents so they cannot pollute one another. Enforcing these four rules about what goes in and what stays out is far more reproducible than polishing prompt wording.
Context as an Evolving Document: ACE
ACE (Agentic Context Engineering), submitted to ICLR 2026, treats context not as a fixed prompt but as a document that keeps evolving. By accumulating and updating strategies drawn from execution experience, the approach reported a +10.6% gain on a coding benchmark and +8.6% on financial reasoning. Where prompt engineering was a one-time act of writing something well, context engineering is an operational discipline of continuously deciding what to keep and what to discard.
A Full Guide: From Planning to Operations
In planning, pin the context budget to numbers. For example, cap total input at 8,000 tokens per call, the system prompt at 2,000 tokens, retrieval at the top 5 results, and conversation-history summaries at 800 tokens. Set target metrics — zero budget-overrun violations, a task pass rate of at least 90%, and p95 latency under 4 seconds — and confirm at every deploy that these hold without regression.
The signature failure mode is context contamination. When select pulls in irrelevant documents, accuracy drops; when compress over-summarizes, key facts disappear. Make the recovery branches explicit. If retrieval confidence falls below threshold, retry; if a summary drops a required item, fall back to a safe compression that restores part of the source. If two retries still cannot produce an answer within budget, stop the automatic run and hand off to a human for review.
The operations checklist rests on standard logging. For each call, record input token count, usage against budget, selected document IDs, tokens before and after compression, and whether a subagent isolation occurred. Apply PII masking so logs never carry personal data, and block masking violations at the deploy gate. Tag calls that exceeded budget separately so their causes can be traced.
The continuous-improvement loop runs weekly: which documents were frequently selected yet never contributed to the answer, and which summaries lost facts? Drop low-contribution sources from select, and lower compression strength where fact loss recurs. Context composition is not a fixed constant but something you keep tuning against pass rate and violation counts.
The Bottom Line
In short, context is not something you write long — it is something you manage as a budget. Discipline what goes in and out with write, select, compress, and isolate; keep the system prompt under 2,000 tokens; and, as ACE shows, treating context as an evolving document can yield gains like +10.6% on coding and +8.6% on financial reasoning. Guarding against regression with metrics like zero budget violations and a 90% pass rate is the discipline that replaced prompt engineering.