Context Compression

The common assumption is that context compression is about fitting more content into a model's token limit. While this is one benefit, the more significant effect is on attention allocation. Transformer-based models distribute attention across all tokens in the context window. Redundant content — the same point restated in different words, filler phrases, repeated emphasis — dilutes the model's attention across information it has already processed rather than concentrating it on unique, decision-relevant content.

A 2,000-word document that repeats its core message in slightly different phrasings across every section gives the model abundant signal about tone and general intent but limited signal about specific requirements, edge cases, and constraints. Compressing this to 600 words that state each point exactly once forces the model to engage with the specifics rather than the repetitions. The result is typically more precise, more actionable outputs — not because the model received different information, but because the information it received was less noisy.

This principle applies regardless of context window size. Even models with very large context windows — hundreds of thousands of tokens — produce better outputs from compressed, high-density inputs than from verbose, repetitive ones. The bottleneck is attention quality, not attention capacity.

The highest-impact technique is deduplication, as illustrated in this prompt teardown: identify and remove repeated statements, restated requirements, and reiterated principles. In meeting transcripts, the same point is often made by multiple speakers or rephrased for emphasis across the conversation — the compressed version states it once, in its most precise formulation. In project documents, principles declared in the introduction are frequently restated in every section — keep the most specific instance and remove the rest.

The second technique is abstraction: replace detailed examples with the pattern they illustrate, unless the specific details are relevant to the task at hand. "We've had issues with three customers — Acme lost data, Beta had slow load times, and Gamma had login failures" compresses to "recurring reliability issues: data loss, performance, authentication" when the specific customer names are not needed for the analysis. If the customer names matter, keep them; if they are just instances of a pattern, the pattern is sufficient.

A third technique is scope filtering: remove sections that are irrelevant to the specific question being asked. If you are asking the model to analyse pricing strategy, the sections about engineering architecture can be removed from the input document entirely. This is where <a href="/aisapedia/conversation-chunking">conversation chunking</a> intersects with compression — each conversation chunk should contain only the context relevant to its specific task, not the full project history.

A fourth technique, often overlooked, is format compression: converting prose into structured output formats like bullet points, tables, or key-value pairs. "The project started in January, involves three teams — engineering, design, and marketing — has a budget of $500K, and must ship before Q3" compresses to a four-line structured summary that the model can parse faster and reference more precisely. Structured formats also reduce the risk of the model misinterpreting ambiguous prose.

Compression fails when it removes context that the model needs to understand constraints, exceptions, or nuances that would change the output. The most dangerous removals are implicit constraints — requirements that are "obvious" to the human compressor but that the model cannot infer from the remaining text. Removing the sentence "this must work on devices with limited bandwidth" because it seems minor could lead to an output that assumes fast network access throughout.

Another failure mode is removing the reasoning behind a decision while keeping the decision itself. "We chose React because our team has deep expertise and our hiring pipeline is React-focused" compresses to "we use React" — but the reasoning might be essential if the model is asked to evaluate alternatives, plan a migration, or advise on a new project. The compressed version tells the model what was decided but not why, which limits its ability to reason about the decision in context.

A practical test: after compressing, read the compressed version as if you had no prior knowledge of the project and ask whether you could understand the key constraints, decisions, and their rationale. If significant context is only available in your head and not in the compressed text, the compression has gone too far. The model has no access to your mental context — it works only with what is in the prompt.

For one-off tasks, compression can be aggressive and context-specific: strip everything that is not directly relevant to the immediate question. The compressed context only needs to serve this single interaction, so there is no cost to removing background information that might be relevant to different questions about the same material.

For recurring workflows — where the same base context is reused across many interactions over time — compression must be more conservative. A project context document that will be referenced for weeks of AI-assisted work needs to preserve the reasoning behind decisions, the constraints that might become relevant in future questions, and the background that different team members might need for different tasks. Over-compressing a shared context document optimises for one person's immediate needs at the expense of everyone else's future needs.

A practical pattern for recurring contexts is layered compression: a highly compressed core summary that covers the essentials for any interaction, paired with expandable sections that provide deeper context for specific topic areas. The core summary goes into every prompt; the relevant expanded section gets added only when the task requires it. This balances token efficiency with comprehensive coverage across diverse tasks.

When multiple team members work with the same base context, establish a shared compressed version that is reviewed collectively rather than letting each person create their own. Individual compressions reflect individual assumptions about what matters, and those assumptions may not hold for colleagues working on different aspects of the same project. A collectively reviewed compression ensures that no critical constraint is silently dropped because one person considered it unimportant.