Conversation Chunking

Transformer-based models process the entire conversation history in their context window for every response. As the conversation grows, early messages move further from the generation point in the token sequence. While the content is technically still in the window, the model's attention mechanism allocates progressively less weight to distant context, effectively "forgetting" early instructions, constraints, and decisions — even though they remain visible in the conversation.

In practice, this manifests as contradictions: the model makes recommendations at message 30 that violate constraints established at message 5. Or it re-asks questions that were already answered. Or it loses the thread of a multi-part task, producing outputs that no longer align with the project's agreed direction. These failures increase gradually rather than occurring at a specific breakpoint, making them hard to pinpoint — the conversation does not break at a discrete moment, it slowly drifts.

The degradation is content-dependent: conversations with highly repetitive content degrade faster because the repeated tokens dominate attention at the expense of unique constraints. Conversations with diverse, non-overlapping content hold up better but still eventually suffer from the attention dilution effect. As a general pattern, most users notice quality degradation somewhere between 15 and 30 exchanges, though this varies by model and conversation complexity.

The best moments to chunk are at natural decision boundaries: when a set of conclusions has been established and the work is transitioning to a new phase. Architecture decisions locked in one session, implementation details in the next, testing and validation in a third. Each chunk starts with a clear statement of what was decided and why, then focuses on its specific phase without the accumulated history of previous phases.

Warning signs that a conversation needs chunking include: the model contradicts something it said earlier, it forgets a constraint you established, it re-generates content that was already finalised, or its outputs become noticeably less specific, less relevant, or more generic. These signals indicate that the model's effective context has degraded to the point where a fresh start with synthesised context will produce better results than continuing the current thread.

A proactive approach — chunking at planned intervals or phase transitions rather than waiting for degradation — produces more consistent results than a reactive approach. Some practitioners chunk after every major decision milestone regardless of conversation length, treating each chunk as a focused work session with a clear objective.

The handoff summary is the bridge between conversation chunks, and its quality determines whether continuity is maintained or lost. An effective summary captures three categories: decisions made (what was concluded and why, including alternatives that were considered and rejected — a form of context compression), constraints active (what rules, requirements, or limitations apply going forward), and current state (where the work stands, what is complete, what remains, and what the next step should be).

Keep handoff summaries concise and structured. A common mistake is copying large portions of the previous conversation into the new one, which reintroduces the noise problem that chunking was meant to solve. Instead, distil the conversation to its essential outputs — usually one to three focused paragraphs or a structured list of decisions and constraints. The goal is to give the new conversation the minimum context needed to continue effectively, not a transcript of the previous session.

For complex projects that span many sessions, consider maintaining a persistent project document — in a tool like <a href="/aisapedia/claude-projects">Claude Projects</a> — that accumulates decisions and constraints across all chunks. Each new conversation references this document rather than a per-session summary, ensuring that decisions from session one are still accessible in session ten. This approach scales better than per-session handoffs for long-running projects.

For technical projects (building software, designing systems), chunk by development phase: requirements analysis, architecture design, implementation, testing, deployment. Each phase has distinct context needs, and carrying implementation details into the architecture session — or vice versa — adds noise without value.

For analytical projects (research, strategy, competitive analysis), chunk by analysis stage: data gathering, pattern identification, hypothesis formation, validation. The data-heavy gathering phase fills context with raw material that the synthesis phases do not need verbatim — they need the extracted insights, not the source data.

For creative projects (content creation, campaign development), chunk by creative phase: brief development, ideation, drafting, refinement. The ideation phase benefits from a wide-open context with many possibilities; the refinement phase benefits from a focused context containing only the selected direction and quality criteria. Carrying rejected ideas into the refinement session can cause the model to reintroduce them.

The most common mistake is writing handoff summaries that are too thin, omitting constraints or rejected alternatives that become relevant in the next session. When the new session proposes an approach that was already considered and rejected, the team loses time re-evaluating it. Include not just what was decided but why alternatives were ruled out — this prevents the new conversation from revisiting settled ground.

Another mistake is chunking too frequently, which fragments context unnecessarily. If a conversation is flowing productively and the model is maintaining coherence, interrupting it to chunk introduces overhead without benefit. Chunk when you observe degradation signals or reach a natural phase boundary, not on a rigid schedule that ignores the conversation's actual state.

A third mistake is inconsistent handoff ownership. In team settings, if different people write handoff summaries with different levels of detail and different assumptions about what the next person needs, the quality of cross-session continuity varies unpredictably. Establish a handoff template that standardises what every summary includes, ensuring consistent quality regardless of who writes it. The template should cover: key decisions (with rationale), active constraints, current status, and the next action to take — providing a minimum viable context for any team member starting the next chunk.