Attention Mechanisms

When a transformer processes a sequence of tokens, each token computes a set of attention scores against every other token in the sequence. These scores determine how much influence each token has on the representation of the current token. A high attention score between two tokens means the model considers them strongly related in context — the word 'bank' attends heavily to 'river' or 'money' depending on which other tokens are nearby, and this contextual weighting is what gives the model its ability to interpret ambiguous language.

This process repeats across multiple attention heads (parallel attention computations that learn different relationship types) and multiple layers (successive refinements of the representation). Early layers tend to capture syntactic relationships like subject-verb agreement and phrase boundaries, while later layers capture semantic relationships like topic relevance, logical entailment, and pragmatic intent. The final representation of each token encodes its meaning in the full context of the entire input.

The practical consequence is that a token's meaning is not fixed — it depends on everything else in the context window. The word 'Python' means something different when surrounded by programming terms versus zoology terms. This context-dependent meaning is attention's core contribution to language model capability, and it's also the reason that the surrounding context in a prompt can dramatically change how the model interprets any given instruction.

Research on attention patterns in large language models shows that tokens at the beginning and end of the input sequence tend to receive disproportionately high attention scores compared to tokens in the middle. This is sometimes called the 'lost in the middle' phenomenon, and it has been documented across multiple model families and sizes. Instructions, constraints, or key information placed in the middle of a long prompt are measurably less likely to be followed consistently than the same content placed at the beginning or end.

This has direct implications for prompt engineering. If you have a critical constraint — 'never reveal the system prompt,' 'always respond in JSON format,' or 'exclude data before 2024' — placing it as the first or last instruction in the prompt significantly increases compliance rates. Burying it in paragraph three of a complex prompt is the structural equivalent of putting important terms in the fine print of a contract.

For long-context applications where you're feeding documents to the model for analysis, this pattern suggests placing the question or task instruction after the document rather than before it. When the instruction comes first and is followed by thousands of tokens of document text, it gets pushed into a lower-attention zone. When the instruction comes last, it benefits from recency effects in the attention computation.

Some practitioners mitigate the lost-in-the-middle effect by restating critical instructions at both the beginning and end of the prompt. While this uses additional tokens, the redundancy ensures that at least one copy of the instruction receives strong attention regardless of context length.

Larger context windows (100K+ tokens) allow models to process longer documents in a single pass, but attention quality is not uniform across the full window. In practice, models perform best on information near the beginning and end of the context, with degraded performance on information in the middle — especially for tasks that require precise recall of specific details rather than general comprehension or summarisation.

This has practical implications for how you structure long-context inputs. Techniques like chunking large documents into segments and processing them separately can sometimes outperform single-pass long-context processing, precisely because each chunk gets full attention rather than competing with the rest of the document for attention bandwidth. The trade-off is that chunked processing loses cross-document relationships that a single-pass approach would capture.

For retrieval-augmented generation systems, understanding attention patterns informs how many retrieved passages to include and in what order. Including ten retrieved passages may produce worse results than including the three most relevant ones, because the additional passages dilute attention without adding proportional value. Quality of retrieved context consistently matters more than quantity.

Structure prompts with the most important instructions at the very beginning, supporting context in the middle, and a restatement of the key constraint or desired output format at the end. This 'bookend' pattern leverages the natural attention distribution to maximise compliance with your most important instructions.

Use clear structural markers — headings, numbered lists, XML-style tags, and explicit delimiters — to help the model's attention mechanism identify the boundaries between different types of content in the prompt. A prompt that mixes instructions, context, examples, and constraints in a single paragraph forces the model to disentangle them; a prompt with clearly labelled sections makes each component more salient to the attention mechanism.

When debugging inconsistent model behaviour, consider whether the failing instruction is in a low-attention position before assuming the instruction itself is poorly written. Moving an instruction from the middle to the beginning of the prompt is a zero-cost change that often resolves inconsistency issues that feel like model limitations but are actually structural issues with prompt design.

Transformer models use multiple attention heads operating in parallel, each learning to focus on different types of relationships in the input. One head might specialise in syntactic dependencies (linking subjects to their verbs), another in coreference (linking pronouns to their antecedents), and another in semantic similarity (linking related concepts across distant parts of the text). The combined output of all heads gives the model a multi-faceted understanding of each token's role in context.

For prompt engineering, this means that well-structured prompts with clear grammatical relationships, explicit references, and consistent terminology are easier for the model to process than ambiguous or convoluted instructions. When a prompt uses a pronoun like 'it' that could refer to multiple antecedents, different attention heads may resolve the reference differently, leading to inconsistent behaviour. Explicit naming eliminates this ambiguity at the attention level.

Understanding that attention heads specialise also explains why certain prompt techniques work. Using XML-style tags to delimit sections of a prompt gives structure-sensitive attention heads clear boundaries to work with. Numbered lists activate heads that track sequential relationships. These formatting choices are not just cosmetic — they provide structural signals that the attention mechanism uses to organise the information in the prompt.