Transformer Architecture

Before transformers, language models used recurrent neural networks (RNNs) that processed text one token at a time, left to right. This sequential processing created two fundamental problems: information from early tokens degraded as the sequence grew longer (the vanishing gradient problem), and training was slow because each token had to wait for the previous one to be processed — no parallelism was possible.

Transformers solved both problems with the self-attention mechanism, which processes all tokens in parallel and computes direct relationships between any two positions in the sequence. Token one can attend to token ten thousand with the same computational directness as to token two. This parallelism also made training dramatically faster on GPU hardware, enabling the massive scale that produced today's large language models.

The practical consequence for users is that transformers can handle long documents, maintain coherence across extended conversations, and connect concepts that appear far apart in a prompt — capabilities that earlier architectures struggled with. The architecture's ability to see the entire input simultaneously, rather than one word at a time, is why modern models can perform tasks like summarising long documents, translating between languages, and following complex multi-part instructions.

Understanding this architectural foundation helps explain many observable model behaviours — from why prompt structure matters to why models have context window limits to why they excel at pattern matching but struggle with tasks requiring genuine sequential reasoning like multi-digit arithmetic.

Self-attention works by computing a relevance score between every pair of tokens in the input. When the model encounters the word 'bank' in a sentence about rivers, the attention mechanism assigns high relevance to nearby words like 'river' and 'water', steering the model toward the geographical meaning rather than the financial one. This contextual disambiguation happens across multiple attention heads, each learning to track different types of relationships — syntactic structure, semantic meaning, co-reference, and more.

This mechanism explains several observable model behaviours. Models are good at following instructions placed at the beginning or end of prompts because attention scores tend to be highest at sequence boundaries — a phenomenon often called the 'lost in the middle' effect, which is directly relevant to context window management. It also explains why models can struggle when a prompt defines many similar terms: the attention mechanism must distribute its capacity across competing definitions, and ambiguity increases with the number of similar concepts.

Understanding attention also explains why clear, well-structured prompts outperform verbose ones. Every token in the prompt competes for attention. Filler words, redundant instructions, and irrelevant context dilute the attention available for the tokens that actually matter to the task. A concise prompt with high signal-to-noise ratio allows the attention mechanism to focus on what matters rather than spreading thin across padding.

Multi-head attention is why models can track multiple aspects of a sentence simultaneously. One head might track subject-verb agreement across a long sentence while another tracks sentiment and a third tracks topic. This parallel processing of different relationship types is what gives transformers their remarkable ability to understand nuanced text — but it also means that confusing or contradictory inputs create conflict between heads, reducing output quality.

Several practical prompt engineering techniques — like those in this score-9 prompt teardown — follow directly from how transformers process input. Placing critical instructions at the beginning of the prompt exploits the attention bias toward early tokens. Using clear structural markers — headers, numbered lists, explicit delimiters — helps the attention mechanism parse the prompt's organisation, reducing the chance that the model misinterprets which instruction applies to which section.

The architecture also explains why few-shot prompting works so effectively. When you include examples in your prompt, the attention mechanism can directly compare the structure of your examples with the task at hand, reproducing patterns with high fidelity. This is more reliable than abstract instructions because the model is matching concrete patterns rather than interpreting natural language descriptions of desired behaviour.

Token limits are a direct consequence of the architecture. The attention mechanism computes relationships between every pair of tokens, meaning computational cost grows quadratically with sequence length. Context windows represent the practical boundary where this computation remains feasible — a constraint that system architects must design around in production systems. Various techniques like sliding window attention and sparse attention reduce this cost, enabling the larger context windows in modern models.

The architecture's pattern-matching nature also explains both strengths and limitations. Models excel at tasks that can be solved by recognising and reproducing patterns — translation, summarisation, code generation in familiar languages, style matching. They struggle with tasks that require genuinely novel reasoning — mathematical proofs, complex logical chains, and situations where the correct answer violates common patterns in the training data.

Transformers do not process words — they process tokens, which are sub-word units determined by the model's tokenizer. The word 'unbelievable' might be three tokens ('un', 'believ', 'able'), while common words like 'the' are a single token. This tokenization is invisible to users but has practical consequences for prompt design and cost management.

Non-English text, technical jargon, and code typically require more tokens per concept than standard English prose. A prompt that fits comfortably in the context window in English might exceed it when translated to another language, even though the semantic content is identical. Similarly, dense technical content with specialised terminology costs more per concept than conversational text.

Understanding the token model also explains why models sometimes produce unusual behaviour with rare words, proper nouns, or novel terminology. If a company name or technical term is tokenized into fragments that individually have different meanings, the model must use context to reassemble the intended meaning — a process that sometimes fails. Adding a brief definition or gloss for unusual terms helps the model process them correctly.

At scale, token awareness becomes a token economics skill. Teams processing large volumes of text through AI APIs can reduce costs significantly by being deliberate about what enters the context window. Removing redundant context, compressing verbose instructions, and structuring prompts efficiently are all token-level optimisations that compound across thousands of requests.