Context Caching

When you send a prompt to an AI API, the model processes every token in the request — system prompt, reference documents, few-shot examples, and the user's actual message. If your system prompt is 10,000 tokens and the user's message is 200 tokens, you're paying to process 10,200 tokens on every request, even though 10,000 of them are identical across every call. For high-volume applications, this redundant processing dominates both cost and latency.

Context caching allows you to mark the static portion of the prompt, similar to how context engineering separates stable from dynamic content (the system instructions and reference documents) as cacheable. The API processes this content once, stores the intermediate computation state, and reuses it on subsequent requests that share the same prefix. You pay the full processing cost on the first request; subsequent requests pay a reduced rate for the cached tokens and full price only for the new, variable tokens.

The cache typically has a time-to-live (TTL) that extends automatically with use. If you're making requests every few minutes, the cache stays warm. If there's a long gap between requests, the cache expires and must be rebuilt on the next call. This means caching provides the most benefit for applications with steady, frequent traffic — exactly the pattern that makes cost optimisation most important.

The savings scale with two factors: the ratio of static to dynamic content and the frequency of requests. A customer support system that sends a 40-page troubleshooting guide with every ticket benefits enormously — the guide is static, the tickets are dynamic, and there are many tickets per hour. The cost per ticket drops by an order of magnitude because the guide is processed once and reused hundreds of times.

Conversely, caching provides little benefit for applications where most of the prompt is dynamic (each request is unique) or where requests are infrequent enough that the cache expires between calls. A one-off analysis task with a custom prompt doesn't benefit from caching because there's no repetition to amortise the initial processing cost against.

Applications with large few-shot example sets see particularly strong benefits. A classification system that includes 50 labelled examples in every prompt can cache the examples and only process the new item being classified. This pattern is common in enterprise applications where the examples represent the company's specific taxonomy or classification rules and change infrequently.

Latency reduction is often as valuable as cost reduction. By skipping the processing of cached tokens, the model begins generating the response sooner. For interactive applications where users are waiting for a response, reducing time-to-first-token from 3 seconds to under 1 second materially improves the user experience.

Claude Projects, custom GPTs, and similar product features implement a form of context caching at the user experience level. When you upload documents to a Claude Project, those documents persist across conversations — you don't re-upload or re-explain them each session. The underlying mechanism achieves a similar effect: the platform stores the project documents in a way that makes them available to every conversation without full reprocessing from scratch.

The distinction matters for different audiences. End users benefit from product-level caching through features like Projects — no code required, just upload documents and start conversations. Developers building applications benefit from API-level caching by structuring their prompts so that static content is flagged as cacheable, reducing costs and latency programmatically. Both approaches serve the same principle: identify the content that doesn't change between requests and process it only once.

For developers, the prompt structure matters. The cacheable prefix must be identical across requests — the same tokens in the same order. Any variation in the static portion (even whitespace differences) creates a cache miss and triggers full reprocessing. This means prompts should be designed with a clear separation between the stable prefix (system instructions, reference documents, examples) and the variable suffix (user message, dynamic context).

Place all static content at the beginning of the prompt and all dynamic content at the end. The cache matches on the prompt prefix — each token must match exactly — it checks whether the start of the new request matches a cached computation. If your system prompt is 10,000 tokens followed by a 200-token user message, the 10,000-token prefix is cached. If the user message were placed first, nothing would be cacheable because the prefix changes with every request.

Order static content from most stable to least stable. System instructions that never change come first, followed by reference documents that change monthly, followed by few-shot examples that change weekly, followed by the dynamic user input. This maximises the length of the cacheable prefix even when some components are updated, because only the portions after the changed component need reprocessing.

Avoid including timestamps, random session identifiers, or other per-request values in the static portion of the prompt. A common mistake is including a 'Current date: 2026-05-13' line in the system prompt, which changes daily and invalidates the cache every 24 hours. Move dynamic metadata like dates and session IDs to the variable suffix to keep the static prefix stable.

Cache warming — proactively building the cache before user requests arrive — is valuable for applications with predictable traffic patterns. If your customer support system receives most requests during business hours, warming the cache with the system prompt and knowledge base before the first morning request ensures that the first user of the day gets the same low-latency experience as every subsequent user. Without warming, the first request after a cache expiry pays the full latency and cost penalty.

Cache invalidation is necessary when the cached content changes. If your system prompt is updated, the knowledge base is refreshed, or the few-shot examples are modified, the cached version is stale and must be rebuilt. Invalidation should be triggered automatically by content changes rather than relying on TTL expiry, which may serve stale content until the time-based expiry occurs. Linking cache invalidation to your deployment pipeline ensures that prompt changes take effect immediately rather than being delayed by the cache TTL.

For applications with multiple cache segments (different knowledge bases for different product lines, different few-shot sets for different classification tasks), manage each segment independently. Updating the knowledge base for one product line should invalidate only that segment's cache, not force a rebuild of all segments. This granular cache management reduces the latency impact of content updates by limiting the scope of invalidation.