API Integration Patterns

Traditional REST APIs return consistent responses in predictable timeframes. An e-commerce product API returns a JSON object in under 100 milliseconds, every time. AI model APIs behave differently: response times vary by an order of magnitude depending on prompt length and output complexity, rate limits are aggressive and shared across all customers, outputs are non-deterministic by design, and the service may experience degraded performance during traffic spikes that affect all users simultaneously.

These characteristics break standard integration assumptions. A timeout set for a traditional API will fire prematurely on an AI API that legitimately needs 30 seconds for a complex generation. A retry strategy that works for a 500 error on a traditional API can trigger rate limiting on an AI API that returns 429s during traffic spikes. And caching strategies must account for non-deterministic outputs — caching an AI response is valid for some use cases (factual reference lookups) but inappropriate for others (creative generation where variety matters).

Streaming adds another dimension of complexity. Understanding token prediction helps explain why responses arrive incrementally. Many AI APIs support streaming responses (returning tokens as they're generated), which improves perceived latency but requires different error handling — a stream can fail partway through, leaving the application with a partial response that must be detected and handled.

Exponential backoff with jitter is the standard starting point: wait 1 second, then 2, then 4, with a random offset to prevent multiple clients from retrying simultaneously and creating a thundering herd. But AI APIs require additional nuance beyond generic backoff. Rate limit responses (HTTP 429) often include a Retry-After header that specifies exactly how long to wait — respecting this header is more efficient and more polite than generic backoff.

Circuit breaker patterns prevent cascading failures when the AI service is experiencing prolonged issues. After a threshold of consecutive failures (typically three to five), the circuit 'opens' and subsequent calls return immediately with a fallback response rather than waiting for another timeout. The circuit 'half-opens' after a cooldown period, allowing a single test request through to check whether the service has recovered. This protects both your application from accumulating hung requests and the AI service from being overwhelmed by retry storms.

The fallback itself is a critical design decision. Options range from returning cached previous responses (works for personalisation and recommendations), falling back to a simpler model or rule-based system (works for classification and extraction), queuing the request for later processing (works for batch operations), or showing a graceful degradation message to the user (necessary for real-time interactive features). The right choice depends on the use case — a chatbot needs a real-time fallback while a background data pipeline can queue and retry transparently.

AI API costs scale with usage in ways that can produce surprise bills. A prompt injection attack that causes your application to generate unusually long responses, a bug that triggers retry loops, a traffic spike from a viral marketing campaign, or a misconfigured batch job that sends thousands of duplicate requests can each multiply costs by orders of magnitude. Production integrations need usage controls independent of the API provider's billing alerts, which may lag by hours or days.

Implement per-request cost estimation before sending the request (based on input token count and expected output length), daily and monthly spending caps that circuit-break when exceeded, per-user rate limits that prevent individual users from consuming disproportionate resources, and anomaly detection that flags unusual usage patterns. These controls should fail closed — when the spending cap is hit, the system should stop making API calls and serve fallback responses, not log a warning and continue accumulating charges.

Context caching and prompt optimisation serve as both cost controls and performance improvements. By reducing the number of tokens sent per request through caching static content, optimising prompt length, and batching related requests, teams can reduce costs significantly without changing the user experience. These optimisations compound — a 30% reduction in tokens per request across thousands of daily requests translates to substantial savings.

Running the same workload across multiple AI providers — a decision informed by model selection criteria — (Anthropic, OpenAI, Google) provides resilience against single-provider outages and leverage in pricing negotiations. However, it introduces significant complexity: different providers have different APIs, different model capabilities, different token counting methods, different content policies, and different failure modes. The integration layer must abstract these differences while handling provider-specific edge cases.

In practice, most teams start with a single provider and add a second only when they have a concrete reliability or capability reason — either the primary provider has experienced outages that affected production, or different tasks benefit from different providers' strengths. A reasonable middle ground is designing the integration layer with provider abstraction from the start (so switching is possible) without actually maintaining active integrations with multiple providers until the need is proven.

This avoids paying the ongoing complexity cost of multi-provider management (testing against both providers, maintaining compatibility layers, handling discrepancies in output quality) while preserving the option to add providers quickly when the need arises. The abstraction layer should normalise request and response formats, handle authentication per provider, and provide a consistent error taxonomy across providers.

Beyond standard uptime monitoring, AI API integrations benefit from quality monitoring that tracks whether the responses are useful, not just whether they arrive. A 200 OK response that contains a hallucinated or irrelevant answer is functionally equivalent to a failure for the end user. Quality monitoring samples responses, evaluates them against expected patterns or quality thresholds, and alerts when quality degrades — even if the API itself appears healthy.

Latency monitoring should track percentiles (p50, p95, p99) rather than averages, because AI API response times have heavy tails. An average latency of 2 seconds may conceal a p99 of 30 seconds, meaning one in a hundred users waits half a minute for a response. Setting alerts on tail latency rather than average latency catches the degradation patterns that affect real user experience.

Cost monitoring should operate in near-real-time, not on billing cycle delays. A runaway loop, a prompt injection attack that inflates output length, or a traffic spike can multiply costs within hours. Dashboards that show current-day spend against budget, with alerts at threshold percentages, give teams time to intervene before a cost anomaly becomes a billing surprise. Pairing cost alerts with automatic circuit breakers that throttle requests when spending exceeds thresholds provides a safety net that does not depend on human response time.