Reliability without effort

A single provider outage, rate limit, or degraded model should not take a feature offline. Yet building failover and caching into every agent and service is repetitive work, and each team ends up reimplementing it inconsistently, or not at all.

What you can do

• Fail over to backup providers of a different vendor or model so that a single provider outage degrades performance instead of breaking the feature.
• Load-balance across a pool of providers by lowest latency or lowest cost so that traffic follows the fastest or cheapest healthy option.
• Serve repeat and near-duplicate prompts from a cache so that common requests skip the provider call entirely and return faster.
• Apply all of this per surface with no external infrastructure so that no team re-implements resilience in its own code.
• Detect when a model’s behaviour drifts from a pinned baseline, with drift alerts through your integrations, so that you learn about a silent provider change from an alert rather than from user reports.

How it works

Before a request reaches a provider, the surface can serve it from a response cache. An exact cache matches the normalised request, and a semantic cache matches paraphrased duplicates above a similarity threshold, so a cache hit returns without any provider call. On a miss, failover sends the request to ordered backup providers of a different vendor or model, stopping at the first that succeeds, while a load-balancing pool can instead distribute traffic by weighted split, lowest observed latency, or lowest price. Each surface uses either failover or load balancing, chosen in its Failover & Load Balancing settings.

Failover is content-safety-aware: a request blocked by a guardrail is not sent to a backup provider, and if every eligible backup fails, the request fails rather than returning a degraded or unsafe answer.

Related

• Resilience and caching: How failover, load balancing, and the response cache work together on a surface.
• Fail over to a backup provider: Step-by-step configuration of an ordered failover chain across vendors.
• Balance traffic across providers: How to distribute requests across a pool by latency or cost.
• Cache repeated responses: How to turn on exact and semantic caching and measure the spend it avoids.
• LLM drift detection: How a surface is measured against a pinned baseline and alerts on drift.