Resilience and caching

A single provider outage, rate limit, or degraded model should not take a feature down. Resilience and caching give a surface production-grade traffic handling with no external dependencies to run alongside it. LLM drift detection →

How routing and resilience work together

Before a request reaches an upstream provider, routing mechanics determine which surface variant handles it, and resilience controls protect the execution against provider degradation or outages.

Routing Rule

A Routing Rule is an ordered rule on the surface’s Router element that matches a request header, a top-level body field, or an attachment type (image, audio, document, or realtime voice), and routes to a variant. First match wins. Routing rules take precedence over canary splits.

Canary

A Canary is a weighted split across variants and the base Surface, so two full-config postures can be A/B-tested by percentage. The precedence is: an explicit $alias in the route always wins, then a routing rule, then a canary split. See Surface Variants →.

How Failover and Load Balancing Pools protect a provider call

A surface can define ordered backup providers of a different vendor or model, or distribute requests across a pool of providers.

Failover

Failover retries a failed request against ordered backup providers of a different vendor or model. It is content-safety-aware: a request blocked by a guardrail or by the surface’s own rate limit is never silently retried against another vendor.

A failover chain falls through its backups on an eligible error, stopping at the first one that succeeds:

Load Balancing Pool

A Load Balancing Pool distributes requests across a pool of providers with a selectable strategy: a predictable weighted split, the lowest observed latency, or the lowest price. See Resilience and caching → for the full strategy reference.

A surface uses at most one of the two: the Failover & Load Balancing selector offers Off, Failover, or Load balancing.

How response caching avoids repeat provider calls

A surface can serve a repeat request from its response cache instead of calling the provider at all.

Exact Cache

Exact Cache matches a SHA-256 hash of the normalised request (object-key order and volatile fields ignored). Two requests that differ only in field ordering hash the same.

Semantic Cache

Semantic Cache embeds the request text via a configured embedder and returns the highest-similarity prior answer above a threshold. It catches paraphrased duplicates that exact-match would miss.

Cache Provenance

Cache Provenance is the record of which provider and model produced a cached entry. Provenance is surfaced in the response headers when a cached entry is returned, so a caller can tell what generated the answer they are receiving.

Cache savings

Cache Savings is the spend that the response cache prevented, calculated from the cached entries’ original provider/model pricing. Once the cache has served a hit, it appears alongside Cache Hits, Cache Hit Rate, and Tokens Saved on the Dashboard page and on each surface’s Monitoring tab.

Cache entries are scoped per surface, per variant, and per effective provider target, expire by time-to-live and maximum entry count, and are purged when surface configuration changes. The cache is checked before the streaming path, so a streamed request can be served from the cache. A streamed answer is written to the cache only when the request offers no tools.

Why this matters

• Failover and load balancing mean a provider outage degrades a surface’s performance instead of breaking it outright, without every team re-implementing retry logic.
• Response caching cuts cost and latency on repeated or near-duplicate prompts, such as FAQs, retried agent steps, or common tool calls, without changing what the client sends.

Related

• Pipeline and stages: Where the cache lookup and the provider call sit in the overall stage order.
• Cost and usage governance: How a cache hit affects spend and quota differently from a provider call.
• Surface Variants: How routing rules and canary splits pick a variant before the provider call.
• Providers and Models: The provider adapters that failover and load balancing select between.
• LLM drift detection: How replay evaluation tests a resilience or caching change against a surface’s baseline.