Resilience and caching

A single provider outage, rate limit, or degraded model should not take a feature down. Resilience and caching give a surface production-grade traffic handling with no external dependencies to run alongside it. The response cache and rate limiter are process-local, in-memory state with no Redis or shared backend, so cache hit rate and rate-limit windows are per-instance in a horizontally-scaled deployment. LLM drift detection →

How routing and resilience work together

Before a request reaches an upstream provider, routing mechanics determine which surface variant or target handles it, and resilience controls protect the execution against provider degradation or outages.

Routing Rule

A Routing Rule is an ordered rule that matches a request header, a top-level body field, or the request’s detected modality (image, audio, document, text, realtime), and routes to a variant or an inline provider target. First match wins. Routing rules take precedence over canary splits.

Canary

A Canary is a weighted split across variants and the base Surface, so two full-config postures can be A/B-tested by percentage. The precedence is: an explicit $alias in the route always wins, then a routing rule, then a canary split. See Surface Variants →.

How Failover and Load Balancing Pools protect a provider call

A surface can define ordered backup providers of a different vendor or model, or distribute requests across a pool of providers.

Failover

Failover retries a failed request against ordered backup providers of a different vendor or model. It is content-safety-aware: a request blocked by a guardrail or by the gateway’s own rate limit is never silently retried against another vendor.

A failover chain falls through its backups on an eligible error, stopping at the first one that succeeds:

Load Balancing Pool

A Load Balancing Pool distributes requests across a pool of providers with a selectable strategy: a predictable weighted split, the lowest observed latency, or the lowest catalogue cost. See Resilience and caching → for the full strategy reference.

Load balancing and failover are mutually exclusive at runtime (load balancing wins if both are configured).

How circuit breakers and retry policies handle failed calls

Circuit Breaker

A Circuit Breaker is a state machine that tracks consecutive upstream failures. It opens on a configured threshold, half-opens after a cooldown to test recovery, and closes on success, so a degraded provider does not cause cascading failures across the Surface.

Retry Policy

A Retry Policy is the exponential-backoff configuration for the LLM upstream call, with a maximum attempt count, initial delay, maximum delay, and a backoff multiplier. Unlike failover and the circuit breaker, which explicitly exclude guardrail rejections, retry does not filter by error type: it retries any failed attempt, including a request a guardrail just blocked, against the same provider up to the configured attempt count.

How response caching avoids repeat provider calls

A surface can serve a repeat request from an in-memory cache instead of calling the provider at all.

Exact Cache

Exact Cache matches a SHA-256 hash of the normalised request (object-key order and volatile fields ignored). Two requests that differ only in field ordering hash the same.

Semantic Cache

Semantic Cache embeds the request text via a configured embedder and returns the highest-similarity prior answer above a threshold. It catches paraphrased duplicates that exact-match would miss.

Cache Provenance

Cache Provenance is the record of which provider and model produced a cached entry. Provenance is surfaced in the response headers when a cached entry is returned, so a caller can tell what generated the answer they are receiving.

Cache savings

Cache Savings is the cumulative spend that the response cache prevented, calculated from the cached entries’ original provider/model pricing. It is surfaced on the cache dashboards alongside hit count, hit rate, and tokens saved.

Cache entries are scoped per surface, per variant, and per effective provider target, expire by time-to-live and maximum entry count, and are purged when surface configuration changes. Streaming responses are not cached.

Why this matters

• Failover and load balancing mean a provider outage degrades a surface’s performance instead of breaking it outright, without every team re-implementing retry logic.
• Response caching cuts cost and latency on repeated or near-duplicate prompts, such as FAQs, retried agent steps, or common tool calls, without changing what the client sends.

Related

• Pipeline and stages: Where the cache lookup and the provider call sit in the overall stage order.
• Cost and usage governance: How a cache hit affects spend and quota differently from a provider call.
• Surface Variants: How a load-balanced pool of providers relates to surface variants.
• Providers and Models: The provider adapters that failover and load balancing select between.
• LLM drift detection: How replay evaluation tests a resilience or caching change against a surface’s baseline.