Resilience and caching
See Resilience and caching → for the conceptual model. Content rejections from Prompt Guard and the gateway’s own rate limit never trigger failover or load-balancing fallthrough.
Failover and load balancing are set in the Failover & Load Balancing card of the LLM editor, opened with Configure LLM… on the LLM node. The card’s mode selector offers Off, Failover, and Load balancing, so a surface uses at most one of the two. It starts at Off.
Failover
Failover retries a request against ordered backup providers when the primary fails with a transport- or upstream-level error. It keeps a surface answering through an outage or transient error instead of the caller seeing a failure.
- Failover fits a clear primary provider with one or more backups held in reserve, used only when the primary fails. Order backups by preference, since they’re tried in that order.
- To spread normal traffic across multiple providers instead of holding backups in reserve, choose load balancing instead.
| Field | What it does | Default |
|---|---|---|
| Failover | Choosing Failover in the mode selector turns failover on. | Off |
| Targets | Ordered backup providers, tried after the primary fails. At least one is needed. Shown as numbered Backup 1, Backup 2, … cards with an Add backup provider button. | One empty Backup 1 card |
Each backup target is a provider/model/endpoint target (see Guardrails → for its fields), set with the Provider, Endpoint, Model, and API key controls. The API key field takes the provider’s own label (for example OpenAI API Key), or API Key Secret for a provider without one. A target needs a provider, model, and endpoint. A relative endpoint is resolved against the provider’s base URL, which for a custom provider comes from its definition on the Models page. The resolved endpoint must be an absolute http or https URL that the appliance can reach. Localhost and cloud instance-metadata addresses are rejected.
Load balancing
Load balancing distributes requests across a pool made up of the base provider plus weighted backup targets, with failover-style fallthrough on transport or upstream errors. Primary and backups act as one pool that shares normal traffic, rather than backups sitting in reserve. Useful for spreading load across comparable providers for capacity or cost, not just disaster recovery.
Pick a strategy based on what matters most for the pool: a predictable proportional split, the fastest response, or the lowest cost. See the “When to pick it” column below.
| Field | What it does | Default |
|---|---|---|
| Load balancing | Choosing Load balancing in the mode selector turns load balancing on. | Off |
| Strategy | Selection strategy across the pool (below). | Weighted round-robin |
| Primary weight | Weight of the surface’s own provider in the pool. A weight of 0 keeps it only as a fallthrough target. | 1 |
| Targets | Weighted backup providers. At least one is needed. Shown as numbered Pool member 1, Pool member 2, … cards with an Add pool member button. | One empty Pool member 1 card, weight 1 |
Strategies:
| Option | Behaviour | When to pick it |
|---|---|---|
| Weighted round-robin (default) | Deterministic weighted round-robin. | You want predictable, proportional traffic split across a pool of comparable providers, for example spreading load across two contracts with the same model for capacity reasons. |
| Lowest latency | Routes to the pool member with the lowest recent average latency over completed requests. Members with no measurements yet are tried first to gather a baseline; ties go to the earliest pool member. | Response speed matters more than which provider serves the request, and the pool members are otherwise interchangeable (similar model quality and cost). |
| Lowest cost | Routes to the pool member with the lowest price: its declared price when one is set, otherwise the catalogue price for its model (input plus output per 1K tokens). Members with neither come last; ties go to the earliest pool member. | You’re pooling multiple providers or models that can all serve the request adequately, and you want spend minimised above latency or any single provider’s characteristics. |
Each pool member is a provider/model/endpoint target (same Provider / Endpoint / Model / API key controls as the failover target above), plus a Weight (a whole number, default 1; 0 keeps the member only as a fallthrough target). The weights still decide which members take part under Lowest latency and Lowest cost.
Validation: an enabled pool needs at least one target and a positive total weight (primary weight plus the sum of target weights); every target is checked the same way as a failover target.
Streaming passthrough
Streaming passthrough sends a streaming response straight back to the caller as the provider produces it. It’s set in the SSE streaming passthrough section of the Surface Configuration editor.
Passthrough needs a base provider whose wire format is OpenAI, Anthropic, or Microsoft 365 Copilot. It works with no routing stage, with a Judge, or with a Decider in Decide LLM based on content mode that has at least one candidate. The surface answers a streaming request with a buffered response, rather than rejecting it, in these cases:
- Streaming is turned off for the surface.
- A Jury is enabled.
- Mode is Compare output from multiple LLMs or Summarise output from multiple LLMs.
- The surface has response-side Prompt Guard rules or a response-side Expert Witness.
- Governed server tools are active.
| Field | What it does | Default |
|---|---|---|
| Allow SSE streaming passthrough | When a client requests a streaming response and the surface is eligible (see above), the upstream token stream is proxied straight back to the client. Turn this off to send every request through the buffered path, even for streaming clients. | On |
Response cache
The response cache serves a repeat request from cache instead of calling the provider. That’s a real win for cost and latency on any surface that sees the same questions more than once, from FAQ-style traffic to retried near-identical prompts. A cache hit is billed at zero spend. The cache is checked before the streaming path, so a streamed request can be served from the cache too. A streamed answer is written to the cache only when the request offers no tools. The cache is set in the Response cache section of the Surface Configuration editor, and is off until you turn it on.
- Exact — identical normalized request, the default, matches repeated requests with no extra infrastructure and nothing to tune.
- Semantic — embedding similarity also catches near-duplicate (not identical) requests, at the cost of an embedder to configure and an embedding call per request to check.
- Start the similarity threshold at the 0.95 default and adjust by what you observe: lower it a little (e.g. to 0.9) if requests you’d expect to be near-duplicates aren’t hitting the cache; raise it (e.g. to 0.98) if clearly unrelated requests are getting served the same cached answer.
Once the cache has served a hit, its savings appear on the Dashboard page and on the surface’s Monitoring tab.
| Field | What it does | Default |
|---|---|---|
| Enable response cache | Master switch. | Off |
| TTL (seconds) | Time-to-live for a cached response. | 3,600 seconds (1 hour) |
| Max entries | Maximum cached responses retained before the oldest are evicted. | 1,000 |
| Matching mode | Matching strategy (below). | Exact — identical normalized request |
| Similarity threshold | Minimum similarity for a semantic cache hit, set with a slider from 0.5 to 1. The label shows the current value. Semantic mode only. | 0.95 |
| Embedder provider | Embeddings provider used for semantic matching, picked from providers that offer an embeddings endpoint. Choosing one fills in its endpoint; Custom / other lets you enter the endpoint yourself. Required in semantic mode. | Not set |
Matching modes:
| Option | Behaviour |
|---|---|
| Exact — identical normalized request (default) | Requests that are identical once normalised match. |
| Semantic — embedding similarity | Requests whose embedding similarity clears the similarity threshold match. |
Embedder fields, in semantic mode:
| Field | What it does |
|---|---|
| Embedder endpoint URL | Full embeddings endpoint URL, e.g. https://api.openai.com/v1/embeddings. Shown only after choosing Custom / other in the provider picker. |
| Embedder model | Embedding model name, e.g. text-embedding-3-small. |
| API key (secret) | A secret holding the embedder’s API key, or None (no auth). |
Validation: an enabled cache needs a positive TTL and max entries; semantic mode also needs an embedder with a model and an endpoint. The embedder endpoint must be an absolute http or https URL that the appliance can reach, and localhost and cloud instance-metadata addresses are rejected.
Related
- Guardrails: The provider/model/endpoint target shape shared by failover and load-balancing targets.
- Providers: The provider fields shared by every failover and load-balancing target.
- Routing and variants: Canary and routing rules, which select among variants rather than a resilience pool.
- Resilience and caching concept: Why content rejections never trigger failover.
Glad to hear it! Please tell us how we can improve more.
Sorry to hear that. Please tell us how we can improve.
Thank you for sharing your feedback so we can improve your experience.