Resilience and caching
See Resilience and caching → for the conceptual model. Content rejections from Prompt Guard and the gateway’s own rate limit never trigger failover or load-balancing fallthrough.
Failover
Failover retries a request against ordered backup providers when the primary fails with a transport- or upstream-level error. It keeps a surface answering through an outage or transient error instead of the caller seeing a failure.
- Failover fits a clear primary provider with one or more backups held in reserve, used only when the primary fails. Order backups by preference, since they’re tried in that order.
- To spread normal traffic across multiple providers instead of holding backups in reserve, use load balancing instead. The two are mutually exclusive in the dashboard’s mode selector, and load balancing takes precedence if both are somehow configured.
| Field | What it does | Default |
|---|---|---|
| Enabled | Master switch. Not exposed as its own toggle in the dashboard. The builder instead offers a single Off / Failover / Load balancing mode selector, and choosing “Failover” sets this implicitly. | On |
| Targets | Ordered backup providers, tried after the primary fails. Required (non-empty) when enabled. In the dashboard these render as numbered “Backup 1”, “Backup 2”, … cards with an “Add backup provider” button. There is no field literally labeled “Targets”. | None added |
Each backup target: a provider/model/endpoint target (see Guardrails → for its fields). The dashboard has no single “LLM Target” label for this; it renders separate Provider, Endpoint, Model, and API Key Secret controls. Validation requires a non-empty provider, model, and endpoint; a relative endpoint is resolved against the provider’s default base URL for a built-in provider, but a custom/catalogue provider has no built-in base, so a relative path is rejected with an actionable error. The resolved URL is also checked against the SSRF guard.
Load balancing
Load balancing distributes requests across a pool made up of the base provider plus weighted backup targets, with failover-style fallthrough on transport/upstream errors. Primary and backups act as one pool that shares normal traffic, rather than backups sitting in reserve. Useful for spreading load across comparable providers for capacity or cost, not just disaster recovery. Takes precedence over failover when both are configured.
Pick a strategy based on what matters most for the pool: a predictable proportional split, the fastest response, or the lowest cost. See the “When to pick it” column below.
| Field | What it does | Default |
|---|---|---|
| Enabled | Master switch. Not exposed as its own toggle in the dashboard. The same Off / Failover / Load balancing mode selector described under Failover sets this implicitly. | On |
| Strategy | Selection strategy across the pool (below). | Weighted |
| Primary Weight | Weight of the base surface’s own provider in the pool. A weight of 0 drains it from first-pick selection; it remains a last-resort fallthrough target. | 1 |
| Targets | Weighted backup providers. Required (non-empty) when enabled. In the dashboard these render as numbered “Pool member 1”, “Pool member 2”, … cards with an “Add pool member” button. There is no field literally labeled “Targets”. | None added |
Strategies:
| Value | Behaviour | When to pick it |
|---|---|---|
| weighted (default) | Deterministic weighted round-robin. | You want predictable, proportional traffic split across a pool of comparable providers, for example spreading load across two contracts with the same model for capacity reasons. |
| lowest_latency | Routes to the pool member with the lowest observed EWMA latency over completed requests. Unsampled members are tried first to gather a baseline; ties resolve to the lowest pool index. | Response speed matters more than which provider serves the request, and the pool members are otherwise interchangeable (similar model quality and cost). |
| lowest_cost | Routes to the pool member with the lowest catalogue price for its model (input + output per 1k tokens). Members without catalogue pricing are deprioritised; ties resolve to the lowest pool index. | You’re pooling multiple providers or models that can all serve the request adequately, and you want spend minimized above latency or any single provider’s characteristics. |
Each pool member: a provider/model/endpoint target (same Provider / Endpoint / Model / API Key Secret controls as the failover target above, with no single “LLM Target” label), Weight (integer, default 1; 0 excludes the target from first-pick selection while keeping it as a fallthrough target).
Validation: an enabled pool requires at least one target and a positive total weight (primary weight plus the sum of target weights); every target is validated the same way as a failover target.
Streaming passthrough
Streaming controls SSE streaming passthrough.
| Field | What it does | Default |
|---|---|---|
| Allow SSE Streaming Passthrough | When a client requests a streaming response and the surface is eligible (an OpenAI-schema base provider with no response-side processing), the upstream token stream is proxied straight back to the client. Turn this off to force every request through the buffered path even for streaming clients. Dashboard section heading: “SSE streaming passthrough”; checkbox label: “Allow SSE streaming passthrough”. | On |
Leaving this block absent means streaming is allowed by default.
Response cache
The response cache serves an identical (normalized) request from cache instead of calling the provider. That’s a real win for cost and latency on any surface that sees the same questions more than once, from FAQ-style traffic to retried near-identical prompts. Applies to the buffered (non-streaming) chat path only. Opt-in (defaults off) and variant-overridable.
- exact, the default, matches byte-identical repeated requests, with no extra infrastructure and nothing to tune.
- semantic matching turns this into a semantic cache that also catches near-duplicate (not identical) requests, at the cost of an embedder to configure and an embedding call per request to check.
- Start Similarity Threshold at the 0.95 default and adjust by what you observe: lower it a little (e.g. to 0.9) if requests you’d expect to be near-duplicates aren’t hitting the cache; raise it (e.g. to 0.98) if clearly unrelated requests are getting served the same cached answer.
| Field | What it does | Default |
|---|---|---|
| Enable Response Cache | Master switch. | Off |
| TTL (Seconds) | Time-to-live for a cached response. | 3,600 seconds (1 hour) |
| Max Entries | Maximum cached responses retained before the oldest are evicted. | 1,000 |
| Matching Mode | Matching strategy (below). | Exact |
| Similarity Threshold | Minimum cosine similarity (0.0–1.0] for a semantic cache hit. Only used in semantic matching mode. | 0.95 |
| Embedder Provider | Vectoriser used for semantic matching. Required in semantic matching mode. In the dashboard this is an “Embedder provider” picker (choosing a catalogue provider auto-fills its endpoint and model) plus the separate controls below. | Not set |
Matching modes:
| Value | Behaviour |
|---|---|
| exact (default) | Byte-identical (normalized) request bodies match. |
| semantic | Requests whose embedding cosine similarity clears similarity_threshold match. |
Embedder fields, when matching mode is semantic:
| Field | What it does |
|---|---|
| Embedder Endpoint URL | Full embeddings endpoint URL, e.g. https://api.openai.com/v1/embeddings; shown only after choosing “Custom / other” in the provider picker. |
| Embedder Model | Embedding model name, e.g. text-embedding-3-small. |
| API Key (Secret) | Secret id for the embedder’s API key, if required. |
Validation: an enabled cache requires a positive TTL and max-entries value; semantic mode additionally requires a similarity threshold in (0.0, 1.0] and a valid embedder (a non-empty, SSRF-clear endpoint and non-empty model).
Retry and circuit-breaking for the upstream call itself are not exposed in the dashboard and not configurable by you.
Related
- Guardrails: The provider/model/endpoint target shape shared by failover and load-balancing targets.
- Providers: The provider fields (Temperature, Max Tokens, API Key Secret, …) shared by every failover and load-balancing target.
- Routing and variants: Canary and routing rules, which select among variants rather than a resilience pool.
- Resilience and caching concept: Why content rejections never trigger failover.
Glad to hear it! Please tell us how we can improve more.
Sorry to hear that. Please tell us how we can improve.
Thank you for sharing your feedback so we can improve your experience.