Fail over to a backup provider on error

Add ordered backup providers to a surface, so a request retries against a backup instead of failing outright when the primary provider errors.

This guide adds one or more ordered backup providers to a surface, so a transport- or upstream-level failure on the primary provider falls through to a backup instead of failing the request. For the conceptual model, see Resilience and caching →.

Without a configured backup, a single provider outage, rate limit, or degraded model takes the whole surface down for every caller, with no path to recovery beyond waiting for the provider itself. Every team building this retry logic independently produces inconsistent, unaudited failure handling across an organisation’s surfaces.

Use this guide when:

  • A surface’s primary provider has an availability or rate-limit history you want to route around automatically.
  • You have a second provider or model that can serve as an acceptable substitute when the primary fails.
  • You want provider failures handled the same way, and observed the same way, across every surface.

You do not need failover if your surface already uses load balancing; load balancing takes precedence over failover when both are configured, and its pool already falls through on the same errors.

Prerequisites

  • An active LLM Surface with its primary provider configured.
  • Credentials for each backup provider, stored as secrets.

Steps

Switch the surface to Failover mode

Under SURFACES in the dashboard sidebar, select LLM, open your surface, and select its resilience mode selector: Off, Failover, or Load balancing. Choose Failover.

Add your first backup provider

The surface's resilience settings in Failover mode, showing the auto-created Backup 1 card with its Provider, Endpoint, Model, and API Key Secret fields

Switching to Failover mode already adds a Backup 1 card automatically. On it, set Provider, Endpoint, Model, and API Key Secret. A relative endpoint resolves against a built-in provider’s default base URL; a custom or catalogue provider has no built-in base, so give it a complete endpoint URL.

Add further backups in trial order

Select Add backup provider again for each additional fallback. Backups are tried in the order they appear, Backup 1 first, so order the list from most to least preferred.

Save the surface

Select the save icon in the toolbar. The change applies immediately; no restart is required.

Confirm

Replace <YOUR_APPLIANCE_HOST> and <YOUR_SURFACE_ROUTE> with your surface’s values. To force a primary-provider failure for this test, temporarily point the surface’s primary API Key Secret at an invalid value, then restore it afterwards.

Test 1: a primary-provider failure returns 200 from Backup 1, not the primary

curl -k -i -X POST "https://<YOUR_APPLIANCE_HOST><YOUR_SURFACE_ROUTE>/v1/chat/completions" \
  -H "Content-Type: application/json" \
  -d '{
    "messages": [
      { "role": "user", "content": "Say hello from the failover test." }
    ]
  }'

The -k flag disables TLS certificate verification. Use this for local testing only. Remove it in production.

Expected: 200 OK. Check the x-agent-stream-provider and x-agent-stream-model response headers; they should name Backup 1’s provider and model, not the primary’s, confirming the fallthrough happened.

Test 2: a guardrail rejection keeps its own rejection response, with no attempt against any backup

Send a request that a configured Prompt Guard rule rejects, or that exceeds the surface’s rate limit. Expected: the guardrail’s or rate limiter’s own rejection response, unchanged, with no attempt against any backup. Content-safety rejections and Agent Stream’s own rate limit never trigger failover.

Troubleshooting

SymptomLikely causeFix
The request fails outright instead of falling through to a backup.The primary failed with a content-safety rejection or a rate-limit block, neither of which triggers failover, or no backup targets are configured.Confirm the failure is a transport- or upstream-level error, and that at least one backup target is saved under Failover.
Adding a backup fails validation with a relative-endpoint error.The selected provider is a custom or catalogue provider with no built-in base URL, and the Endpoint field holds only a relative path.Enter the backup’s complete endpoint URL instead of a relative path.
A backup never seems to get used, even during a real primary outage.Load balancing mode is also configured, which takes precedence over failover at runtime.Confirm the surface’s mode selector actually reads Failover, not Load balancing, if this is the intended behaviour.
Every backup also fails.The failover chain is exhausted: every configured target failed on this request.Add a further backup, or confirm the existing backups’ credentials and endpoints are correct.

Next steps