Balance traffic across providers by cost or latency

Distribute requests across a weighted pool of providers, selecting members by weighted round-robin, lowest observed latency, or lowest catalogue cost.

This guide switches a surface from a single fixed provider to a pool of providers, selected on every request by a strategy you choose. For the conceptual model, see Resilience and caching →.

A surface with one fixed provider sends every request to the same place regardless of that provider’s current latency or relative cost against alternatives. Without load balancing, taking advantage of a cheaper or faster alternative model means manually re-pointing the surface, rather than letting traffic find the best option automatically.

Use this guide when:

  • You want normal traffic spread across more than one provider or model, not only a fallback used on failure.
  • Cost or latency should drive which pool member handles a request, rather than a fixed, manually chosen split.
  • You are comparing two or more equivalent models under real production traffic.

You do not need load balancing if you only want a backup used when the primary fails; use Fail over to a backup provider on error instead. Load balancing takes precedence over failover when both are configured.

Prerequisites

  • An active LLM Surface with its primary provider configured.
  • Credentials for each additional pool member, stored as secrets.

Steps

Switch the surface to Load balancing mode

Under SURFACES in the dashboard sidebar, select LLM, open your surface, and select its resilience mode selector: Off, Failover, or Load balancing. Choose Load balancing.

Choose a selection strategy

Set Strategy to one of:

StrategyBehaviour
weighted (default)Deterministic weighted round-robin across the pool.
lowest_latencyRoutes to the pool member with the lowest observed EWMA latency. Unsampled members are tried first to gather a baseline.
lowest_costRoutes to the pool member with the lowest catalogue price for its model. Members without catalogue pricing are deprioritised.

Set the primary’s weight

Set Primary Weight for the base surface’s own provider (default 1). Set it to 0 to drain the primary from first-pick selection while keeping it as a last-resort fallthrough target.

Add pool members

The surface's Load balancing settings, showing the Strategy dropdown set to Weighted round-robin and a Pool member card with Weight, Provider, Endpoint, Model, and API Key Secret set

Switching to Load balancing mode already adds a Pool member 1 card automatically. On it, and on each further card added with Add pool member, set Provider, Endpoint, Model, API Key Secret, and Weight (default 1; 0 excludes it from first-pick selection while keeping it as a fallthrough target).

Save the surface

Select the save icon in the toolbar. The change applies immediately; no restart is required.

Confirm

Replace <YOUR_APPLIANCE_HOST> and <YOUR_SURFACE_ROUTE> with your surface’s values.

Test 1: the x-agent-stream-provider header shows the pool member roughly three times more often than the primary

With Strategy set to weighted, Primary Weight at 1, and one pool member at Weight 3, send a batch of requests and tally the x-agent-stream-provider header on each response:

for i in $(seq 1 20); do
  curl -k -s -o /dev/null -D - -X POST "https://<YOUR_APPLIANCE_HOST><YOUR_SURFACE_ROUTE>/v1/chat/completions" \
    -H "Content-Type: application/json" \
    -d '{"messages":[{"role":"user","content":"Say hello."}]}' \
    | grep -i x-agent-stream-provider
done

The -k flag disables TLS certificate verification. Use this for local testing only. Remove it in production.

Expected: across the 20 requests, the pool member’s provider appears roughly three times as often as the primary’s, matching the 1:3 weight ratio.

Test 2: the x-agent-stream-provider header names the cheaper pool member on every request

Switch Strategy to lowest_cost and repeat the same batch. Expected: every response’s x-agent-stream-provider and x-agent-stream-model headers name whichever pool member has the lower catalogue price, on every request, rather than a mix.

Troubleshooting

SymptomLikely causeFix
Every request still goes to the primary, even with pool members configured.The resilience mode selector still reads Off or Failover, or Primary Weight dominates the pool’s total weight.Confirm the mode selector reads Load balancing, and lower Primary Weight relative to the pool members’ weights if you want more traffic to shift away from it.
Traffic looks unevenly distributed right after enabling lowest_latency.Unsampled pool members are tried first to gather a latency baseline; distribution stabilises only once every member has been sampled.This is expected warm-up behaviour; allow enough traffic through for every member to be sampled before judging the steady-state distribution.
A pool member never gets picked under lowest_cost.The member’s model has no catalogue pricing, so it is deprioritised regardless of its actual cost.Confirm the model is present in the model catalogue with pricing, or accept that an unpriced model is not eligible for cost-based selection.
Saving the surface fails with a total-weight validation error.An enabled pool has a zero total weight: every target’s weight, plus the primary’s, sums to zero.Set a positive weight on at least one target or on the primary.

Next steps