# Balance traffic across providers by cost or latency

> Distribute requests across a weighted pool of providers, selecting members by weighted round-robin, lowest observed latency, or lowest catalogue cost.

This guide switches a surface from a single fixed provider to a pool of providers, selected on every request by a strategy you choose. For the conceptual model, see [Resilience and caching →](/products/affinidi-trust-fabric/agent-stream/concepts/resilience-and-caching.md).

A surface with one fixed provider sends every request to the same place regardless of that provider’s current latency or relative cost against alternatives. Without load balancing, taking advantage of a cheaper or faster alternative model means manually re-pointing the surface, rather than letting traffic find the best option automatically.

Use this guide when:

- You want normal traffic spread across more than one provider or model, not only a fallback used on failure.

- Cost or latency should drive which pool member handles a request, rather than a fixed, manually chosen split.

- You are comparing two or more equivalent models under real production traffic.

You do not need load balancing if you only want a backup used when the primary fails; use [Fail over to a backup provider on error](/products/affinidi-trust-fabric/agent-stream/how-to-guides/resilience/fail-over-to-a-backup-provider.md) instead. Load balancing takes precedence over failover when both are configured.

## Prerequisites

- An active LLM Surface with its primary provider configured.

- Credentials for each additional pool member, stored as secrets.

Secret hygiene

Never hardcode a pool member’s API key into a policy, prompt, or configuration field. Store each credential as a secret, reference it by ID via API Key Secret, and rotate it through the secrets manager. Confirm no credential value appears in surface request logs or debug output before enabling logging on a surface with load balancing enabled.

## Steps

Switch the surface to Load balancing mode

Under SURFACES in the dashboard sidebar, select LLM, open your surface, and select its resilience mode selector: Off, Failover, or Load balancing. Choose Load balancing.

Choose a selection strategy

Set Strategy to one of:

| Strategy | Behaviour |
| weighted (default) | Deterministic weighted round-robin across the pool. |
| lowest_latency | Routes to the pool member with the lowest observed EWMA latency. Unsampled members are tried first to gather a baseline. |
| lowest_cost | Routes to the pool member with the lowest catalogue price for its model. Members without catalogue pricing are deprioritised. |

Set the primary’s weight

Set Primary Weight for the base surface’s own provider (default 1). Set it to 0 to drain the primary from first-pick selection while keeping it as a last-resort fallthrough target.

Add pool members

Switching to Load balancing mode already adds a Pool member 1 card automatically. On it, and on each further card added with Add pool member, set Provider, Endpoint, Model, API Key Secret, and Weight (default 1; 0 excludes it from first-pick selection while keeping it as a fallthrough target).

Save the surface

Select the save icon in the toolbar. The change applies immediately; no restart is required.

## Confirm

Replace <YOUR_APPLIANCE_HOST> and <YOUR_SURFACE_ROUTE> with your surface’s values.

### Test 1: the x-agent-stream-provider header shows the pool member roughly three times more often than the primary

With Strategy set to weighted, Primary Weight at 1, and one pool member at Weight 3, send a batch of requests and tally the x-agent-stream-provider header on each response:

```bash
for i in $(seq 1 20); do
  curl -k -s -o /dev/null -D - -X POST "https:///v1/chat/completions" \
    -H "Content-Type: application/json" \
    -d '{"messages":[{"role":"user","content":"Say hello."}]}' \
| grep -i x-agent-stream-provider
done
```

The -k flag disables TLS certificate verification. Use this for local testing only. Remove it in production.

Expected: across the 20 requests, the pool member’s provider appears roughly three times as often as the primary’s, matching the 1:3 weight ratio.

### Test 2: the x-agent-stream-provider header names the cheaper pool member on every request

Switch Strategy to lowest_cost and repeat the same batch. Expected: every response’s x-agent-stream-provider and x-agent-stream-model headers name whichever pool member has the lower catalogue price, on every request, rather than a mix.

## Troubleshooting

| Symptom | Likely cause | Fix |
| Every request still goes to the primary, even with pool members configured. | The resilience mode selector still reads Off or Failover, or Primary Weight dominates the pool’s total weight. | Confirm the mode selector reads Load balancing, and lower Primary Weight relative to the pool members’ weights if you want more traffic to shift away from it. |
| Traffic looks unevenly distributed right after enabling lowest_latency. | Unsampled pool members are tried first to gather a latency baseline; distribution stabilises only once every member has been sampled. | This is expected warm-up behaviour; allow enough traffic through for every member to be sampled before judging the steady-state distribution. |
| A pool member never gets picked under lowest_cost. | The member’s model has no catalogue pricing, so it is deprioritised regardless of its actual cost. | Confirm the model is present in the model catalogue with pricing, or accept that an unpriced model is not eligible for cost-based selection. |
| Saving the surface fails with a total-weight validation error. | An enabled pool has a zero total weight: every target’s weight, plus the primary’s, sums to zero. | Set a positive weight on at least one target or on the primary. |

## Next steps

- [Fail over to a backup provider on error](/products/affinidi-trust-fabric/agent-stream/how-to-guides/resilience/fail-over-to-a-backup-provider.md): The simpler, backup-on-failure alternative to a weighted pool.

- [Roll out a new model with a canary split](/products/affinidi-trust-fabric/agent-stream/how-to-guides/resilience/roll-out-a-new-model-with-a-canary-split.md): Compare a full configuration change on a percentage of traffic, rather than balancing every request across a pool.

- [Cache repeated responses to cut cost and latency](/products/affinidi-trust-fabric/agent-stream/how-to-guides/cost-and-usage/cache-repeated-responses.md): Avoid a pool call altogether on repeated prompts.

## Related

- [Resilience and caching reference](/products/affinidi-trust-fabric/agent-stream/reference/surfaces/resilience-and-caching.md): Every load-balancing field and validation rule.

- [Response headers](/products/affinidi-trust-fabric/agent-stream/reference/observability/response-headers.md): The x-agent-stream-provider and x-agent-stream-model headers this guide’s Confirm section checks.

- [Providers](/products/affinidi-trust-fabric/agent-stream/reference/surfaces/providers.md): How catalogue pricing resolves, which the lowest_cost strategy depends on.