Roll out a model with a canary split

This guide creates a variant that overrides a surface’s model, then routes a small percentage of alias-less traffic to it with a canary split, so a change is tested against real production traffic before it becomes the default for every caller. For the conceptual model, see Variants and progressive rollout →.

Rolling out a new model, provider, or guardrail posture usually means redeploying the client or standing up a parallel endpoint. Without a canary split, the only way to compare a candidate change against production traffic is to switch the whole surface over and watch what happens to every caller at once.

Use this guide when:

• You want to test a new model, provider, or guardrail posture against a small percentage of real traffic before promoting it.
• The change should be reachable at the same route clients already call, with no client-side redeploy.
• You need per-variant cost, latency, and rejection figures to decide whether to promote, adjust, or roll back.

You do not need a canary split if you only need one caller group to explicitly opt into a variant; address the variant directly with $alias instead, and skip the canary weighting.

Prerequisites

• An active LLM Surface.
• The candidate provider, model, or guardrail configuration you want to test.

Steps

Create a variant

Creating a variant adds a weighted row for it alongside the base surface in the canary split panel:

Under SURFACES in the dashboard sidebar, select LLM, open your surface, and use the variants widget on the canvas to create a new variant. Set its Alias (lowercase letters, digits, and dashes, 1–32 characters) and Name.

Override the variant’s configuration

While editing the variant, change only what differs from the base surface, for example the Pipe Type to point at the candidate provider and model. A variant built in the dashboard is a frozen snapshot of the base surface’s provider/model configuration, rate limit, cost tracking, override policy, OPA policy, Prompt Guard, Judge, and Jury, taken at creation time: those fields are copied as-is, and a later edit to the base surface does not propagate to the variant. The variant’s identity, listener, and route always stay fixed to the base surface’s.

Several other fields on the base surface, including source authentication, circuit breaker, retry policy, response cache, usage alerts, Expert Witnesses, and Forward Caller Token, are not captured by the dashboard’s variant editor: a variant resolves those specific fields as empty, regardless of what the base surface has configured. If your base surface relies on source authentication, do not assume a canary variant inherits it; there is currently no dashboard control to add it to a variant.

Save the variant

Save the surface. The variant is now reachable directly at {route}$alias/... alongside the base surface’s own route.

Enable the canary split

Select the Surface node on the canvas (the root container the rest of the pipeline sits inside) and select Configure Surface Features in its side panel. Scroll to Canary / A-B traffic split and turn on Enable canary split.

Set the split percentage

In the canary rows, one per target (the base surface plus each variant), enter a small percentage for your new variant, such as 5. Entering a value rebalances the other rows automatically so every row’s percentage still sums to 100%.

Save the surface

Select the save icon in the toolbar. The change applies immediately; no restart is required.

Confirm

Replace <YOUR_APPLIANCE_HOST>, <YOUR_SURFACE_ROUTE>, and <ALIAS> with your surface’s values and the variant’s alias.

Test 1: an explicit alias always returns 200 from the variant

curl -k -i -X POST "https://<YOUR_APPLIANCE_HOST><YOUR_SURFACE_ROUTE>\$<ALIAS>/v1/chat/completions" \
  -H "Content-Type: application/json" \
  -d '{
    "messages": [
      { "role": "user", "content": "Say hello from the canary variant." }
    ]
  }'

The -k flag disables TLS certificate verification. Use this for local testing only. Remove it in production.

Expected: 200 OK, with x-agent-stream-provider and x-agent-stream-model naming the variant’s overridden provider and model. An explicit $alias in the route always wins over any canary split.

Test 2: alias-less traffic reaches the variant in roughly the configured percentage of responses

Send a batch of requests to the base route with no $alias, and tally the x-agent-stream-provider/x-agent-stream-model headers across responses, the same way as in Balance traffic across providers by cost or latency. Expected: roughly the configured percentage of responses show the variant’s provider and model, with the remainder showing the base surface’s.

Troubleshooting

Next steps

• Fail over to a backup provider on error: Add resilience to the variant once it is promoted to production traffic.
• Balance traffic across providers by cost or latency: Compare the canary approach to distributing every request across a weighted pool.
• Observability: Watch the variant’s per-variant cost, latency, and rejection figures before promoting it.

Related

• Routing and variants reference: Every canary, routing-rule, and variant field.
• Variants and progressive rollout: The precedence order between an explicit alias, routing rules, and a canary split.
• LLM drift detection: Replay captured traffic through a candidate variant to check it against a baseline before it takes any live traffic.