Pipeline and stages

Every LLM request rides a governed pipeline. The Surface is a declarative description of which controls are switched on. Anything not configured is simply skipped, so a minimal Surface is a thin, low-latency proxy and a fully loaded one is a governed, guard-railed, multi-model router: the same pipeline, driven by configuration. Security and access control →

What order pipeline stages run in

For an LLM Surface, the inbound path runs in this order, grouped into five phases. The routing stage and the Judge are shown together because, in Decider mode on a buffered request, they run concurrently rather than one after the other. In Comparer or Summariser mode, and on a streamed request, the Judge completes first.

Running the routing stage and Judge side by side, in Decider mode on a buffered request, keeps the routing decision from adding a second round-trip of latency on top of the model call itself. In Comparer or Summariser mode, the Judge runs first instead, so its check still lands before the larger cost of calling every candidate.

What a stage is and how each one is measured

A stage is a named step in the request pipeline. Four stages meter their own cost, tokens, and latency:

• LLM: The main upstream model call.
• Judge: Pre-flight LLM review.
• Jury: Post-flight LLM review.
• Decider: Content-based routing.

Per-stage counters make it clear which of these four is consuming budget. Prompt Guard and Expert Witness also run as named pipeline stages, outside that metered set:

• Prompt Guard: Regular Expression rules, plus AI Matching detection on the request side.
• Expert Witness: External safety classification.

Prompt Guard

Prompt Guard checks requests and responses for sensitive content. A Regular Expression guard matches a Built-in Pattern, such as email addresses, phone numbers, credit card numbers, dates of birth, IPv4 addresses, US passport numbers, or US street addresses, or a Custom Regex of your own. On the request side, an AI Matching guard can also detect entities such as people, organisations, locations, addresses, email addresses, and phone numbers. When a guard detects a match, it rejects the call, redacts the match with a fixed token such as <EMAIL>, or replaces it with a NER ID such as [EMAIL_NER_001] so the request can continue with its context preserved. Prompt Guard runs on the request seam and response seam independently. See Guardrails →.

Expert Witness

An Expert Witness is a first-class, purpose-trained external safety classifier consulted on the request seam, response seam, or both. Built-in providers include EnkryptAI (Detect API, named-Guardrail, AI-Proxy, and legacy Policy modes), Lakera Guard, Azure Prompt Shields (request-side injection detection), Azure Content Safety (response-side harm moderation), and a generic OpenAI moderation shape. Multiple witnesses can run on one Surface. See Guardrails →.

Decider

The Decider is a router model that inspects each request and selects the best upstream model from an allow-list of candidates (for example, routing simple requests to a smaller model and complex requests to a frontier model), with a default candidate as the fallback. The Decider runs in parallel with the Judge on a buffered request to absorb latency, and the chosen candidate’s provider overrides the base provider for that request only.

Judge

The Judge is an LLM-based (or OpenAI-Moderation-based) pre-check that moderates or blocks a request before the main model is called. It runs in parallel with the Decider on a buffered request. On the streaming path, and before a Comparer or Summariser fan-out, the Judge completes first. If the Judge blocks a request, the LLM call is skipped and the request is rejected with a canonical rejection category.

Jury and Juror

A Jury is a multi-juror LLM post-review that can approve, block, or re-drive (regenerate) the model’s response before it returns. Each Juror is an LLM evaluation with its own model, prompt template, and decision rule; retry (redrive) is configured once at the Jury level and applies to the whole panel, not per Juror. The Jury’s execution mode determines how jurors combine into a final verdict: Parallel (every juror must approve), Serial (jurors run in sequence and stop at the first rejection), or Majority (more than half must approve).

What each stage does

How Direct, Decider, Comparer, and Summariser routing modes differ

The routing stage is governed by a single Mode setting on the surface’s LLM element. The dashboard’s Mode dropdown offers LLM Direct, Decide LLM based on content, Compare output from multiple LLMs, and Summarise output from multiple LLMs. On the canvas, the other three options appear as Decider, Comparer, and Summariser. LLM Direct (Direct) has no separate canvas node of its own: the canvas simply shows the plain LLM node.

Choosing Decider, Comparer, or Summariser also relabels the canvas’s routing node to match and fans it out to each configured candidate model:

The Decider’s own fullscreen panel is where you point it at a router model and write the routing instructions it uses to pick a candidate. Judge and Jury have their own fullscreen panels too, covered in Guardrails → rather than repeated here.

• Direct is a straight pass-through to the one model you configure on the surface, provider, endpoint, and model. No routing decision is made, so this mode adds nothing to the pipeline beyond the provider call itself.
• Decider uses a small router model that reads the incoming request and selects exactly one candidate to answer it, for example sending simple requests to a cheaper model and complex ones to a frontier model. A configured default candidate serves as fallback when the router’s own choice is unknown or empty.
• Comparer fans the request out to every candidate model concurrently and returns every response to the caller side by side, for a person to compare and choose from manually. Because there is no single response for the Jury to review, the Jury stage does not run in Comparer mode. Response Prompt Guard rules and NER ID restoration apply to each candidate’s answer, and a response guard rejection or a response Expert Witness flag on any candidate’s answer blocks the whole response.
• Summariser also fans the request out to every candidate concurrently, but the individual candidate answers are never returned to the caller. Instead, the Merger LLM set on the Decider panel consolidates them into one answer, guided by its Merge instructions. That consolidated answer then continues through the pipeline exactly like any other single response: the Jury stage reviews it, and response guardrails and NER ID restoration apply to the merged answer, not to each candidate’s raw output.

What Comparer and Summariser have in common, because both call every candidate on every request:

• Every candidate is billed and metered independently. Cost scales with the number of candidates, and usage is recorded per candidate so you can see exactly where spend went.
• Latency reflects the slowest candidate, and for Summariser, the Merger LLM’s own pass afterwards, rather than the fastest response.
• Streaming is unavailable. Because every candidate’s response, and for Summariser the merge itself, must be gathered before anything can be returned, a request with "stream": true against a Comparer or Summariser surface is served as one complete, buffered response instead of a token stream.
• A configured maximum bounds the fan-out. A surface can fan out to a bounded number of candidate models per request, up to ten, so cost and load stay predictable; the surface builder prevents adding more than the limit.
• Failure is best-effort, not all-or-nothing. If some candidates fail, the surface continues with the ones that succeeded, Comparer shows the successes and flags the failures, Summariser consolidates the successful answers, and the request only fails outright if every candidate fails.
• Request-side guardrails run once, before fan-out, so every candidate model sees the same protected input rather than each candidate re-triggering its own guardrail pass.
• Governance records capture the whole multi-model interaction, including each candidate’s contribution and the routing or consolidation decision, so a Comparer or Summariser run is as fully auditable as a single-model one. See Governance records →.

Why the order matters

Cheap, deterministic checks run before expensive ones. Source auth, inbound policy, and the Team Gate reject unauthorised or over-quota callers before any guardrail or provider cost is incurred. The surface’s own rate limit is checked later, after the cache lookup and before Prompt Guard. The cache lookup runs before the request ever reaches a guardrail or a provider, so a cache hit costs nothing beyond the lookup itself and still counts against team quotas. In Decider mode on a buffered request, the routing stage and Judge run in parallel because both would otherwise add a full round-trip of latency before the real provider call. In Comparer or Summariser mode, the Judge completes before the fan-out begins, so its check still lands before the larger cost of calling every candidate.

Streaming is a shorter path

When a client requests "stream": true and the surface is eligible, the upstream Server-Sent-Events byte stream is proxied back to the caller unbuffered, and usage is metered from the terminal usage chunk instead of a buffered response body. Streaming still runs every request-side stage and still emits the request-time observability headers.

A surface is streaming-eligible when its base provider uses the OpenAI, Anthropic, or Microsoft 365 Copilot wire format. Streaming works with no routing stage, with a Judge, or with a Decider in Decide LLM based on content mode that has at least one candidate. On the streaming path the Judge completes before the Decider runs. The surface answers with a buffered response, rather than rejecting the request, when any of the following applies:

• Streaming is turned off for the surface.
• A Jury is enabled.
• Mode is Compare output from multiple LLMs or Summarise output from multiple LLMs.
• The surface has response-side Prompt Guard rules or a response-side Expert Witness.
• Governed server tools are active.

The response cache is checked before the streaming path, so a streamed request can be served from the cache. A streamed answer is written to the cache only when the request offers no tools. See Routing modes above for the full behaviour of each mode.

Related

• Surfaces: What a surface is and what it can compose.
• Guardrails: The request- and response-side safety stages in detail.
• Cost and usage governance: How the meter stage turns into budgets and alerts.
• OPA policies: The inbound policy evaluation this pipeline’s gate stage runs.
• Resilience and caching: The cache lookup and failover behind the provider-call stage.