Frequently asked questions
Providers & routing
Which LLM providers does Agent Stream support?
An LLM Surface reaches OpenAI, Anthropic, Google Gemini, AWS Bedrock, and Ollama, plus a set of OpenAI-compatible providers: Groq, xAI (Grok), OpenRouter, Fireworks AI, Cerebras, DeepSeek, Moonshot AI (Kimi), SambaNova Cloud, Hyperbolic, NVIDIA NIM, and Nebius AI Studio. Microsoft 365 Copilot is supported as its own provider type. Azure OpenAI is listed in the provider catalogue but isn’t available on this appliance. To use a provider outside the built-in catalogue, an administrator defines it as JSON on the Models page’s JSON tab, where it then appears under Custom, and you pick it as the provider on the surface.
Can I use a provider that isn't in the built-in list?
Yes. An administrator defines the provider as JSON on the Models page’s JSON tab, then you pick it as the provider when you create or edit an LLM Surface from the dashboard, with no code change or redeploy. Agent Stream still normalises your application’s single request shape for that target, aliasing parameters and transforming the wire format for a non-OpenAI-schema provider such as Anthropic Messages or Gemini generateContent. Provider pricing, capabilities, and request profiles come from a hot-reloadable catalogue, so a new model becomes available the same way, through the dashboard, without shipping code.
What happens if a provider is slow, rate-limited, or down?
A surface can fail over to ordered backup providers, or load balance across a pool by weighted round-robin, lowest observed latency, or lowest price. A per-minute rate limit applies independently of routing. Failover is triggered by provider and network errors, so a request a guardrail blocks is not sent to a backup provider.
Does Agent Stream cache responses to save cost and latency?
Yes. A surface can serve a repeat or near-duplicate prompt from a cache instead of calling the provider, in exact or semantic (embedding-similarity) mode. A cache hit still counts against team quotas but at zero provider spend. A streamed request can also be served from the cache, and a streamed answer is written to the cache only when the request offers no tools.
Can I roll out a new model or provider to only some traffic first?
Yes, through surface variants. A routing rule on the surface’s Router element can send traffic to a variant by header, body field, or attachment type, or a weighted canary split can send a small percentage of traffic to a new variant while you watch its cost, latency, and rejection figures before promoting it to everyone. An explicit $alias in the route always takes precedence over both.
API coverage & streaming
Which API endpoints does Agent Stream support?
OpenAI-compatible providers support buffered and streaming chat completions through /v1/chat/completions. The same governed Surface can also serve embeddings, moderations, reranking, image generation, transcription, translation, batches, and files without separate endpoint configuration. Text-to-speech uses a binary passthrough that still applies request-side gates. GET /v1/models is answered by Agent Stream and lists the models exposed by the Surface, so model discovery does not require a provider round trip.
Non-OpenAI-schema providers, such as Anthropic Messages, Gemini generateContent, and Microsoft 365 Copilot Chat, are supported for chat requests through the appropriate wire-format transformation.
By default, callers can also use the Anthropic Messages (/v1/messages) or OpenAI Responses (/v1/responses) format, whatever the surface’s provider, as set in the surface’s Client API formats section. Audio and images in a chat request are accepted only on a surface with the matching Audio File or Image File element.
What happens when a client requests a streaming response?
When the Surface is eligible, stream: true proxies the upstream Server-Sent Events stream to the caller without buffering. Request-time trace, provider, and model headers are preserved, and usage is metered from the terminal usage chunk. Streaming passthrough needs a base provider whose wire format is OpenAI, Anthropic, or Microsoft 365 Copilot. It works with no routing stage, with a Judge, or with a Decider in Decide LLM based on content mode that has at least one candidate.
Streaming is not used when:
- Allow SSE streaming passthrough is turned off for the surface.
- A Jury is enabled.
- The Mode is Compare output from multiple LLMs or Summarise output from multiple LLMs.
- The surface has response-side Prompt Guard rules or a response-side Expert Witness.
- Governed server tools are active.
In those cases the surface answers with a buffered response rather than rejecting the request.
Safety & guardrails
What stops an unsafe prompt or response from reaching a model or a caller?
Four layered stages: Prompt Guard (pattern and AI-matched PII detection), Expert Witnesses (external safety classifiers on either seam), Judge (a request-side pre-check that runs before the main model call, in parallel with the Decider on a buffered request in Decide LLM based on content mode), and Jury (a multi-juror response review that can approve, block, or regenerate an answer). A surface can run just Prompt Guard for a low-risk internal tool, or all four for a customer-facing deployment.
What happens if a guardrail provider itself is unreachable?
Two guardrails have a fail-open or fail-closed choice, and both default to fail closed: each Expert Witness, and the AI Matching detector in Prompt Guard. When one of them fails closed and its check can’t run, the request is blocked rather than allowed through with protection silently disabled. Either one can be set to fail open where the risk of a false block outweighs the risk of an unchecked call, but the default favours safety over availability. Other guardrails have no fail-open setting.
Which third-party safety providers can I plug in as Expert Witnesses?
Built-in providers include EnkryptAI, Lakera Guard, Azure Prompt Shields for request-side injection detection, Azure Content Safety for response-side harm moderation, and a generic OpenAI-moderation shape.
Can I see why a specific request was blocked?
Every guardrail block is classified into a stable rejection taxonomy, covering categories such as hate, toxicity, prompt injection, jailbreak, PII, secrets, and policy violation, with Uncategorized as the fallback, and charted per team, member, surface, and stage. This answers why requests were blocked, not just how many.
Privacy & PII protection
How does Agent Stream stop personal data in a prompt from reaching a model?
Prompt Guard combines deterministic pattern matching (emails, card numbers, phone numbers, custom patterns) with a model-backed AI detector for free-form PII such as names and addresses. The When detected setting chooses what happens to a match: Reject the call completely, Redact the text that matches the pattern (replacing it with a fixed token such as <EMAIL>), or Use a NER ID to preserve context (replacing it with a stable pseudonym such as [EMAIL_NER_001]).
What's the difference between redaction and a NER ID?
Redaction replaces every match of a type with the same fixed token, such as <EMAIL>, so two different people’s details become indistinguishable and the model can no longer tell whose address belongs to whose email. A NER ID gives each distinct value its own stable pseudonym instead, so the model can still reason over the correct pairings without ever seeing the real values.
Is the model's response also scanned for PII, and can the real values be restored for the caller?
Yes, though request and response are not symmetric: the response seam only runs pattern matching, since AI matching is a request-side-only detector. If restore-in-response is enabled, real values are swapped back into the answer after every model-facing stage, including the message text of streamed responses.
Is my data sent anywhere to run the AI-backed PII detector?
You point the AI detector at a model you control, ideally one you host yourself. The endpoint is SSRF-checked the same way as any other operator-supplied endpoint. Input is bounded so a large prompt cannot run up cost or latency. The inspected text is wrapped in an injection-resistant frame, and no raw values are persisted beyond the life of the request.
Does linking pronouns and aliases to the same NER ID cost more?
Deterministic pronoun linking is free. AI coreference linking, which collapses aliases like “Bob” and “Robert Smith” to one shared NER ID, has a real cost: it forces the detector cache to work at whole-prompt granularity instead of per-message, so every turn re-scans the entire, ever-growing conversation. Leave it off unless the value of linking aliases outweighs that cost.
Security & access control
How does Agent Stream authenticate callers?
Each surface authenticates with a JWT bearer token, an API key, or mTLS. With mTLS, the surface pins the caller’s client certificate and identifies the caller by its fingerprint.
Can an organisation use anonymous access or bind mTLS through a proxy?
Yes. When team attribution is on for a Surface, a caller that can’t be matched to a member is attributed to the built-in anonymous member and team, so quotas and rate limits still apply.
An mTLS surface can take the client certificate from a direct TLS handshake, or from a trusted proxy that forwards it. Settings → Networking shows which of these the appliance accepts and lists its trusted proxies.
How are OAuth sign-in and upstream credentials separated?
OAuth Identifiers store reusable public OAuth 2.0 and OpenID Connect sign-in details, such as the authority, client ID, and scopes, without storing secrets. A Surface can use one identifier for front-door sign-in and another for an upstream provider. In a dual-token flow, the client sends the upstream credential separately, allowing policy to require it for selected models without confusing it with the caller’s sign-in identity.
Who can administer Agent Stream itself?
An ordered role ladder: Administrator, Power User, and User.
- Surfaces: creating and editing LLM and IDE Surfaces defaults to the User role.
- Other management permissions: most default to Administrator, including:
- Users, teams, and members.
- Settings, secrets, API keys, and access tokens.
- Policies, including publishing a global policy.
- Prompt editing, setting a surface’s price, and editing the model catalogue.
- Terms and conditions, the audit and governance views, and LLM Drift.
- Metrics: any signed-in account sees aggregate figures, while a per-member or per-team breakdown needs Administrator access.
Where are provider API keys and other credentials stored?
In an encrypted secrets store, referenced by ID from a surface, never embedded directly in application code or surface configuration.
Does policy evaluation require an external OPA service?
No. Agent Stream evaluates policy in-process, so a decision adds no network round trip and there is no separate service to keep available. For an LLM Surface request, the appliance-wide (global) policies for that surface type evaluate first, then the surface’s own OPA policy, which only affects that surface. Both are deny-overrides, so a deny from either one blocks the request, and both fail closed. The exception is a global policy set to Monitor only, which records its decision but never blocks.
What context can an Agent Stream policy evaluate?
For an LLM Surface request, policy input includes:
- The authenticated caller: the JWT subject and claims, the API key name, or the client certificate details.
- The request path and non-sensitive headers.
- The Surface’s identifier and name, the request time, and the action.
- The request target: the Surface type, the requested model, active Web Search or Web Fetch tools, and the types of any attached documents, images, or audio.
When VP evidence is turned on for the Surface and the caller presents a verifiable presentation, identity_binding adds its holder and issuer DIDs, a verified flag, and the credential’s claims.
IDE Surfaces distinguish model discovery from dispatch, so a policy can allow GET /v1/models while requiring authentication for POST /v1/chat/completions.
Policy definitions are versioned immutably and referenced by content hash, allowing an audit record to identify the exact policy bytes that ran.
What stops a malicious or misconfigured custom endpoint from reaching internal infrastructure?
Every operator-supplied endpoint, such as a custom provider URL, is validated against cloud-metadata addresses and other unsafe targets, both when it is first entered and before every outbound call.
Cost & attribution
Can I cap spend before it happens, rather than review it afterwards?
Yes. A surface can enforce a monthly USD budget and a monthly token cap, at the surface level, per stage, or even per individual juror. Once spend or tokens have reached a configured cap, further requests are blocked with a 429 response before they reach the provider. The request that crosses the cap is itself allowed through.
How do I know which team or caller is driving cost?
On a surface with team attribution on, each caller is matched to a registered member, and usage is attributed to the member and its teams, aggregated across every surface the member calls. A caller that can’t be matched is attributed to the built-in anonymous member and team.
Does Agent Stream double-count a provider's own prompt-caching discount?
No. Provider-side cached-token figures, such as OpenAI’s cached_tokens, Anthropic’s cache_read_input_tokens, and DeepSeek’s prompt_cache_hit_tokens, are surfaced explicitly and kept distinct from Agent Stream’s own response cache, so a provider’s cost saving is never counted as if it were Agent Stream’s.
Can I see cost per stage and receive cost alerts?
Yes. Cost, token usage, and latency are counted per stage for the LLM, Judge, Jury, and Decider stages. Threshold and anomaly alerts can notify configured integrations about cost, token, rolling error-rate, or cost-spike conditions.
Governance, audit & compliance
Can I prove exactly what a model received and returned for a specific interaction?
Yes, through governance records: a signed W3C Verifiable Credential produced for each recorded interaction, capturing the prompt, every processing stage’s verdict, the completion, and per-interaction metrics. Recording is switched on per surface with Record signed governance records, which is off on a new surface. Its Record setting chooses Every interaction, Blocked interactions only, or Blocked and failed interactions, and streamed interactions are recorded while Record streamed interactions is on.
Are governance records tamper-proof?
Trust rests on two independent mechanisms. Each record is signed with the appliance’s own key, tied to its Decentralised Identifier, and each record also embeds the content hash of the previous record on the same surface. Deleting or reordering a record breaks that chain, so gaps and tampering are both detectable, not just individual forged records. Records removed by the surface’s retention window are replaced by a signed chain anchor, so the remaining chain still verifies. A surface’s chain can be verified on demand from the dashboard.
Do governance records store the raw prompt and completion text?
By default, yes. When recording is on, Keep prompt & completion bodies defaults to For every recorded interaction. Set it to For blocked interactions only or Never (digests only) to keep less. With Never (digests only), the appliance keeps cryptographic digests, fingerprints that still prove what content flowed without storing the raw text. Records are encrypted at rest. Reading a record needs its own dedicated permission, which defaults to Administrator and is separate from general log access.
Configuration, deployment & extending access
How do developers use approved models from their editor without holding a provider key?
Through an IDE Surface: a governed, per-caller-filtered, OpenAI-compatible model menu that aggregates one or more LLM Surfaces behind a single Access Point URL. The client presents only the developer’s own SSO token; the IDE Surface resolves that identity to a member and dispatches accordingly, with no provider key ever held in the editor.
Can Agent Stream ground answers on our own SharePoint or OneDrive content without exposing the documents to a model?
Yes, through the Microsoft 365 Copilot provider adapter. Retrieval and grounding happen inside Microsoft’s own Copilot boundary, under the signed-in user’s existing access control lists, and only the synthesised answer text crosses back to the caller. The editor, agent framework, and any other surface’s model never receive the raw documents.
How do I monitor Agent Stream, and can I export telemetry to tools I already use?
A built-in dashboard shows cost, tokens, latency, and throughput per surface, and every buffered response carries x-agent-stream-* headers with trace ID, provider, model, tokens, and cost. OpenTelemetry (OTLP) exports traces, metrics, and audit logs, a Prometheus scrape endpoint serves metrics, and webhook integrations deliver events.
Can separate Agent Stream instances, or Agent Stream and Agent Gateway, talk to each other?
Yes, over the Fabric. An appliance connects to a peer through a Connection Point over DIDComm v2.1, with a mediator where the two can’t connect directly, the same model used elsewhere on the Affinidi Trust Fabric. Agent Stream uses these gateway-to-gateway (G2G) connections to browse and buy models from a connected peer’s model marketplace, settled through the x402 payments protocol. See Agent Gateway for the equivalent gateway-to-gateway model on agent traffic.
Where does Agent Stream run?
Agent Stream is an appliance hosted by Affinidi. You configure and operate it from its browser-based dashboard.
How does Agent Stream protect outbound network connections?
The appliance accepts inbound HTTPS connections at a Surface Access Point and reaches providers and integrations over outbound HTTPS. Operator-supplied endpoints are checked against cloud-metadata, loopback, and other unsafe targets when entered and before each send, including DNS-resolution checks.
Glad to hear it! Please tell us how we can improve more.
Sorry to hear that. Please tell us how we can improve.
Thank you for sharing your feedback so we can improve your experience.