Frequently asked questions
Providers & routing
Which LLM providers does Agent Stream support?
An LLM Surface reaches OpenAI, Anthropic, Google Gemini, AWS Bedrock, Azure OpenAI, Cohere, Mistral, Together AI, Replicate, Hugging Face, and Ollama, plus a set of OpenAI-compatible presets: Groq, xAI, OpenRouter, Fireworks, Cerebras, DeepSeek, Moonshot, SambaNova, Hyperbolic, NVIDIA NIM, and Nebius. Microsoft 365 Copilot is supported as its own provider type, and any custom OpenAI-compatible endpoint can be added directly.
Can I use a provider that isn't in the built-in list?
Yes. When you create or edit an LLM Surface from the dashboard, set its provider type to a custom OpenAI-compatible endpoint and supply the base URL and credentials, no code change or redeploy required. Agent Stream still normalises your application’s single request shape for that target, aliasing parameters and transforming the wire format for a non-OpenAI-schema provider such as Anthropic Messages or Gemini generateContent. Provider pricing, capabilities, and request profiles come from a hot-reloadable catalogue, so a new model becomes available the same way, through the dashboard, without shipping code.
What happens if a provider is slow, rate-limited, or down?
A surface can fail over to ordered backup providers, or load balance across a pool by weighted round-robin, lowest observed latency, or lowest catalogue cost. Retry with backoff, a circuit breaker, and token-bucket rate limiting all apply independently of routing. None of this retries a request a guardrail already blocked, since retrying would defeat the guardrail rather than recover from a transient failure.
Does Agent Stream cache responses to save cost and latency?
Yes. A surface can serve a repeat or near-duplicate prompt from an in-memory cache instead of calling the provider, in exact (SHA-256 hash) or semantic (embedding-similarity) mode. A cache hit still counts against team quotas but at zero provider spend. Streaming responses are not cache-eligible, since there is no token stream to replay from a cached entry.
Can I roll out a new model or provider to only some traffic first?
Yes, through surface variants. A routing rule can tier traffic by header or body field, or a weighted canary split can send a small percentage of traffic to a new variant while you watch its cost, latency, and rejection figures before promoting it to everyone. An explicit $alias in the route always takes precedence over both.
API coverage & streaming
Which API endpoints does Agent Stream support?
OpenAI-compatible providers support buffered and streaming chat completions through /v1/chat/completions. The same governed Surface can also serve embeddings, moderations, reranking, image generation, transcription, translation, batches, and files without separate endpoint configuration. Text-to-speech uses a binary passthrough that still applies request-side gates. GET /v1/models is answered by Agent Stream and lists the models exposed by the Surface, so model discovery does not require a provider round trip.
Non-OpenAI-schema providers, such as Anthropic Messages, Gemini generateContent, and Microsoft 365 Copilot Chat, are supported for chat requests through the appropriate wire-format transformation.
What happens when a client requests a streaming response?
When the Surface is eligible, stream: true proxies the upstream Server-Sent Events stream to the caller without buffering. Request-time trace, provider, and model headers are preserved, and usage is metered from the terminal usage chunk. Streaming is available for OpenAI-schema base providers and bypasses response-side review stages such as the Jury, so the eligibility gate rejects combinations that cannot be governed safely.
Safety & guardrails
What stops an unsafe prompt or response from reaching a model or a caller?
Four layered stages: Prompt Guard (pattern and AI-matched PII detection), Expert Witnesses (external safety classifiers on either seam), Judge (a request-side pre-check that runs in parallel with routing), and Jury (a multi-juror response review that can approve, block, or regenerate an answer). A surface can run just Prompt Guard for a low-risk internal tool, or all four for a customer-facing deployment.
What happens if a guardrail provider itself is unreachable?
Guardrails fail closed by default: if a guardrail provider errors or times out, the request is blocked rather than allowed through with protection silently disabled. This can be set to fail open per surface where the risk of a false block outweighs the risk of an unchecked call, but the default favours safety over availability.
Which third-party safety providers can I plug in as Expert Witnesses?
Built-in providers include EnkryptAI, Lakera Guard, Azure Prompt Shields for request-side injection detection, Azure Content Safety for response-side harm moderation, and a generic OpenAI-moderation shape. A new provider onboards through a data-driven profile that declares its auth header, request template, and verdict rule, rather than through a code change.
Can I see why a specific request was blocked?
Every guardrail block is classified into a stable rejection taxonomy, covering categories such as hate, toxicity, prompt injection, jailbreak, PII, secrets, and policy violation, and charted per team, member, surface, and stage. This answers why requests were blocked, not just how many.
Privacy & PII protection
How does Agent Stream stop personal data in a prompt from reaching a model?
Prompt Guard combines deterministic pattern matching (emails, card numbers, phone numbers, custom patterns) with a model-backed AI detector for free-form PII such as names and addresses. For each match, you choose to reject the call, redact it to a static placeholder, or mask it as a stable NER ID pseudonym.
What's the difference between redaction and a NER ID?
Static redaction replaces every match with the same generic placeholder, so two different people’s details become indistinguishable and the model can no longer tell whose address belongs to whose email. A NER ID gives each distinct value its own stable pseudonym instead, so the model can still reason over the correct pairings without ever seeing the real values.
Is the model's response also scanned for PII, and can the real values be restored for the caller?
Yes, though request and response are not symmetric: the response seam only runs pattern matching, since AI matching is a request-side-only detector. If restore-in-response is enabled, real values are swapped back into the answer after every model-facing stage, including a chunk-boundary-aware restoration path for streamed responses.
Is my data sent anywhere to run the AI-backed PII detector?
You point the AI detector at a model you control, ideally self-hosted. The endpoint is SSRF-checked the same way as any other operator-supplied endpoint. Input is bounded so a large prompt cannot run up cost or latency. The inspected text is wrapped in an injection-resistant frame, and no raw values are persisted beyond the life of the request.
Does linking pronouns and aliases to the same NER ID cost more?
Deterministic pronoun linking is free. AI coreference linking, which collapses aliases like “Bob” and “Robert Smith” to one shared NER ID, has a real cost: it forces the detector cache to work at whole-prompt granularity instead of per-message, so every turn re-scans the entire, ever-growing conversation. Leave it off unless the value of linking aliases outweighs that cost.
Security & access control
How does Agent Stream authenticate callers?
Each surface authenticates with a JWT bearer token, an API key, or mTLS. mTLS supports pinned certificates or CA-validated chains, with identity bound to a fingerprint, subject name, or a DNS, URI, or IP subject alternative name, including SPIFFE identifiers. Managed API keys can carry an expiry, and rotating a key preserves its remaining lifetime rather than resetting it.
Can an organisation use anonymous access or bind mTLS through a proxy?
Yes, when explicitly enabled, unauthenticated traffic is attributed to a built-in anonymous member and team so quotas and rate limits still apply. mTLS identity can bind to a certificate fingerprint, subject common name, DNS, URI, or IP subject alternative name, including SPIFFE identifiers. Both direct TLS handshakes and forwarded client-certificate information from a trusted proxy are supported.
How are OAuth sign-in and upstream credentials separated?
OAuth Identifiers store reusable public OAuth 2.0 and OpenID Connect sign-in details, such as the authority, client ID, and scopes, without storing secrets. A Surface can use one identifier for front-door sign-in and another for an upstream provider. In a dual-token flow, the client sends the upstream credential separately, allowing policy to require it for selected models without confusing it with the caller’s sign-in identity.
Who can administer Agent Stream itself?
An ordered role ladder: Administrator, Power User, and User. Mutating admin operations default to Administrator-only, and metrics access is split: any signed-in account sees aggregate figures, while a per-member or per-team breakdown needs Administrator-only access. A missing or incomplete access-control configuration fails toward the more restrictive setting.
Where are provider API keys and other credentials stored?
In an encrypted secrets store, referenced by ID from a surface, never embedded directly in application code or surface configuration. Local storage can be encrypted with AES-256-GCM, using either a local key or a per-value AWS KMS envelope.
Does policy evaluation require an external OPA service?
No. Agent Stream evaluates policy in-process, so a decision adds no network round trip and there is no separate service to keep available. Gateway-level policy evaluates first and a gateway-level deny is final; surface-level policy evaluates after and only affects that surface. This also means policy enforcement works the same way in an air-gapped deployment as in a fully connected one.
What context can an Agent Stream policy evaluate?
Policy input can include the authenticated caller and claims, the Member and Team identity binding, the appliance environment, the Surface, provider, model, tool, route, action, and relevant request or response data. IDE Surfaces distinguish model discovery from dispatch, so a policy can allow GET /v1/models while requiring authentication for POST /v1/chat/completions. Policy definitions are versioned immutably and referenced by content hash, allowing an audit record to identify the exact policy bytes that ran.
What stops a malicious or misconfigured custom endpoint from reaching internal infrastructure?
Every operator-supplied endpoint, such as a custom provider URL, is validated against cloud-metadata addresses and other unsafe targets, both when it is first entered and before every outbound call.
Cost & attribution
Can I cap spend before it happens, rather than review it afterwards?
Yes. A surface can enforce a monthly USD budget and a monthly token cap, at the surface level, per stage, or even per individual juror. A request that would exceed a configured cap is blocked with a 429 response before it reaches the provider, rather than being allowed through and billed afterwards.
How do I know which team or caller is driving cost?
Every caller is a registered member, grouped into a team, and usage is attributed to both, aggregated across every surface the member calls.
Does Agent Stream double-count a provider's own prompt-caching discount?
No. Provider-side cached-token figures, such as OpenAI’s cached_tokens, Anthropic’s cache_read_input_tokens, and DeepSeek’s prompt_cache_hit_tokens, are surfaced explicitly and kept distinct from Agent Stream’s own response cache, so a provider’s cost saving is never counted as if it were Agent Stream’s.
Can I export chargeback reports and receive cost alerts?
Yes. Cost, token usage, and latency are counted per stage, including the LLM, Judge, Jury, Decider, and configured Expert Witness stages. Threshold and anomaly alerts can notify configured integrations about cost, token, rolling error-rate, or cost-spike conditions. Chargeback reports can be exported as JSON or CSV for a date range and rolled up by Team, Member, and Surface.
Governance, audit & compliance
Can I prove exactly what a model received and returned for a specific interaction?
Yes, through governance records: a signed W3C Verifiable Credential produced for every interaction on a surface where recording is enabled, capturing the prompt, every processing stage’s verdict, the completion, and per-interaction metrics. Recording is off by default and switched on per surface.
Are governance records tamper-proof?
Trust rests on two independent mechanisms. Each record is signed with the appliance’s own key, tied to its Decentralised Identifier, and each record also embeds the content hash of the previous record on the same surface. Deleting or reordering a record breaks that chain, so gaps and tampering are both detectable, not just individual forged records. A surface’s chain can be verified on demand from the dashboard.
Do governance records store the raw prompt and completion text?
Only if you enable body capture on that surface. With it off, the appliance keeps cryptographic digests only, fingerprints that still prove what content flowed without storing the raw text. When body capture is on, every record is encrypted at rest, and reading a record is gated by its own dedicated permission, restricted to administrators by default and distinct from general log access.
Configuration, deployment & extending access
How do developers use approved models from their editor without holding a provider key?
Through an IDE Surface: a governed, per-caller-filtered, OpenAI-compatible model menu that aggregates one or more LLM Surfaces behind a single Access Point URL. The client presents only the developer’s own SSO token; the IDE Surface resolves that identity to a member and dispatches accordingly, with no provider key ever held in the editor.
Can Agent Stream ground answers on our own SharePoint or OneDrive content without exposing the documents to a model?
Yes, through the Microsoft 365 Copilot provider adapter. Retrieval and grounding happen inside Microsoft’s own Copilot boundary, under the signed-in user’s existing access control lists, and only the synthesised answer text crosses back to the caller. The editor, agent framework, and any other surface’s model never receive the raw documents.
How do I monitor Agent Stream, and can I export telemetry to tools I already use?
A built-in dashboard shows cost, tokens, latency, and throughput per surface, and every buffered response carries x-agent-stream-* headers with trace ID, provider, model, tokens, and cost. Traces, metrics, and logs also export through a native Langfuse connector, OpenTelemetry OTLP, a Prometheus scrape endpoint, or a content-capturing webhook.
Can separate Agent Stream deployments, or Agent Stream and Agent Gateway, talk to each other?
Yes. Instances federate over DIDComm v2.1 through connection points and mediators, the same model used elsewhere on the Affinidi Trust Fabric. Agent Stream also includes an embedded model marketplace for provisioning purchased models directly, and supports the x402 protocol for metered, pay-per-use settlement at the request boundary. See Agent Gateway for the equivalent gateway-to-gateway model on agent traffic.
Can Agent Stream run in a regulated or air-gapped environment?
Agent Stream can run as a managed appliance in the enterprise’s environment or as a self-hosted single Rust binary. The self-hosted deployment does not require an external database, cache, or broker. State is persisted locally and can be encrypted with a local key or an AWS Key Management Service-backed envelope. Most configuration changes reload without a restart, while listener bindings, TLS certificates, failover table names, and the identity hash pepper are startup settings.
How does high availability work?
Warm-standby failover uses an active-passive replica pair and a DynamoDB conditional lease so only one replica serves traffic. A standby reloads shared time-series snapshots when promoted. Connection-point generations and demotion fencing prevent stale tasks from reconnecting or removing replacements. Filesystem election is intended for explicit single-process local development and is not an availability mechanism.
How does Agent Stream protect outbound network connections?
The appliance accepts inbound HTTPS connections at a Surface Access Point and reaches providers and integrations over outbound HTTPS. Operator-supplied endpoints are checked against cloud-metadata, loopback, and other unsafe targets when entered and before each send, including DNS-resolution checks. Outbound connector requests bypass ambient HTTP proxy variables so an unchecked proxy cannot resolve a forbidden target independently.
Glad to hear it! Please tell us how we can improve more.
Sorry to hear that. Please tell us how we can improve.
Thank you for sharing your feedback so we can improve your experience.