Telemetry and exports
This is the field-level detail behind Observability →: how to send Agent Stream’s telemetry to your own observability stack, and how to read the usage and rejection data behind the dashboard’s own charts.
Exporting to your own stack
Send Agent Stream’s telemetry into an observability stack you already run, instead of relying on the dashboard’s own charts alone. Doing so lets you correlate its traces, metrics, and audit logs with the rest of your systems, in the tools your team already watches. Each channel carries a different signal:
| Channel | Carries | Reach for it when |
|---|---|---|
| OpenTelemetry (OTLP) | Traces, metrics, and audit logs: policy decisions, governance records, and credential delegation records | You already run an OTLP-compatible collector, such as Grafana, Datadog, or Honeycomb. |
| Prometheus | Metrics | A Prometheus server you already operate should scrape the appliance. |
| Webhook | Events | You forward events into your own system, such as a ticketing tool, an internal pipeline, or a SIEM. |
The channels are independent, so you can use more than one at a time.
OpenTelemetry (OTLP)
Send traces, metrics, and audit logs to any OTLP-compatible collector. Configure it under Monitoring → Metrics, on the OpenTelemetry tab of the Metrics Configuration page. The other settings appear once Enable OpenTelemetry is on.
- Set Transport Protocol to whichever your collector is listening on: OTLP / gRPC (default port 4317) or OTLP / HTTP (default port 4318).
- Turn on Authenticate to the collector when your collector requires a credential.
- 100% is a reasonable Sample Rate to start with on a low-traffic appliance. Lower it if trace volume becomes a cost or storage concern on your collector.
- A shorter Export Interval (seconds) gives more responsive dashboards on the collector’s side, at the cost of more export traffic.
- A governance record reaches your collector only when the surface’s OpenTelemetry exporter switch (in its Governance recording section) and Enable Log Export are both on.
| Field | Default | Description |
|---|---|---|
| Enable OpenTelemetry | Off | Master checkbox for OTLP export. |
| OTLP Endpoint (gRPC / HTTP) | — | OTLP collector endpoint (host:port or URL, interpreted per Transport Protocol). The dashboard label switches between “OTLP Endpoint (gRPC)” and “OTLP Endpoint (HTTP)” to match the selected transport. For HTTP, the signal path is appended automatically. Test Connection checks that the endpoint is reachable. |
| Transport Protocol | OTLP / gRPC | Dropdown: OTLP / gRPC or OTLP / HTTP. |
| Authenticate to the collector | Off | Checkbox that adds an authentication header to every export, set in its own sub-form (see below). |
| Service Name | — | Service name attached to every trace, metric, and log record. Use a name that identifies this appliance in your collector’s service list. |
| Environment | — | Deployment environment label (e.g. production, staging); use whatever convention your existing observability stack already applies to other services. |
| Enable Trace Export | — | Checkbox for trace export. |
| Sample Rate | 100% | Trace sampling rate, shown as a percentage slider (“Sample Rate: N%”). Shown when trace export is on. |
| Enable Metrics Export | — | Checkbox for metrics export. |
| Export Interval (seconds) | — | How often metrics are sent to the collector. Shown when metrics export is on. |
| Enable Log Export | — | Checkbox for audit log export: policy decisions, governance records, and credential delegation records. |
Settings with no listed default open with your appliance’s current values.
The collector credential is read from a secret rather than typed in:
| Field | Default | Description |
|---|---|---|
| Header Name | Not set | Header sent to the collector, e.g. Authorization or X-Api-Key, depending on what your collector expects. |
| Secret | Not set | The secret holding the credential value, chosen from a secret picker labelled “Secret”. Create it first on the Secrets page using the token or key your collector issued you. |
| Header Value Format | {value} (the secret’s value, unmodified) | Format string interpolated with the resolved secret; use Bearer {value} for bearer-token auth. |
Prometheus
The appliance exposes its metrics for scraping at /v1/metrics/prometheus, covering request counts and latency, GenAI token and cost usage, policy decisions, and live gauges such as active connections and throughput. Prometheus itself holds the history. Point your own Prometheus server’s scrape config at that path to start collecting.
The Prometheus Endpoint Authentication card, also on the OpenTelemetry tab, requires scrapers to send HTTP Basic credentials. Turn it on for any appliance reachable from outside a trusted network.
| Field | Default | Description |
|---|---|---|
| Prometheus Endpoint Authentication | Disabled | Switch that requires credentials on the scrape endpoint. |
| Username | — | Username your Prometheus server sends. |
| Password | — | Password your Prometheus server sends. Leave it empty when editing to keep the current password. Select Save Credentials to apply. |
Webhook
A webhook is configured as an integration of type Webhook:
| Field | Default | Description |
|---|---|---|
| Webhook URL | Not set | Destination URL. It must be a public address: private, link-local, and localhost addresses are rejected. Use the ingestion endpoint your receiving system provides. |
The webhook integration also supports request signing (Signing Secret), Custom Headers, and a Payload Template, resolved with the same runtime variables used across every integration type. See Notifications and alerts → for the full integration shape and the other delivery types (Slack, email, streaming platforms).
Reading the usage and rejection charts
Usage time ranges
The dashboard’s usage charts (spend, tokens, request counts) can plot any window from now back to roughly five years, because Agent Stream keeps this data at several fixed resolutions rather than one raw log. Zooming out to a longer window doesn’t lose accuracy: a coarser point on the chart is always the exact sum of the finer points it covers, so the numbers stay consistent no matter which time range you pick.
The finest detail, 30-second resolution, covers the last two hours. Minute-level detail covers the last six hours, 5-minute detail the last day, and resolution grows coarser the further back you look, out to about five years of history. Deleting a surface, member, or team also removes its historical usage data.
Rejection categories
When a request is rejected (by a guardrail, Judge, or Jury stage), the dashboard groups the reason into one of a small, stable set of categories, regardless of which underlying detector produced it, so counts stay comparable across providers and over time:
Prompt Injection · Jailbreak · Toxicity · Hate Speech · Violence · Sexual / NSFW · Self-Harm · PII · Secrets / Credentials · Bias · Policy Violation · Banned Keyword · Off-Topic / Relevancy · Copyright / IP · System Prompt Leak · Sponge / DoS · Hallucination · Redrive (recovered) · Uncategorized
Redrive (recovered) means a Judge/Jury rejection was later recovered by a redrive and the request ultimately succeeded. It’s recorded distinctly from the real content category so the two aren’t conflated. Uncategorized is the fallback when nothing else matches.
Related
- Observability concept: What gets recorded and why, in plain terms.
- Response headers: The per-request
x-agent-stream-*headers derived from the same execution this telemetry captures. - Notifications and alerts: The integration record shape and delivery-type configuration referenced above.
- Cost and usage limits: The budgets and alerts built on this same usage telemetry.
- Guardrails: The Prompt Guard, Judge, and Jury stages whose rejections populate these categories.
Glad to hear it. Thanks for letting us know.
Sorry to hear that. Thanks for letting us know.