Observability

The Agent Gateway produces three telemetry signals on every request: distributed traces, metrics, and structured logs. Each signal carries agent identity context. A policy denial or a latency spike can therefore be attributed to a specific surface, identity, or routing decision rather than appearing as an undifferentiated failure. All three signals are exportable to any OTLP-compatible backend.

Two planes of observability

The gateway provides telemetry at two independent levels. You can use either without configuring the other.

Dashboard tools

Dashboard overview

The Dashboard home screen provides a real-time, gateway-wide snapshot of connection state, latency distribution, identity resolution, and surface performance. It refreshes automatically on a timer and is the natural starting point when opening the gateway.

Four stat cards summarise current gateway activity. For field definitions, see Dashboard.

Below the stat cards, three time-series charts provide operational context:

• Connections over time: Requests split into accepted, denied, and faulted. Use the Bucket selector to zoom between 30-second and 30-minute windows. Selecting a bucket filters the stat cards, latency chart, identity resolution breakdown, and top surfaces table to that time slice. Selecting the same bucket again clears the filter.
• Latency over time: End-to-end response times from median through to the slowest few percent, measured in milliseconds. A spike in the upper percentiles often signals load on a specific surface or external service even when the median remains stable.
• Identity resolution: A breakdown of authentication methods active in live traffic: API key, DID, certificate, and request-content extraction. A sudden shift in this distribution indicates a new integration has gone live or an authentication method has changed.

The Top surfaces table lists the gateway’s busiest surfaces with their name, identifier, enabled state, live connection status, accept/deny ratio, and uptime. Selecting any row opens that surface’s configuration editor at the Monitoring tab, where the same connection and latency charts are scoped to that surface and accompanied by a live log stream.

Use the home screen to confirm that traffic is flowing normally, identify which surface is over-subscribed, and spot sudden changes in error rate or authentication distribution before drilling into a specific surface or log stream.

Logs

The Logs page streams a unified, live output from all surfaces and gateway system components. Each entry carries a severity level, a timestamp, and the message text. A keyword filter focuses the stream on a specific surface name, agent identity, or error string without interrupting the live feed.

Use the Logs page when you need to understand what the gateway did on a specific request: which authentication outcome it produced, which policy decision it made, and which target it routed to. The sequence of log entries shows the order of decisions, which is often faster for diagnosis than numeric charts alone.

Audit log

The Audit page provides a searchable, filterable record of significant gateway events: policy decisions (allow and deny), trust check outcomes, and credential delegation operations (consent granted, credential injected, credential refreshed, and revocation). Events are persisted to disk in daily-rotating JSONL files and are not capped in volume.

Each audit entry carries a timestamp, surface identifier, caller identity, protocol, and a trace ID that links back to the corresponding distributed trace and log entries. The page supports filtering by event category, decision outcome (allow or deny), request flow (access point, transit point, fabric), and free-text search.

Use the Audit page when you need a durable, queryable record of what the gateway authorised or denied: post-incident review, credential delegation tracing, and compliance reporting.

System health

The System page shows CPU usage and memory consumption for both the gateway process and its host machine, alongside process uptime.

Use the System page when the gateway is slow or unstable and you need to separate an infrastructure problem from a configuration or policy one. Sustained memory growth or high process CPU often explains latency that appears in surface charts.

Runtime tasks

The Tasks page lists every active listener and worker: inbound surface listeners, outbound routes, MCP proxy listeners, connection-point workers, and notification listeners. Each row shows connection state, in-flight request count, cumulative error count, and last activity time.

Use the Tasks page to confirm that a configured surface is actually listening. A surface that is configured but not running appears with a failed or stopped connection state and can be restarted directly from this page without modifying configuration.

Traffic and latency

The Metrics page shows a hierarchical view of request flow across all surfaces: source, caller identity, destination, request counts, acceptance and denial rates, and fault counts.

For per-surface detail, each surface has a Monitoring tab that shows time-series charts for request counts, rule accepts, rule denies, failures, gateway faults, and response latency at p50, p95, and p99.

Use the Metrics page to identify which surface is receiving the most traffic or producing the most faults. Use the surface Monitoring tab to investigate how a specific surface has behaved over a time window.

Request pipeline inspection

Each surface has a Capture feature, accessible from its Monitoring tab. Capture streams individual request payloads through each stage of the processing pipeline as they arrive. It is asynchronous and adds no latency to the request itself.

Four stages are visible per captured request:

• Inbound request: the payload as received from the source agent.
• Outbound request: the request forwarded to the backend, after identity injection and any gateway transformations.
• Inbound response: the raw response returned by the backend.
• Outbound response: the final response delivered to the caller.

Use Capture when a surface behaves unexpectedly and you need to see exactly what changed between what the caller sent and what the backend received. Comparing the inbound and outbound stages shows what identity fields the gateway injected, what metadata it added, and what the backend actually saw.

Connection tracking and audit

Every request is recorded with the following fields:

This data feeds the dashboard charts and powers log correlation via the trace ID. To investigate a specific request outcome, find its trace ID in the Logs page, then cross-reference it with the corresponding distributed trace or policy audit entry.

Multiple backend export targets

Metrics and traces can simultaneously persist to four independent targets, supporting diverse deployment patterns:

1. Local files (JSON format): Used to populate the Agent Gateway’s in-dashboard views and provide immediate point-in-time visibility of gateway state.
2. Prometheus/Grafana: For real-time dashboards, alerting rules, and metric-based SLOs in on-premises or hybrid environments.
3. Cloud loggers: Direct integration with AWS CloudWatch, Google Cloud Logging, Azure Monitor, and similar services for centralized cloud environments.
4. OpenTelemetry (OTLP) collectors: Generic integration point supporting Jaeger, Grafana Tempo, Datadog, New Relic, Splunk, and any OTLP-compatible backend.

Real-time streaming

The gateway streams live updates to the dashboard over WebSocket, with no polling required. This enables:

• Live task monitoring: the Tasks page updates in real time as listeners start, stop, or transition between states.
• Live log streaming: new log entries appear immediately in the Logs page as they are produced, with no refresh required.
• Live metrics updates: dashboard charts and stat cards refresh automatically without manual page reload.

Structured logging

Every request gets a unique trace ID that links it across all telemetry signals:

• Injected into outbound requests as the X-Gateway-Trace-Id header.
• Included in all gateway log entries for that request.
• Carried through the distributed trace in OTLP export.

This enables deterministic correlation: a single trace ID allows you to find all logs, metrics, and spans belonging to a single request across the gateway’s local logs, external log aggregators, and OTLP backends.

Exported telemetry signals

Distributed traces

The gateway creates a root span for every A2A, AP2, and MCP request. Each span carries routing context, outcome status, latency, and agent identity context when identity is enabled.

The gateway propagates the W3C traceparent header, so gateway spans connect to the upstream caller’s trace and to any downstream service that also participates in distributed tracing. This produces a single end-to-end trace tree across the full request path, visible in Jaeger, Grafana Tempo, or another OTLP-compatible backend.

Metrics

The gateway records the following metrics. All are exported to every configured backend (Prometheus/Grafana, AWS CloudWatch, OTLP collectors, and local JSON files) and are also the source for the in-dashboard live views.

Metrics export on a configurable interval and also feed the dashboard’s live views.

Structured logs

Every log entry carries a trace identifier that matches the corresponding OTLP span. This links a specific log entry to its trace in an external backend, so a request can be correlated across logs, traces, and metrics using a single identifier.

Log entries include request route, direction, status, policy outcome, and agent identity context when available. All exported signals carry resource attributes identifying the gateway instance: service name, binary version, and deployment environment. These ensure that signals from different gateway deployments remain distinguishable in a shared external backend.

Log examples

Every trust check query emits a structured log entry with the verification result and context:

    WARN trust check: denied surface_id="1f4802ef-6734-4ffb-9003-42fc86e4b3ad" 
     leg="caller" element_id="8008b0e2-1395-437a-9495-be105e7e2b83" 
     element_name="" trust_registry_id="323ae2cb-cffc-4d81-8018-debcea3770e5" 
     query_type="recognition" result="denied" ok=false latency_ms=101

The structured fields enable filtering and correlation:

When a trust check result flows to the policy engine, it appears in the OPA input as part of the context object:

"trust_check_results": {
  "caller": [
    {
      "id": "8008b0e2-1395-437a-9495-be105e7e2b83",
      "trust_registry_id": "323ae2cb-cffc-4d81-8018-debcea3770e5",
      "query_type": "recognition",
      "ok": true,
      "error": null,
      "authority_id": "did:web:authority.example.com",
      "entity_id": "did:web:caller.example.com",
      "action": "is",
      "resource": "ownedAgent",
      "query_resolved": true
    }
  ],
  "target": []
}

Your OPA rule can then branch on this result:

# Allow only if the caller passed trust verification
allow {
    input.trust_check_results.caller[_].ok == true
}

When a policy decision is made, a corresponding log entry records the outcome:

    WARN policy decision: deny policy_scope="surface" policy_flow="access_point" 
     policy_decision="deny" policy_id="surface.policy" 
     deny_reason="Policy denied request" surface_id="1f4802ef-6734-4ffb-9003-42fc86e4b3ad" 
     caller_did="did:webvh:QmTnW9VsBkDghqPSabm5mgKnw4mFd2QvTKPxtciMeE7P5i:..." 
     trace_id="4dfcbc74-9697-4e21-9434-d1a5045cdf80"

Key policy decision fields:

Correlating logs with traces

The trace_id field is the key to end-to-end debugging. When you see a denial in the logs, copy the trace_id and search for it in your OTLP backend:

1. Gateway logs (stdout) show the decision and reason.
2. Distributed trace (Jaeger, Tempo, Datadog) shows timing of each stage: authentication, trust check query, policy evaluation, and backend routing.
3. Request capture (dashboard Capture tab) shows the exact payload transformations.

Together, these three views explain not just what happened, but why and how long it took.

Log redaction

Log redaction runs by default on every gateway instance. Before any output leaves the process, an ordered pipeline of rules matches sensitive patterns and replaces them with [REDACTED].

Redaction applies to:

• Formatted log lines written to stdout.
• Span attribute values exported via OTLP.
• The WebSocket log stream delivered to the dashboard.

Sensitive data is sanitised at source and never reaches an external log backend unredacted. The default rule set covers common credential patterns such as bearer tokens and API keys. Custom redaction rules are not configurable in the current deployment model.

Sampling

The OTLP trace exporter supports a configurable sampling rate between 0.0 and 1.0. At 1.0, every request produces a trace span. At lower rates such as 0.1 for 10%, export volume is reduced while statistically representative traces are retained. Reducing the sampling rate is typical for high-throughput production deployments.

Metrics and logs are not sampled. Every request produces a metrics data point and a log entry regardless of the trace sampling setting.

Event notifications

Event notifications deliver push alerts to an external destination when gateway lifecycle events occur: a user is approved, a gateway is created, or a connection point changes state. Unlike telemetry signals, they fire on configuration and lifecycle changes, not on individual requests.

An event notification combines a delivery channel (email, Slack, a webhook endpoint, or a streaming platform), an event category that determines which events trigger it, and a message template populated with event-specific data at delivery time. When a matching event occurs, the gateway renders the template and delivers the message. Notifications are push-only and retried on failure, so a transient outage at the destination does not drop a notification.

Use event notifications when gateway activity needs to reach people or systems outside the dashboard. Use the dashboard tools for real-time request-level diagnosis. The two are independent and can be used together.

Configure an event notification →

Related

• Set up event notifications: Configure push alerts for gateway events.
• Metrics reference: Field-level reference for emitted metrics records.
• Surfaces: Understand where Capture sits in Agent Surface processing.