Alert on budget breaches and cost spikes

This guide creates an integration and attaches a usage alert to a surface, so a budget breach or a cost anomaly reaches the people who need to act on it. For the conceptual difference between this and a hard cap, see Cost and usage governance →.

A usage limit, such as the one added in Add guardrails and a budget, blocks a request once a hard ceiling is reached. Without a usage alert, everything short of that hard ceiling, a rising error rate or a single execution far above a surface’s typical cost, produces no signal until someone happens to check a dashboard.

Use this guide when:

• You want to know about a cost or token threshold being crossed, or a rising error rate, before it becomes a hard block.
• A single anomalous execution, far more expensive than a surface’s typical call, should page someone rather than pass silently.
• Alerts need to reach an existing channel your team already watches, such as Slack, email, or a webhook into your own system.

You do not need usage alerts if a hard usage limit already gives you the coverage you need and nobody needs to be notified before it triggers.

Prerequisites

• An active LLM Surface with cost tracking enabled.
• Dashboard access with permission to create integrations and edit surfaces.
• A destination for alerts: a Slack incoming-webhook URL, SMTP credentials for email, or a webhook endpoint.

Steps

Create an integration

Select Integrations in the dashboard sidebar, then select Add Integration. Set Integration Name, Category, and Integration Type (webhook, slack, email, or stream), then fill in the type-specific configuration: a Slack Webhook Configuration URL for Slack, SMTP Configuration for email, or a destination URL for a webhook. Save; a new integration is active immediately, with no separate Enabled switch until you come back to edit it.

Add a usage alert to the surface

Under SURFACES in the dashboard sidebar, select LLM, and open your surface. On the canvas, select the Surface node (the root container the rest of the pipeline sits inside) and select Configure Surface Features in its side panel to open the full-page editor. Scroll to Breach & anomaly alerts and turn on Enable breach & anomaly alerts, then add a usage alert and select an Alert Type: Cost per request, Tokens per request, Error rate, or Cost spike.

Set the threshold

The threshold field’s label changes with the alert type: Cost ceiling (USD) for cost per request, Token ceiling for tokens per request, Error-rate ceiling (0–1) for error rate, or Spike multiplier (× mean) for cost spike. A cost-spike multiplier must be strictly greater than 1.0; an error-rate ceiling must be a positive fraction no greater than 1.0.

Select where the alert is delivered

Under Notify via, select the integration you created. You can add more than one usage alert, each with its own type, threshold, and delivery channels.

Save the surface

Select the save icon in the toolbar. Usage alerts are evaluated on the response path but dispatched off it, so they never add latency to the request they were triggered by.

Confirm

Replace <YOUR_APPLIANCE_HOST> and <YOUR_SURFACE_ROUTE> with your surface’s values. Set a low Cost ceiling (USD) for this test, so a single ordinary request is enough to cross it.

Test 1: a request that crosses the threshold still returns 200, and a notification arrives

curl -k -X POST "https://<YOUR_APPLIANCE_HOST><YOUR_SURFACE_ROUTE>/v1/chat/completions" \
  -H "Content-Type: application/json" \
  -d '{
    "messages": [
      { "role": "user", "content": "Say hello." }
    ]
  }'

The -k flag disables TLS certificate verification. Use this for local testing only. Remove it in production.

Expected: 200 OK. The request completes normally; a usage alert never blocks it. Check the integration’s destination, your Slack channel, inbox, or webhook receiver, for a message carrying ALERT_TYPE, THRESHOLD, OBSERVED, and MESSAGE.

Test 2: a cost-spike alert fires only after 5 baseline requests exist

Send at least 5 ordinary requests first to establish a rolling baseline, then send one unusually expensive request (for example, a much longer prompt or a pricier model). Expected: the cost-spike alert fires only after the baseline exists; a single early request against an empty history cannot trip it.

Troubleshooting

Next steps

• Cap spend per pipeline stage: Pair this alert with a hard ceiling on the stage it is watching.
• Cache repeated responses to cut cost and latency: Reduce the spend a cost alert would otherwise be watching.
• Cost and usage governance: Why usage limits and breach alerts are two different mechanisms.

Related

• Notifications and alerts reference: The full Integration record shape and event-type taxonomy.
• Cost and usage limits reference: Every AlertType and its threshold semantics.
• Observability: Where usage telemetry and per-request cost headers are exposed alongside these alerts.