LLM drift

This is the field-level detail behind LLM drift detection →: the surface’s Drift Harness settings, the nine dimensions it tracks, exactly how a baseline turns into a stable/warn/drift verdict, and the retention defaults behind “200 sessions and 50 runs.”

Surface configuration

Turning this on feeds everything else on this page: the nine dimensions, alerting, and replay evaluation all depend on drift monitoring being switched on for a surface. It turns “does this surface still behave the way it did” into an ongoing measurement, rather than something you only learn from a support ticket or a quality complaint. Absent means the surface emits no drift data at all.

The Drift Harness settings and the drift chart area are two separate screens: the settings live in the surface builder’s Drift Harness element, shown below, and the charts live on the standalone LLM Drift monitoring page as a grid of per-dimension charts, each carrying its own stable, warn, drift, or no-baseline badge.

• Customer-facing, compliance-sensitive, or otherwise high-stakes surfaces are the best candidates for Monitor drift on live traffic, since a quiet quality regression there is costly to miss. A low-stakes or experimental surface may not need the always-on sampling overhead.
• Warn threshold and Drift threshold start at 25% and 50%; adjust them once you’ve seen how the surface actually behaves. Choosing thresholds below covers when to tighten or loosen them.
• Block saves on regression suits a surface where a drifting configuration should never accumulate further changes before the regression gets addressed. Details in Regression gating.
• Alert integrations stays empty by default, reusing the surface’s own integration mappings. Give it a value only to route drift alerts somewhere different, like a dedicated on-call channel.

Warn threshold and Drift threshold are resolved together: an invalid override (not a real number, or zero/negative) is ignored in favor of the built-in default. Also, Drift threshold can never end up lower than Warn threshold. If you set Drift below Warn, Drift is automatically raised to match it, so the two can’t invert.

Replay-based evaluation settings

Needs an embeddings provider to measure semantic distance.

• Turn on Run automatically if you want drift evidence to build up on a schedule without someone remembering to click Run; leave it off if you’d rather trigger replays manually after a prompt or model change.
• Lower Max sessions per run if replay runs are taking too long or costing too much; the default cap is usually fine for a low-to-moderate-traffic surface.

Alert cadence overrides

Absent fields fall back to the appliance-wide defaults. Most surfaces should leave both unset; set them individually only when a surface genuinely needs a faster or slower re-alert cadence than the rest of the appliance.

Nine dimensions

Quality score and semantic distance only populate once a replay evaluation has run against the surface’s captured sessions; the other seven fill in from ordinary production traffic (no extra sampling cost). If you turn on monitoring but never run a replay evaluation, expect those two dimensions to stay unpopulated.

Baseline and verdict mechanics

A baseline is a per-surface snapshot: the surface, when it was captured, the window it covers, and a mean value per dimension over the window that was current when it was pinned.

A dimension’s verdict compares how far its current mean has moved from the pinned baseline, as a share of the baseline’s size. If that change reaches the Drift threshold it’s flagged drift; if it only reaches the Warn threshold, warn; otherwise stable. A dimension with no pinned baseline or no samples in the window is unknown, never drift.

A surface’s overall status is the worst of its per-dimension verdicts, ranked from least to most severe: stable, unknown, warn, drift. So one regressing axis is enough to flag the whole surface.

Choosing thresholds, and what to do about warn/drift

Picking Warn threshold / Drift threshold. Start at the defaults (25% warn, 50% drift) and adjust from what you observe:

• Tighten both thresholds for a surface where subtle quality regressions matter a lot (customer-facing answers, compliance-sensitive output, anything where a quiet drop in quality is expensive to catch late). Lowering the thresholds surfaces smaller relative changes sooner.
• Loosen both thresholds for a noisy or low-volume surface where the defaults produce false alarms. Fewer samples per window means more variance in the computed means, so ordinary noise can look like drift at default sensitivity.
• Change them together, not just one: a Drift threshold set below the current Warn threshold is raised automatically to match it, so tightening drift alone has no effect until warn moves too.

Responding to a warn or drift verdict. Treat this as an investigation prompt, not a false alarm to dismiss:

1. Investigate first. Check whether a prompt, model, provider, or routing change landed on this surface around the time the dimension moved. Most real drift traces back to a change someone made.
2. Re-baseline once you’ve confirmed the new behavior is intentional. Pin a new baseline so future comparisons measure against the current normal, not the old one.
3. If you can’t explain the change, don’t just re-baseline it away. An unexplained regression is exactly what this feature exists to catch.
4. Consider Block saves on regression when a bad save compounding on top of an existing regression would be costly to unwind.

Regression gating

When Block saves on regression is on, saving the surface’s configuration is refused while its current overall verdict (compared against its pinned baseline) resolves to drift. The check is default-safe: a missing drift/baseline store, a surface with no pinned baseline, or an empty window all resolve to unknown, never drift, so gating can only block on a genuine, positive regression signal, never a data gap.

Alerting

A periodic evaluator re-judges every drift-monitored surface every 300 seconds, comparing each surface’s trailing 3,600-second (1-hour) window against its pinned baseline. Per-surface alert state (current status, alerts fired, last-alert time) is held in memory and reset when the surface recovers out of drift:

• Entering drift fires drift.detected immediately.
• While the surface stays in drift, it re-fires no more often than Re-alert every (minutes) (default 60).
• Firing stops once Max alerts per baseline (default 5, 0 = unlimited) is reached for the current baseline.
• Recovering to stable resets the alert state, so the next entry into drift alerts again from zero.

The event carries the surface, the dimension, the baseline, the current value, and the delta. See Notifications and alerts → for the full drift.detected variable set and routing.

Replay evaluation

While monitoring is on, sampled interactions are captured as recorded sessions: the real request/response pairs a replay evaluation re-executes.

Each run appears in run history with these fields:

The replay separates the source of the captured sessions from the target surface or variant that executes them, defaulting to the same surface as a self-check. A self-run folds its result into the source surface’s own dimensions; pointing the target elsewhere produces an inspect-only cross-surface run, saved to run history without touching the baseline it was compared against. Use this to test a candidate model or prompt change against real recorded traffic before rolling it out on the source surface itself.

Retention

Retention is an appliance-wide setting, not a per-surface one, and combines two caps:

• Count cap: run history and captured sessions are capped at 50 runs and 200 sessions per surface, pruned automatically as new ones are recorded. This cap is fixed and not configurable by you.
• Age cap: an optional setting that drops runs or sessions older than a configured number of days; it defaults to no age limit, which is useful when storage volume or a compliance/retention policy requires bounding how long drift history is kept.

Both caps are enforced whenever a surface writes a new run or session, and a periodic sweep re-applies them across all surfaces, so an idle surface that has stopped writing new records is still pruned. Both runs and sessions are encrypted at rest.

Related

• LLM drift concept: Why drift monitoring exists and how the Drift Harness fits into the surface builder.
• Notifications and alerts: The drift.detected event’s full variable set and integration routing.
• Variants and progressive rollout: The variant a cross-surface replay’s target can point at.
• Cost and usage limits: The budgets a replay run spends through like any other call.