Cache repeated responses to cut cost and latency

Serve an identical or near-duplicate request from an in-memory response cache instead of calling the provider again.

This guide enables a surface’s response cache, so a repeated or near-duplicate prompt is served from memory instead of incurring another provider call. For the conceptual model, see Resilience and caching →.

Common patterns such as FAQs, retried agent steps, and repeated tool calls send the same or a near-identical prompt to a provider many times over. Without a response cache, every one of those repeats calls the provider again, adding avoidable latency and cost on top of whatever a usage limit or budget alert is already tracking.

Use this guide when:

  • Your surface receives a meaningful share of exact or near-duplicate requests, such as FAQs or repeated agent steps.
  • You want to cut latency and cost on those repeats without changing anything the client sends.
  • You can tolerate serving a cached answer for a bounded period, rather than always calling the provider fresh.

You do not need response caching if your surface’s traffic is already highly varied, or if every request must always reach the live provider, such as a surface answering with strictly time-sensitive information.

Prerequisites

  • An active LLM Surface.
  • For semantic matching only: an embeddings endpoint and model, plus a secret for its API key if required.

Steps

Enable the response cache

The Response cache panel fully expanded, with Enable response cache on, TTL, Max entries, Matching mode set to Semantic, and the Embedder provider, Embedder model, and Similarity threshold fields

Under SURFACES in the dashboard sidebar, select LLM, open your surface, select its Surface node (the root container the rest of the pipeline sits inside), and select Configure Surface Features in its side panel. Scroll to Response cache and turn on Enable response cache.

Set the cache lifetime and size

Set TTL (Seconds) for how long a cached response stays valid (default 3600), and Max Entries for how many responses the cache retains before the oldest are evicted (default 1000).

Choose a matching mode

Set Matching Mode to exact (the default, matching a byte-identical normalized request) or semantic (matching by embedding similarity across differently worded but equivalent requests).

Configure the embedder, if using semantic matching

If Matching Mode is semantic, select an Embedder Provider (choosing a catalogue provider auto-fills its endpoint and model), or choose Custom / other and enter an Embedder Endpoint URL and Embedder Model directly. Set API Key (Secret) if the embedder requires one, and adjust Similarity Threshold (default 0.95) for how close two requests must be to count as a match.

Save the surface

Select the save icon in the toolbar. The change applies immediately; no restart is required.

Confirm

Replace <YOUR_APPLIANCE_HOST> and <YOUR_SURFACE_ROUTE> with your surface’s values.

Test 1: an identical repeated request returns a cache hit

curl -k -i -X POST "https://<YOUR_APPLIANCE_HOST><YOUR_SURFACE_ROUTE>/v1/chat/completions" \
  -H "Content-Type: application/json" \
  -d '{
    "messages": [
      { "role": "user", "content": "Say hello from the response cache test." }
    ]
  }'

The -k flag disables TLS certificate verification. Use this for local testing only. Remove it in production.

Send this exact request twice. Expected: the first response’s x-agent-stream-cache header reads miss. The second, identical request’s x-agent-stream-cache header reads hit, and it also carries x-agent-stream-cache-saved-usd, the provider spend this hit avoided.

Test 2: a streamed request always returns a cache miss

Resend the same request with "stream": true added to the body. Expected: x-agent-stream-cache reads miss, even if the identical non-streaming request was cached moments earlier. Only the buffered, non-streaming path is cache-eligible.

Troubleshooting

SymptomLikely causeFix
An identical repeated request still shows miss.The cache’s TTL expired between requests, Max Entries was exceeded and the entry was evicted, or the request differs in a field the normalizer does not ignore.Confirm the requests are truly identical, increase TTL or Max Entries, and resend within the TTL window.
A near-duplicate request never matches in semantic mode.Similarity Threshold is set too high for the phrasing difference, or the embedder is misconfigured.Lower Similarity Threshold, and confirm the embedder endpoint and model resolve correctly.
Cache hits stop after a load-balanced pool routes to a different provider.Cache entries are scoped per surface, per variant, and per effective provider target, so routing to a different pool member is a cache miss by design.This is expected; a cache entry from one provider target never leaks into another’s slot.
A cached answer still reflects an old configuration after editing the surface.This should not happen: cache entries are purged automatically when a surface’s configuration changes.Confirm the surface was actually saved after the edit, and that you are testing against the intended surface and variant.

Next steps