Prompt caching with OpenAI and Anthropic

· 3 min read

Prompt caching lets the provider reuse the start of a prompt it has already processed. When it works, the repeated part is billed at a fraction of the normal input price and the answer starts sooner. When it does not, you pay full price for the same system prompt on every call, and nothing tells you.

This guide explains how each provider caches, the mistakes that stop it from working, and how to check.

How caching works

Both providers cache a prefix: the beginning of the request, up to some point. The next request reuses the cache only if its beginning is identical, byte for byte, up to that point. Anything that changes early in the prompt (a date, a user name, a request id, tools in a different order) breaks the match for everything after it.

So the rule is the same everywhere:

  1. Stable content first: system prompt, tool definitions, reference documents, few-shot examples.
  2. Changing content last: the user's message, the conversation so far, today's date.

OpenAI

  • Caching is automatic for prompts whose stable start is long enough. On current models the minimum is 1,024 tokens; for some earlier models it depends on request settings.
  • Newer models also support explicit breakpoints and a choice of cache lifetime, so you can say exactly where the stable part ends and how long to keep it.
  • prompt_cache_key groups related requests (for example, one per customer) so they reuse the same cache and keep separate cache accounting.
  • The response's usage shows how many input tokens came from the cache (cached_tokens).

Anthropic

  • Caching needs cache_control. Add it once at the top level of the request for automatic caching (the API places the breakpoints as the conversation grows), or on individual content blocks for exact control.
  • The default cache lives 5 minutes, renewed each time it is used. A 1-hour lifetime is available.
  • Writing to the cache costs a little more than normal input; reading from it costs much less. With the 5-minute cache, it pays for itself after one cache read. Check Anthropic's pricing page for the current multipliers.
  • The response's usage reports cache_creation_input_tokens (written) and cache_read_input_tokens (read).
python
import anthropic

client = anthropic.Anthropic()
response = client.messages.create(
    model="claude-sonnet-5-5",
    max_tokens=1024,
    cache_control={"type": "ephemeral"},  # automatic caching
    system=LONG_STABLE_SYSTEM_PROMPT,
    messages=[{"role": "user", "content": question}],
)
print(response.usage.cache_read_input_tokens)

Why the cache misses

  • A timestamp or id at the top of the system prompt. Move it to the end of the user message.
  • Tools built in a different order each time. Sort them, or build the list once.
  • Personalised system prompts. "You are helping Priya" changes the prefix for every user. Put the name in the user message.
  • Calls too far apart. A cache that lives 5 minutes does not help a feature called once an hour. Use a longer lifetime where the provider offers one, or accept the miss.
  • Prefix too short. Below the provider's minimum nothing is cached.
  • Different models. Caches are per model.

Check that it works

Look at the cached-token counts in the usage of each response, per feature. A feature that sends the same long prompt many times a day with a cached share near zero is paying full price for nothing. That is the signal to fix, and it can be found from token counts alone, without reading any prompt.

How Zexl finds missed caching

Zexl groups calls that share the same system prompt, tools and model (using salted hashes, never the text). When a group has 20 or more calls in a day, close enough together for the cache to help, a long enough shared start, and under 10% of input read from cache, the calls are marked prompt caching not used, with the money caching would have saved. See waste patterns.

Create a free account to see it on your own calls, or read How to reduce your LLM API costs for the other big savings.

See where your own AI spend goes

One import in a Python app, or five environment variables for Claude Code. Prompts never leave your servers.