# LLM Token Engineering: Prompt Caching, Batch APIs and the Silent Cache Busters

> How Anthropic, OpenAI and Google price tokens, why cached and batched tokens are cheaper, how to route for cache hits, and the easy-to-miss bugs (timestamps, unsorted JSON, reordered tools) that quietly wreck your hit rate.

Author: Rahul Sumbly | Published: 2026-10-04 | Updated: 2026-10-08 | Tags: LLM, Cost Optimization, Prompt Caching, Batch API
Canonical URL: https://rahul.sumbly.tech/blog/llm-token-engineering/
Resources: https://github.com/Rahul-s-007/resources/tree/main/token-engineering

Most LLM bills aren't high because of the model you picked. They're high because the same tokens get paid for over and over: the same 20K-token system prompt, the same tool definitions, the same document, re-processed from scratch on every call.

All three major providers now give you two levers against this:

1. **Prompt caching:** pay a fraction of the price for input the provider has already processed.
2. **Batch / flex processing:** pay about half price if you can wait.

Used well, the two together cut input costs by 80–95%. Used carelessly, caching silently does nothing, because one byte you didn't notice changed at the top of your prompt.

This post covers the fundamentals first, then the actual numbers from each provider's docs, then routing, then batch, then a long list of ways caches get busted in real code, and finally fingerprinting: a way to measure all of it in your own traffic.

> **Pricing note:** all prices and limits below come from the official Anthropic, OpenAI and Google docs as of **October 4, 2026**. Providers change these often, so check the [sources](#sources) before making a decision.

## Part 1: How an LLM actually spends compute

### Tokens

Models don't see characters; they see **tokens**, sub-word chunks from a fixed vocabulary. A rough rule for English is ~4 characters or ~0.75 words per token, but every provider uses its own tokenizer. The same text yields different counts across vendors and sometimes across model generations: Anthropic notes its Claude 4.7+ tokenizer produces roughly 30% more tokens for the same text than earlier models. Price per token only means something next to the tokenizer.

### Two phases: prefill and decode

Every request runs in two very different phases:

```mermaid
flowchart LR
  A["Your prompt<br/>(N input tokens)"] --> B["PREFILL<br/>all N tokens processed in parallel<br/>compute-bound"]
  B --> C["KV cache<br/>(keys + values for every<br/>layer and every token)"]
  C --> D["DECODE<br/>one output token at a time<br/>memory-bandwidth-bound"]
  D -->|append new token's K/V| C
  D --> E[Response]
```

- **Prefill.** The model reads your whole prompt at once. Each token passes through every transformer layer, and at every attention layer the model computes a **key** and **value** vector for it. This is a big, highly parallel matrix multiplication, which GPUs are good at.
- **Decode.** The model generates output one token at a time. Each new token attends over the keys and values of everything before it, so instead of recomputing them the server keeps them in GPU memory. That stored state is the **KV cache**. Decode is sequential and limited by memory bandwidth, which is a big reason **output tokens cost 4–6× more than input tokens** at every provider.

### Prompt caching = keeping the KV cache across requests

Inside one request, the KV cache always exists. **Prompt caching** keeps the KV cache for a prompt prefix *after the request finishes*, so the next request starting with the same prefix can skip prefill for that part. OpenAI's docs describe it directly: the cache stores the KV tensors for the prefix, not the raw tokens.

That explains the economics:

- **Reads are cheap** because the provider skips the prefill compute for those tokens.
- **Caches expire** because KV state is large and sits in expensive GPU or host memory. Every second it's kept costs the provider something, so caches have a time-to-live (TTL).
- **Writes can cost extra** (Anthropic, and OpenAI on newer models), and **storage can be billed by the hour** (Google's explicit caching), because the provider pays for that memory.

### Why caching is prefix-only

This is the most important idea in the post, and it comes straight out of how attention works.

LLMs use *causal* attention: the key/value for token *i* depends on **every token before it**. Change one token at position 500 and the KV state for token 501 onwards is different, even if those tokens' text is identical. So caches can only be reused for an **exact, byte-identical prefix**.

```mermaid
flowchart TB
  subgraph R1["Request 1"]
    direction LR
    a1[Tools] --> b1[System prompt] --> c1[Docs] --> d1["Question A"]
  end
  subgraph R2["Request 2 - cache HIT"]
    direction LR
    a2[Tools] --> b2[System prompt] --> c2[Docs] --> d2["Question B"]
  end
  subgraph R3["Request 3 - cache MISS from here"]
    direction LR
    a3[Tools] --> b3["System prompt<br/>+ timestamp"] --> c3[Docs] --> d3["Question C"]
  end
  R1 ~~~ R2 ~~~ R3
```

In Request 3, the docs and everything after them are byte-identical to Request 1, but the changed timestamp sits *before* them, so none of it can be reused. **Stable content first, volatile content last.** Nearly every cache-busting bug in Part 6 is a variation on this.

## Part 2: Implicit vs explicit caching (and each provider's default)

There are two models of caching:

- **Implicit:** the provider caches prefixes automatically. You change nothing, but you get no guarantees about hits.
- **Explicit:** you mark what to cache (a breakpoint, or a named cache object), you control the TTL, and you get predictable behavior. Sometimes you pay for writes or storage.

| | **Anthropic (Claude)** | **OpenAI** | **Google (Gemini)** |
|---|---|---|---|
| **Default** | **Off.** Opt in with `cache_control` | **On**, implicit, for supported models | **On**, implicit, for Gemini 2.5+ |
| **Explicit option** | `cache_control` breakpoints on blocks (max 4), or one top-level `cache_control` that places the breakpoint for you | GPT-5.6+: `prompt_cache_options` + `prompt_cache_breakpoint` (up to 4 writes/request) | `cachedContents` API: create a named cache, reference it by name |
| **Guarantee** | Deterministic prefix match at your breakpoints | Best-effort (routing-dependent) | Implicit: *"no cost saving guarantee"*. Explicit: *"cost saving guarantee"* |
| **TTL** | 5 min (default) or 1 h; refreshed on every hit | GPT-5.6+: `ttl: "30m"`. Older: in-memory ~5–10 min idle, up to 1 h, or `prompt_cache_retention: "24h"` | Explicit: default **1 h**, no min/max bounds, adjustable. Implicit: not specified |
| **Min. prefix** | 512 – 4,096 tokens depending on model | 1,024 tokens | Implicit: 2,048 (2.5) or 4,096 (3.x). Explicit: model-dependent |

### Anthropic

Claude doesn't cache anything unless you ask. There are two ways to ask:

1. **Automatic caching:** one top-level `cache_control` field. The API puts the breakpoint on the last cacheable block and moves it forward as a conversation grows. Despite the name it's still opt-in; it automates *where* the breakpoint goes, not *whether* caching happens.
2. **Explicit breakpoints:** `cache_control` on specific content blocks (tool definitions, system blocks, messages), up to 4 per request.

```python
import anthropic

client = anthropic.Anthropic()

response = client.messages.create(
    model="claude-sonnet-5",
    max_tokens=1024,
    system=[
        {
            "type": "text",
            "text": LARGE_STABLE_INSTRUCTIONS,           # 20K tokens of policy, examples, docs
            "cache_control": {"type": "ephemeral"},      # 5-minute TTL; add "ttl": "1h" for 1 hour
        }
    ],
    messages=[{"role": "user", "content": user_question}],  # volatile content goes AFTER the breakpoint
)

u = response.usage
print(u.cache_creation_input_tokens, u.cache_read_input_tokens, u.input_tokens)
```

Things the Anthropic docs call out:

- **Render order is `tools` → `system` → `messages`.** A breakpoint on the last system block caches tools and system together.
- **The minimum cacheable prefix depends on the model and doesn't go up or down steadily across generations:** 512 tokens on Claude Opus 5, 1,024 on Sonnet 5 / Opus 4.8, 2,048 on Opus 4.7, 4,096 on Opus 4.6 and Haiku 4.5. If the prefix is shorter, it **silently** isn't cached: there's no error, just `cache_creation_input_tokens: 0`.
- **20-block lookback:** each breakpoint looks back at most 20 positions for a previous cache entry. A single turn that appends more than 20 blocks can miss the previous cache.
- **Caches are isolated per workspace** on the Claude API. The same prompt sent from two workspaces writes two separate caches.

### OpenAI

Caching is on by default. OpenAI's docs say cached input is *"discounted up to 95%"*. The details depend on the model generation:

- **GPT-5.6 and later:** implicit *or* explicit breakpoints via `prompt_cache_options`, up to 4 cache writes per request, a `30m` TTL measured from the last write or reuse, and a `prewarm` option. Writes are billed at **1.25×** the uncached rate and reads at **0.1×**.
- **Earlier models:** implicit only, with no write charge. Entries stay in memory for about 5–10 minutes of inactivity (up to an hour), or you can set `prompt_cache_retention: "24h"` on supported models (gpt-5.5, gpt-5.4, gpt-5.1, gpt-5, gpt-4.1 and others). Organizations without Zero Data Retention default to `24h`.

The cache key covers more than the text. OpenAI lists `model`, `tools`, `parallel_tool_calls`, the structured-output schema (`text.format`), `reasoning.effort` and `text.verbosity` as part of the cached context. **Changing your JSON schema or reasoning effort changes the prefix.**

Hits show up in `usage.input_tokens_details.cached_tokens`, plus `cache_write_tokens` on newer models.

### Google Gemini

Gemini has both modes side by side:

- **Implicit caching** is on by default for Gemini 2.5 and newer. Google *"automatically pass[es] on cost savings"* when you hit, with no guarantee. Their two tips: put large, common content at the **start** of the prompt, and send similar-prefix requests **close together in time**.
- **Explicit caching** (the `cachedContents` API, currently `v1beta`) gives you a named cache with a TTL you choose:

```python
from google import genai
from google.genai import types

client = genai.Client()

cache = client.caches.create(
    model="gemini-2.5-pro",
    config=types.CreateCachedContentConfig(
        system_instruction="You are a contract analyst.",
        contents=[contract_text],       # the big, reused part
        ttl="3600s",                    # default is 1 hour; no min/max bound
    ),
)

response = client.models.generate_content(
    model="gemini-2.5-pro",
    contents="List every termination clause.",
    config=types.GenerateContentConfig(cached_content=cache.name),
)
```

Explicit Gemini caching has **three cost parts**: cached tokens at a reduced rate, **storage billed per token per hour** of TTL, and normal rates for uncached input and output. A long TTL on a cache nobody reads is pure cost.

## Part 3: The numbers

Prices per **1M tokens**, standard tier, short context. Representative current models only.

### Anthropic

| Model | Input | 5m cache write | 1h cache write | Cache read | Output |
|---|---|---|---|---|---|
| Claude Opus 5 | $5.00 | $6.25 | $10.00 | $0.50 | $25.00 |
| Claude Sonnet 5 | $2.00 | $2.50 | $4.00 | $0.20 | $10.00 |
| Claude Haiku 4.5 | $1.00 | $1.25 | $2.00 | $0.10 | $5.00 |

Writes cost **1.25×** (5 min) or **2×** (1 h); reads cost **0.1×** (0.05× on Opus 5.5, 0.025× on Fable 5.1). Per Anthropic, a 5-minute cache pays off after **one** read and a 1-hour cache after **two**. Every read also resets the TTL, so steady traffic keeps a 5-minute cache warm indefinitely.

### OpenAI

| Model | Input | Cached input | Cache write | Output |
|---|---|---|---|---|
| gpt-6.1-sol | $2.00 | $0.10 | $2.50 | $10.00 |
| gpt-5.6-sol | $4.00 | $0.40 | $5.00 | $20.00 |
| gpt-5.5 | $5.00 | $0.50 | – | $30.00 |

The cached discount on older models varies: gpt-4.1 is 75% off and gpt-4o is 50% off, while GPT-5.x is 90% off. GPT-5.6 Sol is on promotional pricing through at least Nov 21, 2026.

### Google

| Model | Input | Cache read | Storage (per 1M tok / hour) | Output |
|---|---|---|---|---|
| gemini-3.1-pro-preview (≤200K) | $2.00 | $0.20 | $4.50 | $12.00 |
| gemini-3.8-flash | $0.75 | $0.075 | $0.50 | $3.75 |
| gemini-2.5-pro (≤200K) | $1.25 | $0.125 | $4.50 | $10.00 |

Across these models a cache read is 10% of the input price. Gemini 3.8 Flash prices **double on January 1, 2027**.

### Watch the size of each request: long-context pricing

Per-token prices aren't flat. On some models, once a single request's **input** crosses a threshold, you move to a more expensive price tier, and the higher rate covers output and cached tokens too.

| Provider | Threshold (input tokens per request) | Input | Cached input | Output |
|---|---|---|---|---|
| **OpenAI** (gpt-6.1-sol, gpt-5.6-sol, gpt-6-astra) | > 272K | 2× | 2× | 1.5× |
| **Google** (gemini-3.1-pro-preview, gemini-2.5-pro) | > 200K | 2× | 2× | 1.5× |
| **Google** Flash models | none | – | – | – |
| **Anthropic** (Claude 4.6 and later) | none, up to 1M | – | – | – |

In dollars: gpt-6.1-sol goes from $2 / $10 (input / output) to $4 / $15, and gemini-3.1-pro-preview from $2 / $12 to $4 / $18, with its cache reads going from $0.20 to $0.40. Anthropic's pricing page says the opposite for current Claude models: a 900K-token request is billed at the same per-token rate as a 9K-token one.

Two things follow from this:

- **Track input tokens per request, not just total spend.** Log the input token count from every response's usage data and alert as a route approaches the threshold. An agent whose conversation history grows by a few thousand tokens per turn can drift across the line without any code changing, and the bill for that route jumps.
- **Caching doesn't keep you under the threshold.** The tiers are defined by the size of the prompt, and cached tokens are still part of the prompt. A 300K-token prompt that is 95% cache hits is still a long-context request, paying the doubled cached rate.

Neither OpenAI's nor Google's pricing page spells out whether the higher rate applies to the whole request or only to the tokens past the threshold. Both describe the tiers by request size ("prompts > 200k tokens", "long context: >272K input tokens"), so budget as if the whole request is billed at the higher rate, and confirm on a real invoice if you run close to the line.

If a route sits near a threshold, the usual fixes are trimming or summarizing history in large chunks (see Part 6 for why not one message at a time), retrieving fewer or smaller documents, or moving that route to a model with flat pricing.

### A worked example

Say you have a **50K-token** system prompt (policies, few-shot examples, tool schemas) and send **1,000 requests an hour** on Claude Sonnet 5:

- **No caching:** 50M input tokens × $2.00 = **$100.00/hour**
- **With a 5-minute cache:** 1 write (50K × $2.50/M = $0.125) + 999 reads (≈50M × $0.20/M = $9.99) ≈ **$10.12/hour**

That's about **90% off** the prefix cost for one field in the request. It only holds if the cache actually hits, which is what the rest of this post is about.

## Part 4: Routing and `prompt_cache_key`

A cache isn't a global database. **KV state lives in the memory of the particular machine that computed it.** For a request to hit, the load balancer has to send it to a machine that holds the matching prefix.

OpenAI's docs describe this most clearly. Requests are routed using machine load, a **hash of the first tokens of the prompt** (after OpenAI's hidden system content, including your tool definitions), and an optional **`prompt_cache_key`** you provide.

```mermaid
sequenceDiagram
  participant App
  participant LB as Router
  participant M1 as Machine A (has prefix)
  participant M2 as Machine B (cold)
  App->>LB: request (prefix hash + prompt_cache_key)
  LB->>M1: route by hash(prefix, key)
  M1-->>App: cache HIT - cheap and fast
  Note over LB,M1: Above ~15 req/min for one prefix+key,<br/>traffic overflows to other machines
  App->>LB: 16th request in the same minute
  LB->>M2: overflow
  M2-->>App: cache MISS - full price
```

Things to know:

- **On models before GPT-5.6,** the key affects routing. OpenAI says cached states live on individual machines, *"where traffic above 15 requests per minute can lead to overflow routing,"* and recommends about **15 requests per minute per key** across all prefixes that use it. For heavier traffic, split across several keys with a **deterministic** mapping, for example `f"support-bot:{hash(tenant_id) % 8}"`. Keys *"do not pin requests to a machine or guarantee a cache hit."*
- **On GPT-5.6 and later,** routing for caching is automatic and the key is optional. It now mainly isolates cache accounting per user or tenant, which also prevents *cache-hit probing*: an attacker inferring another user's prompt from response latency.
- **Choosing a key:** key by *what shares a prefix* (an agent version, a tenant's knowledge base), not by request ID. A unique key per request guarantees misses.

```python
from openai import OpenAI

client = OpenAI()

resp = client.responses.create(
    model="gpt-5.5",
    instructions=STABLE_INSTRUCTIONS,                  # identical bytes every call
    input=user_message,
    prompt_cache_key=f"kb-assistant:{tenant_shard}",   # stable, shared by requests with the same prefix
    prompt_cache_retention="24h",                      # supported on gpt-5.5, gpt-5.1, gpt-4.1, ...
)
print(resp.usage.input_tokens_details.cached_tokens)
```

**Anthropic** and **Google** don't expose a routing key. On Claude, the breakpoint is the contract: if the bytes match within the TTL and workspace, you hit. On Gemini's implicit cache, the closest thing to routing is Google's advice to send similar-prefix requests close together; if you need certainty, use explicit caching.

```subscribe
```

## Part 5: Batch APIs (and their real-time cousin, Flex)

### Why batch tokens are half price

All three providers charge **50% of standard rates** for batch work. The docs don't explain the scheduling internals, but the economics are simple:

- **Interactive traffic needs headroom.** Providers have to keep enough GPUs to serve peak demand within seconds, and demand rises and falls during the day and across time zones. Off-peak, a lot of that capacity sits idle.
- **Batch work has no latency deadline.** With a 24-hour window, the provider can fill the gaps: run your jobs when interactive demand drops, pack GPUs to high utilization, and group requests efficiently.
- **You're selling flexibility.** The provider gets work that smooths out its utilization curve, and you get half price. That's the whole trade.

### How each provider does it

| | **Anthropic** | **OpenAI** | **Google** |
|---|---|---|---|
| Discount | 50% on all tokens | 50% | 50% |
| Submit as | List of Messages requests, each with a `custom_id` | `.jsonl` file upload (`custom_id`, `method`, `url`, `body`) | Inline list (< 20 MB) or JSONL file (up to 2 GB) |
| Size limit | 100,000 requests **or** 256 MB | 50,000 requests **or** 200 MB | 20 MB inline / 2 GB per file |
| Turnaround | "Most" within 1 hour; expires at **24 h** | 24 h window, "often more quickly" | Target 24 h; expires after **48 h** running/pending |
| Results kept | 29 days | – | 6 weeks |
| Rate limits | Separate batch limits (HTTP + queued requests) | Separate pool; per-model cap on queued tokens | Separate batch-mode limits |
| Works with caching? | Yes, discounts **stack**; hits are **best-effort** | Pricing table lists half-price cached input for GPT-5.x/6 batch | Yes; check pricing per model (see below) |

### The nuances that bite

**1. It's not instantaneous, and nothing is promised.** Even if most batches finish in an hour, your design has to work with the worst case: a full day, plus expiry. Anthropic and OpenAI give up at 24 hours. Unfinished requests come back as `expired` (Anthropic, not billed) or as `batch_expired` errors (OpenAI). Completed requests are still returned and billed, so **expect partial results and plan to resubmit.**

**2. Results come back out of order.** All three providers say so. Always join on `custom_id`, never on position. Make `custom_id` meaningful (`ticket-48213`, not `req-7`) so retries and debugging stay simple.

**3. Validation is asynchronous.** On Anthropic, each request's `params` are validated *during processing*, so a typo in request #40,000 shows up as an error hours later. **Send one request through the normal synchronous API first** to check the request shape.

**4. Caching in a batch is best-effort.** Batch requests run concurrently and in any order, so they don't neatly reuse each other's caches. Anthropic suggests the **1-hour TTL** for batches with shared context, because a batch can easily run longer than 5 minutes.

```python
# Anthropic: a batch where every request shares a cached rubric
batch = client.messages.batches.create(
    requests=[
        {
            "custom_id": f"ticket-{t.id}",
            "params": {
                "model": "claude-haiku-4-5",
                "max_tokens": 512,
                "system": [{
                    "type": "text",
                    "text": CLASSIFICATION_RUBRIC,
                    "cache_control": {"type": "ephemeral", "ttl": "1h"},  # outlives a long batch
                }],
                "messages": [{"role": "user", "content": t.body}],
            },
        }
        for t in tickets
    ]
)
# Poll until batch.processing_status == "ended", then stream results and join on custom_id.
```

**5. The batch-plus-cache price isn't always obvious.** Anthropic explicitly says the two discounts **multiply**. OpenAI's Batch pricing table lists half-price cached input for GPT-5.x and GPT-6 models, but shows no batch cached rate for gpt-4.1/gpt-4o. Google's batch page says cached tokens in batch are *"priced the same as for non-batch mode traffic,"* yet its pricing page lists a halved batch cache-read price for Gemini 3.8 Flash. When it matters, run a small batch and read the invoice.

### Flex: batch prices, synchronous calls

OpenAI and Google both offer a **flex** service tier: a normal synchronous request with `service_tier: "flex"`, billed at roughly batch prices, but *sheddable*. When capacity is tight you get a `429` (OpenAI: not charged) or a `503`/`429` (Google: no server-side fallback) and you retry later. Google targets 1–15 minutes of latency. Raise your client timeouts (OpenAI suggests 15 min; Google suggests 10+) and fall back to the standard tier if you need to.

Flex suits background agents and pipelines where you'd rather not build batch file handling but can tolerate minutes of delay.

### What batch is good (and bad) for

**Good fits:**
- Offline evals and regression suites across thousands of prompts
- Classification, tagging and extraction over existing data (tickets, logs, transactions, documents)
- Generating embeddings or synthetic data, and enrichment of historical records
- Nightly reports, summaries, content moderation backfills
- LLM-as-judge scoring of model outputs

**Bad fits:**
- Anything a user is waiting on
- Multi-step agent loops where step N+1 depends on step N (each step would wait up to 24 h)
- Work with a hard deadline shorter than the expiry window
- Tiny jobs where the batch file handling costs more engineering time than the 50% saves

## Part 6: Cache busters, the bugs nobody notices

This is where the real money goes. Every item below produces correct responses and passes code review while quietly sending your cache hit rate to zero. Each one boils down to the same thing: **some byte in the prefix changed.**

### 1. The timestamp at the top of the system prompt

The classic.

```python
# ❌ Busts the cache on every call (the prefix changes every second)
system = f"""You are SupportBot. Current time: {datetime.now().isoformat()}
<20,000 tokens of policies and examples>"""

# ✅ Stable prefix first; volatile context goes at the end, after the breakpoint
system = "You are SupportBot.\n<20,000 tokens of policies and examples>"
messages = [{"role": "user", "content": f"(Current date: {date.today()})\n\n{question}"}]
```

Even `date.today()` at the top is a trap. It busts the cache **once a day at midnight UTC**, which looks like a mysterious daily cost spike. Time-of-day greetings, "days until end of quarter" and countdowns all count.

### 2. Request IDs, trace IDs and nonces in the prompt

`uuid4()`, `request_id`, a correlation ID "for debugging" in the system prompt, a random seed in an instruction. Unique per request means never cacheable. Put them in **request metadata or headers**, not in the prompt text.

### 3. JSON with non-deterministic key order

You serialize a config or user profile into the prompt. The data is identical, but the bytes aren't:

```python
# Same data, different bytes, different cache entries
json.dumps({"plan": "pro", "region": "eu"})   # '{"plan": "pro", "region": "eu"}'
json.dumps({"region": "eu", "plan": "pro"})   # '{"region": "eu", "plan": "pro"}'

# ✅ Canonical serialization
json.dumps(obj, sort_keys=True, separators=(",", ":"), ensure_ascii=False)
```

Why the key order changes when "nothing changed":

- **Python dicts keep insertion order,** so the order depends on *how the dict was built*. `{**defaults, **overrides}` versus `{**overrides, **defaults}`, the order fields were set across a code path, a dict rebuilt from a DB row whose column order changed in a migration, or an ORM's `to_dict()` that changed in a library upgrade.
- **JavaScript objects** keep insertion order *except* integer-like keys, which always come first in numeric order. `{"b":1,"10":2,"a":3}` serializes as `{"10":2,"b":1,"a":3}`. Mixing numeric IDs and string keys gives orderings that surprise people.
- **Merging API responses:** objects from upstream services can reorder fields between deploys. Your prompt inherits their non-determinism.
- **Sets:** iterating a Python `set` of strings can produce a different order **across processes**, because string hashing is randomized per process (`PYTHONHASHSEED`). Your local tests pass and production pods disagree with each other. Use `sorted(...)`.
- **Formatting differences:** `json.dumps` defaults to `", "` and `": "` separators, while a JS `JSON.stringify` produces none. Two services rendering "the same" prompt produce different bytes. `indent=2` versus compact, `ensure_ascii=True` versus `False` (`"é"` vs `"é"`), and float formatting (`1.0` vs `1`, `1e-05` vs `0.00001`) all matter.

Go's `encoding/json` already sorts map keys, which is one of the few cases where the default is safe. Struct fields, though, follow struct declaration order, so reordering fields in a refactor busts the cache.

### 4. Tool definitions that change order or content

Tools render **first** (Anthropic: `tools` → `system` → `messages`; OpenAI includes them in the routing hash). So any change to tools invalidates *everything*:

- Building the tool list from a dict or registry whose iteration order varies → **sort tools by name.**
- Giving each user or plan a different tool set (`if user.is_admin: tools.append(...)`) → every combination is its own cache. Prefer one superset of tools and enforce permissions in your tool handler.
- Tool schemas generated from Pydantic or Zod models: a library upgrade that reorders properties, adds a `title`, or changes `$defs` handling rewrites every schema byte.
- Adding or removing a tool mid-conversation, or swapping tool sets for "modes", invalidates the whole cache.
- Descriptions with interpolated values ("Search the catalog of **{n_products}** products").

### 5. Conditional sections in the system prompt

```python
system = BASE
if user.locale != "en": system += LOCALE_RULES[user.locale]
if feature_flags.new_tone: system += NEW_TONE
if user.tier == "enterprise": system += ENTERPRISE_RULES
```

Three flags make 8+ distinct prefixes, each with its own cache that's colder because traffic is split. Order sections by **how often they change**: universal content first, then per-tenant, then per-user, then per-request. Put a breakpoint at each boundary (Anthropic allows 4).

Personalization at the top ("You are helping **Priya** from **Acme**") has the same problem. It makes the cache per-user when it could be shared across all users.

### 6. Putting the question before the document

```text
❌  [instructions] [user question] [50K-token document]
✅  [instructions] [50K-token document] [user question]
```

Same tokens, same answer quality, very different bill. The document can only be reused if everything before it is identical, and the question is different every time.

### 7. Rewriting conversation history

Multi-turn caching works because each turn *appends* to the previous prefix. Anything that edits earlier turns breaks that:

- **Sliding-window truncation:** dropping the oldest message to stay under a budget changes position 0 of the message history, so the **whole conversation** misses on every turn after the window fills. Truncate in large, infrequent chunks (or summarize once) instead of trimming one message per turn.
- **Re-rendering old messages:** re-templating history from your DB with a new format, normalizing whitespace, or redacting PII after the fact.
- **Injecting and removing "reminder" messages** each turn. Removing a previously sent message is a history edit.
- **Re-serializing tool results** differently on replay (see JSON above).

### 8. Non-deterministic retrieval (RAG)

Your vector search returns the same 5 chunks in a different order because of score ties, approximate-nearest-neighbor non-determinism, or index rebuilds. Sort retrieved chunks by a stable key (document ID + chunk index) after you've chosen the top-k. And put retrieved context **after** the static instructions, never before them.

### 9. Invisible byte differences

- Trailing whitespace or a trailing newline added by one template engine and not another
- `\r\n` vs `\n` (a prompt file edited on Windows)
- Unicode normalization: `é` as one code point (NFC) vs `e` plus a combining accent (NFD). They look identical and are different bytes.
- Smart quotes introduced by a CMS or a doc editor
- A prompt loaded from a file whose contents changed in a deploy you didn't think touched prompts

### 10. Images and files re-encoded on each request

Re-resizing an image, re-compressing a JPEG, or re-encoding a PDF produces different bytes even when it looks the same. Encode once, store the bytes or a file ID, and reuse exactly that.

### 11. "Harmless" parameter changes

The cache key covers more than your text:

- **Model:** caches are model-scoped. A cascade that sends some traffic to a cheaper model splits your cache.
- **OpenAI:** `tools`, `parallel_tool_calls`, the structured-output schema (`text.format`), `reasoning.effort` and `text.verbosity` are all part of the prefix. Tweaking reasoning effort per request or reshuffling your JSON schema busts it.
- **Anthropic:** changing thinking or effort settings invalidates the messages cache (and on some models more than that). Switching `tool_choice` keeps the tools and system caches but invalidates the messages cache. Pin these settings per route instead of varying them per request.

### 12. Doing everything right and still missing

Not a bug in your prompt, but just as invisible:

- **Below the minimum length:** a 3,000-token prefix caches on Claude Opus 5 (min 512) but **silently doesn't** on Haiku 4.5 (min 4,096), and doesn't qualify for Gemini 3.x implicit caching (min 4,096).
- **TTL gaps:** traffic every 7 minutes against a 5-minute TTL is a 0% hit rate. Use the longer TTL, or (on Anthropic) send a cheap keep-alive request before the cache expires.
- **Split traffic:** two Anthropic workspaces, or OpenAI keys with too much traffic (more than about 15 RPM per prefix+key on pre-5.6 models), mean separate or overflowing caches.
- **Breakpoint on the unique tail:** caching the *whole* prompt, including the per-request question, writes a new cache entry every time and never reads one. On Anthropic you pay the 1.25× write premium for nothing. Put the breakpoint at the end of the **shared** part.

### Test yourself

Think you'd catch these in code review? Each round shows two consecutive requests. Find where the cache breaks, or decide that it doesn't.

```game
cache-buster
```

## Part 7: Fingerprinting — measure before you cache

Everything in Part 6 is easy to recognize once it has been pointed out and hard to spot in a live system. In production you don't want to guess; you want answers to four questions:

1. Which parts of my prompt are actually stable, and how stable?
2. Where should the cache breakpoints go?
3. When the cache misses, which part changed?
4. What hit rate and savings should I expect *before* I change anything?

**Prompt fingerprinting** answers all four without logging a single prompt. It works the same for every provider, and you can run it before you've turned caching on at all.

### The idea: hash each segment, and hash the prefix

Split each request into its ordered **segments** (model and settings, each tool, each system block, retrieved docs, each message) and record two hashes per segment:

- **Segment hash:** the hash of that segment alone. It tells you *what* changed.
- **Prefix hash:** a running hash that chains in every segment before it. This mirrors how the provider's cache works: two requests share a cacheable prefix up to segment N exactly when their prefix hashes at N match.

```mermaid
flowchart TB
  subgraph A["Request A"]
    direction LR
    A1["tools<br/>seg 9f2c"] --> A2["system<br/>seg 41ab"] --> A3["docs<br/>seg 7d10"] --> A4["question<br/>seg c3e8"]
  end
  subgraph B["Request B"]
    direction LR
    B1["tools<br/>seg 9f2c"] --> B2["system<br/>seg 41ab"] --> B3["docs<br/>seg 55f0"] --> B4["question<br/>seg 0b9a"]
  end
  A ~~~ B
  classDef changed stroke:#f87171,stroke-width:2px
  class B3,B4 changed
```

Requests A and B have the same prefix hashes for tools and system, then diverge at docs. So the cacheable prefix they share is tools + system, and the segment hash at the divergence point tells you exactly which part changed.

### A minimal implementation

```python
import hashlib, hmac, json, os

FP_KEY = os.environ["PROMPT_FP_KEY"].encode()  # keyed hash: safe to log, not reversible by guessing

def _h(data: bytes) -> str:
    return hmac.new(FP_KEY, data, hashlib.sha256).hexdigest()[:16]

def _strip_markers(obj):
    # cache_control moves every turn; it is not a cache buster, so leave it out of the fingerprint
    if isinstance(obj, dict):
        return {k: _strip_markers(v) for k, v in obj.items() if k != "cache_control"}
    if isinstance(obj, list):
        return [_strip_markers(v) for v in obj]
    return obj

def segments_for(payload: dict) -> list[tuple[str, object]]:
    """Ordered (label, value) pairs, in the order the provider renders them."""
    segs = [("config", {k: payload.get(k) for k in ("model", "thinking", "output_config", "tool_choice")})]
    segs += [(f"tool:{t['name']}", t) for t in payload.get("tools", [])]
    system = payload.get("system", [])
    segs += [(f"system[{i}]", b) for i, b in enumerate(system if isinstance(system, list) else [system])]
    segs += [(f"msg[{i}]:{m['role']}", m) for i, m in enumerate(payload["messages"])]
    return segs

def fingerprint(payload: dict) -> list[dict]:
    chain, out = "", []
    for label, value in segments_for(payload):
        value = _strip_markers(value)
        # Hash the bytes you SEND. Plain json.dumps keeps insertion order, like your HTTP client does.
        raw = value if isinstance(value, str) else json.dumps(value, ensure_ascii=False)
        seg = _h(raw.encode("utf-8"))
        chain = _h((chain + seg).encode())
        out.append({"label": label, "seg": seg, "prefix": chain, "chars": len(raw)})
    return out
```

Call `fingerprint(payload)` right before every API call and log the result next to a timestamp, the route, a tenant ID, and the cache numbers from the response.

Two details matter more than they look:

- **Don't canonicalize inside the fingerprint.** It's tempting to `sort_keys=True` here, but then the fingerprint would hide the very key-order bugs you're hunting. Hash what actually goes over the wire, in the order it goes. The only thing to strip is the `cache_control` marker, which moves forward every turn by design.
- **Use a keyed hash (HMAC), not plain SHA-256.** Short, predictable segments like `{"plan": "pro"}` can be recovered from a plain hash by guessing. With a secret key, the logs are safe to keep alongside other telemetry.

Adapt `segments_for` to each provider's request shape: `instructions`/`input` items on OpenAI's Responses API, `system_instruction`/`contents` on Gemini. Put any setting that is part of the cache key into `config` (OpenAI's `text.format` schema and `reasoning.effort`, for example).

### Reading the logs

**1. Stability profile: where to put breakpoints.** Group by segment label and count distinct segment hashes over a day, and per tenant:

| Segment | Distinct hashes / 24h | Distinct per tenant | What it tells you |
|---|---|---|---|
| `config`, `tool:*` | 1 | 1 | Changes only on deploy. Cache it. |
| `system[0]` (policies) | 1 | 1 | Same. **Breakpoint 1 goes here.** |
| `system[1]` (tenant KB) | 37 | 1 | Stable per tenant. **Breakpoint 2 goes here.** |
| `system[2]` (user profile) | 4,112 | 118 | Per user. Move it after the breakpoints. |
| `msg[*]` | every request | every request | Volatile. Rely on conversation caching. |

*(Illustrative numbers.)* The rule: put a breakpoint after the **last segment that is stable for the traffic you want to share it**. A segment that looks static but has thousands of distinct hashes is a bug report: someone is interpolating per-user or per-request data into it.

**2. First divergence: which part changed.** Within a conversation, every request should start with the previous request's prompt, unchanged. Compare consecutive fingerprints:

```python
def first_divergence(prev: list[dict], curr: list[dict]) -> str | None:
    for a, b in zip(prev, curr):
        if a["prefix"] != b["prefix"]:
            return b["label"]  # the cache buster lives in this segment
    return None  # curr cleanly extends prev
```

Count the results across conversations. A healthy route returns `None` almost every time. If `system[0]` shows up in 100% of comparisons, you have a timestamp or ID in the system prompt. If `msg[0]` appears only once a conversation passes 40 messages, your history truncation is trimming one message per turn. If `tool:*` labels show up in a different *order* between requests, your tool list isn't sorted.

**3. Simulate the hit rate before turning caching on.** Replay the log against each provider's cache lifetime and minimum size:

```python
def simulate(requests: list[dict], ttl_s: float, min_chars: int) -> float:
    """Optimistic estimate: one machine, a cache entry at every segment boundary, no eviction."""
    last_seen: dict[str, float] = {}
    cached = total = 0
    for req in sorted(requests, key=lambda r: r["ts"]):
        size = hit = 0
        for seg in req["fingerprint"]:
            size += seg["chars"]
            t = last_seen.get(seg["prefix"])
            if t is not None and req["ts"] - t <= ttl_s and size >= min_chars:
                hit = size  # longest still-warm prefix so far
            last_seen[seg["prefix"]] = req["ts"]  # written or refreshed
        cached += hit
        total += size
    return cached / total

for name, ttl in [("5 min", 300), ("30 min", 1800), ("1 hour", 3600)]:
    print(name, f"{simulate(log, ttl, min_chars=4 * 1024):.0%}")  # ~4 chars per token
```

That gives you the decision in numbers: if the 5-minute and 1-hour rates are nearly the same, your traffic keeps a 5-minute cache warm on its own and the 1-hour write premium is wasted. If the 1-hour rate is far higher, your gaps are longer than 5 minutes. For explicit caching, only record `last_seen` at the labels where you'd place breakpoints. Treat the result as an upper bound: it ignores routing, overflow and eviction.

**4. Check your prompt against what the provider reports.** Compare the prefix size your fingerprints say should be cached with what the provider actually reports (`cache_read_input_tokens`, `cached_tokens`, `usage_metadata`). The gap tells you where to look:

- **Fingerprints say "should hit" and the provider reports a hit:** healthy.
- **Fingerprints diverge early:** your prompt is the problem. The first-divergence label tells you which segment.
- **Fingerprints match but the provider misses:** your prompt is fine. Look at TTL expiry, the minimum cacheable length, traffic split across Anthropic workspaces, or OpenAI keys overflowing past roughly 15 requests per minute.

### Tools that complement fingerprinting

- **Anthropic cache diagnostics (beta).** Send the beta header on every request, pass the previous response's ID, and the API tells you where two requests diverged: model, system, tools or messages. Coarser than per-segment fingerprints and Claude-only, but zero setup.
- **Payload diffs.** Once fingerprinting tells you *which* segment changed, diff two logged payloads (from a sampled, access-controlled log) to see the exact characters.
- **Snapshot tests in CI.** Build the same prompt twice, in separate processes, and assert the bytes are identical. Different processes catch Python's per-process set ordering, which a single-process test misses.
- **Lint rules** for `datetime.now`, `uuid`, `random`, `json.dumps` without `sort_keys`, iterating sets, and f-strings with user fields in anything that builds a system prompt.
- **Cache dashboards.** Alert on the hit rate per route. A cache regression is a cost regression, and it should page someone like a latency regression would.

## TL;DR

- **Caching skips prefill.** Reads cost about 10% of the input price (OpenAI says up to 95% off; some Claude models go below 10%). The catch is that it's **exact-prefix only**, because attention is causal.
- **Defaults differ:** OpenAI and Gemini cache implicitly; Claude caches only when you add `cache_control`. Explicit caching (Claude breakpoints, OpenAI GPT-5.6+ breakpoints, Gemini `cachedContents`) gives you predictability at the cost of write or storage fees.
- **Routing matters:** caches live on particular machines. On OpenAI, use a stable `prompt_cache_key` and keep about 15 RPM per key on pre-5.6 models.
- **Batch = 50% off** for work that can wait up to 24 h (48 h on Gemini). Results arrive out of order, partial results are normal, and validation errors show up late. Flex gives similar pricing synchronously, best-effort.
- **Most cache misses are self-inflicted:** timestamps, IDs, unsorted JSON, tool reordering, per-user system prompts, history truncation, re-encoded images. **Stable first, volatile last, serialize canonically, and watch your cache counters.**
- **Watch request size:** above 272K input tokens (OpenAI) or 200K (Gemini Pro), input and cached-input rates double and output costs 1.5×. Current Claude models are flat up to 1M. Log input tokens per request.
- **Measure, don't guess:** fingerprint each prompt segment with a keyed hash. Change rates tell you where to put breakpoints, the first divergence names the cache buster, and replaying the log predicts your hit rate before you turn caching on.

## Resources

Everything in this post is packaged as a set of [Claude Code](https://claude.com/claude-code) skills in a public repo: [Rahul-s-007/resources](https://github.com/Rahul-s-007/resources/tree/main/token-engineering). A main skill routes to five focused ones: cache audit, cache setup, prompt fingerprinting (with the scripts from Part 7), cost estimates that fetch live prices instead of hardcoding them, and a batch planner.

```text
/plugin marketplace add Rahul-s-007/resources
/plugin install token-engineering@rahul-resources
```

## Sources

Read from the official docs on October 4, 2026:

**Anthropic**
- [Pricing](https://platform.claude.com/docs/en/about-claude/pricing)
- [Prompt caching](https://platform.claude.com/docs/en/build-with-claude/prompt-caching)
- [Batch processing](https://platform.claude.com/docs/en/build-with-claude/batch-processing)

**OpenAI**
- [Prompt caching](https://developers.openai.com/api/docs/guides/prompt-caching)
- [Batch API](https://developers.openai.com/api/docs/guides/batch)
- [Flex processing](https://developers.openai.com/api/docs/guides/flex-processing)
- [Pricing](https://developers.openai.com/api/docs/pricing)

**Google**
- [Context caching](https://ai.google.dev/gemini-api/docs/generate-content/caching)
- [Batch Mode](https://ai.google.dev/gemini-api/docs/batch-mode)
- [Flex inference](https://ai.google.dev/gemini-api/docs/flex-inference)
- [Pricing](https://ai.google.dev/gemini-api/docs/pricing)