Cutting Inference Cost by 36% with Prompt Caching
We built a human-in-the-loop ReAct agent for a gaming company, handling VIP customer support. VIP support is not a FAQ bot. Before the agent can say anything useful, it needs to know who it's talking to: their play persona, account standing, lifetime value, which deals they're eligible for, what happened the last four times they contacted support, and which human agent persona it should speak as. That's a lot of context per conversation, and almost none of it is optional.
The agent worked. Then we put it on production traffic and looked at the bill.
Cost per interaction came in above what we'd modeled. Not a crisis, but enough to become one at the volume they wanted to reach. We hadn't done any optimization yet, on purpose: there's no point tuning a system whose behavior you haven't settled. Now the behavior was settled. So we started where you should always start, with the prompt.
Two passes later the bill was down 36%, with no measurable change in output quality.
Here's what we did, in order, and what didn't work.
What you're actually paying for
The thing that usually takes a while to sink in about ReAct agents: you re-send the entire prompt on every model call.
One "interaction" is one message to the user. To the API it's a loop. The agent reads the context, decides to call a tool, gets a result, then reads everything again plus that result, and decides what to do next. A conversation with two tool calls is three model requests. Request three carries the full system prompt, the full tool schema, the full player profile, and the whole conversation so far.
That fixed preamble is billed at full input price every time. It doesn't matter that it's byte-for-byte identical to what you sent nine seconds ago.
Our prompt had a natural three-tier structure, which turned out to matter a lot:
The rough token split looked like this:
| tier | approx. tokens | changes | re-sent per request |
|---|---|---|---|
| 1 — fixed | ~14,000 | never | always |
| 2 — ticket context | ~3,500 | per ticket | always |
| 3 — turn context | ~800 and growing | per turn | always |
Tier 1 is 76% of the input on the first request of a ticket. It's the same bytes as the last ticket's tier 1, and the one before that, and every ticket we will ever run. We were paying full price for it a few hundred thousand times a day.
Pass one: make the prompt smaller
Before caching anything, shrink it. Caching a bloated prompt just stores the bloat more cheaply.
Three things help:
- Don't repeat yourself. This is the big one, and it's easy to miss, because prompts don't get written so much as pile up. Someone hits an edge case and adds a rule. Someone else hits a similar case in a different section and adds a similar rule, worded slightly differently. Six months later the same instruction sits in three places and two of them disagree. That's not just wasted tokens. It's a behavior bug you haven't noticed.
- Say it once, plainly. Models love to pad prompts, so if you've been iterating with a model's help, yours is padded. "It is absolutely critical that you must always ensure that you verify" is nine tokens of nothing.
- Prefer rules over examples. A worked example is expensive and narrow. A stated rule is cheap and general. Keep examples only where the output format is tricky and a rule can't pin it down.
Our approach was deliberately lazy: let the model compact it first, then review the important logic by hand. Feed the prompt back with "remove redundancy and simplify phrasing, change no behavior," then read the diff carefully. The model will happily delete a load-bearing constraint that looked like filler. The mechanical part is automatable. The judgment isn't.
That got us 15% fewer tokens, on prompts we already thought were tight. If you've never done this pass, expect more.
This is the best kind of saving, because it's free and it shrinks the base that every later optimization works on.
How prompt caching is priced
You can't reason about any of what follows without the pricing model, so:
Caching is a prefix match. The provider hashes your prompt from the start up to a marker you place. On the next request, if the bytes before that marker are identical, the prefix is served from cache. Change one character anywhere in the prefix and everything after it is invalidated.
The render order is tools → system → messages. That's why the diagram above is drawn the way it is.
The economics, for Claude on Bedrock:
| cost, relative to normal input | |
|---|---|
| cache write (5-minute TTL) | 1.25× |
| cache write (1-hour TTL) | 2× |
| cache read | ~0.1× |
| no cache | 1× |
Reads are nearly free. Writes cost extra. So caching is a bet: pay 25% more once, to pay 90% less every time after. And like any bet, you can lose it. Write an entry that never gets read and you've simply paid 25% more for that request.
The break-even is simple arithmetic. For a segment read times after one write:
So on paper you break even after barely more than one reuse. Remember that number. It's about to mislead us.
Two more mechanics worth knowing:
- There's a minimum cacheable prefix. It depends on the model, somewhere in the 512–4096 token range. Below it, nothing caches. You get no error, just a cache-write count of zero while you wonder why your dashboard never moves.
- On Bedrock you place cache breakpoints yourself. The first-party API has an automatic mode that caches the last cacheable block. Bedrock doesn't. You get at most four breakpoints and you decide where they go. That's arguably a good thing, since it forces you to think about your tier boundaries.
Pass two: caching, and the tier that didn't pay
The obvious move was to cache tier 1 and tier 2. One breakpoint at the end of the fixed block, one at the end of the ticket context. Tier 3 stays below the last marker, where volatile content belongs.
We expected close to half the remaining input cost to disappear.
After a few days of production traffic, tier 1 was working as intended. Tier 2 was doing almost nothing.
Here's why.
Go back to that break-even of . The number is right, but it answers the wrong question. The question isn't "does this segment break even?" It's "how many times does this particular cache entry get read before it expires?"
For tier 1, the answer is: a lot. One write, then every request from every ticket for the next five minutes reads it. Over a busy hour that's thousands of reads per write, and the write premium disappears into the noise.
For tier 2, the answer is: as many requests as this one ticket makes. The entry is written when the ticket starts and is worthless the moment it ends, because the next ticket is a different player. And our agent usually made only 1–2 tool calls per ticket. That's the number we hadn't taken seriously enough.
1–2 tool calls means 2–3 model requests, which means 1–2 reads per write.
| requests/ticket | uncached | cached | saving on tier 2 |
|---|---|---|---|
| 2 | 2.00× | 1.35× | 33% |
| 3 | 3.00× | 1.45× | 52% |
| 10 | 10.00× | 2.15× | 79% |
The saving is real. 33% at two requests isn't nothing. But tier 2 is only about a fifth of the input, so we were claiming a third of a fifth. Call it 6% of input on a good day. Against that: more complexity, one of our four breakpoints spent, and one more thing that breaks silently when someone reorders a field in the player profile serializer.
We kept the tier 1 breakpoint and dropped the tier 2 one.
Pass two came out at 25% off what pass one left us.
That's where the 36% comes from. Compounding percentages is where cost-saving posts tend to get sloppy, so here it is in full:
15% off the original, then another 25% off what remained, which is 21 points of the original bill. 36% total, with the same outputs and the same eval scores.
The one we left on the table
Once we had cache metrics on a dashboard, something else showed up: during slow hours, the cache hit rate collapsed.
Obvious in hindsight. The default TTL is five minutes. At peak, tickets arrive constantly and every request refreshes the tier 1 entry, so it never expires. At 3am, tickets arrive every ten or fifteen minutes. Every ticket pays for a fresh cache write, gets one or two reads out of it, and the entry dies before the next ticket arrives.
Bedrock offers a 1-hour TTL at a 2× write premium instead of 1.25×. Here's a slow hour, assuming tickets 12 minutes apart at 3 requests each:
| writes/hour | reads/hour | relative cost | |
|---|---|---|---|
| 5-minute TTL | 5 × 1.25 | 10 × 0.10 | 7.25× |
| 1-hour TTL | 1 × 2.00 | 14 × 0.10 | 3.40× |
Under the 5-minute TTL every ticket writes its own entry and gets two reads from it. Under the 1-hour TTL, one write covers the whole hour. That's roughly a 2× improvement on tier 1 off-peak.
Which sounds good until you notice it's 2× on the cheapest hours of the day. And making the same switch at peak would cost us money: a 2× write premium for an entry that a 1.25× write would have kept alive anyway.
Doing it properly means picking the TTL based on traffic rate. That's a scheduler, a metric, and a new failure mode. We haven't built it, and as the client scales, off-peak stops existing and the whole thing becomes pointless. That's a good reason not to build something.
I'm mentioning it because your traffic might look nothing like ours. If you run bursty, low-volume, or scheduled batch workloads with long gaps between calls, the 1-hour TTL is probably the biggest win available to you.
Do these first
Everything above assumes the boring things are already done. We do them by default on every build, which is why caching was the first thing left to optimize rather than the first thing we reached for. If that's not true for you, start here instead. These are cheaper to implement than caching, and they stack with it.
Consider a cheaper model. The best token optimization is a lower price per token. Route the easy paths to something small and fast: classification, routing, extraction, "does this need a human." Save the expensive model for the turn that actually needs to reason. In a ReAct loop that's often most of your calls.
Drop JSON where you actually have the choice. JSON is a wasteful format to pay per token for. Every key repeats on every object, and you're billed for each brace, bracket, quote, and comma. YAML drops most of the punctuation. TOON goes further for tabular data by lifting the keys into a header row, so a hundred-row result carries its field names once instead of a hundred times.
The catch is that you can't apply this everywhere. Tool calls are JSON by definition: the model fills in your input_schema, and the same is true for structured outputs and strict tool use. Fighting that is a bad trade. You'd give up schema validation, and you'd be pushing the model off the format it's most reliable at producing, to save tokens on what is usually the smallest part of the payload anyway.
Where you do have a choice is tool results, and any bulk data you paste into the prompt. Nothing constrains their format, and they're usually much bigger than the calls that produced them. That's also where the savings compound: a tool result gets pushed back into context on every later request in the loop, so a verbose one is a cost you keep paying for the rest of the conversation.
Load context lazily. Tier 2 is the interesting case. We front-load the whole player profile because we assumed the agent would want it. Does it? For plenty of tickets it never touches half of it. The alternative is a get_player_details tool, or a skill that loads on demand, fetching the expensive fields only when the agent asks. You trade a guaranteed cost on every ticket for an occasional extra round trip.
We haven't measured this properly yet. I suspect it beats caching tier 2 comfortably, for the same reason tier 2 caching failed: content that's expensive and rarely read shouldn't be in your prefix at all.
Conclusion
Two things to take away.
First, caching isn't a switch you flip. It's how you lay out the prompt. Stable content first, changing content last, breakpoints in between. Add cache_control to a prompt with a timestamp in the header and nothing will ever be read back, because the prefix is different every time. You won't get an error. You'll just write a fresh entry on every single call, at 1.25× — which is 25% more than you were paying before you turned caching on.
Second, what matters is how often a block is read, not how big it is. The instinct is to cache the biggest block. Cache the most-read one instead. Often they're the same block. When they aren't, like our tier 2, you get the complexity and none of the saving. So before adding a breakpoint, ask how many times that entry gets read before it expires. One or two? Skip it.
And before any of that, read your prompt. Top to bottom. Ours had 15% dead weight in it and we thought it was tight. Yours probably does too.
Related: How to Build ReAct Agents in 2026 · Measure What Counts: The AI Engineering Approach to Agent Evaluation