Prompt caching: the AI cost comparison that a token price misses
How to evaluate repeated AI workloads using Gemini's caching documentation, a worked cost example and an acceptance-based comparison method.
CryptoXAI Editorial · Published 2026-09-10 · 6 min read · Source-based analysis, AI-assisted
A fictional cost comparison
Illustrative arithmetic only, not provider prices or measured results.
| Measure | Configuration A | Configuration B |
|---|---|---|
| Attempted tasks | 20 | 20 |
| Total API cost | $0.40 | $0.30 |
| Accepted tasks | 16 | 10 |
| API cost per accepted task | $0.025 | $0.03 |
| Still to measure | Retries and review effort | Retries and review effort |
The repeated document problem
Imagine an assistant answering several questions about the same long manual. Much of the input stays constant while the question changes. A comparison based only on a headline token price may miss how that repeated workload behaves. Conversely, one unusually cheap repeat request can make a system look better than an entire day of mixed work.
This is a measurement guide, not a cross-provider benchmark. We have not run the controlled experiments described here. The aim is to give buyers and builders a reusable method that keeps correctness, operating effort and cost together. A provider's documented capability starts the investigation; observed performance on your task should inform the eventual choice.
What Google's documentation supports
Google's Gemini API caching guide describes implicit caching for repeated input and says savings are applied when requests hit the cache. It lists model-specific minimum input sizes and recommends placing substantial shared content near the beginning of a prompt and sending similar prefixes close together. These are useful conditions to investigate, not a promise that every repeat request will hit.
The guide distinguishes API surfaces: Interactions supports implicit caching, while explicit cache management requires generateContent. This source was checked on 10 September 2026. Verify the exact model and interface before implementing a workflow; documentation for one API should not be treated as evidence that another exposes identical controls.
- Establish baselineMeasure first questions and new documents.
- Repeat contextTrack cache usage and traffic conditions.
- Accept or rejectReview answers against fixed criteria.
- CalculateDivide total workflow cost by accepted tasks.
Design a representative comparison set
Start with four kinds of task: the first question about a document, follow-up questions about that document, a question about an unrelated document, and a question after the original document changes. Use material that resembles your actual workload. A benchmark consisting entirely of repeated content will answer a narrower question than a support assistant facing varied enquiries.
Write acceptance criteria before running the tasks. For a document assistant, a useful answer might need to be correct, identify the relevant passage and admit when the document does not contain the answer. Keep those requirements consistent across tools. An inexpensive answer that confidently invents a policy should not count as a successful result.
Record the model, product, API, configuration and date. Keep the same questions and documents across candidates while documenting any interface differences. If one product requires additional preparation, include that effort. Do not compare a carefully tuned implementation with another tool's first attempt and present the difference as model capability alone.
Worked example: why cost per answer can mislead
The following numbers are invented arithmetic examples, not Gemini prices or measured provider results. Suppose configuration A processes twenty requests for $0.40 and sixteen answers pass review. Its API cost per accepted answer is $0.025. Configuration B processes the same twenty requests for $0.30, but only ten pass. Its API cost per accepted answer is $0.03.
B has the lower total bill and lower cost per attempted request. A has the lower cost per accepted answer. Neither figure tells the full operational story, because review effort and the consequences of failure may differ. The useful comparison reports the denominator rather than letting the reader assume every output was valuable.
Now suppose the rejected answers need retries. Add the retry cost to the total and count accepted outcomes carefully. If a retry corrects an earlier failure, do not count the attempted answers as separate completed tasks. If the user abandons the task, retain its incurred cost. This method prevents unsuccessful work disappearing from a flattering headline.
Separate first requests from repeat traffic
Keep separate result groups for first requests and repeated context. Then combine them using the mix you expect to operate. In another fictional example, a system with mostly new documents faces a different workload from one answering hundreds of questions about a stable handbook. A result from one mix cannot establish the cost of the other.
Run the same mixture at more than one time pattern if timing is relevant to the documentation. A burst of near-identical requests is not the same workload as enquiries spread across a working day. Label the conditions clearly. The goal is to estimate your application's behaviour, not to manufacture the largest possible saving.
Capture reported cache usage where the API provides it, along with input and output usage and the charged or calculated amount. Keep calculations distinct from invoices. If you cannot establish whether caching occurred, do not attribute a cost difference to it. Mark that part of the explanation as unverified.
Include the human work
For each task, record minutes spent preparing context, reviewing the answer and repairing errors. Decide whether these are included in a monetary estimate or reported separately. If you apply a labour rate, show the assumption. A reader should be able to change it and see whether your conclusion still holds.
A useful pilot result can show both API cost per accepted task and review minutes per accepted task. Those measures may favour different candidates. For low-consequence internal drafting, a team might tolerate more review. For a workflow where a wrong answer causes a significant problem, correctness and escalation behaviour may be decisive constraints.
Set critical requirements as pass-or-fail checks before optimising cost. Do not allow a low average bill to conceal an unacceptable failure mode. The requirements depend on the actual application; this article does not certify any tool for a regulated or sensitive use.
A practical build-or-buy decision
If you are comparing finished AI tools, test the workflow those products actually expose. A tool may manage context on your behalf, leaving fewer implementation choices to measure directly. Record that limitation. Do not assume an API capability is controllable through the consumer interface of a product using the same brand.
If you are building an application, measure your integration as part of the system. Context assembly, retries, logging and output validation affect whether the result is usable. Start with a correct baseline, then change one thing at a time. If several settings change together, you may improve the outcome without knowing which change caused the improvement.
Our conditional recommendation is to choose the configuration that satisfies the written acceptance standard at an acceptable total operating cost. A cache-friendly design is worth investigating when your workload repeats input. It is not a reason to compromise answer quality or assume the same gain on unrelated tasks.
Turn the result into an honest comparison page
Publish the task set, sample size, versions, dates and calculation method with the conclusion. Include rejected outcomes and identify missing measurements. Distinguish a documented feature from an observed result. Readers should be able to tell whether a statement came from provider documentation, your own experiment or a proposed framework such as this one.
Revisit the evaluation when prompts, models, documents or traffic patterns change. Keep the earlier result dated rather than silently replacing its assumptions. A specific conclusion—this configuration improved measured cost on these tasks while meeting this standard—is more useful than an unsupported best-tool ranking, because another team can test whether the same conditions apply.
Sources & method
Linked sources support the documented facts. Evaluation frameworks are CryptoXAI analysis; hypothetical examples are labelled. No hands-on benchmark results are claimed. An expansion date records an editorial change, not a new verification of every source.