The model bill grows faster than the value
You picked the strongest model available because it worked. Then volume grew, prompts got longer, and the invoice started showing up in reviews. The question has changed from whether the model works to whether every call needs that model.
Most teams answer that by feel. A little measurement answers it better, and the first measurement is not a price.
Set the accuracy bar before you compare prices
OpenAI’s model selection guide treats cost and latency as secondary, because a model that misses your accuracy target makes those concerns moot. Fix accuracy first, then optimize cost inside that constraint (OpenAI’s model selection guide).
The Overqualification study applies the same rule from the other side. Set the acceptable miss rate first, then compare prices only among models that meet it. Its allergen detection task shows why. Cheap-tier models answered NO on a median 14.5% of lines that definitely contained gluten, dairy or seafood, most at stated confidence of 95 or higher, while the frontier tier did so 0 times out of 1,972 (the Overqualification study).
Where the work is routine, the arithmetic reverses. Across 9 small language models and 29 variants tested against a customer-facing GPT-4 implementation, researchers reported competitive quality, more consistent performance and a cost reduction of 5x to 29x (Scaling Down to Scale Up).
Cost per answer you can use
Cost per token is the supplier’s number. Yours is cost per answer you can use. Divide the price of a call by the probability the answer is correct, then add the cost of a wrong answer where the task carries risk.
A decision-theoretic model of LLM selection uses the same inputs: cost per token, probability of success on the task, and the gain and loss attached to using the model. It finds that higher accuracy from a more expensive model can justify the larger spend through bigger earnings, though not necessarily a better return on investment (The Economic Implications of Large Language Model Selection).
A checklist you can run on Monday
- Write the accuracy target down as a number, with the miss rate this task can tolerate.
- Sample real traffic and keep the current answers as a baseline. Test prompts flatter cheap models.
- Run the cheap tier on those same prompts. Across 24 comparisons, a cheaper model matched or beat the chosen model in 18 of them (75%), and cheaper models cost roughly 89% less across all 24 (measured model routing).
- Check the pair, not the tier. Claude Sonnet 5 to Gemini 2.5 Flash matched in 9 of 10 comparisons, while swaps below Claude Haiku 4.5 matched in 1 of 4 (measured model routing).
- Fine-tune before you compromise. In OpenAI’s classifier example, gpt-4o with five few-shot examples hit 91.5% accuracy at $11.92 per 1,000 records, and a fine-tuned gpt-4o-mini hit the same 91.5% at $0.21, under 2% of the cost, from 1,000 labeled examples (OpenAI’s model selection guide).
- Route, do not replace everything. FrugalGPT describes three ways to cut inference cost, prompt adaptation, LLM approximation and LLM cascade, and its cascade matched the best single model with up to 98% lower cost (FrugalGPT). A budget-aware policy called TREACLE reports savings up to 85% while holding accuracy (TREACLE).
- Compute your break-even. The published formula is N_r = C_R&D ÷ (ENCS − C_m), where the number of suggestions you need depends on research cost, cost per message and maintenance cost (the LLM economic trade-offs case study).
A small brand sending 500,000 agent messages a year, with savings around $24,000 a year, breaks even fastest by cutting upfront research cost. A large enterprise saving $950,000 a year over 20 million messages breaks even on almost any research budget, and there the inference cost dominates (the LLM economic trade-offs case study).
What this costs, and where it fails
Routing is not free. You maintain an evaluation set, a router and a fallback path, and every model update resets part of that work. Read the savings claims carefully too. The published estimate is 40% to 70% of spend on a workload that is mostly simple requests, and it is an estimate, not a measurement. On the one live account with data, 34% of requests went to a cheaper model and the bill fell about 11%, because routed calls were smaller than average (measured model routing).
The failure mode costs more than the savings. When a cheap model is confidently wrong on something that matters, you pay once and remember it for years. That is why we build and operate intelligent systems around the accuracy bar and the cost of failure for each workflow, not around a model catalogue (our framework for AI in CRM). Human+AI setups hold up better when the cheap tier handles routine traffic and people take the exceptions. We have spent more than 20 years in CRM and data in Barcelona, and that balance shifts every time a provider releases a new version.
If your model spend is climbing while returns stay flat, and you need a routing decision that survives a review, we can help you build the evaluation that settles it (tell us what you are running).