# Runtime cost baseline

A one-page procedure for finding out what the software around your model is costing you.
Free to use and to copy. From Data Innovation, Intelligence² #130, 22 September 2026.

Why bother: UC Berkeley and Arena measured identical models on identical coding tasks inside
different agent runtimes in September 2026. Cost per attempt varied by up to 5x while success
rates stayed close, and models did not always score best inside their own vendor's runtime.
Nothing about that is specific to coding. If you run agents against a CRM, a warehouse or a
ticket queue, the same spread is sitting in your bill.

## What you need

- Your three most frequent agent tasks. Not the impressive ones, the ones that run hundreds of
  times a week. Those carry the cost.
- Twenty real inputs per task, taken from production, with the correct output already known.
- Two runtimes: the one you use now, and one deliberately plainer.
- A way to read cost per call. Most gateways report it per request.

## The procedure

1. **Write down the success test before you run anything.** One line per task, checkable by a
   script or by one person in under ten seconds. "Correct account matched" beats "good answer".
2. **Freeze the model.** Same model, same version, same temperature across both runtimes. You are
   measuring the wrapper, so nothing else may move.
3. **Run all twenty inputs through runtime A.** Record, per call: cost, latency, pass or fail.
4. **Run the same twenty through runtime B.** Same inputs, same order.
5. **Compare three numbers, not one:** cost per attempt, pass rate, and cost per *successful*
   attempt. The third is the one that decides. A runtime that halves cost and drops ten points of
   pass rate has saved you nothing.
6. **Repeat the winner on a second day** before you move anything. One run is an anecdote.

## The table to fill in

| | Runtime A (current) | Runtime B (plainer) |
|---|---|---|
| Model and version | | |
| Cost per attempt | | |
| Pass rate over 20 | | |
| **Cost per successful attempt** | | |
| Median latency | | |
| Failures worth reading | | |

## Where the money usually hides

- **The system prompt, resent on every call.** Long preambles are cheap once and expensive ten
  thousand times.
- **Tool descriptions.** Every tool you expose is tokens on every call, whether or not it is used.
  Most agents are handed tools they never call.
- **Conversation memory carried forward whole** rather than summarised or windowed.
- **Retries that are not counted.** A silent retry doubles the cost of that call and rarely shows
  up in the dashboard next to the original.
- **Classification paid at generation prices.** If the answer is one of five labels, a model that
  writes prose to say it is the wrong tool. Classifier models price this differently.

## What to do with the result

If the plainer runtime wins on cost per successful attempt, move the three tasks and leave
everything else alone. Rerun the baseline when you change model or when volume doubles. The number
you want on a slide is not "we cut AI spend", it is "same pass rate, this much less per successful
task, measured on twenty real cases".

---

Built by a human team working with AI agents at Data Innovation SL, Barcelona.
Questions, corrections or your own numbers: hello@datainnovation.io