Why your LLM bill beats your estimate: five mistakes in order of size

Nobody sets out to under-budget. The estimate is wrong in a consistent direction because the same five things get left out, and four of them are about token counts rather than token prices.

By Soufiane Bellahmar · Founder and editor, LLM Cost Lab
Figures recomputed from the 60-model dataset on every build. Rates last verified 2026-08-16.

There is a predictable rhythm to AI budgets. A number is produced early, it is sensible, it is based on real rate cards. Then the feature ships and the invoice arrives at some multiple of it. Not double the price of a token — the rates were right. Double or triple the number of tokens.

Here are the five reasons, ordered by how much damage each one typically does. All figures are priced against Doubao Seed 2.1 Pro at $0.85 input and $4.23 output, which sits near the middle of our 60-model distribution.

1. Modelling conversation cost as linear

Covered at length in the piece on hidden costs, but it belongs at the top of this list because it is the largest and the most common.

Chat APIs are stateless. Every turn re-sends the full history. Twenty turns is not twenty times the cost of one turn; it is closer to two hundred times, because turn n carries everything from turns 1 through n−1. A spreadsheet that multiplies "messages per month" by "cost per message" is describing a product that does not exist.

Measure instead: the distribution of conversation lengths in production, and the input token count of the last turn rather than the average turn. That final number is what your heavy users cost.

2. Guessing the input-to-output ratio

Output typically costs four to five times input. So the split between them dominates your blended rate, and it is almost always assumed rather than measured.

One million input plus one million output tokens costs $5.08. Shift that same two-million-token volume to be input-heavy (a RAG or classification shape, 7:1) and it costs $2.54. Shift it output-heavy (a content generation shape, 1:7) and it costs $7.62 — 3.0× the input-heavy figure for identical total volume.

Same token count, same model, same month. Only the ratio changed.

Measure instead: pull the actual input and output totals from your provider's usage API for a representative week and compute the real ratio. Do not carry an assumed one into a forecast.

3. Budgeting the average user

Usage of AI features is power-law distributed. A small minority of users generate most of the tokens. If you budget the mean and your heavy tail is fat, you will be wrong by the size of the tail — and the tail is precisely the part that grows when a feature succeeds.

The failure mode is subtle: the budget is not wrong at launch, when usage is uniform and low. It becomes wrong as adoption deepens and power users emerge, which reads as a cost problem arriving at the same time as a success signal.

Measure instead: p50, p90, and p99 tokens per user, tracked over time. Budget from a weighted blend of those, and watch the p99 as a leading indicator. See cost per user for the full arithmetic.

4. Forgetting the calls that aren't the main call

Every AI feature has a headline call — the one that generates the answer — and it is the one that gets budgeted. Around it sit the calls nobody counts:

• Query rewriting or expansion before retrieval
• Embedding the query at request time
• Reranking retrieved chunks
• A safety or moderation pass on input, output, or both
• Title and summary generation for the conversation list
• Follow-up suggestion generation
• Evaluation or quality-scoring calls in the background

Individually each is cheap. Collectively they frequently match or exceed the main call, because there are six of them and they run on every request. In agent architectures the multiplier is worse still: an agent that takes eight tool-calling steps makes at least eight model calls per user action, each carrying the accumulated context of the previous ones.

Measure instead: count model calls per user action, not per feature. If the number surprises you, that is the finding. The agent cost calculator models the multi-step case explicitly.

5. Assuming caching and batching are already working

This one is different from the others: it is not an underestimate of usage but an overestimate of savings.

Prompt caching is available on 27 of the 60 models we track, at a median 90% discount on cached input. That headline number gets folded into budgets as though it applies to all input tokens. It applies only to the cached prefix, only when the prefix matches exactly, and only within the cache's time-to-live — often just a few minutes.

A cache hit rate of 40% with a 90% discount on the prefix portion of your input is a much smaller saving than "90% off input," and the difference between those two readings has broken more than one forecast. The same applies to batch discounts, which only apply to work you actually moved to the batch endpoint.

Measure instead: your real cache hit rate from the usage object, and the fraction of input tokens that are actually in the cached prefix. Then apply the discount to that fraction only. The caching calculator takes a hit rate as an input for exactly this reason.

The pattern behind all five

Four of the five are about token counts and one is about a discount rate. None of them is about the price of a token being wrong. Rate cards are accurate and public; the uncertainty is entirely on your side of the API.

Which is the useful conclusion, because it means the fix is instrumentation rather than negotiation. Log the usage object from every response into whatever you already use for metrics, tagged by feature and user. A week of that data replaces every assumption on this page with a measurement, and it is the single highest-return afternoon of work available to a team that is worried about its AI bill.

Once you have real numbers, the cost reduction guide covers the levers in order of effort, and the calculators will price any of them against your measured usage.