Six LLM API costs that never appear on the rate card

A provider's pricing page has two numbers on it. An invoice has considerably more than two causes. These are the six that most often account for the difference.

By Soufiane Bellahmar · Founder and editor, LLM Cost Lab
Figures recomputed from the 60-model dataset on every build. Rates last verified 2026-08-16.

Every LLM pricing page works the same way: a price per million input tokens, a price per million output tokens, and a table of models. It is an honest presentation and it is not hiding anything. It is simply answering a narrower question than the one you are asking. You want to know what your feature will cost. The rate card tells you what a token costs.

The gap between those two is where budgets die. Here is what lives in it, roughly in order of how much damage each one does.

1. Conversation history, re-sent every turn

This is the big one, and it is big because it compounds. LLM APIs are stateless. The model does not remember your previous message; your client re-sends the entire conversation on every request. So turn one sends the system prompt plus one message. Turn twenty sends the system prompt plus all nineteen previous exchanges plus the new message.

Input tokens therefore grow with the square of conversation length, not linearly with it. Most people model it linearly, because linear is what intuition suggests.

A concrete case, priced against Doubao Seed 2.1 Pro because it sits near the middle of our distribution. Take a 1,500-token system prompt, 150-token user messages, 400-token replies, and a twenty-turn conversation. If you budget it as twenty independent requests, you get $0.062. The actual cost, with history re-sent each turn, is $0.151 — 2.4× the estimate, from exactly the same twenty exchanges.

Nothing was hidden. The arithmetic was just harder than it looked. Our chatbot monthly cost estimator does this growth properly rather than assuming independence.

2. Reasoning tokens you are billed for but never see

Models with a reasoning or extended-thinking mode generate a private chain of intermediate tokens before producing the answer. Those tokens are billed at the full output rate. They are typically not returned in the response body, which means if you are metering your own usage by counting the characters you received, you are undercounting — sometimes by a factor of five or more on hard problems.

The trap is that reasoning modes are increasingly on by default or triggered automatically by perceived difficulty. A workload can start reasoning more without anyone having changed a setting, simply because the inputs got harder. Check the usage object your provider returns rather than the text length; the reasoning token count is reported there even when the content is not.

3. The system prompt, multiplied by every request

A system prompt feels like configuration, so it gets treated like configuration — written once, then forgotten. But it is input tokens, and it is sent on every call.

A two-thousand-token system prompt across a million monthly requests is two billion input tokens per month before a single user has typed a word. Whether that matters depends entirely on your model: at the cheap end of our dataset that is a rounding error, and at the expensive end it is a real line item. Either way it is worth knowing, and it is usually the single easiest thing to fix, because prompt caching exists precisely for this.

27 of the 60 models we track publish a cached input rate, at a median discount of 90% off the standard input price. The remaining 33 do not, which is itself a cost factor worth weighing when you pick a model for a system-prompt-heavy workload. The prompt caching calculator will price it at your actual hit rate; the mechanics are explained on the prompt caching guide.

4. Retries, timeouts, and results you throw away

If the model generated a response, you are paying for it — whether or not your code kept it. That includes:

• Responses that failed your JSON schema validation and triggered a retry.
• Requests your own client timed out on after the provider had already generated the tokens.
• Speculative parallel calls where you keep the fastest and discard the rest.
• Requests dropped by a downstream failure after the LLM call succeeded.

None of these appear in a naive model of "requests times cost per request," because that model counts successful user-visible outcomes. The provider counts generations. A system with a 10% retry rate is 10% more expensive than its own success metrics suggest, and retry rates tend to be highest on exactly the expensive, complex prompts where each retry costs most.

5. Output length you did not ask for

Output is the expensive side of almost every rate card — frequently four to five times the input rate. And output length is the variable you control least directly. A model that decides to be thorough, restate the question, add caveats, or produce a summary you did not request is spending your money at the highest rate on the card.

max_tokens is a safety net, not a control — it truncates, which usually means you pay for a useless partial response and then pay again for the retry. The real control is instructional: asking for a specific format, a length bound in the prompt, or a structured output schema that has no room for preamble. This is the highest-leverage token reduction available to most teams and it costs nothing to try.

6. The tokens you pay for twice

In a retrieval pipeline, the same text is often billed more than once. You pay to embed a document when you index it. You pay again to send the retrieved chunks as input at query time. If you re-index — because you changed the chunking strategy, or switched embedding models, or the source updated — you pay the embedding cost again for the whole corpus.

Re-indexing is the one that surprises people, because it is a one-off cost that arrives at unpredictable moments and is never in the monthly model. Corpus size times embedding rate, every time you change your mind about chunking. The RAG pipeline estimator and the embeddings calculator both handle this explicitly.

What to do about it

The unifying point is that a rate card prices tokens, and none of these six are about the price of a token. They are all about the number of tokens, which is the variable you actually control. That is good news: the rate is set by someone else, but the count is set by you.

For latency-tolerant work there is one more lever worth knowing: 18 of the models we track offer a discounted batch tier, which is a straight reduction with no change to your prompts at all. The batch savings calculator prices it, and the cost reduction guide covers the full set of levers in order of effort.

Frequently asked questions