The real cost of running LLMs at scale, and how to control it

Why your LLM bill grows faster than your usage

Here is the thing most finance teams discover three months into a serious AI rollout: the bill is not linear. Traffic goes up 40 percent and the invoice doubles. That gap is where the money quietly leaks, and it almost always traces back to how the system is built rather than how many users showed up.

The pattern we see across the companies we work with is consistent. Early on, nobody is watching tokens because the numbers are tiny. A feature that costs a fraction of a cent per call feels free. Then it ships to every customer, gets wrapped in an agent loop that calls the model five times per task, and starts dragging in a 30,000 token context window on every single request. The unit economics that looked fine in the demo fall apart at volume.

Token cost is the new compute cost, and it behaves like one. You would not run a cloud workload without knowing your instance hours. Running LLMs without knowing your tokens per request is the same mistake wearing a newer outfit.

• Tokens in (your prompt, system instructions, retrieved context, conversation history) are usually the silent majority of the cost.
• Tokens out (the model's response) are priced higher per token on most providers, so verbose outputs hurt more than they look.
• Every retry, reflection step, and tool call is another full round trip you pay for in full.
• Context that travels with every request (long system prompts, RAG chunks, chat history) multiplies cost by your request volume.

What actually drives the number on your invoice

Before you can cut cost you have to know where it comes from. There are really only four levers underneath every LLM bill, and almost all waste lives in two of them.

Model choice and call count are the dominant factors at scale, and they are the ones teams reach for last. People will spend a week shaving 200 tokens off a prompt while a flagship model handles a classification task that a model a tenth the price could do just as well. The prompt diet feels productive. The model is where the actual money is.

• Model choice: a frontier model can cost 15 to 30 times more per token than a small one. Using it for tasks that do not need it is the single most expensive habit in production AI.
• Context size: long prompts and large retrieved chunks get charged on every call. A bloated system prompt is a tax you pay forever.
• Output length: unbounded responses cost more and run slower. Asking for structured, capped output is free money.
• Call multiplication: agent loops, retries, self-critique passes, and multi-step chains can turn one user action into a dozen billed requests.

The cost-control playbook we actually use

When we audit an AI system for spend, we work through the same ordered checklist every time. The order matters: the early items return far more money for far less effort, and we have seen teams burn weeks on prompt micro-optimization while ignoring a routing change that would have cut the bill in half.

None of this requires switching providers or rewriting your stack. Most of it is configuration and architecture discipline you can apply this quarter.

• Route by difficulty: send easy tasks (classification, extraction, short rewrites, routing decisions) to a small or distilled model, and reserve the frontier model for genuinely hard reasoning. This one change is usually the biggest win.
• Cap retries and agent steps: set a hard ceiling on loop iterations and tool calls. An agent with no step limit is an open invoice. Three to five steps covers the vast majority of real tasks.
• Cache aggressively: use prompt caching for stable system instructions and shared context, and cache full responses for repeated or near-identical queries. Repeated context should be paid for once, not every call.
• Compress context: trim system prompts, summarize long histories instead of resending them raw, and retrieve fewer, better chunks rather than dumping everything in.
• Bound the output: ask for structured responses with explicit length limits. Shorter outputs cost less and parse cleaner.
• Batch where latency allows: offline and async jobs often qualify for cheaper batch pricing, which can halve cost on bulk workloads.

Model routing without wrecking quality

The objection we hear most when we suggest routing is fair: won't cheaping out on the model hurt the output? It can, if you do it blindly. The discipline is to route on task difficulty, not on vibes, and to measure quality at each tier so you know exactly what you are trading.

A clean way to think about it is a tiered ladder. Most production traffic is not hard. The skill is building a router that recognizes the easy 80 percent and reserves your expensive model for the 20 percent that earns it.

• Tier 1, small model: classification, intent detection, formatting, short summaries, yes or no decisions. Cheap, fast, good enough.
• Tier 2, mid model: standard generation, moderate reasoning, most chat turns. The workhorse for general work.
• Tier 3, frontier model: complex reasoning, long-context synthesis, high-stakes output where a mistake is expensive.
• Build an evaluation set per tier so a routing change is a measured decision, not a guess, and you can prove quality held.

How to budget and monitor AI spend like a real line item

You cannot control what you cannot see, and the default provider dashboard shows you a single total that tells you nothing about which feature is bleeding. The fix is to attribute cost the way you would attribute any other operating expense: per feature, per customer, per request type.

We tell every client the same thing. Decide your unit economics before you scale, not after. If a feature costs more per use than it earns or saves, that is a product decision to make on purpose, not a surprise you find in next month's invoice.

• Tag every request with metadata (feature, customer tier, model used) so you can break the bill down by what actually drives it.
• Set a target cost per request or per task and track it as a core metric, the same way you track latency or error rate.
• Alert on anomalies: a sudden spike in tokens per request usually means a prompt bug, a runaway loop, or context that stopped getting trimmed.
• Forecast against real traffic projections so finance is not blindsided, and so a 10x growth plan has a cost number attached to it.

The hidden costs nobody puts in the spreadsheet

The per-token rate is the part everyone sees. The costs that actually decide whether your AI product is profitable tend to sit just outside the invoice, and they are the ones we spend the most time digging out during an audit.

These do not show up as a line item from your provider, which is exactly why they get missed. They show up as a margin that is thinner than your model expected, and by then the architecture is hard to change.

• Failed and retried calls: a request that errors halfway still bills you for the tokens it consumed before failing, and your retry logic pays again.
• Context window creep: chat histories and growing RAG indexes mean the same feature costs more per call every month unless you actively trim.
• Over-provisioned quality: paying frontier prices for output users cannot tell apart from a cheaper model is pure waste.
• Human review and rework: low-quality cheap output that someone has to fix can cost more in labor than the model fee you saved.
• Idle observability gaps: months of unmonitored spend before anyone notices is a real cost, and it compounds quietly the whole time.

A 30-day plan to get costs under control

If your AI spend is growing and nobody owns the number, you do not need a six-month transformation. You need a month of focused work and one person accountable for the line item. Here is the sequence we run with teams who want results fast.

Done in this order, most teams cut 30 to 60 percent off their bill in the first month without users noticing a difference in quality, because the savings come from waste, not from cutting corners.

• Week 1: instrument everything. Tag requests, build a cost-per-feature view, and find your top three most expensive call paths.
• Week 2: cap and cache. Set retry and step limits, turn on prompt caching, and bound output lengths across the board.
• Week 3: route. Move your easy, high-volume tasks to a smaller model and validate quality against an eval set before and after.
• Week 4: budget and alert. Set per-request targets, wire up anomaly alerts, and hand ongoing ownership of the number to a named person.