You priced the feature. One call, one system prompt, five cents a request. Staging agreed with you. Two months into production the invoice does not.
Say it went from $80 a month to $9,000 for the same feature. Those numbers are illustrative, but the shape is one we keep seeing, and it is never one big mistake. It is seven small things multiplying.
1. Retries
Every layer retries. The OpenAI SDK retries some 5xx responses. LangChain retries failed chains. Bedrock retries when you're throttled. Each retry re-sends the whole input and pays for it again. The single call in your spreadsheet is sometimes three, and only on the days your provider is having a bad time.
2. Tool use
A tool call is never one call. The model asks for the tool, you feed the result back, the model answers. Two calls, minimum. Add a routing model, a guardrail, a re-ranker, and one ReAct step that decides whether to call another tool, and the one request your user sees is six underneath.
3. Context accumulation
Most agent frameworks resend the whole conversation at every step. Step eight pays for roughly eight times the input of step one, and you paid the steps in between on the way up. Open a long run in the trace view and per-call cost climbs while answers stay the same length.
4. Cache misses
Prompt caching only helps when the cache hits. Cold starts, first-of-day traffic, and any prompt whose prefix shifted by a word all pay full input price, and on some providers writing to the cache costs more than the plain input rate.
5. Reasoning tokens
Reasoning models bill for tokens you never see. The visible answer can be a small fraction of what you're charged for. If someone moved a workload to a reasoning model to get better answers, the per-request cost changed by a multiple, not a percentage.
6. Image input
One 1024x1024 image is roughly 765 input tokens on OpenAI. Ship an "attach a screenshot" button and your input bill moves while your request count sits perfectly still. Most dashboards count requests.
7. Loops
One incident here outweighs everything above. An agent that repeats the same step. A chain that hits a rate limit and retries with the full context. A plan and execute pair that never terminates. There are documented cases of a single loop running past $700 and Gemini calls looping for 37 hours.
What a rule looks like for this
A TokenPolice rule is a limit with an amount and a unit, USD or tokens or requests, applied to one shared pool or to one pool per user, per plan, per feature (metadata.<key>), or per session. The SDK checks it before the provider call goes out, so a blocked call costs nothing. New rules start in dry-run, which means the first thing you get is a list of the calls it would have stopped, not an outage. Details in the rules docs; the shapes people actually use are on Solutions.
Multiplier 7 gets its own tab. A hard cap per trace, one agent run, in calls or dollars or tokens. And a detector for the same prompt going out over and over inside a run. Either can be notify-only. The stop lands a call late by design, so treat it as a ceiling on runaway cost rather than an exact counter. How loop rules behave.
Every new workspace already has three loop rules, pre-made, on notify-only. Sign up, point the SDK at one service, and leave everything in dry-run for a week before you turn anything on.