Status note (19 July 2026): Model prices and service terms change faster than evergreen articles. This guide treats cost as a measured property of a workload, not a permanent vendor ranking. Pin the model, region, date, contract, quality target, and traffic shape before making a migration or routing decision.
Key Takeaways
- A price table without a model snapshot, date, region, and token mix is not a reliable budget.
- API versus private-runtime economics depends on utilization and operational scope, not a universal token threshold.
- Caching, routing, compression, batching, and smaller models are experiments with quality and safety gates.
- Cost observability must reconcile provider usage with application events, retries, review, storage, and infrastructure.
- Optimize cost per accepted outcome, not cost per generated token.
Build a Dated Cost Ledger
For each provider or runtime, record:
| Field | Example |
|---|---|
| Identity | Provider, model ID/checkpoint, revision, endpoint, region |
| Metering | Input, output, reasoning, cached, batch, image/audio units |
| Conditions | Context or resolution, quality tier, rate limits, concurrency |
| Commercial terms | Effective date, currency, taxes, credits, contract discounts |
| Reliability | Retries, failed jobs, timeout behavior, duplicate side effects |
| Evidence | Rate card URL, invoice line, usage export, access date |
Do not compare a public list price with a private contract or mix input and output tokens into one “average” without showing the weighting. Re-run the ledger when a provider changes a model ID, tokenizer, cache policy, or billing rule.
Define the Unit of Value
Token cost is only an input. Choose a business unit such as:
cost_per_accepted_outcome =
(provider + runtime + retries + review + operations + infrastructure)
/ accepted_outcomes
An accepted outcome needs a deterministic or human-verifiable acceptance rule. For an agent, count failed tool calls, duplicate actions, manual correction, and unresolved outcomes. For generation, count usable assets rather than attempts.
API Versus Private Runtime
Compare both options against the same service contract:
| Dimension | Hosted API | Private or local runtime |
|---|---|---|
| Variable charge | Dated provider metering | Compute, energy, storage, network |
| Fixed charge | Usually lower at low utilization | Engineering, SRE, security, support |
| Elasticity | Provider capacity and quotas | Capacity planning and scaling |
| Change cost | Vendor model and contract changes | Checkpoint, kernel, runtime, and hardware changes |
| Data boundary | Provider terms and region | Your controls, vendors, backups, and operators |
| Quality risk | API/model drift | Self-managed update and evaluation burden |
Break-even is an output of a sensitivity model. Vary traffic, utilization, batchability, latency target, replica count, GPU price, staff allocation, and acceptance rate. Do not publish a threshold until readers can reproduce those assumptions.
Cost Reduction Experiments
Routing and smaller models
Route only when the classifier and fallback are measurable:
request
-> classify workload and risk
-> select verified model or deterministic path
-> enforce budget, authorization, and timeout
-> evaluate outcome
-> escalate or retry with an explicit reason
Compare quality, tool-call validity, refusal, recovery, latency, cost, and escalation by task slice. A smaller model is not a drop-in replacement merely because it is cheaper.
Caching
Measure exact and semantic caches separately. A valid cache key may include tenant, authorization scope, policy version, model revision, locale, source revision, and freshness window. Never let semantic similarity bypass permission checks or return stale private data.
Prompt compression and context selection
Compression can reduce input units while damaging instructions, citations, or safety constraints. Use an A/B fixture with answer quality, evidence coverage, refusal behavior, input tokens, latency, and review rate. Do not transfer a compression ratio or quality-loss percentage from another dataset.
Batch processing
Batch may reduce unit price while increasing queue delay, storage, retries, and cancellation complexity. Record the provider's current batch terms and measure completion-time distribution rather than promising a fixed turnaround.
Observability and Reconciliation
Join provider usage with an application event:
event_id
request_id
tenant_id_hash
model_revision
input_units
output_units
cached_units
retry_count
review_required
accepted_outcome
provider_cost
infrastructure_cost
Redact prompts, secrets, personal data, and generated content according to a retention purpose. Keep enough metadata to reconcile an invoice and investigate a failure. A trace is evidence for accounting and debugging, not permission to retain every raw model message forever.
A Reproducible Budget Model
monthly_cost =
Σ(input_units × input_rate)
+ Σ(output_units × output_rate)
+ cache_and_batch_charges
+ retries_and_failed_work
+ storage_and_egress
+ moderation_and_human_review
+ observability_and_support
+ infrastructure_and_hardware_amortization
Report a base, low, and high scenario. Vary input/output mix, demand, cache hit rate, retry rate, acceptance rate, utilization, and exchange rate. A spreadsheet or notebook should emit the assumptions alongside the result so a new model or price can be substituted without rewriting the conclusion.
Decision Gates
- Baseline: reconcile a representative period and classify costs by workload.
- Experiment: change one lever and measure quality, safety, latency, and cost.
- Release: set a budget and acceptance gate with an owner and rollback path.
- Recheck: trigger evaluation after model, price, policy, traffic, or data changes.
Conclusion
Inference economics is an accounting and evaluation problem. A lower rate can increase total spend if it attracts more traffic, causes more retries, or reduces accepted outcomes. A private runtime can improve control while increasing operational risk. Keep the ledger dated, test each optimization on representative work, and optimize the cost of a verified result rather than a headline token price.