How to Track LLM Token Costs and Latency in Production
To track LLM token costs and latency in production, capture token counts and timing on every model call, attach business context to each record, convert tokens to cost with a versioned price table, and aggregate the results into percentile latency and cost-per-feature dashboards with alerts. In practice, LLM cost monitoring comes down to one structured telemetry event per request that holds the model, input tokens, output tokens, cached tokens, time to first token, total duration, and tags like tenant, feature, and prompt version. Everything else (dashboards, budgets, anomaly alerts) is built from that event.
The rest of this post covers how we approach each piece, with code you can adapt.
Why LLM Cost Monitoring Is Harder Than It Looks
Most teams start with the provider's billing console. It shows that spend went up. It rarely tells you why. The common gaps we see:
- No attribution. The invoice is per API key or project, not per feature, customer, or prompt.
- Variable output length. The same endpoint can cost very different amounts depending on how verbose the model gets.
- Hidden multipliers. Retries, agent loops, tool calls, and RAG context stuffing multiply tokens without an obvious change in user traffic.
- Streaming hides latency. Total request duration says little about perceived speed when users see the first token early.
- Price changes. Providers adjust pricing and introduce new tiers such as cached input, so a hardcoded rate goes stale.
Billing data is a lagging, coarse signal. You need request-level telemetry that you own.
Step 1: Define the Metrics That Matter
Before writing code, agree on a minimal schema. These are the fields we treat as non-negotiable for production LLM metrics.
Token and cost fields
- Input tokens: prompt plus system message plus retrieved context.
- Output tokens: completion tokens, including reasoning tokens where the provider reports them separately.
- Cached input tokens: tokens served from a prompt cache, usually billed at a different rate.
- Computed cost: derived from tokens and a price table, never estimated from character counts.
Latency fields
- Time to first token (TTFT): what users perceive in streaming UIs.
- Time per output token: generation speed after the first token.
- Total duration: end-to-end time for the call, including network.
- Attempt count: how many retries occurred before success.
Context fields
feature,tenant_id,prompt_version,model,environment, and atrace_idlinking the call to the parent request.
Use the token counts returned by the provider, not your own tokenizer. Local tokenizers are useful for pre-flight estimates, but the response's usage block is what you are billed on.
Step 2: Instrument Every Model Call
Wrap the client once and route all calls through the wrapper. The example below uses the OpenAI Python SDK and OpenTelemetry. Attribute names follow the OpenTelemetry Semantic Conventions for Generative AI, which are still marked as in development, so pin a version and review changes when you upgrade.
import time
from openai import OpenAI
from opentelemetry import trace, metrics
client = OpenAI()
tracer = trace.get_tracer("llm.client")
meter = metrics.get_meter("llm.client")
token_usage = meter.create_histogram(
"gen_ai.client.token.usage", unit="{token}"
)
op_duration = meter.create_histogram(
"gen_ai.client.operation.duration", unit="s"
)
ttft_hist = meter.create_histogram("llm.time_to_first_token", unit="s")
def chat(messages, model, feature, tenant_id, prompt_version):
attrs = {
"gen_ai.operation.name": "chat",
"gen_ai.request.model": model,
"app.feature": feature,
"app.prompt_version": prompt_version,
}
with tracer.start_as_current_span(f"chat {model}") as span:
span.set_attributes({**attrs, "app.tenant_id": tenant_id})
start = time.perf_counter()
first_token_at = None
chunks, usage = [], None
stream = client.chat.completions.create(
model=model,
messages=messages,
stream=True,
stream_options={"include_usage": True},
)
for chunk in stream:
if chunk.choices and chunk.choices[0].delta.content:
if first_token_at is None:
first_token_at = time.perf_counter()
chunks.append(chunk.choices[0].delta.content)
if chunk.usage:
usage = chunk.usage
end = time.perf_counter()
op_duration.record(end - start, attrs)
if first_token_at:
ttft_hist.record(first_token_at - start, attrs)
if usage:
cached = getattr(
usage.prompt_tokens_details, "cached_tokens", 0
) or 0
span.set_attributes({
"gen_ai.usage.input_tokens": usage.prompt_tokens,
"gen_ai.usage.output_tokens": usage.completion_tokens,
"app.usage.cached_input_tokens": cached,
})
token_usage.record(usage.prompt_tokens,
{**attrs, "gen_ai.token.type": "input"})
token_usage.record(usage.completion_tokens,
{**attrs, "gen_ai.token.type": "output"})
return "".join(chunks), usage
A few deliberate choices here:
stream_options={"include_usage": True}makes the OpenAI API emit a final chunk with usage data. Without it, streamed responses do not report token counts. See the OpenAI API reference.tenant_idgoes on the span, not the metric. High-cardinality labels on metrics can overwhelm a time-series database. Keep them on traces or logs and aggregate offline.- Other providers use different field names. Anthropic's Messages API, for example, returns
input_tokens,output_tokens,cache_creation_input_tokens, andcache_read_input_tokens(Anthropic docs). Normalize these in the wrapper so downstream queries stay provider-agnostic.
Step 3: Convert Tokens to Cost With a Versioned Price Table
Do not multiply tokens by a constant buried in application code. Keep pricing in configuration, versioned with an effective date, and compute cost in your pipeline.
# pricing.yaml — values are placeholders; load from provider pricing pages
- model: example-model-large
effective_from: 2025-01-01
per_million:
input: 0.00
cached_input: 0.00
output: 0.00
- model: example-model-small
effective_from: 2025-01-01
per_million:
input: 0.00
cached_input: 0.00
output: 0.00
def compute_cost(rec, prices):
p = prices.lookup(rec["model"], rec["timestamp"])["per_million"]
uncached = rec["input_tokens"] - rec["cached_input_tokens"]
return (
uncached * p["input"]
+ rec["cached_input_tokens"] * p["cached_input"]
+ rec["output_tokens"] * p["output"]
) / 1_000_000
Look up prices by model and timestamp so historical costs stay accurate after a price change. Check whether your provider reports cached tokens as a subset of input tokens (as in the example above) or as a separate count, and adjust the arithmetic accordingly. Reconcile your computed totals against the provider invoice monthly. If they drift, you are missing calls or using stale rates.
Step 4: Build Dashboards Around Questions, Not Charts
Good LLMOps observability answers specific questions. We organize dashboards around these:
Cost questions
- What does each feature cost per day, and per successful user action?
- Which tenants drive the most spend relative to their plan?
- Did a new prompt version change average input or output tokens?
Latency questions
- What are p50, p95, and p99 TTFT per model and feature?
- Is total duration dominated by generation, retries, or upstream retrieval?
Use percentiles, never averages, for LLM latency. Output length varies widely, so means hide the tail your users actually feel. With Prometheus-compatible storage, a p95 TTFT query looks like this (exact metric names depend on how your exporter translates OpenTelemetry names):
histogram_quantile(
0.95,
sum by (le, gen_ai_request_model) (
rate(llm_time_to_first_token_seconds_bucket[5m])
)
)
Configure histogram bucket boundaries for your workload. Default buckets designed for millisecond HTTP calls are a poor fit for requests that can take many seconds.
Step 5: Alert on Budgets and Anomalies
Dashboards are for investigation. Alerts catch problems before the invoice does. The alerts we find most useful:
- Spend rate vs. budget: projected daily cost per feature exceeds a threshold.
- Tokens per request drift: average input tokens jump after a deploy, often a sign of runaway context or a broken truncation step.
- Retry ratio: attempts per successful call rises, which inflates both cost and latency.
- Tail latency: p95 TTFT breaches your SLO for a sustained window.
- Loop detection: a single trace exceeds a maximum number of model calls, common in agent workflows.
Pair alerts with hard guardrails in code: max_tokens on every request, a cap on agent iterations, and per-tenant rate limits. Monitoring tells you something went wrong. Limits stop it from getting expensive while you investigate.
Common Pitfalls
- Logging full prompts by default. Prompts often contain personal or regulated data. Log token counts and hashes by default, and sample full content only where policy allows.
- Ignoring failed calls. Errored requests can still consume input tokens. Record them.
- Measuring only the model call. Retrieval, reranking, and post-processing add latency. Trace the full request so you can see where time actually goes.
- One shared API key. Separate keys or projects per environment make reconciliation far easier.
Putting It Into Practice
The pattern is consistent across stacks: one wrapper, one normalized event, one price table, and dashboards tied to real questions. Start with a single high-traffic feature, validate your computed costs against the invoice, then expand coverage.
If you want help designing this kind of telemetry pipeline alongside your broader platform work, our data, AI, and MLOps capabilities cover observability, evaluation, and production hardening. Requirements around data handling and audit trails also vary by sector, so it is worth reviewing how these practices apply in regulated and high-scale industries before you finalize retention and logging policies.
FAQ
What is LLM cost monitoring?
LLM cost monitoring is the practice of recording token usage for every model call, converting it to cost with current pricing, and attributing that cost to features, tenants, and prompt versions. It gives you request-level visibility that provider billing consoles do not provide.
Should I count tokens myself or use the provider's numbers?
Use the usage data returned in the provider's API response for cost tracking, because that is what you are billed on. Local tokenizers are useful for estimating prompt size before sending a request or for enforcing context limits.
Which latency metric matters most for LLM applications?
For streaming, user-facing applications, time to first token is usually the most important because it reflects perceived responsiveness. Track it alongside total duration and time per output token, and report all of them as percentiles rather than averages.
How do I track costs for streamed responses?
Many providers only return usage data in streaming mode when you request it. With the OpenAI API, set stream_options={"include_usage": true} to receive a final chunk with token counts. Check your provider's documentation for the equivalent option.
Is OpenTelemetry ready for LLM observability?
OpenTelemetry publishes semantic conventions for generative AI that define standard attribute and metric names for model calls. They are still evolving, so pin the version you adopt and review changes on upgrade, but they are a sound foundation for vendor-neutral LLMOps observability.
