How to Monitor LLM Inference Latency and Token Usage with Prometheus and Grafana

If you are running large language models in production, the fastest way to get LLM telemetry into a system you can act on is to instrument your inference service to expose Prometheus metrics for latency and token usage, then visualize and alert on them in Grafana. Concretely: emit a histogram for request latency, counters for prompt and completion tokens, and a gauge for in-flight requests. Scrape those with Prometheus, build a Grafana dashboard on top, and wire alerts for p95 latency and token spend. The rest of this post shows exactly how to do that, with code you can adapt today.

Why LLM telemetry is different from ordinary service monitoring

Most of us already monitor HTTP services with request rate, error rate, and duration. LLM workloads add two problems that break the assumptions behind those defaults.

• Latency is bimodal and long-tailed. A short completion returns in a few hundred milliseconds. A long, streamed response can take tens of seconds. Averaging those together hides everything that matters. You need histograms and percentiles, not means.
• Cost is driven by tokens, not requests. Two requests can differ by 100x in cost because one generated 20 tokens and the other generated 2,000. If you only count requests, you are blind to the thing your finance team cares about.

Good LLM telemetry therefore tracks three signal families:

1. Latency, split by phase where possible (queue time, time-to-first-token, total generation time).
2. Token usage, split into prompt and completion tokens, ideally labeled by model.
3. Throughput and saturation, meaning requests per second and concurrent in-flight requests.

If you are building this into a broader platform, it fits naturally alongside the observability work we describe under our data, AI, and MLOps capabilities.

Instrumenting your inference service

The examples below use Python and the official prometheus_client library, but the metric shapes translate directly to Go, Node, or Java clients.

Choosing the right metric types

• Use a Histogram for latency so Prometheus can compute quantiles across instances.
• Use Counters for token totals so you can derive rates with rate().
• Use a Gauge for concurrent requests, which goes up and down.

Defining the metrics

from prometheus_client import Counter, Histogram, Gauge

# Latency in seconds. Buckets tuned for LLM response times.
LLM_LATENCY = Histogram(
    "llm_request_duration_seconds",
    "End-to-end LLM inference latency",
    labelnames=("model", "route"),
    buckets=(0.1, 0.25, 0.5, 1, 2, 4, 8, 16, 32, 64),
)

# Time to first token, useful for streamed responses.
LLM_TTFT = Histogram(
    "llm_time_to_first_token_seconds",
    "Time until the first token is emitted",
    labelnames=("model",),
    buckets=(0.05, 0.1, 0.25, 0.5, 1, 2, 4),
)

LLM_PROMPT_TOKENS = Counter(
    "llm_prompt_tokens_total",
    "Total prompt (input) tokens processed",
    labelnames=("model",),
)

LLM_COMPLETION_TOKENS = Counter(
    "llm_completion_tokens_total",
    "Total completion (output) tokens generated",
    labelnames=("model",),
)

LLM_INFLIGHT = Gauge(
    "llm_inflight_requests",
    "Currently executing inference requests",
    labelnames=("model",),
)

LLM_ERRORS = Counter(
    "llm_request_errors_total",
    "Failed inference requests",
    labelnames=("model", "reason"),
)

Two design notes that will save you pain later:

• Keep label cardinality low. Do not put user IDs, prompts, or request IDs into labels. Each unique label combination is a separate time series, and high cardinality is the most common way people blow up their Prometheus memory. Model name and route are safe. A free-text field is not.
• Custom histogram buckets matter. The default prometheus_client buckets top out around 10 seconds, which is fine for web APIs but too coarse for long generations. Pick buckets that straddle your real latency distribution.

Recording metrics around a call

Wrap the actual inference call so you capture latency, tokens, and errors in one place.

import time
from contextlib import contextmanager

@contextmanager
def track_inference(model: str, route: str):
    LLM_INFLIGHT.labels(model=model).inc()
    start = time.perf_counter()
    try:
        yield
    except Exception as exc:
        LLM_ERRORS.labels(model=model, reason=type(exc).__name__).inc()
        raise
    finally:
        LLM_LATENCY.labels(model=model, route=route).observe(
            time.perf_counter() - start
        )
        LLM_INFLIGHT.labels(model=model).dec()


def handle_chat(request, model="gpt-oss-20b"):
    with track_inference(model, route="/v1/chat"):
        response = call_model(request)  # your provider or self-hosted call

    usage = response["usage"]
    LLM_PROMPT_TOKENS.labels(model=model).inc(usage["prompt_tokens"])
    LLM_COMPLETION_TOKENS.labels(model=model).inc(usage["completion_tokens"])
    return response

Most hosted APIs and popular self-hosted servers such as vLLM return a usage block with token counts. If yours does not, count tokens with the same tokenizer the model uses so your numbers reconcile with any billing you receive.

Exposing the metrics endpoint

from prometheus_client import start_http_server

# Serve /metrics on port 8000 in a background thread.
start_http_server(8000)

If you already run FastAPI, mount the ASGI app instead:

from prometheus_client import make_asgi_app
app.mount("/metrics", make_asgi_app())

Configuring Prometheus LLM monitoring

Point Prometheus at your service. A minimal scrape job:

# prometheus.yml
global:
  scrape_interval: 15s

scrape_configs:
  - job_name: "llm-inference"
    metrics_path: /metrics
    static_configs:
      - targets: ["llm-service:8000"]
        labels:
          service: "chat-api"
          env: "production"

In Kubernetes with the Prometheus Operator, use a ServiceMonitor instead of static targets so scraping tracks your pods automatically:

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: llm-inference
  labels:
    release: prometheus
spec:
  selector:
    matchLabels:
      app: llm-service
  endpoints:
    - port: metrics
      interval: 15s

Recording rules for the queries you run often

Percentile and cost queries are expensive to compute on the fly across many dashboards. Precompute them with recording rules.

groups:
  - name: llm-telemetry
    interval: 30s
    rules:
      - record: llm:latency_p95:5m
        expr: |
          histogram_quantile(0.95,
            sum by (le, model) (rate(llm_request_duration_seconds_bucket[5m])))

      - record: llm:completion_tokens:rate5m
        expr: sum by (model) (rate(llm_completion_tokens_total[5m]))

Building the Grafana LLM dashboard

Add Prometheus as a data source in Grafana, then create panels backed by PromQL. Here are the queries I reach for first.

p50, p95, and p99 latency (time series panel):

histogram_quantile(0.95,
  sum by (le, model) (rate(llm_request_duration_seconds_bucket[5m])))

Duplicate the query with 0.50 and 0.99 for the other lines.

Token throughput (stacked time series):

sum by (model) (rate(llm_prompt_tokens_total[5m]))
sum by (model) (rate(llm_completion_tokens_total[5m]))

Estimated hourly token spend (stat panel). If your price is, say, $0.60 per million completion tokens, express it as a constant. Replace the numbers with your own contracted rates. Do not hard-code guesses.

sum(rate(llm_completion_tokens_total[1h])) * 3600 / 1e6 * 0.60

In-flight requests (gauge or time series):

sum by (model) (llm_inflight_requests)

Error rate (time series):

sum by (reason) (rate(llm_request_errors_total[5m]))

Group these into a Grafana LLM dashboard with a row per concern: latency at the top, tokens and cost in the middle, saturation and errors at the bottom. Add a model template variable so you can filter across models from a dropdown. This layout maps to how teams in regulated and cost-sensitive industries actually triage incidents: performance first, then cost, then reliability.

Alerting on what actually hurts

Dashboards are for investigation. Alerts are for waking someone up. Keep them few and specific.

groups:
  - name: llm-alerts
    rules:
      - alert: LLMHighLatencyP95
        expr: llm:latency_p95:5m > 8
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "p95 latency above 8s for {{ $labels.model }}"

      - alert: LLMErrorSpike
        expr: |
          sum(rate(llm_request_errors_total[5m]))
            / sum(rate(llm_request_duration_seconds_count[5m])) > 0.05
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "Error rate above 5%"

For cost, an alert on a sudden jump in completion token rate often catches runaway retry loops or a prompt template change before the invoice does.

Common pitfalls to avoid

• Averaging latency. Means lie for long-tailed distributions. Always publish percentiles from histograms.
• High-cardinality labels. Never label by user, session, or prompt content.
• Counting requests instead of tokens. Cost lives in tokens. Track both.
• Ignoring time-to-first-token for streaming. For chat UIs, perceived latency is TTFT, not total duration. Measure both.
• Scraping too aggressively. A 15 to 30 second interval is plenty for these signals and keeps storage sane.

Once these fundamentals are in place, you have a durable foundation for capacity planning, cost attribution, and SLOs. That is the payoff of treating LLM telemetry as a first-class part of your platform rather than an afterthought.

FAQ

What is the difference between latency and time-to-first-token in LLM telemetry?

Total latency measures the full request from start to the last token. Time-to-first-token measures how long a user waits before any output appears. For streamed chat interfaces, time-to-first-token drives perceived responsiveness, so I recommend tracking both as separate histograms.

How do I get token counts if my model server does not report them?

Count tokens client-side using the same tokenizer the model uses, for example the model's published tokenizer library. Increment your prompt and completion counters with those values. Reconcile against provider billing periodically to confirm your counts match.

Will Prometheus histograms give accurate p99 latency across multiple instances?

Yes, as long as every instance uses identical bucket boundaries. Prometheus aggregates the _bucket series across instances and histogram_quantile() estimates the percentile from the combined buckets. Accuracy depends on bucket granularity near your latency range, so tune buckets accordingly.

How do I avoid high cardinality in LLM metrics?

Only use labels with a small, bounded set of values, such as model name, route, and environment. Never put user IDs, request IDs, or prompt text into labels. If you need per-request detail, send that to logs or traces, not to Prometheus metrics.

Can I estimate cost directly in Grafana?

You can, by multiplying token rate counters by your contracted per-token price expressed as a constant in the query. Use your actual negotiated rates rather than assumptions, and update the constant when pricing changes so the panel stays accurate.

---

Production-grade cloud, software, and engineering teams for scaling companies.