To monitor LLM applications with Prometheus and Grafana, you instrument your application code to emit metrics (request latency, token counts, error rates, cost), expose them on a /metrics endpoint that Prometheus scrapes, and then visualize and alert on those metrics in Grafana. That is the short version. Effective LLM monitoring goes deeper: you need to track quality signals, provider dependencies, and cost per request, not just CPU and memory. This guide walks through the full setup, from instrumenting a Python service to building dashboards and alerts that catch real problems before your users do.
Why LLM Monitoring Is Different
Traditional application monitoring assumes deterministic behavior. Given the same input, a function returns the same output in roughly the same time. LLM-backed services break that assumption in several ways:
- Latency is highly variable. Token generation time depends on prompt length, output length, and provider load. A p50 of 800ms can sit alongside a p99 of 12 seconds.
- Cost scales with usage in real money. Every request consumes tokens you pay for. Unmonitored, a runaway retry loop or a prompt-injection attack can produce a large invoice.
- Failures are diverse. You deal with rate limits, context-window overflows, content filter blocks, timeouts, and malformed JSON responses, each needing different handling.
- Quality degrades silently. A model or prompt change can produce worse answers with zero errors and normal latency. Nothing in your infrastructure alerts on that.
So your LLM telemetry has to capture operational metrics and application-specific signals together. Prometheus and Grafana handle the collection, storage, and visualization. Your job is deciding what to measure.
The Metrics That Matter
Before writing code, define the signals. I group LLM metrics into four categories.
Operational metrics
- Request rate (requests per second by endpoint and model)
- Latency distribution (histogram, so you can compute percentiles)
- Error rate broken down by error type
- In-flight requests (concurrency)
Token and cost metrics
- Prompt tokens and completion tokens per request
- Total tokens counter (for cost estimation)
- Estimated cost, derived from token counters and per-model pricing
Provider and dependency metrics
- Upstream provider latency versus your total handler latency
- Rate-limit hits (HTTP 429 counts)
- Retry counts and fallback activations
Quality and safety metrics
- Content filter or moderation blocks
- Empty or truncated responses
- Optional evaluation scores if you run inline checks
Instrumenting a Python LLM Service
The prometheus_client library is the standard way to expose metrics from a Python service. Install it alongside your app:
pip install prometheus-client
Define your metrics once, at module load. Use Counter for monotonic values, Histogram for distributions, and Gauge for values that go up and down.
from prometheus_client import Counter, Histogram, Gauge
# Operational
LLM_REQUESTS = Counter(
"llm_requests_total",
"Total LLM requests",
["model", "endpoint", "status"],
)
LLM_LATENCY = Histogram(
"llm_request_duration_seconds",
"LLM request latency in seconds",
["model", "endpoint"],
buckets=(0.25, 0.5, 1, 2, 5, 10, 20, 30, 60),
)
LLM_INFLIGHT = Gauge(
"llm_inflight_requests",
"Requests currently in flight",
["model"],
)
# Tokens and cost
LLM_PROMPT_TOKENS = Counter(
"llm_prompt_tokens_total", "Prompt tokens consumed", ["model"]
)
LLM_COMPLETION_TOKENS = Counter(
"llm_completion_tokens_total", "Completion tokens generated", ["model"]
)
# Errors by type
LLM_ERRORS = Counter(
"llm_errors_total", "LLM errors by type", ["model", "error_type"]
)
A note on histogram buckets: the defaults in prometheus_client top out around 10 seconds, which hides the long tail typical of LLM responses. Set buckets that reflect your real latency range, as above. Percentile accuracy in Prometheus depends entirely on bucket boundaries.
Now wrap your model call. This example uses a generic client, but the pattern applies to any provider SDK.
import time
def call_llm(model: str, endpoint: str, messages: list) -> dict:
LLM_INFLIGHT.labels(model=model).inc()
start = time.perf_counter()
status = "success"
try:
response = client.chat.completions.create(
model=model, messages=messages
)
usage = response.usage
LLM_PROMPT_TOKENS.labels(model=model).inc(usage.prompt_tokens)
LLM_COMPLETION_TOKENS.labels(model=model).inc(usage.completion_tokens)
return response
except RateLimitError:
status = "error"
LLM_ERRORS.labels(model=model, error_type="rate_limit").inc()
raise
except TimeoutError:
status = "error"
LLM_ERRORS.labels(model=model, error_type="timeout").inc()
raise
except Exception:
status = "error"
LLM_ERRORS.labels(model=model, error_type="unknown").inc()
raise
finally:
duration = time.perf_counter() - start
LLM_LATENCY.labels(model=model, endpoint=endpoint).observe(duration)
LLM_REQUESTS.labels(model=model, endpoint=endpoint, status=status).inc()
LLM_INFLIGHT.labels(model=model).dec()
Finally, expose the metrics endpoint. If you run FastAPI, mount the ASGI app:
from prometheus_client import make_asgi_app
metrics_app = make_asgi_app()
app.mount("/metrics", metrics_app)
Keep label cardinality low. Never use user IDs, request IDs, or raw prompt text as label values. Each unique combination creates a new time series, and high cardinality is the fastest way to overwhelm Prometheus.
Configuring Prometheus to Scrape LLM Metrics
Point Prometheus at your service with a scrape job. The Prometheus LLM scrape config is no different from any other target; the value is in the metrics you defined.
# prometheus.yml
global:
scrape_interval: 15s
evaluation_interval: 15s
scrape_configs:
- job_name: "llm-service"
metrics_path: /metrics
static_configs:
- targets: ["llm-service:8000"]
labels:
service: "chat-api"
environment: "production"
In Kubernetes, use kubernetes_sd_configs or the Prometheus Operator's ServiceMonitor custom resource instead of static targets. Verify scraping works by visiting the Prometheus UI at /targets and confirming your endpoint shows UP.
Building Grafana Dashboards for LLMs
Connect Grafana to Prometheus as a data source, then build panels around the four metric categories. Here are the core PromQL queries I start every dashboard with.
Request rate by model:
sum(rate(llm_requests_total[5m])) by (model)
p95 latency:
histogram_quantile(0.95,
sum(rate(llm_request_duration_seconds_bucket[5m])) by (le, model)
)
Error rate as a percentage:
sum(rate(llm_requests_total{status="error"}[5m])) by (model)
/
sum(rate(llm_requests_total[5m])) by (model) * 100
Token throughput:
sum(rate(llm_completion_tokens_total[5m])) by (model)
Estimated hourly cost (assuming a recording rule or fixed price per 1K tokens):
sum(rate(llm_prompt_tokens_total[1h])) by (model) * 0.0000015
+
sum(rate(llm_completion_tokens_total[1h])) by (model) * 0.000002
Adjust the multipliers to match current provider pricing. I keep pricing in a config map so I update it in one place when rates change.
For layout, I recommend a top row of single-stat panels (current RPS, p95 latency, error rate, hourly cost), a middle row of time-series graphs, and a bottom row breaking down errors by type. This gives on-call engineers a health read in under five seconds. Good Grafana dashboards answer "is it healthy" before they answer "why."
Teams standing up this kind of LLMOps observability stack from scratch often benefit from a structured engagement. Our Data, AI & MLOps capabilities cover instrumentation patterns and platform setup end to end.
Alerting on the Signals That Cost You
Dashboards are for investigation. Alerts are for waking someone up. Define Prometheus alerting rules for the failure modes that hurt.
groups:
- name: llm-alerts
rules:
- alert: HighLLMErrorRate
expr: |
sum(rate(llm_requests_total{status="error"}[5m])) by (model)
/ sum(rate(llm_requests_total[5m])) by (model) > 0.05
for: 5m
labels:
severity: critical
annotations:
summary: "Error rate above 5% for {{ $labels.model }}"
- alert: LLMLatencyDegraded
expr: |
histogram_quantile(0.95,
sum(rate(llm_request_duration_seconds_bucket[5m])) by (le, model)
) > 15
for: 10m
labels:
severity: warning
annotations:
summary: "p95 latency above 15s for {{ $labels.model }}"
- alert: TokenSpike
expr: |
sum(rate(llm_completion_tokens_total[5m])) by (model)
> 3 * sum(rate(llm_completion_tokens_total[1h] offset 1h)) by (model)
for: 5m
labels:
severity: warning
annotations:
summary: "Token throughput 3x above baseline"
The TokenSpike alert is the one that saves money. A sudden jump in token consumption usually means a bug, an abusive client, or a prompt loop, and catching it early limits the damage.
Practical Considerations
- Correlate with traces. Metrics tell you something is wrong. To see why, pair Prometheus with distributed tracing (OpenTelemetry) so you can jump from a latency spike to the specific slow request.
- Watch cardinality over time. Audit your label sets periodically. A label added in good faith can explode series count once traffic grows.
- Version your prompts as a label. A low-cardinality
prompt_versionlabel lets you compare quality and latency across prompt changes, which is invaluable during rollouts. - Retention and storage. Token counters accumulate quickly. Use recording rules to precompute expensive queries and consider remote storage for long retention.
Domain requirements shape what you monitor. Regulated sectors need audit-friendly logging and stricter cost controls; see how we approach this across regulated and data-intensive industries for context on compliance-aware observability.
Set up correctly, this stack turns an opaque LLM service into one you can reason about. You see cost accrue in real time, catch quality regressions through proxy metrics, and get paged before your users complain.
FAQ
What metrics should I prioritize when I start LLM monitoring?
Start with the four operational basics: request rate, latency histogram, error rate by type, and total token counters. These give you health and cost visibility immediately. Add quality and provider-specific metrics once the foundation is stable, since those require more application-specific instrumentation.
How do I estimate LLM cost in Prometheus without invented pricing?
Track prompt and completion tokens as separate Counter metrics, then multiply token rates by your provider's published per-token price inside a PromQL query or recording rule. Keep the price values in configuration so you update them when providers change rates. This gives an estimate grounded in your actual token usage rather than guesswork.
Why are my latency percentiles inaccurate in Grafana?
Prometheus computes percentiles from histogram buckets, so accuracy depends entirely on bucket boundaries. LLM responses often exceed the default buckets, which top out around 10 seconds. Define custom buckets that span your real latency range, for example up to 60 seconds, so histogram_quantile has the resolution to compute the tail correctly.
Can I monitor response quality with Prometheus alone?
Prometheus is built for numeric time series, so it can track quality proxies like truncation rate, empty-response rate, moderation blocks, and inline evaluation scores if you emit them as metrics. For richer quality analysis, pair it with logging and tracing systems that can store the actual prompts and responses. Use Prometheus for the aggregate signal and alerting.
How do I avoid high cardinality in LLM metrics?
Never use unbounded values such as user IDs, request IDs, or raw prompt text as label values. Stick to low-cardinality labels like model name, endpoint, error type, and a bounded prompt version identifier. Audit your series count regularly, because cardinality that is fine at launch can overwhelm Prometheus as traffic grows.
Production-grade cloud, software, and engineering teams for scaling companies.
