To build a production-ready MLOps dashboard with Grafana and Prometheus, you instrument your model serving layer to expose metrics in Prometheus format, scrape those metrics on a schedule, store them in Prometheus, and visualize them in Grafana with panels that track latency, throughput, error rates, and model-specific signals like prediction drift. That is the short answer. The rest of this post walks through how I set this up for production systems, what to measure, and the mistakes that cost teams real incident time.
Most ML monitoring failures I see are not model quality problems. They are observability gaps. The model degrades silently, prediction latency creeps up, a feature pipeline starts shipping nulls, and nobody notices until a downstream team files a ticket. Prometheus machine learning monitoring combined with Grafana closes that gap with the same tooling your platform team already trusts for infrastructure.
Why Prometheus and Grafana for Production ML Monitoring
The appeal here is boring, which is exactly the point. Prometheus and Grafana are mature, widely deployed, and operationally well understood. You are not introducing a bespoke observability stack that only the ML team can operate at 3 a.m.
- Pull-based scraping fits stateless model servers well. Your service exposes a
/metricsendpoint, Prometheus scrapes it, and you avoid the complexity of pushing metrics from every replica. - Dimensional metrics via labels let you slice by model name, version, region, and endpoint without exploding your schema.
- PromQL gives you a query language expressive enough for rate calculations, percentiles, and drift ratios.
- Grafana unifies ML metrics with the rest of your platform so on-call engineers see one pane of glass.
For teams running multiple models across environments, this consolidation matters. If you want help standing up this foundation across a fleet of services, our data and MLOps capabilities cover the platform work that sits underneath the dashboards.
What to Instrument Before You Open Grafana
A dashboard is only as good as the metrics behind it. I group model observability signals into four layers, and I recommend instrumenting all four before building any panels.
1. Service health metrics
These are the classic RED signals: Rate, Errors, Duration. For a model serving endpoint, that means request count, error count, and request latency. These tell you whether the service is up and responsive, independent of whether predictions are correct.
2. Inference metrics
This is where ML diverges from ordinary web services. You want:
- Prediction latency broken out from total request latency, because feature fetching and post-processing often dominate.
- Batch size distribution if you support batching.
- Model version in use, exposed as a label so you can confirm a rollout actually took effect.
3. Data and feature quality metrics
- Null or missing feature rates
- Out-of-range feature counts
- Feature freshness (age of the most recent feature value)
4. Model quality and drift signals
- Prediction distribution statistics (mean, histogram buckets)
- Input drift measures such as population stability index computed in a sidecar job
- Ground-truth-based accuracy when labels arrive, usually delayed
Instrumenting a Python Model Server
Here is a minimal example using the official prometheus_client library in a FastAPI service. The key decisions are choosing the right metric types and keeping label cardinality under control.
from prometheus_client import Counter, Histogram, Gauge, make_asgi_app
from fastapi import FastAPI
import time
app = FastAPI()
# Mount the Prometheus scrape endpoint
app.mount("/metrics", make_asgi_app())
PREDICTIONS = Counter(
"model_predictions_total",
"Total predictions served",
["model_name", "model_version", "outcome"],
)
PREDICTION_LATENCY = Histogram(
"model_prediction_latency_seconds",
"Prediction latency in seconds",
["model_name", "model_version"],
buckets=(0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1.0, 2.5),
)
PREDICTION_SCORE = Histogram(
"model_prediction_score",
"Distribution of model output scores",
["model_name"],
buckets=(0.0, 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1.0),
)
NULL_FEATURES = Counter(
"model_null_features_total",
"Count of null feature values received",
["model_name", "feature"],
)
MODEL_INFO = Gauge(
"model_info",
"Deployed model metadata",
["model_name", "model_version"],
)
MODEL_INFO.labels("fraud_v2", "2024.11.1").set(1)
@app.post("/predict")
def predict(payload: dict):
model_name, model_version = "fraud_v2", "2024.11.1"
start = time.perf_counter()
for feature, value in payload.items():
if value is None:
NULL_FEATURES.labels(model_name, feature).inc()
score = run_inference(payload) # your model call
outcome = "flagged" if score > 0.8 else "clear"
PREDICTION_LATENCY.labels(model_name, model_version).observe(
time.perf_counter() - start
)
PREDICTION_SCORE.labels(model_name).observe(score)
PREDICTIONS.labels(model_name, model_version, outcome).inc()
return {"score": score, "outcome": outcome}
A note on cardinality. Do not put user IDs, request IDs, or raw feature values in labels. High-cardinality labels will blow up Prometheus memory and storage. Keep labels bounded to things like model name, version, region, and coarse outcome buckets.
Configuring Prometheus to Scrape Your Models
Point Prometheus at your service. If you run on Kubernetes, use service discovery so new replicas are picked up automatically.
scrape_configs:
- job_name: "model-servers"
scrape_interval: 15s
kubernetes_sd_configs:
- role: pod
relabel_configs:
- source_labels: [__meta_kubernetes_pod_label_app]
regex: model-server
action: keep
- source_labels: [__meta_kubernetes_pod_label_model]
target_label: model
A 15-second scrape interval is a reasonable default for serving metrics. For slow-moving drift metrics computed in batch jobs, you can expose them through the Pushgateway or a separate exporter with a longer interval.
Building the MLOps Dashboard in Grafana
Now the part that your on-call team actually looks at. I organize a production MLOps dashboard in Grafana into rows that mirror the four metric layers above, so an engineer can scan top to bottom from service health down to model quality.
Row 1: Service health
- Request rate panel:
sum(rate(model_predictions_total[5m])) by (model_name) - Error rate panel: ratio of errors to total requests, with a threshold color at 1 percent.
- p95 latency panel using the histogram:
histogram_quantile(
0.95,
sum(rate(model_prediction_latency_seconds_bucket[5m])) by (le, model_name)
)
Row 2: Inference behavior
- Throughput by model version to confirm traffic shifts during a canary rollout.
- Prediction score distribution as a heatmap using the
model_prediction_scorehistogram. This is one of the most valuable panels for catching silent drift. When the score distribution shifts without a deploy, something upstream changed.
Row 3: Data quality
- Null feature rate by feature name:
sum(rate(model_null_features_total[15m])) by (feature). A spike here usually means an upstream pipeline broke before the model noticed.
Row 4: Drift and model quality
- Prediction mean over time to spot gradual drift.
- Population stability index gauge, computed by a scheduled job and exposed as a Prometheus metric, with alert thresholds you agree on with data science.
Alerting That Respects On-Call Sanity
Dashboards are for investigation. Alerts are for waking people up, so be deliberate. Define alerting rules in Prometheus and route through Alertmanager.
groups:
- name: model-serving
rules:
- alert: HighPredictionLatency
expr: |
histogram_quantile(0.95,
sum(rate(model_prediction_latency_seconds_bucket[5m])) by (le, model_name)
) > 0.5
for: 10m
labels:
severity: warning
annotations:
summary: "p95 latency above 500ms for {{ $labels.model_name }}"
- alert: FeatureNullSpike
expr: sum(rate(model_null_features_total[15m])) by (feature) > 10
for: 5m
labels:
severity: critical
annotations:
summary: "Null spike in feature {{ $labels.feature }}"
The for clause matters. It prevents flapping alerts from transient blips. Tune thresholds against real baseline data rather than guessing, and review them after every incident.
Operational Practices That Keep the Stack Healthy
- Version your dashboards as code. Export Grafana dashboards to JSON and store them in Git. Provision them with the Grafana provisioning config or Terraform so changes are reviewed.
- Set retention deliberately. Serving metrics at 15-second resolution add up. Use recording rules to pre-aggregate expensive queries and downsample long-term data.
- Separate drift computation from serving. Keep heavy statistical jobs out of the request path so dashboard pressure never degrades inference.
- Agree on ownership. Decide who responds to a data-quality alert versus a latency alert. Observability without clear ownership just generates noise.
These patterns apply whether you serve a single model or hundreds. For teams in regulated environments such as finance or healthcare, the same stack supports audit and compliance reporting. Our work across regulated and data-intensive industries leans heavily on exactly this kind of model observability.
Putting It Together
A production MLOps dashboard is not one giant screen. It is a layered view that lets an engineer move from "is the service up" to "is the model still trustworthy" in a few glances. Instrument the four layers, keep label cardinality disciplined, build Grafana rows that follow the same structure, and back it all with alerts you have actually tuned. Do that, and silent model degradation stops being an incident you discover from a downstream complaint.
FAQ
Do I need a specialized ML monitoring tool, or is Prometheus and Grafana enough?
For most teams, Prometheus and Grafana cover service health, inference metrics, and basic drift. Specialized platforms add convenience for complex drift detection and label-delayed accuracy tracking. Start with Prometheus and Grafana because your platform team already operates them, then add specialized tooling only when you hit a concrete limitation.
How do I monitor model drift when ground-truth labels arrive late?
Compute input and prediction drift in near real time using distribution-based measures like population stability index, which do not need labels. Track label-based accuracy separately as a delayed metric once ground truth lands, often hours or days later. Expose both to Prometheus so your Grafana dashboard shows the fast proxy and the slower truth side by side.
What is the biggest mistake teams make with Prometheus machine learning metrics?
High label cardinality. Putting user IDs, request IDs, or raw feature values into labels causes Prometheus memory and storage to grow uncontrollably and can crash the server. Keep labels bounded to model name, version, region, and coarse outcome categories.
How often should Prometheus scrape model serving metrics?
A 15-second interval works well for serving metrics such as latency and throughput. For batch-computed drift metrics, use a longer interval through an exporter or Pushgateway, since those values change slowly and do not justify frequent scraping.
