Data, AI & MLOps6 min read · Oct 14, 2026

How to Build a Production-Ready ML Monitoring Stack with Prometheus and Grafana

Techsense DevelopersEngineering

Building a production-ready ML monitoring stack with Prometheus and Grafana comes down to four moves. First, instrument your model service to emit operational and prediction metrics. Second, run a separate job that computes drift and quality metrics against a reference baseline. Third, let Prometheus scrape both and evaluate alert rules. Fourth, use Grafana to visualize the results and route alerts to the people who own the model. The hard part is not the tooling. It is deciding which signals actually indicate a model is failing, and keeping metric cardinality under control so the stack stays reliable as you add models.

This guide walks through that design end to end, with code you can adapt.

Why ML Monitoring Is Different from Service Monitoring

A model can return HTTP 200 in 40 ms and still be wrong. Traditional service monitoring catches crashes, latency spikes, and error rates. Models also fail silently: input distributions shift, upstream features break, or the relationship between inputs and outcomes changes over time.

Sculley et al. described this class of problem in Hidden Technical Debt in Machine Learning Systems (NeurIPS 2015). They noted that ML systems carry dependencies on data that are harder to detect than code dependencies. Breck et al.'s The ML Test Score (Google, 2017) lists monitoring tests for production models, including checks for data invariants, training/serving skew, and prediction quality.

In practice, we group production ML metrics into three layers:

Operational health: request rate, latency, errors, resource saturation.
Data and prediction behavior: input feature distributions, missing values, prediction score distributions, and drift against a baseline.
Model quality: accuracy, precision/recall, or business KPIs, computed once ground-truth labels arrive. This is often hours or weeks later.

Prometheus handles all three well, as long as you respect what it is good at. It works best with numeric time series and low-to-moderate cardinality. It is not a store for raw predictions.

Reference Architecture for an MLOps Monitoring Stack

Here is the shape we recommend for most teams:

Model service (FastAPI, BentoML, KServe, etc.) exposes /metrics with operational and prediction histograms.
Prediction log (object storage, Kafka, or a warehouse table) captures sampled inputs and outputs for offline analysis.
Drift and quality job reads the prediction log, compares it to a training baseline, and exposes results as Prometheus gauges.
Prometheus scrapes everything and evaluates recording and alerting rules.
Alertmanager routes alerts by model owner and severity.
Grafana provides dashboards provisioned as code.

The key design choice is the split between the online path and the offline path. Keep heavy statistical computation out of the request path. Your inference service should only increment counters and observe histograms.

Step 1: Instrument the Model Service

Use the official prometheus_client library. Track latency, throughput, errors, and the distribution of prediction scores.

from prometheus_client import Counter, Histogram, Gauge, start_http_server

MODEL_LABELS = ["model", "version"]

PREDICTIONS = Counter(
    "model_predictions_total",
    "Total predictions served",
    MODEL_LABELS,
)

ERRORS = Counter(
    "model_prediction_errors_total",
    "Prediction failures",
    MODEL_LABELS + ["reason"],
)

LATENCY = Histogram(
    "model_inference_latency_seconds",
    "Inference latency",
    MODEL_LABELS,
    buckets=(0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1.0, 2.5),
)

SCORE = Histogram(
    "model_prediction_score",
    "Distribution of predicted probabilities",
    MODEL_LABELS,
    buckets=[i / 10 for i in range(1, 11)],
)

MISSING_FEATURES = Counter(
    "model_missing_feature_total",
    "Requests with a null or missing feature",
    MODEL_LABELS + ["feature"],
)

def predict(model, features, name="churn", version="2024-06-01"):
    labels = {"model": name, "version": version}
    with LATENCY.labels(**labels).time():
        try:
            for f, v in features.items():
                if v is None:
                    MISSING_FEATURES.labels(feature=f, **labels).inc()
            score = model.predict_proba([features])[0][1]
        except Exception as exc:
            ERRORS.labels(reason=type(exc).__name__, **labels).inc()
            raise
    PREDICTIONS.labels(**labels).inc()
    SCORE.labels(**labels).observe(score)
    return score

start_http_server(8000)

Watch your cardinality

The Prometheus documentation warns against using labels with unbounded values, such as user IDs or request IDs. Every unique label combination creates a new time series. For ML, this means:

Do label by model, version, and a bounded set of feature names.
Don't label by customer, raw input values, or free-text error messages.
Keep the feature label to the features that matter most. A model with 2,000 sparse features should not emit 2,000 series per metric.

Step 2: Compute Model Drift Detection Metrics Offline

The score histogram above gives you a cheap, real-time view of prediction drift. For feature-level drift, run a scheduled job that compares recent production data to the training baseline. The Population Stability Index (PSI) is a common, interpretable choice. Kolmogorov–Smirnov tests and Jensen–Shannon distance are reasonable alternatives.

import numpy as np
from prometheus_client import Gauge, start_http_server
import time

PSI = Gauge("model_feature_psi", "PSI vs training baseline",
            ["model", "version", "feature"])
LABELED_ACCURACY = Gauge("model_labeled_accuracy",
            "Accuracy on predictions with ground truth",
            ["model", "version", "window"])

def psi(expected, actual, bins=10, eps=1e-6):
    edges = np.quantile(expected, np.linspace(0, 1, bins + 1))
    edges[0], edges[-1] = -np.inf, np.inf
    e = np.histogram(expected, edges)[0] / len(expected) + eps
    a = np.histogram(actual, edges)[0] / len(actual) + eps
    return float(np.sum((a - e) * np.log(a / e)))

def run_once(baseline, recent, model, version):
    for feature in baseline.columns:
        PSI.labels(model, version, feature).set(
            psi(baseline[feature].dropna(), recent[feature].dropna())
        )

if __name__ == "__main__":
    start_http_server(9100)
    while True:
        # load_baseline() / load_recent() read from your feature store or warehouse
        run_once(load_baseline(), load_recent(hours=24), "churn", "2024-06-01")
        time.sleep(3600)

A few practical notes:

Use quantile-based bins from the baseline, as above. This keeps bins stable and avoids empty buckets.
Version the baseline alongside the model. When you retrain, the reference distribution changes too.
Running the drift job as a long-lived service that Prometheus scrapes is simpler than pushing. The Prometheus docs recommend the Pushgateway mainly for service-level batch jobs. If your job is a short-lived cron task, Pushgateway is acceptable.
Ground-truth quality metrics belong in this same job. Join predictions with labels as they arrive, and export accuracy or AUC with a window label such as 7d.

PSI thresholds of roughly 0.1 (moderate shift) and 0.25 (significant shift) are a widely used convention in credit risk modeling. Treat them as a starting point and calibrate them against your own model's history.

Step 3: Configure Prometheus Scraping and Rules

# prometheus.yml
scrape_configs:
  - job_name: model-serving
    scrape_interval: 15s
    static_configs:
      - targets: ["churn-model:8000"]
  - job_name: model-drift
    scrape_interval: 60s
    static_configs:
      - targets: ["drift-job:9100"]

rule_files:
  - "ml_rules.yml"

Recording rules precompute expensive queries, which keeps dashboards fast:

# ml_rules.yml
groups:
  - name: ml-recording
    rules:
      - record: model:inference_latency_seconds:p99_5m
        expr: |
          histogram_quantile(0.99,
            sum by (le, model, version) (rate(model_inference_latency_seconds_bucket[5m])))
      - record: model:positive_rate:1h
        expr: |
          1 - (
            sum by (model, version) (rate(model_prediction_score_bucket{le="0.5"}[1h]))
            /
            sum by (model, version) (rate(model_prediction_score_count[1h]))
          )

  - name: ml-alerts
    rules:
      - alert: ModelHighLatency
        expr: model:inference_latency_seconds:p99_5m > 0.5
        for: 10m
        labels: { severity: page }
        annotations:
          summary: "p99 latency above 500ms for {{ $labels.model }}"
      - alert: FeatureDriftSignificant
        expr: model_feature_psi > 0.25
        for: 2h
        labels: { severity: ticket }
        annotations:
          summary: "PSI {{ $value | humanize }} on {{ $labels.feature }}"
      - alert: PositiveRateShift
        expr: |
          abs(model:positive_rate:1h - model:positive_rate:1h offset 7d) > 0.1
        for: 1h
        labels: { severity: ticket }

Route by severity. Latency and error alerts page on-call engineers. Drift alerts usually open a ticket for the model owner, because drift is a signal to investigate, not necessarily an outage.

Step 4: Build Grafana Dashboards as Code

Provision dashboards through Grafana's file-based provisioning or its HTTP API so they live in version control next to the model. A useful per-model dashboard includes:

Golden signals row: request rate, error ratio, p50/p99 latency.
Prediction behavior row: score distribution heatmap from model_prediction_score_bucket, positive rate versus last week.
Data quality row: missing-feature rate by feature, top-N features by PSI.
Model quality row: labeled accuracy by window, annotated with deployment events.

Use a model and version template variable so one dashboard serves every model. Grafana annotations for deployments make it obvious when a metric change lines up with a release.

Hardening the Stack for Production

Retention: Local Prometheus storage is not meant for long-term history. For quarter-over-quarter drift analysis, add a remote-write backend such as Thanos, Mimir, or Cortex.
Self-monitoring: Alert on stale data. For example, absent(model_feature_psi) or time() - timestamp(...) checks will catch a drift job that silently stopped.
Runbooks: Link every alert annotation to a runbook covering likely causes, how to check upstream pipelines, and the rollback or retrain path.
Ownership: Add a team label at scrape time so Alertmanager can route without hardcoding model names.

If you are standardizing this across many models or regulated workloads, our data, AI, and MLOps capabilities cover the platform patterns behind it. Domain-specific considerations, such as fairness monitoring in lending or latency budgets in real-time fraud, are discussed on our industries page.

FAQ

Can Prometheus store raw predictions for ML monitoring?

No, and it should not. Prometheus is designed for numeric time series with bounded labels. Store raw or sampled predictions in a log, warehouse, or object store. Export aggregated metrics such as histograms, PSI, and accuracy to Prometheus.

How do I monitor model quality when labels arrive weeks later?

Compute quality metrics in a scheduled job that joins predictions with ground truth as it lands. Export them with a window label. In the meantime, use prediction distribution and feature drift as leading indicators.

Which drift metric should I start with?

PSI is a practical default because it is easy to interpret and works on both numeric and binned categorical features. Add KS tests or Jensen–Shannon distance if you need more sensitivity on specific features.

Do I need a dedicated ML observability tool instead?

Not necessarily. Prometheus and Grafana cover operational, drift, and quality metrics well. Dedicated tools can add value for explainability, slice analysis, and raw prediction exploration. Evaluate them against the gaps you actually hit.