Alerting That People Don't Ignore: Uptime Checks, Prometheus Alertmanager, SLOs and Error Budgets for Django

Most alerting setups die the same way: a pile of CPU, memory and "worker restarted" alerts fires every night, nobody can act on them, and the team learns to mute the channel. Then the one alert that matters gets muted along with the rest. The fix is to alert on what users actually feel, measured against an explicit objective, route everything that can wait to a ticket queue, and delete the rest without sentiment.

Symptoms, not causes

A symptom is something a user experiences: the site is unreachable, requests fail with a 5xx, pages take four seconds. A cause is something that might produce a symptom: CPU at 80%, a nearly full connection pool, a worker killed by the OOM killer. CPU at 80% with fast, successful responses is a capacity-planning note, not an emergency. CPU at 30% while every checkout returns a 502 because a migration dropped a column is an emergency that no cause-based alert will ever catch.

The rule is simple: page on symptoms, and keep cause metrics on dashboards for diagnosis after a symptom alert has fired. There are only a few cause-based alerts worth keeping, and they share one property: they predict a user-facing outage with enough lead time for someone to act. A disk that will be full in four hours qualifies. A TLS certificate that expires in ten days qualifies. A load average of 6 does not, because nobody knows what to do about it and it usually resolves itself.

Every alert also needs a severity that decides how a human hears about it. Two levels are enough for most teams; a third level almost always turns into a dumping ground.

If a proposed alert fits neither row, it is not an alert. It is a graph, and it belongs on a dashboard.

Exposing Django and host metrics

The reference layout here is small but realistic: one application host (app.example.com) running nginx, Gunicorn and PostgreSQL 16, and a separate monitoring host (203.0.113.10) running Prometheus, Alertmanager and blackbox_exporter. Keeping monitoring off the application host matters: if the two share a machine, the outage that takes the site down also takes out the thing meant to tell you about it.

Install django-prometheus and wrap your middleware stack. The "before" middleware must come first and the "after" middleware last, so together they time the whole request, including everything your other middleware does.

# myproject/settings.py
INSTALLED_APPS += ["django_prometheus"]

MIDDLEWARE = [
    "django_prometheus.middleware.PrometheusBeforeMiddleware",
    # ... your existing middleware, unchanged ...
    "django_prometheus.middleware.PrometheusAfterMiddleware",
]

# Gunicorn runs 4 workers, each with its own metrics registry.
# Each worker binds the first free port in this range.
PROMETHEUS_METRICS_EXPORT_PORT_RANGE = range(8001, 8005)
PROMETHEUS_METRICS_EXPORT_ADDRESS = "0.0.0.0"

The multi-process point is the one most teams miss. The alternative, exposing /metrics through a URL with path("", include("django_prometheus.urls")), means each scrape lands on whichever worker happens to accept it. You end up graphing one process's counters at a time, and rates jump around as counters appear to reset. Per-worker ports give Prometheus one target per worker, and rate() handles the counter reset when a worker is recycled. Size the port range to your worker count exactly; spare ports show up as permanently down targets. Do not start Gunicorn with --preload in this mode, because the exporter is started when the Django app loads, and with preloading that happens once in the master rather than in each worker.

These ports must never be reachable from the internet: they reveal every view name, traffic volume and error rate. Restrict them, and the other exporters, to the monitoring host:

sudo ufw allow from 203.0.113.10 to any port 8001:8004 proto tcp
sudo ufw allow from 203.0.113.10 to any port 9100 proto tcp   # node_exporter
sudo ufw allow from 203.0.113.10 to any port 9187 proto tcp   # postgres_exporter

If you do serve metrics through a URL (a single-process server, for example), block location = /metrics in nginx with allow 203.0.113.10; deny all; rather than trusting Django to do it.

Alongside Django, run node_exporter for filesystem, memory and network metrics, and postgres_exporter for pg_up, connection counts and replication lag. Both are cause metrics. They earn their keep on dashboards during an incident and in the two or three predictive alerts below; they should not page on their own thresholds.

External uptime checks with blackbox_exporter

Internal metrics have a blind spot the size of your front door. If DNS is broken, nginx will not start, the certificate has expired or the firewall is dropping port 443, Django receives no requests at all, and an error ratio computed from Django metrics looks perfect. You need a probe that behaves like a user. Give it a cheap, honest health endpoint first:

# myproject/health.py
from django.contrib.auth.decorators import login_not_required
from django.db import connection
from django.http import JsonResponse
from django.views.decorators.cache import never_cache


@never_cache
@login_not_required
def healthz(request):
    try:
        with connection.cursor() as cursor:
            cursor.execute("SELECT 1")
    except Exception:
        return JsonResponse({"status": "database unavailable"}, status=503)
    return JsonResponse({"status": "ok"})


# myproject/urls.py
from django.urls import path
from myproject.health import healthz

urlpatterns = [
    path("healthz", healthz, name="healthz"),
    # ...
]

The login_not_required decorator only matters if you use Django's LoginRequiredMiddleware; it is harmless otherwise. Keep the check shallow. A health check that calls three third-party APIs will page you whenever one of them wobbles, which is the cause-based trap again in a different costume. Checking the database is justified because without it no page renders.

Configure a blackbox_exporter module on the monitoring host:

# /etc/prometheus/blackbox.yml
modules:
  http_healthz:
    prober: http
    timeout: 5s
    http:
      method: GET
      valid_status_codes: [200]
      fail_if_not_ssl: true
      preferred_ip_protocol: ip4

Then the full Prometheus configuration, including the relabelling that turns a URL target into a probe request:

# /etc/prometheus/prometheus.yml
global:
  scrape_interval: 30s
  evaluation_interval: 30s

rule_files:
  - /etc/prometheus/rules/*.yml

alerting:
  alertmanagers:
    - static_configs:
        - targets: ["127.0.0.1:9093"]

scrape_configs:
  - job_name: django
    static_configs:
      - targets:
          - app.example.com:8001
          - app.example.com:8002
          - app.example.com:8003
          - app.example.com:8004

  - job_name: node
    static_configs:
      - targets: ["app.example.com:9100"]

  - job_name: postgres
    static_configs:
      - targets: ["app.example.com:9187"]

  - job_name: blackbox_https
    metrics_path: /probe
    params:
      module: [http_healthz]
    static_configs:
      - targets: ["https://app.example.com/healthz"]
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement: 127.0.0.1:9115

Every HTTPS probe also exports probe_ssl_earliest_cert_expiry, the Unix timestamp at which the soonest-expiring certificate in the served chain expires. Subtract time() and you have an alert that catches a broken renewal timer weeks before browsers start showing warnings. Because it measures the certificate actually served, it also catches the classic failure where renewal succeeded but nginx was never reloaded.

Defining the SLI, SLO and error budget

The SLI (service level indicator) is the proportion of good events. Here an event is an HTTP request to Django, and good means "not a 5xx and served in under 500 ms". The SLO is the target for that proportion: 99.5% of requests good over a rolling 30 days. The error budget is the complement, the amount of badness you have agreed to tolerate:

budget        = 1 - 0.995 = 0.005   (0.5% of requests may be bad)
30 days       = 30 x 24 x 60 = 43,200 minutes
budget (time) = 43,200 x 0.005 = 216 minutes = 3h 36m of total outage

The minutes figure is an intuition aid, not an accounting identity. Because the SLI counts requests, a full outage at 03:00 with a trickle of traffic costs far less budget than the same outage at peak. That is usually what you want, since it mirrors how many users were hurt, but it means a quiet-hours outage can consume only a few "minutes" of budget while lasting an hour. The external probe covers that gap.

The budget turns arguments into arithmetic. If a bad deploy burned 150 of 216 minutes this month, the team slows releases and spends the next sprint on reliability. If you have used 10 minutes, you have room to ship risky changes. Choose the SLO from what users tolerate, not from what the system currently achieves. A 99.99% target on a single-host Django deployment gives a budget of about 4 minutes a month; every deploy restart and every PostgreSQL minor upgrade would page someone, and the alerts would be ignored within a fortnight.

Two practical constraints shape the SLI. First, django-prometheus records status codes and latency in separate metrics, so you cannot count "fast and non-5xx" exactly. The rules below add the 5xx rate to the slow-request rate; a slow 5xx is counted twice, making the SLI slightly pessimistic, which is acceptable for alerting. Second, the latency threshold must be a histogram bucket boundary. The default buckets include 0.5 seconds; if you need 300 ms, set PROMETHEUS_LATENCY_BUCKETS, accepting that the change creates new series and resets history for that threshold.

Also check retention. Prometheus keeps 15 days by default, which cannot answer a question about 30 days. Start it with --storage.tsdb.retention.time=45d or longer.

Recording rules

Burn-rate alerts need the error ratio over six windows. Record the bad and total request rates once at 5-minute resolution, then derive each longer window as a ratio of sums. Summing both sides before dividing weights busy periods correctly, and it keeps every evaluation cheap.

# /etc/prometheus/rules/django-slo.yml
groups:
  - name: django-slo-recording
    rules:
      - record: slo:requests_total:rate5m
        expr: sum(rate(django_http_requests_latency_seconds_by_view_method_count{job="django",view!="healthz"}[5m]))
      - record: slo:requests_bad:rate5m
        expr: |
          (sum(rate(django_http_responses_total_by_status_view_method_total{job="django",view!="healthz",status=~"5.."}[5m])) or vector(0))
          + slo:requests_total:rate5m
          - sum(rate(django_http_requests_latency_seconds_by_view_method_bucket{job="django",view!="healthz",le="0.5"}[5m]))
      - record: slo:sli_error:ratio_rate5m
        expr: slo:requests_bad:rate5m / slo:requests_total:rate5m
      - record: slo:sli_error:ratio_rate30m
        expr: sum_over_time(slo:requests_bad:rate5m[30m]) / sum_over_time(slo:requests_total:rate5m[30m])
      - record: slo:sli_error:ratio_rate1h
        expr: sum_over_time(slo:requests_bad:rate5m[1h]) / sum_over_time(slo:requests_total:rate5m[1h])
      - record: slo:sli_error:ratio_rate2h
        expr: sum_over_time(slo:requests_bad:rate5m[2h]) / sum_over_time(slo:requests_total:rate5m[2h])
      - record: slo:sli_error:ratio_rate6h
        expr: sum_over_time(slo:requests_bad:rate5m[6h]) / sum_over_time(slo:requests_total:rate5m[6h])
      - record: slo:sli_error:ratio_rate1d
        expr: sum_over_time(slo:requests_bad:rate5m[1d]) / sum_over_time(slo:requests_total:rate5m[1d])

  - name: django-slo-budget
    interval: 5m
    rules:
      - record: slo:error_budget_remaining:ratio30d
        expr: |
          1 - (
            sum_over_time(slo:requests_bad:rate5m[30d])
            / sum_over_time(slo:requests_total:rate5m[30d])
          ) / 0.005

A few details are load-bearing. The or vector(0) matters: in a healthy week there are no 5xx series at all, and without the fallback the whole sum silently evaluates to nothing, so your SLI disappears exactly when things are fine and your dashboards show gaps. Rules within a group evaluate in order, so later rules see the current results of earlier ones. The budget rule sits in its own group with a 5-minute interval, because scanning 30 days of samples every 30 seconds buys nothing. Excluding view="healthz" stops the probe's steady, trivially successful traffic from diluting real failures; on a low-traffic site it can otherwise be a large share of all requests.

The trade-off of recording rules is that recorded series only exist from the moment you deploy them. The 1d window is unreliable for its first day and the 30-day budget figure is meaningless for its first month. Deploy the recording rules a few days before you switch on the alerts that depend on them.

Multi-window, multi-burn-rate alerts

A burn rate is how fast you are spending the budget relative to plan. A burn rate of 1 spends exactly the whole budget in 30 days. A burn rate of 14.4 sustained for one hour spends 2% of the monthly budget, because 14.4 divided by the 720 hours in 30 days is 0.02. A naive "error ratio above 0.5% for 5 minutes" alert pages on every brief blip; an alert on the 30-day average fires days too late. Burn rates over paired windows sit between those failures.

Each alert has a long window, which proves the problem is significant, and a short window, which proves it is still happening. Without the short window, the alert keeps firing for up to an hour after you have rolled back, and people learn to ignore "stale" pages. The short window is typically one-twelfth of the long one.

The complete alerting rules, including uptime, TLS, disk, database and a watchdog:

# /etc/prometheus/rules/django-alerts.yml
groups:
  - name: django-alerts
    rules:
      - alert: ErrorBudgetBurnFast
        expr: |
          slo:sli_error:ratio_rate1h > (14.4 * 0.005)
          and
          slo:sli_error:ratio_rate5m > (14.4 * 0.005)
        for: 2m
        labels: {severity: page, service: myproject}
        annotations:
          summary: "Error budget burning at 14.4x: {{ $value | humanizePercentage }} of requests bad over 1h"
          runbook_url: "https://app.example.com/runbooks/error-budget-burn"

      - alert: ErrorBudgetBurnSlow
        expr: |
          slo:sli_error:ratio_rate6h > (6 * 0.005)
          and
          slo:sli_error:ratio_rate30m > (6 * 0.005)
        for: 15m
        labels: {severity: page, service: myproject}
        annotations:
          summary: "Error budget burning at 6x: {{ $value | humanizePercentage }} of requests bad over 6h"
          runbook_url: "https://app.example.com/runbooks/error-budget-burn"

      - alert: ErrorBudgetBurnTicket
        expr: |
          slo:sli_error:ratio_rate1d > (3 * 0.005)
          and
          slo:sli_error:ratio_rate2h > (3 * 0.005)
        for: 1h
        labels: {severity: ticket, service: myproject}
        annotations:
          summary: "10% of the monthly error budget spent in one day"
          runbook_url: "https://app.example.com/runbooks/error-budget-burn"

      - alert: SiteDown
        expr: probe_success{job="blackbox_https"} == 0
        for: 2m
        labels: {severity: page, service: myproject}
        annotations:
          summary: "{{ $labels.instance }} failing external probe"
          runbook_url: "https://app.example.com/runbooks/site-down"

      - alert: TLSCertExpiringSoon
        expr: probe_ssl_earliest_cert_expiry{job="blackbox_https"} - time() < 14 * 86400
        for: 1h
        labels: {severity: ticket, service: myproject}
        annotations:
          summary: "TLS certificate for {{ $labels.instance }} expires in under 14 days"
          runbook_url: "https://app.example.com/runbooks/tls-renewal"

      - alert: DiskWillFillIn4h
        expr: |
          predict_linear(node_filesystem_avail_bytes{job="node",fstype!~"tmpfs|overlay"}[6h], 4 * 3600) < 0
          and
          node_filesystem_avail_bytes{job="node"} / node_filesystem_size_bytes{job="node"} < 0.2
        for: 30m
        labels: {severity: page, service: myproject}
        annotations:
          summary: "{{ $labels.mountpoint }} on {{ $labels.instance }} will be full within 4h at the current rate"
          runbook_url: "https://app.example.com/runbooks/disk-full"

      - alert: PostgresDown
        expr: pg_up{job="postgres"} == 0
        for: 2m
        labels: {severity: page, service: myproject}
        annotations:
          summary: "postgres_exporter cannot reach PostgreSQL on {{ $labels.instance }}"
          runbook_url: "https://app.example.com/runbooks/postgres-down"

      - alert: ScrapeTargetDown
        expr: up == 0
        for: 10m
        labels: {severity: ticket, service: myproject}
        annotations:
          summary: "{{ $labels.job }} target {{ $labels.instance }} not scraped for 10m"
          runbook_url: "https://app.example.com/runbooks/scrape-down"

      - alert: Watchdog
        expr: vector(1)
        labels: {severity: none}
        annotations:
          summary: "Always firing. Proves Prometheus and Alertmanager can deliver notifications."

Look closely at the disk rule. predict_linear fits a straight line through six hours of free-space samples and extrapolates four hours ahead. On its own it fires whenever a volume briefly gains data quickly: a nightly pg_dump writing 20 GB to a half-empty disk extrapolates to "full by morning". Requiring less than 20% free as well removes that noise while still catching the runaway log file. A shorter lookback reacts faster but mistakes every burst for a trend.

Every page carries a summary that says what is wrong in one line and a runbook_url that says what to do. A good runbook fits on one screen: what the alert means for users, the first three commands to run, how to roll back the last deploy, and who to escalate to. An alert without a runbook is a puzzle delivered at 03:00.

When not to use burn-rate alerts: very low-traffic services. At 20 requests an hour, one failed request is a 5% error ratio in the 1h window and a 100% ratio in the 5m window, so a single bad request trips the 14.4x rule. For those services, page on the external probe alone, keep the ticket-level burn alert, and review the budget weekly. The same applies to internal admin tools where "users" means three colleagues who will simply tell you.

A worked incident

Consider a realistic failure. At 14:00 a deploy ships a change to the order history view that raises an exception for customers with a legacy address format. About 10% of all requests now return 500. Baseline badness was 0.1%. Here is how each rule responds.

• Fast burn (14.4x, 1h/5m). The 5m window crosses 7.2% within five minutes. The 1h window is a rolling average, so it climbs roughly 10% times the fraction of the hour elapsed; it crosses 7.2% after about 43 minutes. Add for: 2m and the page arrives around 14:45.
• Slow burn (6x, 6h/30m). The 6h window needs 3%, which takes about 108 minutes at a 10% error ratio. It would fire around 15:50 if nothing else had.
• Site down. Never fires. The health check still returns 200, which is exactly why the probe cannot be your only alert.

Forty-five minutes feels slow, and that is the honest trade-off. By the time the page fires, about 4.5 "minutes" of budget (roughly 2%) are gone. A 10% error rate is serious but not catastrophic, and paging earlier would mean paging for every brief spike. A total outage is detected far faster: at 100% errors the 1h window crosses 7.2% in about 4.3 minutes, and SiteDown fires at around the same time if the health check fails too. The detection time scales inversely with severity, which is precisely the behaviour you want.

The engineer rolls back at 14:55. Within five minutes the 5m ratio falls below threshold, the and condition fails, and the alert resolves, even though the 1h window stays above 7.2% for a while longer. Total cost: 55 minutes at 10% bad is about 5.5 budget-minutes, around 2.5% of the month. The post-incident review asks why the test suite missed the legacy address format, not why the alert was late.

Alertmanager routing, grouping and inhibition

Prometheus decides whether an alert fires; Alertmanager decides who hears about it, how often and in what bundles. The complete configuration:

# /etc/alertmanager/alertmanager.yml
global:
  smtp_smarthost: "smtp.example.com:587"
  smtp_from: "alerts@example.com"
  smtp_auth_username: "alerts@example.com"
  smtp_auth_password: "change-me"
  smtp_require_tls: true

route:
  receiver: ticket-email
  group_by: ["alertname", "service"]
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 12h
  routes:
    - matchers: ['alertname="Watchdog"']
      receiver: heartbeat
      group_wait: 0s
      group_interval: 1m
      repeat_interval: 5m
    - matchers: ['severity="page"']
      receiver: pager
      group_wait: 10s
      repeat_interval: 1h

inhibit_rules:
  # When the site is unreachable, burn-rate pages add nothing.
  - source_matchers: ['alertname="SiteDown"']
    target_matchers: ['alertname=~"ErrorBudgetBurn.*"']
    equal: ["service"]
  # An active page suppresses tickets for the same service.
  - source_matchers: ['severity="page"']
    target_matchers: ['severity="ticket"']
    equal: ["service"]

receivers:
  - name: ticket-email
    email_configs:
      - to: "ops@example.com"
        send_resolved: true
    webhook_configs:
      - url: "https://app.example.com/internal/alertmanager/"
        http_config:
          authorization:
            type: Bearer
            credentials_file: /etc/alertmanager/webhook_token

  - name: pager
    webhook_configs:
      - url: "https://hooks.example.com/pager"
        send_resolved: true
      - url: "https://app.example.com/internal/alertmanager/"
        http_config:
          authorization:
            type: Bearer
            credentials_file: /etc/alertmanager/webhook_token

  - name: heartbeat
    webhook_configs:
      - url: "https://hooks.example.com/heartbeat"

What the grouping settings actually do:

• group_by collapses alerts that share those label values into one notification. Three filesystems filling on the same host arrive as one message, not three.
• group_wait is how long a new group waits for siblings before the first notification. Ten seconds for pages; thirty for tickets, where batching matters more than speed.
• group_interval is how long to wait before notifying about new alerts joining a group that has already been notified.
• repeat_interval re-sends a still-firing, unchanged group. One hour for pages keeps a forgotten incident visible; 12 hours for tickets avoids training people to filter the inbox.

Routes are matched top to bottom and the first match wins, which is why Watchdog sits above the page route. The watchdog is the one exception to "every alert needs a severity": it always fires, and the external heartbeat service raises the alarm when it stops arriving. It is your only protection against the silent failure mode where Prometheus is down, Alertmanager is misconfigured or the monitoring host has run out of disk, and everything looks calm.

Inhibition removes consequences when the cause is already paging. Silences remove noise for a known, bounded period. Alertmanager also deduplicates: run two Prometheus replicas with identical rules against it and you still get one notification per group. For planned work, create a silence with an expiry and a comment rather than editing or commenting out rules:

amtool --alertmanager.url=http://127.0.0.1:9093 silence add \
  alertname=DiskWillFillIn4h instance="app.example.com:9100" \
  --duration=2h --comment="Reindexing orders table, expected disk growth (deploy)"

Keeping the noise down

An alerting system decays unless someone actively prunes it. The mechanism that works is boring: record every notification, then review them weekly. The second webhook in the receivers above posts each notification to a small Django endpoint. It is a log for review, not part of the paging path; if the site is down, the pager still works.

# ops/models.py
from django.db import models


class AlertNotification(models.Model):
    received_at = models.DateTimeField(auto_now_add=True)
    receiver = models.CharField(max_length=100)
    status = models.CharField(max_length=10)  # "firing" or "resolved"
    alertname = models.CharField(max_length=200)
    severity = models.CharField(max_length=20, blank=True)
    fingerprint = models.CharField(max_length=32)
    starts_at = models.DateTimeField()
    summary = models.TextField(blank=True)
    actionable = models.BooleanField(null=True)  # set during weekly review

    class Meta:
        indexes = [models.Index(fields=["alertname", "received_at"])]


# ops/views.py
import hmac
import json

from django.conf import settings
from django.http import HttpResponse, HttpResponseForbidden
from django.utils.dateparse import parse_datetime
from django.views.decorators.csrf import csrf_exempt
from django.views.decorators.http import require_POST

from ops.models import AlertNotification


@csrf_exempt
@require_POST
def alertmanager_webhook(request):
    expected = f"Bearer {settings.ALERT_WEBHOOK_TOKEN}".encode()
    supplied = request.headers.get("Authorization", "").encode()
    if not hmac.compare_digest(supplied, expected):
        return HttpResponseForbidden()
    payload = json.loads(request.body)
    AlertNotification.objects.bulk_create(
        AlertNotification(
            receiver=payload["receiver"],
            status=alert["status"],
            alertname=alert["labels"].get("alertname", ""),
            severity=alert["labels"].get("severity", ""),
            fingerprint=alert["fingerprint"],
            starts_at=parse_datetime(alert["startsAt"]),
            summary=alert["annotations"].get("summary", ""),
        )
        for alert in payload["alerts"]
    )
    return HttpResponse(status=204)

The weekly review then starts from data rather than memory: group last week's firing rows by alertname, counting rows (notifications) and distinct fingerprints (episodes).

Twenty minutes a week, with whoever was on call. For each alert ask three questions: did someone act on it, was the runbook right, and should it have been a ticket instead of a page? Then act on the answers:

• Delete alerts nobody acts on. If an alert fired five times and every response was "looked, fine", it is wrong. Fix the threshold or remove it. Deleting an alert is not a loss of safety; an ignored alert provides none.
• Use for: deliberately. It requires the condition to hold across consecutive evaluations before firing. Two minutes filters scrape blips and restarts on pages; longer durations on tickets cost nothing because nobody acts on them in minutes anyway.
• Demote before you delete. A page that was real but never urgent becomes a ticket for a month; if the ticket is also ignored, delete it.
• Track pages per on-call shift. More than two or three a week means the alerting has a bug, not that the week was busy.
• Read the budget. slo:error_budget_remaining:ratio30d below zero is the agreed signal to prioritise reliability work over features, which removes the argument from the meeting.

Testing alert rules

Rules are code, so test them. promtool test rules feeds synthetic series through your rules and checks which alerts fire at which time, without a running Prometheus.

# /etc/prometheus/rules/tests.yml
rule_files:
  - django-alerts.yml

evaluation_interval: 1m

tests:
  - interval: 1m
    input_series:
      - series: 'probe_success{job="blackbox_https",instance="https://app.example.com/healthz"}'
        values: "1 1 0x10"
    alert_rule_test:
      - eval_time: 3m
        alertname: SiteDown
        exp_alerts: []
      - eval_time: 5m
        alertname: SiteDown
        exp_alerts:
          - exp_labels:
              severity: page
              service: myproject
              job: blackbox_https
              instance: https://app.example.com/healthz
            exp_annotations:
              summary: "https://app.example.com/healthz failing external probe"
              runbook_url: "https://app.example.com/runbooks/site-down"

The probe fails from minute 2. At minute 3 the alert is still pending because of for: 2m, so no alert is expected; by minute 5 it must be firing. Tests like this catch the typical regressions: a renamed label that breaks an inhibition rule, or a for: duration changed by someone who did not realise what depended on it. Run them in CI alongside your Django test suite.

Common mistakes

• Scraping /metrics through a load-balanced URL with several Gunicorn workers, so each scrape reports a different process and error rates are nonsense.
• Leaving metrics ports or /metrics reachable from the internet, leaking view names, traffic volumes and error rates.
• Forgetting or vector(0) on the 5xx sum, so the SLI vanishes whenever there are no errors, and nobody notices the alerts can no longer fire correctly.
• Keeping the default 15-day retention while defining a 30-day SLO.
• Single-window burn alerts: a 1h window alone keeps paging for an hour after the rollback; a 5m window alone pages on every blip.
• Running Prometheus and Alertmanager on the application host with no watchdog, so the outage that takes the site down also silences the alerts.

Verify it

# Validate syntax and run the rule tests
promtool check config /etc/prometheus/prometheus.yml
promtool check rules /etc/prometheus/rules/django-slo.yml /etc/prometheus/rules/django-alerts.yml
cd /etc/prometheus/rules && promtool test rules tests.yml
amtool check-config /etc/alertmanager/alertmanager.yml

# Which receiver would a page for this service reach?
amtool config routes test --config.file=/etc/alertmanager/alertmanager.yml \
  severity=page service=myproject

# From the monitoring host: each worker exposes metrics, the probe works
curl -s http://app.example.com:8001/metrics | grep -c django_http_requests_latency_seconds_by_view_method_bucket
curl -s "http://127.0.0.1:9115/probe?module=http_healthz&target=https://app.example.com/healthz" \
  | grep -E "^probe_success|^probe_ssl_earliest_cert_expiry"


# Reload after changes
sudo systemctl reload prometheus
sudo systemctl reload alertmanager

Then open the Prometheus expression browser and check that slo:sli_error:ratio_rate5m returns a value, that up{job="django"} shows one healthy target per worker, and that the Watchdog alert is firing. Finally, test end to end: on a staging host, stop Gunicorn, confirm SiteDown reaches the pager webhook within about three minutes, start it again and confirm the resolved notification arrives. An alert you have never seen fire is a hypothesis, not a safeguard.

The goal is not more alerts but fewer, better ones: a handful of symptom-based pages tied to a budget that reflects what users feel, predictive tickets for slow-moving failures like disks and certificates, a watchdog for the monitoring itself, and a weekly habit of deleting whatever nobody acted on.