Observability for Django: Sentry, Structured Logging, Prometheus, and OpenTelemetry Tracing

When production breaks at 2am, the question is not whether you have logs — it is whether you can answer "what is happening, right now, and why" in minutes instead of hours. Observability is the practice that makes a running system explainable: errors you see immediately, logs you can actually query, metrics that reveal trends, and traces that follow one request across every service it touches. This tutorial builds all four pillars for Django: Sentry, structured logging, Prometheus metrics, and OpenTelemetry tracing.

Monitoring versus observability

Monitoring tells you that something is wrong — a dashboard goes red, an alert fires. Observability tells you why — it is the property of a system that lets you ask arbitrary questions about its behavior from the outside, without shipping new code to investigate. The distinction matters because modern systems fail in ways you did not predict, and a fixed set of dashboards cannot anticipate every question. Good observability means that when a novel problem appears, you already have the data to explore it. The four pillars below each answer a different kind of question, and together they turn a production incident from a guessing game into an investigation with evidence.

Error tracking with Sentry

The first and highest-value pillar is error tracking, and Sentry is the standard. Without it, errors hide in log files nobody reads until a user complains; with it, every exception is captured with its full stack trace, the request that caused it, the user affected, and the surrounding context, then grouped so a thousand occurrences of one bug are a single actionable issue.

import sentry_sdk
from sentry_sdk.integrations.django import DjangoIntegration

sentry_sdk.init(dsn="https://...@sentry.io/...",
                integrations=[DjangoIntegration()],
                traces_sample_rate=0.1, send_default_pii=False)

Setup is minutes, and the payoff is immediate: you learn about errors before users report them, with everything you need to reproduce and fix them already attached.

A production-grade Sentry configuration

The three-line setup is enough to start, but a production deployment needs a few more decisions made explicitly. Read the DSN from the environment rather than committing it, tag every event with an environment and a release so you can tell whether a bug appeared with Tuesday's deploy, and use before_send as the single choke point where you drop noise and scrub anything sensitive that slipped into the event:

# settings/production.py
import os
import sentry_sdk
from django.core.exceptions import DisallowedHost
from sentry_sdk.integrations.django import DjangoIntegration

IGNORED = (DisallowedHost,)   # scanners hitting the raw IP, not real bugs

def before_send(event, hint):
    exc_info = hint.get("exc_info")
    if exc_info and isinstance(exc_info[1], IGNORED):
        return None                      # drop the event entirely
    request = event.get("request", {})
    request.pop("cookies", None)         # belt and braces on top of send_default_pii=False
    return event

def before_send_transaction(event, hint):
    # Health checks and metrics scrapes are high-volume and uninteresting.
    if event.get("transaction") in {"/healthz", "/metrics"}:
        return None
    return event

sentry_sdk.init(
    dsn=os.environ["SENTRY_DSN"],
    environment=os.environ.get("APP_ENV", "production"),
    release=os.environ.get("GIT_SHA", "unknown"),
    integrations=[DjangoIntegration()],
    traces_sample_rate=0.1,
    send_default_pii=False,
    before_send=before_send,
    before_send_transaction=before_send_transaction,
)

Setting release to the commit SHA your CI builds is what unlocks "suspect commits" and regression detection: Sentry can tell you an issue first appeared in a specific release and reopen a resolved issue if it comes back. Bake the SHA into the image at build time (a build argument written to an environment variable) rather than calling git at runtime, where the repository usually does not exist.

Making error tracking useful

Error tracking degrades fast without hygiene. A flood of noisy, low-value errors trains the team to ignore the tool, so the discipline is to keep the signal high: filter out expected exceptions, resolve or mute issues deliberately, and attach context — the user, the tenant, the relevant IDs — so each error is actionable rather than a bare stack trace. Set up alerting for new or spiking issues so a fresh bug reaches you fast, but tune it so routine noise does not. An error tracker that everyone trusts because its alerts always mean something is worth far more than one that cries wolf and gets muted.

Structured logging

Traditional logs are strings written for humans, which makes them nearly impossible to query at scale — you cannot reliably filter millions of free-text lines by user or request. Structured logging emits logs as machine-readable key-value data (typically JSON), so every log carries queryable fields:

import structlog
log = structlog.get_logger()
log.info("order_placed", order_id=order.id, user_id=user.id,
         total=str(order.total), tenant_id=tenant.id)

Now you can ask your log system "show me all order_placed events for tenant 7 over $500 today" and get an answer. The shift from logs-as-prose to logs-as-data is what makes logs a genuine investigative tool rather than a wall of text you grep in desperation.

Wiring structlog into Django's logging

The snippet above works in isolation, but Django, Gunicorn, Celery and every third-party library log through the standard library's logging module. If only your own code emits JSON, your log pipeline receives a mix of JSON and plain text, and the plain lines — often the most important ones, such as django.request 500 errors — become unparseable. The fix is to route everything through one formatter: configure structlog to hand its events to the stdlib, and have a ProcessorFormatter render both structlog and foreign records as JSON.

# settings.py
import structlog

SHARED_PROCESSORS = [
    structlog.contextvars.merge_contextvars,   # request_id, user_id, ... bound per request
    structlog.stdlib.add_logger_name,
    structlog.stdlib.add_log_level,
    structlog.processors.TimeStamper(fmt="iso", utc=True),
]

structlog.configure(
    processors=SHARED_PROCESSORS + [
        structlog.stdlib.filter_by_level,
        structlog.processors.StackInfoRenderer(),
        structlog.stdlib.ProcessorFormatter.wrap_for_formatter,
    ],
    logger_factory=structlog.stdlib.LoggerFactory(),
    wrapper_class=structlog.stdlib.BoundLogger,
    cache_logger_on_first_use=True,
)

LOGGING = {
    "version": 1,
    "disable_existing_loggers": False,
    "formatters": {
        "json": {
            "()": structlog.stdlib.ProcessorFormatter,
            "foreign_pre_chain": SHARED_PROCESSORS,
            "processors": [
                structlog.stdlib.ProcessorFormatter.remove_processors_meta,
                structlog.processors.format_exc_info,
                structlog.processors.JSONRenderer(),
            ],
        },
    },
    "handlers": {
        "stdout": {"class": "logging.StreamHandler", "formatter": "json"},
    },
    "root": {"handlers": ["stdout"], "level": "INFO"},
    "loggers": {
        "django.db.backends": {"level": "WARNING"},   # never ship SQL at INFO
    },
}

Write to stdout and let the platform (systemd-journald, Docker, Kubernetes) collect it; writing to local files inside a container means rotating, shipping and losing them yourself. A single django.request error now arrives as one JSON object with a level, a logger, an ISO timestamp and the formatted traceback in an exception field, instead of twenty lines of traceback that a line-oriented shipper splits into twenty separate "events".

Redacting secrets before they leave the process

The advice to never log secrets is a policy; a processor makes it enforceable. Add a small redaction step to the shared chain so a careless log.info("login", password=...) or a logged request payload cannot leak:

SENSITIVE_KEYS = {"password", "token", "authorization", "secret", "api_key", "cookie"}

def redact(logger, method_name, event_dict):
    for key in list(event_dict):
        if key.lower() in SENSITIVE_KEYS:
            event_dict[key] = "[REDACTED]"
    return event_dict

Place redact in SHARED_PROCESSORS after merge_contextvars so it also sees bound context. It only inspects top-level keys, which is a deliberate trade-off: recursing into every nested structure on every log call costs CPU on the hot path. The stronger defence is to never pass whole request bodies or model instances to the logger in the first place — log the handful of IDs you need.

Contextual logging and correlation

Structured logs become powerful when every log line from a single request shares a correlation ID, so you can reconstruct the full story of one request across many log statements. Bind request-scoped context — a request ID, the user, the tenant — once, and have it automatically attached to every subsequent log in that request. This means when you find one interesting log line, you can pull every other line from the same request instantly. In a distributed system, propagating that correlation ID across service boundaries lets you follow one user action through every service's logs. Correlation is what turns scattered log lines into a coherent narrative of what happened.

A request-ID middleware

structlog's contextvars support makes request binding straightforward, and because context variables are isolated per thread and per asyncio task, it works for both WSGI and ASGI deployments. Accept an upstream ID from your load balancer when present, but validate it: the header is client-controlled, and an unvalidated value can inject fake fields or megabytes of junk into every log line.

import re
import uuid
import structlog

_VALID_ID = re.compile(r"^[A-Za-z0-9\-]{8,64}$")

class RequestIDMiddleware:
    def __init__(self, get_response):
        self.get_response = get_response

    def __call__(self, request):
        incoming = request.headers.get("X-Request-ID", "")
        request_id = incoming if _VALID_ID.match(incoming) else uuid.uuid4().hex
        request.request_id = request_id
        structlog.contextvars.clear_contextvars()
        structlog.contextvars.bind_contextvars(
            request_id=request_id,
            method=request.method,
            path=request.path,
        )
        response = self.get_response(request)
        response["X-Request-ID"] = request_id
        return response

Put it near the top of MIDDLEWARE so logs from later middleware carry the ID, and bind user_id in a second step after authentication has run. Returning the ID in the response header gives support staff something concrete to ask for: a user who reports "it failed" can paste the ID from the browser's network tab, and you jump straight to that request's logs. The clear_contextvars() call matters in thread-pooled servers, where a worker thread would otherwise carry the previous request's bindings into the next one if anything bound outside a request.

Log levels and what to log

Logging everything is as useless as logging nothing — the signal drowns. Use levels deliberately: DEBUG for development detail, INFO for significant business events, WARNING for recoverable problems worth noticing, ERROR for failures. Log the events that tell the story of your system's behavior — a placed order, a failed payment, a permission denial — not every trivial step. And never log secrets, tokens, passwords, or sensitive personal data, because logs are widely accessible and long-lived; a logged credential is a leaked credential. Thoughtful logging is curation: enough to explain the system, not so much that the important events are buried under noise.

Metrics with Prometheus

Metrics are numeric measurements over time — request rate, error rate, latency, queue depth, active users — and they reveal trends and aggregate health that individual logs and errors cannot. Prometheus is the standard: your app exposes a metrics endpoint, Prometheus scrapes it periodically, and you query and graph the results.

from prometheus_client import Counter, Histogram
orders = Counter("orders_total", "Orders placed", ["status"])
latency = Histogram("request_latency_seconds", "Request latency", ["view"])

Counters only go up (total orders, total errors), gauges move both ways (current connections), and histograms capture distributions (latency percentiles). Metrics are how you see the shape of your system's behavior over time rather than one moment of it.

Recording request metrics with a route label

The most useful single metric for a web app is a latency histogram labelled by route and status class. The trap is the label value: request.path contains IDs (/orders/18231/) and turns one series into millions. Use the URL pattern instead, which Django exposes after resolution as request.resolver_match.route:

import time
from prometheus_client import Histogram

REQUEST_LATENCY = Histogram(
    "http_request_duration_seconds",
    "HTTP request latency",
    ["method", "route", "status"],
    buckets=(0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1, 2.5, 5, 10),
)

class PrometheusMiddleware:
    def __init__(self, get_response):
        self.get_response = get_response

    def __call__(self, request):
        start = time.perf_counter()
        response = self.get_response(request)
        match = getattr(request, "resolver_match", None)
        # URL pattern such as "/orders/<int:pk>/"; unresolved 404s collapse into one series
        route = "/" + match.route if match else "unmatched"
        REQUEST_LATENCY.labels(
            method=request.method,
            route=route,
            status=f"{response.status_code // 100}xx",
        ).observe(time.perf_counter() - start)
        return response

Choose buckets around your latency objective: if the SLO threshold is 300ms, you want bucket boundaries near it (0.25 and 0.5 here) so histogram_quantile() interpolation is accurate where it matters. Grouping status codes into classes keeps the label bounded; collapsing unmatched paths into one value stops vulnerability scanners probing thousands of random URLs from creating thousands of series. If you would rather not maintain this yourself, the django-prometheus package provides equivalent middleware plus database and cache metrics.

Exposing metrics under Gunicorn: multiprocess mode

This is where most first attempts silently go wrong. Gunicorn runs several worker processes, each with its own in-memory registry, and a Prometheus scrape of /metrics reaches whichever worker happens to accept it. Counters appear to jump backwards and forwards between scrapes, and rates become nonsense. prometheus_client solves this with multiprocess mode: each worker writes its values to memory-mapped files in a shared directory, and the metrics view aggregates them.

# views.py
from django.http import HttpResponse
from prometheus_client import CONTENT_TYPE_LATEST, CollectorRegistry, generate_latest
from prometheus_client import multiprocess

def metrics_view(request):
    registry = CollectorRegistry()
    multiprocess.MultiProcessCollector(registry)
    return HttpResponse(generate_latest(registry), content_type=CONTENT_TYPE_LATEST)

# gunicorn.conf.py
from prometheus_client import multiprocess

def child_exit(server, worker):
    multiprocess.mark_process_dead(worker.pid)

Set PROMETHEUS_MULTIPROC_DIR to an empty, writable directory (for example /srv/app/run/prom or a tmpfs) before Gunicorn starts, and wipe it on every start, otherwise stale files from previous runs are aggregated forever. Gauges need an explicit aggregation mode in this setup (the multiprocess_mode argument, such as "livesum" or "max"), because "the current value" is ambiguous across several processes. Finally, do not expose /metrics to the internet: route map, traffic volumes and error rates are useful reconnaissance. Serve it on an internal port, restrict it at the reverse proxy to the Prometheus server's address (such as 10.0.0.5), or require a bearer token.

The golden signals

With metrics you could measure anything, so it helps to start with the signals that matter most. The "four golden signals" are latency (how long requests take), traffic (how many requests), errors (how many fail), and saturation (how full your resources are). Tracked together, these four give a remarkably complete picture of service health and are usually enough to detect most problems. Latency and error rate at the percentiles that matter — p95, p99, not just the average — reveal pain that averages hide. Beginning your metrics with the golden signals, then adding business-specific metrics like orders or signups, gives you broad coverage without drowning in dashboards.

Dashboards and alerting

Metrics are only useful if someone sees them, and the two ways are dashboards and alerts. Dashboards (typically Grafana over Prometheus) visualize trends for at-a-glance health and investigation. Alerts watch metrics and notify you when something crosses a threshold — error rate spiking, latency climbing, a queue backing up — so you respond before users feel it. The art of alerting is the same as error hygiene: alert on symptoms users care about, make every alert actionable, and avoid noise that trains people to ignore the pager. A well-tuned alert that means "users are being hurt right now" is the payoff of all the metrics work.

Distributed tracing with OpenTelemetry

In a system of multiple services, a single user request fans out across many of them, and the hardest question is "where did the time go?" Distributed tracing answers it. A trace follows one request end to end, recording each step — each service call, each database query — as a timed span, assembled into a tree that shows exactly where the request spent its time. OpenTelemetry is the vendor-neutral standard for producing this data, with instrumentation for Django that captures requests and queries automatically. Tracing turns "the request was slow" into "it spent 800ms in the recommendations service waiting on one query," which is the difference between guessing and knowing.

Setting up the OpenTelemetry SDK for Django

The minimal production setup has four parts: a resource that names the service, a tracer provider with a sampler, a batch processor that exports spans to an OpenTelemetry Collector over OTLP, and instrumentors for Django, the database driver and outgoing HTTP. The packages are opentelemetry-sdk, opentelemetry-exporter-otlp, opentelemetry-instrumentation-django, opentelemetry-instrumentation-psycopg and opentelemetry-instrumentation-requests.

# myproject/telemetry.py
import os
from opentelemetry import trace
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
from opentelemetry.instrumentation.django import DjangoInstrumentor
from opentelemetry.instrumentation.psycopg import PsycopgInstrumentor
from opentelemetry.instrumentation.requests import RequestsInstrumentor
from opentelemetry.sdk.resources import Resource
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.sdk.trace.sampling import ParentBased, TraceIdRatioBased

def configure_tracing():
    resource = Resource.create({
        "service.name": "myproject-web",
        "service.version": os.environ.get("GIT_SHA", "unknown"),
        "deployment.environment": os.environ.get("APP_ENV", "production"),
    })
    provider = TracerProvider(
        resource=resource,
        sampler=ParentBased(TraceIdRatioBased(0.1)),
    )
    provider.add_span_processor(BatchSpanProcessor(OTLPSpanExporter()))
    trace.set_tracer_provider(provider)

    DjangoInstrumentor().instrument()
    PsycopgInstrumentor().instrument()
    RequestsInstrumentor().instrument()

OTLPSpanExporter() with no arguments reads OTEL_EXPORTER_OTLP_ENDPOINT, so the same code can point at a Collector sidecar on localhost:4317 in one environment and a central gateway in another. ParentBased is important in a multi-service system: it honours the sampling decision in an incoming traceparent header, so if the upstream service decided to keep a trace, every downstream service keeps its part too and you never end up with traces missing their middle.

Where you call configure_tracing() matters. BatchSpanProcessor exports from a background thread, and threads do not survive fork(). If you initialise in settings.py and run Gunicorn with --preload, the master process creates the thread, forks workers, and the workers silently queue spans that are never exported. Call it from Gunicorn's post_fork hook instead, so each worker builds its own provider (with DJANGO_SETTINGS_MODULE already set in the environment, since the Django instrumentor reads settings.MIDDLEWARE to insert its own middleware):

# gunicorn.conf.py
def post_fork(server, worker):
    from myproject.telemetry import configure_tracing
    configure_tracing()

Custom spans around business operations

Auto-instrumentation shows HTTP requests and SQL queries. It cannot show that the slow part of checkout was the tax calculation, because that is plain Python. Wrap operations you care about in a named span and attach attributes that make it searchable:

from opentelemetry import trace
from opentelemetry.trace import Status, StatusCode

tracer = trace.get_tracer(__name__)

def place_order(cart, user):
    with tracer.start_as_current_span("checkout.place_order") as span:
        span.set_attribute("cart.items", len(cart.items))
        span.set_attribute("tenant.id", str(user.tenant_id))
        try:
            with tracer.start_as_current_span("checkout.calculate_tax"):
                tax = calculate_tax(cart)
            with tracer.start_as_current_span("checkout.charge_payment"):
                charge = payment_gateway.charge(cart.total + tax)
        except PaymentDeclined as exc:
            span.record_exception(exc)
            span.set_status(Status(StatusCode.ERROR, "payment declined"))
            raise
        span.set_attribute("order.charge_id", charge.id)
        return charge

Unlike metric labels, span attributes can safely carry high-cardinality values such as order IDs, because each span is stored individually rather than creating a time series. What they must not carry is sensitive data: card numbers, addresses and tokens do not belong in attributes any more than in logs. Name spans after the operation (checkout.charge_payment), not after the instance (charge order 18231), so that the tracing backend can aggregate latency by span name.

Spans, context, and propagation

A trace is built from spans, each representing a unit of work with a start, end, and metadata, nested to show causality. The key mechanism is context propagation: a trace ID is passed from service to service (in request headers), so every span across every service is linked into one trace. This is what lets you see a single user action stitched together across your whole architecture, even as it crosses network boundaries. Add custom spans around your own expensive operations to make them visible in the trace. Understanding spans and propagation is what lets tracing reveal the true, end-to-end path of a request rather than isolated fragments.

Sampling and cost

Tracing and detailed telemetry generate enormous data volume, and capturing every trace is expensive and usually unnecessary. Sampling keeps a representative fraction — perhaps ten percent of normal traffic — while ensuring you capture the interesting cases like errors and slow requests, which is where tail-based sampling helps by deciding what to keep after seeing the whole trace. The goal is enough data to diagnose problems without paying to store every routine request. Balancing observability depth against cost is a real operational concern; sample the mundane, keep the anomalous, and you get the insight you need at a price you can sustain.

Head sampling versus tail sampling

The TraceIdRatioBased(0.1) sampler above is head sampling: the decision is made when the root span starts, before anyone knows whether the request will fail or be slow. It is cheap and needs no infrastructure, but it discards 90% of your errors along with 90% of your boring requests. Tail sampling defers the decision until the whole trace has been seen, which requires buffering spans somewhere central — in practice the OpenTelemetry Collector (the contrib distribution) with its tail_sampling processor:

# otel-collector.yaml (excerpt)
processors:
  tail_sampling:
    decision_wait: 10s
    num_traces: 50000
    policies:
      - name: keep-errors
        type: status_code
        status_code: {status_codes: [ERROR]}
      - name: keep-slow
        type: latency
        latency: {threshold_ms: 500}
      - name: sample-the-rest
        type: probabilistic
        probabilistic: {sampling_percentage: 5}

A trace is kept if any policy matches, so this keeps every failed trace, every trace over 500ms, and 5% of the rest. To make that work, the applications must send all spans to the Collector (set the SDK sampler to always-on, or a high ratio) and let the Collector drop them. That shifts cost from storage to Collector memory and network, and it introduces a constraint that catches teams out: every span of a given trace must reach the same Collector instance. With more than one Collector you need a first tier that routes by trace ID (the load-balancing exporter) in front of the sampling tier.

Note the last row: Sentry's error events are not subject to traces_sample_rate. Every exception is still reported; only the performance transactions are sampled, so lowering the rate saves quota without hiding bugs.

Tying the pillars together

The four pillars are most powerful when connected. An error in Sentry links to the trace that produced it; that trace links to the structured logs from the same request via a shared correlation ID; and metrics show whether this error is an isolated event or part of a spike. This connected workflow is the goal: you notice a problem from a metric or alert, jump to the error for the stack trace, follow the trace to see where time and failure occurred, and read the correlated logs for the detail. A correlation or trace ID threaded through all four is the glue that turns separate tools into a single investigative surface.

Putting the trace ID into every log line

Once tracing is running, the OpenTelemetry trace ID is a better correlation key than a home-grown request ID, because it already crosses service boundaries via the traceparent header. A structlog processor can read the current span and add its IDs to each event:

from opentelemetry import trace

def add_trace_context(logger, method_name, event_dict):
    ctx = trace.get_current_span().get_span_context()
    if ctx.is_valid:
        event_dict["trace_id"] = format(ctx.trace_id, "032x")
        event_dict["span_id"] = format(ctx.span_id, "016x")
    return event_dict

Add it to SHARED_PROCESSORS. The hex formatting matters: the IDs are stored as integers, and tracing backends display and search them as 32- and 16-character lowercase hex strings. If your log and trace backends support it, configure a derived field that turns trace_id into a link, and you can click from a log line straight to the waterfall. Keep the request ID as well; it is what users and support staff can see in a response header, whereas trace IDs are not guaranteed to exist for unsampled requests in every setup.

A worked incident, end to end

To see why the connections matter, follow one realistic incident. At 14:05 the burn-rate alert for checkout fires: the 5xx ratio on route /checkout/ has jumped from 0.1% to 4%. The golden-signals dashboard shows latency p99 on the same route rising to exactly 30 seconds, while traffic is normal and other routes are healthy — so this is one dependency, not a general overload. In Sentry, a new issue appeared at 14:03 in release a1b2c3d: ReadTimeout from the payment client, tagged across many tenants, not one. Opening a linked trace shows the checkout.charge_payment span taking 30s and ending in error, with the outgoing HTTP span to the gateway accounting for all of it. Searching logs by that trace's trace_id shows the retry wrapper logging three attempts at 10s each. The diagnosis — a degraded upstream provider amplified by a retry policy whose total exceeds any sensible user wait — takes minutes because each tool answered one question and handed you the key to the next. Without correlation, the same investigation starts with grepping timestamps across four systems.

SLIs, SLOs, and error budgets

Observability data becomes a management tool through service-level objectives. A service-level indicator is a measured signal — say the percentage of requests served under 300ms; a service-level objective is the target for it, such as 99.9% over a month; and the error budget is the allowed shortfall. This framework turns vague aspirations about reliability into concrete numbers everyone agrees on, and the error budget gives a principled way to balance shipping features against stability: when you are within budget you can move fast, and when you are burning it you slow down and shore up reliability. Grounding reliability in measured objectives rather than opinion is one of the most valuable things observability enables.

Alerting on error-budget burn rate

SLOs also give you better alerts than fixed thresholds. Instead of "error rate above 1%", alert on how fast you are consuming the budget. With a 99.9% objective over 30 days, the budget is 0.1% of requests; a burn rate of 1 spends exactly the budget in 30 days, while a burn rate of 14.4 spends 2% of the monthly budget in one hour. A widely used pattern pairs a long window (to confirm the problem is significant) with a short one (so the alert clears quickly once it is fixed):

# prometheus/rules/checkout-slo.yml
groups:
  - name: checkout-slo
    rules:
      - record: route:error_ratio:rate5m
        expr: |
          sum by (route) (rate(http_request_duration_seconds_count{status="5xx"}[5m]))
            / sum by (route) (rate(http_request_duration_seconds_count[5m]))
      - record: route:error_ratio:rate1h
        expr: |
          sum by (route) (rate(http_request_duration_seconds_count{status="5xx"}[1h]))
            / sum by (route) (rate(http_request_duration_seconds_count[1h]))
      - alert: CheckoutErrorBudgetFastBurn
        expr: |
          route:error_ratio:rate1h{route="/checkout/"} > (14.4 * 0.001)
          and
          route:error_ratio:rate5m{route="/checkout/"} > (14.4 * 0.001)
        labels:
          severity: page
        annotations:
          summary: "Checkout is burning its 30-day error budget 14x too fast"
          runbook: "https://wiki.example.com/runbooks/checkout-errors"

A second, slower rule (for example a burn rate of 6 over six hours, or 1 over three days) routed to a ticket queue rather than the pager catches slow leaks that never trip the fast one. Recording rules keep the alert expressions cheap to evaluate and give you the same series to graph on the SLO dashboard. Low-traffic routes are the edge case: with ten requests an hour, a single failure is a 10% error ratio, so either require a minimum request count in the expression or aggregate small routes into a shared SLO.

The cardinality trap in metrics

A subtle but expensive mistake with metrics is high cardinality — attaching labels with many possible values, like a user ID or a request ID, to a metric. Each unique combination of label values creates a separate time series, and labels with thousands or millions of values explode the number of series, overwhelming your metrics system and ballooning cost. Labels should be low-cardinality dimensions you slice by — status code, endpoint, region — not unbounded identifiers, which belong in logs and traces instead. Understanding the cardinality trap is essential to using metrics sustainably; it is the difference between a metrics system that stays fast and affordable and one that collapses under its own series count.

Avoiding alert fatigue

The failure mode of alerting is not too few alerts but too many: a flood of noisy, non-actionable alerts trains responders to ignore the pager, so the one alert that matters gets missed among the noise. Good alerting is ruthless about signal — alert on symptoms users actually feel, make every alert require a human action, and route informational signals to dashboards rather than the pager. Tuning alerts so each one means "something is wrong that needs you now" preserves the trust that makes alerting work. An alerting system everyone trusts because it never cries wolf is far more valuable than a comprehensive one that everyone has muted.

Observing the user's experience

Server-side observability tells you what your backend did, but the user experiences the whole journey including the browser, the network, and rendering. Real user monitoring captures performance from the user's side — how long pages actually took to load and become interactive for real visitors on real devices and connections. This often reveals that a backend you measure as fast is slow in users' hands due to network latency, heavy assets, or client rendering. Complementing backend observability with a view of the actual user experience closes the gap between "our servers responded quickly" and "our users had a fast experience," which are not the same thing and sometimes diverge sharply.

Summary

Observability is what makes a running system explainable when it matters most, and it rests on four pillars. Sentry captures errors with the full context to fix them, kept useful through hygiene and tuned alerting. Structured logging turns logs from un-queryable prose into machine-readable data, made coherent by correlation IDs that reconstruct each request. Prometheus metrics reveal trends and aggregate health — start with the golden signals of latency, traffic, errors, and saturation — surfaced through dashboards and actionable alerts. OpenTelemetry tracing follows one request across every service via context propagation, showing exactly where time and failures occur, sampled to control cost. Connect all four with a shared trace or correlation ID, and an incident becomes an investigation with evidence rather than a frantic guess. Invest in observability before you need it, because the time to instrument your system is not while it is on fire.