Linux Performance Troubleshooting for Django Servers: The USE Method, vmstat, iostat, ss and strace

"The site is slow" is a symptom, and the usual first reaction of restarting gunicorn destroys the evidence you need to diagnose it. A Django server has only a few resources that can run out: CPU, memory, disk, network, the worker pool, file descriptors and the database behind it. If you check each of them in a fixed order with standard Linux tools, you'll usually find the bottleneck within minutes. This guide covers which commands to run, how to read their output, and how to tell an application problem from a database problem. The sample outputs are illustrative but realistic, and annotated with what each one tells you.

The first 60 seconds

When an alert fires, work through this list before changing anything. Each command takes a few seconds, and together they cover every major resource. Most come from the sysstat package (sudo apt install sysstat on Ubuntu 24.04). Install it on every server now. Installing packages on a box that is already struggling makes things worse and takes longer than you expect.

uptime                       # load averages: is this new or long-running?
sudo dmesg -T | tail -n 30   # OOM kills, I/O errors, conntrack/queue drops
vmstat 1                     # run queue, blocked tasks, swap, iowait, steal
mpstat -P ALL 1              # one hot core, or all of them?
pidstat 1                    # which processes are actually on CPU
iostat -xz 1                 # per-device latency and queue depth
free -m                      # real available memory
sar -n DEV 1                 # interface throughput
top                          # or htop: a sanity check of all of the above

The order matters. uptime and dmesg tell you whether something dramatic has happened, such as a kernel OOM kill or a failing disk. vmstat gives a one-line summary of CPU, memory and I/O pressure. The next four commands narrow down whichever of those looked abnormal. top comes last on purpose. It's the tool everyone opens first, and it encourages you to stare at whichever process sits at the top of the list, even when the real problem is a process that is waiting rather than working.

What load average really means

On Linux, the load average is an exponentially damped average of tasks that are runnable (on a CPU or waiting for one) plus tasks in uninterruptible sleep (state D). D state usually means waiting on block I/O, although NFS mounts and some kernel locks also cause it. So a load of 12 on a 4-core machine can mean one of two opposite things: the CPUs are saturated with three times more work than they can run, or twelve processes are stuck on a slow disk while the CPUs sit idle. Load average tells you which way to look, not what's wrong.

$ uptime
 14:02:11 up 41 days,  3:17,  1 user,  load average: 0.31, 0.28, 0.25
# 1, 5 and 15 minute averages. Low and flat while the site times out:
# nothing is burning CPU or stuck on disk. The workers are *waiting* on
# something outside this box: the network, a lock, or the database.

$ ps -eo state,pid,wchan:32,cmd | awk '$1 ~ /D/'
D   9121 io_schedule                    /srv/app/venv/bin/python manage.py export_orders
# D-state tasks and the kernel function they sleep in. If this list is long
# and stable, go straight to iostat.

Compare the 1-minute figure with the 15-minute one. If the 1-minute value is much higher, the problem started recently and you should look for a recent deploy, cron job or traffic spike. If all three are high and similar, the box has been overloaded for a while and you're probably looking at a capacity problem.

The USE method

The USE method is a checklist. For every resource, check Utilisation (how much of the time it's busy), Saturation (how much extra work is queued because it's busy) and Errors. Saturation is the one that hurts latency. A CPU at 100% with nothing queued is simply efficient. A disk at 60% utilisation with a deep queue is already adding milliseconds to every request. Working through the whole table also stops you fixating on the first odd number you happen to see.

The last three rows are software resources, and on Django servers they cause more incidents than hardware does. Nine sync workers can serve nine requests at once. The tenth waits in the kernel, however idle the CPUs are. Once you think of the worker pool as a resource with its own utilisation and saturation, a lot of "the server is fine but the site is slow" incidents become easy to explain.

Pressure stall information

Kernels since 4.20 expose PSI files that answer the saturation question directly. They report the percentage of time at least one task (some) or all tasks (full) were stalled waiting for the resource:

$ cat /proc/pressure/io
some avg10=41.27 avg60=38.02 avg300=12.55 total=918273645
full avg10=36.90 avg60=33.14 avg300=10.02 total=801928374
# For ~40% of the last 10 seconds some task was stalled on I/O, and the
# 300-second average is much lower: this started a few minutes ago.

CPU: vmstat, mpstat and pidstat

$ vmstat 1
procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 3  0      0 201344  61204 2209980   0    0     5    41  910 1502 22  4 74  0  0
 9  0      0 182340  61204 2210488   0    0     2    40 3120 5410 71 12  3  0 14
11  0      0 181920  61204 2210520   0    0     0    36 3304 5822 69 13  2  0 16
# Ignore the first line: it is the average since boot, not a sample.

• r: runnable tasks, both running and waiting for a CPU. If it's consistently above your core count, the CPUs are saturated.
• b: tasks blocked in D state, usually on I/O. A non-zero value that doesn't go away means you should check iostat.
• si/so: memory swapped in and out per second. Sustained non-zero values mean memory saturation. A large swpd with zero si/so is harmless: pages were swapped out long ago and nobody needs them.
• us/sy: user and kernel CPU. High sy together with a high cs (context switches) often means too many workers or threads contending for a small machine.
• wa: idle time during which I/O was outstanding. It suggests I/O is involved but doesn't measure disk capacity. On a many-core box it can be low even when a single-threaded process is stuck on disk.
• st: steal, the time your virtual CPU wanted to run but the hypervisor ran another guest. The sample above shows 14–16% steal with only 2–3% idle. Tuning Django won't fix that. A larger or dedicated-CPU instance will. Check steal early on any VPS, because it looks exactly like "our code got slower" when nothing in your code has changed.

$ mpstat -P ALL 1
CPU    %usr  %nice   %sys %iowait   %irq  %soft  %steal  %idle
all   26.10   0.00   3.02    0.00   0.00   0.50    0.00  70.38
  0   98.00   0.00   2.00    0.00   0.00   0.00    0.00   0.00
  1    3.03   0.00   3.03    0.00   0.00   1.01    0.00  92.93
  2    2.00   0.00   4.00    0.00   0.00   0.00    0.00  94.00
  3    1.01   0.00   3.03    0.00   0.00   1.01    0.00  94.95
# (columns trimmed) "all" says 70% idle, but CPU 0 is pinned. A single
# process is CPU-bound: one Python process, which the GIL keeps on one core.

$ pidstat 1
  UID       PID    %usr %system  %guest   %wait    %CPU   CPU  Command
 1001     48377   97.00    1.00    0.00    0.00   98.00     0  gunicorn
 1001     48378    0.00    0.00    0.00    0.00    0.00     1  gunicorn
# Worker 48377 is spinning. Next step: py-spy dump --pid 48377.

This pattern of one pinned core with the others idle is common with Python. A single worker stuck in a pathological loop, a catastrophic regex or a huge in-memory sort shows up as exactly one busy core. Averaged across all cores it looks like light load, which is why mpstat is in the checklist even though vmstat already reports CPU usage. pidstat -d 1 (disk I/O per process) and pidstat -r 1 (memory and page faults per process) are handy variants of the same tool.

Disk: iostat and the %util trap

$ iostat -xz 1
Device    r/s    w/s   rkB/s    wkB/s r_await w_await aqu-sz  %util
nvme0n1  4.00 812.00    64.0  24310.0    0.21    0.35   0.29  98.40
vdb    310.00  95.00  9920.0   3040.0   38.70   52.10  15.62  99.80
# (columns trimmed; -z hides idle devices)
# nvme0n1: "98% util" but 0.35 ms per write and a queue of 0.3. It's fine.
# vdb:     ~40-50 ms per request and 15 requests queued. Saturated.

%util is the percentage of time the device had at least one request in flight. On a single spinning disk that could only serve one request at a time, it was a reasonable saturation signal. NVMe drives and network-attached cloud volumes serve many requests in parallel, so 100% %util can sit alongside sub-millisecond latency and plenty of spare capacity. Decide using these columns instead:

• r_await / w_await: average milliseconds per read or write, including time spent queued. For NVMe, single-digit milliseconds is already slow. Network volumes routinely sit at 1–5 ms, and tens of milliseconds means trouble.
• aqu-sz: average queue length. If it grows while throughput stays flat, the device is saturated.
• r/s + w/s: compare them against the volume's provisioned IOPS. Cloud block storage throttles at a fixed IOPS or throughput ceiling. If latency jumps sharply at a suspiciously round number of IOPS, you've hit the provider's cap, not the physical limit of the hardware.

On a Django host, heavy disk I/O is rarely Django itself. The usual causes are verbose logging to disk, a local PostgreSQL sharing the volume, backups, logrotate compressing large files, or a management command writing a big export. Use pidstat -d 1 to see which process is doing the I/O before you blame the application.

Memory: page cache, swap and the OOM killer

$ free -m
               total        used        free      shared  buff/cache   available
Mem:            3915        2710         143          38        1301        1205
Swap:              0           0           0
# "free" is low by design: the kernel uses spare RAM as page cache and
# drops it instantly when programs need memory. "available" (1205 MB) is
# the number that matters: what new allocations can get without swapping.

A low free figure is normal and healthy. The real warning signs are a low available, sustained non-zero si/so in vmstat, and rising /proc/pressure/memory. Before the kernel runs out entirely it starts evicting page cache, including the cached parts of your Python code, templates and static files. Everything then gets slower before anything crashes, and that gradual decline is easy to mistake for a code regression.

When memory really does run out, the kernel's OOM killer picks the process with the highest oom_score, which is mostly determined by its memory use, and kills it. On a Django box that's usually a gunicorn worker or a Celery child process. The gunicorn arbiter notices and forks a replacement, so the only visible symptom may be an occasional 502 from nginx and a worker PID that keeps changing.

$ sudo dmesg -T | grep -iE 'out of memory|oom-kill'
[Sat Sep 26 03:12:44 2026] oom-kill:constraint=CONSTRAINT_NONE,...,task=gunicorn,pid=48213,uid=1001
[Sat Sep 26 03:12:44 2026] Out of memory: Killed process 48213 (gunicorn) total-vm:1893420kB, anon-rss:1502112kB, ...

# The kernel ring buffer rotates; the journal keeps history across it
$ journalctl -k --since "-2 days" | grep -i 'killed process'

# Who the kernel will pick next: higher oom_score = more likely victim
$ for p in $(pgrep -f gunicorn); do echo "$p $(cat /proc/$p/oom_score)"; done

If a single worker has 1.5 GB of anonymous RSS, look for a view that materialises a huge queryset: list(qs) over a large table, an unbounded CSV export, or an admin changelist with no pagination on a big model. QuerySet.iterator() streams rows from the database cursor instead of caching the whole result, and StreamingHttpResponse keeps exports flat in memory. For slow leaks, such as caches that grow for as long as a worker lives, set --max-requests 1000 --max-requests-jitter 100 so workers are recycled regularly and don't all restart at the same moment. Treat that as a safety net, not a fix. Adding swap to "solve" OOM kills usually turns quick crashes into minutes of thrashing, where every worker gets slow instead of one worker dying.

Network: sockets, listen queues and TIME_WAIT

Bandwidth is rarely the limit for a Django app. sar -n DEV 1 takes a second to rule it out, by comparing rxkB/s and txkB/s against the link speed. The interesting network problems happen at the socket level, where ss is the tool to use:

$ ss -s
Total: 2104
TCP:   1893 (estab 212, closed 1641, orphaned 0, timewait 1638)
# 1638 TIME_WAIT: normal on a busy proxy (see below). 212 established.

$ ss -tan state established '( dport = :5432 )' | wc -l    # conns to PostgreSQL (+1 header)
$ ss -tanp state established '( sport = :8000 )'           # who is connected to gunicorn
$ ss -tan state close-wait                                 # peer closed, our app never did

CLOSE_WAIT deserves a mention because, unlike TIME_WAIT, it's always an application bug. The remote side has closed the connection and your process never called close(). A growing CLOSE_WAIT count on outbound connections usually goes hand in hand with a file descriptor leak.

The accept queue: where requests wait for a free worker

When all gunicorn workers are busy, new connections from nginx don't fail straight away. The kernel completes the TCP handshake and parks them in the listening socket's accept queue until a worker calls accept(). Once that queue is full, the kernel starts dropping connection attempts. ss -lnt shows the queue directly:

$ ss -lnt 'sport = :8000'
State   Recv-Q  Send-Q   Local Address:Port   Peer Address:Port
LISTEN  2048    2048         127.0.0.1:8000        0.0.0.0:*
# For LISTEN sockets only: Recv-Q = connections waiting to be accept()ed,
# Send-Q = the maximum backlog. Recv-Q == Send-Q: the queue is full.

$ nstat -az TcpExtListenOverflows TcpExtListenDrops
TcpExtListenOverflows           18342              0.0
TcpExtListenDrops               18342              0.0
# Absolute counters since boot (-a) including zero values (-z).
# Run plain "nstat TcpExtListenOverflows" repeatedly to see the delta
# since the previous run; a rising number means drops are happening now.

Gunicorn's --backlog defaults to 2048, and the kernel silently caps it at net.core.somaxconn, which is 4096 on current kernels. It's tempting to raise both when you see overflows, but that rarely helps. A longer queue just means requests wait longer before nginx gives up with a 504. The queue fills because the workers are busy, so the fix is whatever is keeping them busy. A large backlog is useful for short bursts. It won't help when every request has become slow.

TIME_WAIT myths

Thousands of TIME_WAIT sockets are normal on any busy server. The side that closes a TCP connection first keeps a small kernel structure for about 60 seconds so that stray packets from the old connection can't corrupt a new one. It isn't a process, it doesn't use a file descriptor, and it costs a few hundred bytes of kernel memory. Some common advice is out of date. net.ipv4.tcp_tw_recycle was removed from the kernel years ago because it broke clients behind NAT, so a sysctl file that sets it just logs an error. tcp_tw_reuse only affects outgoing connections and does nothing for sockets your server accepted.

TIME_WAIT only becomes a real problem when your app opens outbound connections so fast that it runs out of ephemeral ports to the same destination. The classic case is Django opening a new connection to Redis, an internal HTTP API or PostgreSQL on every request. The symptom is EADDRNOTAVAIL ("Cannot assign requested address"). The fix is reusing connections: CONN_MAX_AGE or Django's built-in psycopg pool ("pool": True in OPTIONS, available since Django 5.1), one shared Redis client per process, and a module-level requests.Session instead of calling requests.get() directly.

A stuck worker: strace and py-spy

Once you know the workers are busy, the next question is what they're busy doing. Two tools answer it from different angles. strace shows the system calls a process makes, which tells you what it's asking the kernel for and how long each call takes. py-spy reads the interpreter's memory and prints the Python stack, which tells you which line of your code made the call. Use them together.

$ pgrep -af gunicorn
48370 /srv/app/venv/bin/python /srv/app/venv/bin/gunicorn myproject.wsgi ...   # arbiter
48377 /srv/app/venv/bin/python /srv/app/venv/bin/gunicorn myproject.wsgi ...   # worker
...
$ sudo strace -p 48377 -f -tt -T -yy -s 128 -e trace=network,read,poll
14:03:02.118004 sendto(9<TCP:[127.0.0.1:41822->127.0.0.1:5432]>, "Q\0\0\1\"SELECT ...", 291, MSG_NOSIGNAL, NULL, 0) = 291 <0.000031>
14:03:02.118112 poll([{fd=9, events=POLLIN}], 1, -1 ...
# -p attach to a running PID      -f  follow all threads
# -tt wall-clock timestamps       -T  time spent inside each call <secs>
# -yy decode fds to endpoints     -s  show up to 128 bytes of strings
# The worker sent a query to :5432 and is blocked waiting for the reply
# with no timeout (-1). This worker is waiting on PostgreSQL.

Depending on the driver and its version, you'll see the wait as poll, select or a blocking recvfrom. The call name matters less than the file descriptor it's waiting on and how long it has been there. A socket to port 5432 points to the database. Port 6379 is Redis, port 443 is an outbound HTTP call (a payment API, an S3 upload, a webhook), and futex means the worker is waiting for a lock held by another thread. For a quick overview of where a busy process spends its time, strace -c -p PID collects per-syscall counts and totals. Press Ctrl-C after a few seconds to print the summary.

strace has a cost. It uses ptrace, which stops the traced process on every system call, so a syscall-heavy process can run several times slower while it's attached. Attach to one worker for a few seconds, then detach with Ctrl-C. Don't leave it running against production traffic. On Ubuntu, the Yama ptrace policy (kernel.yama.ptrace_scope = 1) means you need sudo even for your own processes.

$ sudo py-spy dump --pid 48377
Process 48377: /srv/app/venv/bin/python /srv/app/venv/bin/gunicorn myproject.wsgi ...
Python v3.12.x (/srv/app/venv/bin/python)

Thread 48377 (idle): "MainThread"
    wait (psycopg/waiting.py:...)
    execute (psycopg/cursor.py:...)
    _execute (django/db/backends/utils.py:...)
    ...
    __iter__ (django/db/models/query.py:...)
    get_context_data (reports/views.py:88)
    get (django/views/generic/list.py:...)
    ...
    handle_request (gunicorn/workers/sync.py:...)

py-spy dump pauses the process only for the moment it takes to read the stack, so it's safe to use in production. It tells you the exact view and line, which strace can't. Add --locals to see local variables in each frame, which is useful for finding out which customer or which filter triggered the pathological query. If the workers are burning CPU rather than waiting, py-spy top --pid 48377 gives a live sampling profile, and py-spy record writes a flame graph. Taking a dump of every worker in a loop is often the fastest way to find out whether they're all stuck in the same place.

File descriptor leaks and ulimit -n

$ ls /proc/48377/fd | wc -l
1019
$ grep 'open files' /proc/48377/limits
Max open files            1024                 524288               files
# soft limit 1024, hard limit 524288. The worker is at 1019: nearly out.

$ sudo lsof -p 48377 | awk 'NR>1 {print $5}' | sort | uniq -c | sort -rn | head
    842 IPv4       # sockets: a client created per request, never closed
    121 REG        # regular files
     34 unix
$ sudo lsof -p 48377 -a -i TCP -n -P | awk '{print $9}' | sort | uniq -c | sort -rn | head
    831 127.0.0.1:52210->127.0.0.1:6379     # leaking Redis connections

When a process hits its limit, every open(), socket() and accept() fails with EMFILE. Django logs "Too many open files" from unexpected places, and database connections start failing even though PostgreSQL is fine. A descriptor count that climbs steadily over hours and drops only on a worker restart is a leak. The usual causes are file handles opened without a with block, or a Redis, SMTP or HTTP client created inside a view and never closed.

Running ulimit -n in your SSH session tells you nothing about gunicorn. Services started by systemd get their limits from the unit file, so always check /proc/PID/limits for the real process. You can raise the limit with LimitNOFILE=65536 in the [Service] section, but only after fixing the leak. A higher limit just delays the next outage and makes it harder to link to its cause.

Application or database?

If strace and py-spy show workers waiting on port 5432, stop looking at the app host and ask PostgreSQL what each backend is doing and what it's waiting for:

SELECT pid,
       now() - query_start AS runtime,
       state,
       wait_event_type,
       wait_event,
       left(query, 60) AS query
FROM pg_stat_activity
WHERE backend_type = 'client backend'
  AND state <> 'idle'
ORDER BY runtime DESC;

-- Lock chains: who is blocked, and by whom
SELECT pid,
       pg_blocking_pids(pid) AS blocked_by,
       now() - query_start AS waiting,
       left(query, 60) AS query
FROM pg_stat_activity
WHERE cardinality(pg_blocking_pids(pid)) > 0;

The wait_event_type column is how you decide:

• Lock: the backend is blocked by another transaction. Common causes are a migration taking an ACCESS EXCLUSIVE lock, select_for_update() held across a slow external call, or a transaction left open. Follow pg_blocking_pids to the head of the chain. The session at the head is often not slow in itself, it's just holding a lock.
• IO (for example DataFileRead): the backend is reading pages from disk. That's usually a sequential scan on a table larger than shared buffers and page cache. Run EXPLAIN (ANALYZE, BUFFERS) on the query and iostat on the database host.
• wait_event NULL with state = 'active': the backend is running on CPU, typically a bad plan, a huge sort or hash, or a function called per row. Check mpstat on the DB host.
• LWLock: contention on internal shared structures. It's rare at typical Django scale, and when it does happen it often means far more connections than cores.
• state = 'idle in transaction': the application opened a transaction and is now doing something else, such as calling an API or rendering a template inside transaction.atomic() or with ATOMIC_REQUESTS enabled. It's an application bug that shows up in the database, and it holds locks and blocks vacuum. idle_in_transaction_session_timeout puts a limit on it.

If pg_stat_activity shows few active backends and short runtimes while the app workers still look busy, the problem is on the app side. Look at slow outbound HTTP calls, CPU-bound Python, or a lock inside the process. Deciding which side owns the problem early is what stops teams from tuning PostgreSQL during an incident caused by a third-party API.

Worked scenario: every worker blocked on one slow query

At 14:00 on a weekday, nginx starts returning 504s and a few 502s. The gunicorn log fills with lines like [CRITICAL] WORKER TIMEOUT (pid:48377). The deploy an hour earlier added a date-range filter to the orders report. Here's how the checklist plays out.

1. uptime: load 0.31. There's no CPU saturation and no D-state pile-up on the app host. Whatever is wrong, this machine isn't doing much work.
2. dmesg -T | tail: nothing new. No OOM kills, no I/O errors.
3. vmstat 1: r 0–1, b 0, si/so 0, wa 0, id about 95. The host is idle while users get timeouts. That combination points to workers waiting on something outside the box.
4. ss -lnt 'sport = :8000': Recv-Q 2048, Send-Q 2048. The accept queue is full and nstat TcpExtListenOverflows rises every time you run it. All nine workers are busy and requests are queueing in the kernel until nginx's proxy_read_timeout fires.
5. ss -tan state established '( dport = :5432 )': nine connections, one per worker, all open.
6. strace -p on two workers: both sent a SELECT and are blocked in poll on the socket to 127.0.0.1:5432, with no timeout.
7. py-spy dump on every worker: all nine are in reports/views.py:88, iterating the orders queryset.
8. pg_stat_activity: nine active backends running the same query for 12–29 seconds, plus several more whose clients have already disconnected. They're all waiting on IO / DataFileRead. EXPLAIN shows a sequential scan over 40 million rows because the new created_at filter has no index.

There's a feedback loop here that makes this worse over time. Gunicorn kills each worker after its 30-second timeout, but PostgreSQL doesn't notice the client has gone until it next tries to send results, so the orphaned query keeps scanning. The replacement worker picks up the next queued request, which is often a retry of the same report, and starts another identical scan. Disk I/O on the database host climbs, every query gets slower, and more workers time out. Scaling the app tier would make it worse, because more workers means more concurrent sequential scans.

First, stabilise the system by cancelling the runaway queries. pg_cancel_backend cancels the current query and leaves the session open. pg_terminate_backend kills the whole connection, so save it for sessions that won't cancel.

SELECT pid, pg_cancel_backend(pid)
FROM pg_stat_activity
WHERE backend_type = 'client backend'
  AND state = 'active'
  AND query ILIKE 'SELECT %FROM "reports_order"%created_at%'
  AND now() - query_start > interval '10 seconds';

Then fix it at three levels. Make runaway queries fail fast, add the missing index without locking the table, and stop request workers running heavy reports at all.

# myproject/settings.py
DATABASES = {
    "default": {
        "ENGINE": "django.db.backends.postgresql",
        "NAME": "myproject",
        "USER": "deploy",
        "HOST": "127.0.0.1",
        "PORT": "5432",
        "CONN_MAX_AGE": 60,
        "CONN_HEALTH_CHECKS": True,
        "OPTIONS": {
            # Server-side limits for every web connection, in milliseconds.
            # Keep statement_timeout below gunicorn's --timeout (30 s).
            "options": "-c statement_timeout=10000 "
                       "-c lock_timeout=3000 "
                       "-c idle_in_transaction_session_timeout=30000",
        },
    }
}

# reports/migrations/0012_order_created_at_idx.py
from django.contrib.postgres.operations import AddIndexConcurrently
from django.db import migrations, models


class Migration(migrations.Migration):
    atomic = False  # CREATE INDEX CONCURRENTLY cannot run in a transaction

    dependencies = [("reports", "0011_order_status")]

    operations = [
        AddIndexConcurrently(
            "order",
            models.Index(fields=["created_at"], name="order_created_at_idx"),
        ),
    ]

With a 10-second statement timeout, a pathological query raises django.db.utils.OperationalError ("canceling statement due to statement timeout") well before gunicorn kills the worker. The worker survives, the database stops working on the query immediately, and your error tracker gets a clear traceback instead of an unexplained WORKER TIMEOUT. The trade-off is that legitimate long queries fail too. Run batch jobs and management commands under a separate database role with ALTER ROLE ... SET statement_timeout, or override the limit with SET LOCAL statement_timeout inside the transaction that needs it. lock_timeout covers the lock-chain version of this incident, where a migration or a long select_for_update would otherwise make every worker wait indefinitely. Longer term, move the report to a background task that emails a file, so the web workers only ever handle requests that finish in milliseconds.

Common mistakes

• Restarting before capturing evidence. A restart clears the symptoms for a few minutes and throws away the only data that explains them. Capturing vmstat 1 5, ss -lnt, one py-spy dump per worker and a pg_stat_activity snapshot takes under a minute. Script it in advance.
• Adding workers or raising the backlog to fix a database bottleneck. More workers mean more concurrent slow queries against the resource that's already saturated, and a longer backlog just means longer waits before the 504.
• Trusting %util on NVMe or cloud volumes. Decide using r_await/w_await and aqu-sz, and compare IOPS against what the volume is provisioned for.
• Reading the first line of vmstat or iostat. It's the average since boot and can hide an incident that started five minutes ago. Use the interval samples.
• Tuning TIME_WAIT sysctls when the real issue is opening a new outbound connection per request. Reuse connections and TIME_WAIT stops mattering.
• Ignoring steal time on a VPS. If st is consistently above a few percent, no code change will recover the CPU the hypervisor isn't giving you. Move to a larger or dedicated-CPU instance.

Verify it

Rehearse the whole procedure on staging before you need it in production. Check that the tools are installed, that you have permission to use them, and that your safety limits are actually applied:

# Tools present (sysstat, iproute2, strace, lsof, py-spy in the venv or system)
which vmstat mpstat pidstat iostat sar ss nstat strace lsof py-spy
iostat -xz 1 3

# Kernel caps that interact with gunicorn
sysctl net.core.somaxconn kernel.yama.ptrace_scope
ss -lnt 'sport = :8000'

# Effective limits of the real worker, not your shell
grep 'open files' /proc/$(pgrep -nf gunicorn)/limits

# Can you get a Python stack without stopping the worker?
sudo py-spy dump --pid $(pgrep -nf gunicorn)

# Is the statement timeout applied to Django's connections?
cd /srv/app && venv/bin/python manage.py shell -c \
  "from django.db import connection; c = connection.cursor(); c.execute('SHOW statement_timeout'); print(c.fetchone())"
# expected: ('10s',)

# Fire drill: a view that runs SELECT pg_sleep(20), hit it with more
# concurrent requests than you have workers, then watch ss -lnt, strace,
# py-spy and pg_stat_activity tell you exactly what is happening.

Performance incidents end quickly when you follow a fixed order rather than guessing. Check load and the kernel log, then vmstat, then utilisation, saturation and errors for each resource, including the worker pool and database connections. Once you know which queue is full, strace and py-spy show exactly what the stuck workers are waiting on. On most Django servers that turns out to be the database, and pg_stat_activity tells you why.