systemd for Django in Depth: Hardened Services, Timers Instead of Cron, Resource Limits and Restart Policies

Most Django deployments run gunicorn under a five-line systemd unit copied from a blog post years ago: it runs with far too much access to the host, restarts in a tight loop when the database is down, has no memory ceiling, and sits beside a crontab that fails without anyone noticing. systemd already ships sandboxing, resource control, scheduling and failure hooks, and none of it needs another daemon. This guide builds a production unit set for myproject on Ubuntu 24.04 (systemd 255) and explains what each directive buys you and what it costs.

User, directories and secrets

The service runs as a dedicated system account with no shell and no home directory worth protecting. The code belongs to root and the service account can only read it, so a compromised worker cannot change the application on disk.

sudo useradd --system --user-group --no-create-home --shell /usr/sbin/nologin deploy
sudo chown -R root:deploy /srv/app/src /srv/app/venv
sudo chmod -R g+rX,o-rwx /srv/app/src /srv/app/venv
sudo install -d -o deploy -g deploy -m 0750 /srv/app/media

sudo install -d -m 0755 /etc/myproject
sudo install -o root -g root -m 0600 /dev/null /etc/myproject/env
sudo usermod -aG deploy www-data   # nginx must reach the socket

/etc/myproject/env holds DJANGO_SECRET_KEY, DATABASE_URL and similar. Keep it root:root 0600: EnvironmentFile= is read by PID 1 as root before privileges are dropped, so the service account never has to read the file itself. The values still end up in the process environment, which the same user can read through /proc/<pid>/environ. If you need stricter handling, look at LoadCredential=, which exposes secrets as files under $CREDENTIALS_DIRECTORY.

# /etc/myproject/env: not a shell script, so no "export" and no command substitution
DJANGO_SETTINGS_MODULE=myproject.settings.production
DJANGO_SECRET_KEY=change-me
DATABASE_URL=postgres://myproject:secret@127.0.0.1:5432/myproject
ALLOWED_HOSTS=example.com,app.example.com

The complete gunicorn unit

# /etc/systemd/system/myproject-gunicorn.service
[Unit]
Description=myproject gunicorn (Django)
Wants=network-online.target
After=network-online.target postgresql.service
StartLimitIntervalSec=300
StartLimitBurst=5
OnFailure=notify-failure@%n.service

[Service]
Type=notify
NotifyAccess=main
User=deploy
Group=deploy
WorkingDirectory=/srv/app/src
EnvironmentFile=/etc/myproject/env
Environment=PYTHONUNBUFFERED=1
RuntimeDirectory=myproject
RuntimeDirectoryMode=0750
UMask=0027
ExecStart=/srv/app/venv/bin/gunicorn myproject.wsgi:application \
    --bind unix:/run/myproject/gunicorn.sock \
    --workers 4 --timeout 30 --graceful-timeout 25 \
    --max-requests 1000 --max-requests-jitter 100 \
    --access-logfile - --error-logfile -
ExecReload=/bin/kill -s HUP $MAINPID
KillMode=mixed
TimeoutStopSec=35
Restart=on-failure
RestartSec=5s

# Sandboxing
NoNewPrivileges=true
PrivateTmp=true
PrivateDevices=true
ProtectSystem=strict
ReadWritePaths=/srv/app/media
ProtectHome=true
ProtectKernelTunables=true
ProtectKernelModules=true
ProtectKernelLogs=true
ProtectControlGroups=true
ProtectClock=true
ProtectHostname=true
RestrictAddressFamilies=AF_UNIX AF_INET AF_INET6
RestrictNamespaces=true
RestrictRealtime=true
RestrictSUIDSGID=true
CapabilityBoundingSet=
AmbientCapabilities=
LockPersonality=true
MemoryDenyWriteExecute=true
RemoveIPC=true
SystemCallArchitectures=native
SystemCallFilter=@system-service
SystemCallErrorNumber=EPERM

# Resource control
MemoryHigh=1536M
MemoryMax=2G
OOMPolicy=continue
CPUQuota=200%
TasksMax=256

[Install]
WantedBy=multi-user.target

Type=notify works because gunicorn sends READY=1 once the master is up when NOTIFY_SOCKET is set, so dependants and systemctl start wait for a real listener instead of a forked process. KillMode=mixed sends SIGTERM only to the master, which shuts workers down gracefully, then SIGKILLs anything left after TimeoutStopSec. RuntimeDirectory=myproject creates /run/myproject owned by deploy on start and deletes it on stop. With mode 0750, only deploy and members of its group (nginx, via www-data) can reach the socket.

What the hardening directives actually do

The less obvious ones

SystemCallFilter=@system-service is an allow-list of the system calls ordinary services use. Calls outside it, such as mounting filesystems, loading kernel modules, rebooting or tracing other processes, fail with EPERM because of SystemCallErrorNumber=EPERM. Without that setting, the kernel kills the process instead. Returning an error is kinder: a library that probes for an optional feature gets a normal failure it can handle, instead of taking down a worker. SystemCallArchitectures=native blocks the 32-bit compatibility syscall ABI, an old route around seccomp filters. RestrictNamespaces=true prevents creating user, network or mount namespaces, which a lot of container escapes rely on. RestrictSUIDSGID=true stops the service from creating setuid files. RemoveIPC=true removes System V and POSIX IPC objects the user left behind when the service stops. UMask=0027 means files the app creates, such as uploaded media, are not world-readable by default.

Debugging a sandbox failure

When something breaks after hardening, do not remove the whole sandbox. Reproduce the failing command with systemd-run, adding the same properties one at a time, until you find the directive responsible:

sudo systemd-run --pty --wait --collect \
    -p User=deploy -p WorkingDirectory=/srv/app/src \
    -p EnvironmentFile=/etc/myproject/env \
    -p ProtectSystem=strict -p ReadWritePaths=/srv/app/media \
    -p MemoryDenyWriteExecute=true \
    /srv/app/venv/bin/python manage.py shell -c "import myproject.reports; print('ok')"

Syscalls refused with EPERM are not logged by default, so a seccomp problem shows up as an odd PermissionError. To confirm one, remove SystemCallErrorNumber temporarily. The process is then killed with SIGSYS, systemctl status reports status=31/SYS, and the audit log names the syscall. The common Django causes are writes to a path missing from ReadWritePaths (EROFS), a font or cache directory under /home that ProtectHome now hides, and a library that uses AF_NETLINK to list network interfaces.

MemoryDenyWriteExecute and Python

MemoryDenyWriteExecute=true forbids memory mappings that are both writable and executable. The CPython interpreter and ordinary C extensions such as psycopg, Pillow and lxml are fine. Anything that generates machine code at runtime is not: JIT compilers (numba/llvmlite), PyPy, CPython's experimental JIT build, and ctypes or cffi callbacks on libffi builds that allocate closures in writable-executable memory. The failure is usually a crash or MemoryError on a code path that only runs occasionally. Turn it on, run your whole test suite and a PDF or report export under the real unit, and remove it if anything fails. The rest of the sandbox still gives you most of the benefit.

Measuring exposure with systemd-analyze

sudo systemd-analyze verify /etc/systemd/system/myproject-gunicorn.service
systemd-analyze security myproject-gunicorn.service

The report lists every sandboxing check and ends with an overall exposure score from 0.0 to 10.0, where lower is better. A plain unit scores in the 9s and is labelled UNSAFE. The unit above should land in the low range. Do not chase zero: the remaining items, such as PrivateNetwork= or IPAddressDeny=, either break a service that talks to PostgreSQL and Redis or need per-host tuning. Use the score to find directives you missed, and do not treat it as a compliance target.

Drop-in overrides with systemctl edit

Keep the main unit identical across hosts and put host-specific changes in drop-ins:

sudo systemctl edit myproject-gunicorn.service
# opens /etc/systemd/system/myproject-gunicorn.service.d/override.conf

[Service]
MemoryMax=4G
MemoryHigh=3G
ReadWritePaths=/srv/app/exports
# list-type settings append; an empty assignment resets them first
ExecStart=
ExecStart=/srv/app/venv/bin/gunicorn myproject.wsgi:application --bind unix:/run/myproject/gunicorn.sock --workers 8

systemctl cat myproject-gunicorn.service      # merged view, with file names
systemd-delta --type=extended                 # every overridden or extended unit

systemctl edit runs daemon-reload for you when you save. If you write drop-in files by hand or through configuration management, reload yourself. Resetting ExecStart= with an empty assignment is required. Without it, systemd refuses the unit because a non-oneshot service may only have one ExecStart.

Restart policy and start limits

Restart=on-failure restarts after a non-zero exit, a fatal signal, a timeout or a watchdog failure. It does not restart after a clean systemctl stop. Restart=always would also restart after a clean exit, which hides bugs where gunicorn exits 0 because of a bad config. The start limit decides when systemd gives up. Here it allows 5 starts in 300 seconds, then leaves the unit failed with result start-limit-hit, and that failure triggers OnFailure=.

Check the arithmetic: RestartSec × StartLimitBurst must be less than StartLimitIntervalSec, or the limit can never be reached and the unit retries forever. Five attempts 5 seconds apart fit inside 300 seconds. On systemd 254 and later you can also add RestartSteps= and RestartMaxDelaySec= so the delay grows between attempts during a long database outage. Once you have fixed the cause:

sudo systemctl reset-failed myproject-gunicorn.service
sudo systemctl start myproject-gunicorn.service

Reload versus restart on deploy

systemctl reload runs ExecReload and sends SIGHUP to the gunicorn master. The master re-reads its config and replaces workers gradually, so the socket stays open and requests are not dropped. That covers most code deploys, as long as you are not using --preload. With --preload, the app is imported in the master, and HUP does not load new code. systemctl restart closes the socket for a moment and nginx returns 502s for that time. Use restart when the unit file, the virtualenv's interpreter or the environment file changes: EnvironmentFile= is only read when systemd starts the service, so a reload never sees new secrets. Neither reload nor restart counts toward the start limit the way a crash loop does, but a deploy script that restarts several times in a row can still hit it, so batch your changes.

Consider where to stop, too. Restart=on-failure is a safety net for crashes. It does not fix a bad deploy. If the new code raises at import time, systemd retries five times, gives up and alerts you. That is the right result, and it is why deploy scripts should check systemctl is-active and hit a health URL afterwards instead of assuming the restart worked.

Resource control and what OOM looks like

Every service runs in its own cgroup, and these limits apply to the master and all workers together:

• MemoryHigh= is a soft ceiling. Above it, the kernel reclaims memory aggressively and throttles the cgroup. Latency rises, but nothing is killed yet, and that slowdown is your early warning.
• MemoryMax= is the hard limit. When reclaim cannot keep usage below it, the kernel OOM killer runs inside this cgroup only and usually picks the largest worker. The rest of the host is unaffected.
• OOMPolicy=continue matters here. The default, stop, stops the whole service when any of its processes is OOM-killed. With continue, the gunicorn master sees a worker die with SIGKILL and forks a replacement, and the other requests in flight keep going.
• CPUQuota=200% allows at most two CPUs' worth of time, so a runaway report cannot starve PostgreSQL on the same host.
• TasksMax=256 caps processes and threads, which stops thread or fork explosions. Set it well above workers × threads.

An OOM kill shows up in the unit's journal as a line noting that a process was killed by the OOM killer, alongside gunicorn's own "Worker ... was sent SIGKILL" message. Watch live usage with systemd-cgtop. --max-requests handles slow leaks, and MemoryMax handles one request that allocates too much. Use both. Do not set MemoryMax tight enough that normal peaks reach it. Leave it unset until you have measured real usage, because a limit that is too low turns into random 502s.

Worked scenario: the CSV export that ate the box

A staff-only view builds a CSV of every order in the year by calling list(Order.objects.select_related(...)). With four workers at roughly 180 MB each, the service normally uses about 800 MB. One export pushes a worker past 1.4 GB. Here is what happens with the limits above.

• At 1.5 GB for the cgroup, MemoryHigh starts reclaiming. Page cache is dropped first, then the kernel throttles allocations. Every worker in the service slows down, not only the one doing the export, so p95 latency rises across the site for a minute or two. systemd-cgtop shows the unit's memory sitting at the high mark.
• The export keeps allocating, and the cgroup reaches 2 GB. The kernel OOM killer picks the largest process, which is the export worker. It gets SIGKILL, and the request returns a 502 through nginx.
• Because of OOMPolicy=continue, systemd leaves the service running. The gunicorn master logs that the worker was killed and starts a fresh one. The other three workers keep serving throughout, and PostgreSQL on the same host is not affected.

Without any limits, the kernel's global OOM killer would have picked a victim from the whole host, often PostgreSQL, because its shared buffers make it the biggest process. The real fix is in the code: stream the export with StreamingHttpResponse and .iterator(chunk_size=2000), or move it to a background job. What the limits give you is containment and a clear signal in the journal while that fix is written. You can confirm the kill counter afterwards:

systemctl show myproject-gunicorn -p MemoryCurrent -p Result
cat /sys/fs/cgroup/system.slice/myproject-gunicorn.service/memory.events
journalctl -u myproject-gunicorn --since "30 min ago" | grep -iE 'oom|sigkill'

The oom_kill counter in memory.events counts kills since the service started, and the high counter shows how often throttling kicked in. A rising high count with no kills means MemoryHigh is set about right and is catching peaks. If high climbs steadily even under normal traffic, the limit is too low for your worker count.

When to skip memory limits entirely: single-purpose VMs where gunicorn is the only significant process and you would rather use every megabyte. Even then, TasksMax is worth keeping.

Timers instead of cron

Compared with cron, timers log into the journal, use the same sandbox and environment file as the web service, do not start a second run while the previous one is still going, can catch up on runs missed while the host was down, and can report failures. Each job needs a oneshot service and a timer:

# /etc/systemd/system/myproject-clearsessions.service
[Unit]
Description=myproject: purge expired sessions
After=network-online.target postgresql.service
OnFailure=notify-failure@%n.service

[Service]
Type=oneshot
User=deploy
Group=deploy
WorkingDirectory=/srv/app/src
EnvironmentFile=/etc/myproject/env
ExecStart=/srv/app/venv/bin/python manage.py clearsessions
TimeoutStartSec=30min
Nice=10
IOSchedulingClass=idle
NoNewPrivileges=true
PrivateTmp=true
PrivateDevices=true
ProtectSystem=strict
ProtectHome=true
ProtectKernelTunables=true
ProtectKernelModules=true
ProtectControlGroups=true
RestrictAddressFamilies=AF_UNIX AF_INET AF_INET6
CapabilityBoundingSet=
LockPersonality=true
SystemCallArchitectures=native
SystemCallFilter=@system-service
MemoryMax=1G

# /etc/systemd/system/myproject-clearsessions.timer
[Unit]
Description=Nightly session cleanup for myproject

[Timer]
OnCalendar=*-*-* 03:15:00
Persistent=true
RandomizedDelaySec=15min

[Install]
WantedBy=timers.target

The timer starts myproject-clearsessions.service because the names match. Set Unit= only if they differ. Persistent=true saves the last trigger time on disk and runs the job once at boot if a scheduled run was missed. RandomizedDelaySec spreads start times across a fleet so every host does not hit the database at 03:15:00. Oneshot services have no start timeout by default, so set TimeoutStartSec explicitly or a hung command can block every later run.

systemd-analyze calendar '*-*-* 03:15:00'
systemd-analyze calendar --iterations=5 'Mon..Fri *-*-* 08..18:00/30'
sudo systemctl daemon-reload
sudo systemctl enable --now myproject-clearsessions.timer
systemctl list-timers --all 'myproject-*'
sudo systemctl start myproject-clearsessions.service   # run it now, same environment

Calendar syntax, accuracy and overlap

OnCalendar= uses the form DayOfWeek Year-Month-Day Hour:Minute:Second, and any part can be *, a list, a range with .. or a repetition with /. hourly, daily and weekly are shorthands that fire at the start of the period, so every host and every job fires at midnight together, which is a good reason to write explicit times. The calendar uses the system time zone unless you append a zone name, for example *-*-* 03:15:00 Europe/London. Always check a new expression with systemd-analyze calendar before deploying it. It prints the normalised form and the next time it fires, so it catches typos that would otherwise mean the job silently never runs.

By default, timers only guarantee one-minute accuracy (AccuracySec=1min) so the system can batch wake-ups. That is fine for maintenance. Set AccuracySec=1s if a job really does need to start on the second. For "every 10 minutes after the last run finished" semantics, use monotonic timers instead: OnBootSec=5min plus OnUnitInactiveSec=10min measures from when the service last went inactive. A slow run then pushes the next one back instead of queuing behind it, which is exactly what a sync job that sometimes takes 12 minutes needs.

Overlap protection comes from the unit model. While a oneshot service is still activating, another start request joins the job that is already running instead of starting a second copy. Cron would start a second copy of the same command, and two overlapping send_reminders runs send duplicate emails. You do not need flock wrappers.

When to keep cron: jobs that must run as an ordinary user on a shared host you do not administer, or when you need second-level precision for dozens of jobs that change constantly. If you have tasks that run every few seconds, use a proper task queue rather than either.

Failure notification with a templated unit

OnFailure=notify-failure@%n.service starts one instance of a template when the unit enters the failed state, passing the failing unit's full name as the instance. For oneshot jobs, that means every failed run. For the web service, it means the start limit was hit.

# /etc/systemd/system/notify-failure@.service
[Unit]
Description=Failure notification for %i

[Service]
Type=oneshot
DynamicUser=yes
SupplementaryGroups=systemd-journal
ExecStart=/usr/local/bin/notify-failure %i
PrivateTmp=true
ProtectSystem=strict
ProtectHome=true
NoNewPrivileges=true

#!/bin/sh
# /usr/local/bin/notify-failure  (root:root 0755)
set -eu
unit="$1"
body=$(printf 'host: %s\nunit: %s\n\n%s\n\n%s\n' \
    "$(hostname)" "$unit" \
    "$(systemctl status --no-pager --full "$unit" 2>&1 || true)" \
    "$(journalctl -u "$unit" -n 50 --no-pager 2>&1 || true)")
curl -fsS --max-time 10 -X POST -H 'Content-Type: text/plain' \
    --data-binary "$body" https://alerts.example.com/hooks/systemd

DynamicUser=yes runs the notifier as a throwaway UID, and systemd-journal group membership lets it read logs. Test it deliberately with sudo systemctl start notify-failure@myproject-clearsessions.service.service. The doubled suffix is correct, because the instance string is the full unit name.

Reading logs with journalctl

journalctl -u myproject-gunicorn -f
journalctl -u myproject-gunicorn --since "1 hour ago"
journalctl -u myproject-gunicorn --since "2026-01-10 02:00" --until "2026-01-10 03:00"
journalctl -u myproject-gunicorn -b -p err
journalctl -u myproject-clearsessions -n 20 --no-pager
journalctl -u myproject-gunicorn -o json --since today | jq -r '.MESSAGE'
journalctl -u myproject-gunicorn -o json-pretty -n 1

-p err filters on the journal's priority field. Anything written to stdout or stderr is recorded at info by default, so Django's tracebacks never match it. Because SyslogLevelPrefix= is on by default, a line beginning with <3> is stored at priority 3 (err). A small formatter makes priorities meaningful:

# myproject/logging.py
import logging

PRIORITY = {logging.CRITICAL: 2, logging.ERROR: 3, logging.WARNING: 4,
            logging.INFO: 6, logging.DEBUG: 7}

class JournalPrefixFormatter(logging.Formatter):
    def format(self, record):
        return f"<{PRIORITY.get(record.levelno, 6)}>{super().format(record)}"

# settings.py
LOGGING = {
    "version": 1,
    "disable_existing_loggers": False,
    "formatters": {"journal": {"()": "myproject.logging.JournalPrefixFormatter",
                               "format": "%(name)s: %(message)s"}},
    "handlers": {"console": {"class": "logging.StreamHandler", "formatter": "journal"}},
    "root": {"handlers": ["console"], "level": "INFO"},
}

Only the first line of a multi-line record receives the prefix. Traceback lines after it are stored as separate info entries, so use -p err to find incidents and then read the surrounding context without the filter.

Retention and rate limiting

Check that /var/log/journal exists. If it does not, the journal is only kept in memory under /run and is lost on reboot, exactly when you need it most. Cap its size in a drop-in so a noisy access log cannot fill the disk:

# /etc/systemd/journald.conf.d/50-retention.conf
[Journal]
Storage=persistent
SystemMaxUse=2G
MaxRetentionSec=30day

sudo systemctl restart systemd-journald
journalctl --disk-usage

journald also rate-limits each service. A burst of thousands of lines per second, such as a traceback in a tight loop, is dropped with a "suppressed N messages" notice. You can raise the limit per unit with LogRateLimitIntervalSec= and LogRateLimitBurst=, but the suppression usually points to a bug worth fixing rather than a limit worth raising. If your access log volume is high, consider leaving gunicorn's access log off and relying on nginx's, so application errors are not lost among request lines.

Common mistakes

• Setting ProtectSystem=strict and forgetting ReadWritePaths= for media. Uploads then fail with OSError: [Errno 30] Read-only file system, often only on the first upload after a deploy.
• Leaving OOMPolicy at its default of stop with MemoryMax set. One oversized request then takes down every worker instead of one.
• Choosing RestartSec, StartLimitBurst and StartLimitIntervalSec values that can never reach the limit. The unit retries forever and OnFailure= never fires.
• Editing unit files by hand and forgetting systemctl daemon-reload. systemd keeps running the old definition, and systemctl status only shows a small warning that the unit file changed on disk.
• Running migrations from ExecStartPre=. Every crash restart then re-runs them under the web sandbox and timeout. Run them once in the deploy script.
• Making the environment file readable by the service user, or keeping it inside /srv/app/src, where it gets committed or copied into backups along with the code.

Verify it

sudo systemd-analyze verify /etc/systemd/system/myproject-*.service /etc/systemd/system/myproject-*.timer
systemd-analyze security myproject-gunicorn.service | tail -n 1
systemctl show myproject-gunicorn -p User -p MemoryMax -p OOMPolicy -p TasksMax -p NRestarts
sudo -u deploy test -r /etc/myproject/env && echo "BAD: env readable" || echo "ok"
sudo systemd-run --wait -p User=deploy -p ProtectSystem=strict -p ReadWritePaths=/srv/app/media touch /srv/app/media/.probe
sudo systemctl kill -s KILL --kill-whom=main myproject-gunicorn   # expect restart
systemctl status myproject-gunicorn --no-pager
systemctl list-timers 'myproject-*'
journalctl -u myproject-gunicorn -p warning --since "10 min ago"

The probe write should succeed. After killing the master, NRestarts should go up and the site should return within a few seconds. Kill it more than the burst allows, and the unit should reach failed and your alert endpoint should receive a message. Once you have seen each of those, systemd is enforcing your security, resource, scheduling and alerting settings on every restart.