Stop logs filling your disk: cap journald and logrotate, emit JSON from Django with a request ID shared with nginx, ship everything to Loki with Alloy, and query, retain and alert on it without leaking personal data.
Most Django servers go down because of logging long before traffic becomes a problem. A log file nobody rotates, or a journal with no size cap, fills the root filesystem. PostgreSQL then can't write WAL, Redis can't persist, and the site is down at 03:00. The fix is unglamorous but it works: put a limit on every log on the box, write structured JSON with a request ID, ship it to Loki, and query and alert on it from Grafana. After that, debugging production is one query and not an afternoon of grep over SSH.
A typical Ubuntu 24.04 host running Django behind nginx and gunicorn writes logs to three places. Each has different owners, rotation rules and failure modes:
manage.py commands run from systemd timers. Read it with journalctl -u myproject.service. journald rotates and caps its own files, although the defaults may not suit you./var/log/nginx/: access and error logs, written by nginx and rotated daily by the logrotate config the distro package ships./srv/app/logs/django.log, /srv/app/logs/celery.log and the like, created by a FileHandler someone added to LOGGING years ago. Nothing rotates these unless you set it up.Almost every disk-full incident caused by logs starts with the third one. A FileHandler left at DEBUG in production, django.db.backends logging every SQL statement, or a retry loop logging a full traceback several times a second will write gigabytes a day. Nothing complains until the disk is full, and then everything fails at once, often in ways that don't point to logging. Django can't write file-based sessions or uploaded files, PostgreSQL refuses writes, and even sudo and shell history start failing.
The pager reports 500s on app.example.com. You SSH in and find a full disk. The quickest route to the cause:
df -h /
sudo du -xh --max-depth=2 /srv /var/log | sort -h | tail -n 15
journalctl --disk-usage
# Files that are deleted but still held open by a process:
sudo lsof +L1
The lsof +L1 step catches a lot of people. If someone has already run rm /srv/app/logs/django.log in a panic, df still shows the disk as full. The gunicorn workers still hold the file descriptor, so the blocks aren't freed until those processes close it. Truncate the file, don't delete it (: > /srv/app/logs/django.log), or restart the service that holds it. The rest of this tutorial is about never getting into that position again. The rule is simple: every byte of logging on the box must have a limit, and the durable copy of your logs should live somewhere else.
By default journald caps persistent storage at 10% of the filesystem, up to 4G, and keeps entries until it hits that cap. On a 20G VM that's 2G of journal with no time-based retention at all. That's not ideal for a small disk, and not what a data-protection policy wants either. Set explicit limits in a drop-in file instead of editing the packaged /etc/systemd/journald.conf, so upgrades don't overwrite your settings:
# /etc/systemd/journald.conf.d/limits.conf
[Journal]
Storage=persistent
SystemMaxUse=1G
SystemKeepFree=2G
SystemMaxFileSize=100M
MaxRetentionSec=14day
RateLimitIntervalSec=30s
RateLimitBurst=10000
sudo systemctl restart systemd-journald
journalctl --disk-usage
# Emergency cleanup when you are already in trouble:
sudo journalctl --vacuum-size=500M
sudo journalctl --vacuum-time=7d
What each setting does and why it's there:
SystemMaxUse is a hard ceiling on journal files. SystemKeepFree makes journald give way when free space drops below the value, whatever its own cap says. Setting both covers the case where something else is filling the disk.SystemMaxFileSize sets the size of individual journal files. Journald deletes whole archived files, never partial ones, so smaller files make retention more granular.MaxRetentionSec is a time limit. Once logs are shipped to Loki, the local journal only has to cover a Loki outage or a network partition, so a few days to two weeks is enough. It is also part of your GDPR retention story.RateLimitIntervalSec and RateLimitBurst are per-service limits. When a service goes over them, journald drops its messages and logs a note saying how many it suppressed. A service that logs a stack trace per request can burst past the limit during an incident, which is exactly when you want every line; the example above states the values explicitly so you can tune them per host. Raise them, but don't disable them. They also protect you from a crash-looping service.Useful journalctl commands once your apps log there: journalctl -u myproject.service --since "10 min ago", journalctl -u myproject.service -p err for priority error and above, and journalctl -u myproject.service -o cat for the raw message without the syslog prefix. The last one matters when the message is JSON you want to pipe into jq.
For your own processes, log to stdout and stderr and let systemd collect it. You get rotation, size and time limits, timestamps, unit metadata and journalctl -f with no config. There are no file permissions to get wrong, and gunicorn workers can't disagree about rotation. It's also the model containers use, so the same LOGGING dict works if you move to one later. A unit file for gunicorn:
# /etc/systemd/system/myproject.service
[Unit]
Description=myproject gunicorn
After=network.target
[Service]
User=deploy
Group=deploy
WorkingDirectory=/srv/app
Environment=DJANGO_SETTINGS_MODULE=myproject.settings.production
ExecStart=/srv/app/.venv/bin/gunicorn myproject.wsgi:application \
--config /srv/app/gunicorn.conf.py
ExecReload=/bin/kill -s HUP $MAINPID
SyslogIdentifier=myproject
Restart=on-failure
[Install]
WantedBy=multi-user.target
Leave gunicorn's access log off. nginx already records every request with better timing data, and two access logs just double what you ingest. Keep writing to files only when a process genuinely can't write to stdout, or when an audit process needs a separate artefact with its own retention. In that case, use logrotate as described next.
If you must keep file logs, rotate them externally. On Ubuntu 24.04 logrotate runs once a day from logrotate.timer and reads every file in /etc/logrotate.d/:
# /etc/logrotate.d/myproject
/srv/app/logs/*.log {
daily
rotate 14
maxsize 200M
missingok
notifempty
compress
delaycompress
dateext
su deploy deploy
create 0640 deploy deploy
sharedscripts
postrotate
systemctl kill -s USR1 --kill-whom=main myproject.service >/dev/null 2>&1 || true
endscript
}
daily with rotate 14 keeps two weeks. Match this to your retention policy, not to whatever feels safe.maxsize 200M rotates early when a file passes 200M, even before the daily interval is up. It's only checked when logrotate runs, though. A runaway logger can still write 30G between two daily runs. That's another reason to cap things at the source and prefer journald, which enforces its limit on every write.compress gzips old files. delaycompress leaves the most recent rotated file uncompressed for one more cycle, so a process that still has it open can finish writing, and you can grep yesterday's log without zgrep.su deploy deploy is required when a non-root user can write to the log directory. Without it, logrotate refuses to act on the directory because of the security risk.sharedscripts runs postrotate once for the whole glob, not once per file.After renaming a log file, the writer still has the old inode open and keeps appending to django.log.1. There are two ways to deal with that.
Signalling: logrotate renames the file, creates a fresh one, and postrotate tells the process to reopen its logs. The gunicorn master reopens its log files on USR1, and nginx does the same on USR1, which is how the packaged nginx logrotate config works. This is atomic and loses nothing, but the process has to support reopening, and your Python handlers won't reopen on gunicorn's signal. That's why handler choice matters.
copytruncate: logrotate copies the file and truncates the original in place, so the writer never has to know. It works with anything, but it has two costs. Lines written between the copy and the truncate are lost. And if the writer didn't open the file with O_APPEND, it keeps writing at its old offset, which leaves a sparse file that starts with gigabytes of NUL bytes. Python's FileHandler opens in append mode by default, so it's safe there, but copytruncate is still the lossy choice. Keep it for third-party software you can't signal.
Python's RotatingFileHandler and TimedRotatingFileHandler rotate from inside the process. With --workers 4 you have four independent handler instances, one per forked worker, each tracking its own view of the file. Worker 1 hits maxBytes and renames django.log to django.log.1. Workers 2 to 4 carry on writing to what is now django.log.1. Then worker 3 decides to rotate, and renames django.log.1 to django.log.2 and its own file on top of it. You get lost lines, files in the wrong order and workers writing to deleted inodes, which is how you end up running lsof +L1 at 03:00. The Python logging cookbook itself warns that logging to a single file from multiple processes isn't supported.
If you need a file, use WatchedFileHandler. On every emit it checks whether the file's device or inode has changed and reopens it if so. That makes it work with logrotate's rename-and-create and needs no signal:
"handlers": {
"file": {
"class": "logging.handlers.WatchedFileHandler",
"filename": "/srv/app/logs/django.log",
"formatter": "json",
},
},
Even so, stdout to journald remains the better default. Everything below assumes it.
Text logs are fine for a human reading tail. They're poor for a machine that has to answer "how many payment failures for merchant 42 in the last hour?". Parsing "Payment failed for order 1234 (merchant 42): card_declined" with regexes breaks the first time someone changes the wording. Put one JSON object on each line and keep the variable parts in named fields. The message becomes a stable event name, and the data lives next to it.
A small formatter is all you need, and owning it means you control the field names:
# myproject/logging.py
import json
import logging
from datetime import datetime, timezone
# Attributes every LogRecord has; anything else came from extra= or a filter.
_RESERVED = set(vars(logging.makeLogRecord({}))) | {"message", "asctime"}
class JsonFormatter(logging.Formatter):
def format(self, record: logging.LogRecord) -> str:
payload = {
"ts": datetime.fromtimestamp(record.created, tz=timezone.utc).isoformat(),
"level": record.levelname,
"logger": record.name,
"msg": record.getMessage(),
}
for key, value in record.__dict__.items():
if key not in _RESERVED and not key.startswith("_"):
payload[key] = value
if record.exc_info:
payload["exc"] = self.formatException(record.exc_info)
if record.stack_info:
payload["stack"] = self.formatStack(record.stack_info)
return json.dumps(payload, default=str, ensure_ascii=False)
Two details matter. First, default=str. Without it, a single UUID, Decimal or datetime passed in extra makes json.dumps raise. The logging module catches that and prints "--- Logging error ---" plus a traceback to stderr, and you lose the line you most needed. Second, tracebacks go into one exc field inside the JSON object. A multi-line traceback written as plain text would reach journald and Loki as dozens of separate lines, and they'd interleave with other workers' output.
Calling code then looks like this:
import logging
logger = logging.getLogger(__name__)
def capture_payment(order):
try:
charge = gateway.capture(order.payment_intent_id, amount=order.total)
except gateway.CardDeclined as exc:
logger.warning(
"payment.declined",
extra={"order_id": order.pk, "merchant_id": order.merchant_id,
"decline_code": exc.code},
)
raise
logger.info("payment.captured",
extra={"order_id": order.pk, "amount": order.total})
return charge
The python-json-logger package gives you the same result with more options, such as renaming fields and choosing a timestamp format. It's a reasonable choice if you don't want to maintain the formatter above. One trade-off to accept either way: raw JSON is harder to read in a terminal. Keep a plain text formatter in your development settings, and on servers read logs with journalctl -u myproject.service -o cat | jq.
gunicorn's own error log (worker boots, timeouts, SIGKILLs) doesn't go through Django's LOGGING. gunicorn accepts a dictConfig of its own through the logconfig_dict setting in gunicorn.conf.py. Point its gunicorn.error logger at the same JSON formatter so worker timeouts become queryable events rather than free text.
A request ID connects the nginx access line, every Django log line and the exception traceback for one request. Without it, "the user says checkout failed around 14:05" means lining up timestamps across interleaved output from four workers. Store the ID in a contextvars.ContextVar. Context variables are isolated per thread and per asyncio task, so they're correct under gunicorn sync workers, threaded workers and ASGI async views alike. A threading.local is shared between coroutines on the same event loop and gives you the wrong ID.
# myproject/request_id.py
import logging
import re
import uuid
from contextvars import ContextVar
request_id_var: ContextVar[str] = ContextVar("request_id", default="-")
_VALID = re.compile(r"^[A-Za-z0-9-]{8,64}$")
class RequestIdMiddleware:
def __init__(self, get_response):
self.get_response = get_response
def __call__(self, request):
incoming = request.headers.get("X-Request-ID", "")
rid = incoming if _VALID.match(incoming) else uuid.uuid4().hex
token = request_id_var.set(rid)
request.request_id = rid
try:
response = self.get_response(request)
response["X-Request-ID"] = rid
return response
finally:
request_id_var.reset(token)
class RequestIdFilter(logging.Filter):
def filter(self, record: logging.LogRecord) -> bool:
record.request_id = request_id_var.get()
return True
A few decisions are built in here. The incoming header is validated because clients control it. Without validation, someone can send a 10KB header, or one containing newlines, and it gets copied into every log line for that request. The reset(token) in finally keeps the ID from leaking into whatever the worker does next. The ID goes back in the response header, so a user or a support agent can quote it from the browser's network tab. The filter always returns True, because its job is to enrich records, not drop them.
Put the middleware first in MIDDLEWARE so that log lines from security, session and authentication middleware carry the ID too. Then wire up LOGGING:
# myproject/settings/production.py
MIDDLEWARE = [
"myproject.request_id.RequestIdMiddleware",
"django.middleware.security.SecurityMiddleware",
"django.contrib.sessions.middleware.SessionMiddleware",
# ...
]
LOGGING = {
"version": 1,
"disable_existing_loggers": False,
"filters": {
"request_id": {"()": "myproject.request_id.RequestIdFilter"},
"redact": {"()": "myproject.logging.RedactFilter"},
},
"formatters": {
"json": {"()": "myproject.logging.JsonFormatter"},
},
"handlers": {
"stdout": {
"class": "logging.StreamHandler",
"stream": "ext://sys.stdout",
"formatter": "json",
"filters": ["request_id", "redact"],
},
},
"root": {"handlers": ["stdout"], "level": "INFO"},
"loggers": {
"django": {"handlers": ["stdout"], "level": "INFO", "propagate": False},
"django.db.backends": {"level": "WARNING"},
},
}
The filters are attached to the handler, not to a logger. Filters on a logger only run for records created by that exact logger, not for records that propagate up from child loggers. On the handler they see every record that gets written. disable_existing_loggers: False keeps loggers that were created before settings loaded working. django.request already logs 5xx responses at ERROR with the traceback attached, so unhandled exceptions arrive as JSON with exc and request_id filled in and no extra code.
Celery tasks never pass through the middleware, so they'd log request_id: "-". The simplest useful fix is to use the task ID as the context ID, and to pass the originating request ID as an explicit argument when you need to link a task back to the web request:
# myproject/celery_signals.py
from celery.signals import task_prerun
from myproject.request_id import request_id_var
@task_prerun.connect
def bind_task_id(task_id=None, task=None, **kwargs):
request_id_var.set(task_id or "-")
Import this module from your Celery app module so the handler gets registered. In the default prefork pool each child process runs one task at a time, and every task overwrites the value, so it doesn't need a reset.
nginx generates a random 32-character hex $request_id for every request. Log it and pass it upstream, and nginx and Django share one ID. Django's validator accepts it, and the response carries it back to the client:
# /etc/nginx/conf.d/log_format.conf
log_format json_combined escape=json
'{'
'"ts":"$time_iso8601",'
'"remote_addr":"$remote_addr",'
'"request_id":"$request_id",'
'"method":"$request_method",'
'"uri":"$uri",'
'"status":$status,'
'"bytes":$body_bytes_sent,'
'"request_time":$request_time,'
'"upstream_time":"$upstream_response_time",'
'"user_agent":"$http_user_agent"'
'}';
# /etc/nginx/sites-available/myproject
server {
listen 443 ssl;
server_name app.example.com;
access_log /var/log/nginx/app.json.log json_combined;
location = /healthz {
access_log off;
proxy_pass http://127.0.0.1:8000;
}
location / {
proxy_pass http://127.0.0.1:8000;
proxy_set_header Host $host;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header X-Request-ID $request_id;
}
}
Why each part is there:
escape=json escapes quotes, backslashes and control characters in variable values. Without it, a user agent containing " produces a line that no JSON parser will accept.$status, $body_bytes_sent and $request_time are always numeric, so they're left unquoted and LogQL can compare them as numbers. $upstream_response_time is quoted because it can be - (no upstream was contacted) or a comma-separated list when nginx retried more than one upstream.$uri on purpose, not $request_uri. $request_uri includes the query string, and query strings are where password-reset tokens, magic-login links, API keys and ?email= parameters end up. If you need query parameters for analytics, log specific ones with $arg_name variables, not the whole string.access_log off. A load balancer polling every five seconds produces over 17,000 lines a day that tell you nothing.If another proxy or load balancer in front of nginx already sets X-Request-ID, you can keep its ID with a map that falls back to $request_id when the header is empty. Only do that when the upstream is under your control and strips client-supplied values. Otherwise you're back to trusting whatever the client sends. nginx's error_log can't be formatted as JSON. Ship it as plain text with a separate job label and accept that. It's low volume and mostly read by humans.
The stock /etc/logrotate.d/nginx already matches /var/log/nginx/*.log and signals nginx after rotation, so app.json.log is rotated with no extra work.
Loki stores log lines in compressed chunks and indexes only a small set of labels. That's why it's much cheaper to run than a full-text search cluster, and also why label design matters so much (next section). Run Loki on a separate monitoring host, not on the app server. If the logging store lives on the machine whose disk just filled, you lose the evidence and the service together.
Each app server needs an agent that reads journald and the nginx files and pushes them to Loki. Use Grafana Alloy. Promtail, the older Loki-specific agent, is deprecated in favour of Alloy and only receives maintenance fixes, so don't build anything new on it. Alloy's configuration language wires components together: sources read logs, processors transform them, writers send them. With the distro package, the file is /etc/alloy/config.alloy:
loki.write "default" {
endpoint {
url = "http://203.0.113.10:3100/loki/api/v1/push"
}
}
// ---- journald: gunicorn, celery, timers ----
loki.relabel "journal" {
forward_to = []
rule {
source_labels = ["__journal__systemd_unit"]
target_label = "unit"
}
}
loki.source.journal "system" {
forward_to = [loki.process.app_json.receiver]
relabel_rules = loki.relabel.journal.rules
labels = { job = "journal", host = "app1", env = "production" }
}
// Promote the JSON "level" field to a label (5 possible values: safe).
loki.process "app_json" {
stage.json {
expressions = { level = "" }
}
stage.labels {
values = { level = "" }
}
forward_to = [loki.write.default.receiver]
}
// ---- nginx JSON access log ----
local.file_match "nginx" {
path_targets = [{
"__path__" = "/var/log/nginx/app.json.log",
job = "nginx",
host = "app1",
env = "production",
}]
}
loki.source.file "nginx" {
targets = local.file_match.nginx.targets
forward_to = [loki.write.default.receiver]
}
Journal lines that aren't JSON, such as systemd's own "Started myproject.service" messages, just pass through the stage.json step without a level label. The alloy service user needs read access. On Ubuntu the journal is readable by the systemd-journal group and nginx's logs by adm, so run sudo usermod -aG systemd-journal,adm alloy and restart Alloy.
Alloy records how far it has read: the journal cursor, and the file offsets for tailed files. After a Loki outage it resumes where it stopped, as long as the data is still on the local disk. That's the real reason local retention exists. MaxRetentionSec=14day in journald means a two-week Loki outage costs you nothing. Don't expose the push endpoint publicly. Loki accepts whatever is sent to it. Keep it on a private network, or put it behind a reverse proxy with TLS and authentication.
This is where most Loki setups go wrong. Every unique combination of label values creates a separate stream, and each stream has its own chunks and index entries. Take three hosts, six units and five levels: at most 90 streams, which is trivial. Add request_id as a label and every request becomes its own stream. The index grows without limit, chunks stay tiny and poorly compressed, and ingestion starts rejecting pushes for exceeding the stream limit. Labels say where a line came from, never what it says.
| Good labels (small, fixed set of values) | Never labels (unbounded) |
|---|---|
host, env, job | request_id, trace_id |
unit (systemd service) | user_id, IP address, email |
level (five values) | uri, order_id, user agent |
High-cardinality fields stay in the JSON body, and LogQL parses them at query time. You pay that cost only when you run a query, and only over the streams the label selector picks out. A query has three parts: a stream selector in braces (uses the index), optional line filters (cheap substring or regex matching), then parsers and label filters (| json, | status >= 500).
# All Django errors in production
{env="production", unit="myproject.service", level="ERROR"}
# Everything about one request, across nginx and Django
{host="app1"} |= "3f9c2a7e61b04d0c9a1e5b7f2c8d4e10"
# Declined payments for one merchant, printed readably
{unit="myproject.service"} |= "payment.declined" | json | merchant_id = 42
| line_format "{{.ts}} {{.request_id}} order={{.order_id}} code={{.decline_code}}"
# Slow 5xx responses
{job="nginx"} | json | status >= 500 and request_time > 1
# 5xx per minute, for a Grafana time-series panel
sum(count_over_time({job="nginx"} | json | status >= 500 [1m]))
# p95 latency per path over 5-minute windows
quantile_over_time(0.95,
{job="nginx"} | json | unwrap request_time [5m]) by (uri)
Put line filters like |= "payment.declined" before | json. The substring match throws out most lines before the parser runs, and on a busy stream that can make a query ten times faster. The by (uri) grouping in the last query is fine because uri is an extracted label that exists only for that query and is never stored. It can still produce thousands of series if your URLs include IDs, so in Grafana use topk(10, ...) around it.
Support forwards a complaint: checkout failed for a customer at about 14:05, and the customer pasted the X-Request-ID from the error page. One query, {env="production"} |= "<that id>", returns the nginx line (502, upstream_time 30.001), the Django payment.capture_started event, and a gunicorn WORKER TIMEOUT from the error log. The payment gateway call hung past gunicorn's 30-second timeout. That's found in two minutes, without SSH and without having to know which of three hosts served the request.
Under GDPR, logs are personal data as soon as they contain IP addresses, email addresses, user IDs or anything else that can be linked to a person. Almost all logs do. Two principles follow. Data minimisation: don't log what you don't need. Storage limitation: delete what you have on a defined schedule. Centralising logs helps with both, because you have one retention setting instead of scattered files, but it also means that one sloppy logger.info(f"login {request.POST}") ends up in a store that half the engineering team can read.
Never log:
Authorization and Cookie headers.user_id. If support genuinely needs the email domain, log only that.Django already has tools for error reports. Decorate views that handle credentials with @sensitive_post_parameters() and functions with sensitive locals with @sensitive_variables(). SafeExceptionReporterFilter then masks those values in error reports, and it masks settings whose names contain words like PASSWORD, TOKEN or SECRET. On top of that, add a redaction filter as a safety net for fields someone will inevitably pass through extra:
# myproject/logging.py (continued)
import re
SENSITIVE_KEYS = {"password", "token", "authorization", "cookie",
"secret", "api_key", "email", "card_number"}
EMAIL_RE = re.compile(r"([A-Za-z0-9._%+-])[A-Za-z0-9._%+-]*@([A-Za-z0-9.-]+)")
class RedactFilter(logging.Filter):
def filter(self, record: logging.LogRecord) -> bool:
for key in list(record.__dict__):
if key.lower() in SENSITIVE_KEYS:
setattr(record, key, "[redacted]")
if isinstance(record.msg, str) and "@" in record.msg:
record.msg = EMAIL_RE.sub(r"\1***@\2", record.msg)
return True
The filter is a backstop, not a guarantee. It can't see a token hidden inside a longer string or a nested dict, so code review is still the main control. The email mask only covers the message template, not interpolated arguments, which is another reason to put data in extra fields, where it can be redacted by key.
Then have Loki actually delete old data. Retention is enforced by the compactor, and it has to be switched on explicitly:
# loki config.yaml (excerpt)
compactor:
working_directory: /var/lib/loki/compactor
retention_enabled: true
delete_request_store: filesystem
limits_config:
retention_period: 30d
Pick a period you can justify in your records of processing. 30 days covers most debugging and incident review. Security audit trails may justify longer, but put them in their own stream so they don't drag everything else along. Keep journald's MaxRetentionSec and logrotate's rotate count in line with it: a 30-day policy is meaningless if /srv/app/logs holds a year of gzip files. Restrict who can query production logs in Grafana just as you'd restrict database access. Also remember that erasure requests are much easier to satisfy when personal data never went into the logs.
With logs in Loki you can alert on them without instrumenting new metrics. You have two options. Loki's ruler evaluates LogQL expressions written in Prometheus alerting-rule format and sends alerts to Alertmanager (set alertmanager_url in the ruler config). Alternatively, Grafana-managed alert rules can use Loki as a data source. If you already run Grafana alerting with contact points set up, that's the simpler route. The rule logic is the same either way:
groups:
- name: myproject
rules:
- alert: HighServerErrorRate
expr: |
sum(count_over_time({job="nginx", env="production"}
| json | status >= 500 [5m])) > 25
for: 5m
labels:
severity: page
annotations:
summary: "More than 25 5xx responses in 5 minutes on app.example.com"
- alert: GunicornWorkerTimeouts
expr: |
sum(count_over_time({unit="myproject.service"} |= "WORKER TIMEOUT" [10m])) > 3
labels:
severity: ticket
- alert: AppStoppedLogging
expr: |
absent_over_time({unit="myproject.service"}[15m])
labels:
severity: page
Design alerts around what users notice. A 5xx rate pages someone. Worker timeouts open a ticket, because they point to a slow dependency before it becomes an outage. The for: 5m clause means the condition has to hold for five minutes before firing, which filters out single spikes during deploys. absent_over_time catches the silent failure mode: a service that crashed, or an Alloy agent that stopped shipping. Resist alerting on every ERROR line. Within a week that alert is noise and people ignore it, including the one time it mattered. Logs also don't replace an external uptime check. If the host is gone, it sends no logs, and the only signal left is absence.
RotatingFileHandler with multiple gunicorn workers. Workers rotate independently, so lines get lost and deleted inodes keep holding disk space. Use stdout, or WatchedFileHandler with logrotate.request_id, user_id or uri Loki labels. The number of streams explodes, ingestion hits stream limits and queries time out. Keep them in the JSON body and filter at query time.$request_uri or request bodies. Reset tokens and email addresses end up in a system with 30-day retention and wide read access.df doesn't change until the process restarts. Truncate the file instead, and check with lsof +L1.X-Request-ID without validation. It lets anyone inject newlines or pad every log line with kilobytes of junk.# journald limits are in effect
journalctl --disk-usage
systemd-analyze cat-config systemd/journald.conf | grep -E 'MaxUse|KeepFree|Retention'
# logrotate config parses; dry run, then one forced rotation
sudo logrotate -d /etc/logrotate.d/myproject
sudo logrotate -f /etc/logrotate.d/myproject
# nothing holds deleted log files open
sudo lsof +L1 | grep '\.log'
# nginx config valid and access lines are valid JSON
sudo nginx -t
tail -n 1 /var/log/nginx/app.json.log | python3 -m json.tool
# the same request_id appears in the response, Django's log and nginx's log
rid=$(curl -s -o /dev/null -D - https://app.example.com/ \
| awk 'tolower($1)=="x-request-id:" {print $2}' | tr -d '\r')
echo "$rid"
journalctl -u myproject.service -n 100 -o cat | grep "$rid"
grep "$rid" /var/log/nginx/app.json.log
# Django lines are JSON and carry the id
journalctl -u myproject.service -n 5 -o cat | python3 -c \
'import sys, json; [print(json.loads(l)["request_id"]) for l in sys.stdin if l.startswith("{")]'
# Alloy is running and has no push errors
systemctl status alloy
journalctl -u alloy --since "10 min ago" | grep -i error
Last, in Grafana Explore, run {host="app1"} |= "<rid>" and check that the nginx line and the Django line come back together. Then run {unit="myproject.service"} |= "password" over the last day and make sure it returns nothing. Once every log on the box has a size limit and a deletion date, one ID pulls up a whole request, and alerts fire on what users actually feel, logging stops being a disk-space hazard. It becomes the fastest debugging tool you have.