Security Advanced

Web Application Incident Response — Investigating an Active Compromise

From the first alert through containment, eradication, forensics, and post-incident review — a structured playbook for handling web app breaches.

KC Ramo · DjangoZen Team May 10, 2026 20 min read 416 views

Why this matters

Incidents will happen. The companies that survive them well aren't the ones with the fewest incidents — they're the ones with the most practiced response. This tutorial is the operational playbook for handling a web app breach end-to-end.

The phases of incident response

Standard model (NIST SP 800-61):

  1. Preparation — before anything happens
  2. Identification — "is this an incident?"
  3. Containment — stop the bleeding
  4. Eradication — remove the attacker
  5. Recovery — restore safe operations
  6. Lessons learned — improve

Each has specific actions, specific dangers, specific deliverables.

Phase 1 — Preparation

The single highest-leverage investment in IR. Before an incident:

Documented playbook

A written guide covering common scenarios. For a web app shop:

  • Suspected credential compromise
  • Active web shell on a server
  • Suspected data exfiltration
  • DDoS attack ongoing
  • Discovered vulnerability being exploited in the wild

Each scenario has: trigger, who's involved, first 30 minutes of actions, decision points, escalation criteria, communication templates.

Roles defined

  • Incident Commander — runs the response, makes decisions, owns timelines
  • Technical Lead — directs investigation and remediation
  • Communications Lead — internal updates, external customer comms, regulatory notification
  • Legal Lead — counsel for notification obligations
  • Liaison — external (DFIR firm, law enforcement, customers)

Small teams: one person wears multiple hats. The roles still need to be assigned and acknowledged.

Tooling ready

  • Centralized log access (Sentry, your aggregator)
  • Forensic image collection process documented
  • Out-of-band communication channel (Signal, separate Slack workspace)
  • Contact list for legal counsel, DFIR firms, key customers, regulators
  • Pre-drafted communication templates

Backups verified

Quarterly restore tests. Documentation of what's where. Immutable copies (so attackers can't tamper).

Tabletop exercises

Annual at minimum. Run hypothetical incidents through the team. Find gaps.

Phase 2 — Identification

"Is this an incident?" Initial triage.

Common alert sources

  • Sentry exception spike
  • Anomalous traffic in logs (large data downloads, many 500s, unusual API patterns)
  • Customer report ("my account did something I didn't do")
  • Threat intelligence notification (your domain in a breach dump)
  • Third-party notification (Stripe, GitHub, AWS detecting suspicious activity)
  • Self-discovery during routine work
  • Public disclosure (security researcher tweet, news article)

Initial questions

Document timestamps for each:

  1. What is the indicator? (Be precise — "500 errors" isn't enough; "500 errors on /api/orders/ from IP X spiking at HH:MM")
  2. Could this be a false positive? (Sometimes the answer is yes after 10 minutes of investigation. Document the reasoning.)
  3. What's the severity? (Production outage? Data exposure? Suspected compromise of specific account?)
  4. Who needs to know in the first hour? (Engineering on-call? Leadership? Legal? Customers?)
  5. What's our confidence level? ("Suspect" vs "confirmed" matters for what actions you can take.)

Triggers for declaring an incident

Decide in advance. Examples:

  • Evidence of unauthorized access to any system
  • Customer data accessed by unauthorized party
  • Encryption activity on file servers (ransomware indicator)
  • Significant unexplained outbound traffic
  • Public security disclosure naming your company

When a trigger fires: declare. Convene the response team. Open a dedicated incident channel. Start the clock.

Phase 3 — Containment

Stop the active damage. Two goals: prevent expansion AND preserve evidence.

Short-term containment

The first hour. Actions taken under uncertainty, with the goal of limiting blast radius.

  • Isolate compromised hosts — pull from network, NOT power off (lose memory state). For cloud, change security groups to drop all traffic.
  • Disable compromised accounts — but document first; revert if false positive
  • Revoke active sessions — delete the affected users' rows from django_session (or all rows to log everyone out), or rotate SECRET_KEY without fallbacks. Note that python manage.py clearsessions only removes expired sessions and logs no one out.
  • Block known malicious IPs at edge — Cloudflare, AWS WAF, ufw
  • Take database snapshot — preserve state for forensics
  • Capture memory images of affected servers if forensically capable
  • Preserve logs — copy to immutable storage, lock down access

Revoking sessions and tokens in a Django app

Be precise about what each revocation mechanism actually does, because a half-revocation gives false confidence. clearsessions only purges sessions that have already expired; it does not log anyone out. To invalidate every live session with the database or cached-database backend, delete the rows directly. To target only the accounts you believe were abused, decode sessions and match the user id, which keeps everyone else logged in while you investigate.

# python manage.py shell
from django.contrib.sessions.models import Session
from django.utils import timezone

suspect_ids = {"42", "317"}  # user pks as strings, as stored in the session

live = Session.objects.filter(expire_date__gte=timezone.now())
doomed = [s.pk for s in live
          if s.get_decoded().get("_auth_user_id") in suspect_ids]
Session.objects.filter(pk__in=doomed).delete()

# Nuclear option: log out everyone
# Session.objects.all().delete()

Iterating and decoding every session is slow on large tables, so for big sites prefer a full purge or rely on the password change: Django's default authentication verifies a session hash derived from the password, so setting a new password (or set_unusable_password()) invalidates that user's existing sessions on their next request. Signed-cookie sessions cannot be deleted server-side at all; the only global revocation is rotating SECRET_KEY without adding the old value to SECRET_KEY_FALLBACKS, which also invalidates password-reset links and anything else signed with it.

Sessions are rarely the only credential. API tokens (DRF TokenAuthentication rows, JWT signing keys, OAuth access and refresh tokens you issued) survive a session purge. Before you declare containment, list every bearer credential type the application accepts and record, in the timeline, which ones were revoked and when.

What NOT to do in short-term containment

  • Don't reboot servers — destroys memory state that has forensic value
  • Don't delete files — destroys evidence
  • Don't change passwords on all accounts immediately — destroys evidence of which accounts were used
  • Don't communicate externally until you know what happened
  • Don't "clean up" — the urge to restore normalcy will destroy the investigation

Long-term containment

After initial chaos settles (hours to days). Goals: maintain controlled state while preparing eradication.

  • Patch the entry vector if known
  • Maintain enhanced monitoring on affected systems
  • Limit administrative access
  • Document changes in a single source of truth

Phase 4 — Eradication

Remove the attacker entirely. Common mistake: declare victory too early.

Investigation first

Before eradication, you must know:

  • Initial access vector (how did they get in?)
  • Timeline of activity (when did it start? how long were they present?)
  • Scope of access (what did they touch?)
  • Persistence mechanisms (where are their backdoors?)
  • Data exfiltration (what did they take?)

Without this, eradication is whack-a-mole. The attacker is back tomorrow via the same or a similar vector.

Forensic discipline

  • Chain of custody — document every action, who took it, when
  • Working copies, not originals — never mutate evidence
  • Hash everything — every artifact has a SHA-256 hash for integrity verification
  • Timestamp logs — UTC, ISO 8601, source-stamped

Tools

  • Standard forensic tools — Volatility (memory), Plaso/log2timeline (timeline), Autopsy (disk)
  • Application logs — your Sentry + log aggregator
  • Cloud audit logs — CloudTrail (AWS), Cloud Logging (GCP), Activity Log (Azure)
  • Endpoint EDR — CrowdStrike, SentinelOne, Defender if deployed

Triage on a Linux web host

Before running anything on a suspect host, decide whether you trust its binaries. A capable attacker with root can replace ps or hide files from ls. Where possible, snapshot the disk and analyze the snapshot from a clean instance; where live triage is unavoidable, run commands from a known-good static toolkit mounted read-only and write every output to external storage, never to the suspect disk. The goal of live triage is to capture volatile state that a snapshot will not contain: processes, network connections, logged-in users.

# Capture volatile state first, to an evidence volume
OUT=/mnt/evidence/web01-$(date -u +%Y%m%dT%H%M%SZ)
mkdir -p "$OUT"
ps auxwwf                 > "$OUT/ps.txt"
ss -tanp                  > "$OUT/sockets.txt"
{ who -a; last -Fw; }    > "$OUT/logins.txt"
systemctl list-units --type=service --all > "$OUT/services.txt"
systemctl list-timers --all               > "$OUT/timers.txt"
crontab -l -u deploy      > "$OUT/cron-deploy.txt" 2>&1
ls -la /etc/cron.* /var/spool/cron        > "$OUT/cron-dirs.txt" 2>&1

# Files changed in the app tree since the last known-good deploy
find /srv/app -type f -newermt "2026-03-01 00:00" -printf '%TY-%Tm-%Td %TT %p\n' \
  | sort > "$OUT/app-changed.txt"

# Hash what you collected
sha256sum "$OUT"/* > "$OUT/SHA256SUMS"

Treat modification times as hints, not proof: they are trivially forged with touch. A stronger check is comparing the deployed tree against the build artifact you shipped. If you deploy from a versioned release archive or container image, hash every file in the known-good artifact and diff the result against the live tree; any file present only on the server, especially a .py file in a static or media directory, or a template with injected script tags, is a priority lead. For containerized deployments, docker diff on the running container lists every added, changed, or deleted path relative to its image.

Web access logs usually answer the initial-access question. Look for the first request to any file that should not exist, rare user agents, POST requests to endpoints that normally only receive GET, and bursts of 4xx responses from one address immediately preceding a 200 on an unusual path. That 4xx-then-200 shape is the signature of someone probing until something worked, and the timestamp of the first 200 is often your intrusion start time.

Django and PostgreSQL persistence checks

Application-level persistence is easy to overlook because it lives in data, not files. Rebuilding servers does nothing about a rogue superuser in the database.

# python manage.py shell
from datetime import datetime, timezone as tz
from django.contrib.auth import get_user_model
from django.contrib.admin.models import LogEntry

since = datetime(2026, 3, 1, tzinfo=tz.utc)
User = get_user_model()

User.objects.filter(is_superuser=True).values("username", "date_joined", "last_login")
User.objects.filter(is_staff=True, date_joined__gte=since).values("username", "email")
LogEntry.objects.filter(action_time__gte=since).values_list(
    "action_time", "user__username", "content_type__model", "object_repr", "change_message")
-- PostgreSQL: roles and live connections
SELECT rolname, rolsuper, rolcreaterole, rolcanlogin, rolvaliduntil
FROM pg_roles ORDER BY rolname;

SELECT pid, usename, application_name, client_addr, backend_start, state, left(query, 120)
FROM pg_stat_activity WHERE backend_type = 'client backend';

Remember that the admin LogEntry table only records actions made through the admin interface and can itself be edited by anyone with database access, so absence of entries proves nothing. Also review database-level objects an attacker could plant: unexpected functions, triggers on user or payment tables, and extra roles with login rights. Finally, compare installed Python packages against your lock file; a package installed outside the deploy pipeline, or an installed version that differs from the pinned one, deserves investigation as a potential supply-chain or persistence vector.

What to eradicate

  • Web shells, malicious cron jobs, suspicious binaries
  • Backdoor accounts (added admin accounts, SSH keys, OAuth grants)
  • Modified configurations (changed firewall rules, disabled security tools)
  • Persistence mechanisms (scheduled tasks, systemd services, init scripts)
  • Modified application code (injected JavaScript, backdoored views)

Validation

After eradication, validate. Re-scan. Re-image where appropriate. Don't trust a "cleaned" system without verifying.

Phase 5 — Recovery

Bring systems back. Two priorities: business resumption AND defending against the same attack happening tomorrow.

Restoration sources

  • Clean backups (verified to predate the compromise)
  • Rebuilt from scratch (especially compromised servers — never trust a host you've cleaned, only one you've rebuilt)
  • Patched and reconfigured existing systems

Validation before bringing systems back

  • Vulnerability scan
  • Configuration review
  • Patch level confirmation
  • Account audit (no unexpected accounts, all MFA enrolled)
  • Log monitoring tuned for anomalies

Phased return

Critical systems first, broader systems later. Monitor each tier as it comes online.

Password and credential rotation

  • Force password reset for all users
  • Revoke all sessions
  • Rotate API keys, SSH keys, certificates
  • Rotate secrets in secret manager (database passwords, third-party API keys)
  • Re-enroll MFA if compromise included MFA bypass mechanisms

This is the boring, painful work. It's also where many recoveries fail — the attacker had a credential you forgot to rotate.

Phase 6 — Lessons learned

The post-incident review. Done within 1-2 weeks of incident closure, before memory fades.

Blameless review

The goal is improvement, not punishment. People who fear consequences hide information. The review must explicitly be blameless.

Standard agenda

  • Timeline reconstruction — what happened, when, in what order
  • Root cause analysis — beyond "the firewall let it through"; the human and process factors
  • Decision review — what calls were made, what worked, what didn't
  • Detection review — could we have caught this earlier? What signal should have fired?
  • Response review — was the playbook followed? Was it sufficient?
  • Action items — specific, owned, time-bounded improvements

Deliverables

  • Internal incident report — full timeline, root cause, response, action items
  • Executive summary — for leadership and board
  • Customer communication — if data was exposed, what happened and what's been done
  • Regulatory filing — if required by jurisdiction (GDPR within 72 hours)
  • Public disclosure — if appropriate, often months later when context is full

Specific action items typically include

  • Technical: patch the specific vulnerability, add the missing detection rule, harden the broken control
  • Process: update playbook for the gap that hurt, add the missing escalation path
  • People: hire if capacity was the issue, train if knowledge was the issue
  • Tooling: deploy the missing monitoring, integrate the right systems

Regulatory and customer notification

Often time-sensitive. Don't wait for full investigation.

GDPR

  • Notify supervisory authority within 72 hours of becoming aware (not within 72 hours of resolving)
  • Notify affected individuals if high risk to rights and freedoms
  • "Becoming aware" interpretation: when you have reasonable certainty an incident occurred, not when you have full forensic certainty

Sector regulators

Financial services (your national financial regulator, e.g. DNB in the Netherlands), healthcare, critical infrastructure — sector-specific timelines.

Contractual obligations

Your customer contracts may have stricter notification windows than law requires (24-48 hours common in enterprise B2B). Check before you need to know.

What to say

  • What happened (factual, not speculative)
  • What information was affected
  • What you've done to contain and remediate
  • What recipients should do
  • How they can contact you for questions

What NOT to say:

  • Detailed attribution ("It was nation-state actor X") — usually wrong, always speculative
  • Promises you can't keep ("This will never happen again")
  • Minimizing language ("a minor incident," "limited impact")

A first-hour checklist

For when an alert just fired and you need a playbook on the wall:

  • [ ] Document the alert: timestamp, source, indicator
  • [ ] Triage: is this a real incident? (5-10 minutes)
  • [ ] Declare if it is. Open the incident channel.
  • [ ] Page the on-call team
  • [ ] Identify Incident Commander
  • [ ] Start a timeline document — every action gets logged with timestamp
  • [ ] Capture immediate evidence: log queries, screenshots, memory dumps if possible
  • [ ] Implement short-term containment (isolate, revoke, block)
  • [ ] Notify internal stakeholders (engineering leadership, legal counsel)
  • [ ] Reach out to DFIR firm if scope is unclear
  • [ ] Schedule the next checkpoint (30 minutes? 1 hour?)
  • [ ] Do not communicate externally yet

That's enough structure to avoid the worst mistakes while you figure out what's actually happening.

Why preparation decides the outcome

Incidents are inevitable, and what separates organizations that survive them well from those that suffer lasting damage is not how few incidents they have but how prepared they are to respond. The middle of a breach is the worst time to figure out who does what, where the logs are, and how to contain the spread. Preparation — a plan, defined roles, accessible logs, practiced procedures — is what turns a chaotic scramble into a controlled response. Investing in readiness before anything happens is the highest-leverage incident-response work there is, because it determines how quickly and cleanly you can act when the alert finally fires.

Containment before eradication

When a compromise is confirmed, the instinct to immediately clean everything up can backfire — acting too fast can destroy evidence, tip off the attacker, or miss the full extent of the intrusion. The disciplined sequence is to first contain the damage, stopping the bleeding and preventing further spread, while preserving evidence, then to thoroughly understand the scope, and only then to eradicate the attacker and recover. Rushing to eradication before you understand how the attacker got in and how far they reached often leaves them a way back. Methodical response — contain, investigate, eradicate, recover — is what ensures you actually remove the threat rather than merely disturbing it.

Learning from every incident

The final and most often skipped phase of incident response is the lessons-learned review, and it is what makes each incident strengthen rather than merely scar you. Honestly examining how the attacker got in, why it was not caught sooner, and what would have limited the damage produces concrete improvements — a closed gap, a new alert, a better process. Organizations that treat incidents as expensive lessons get steadily harder to compromise; those that just clean up and move on repeat the same mistakes. Building the post-incident review into your process, without blame, is how a breach becomes an investment in resilience rather than only a loss.

Building the incident timeline

The timeline is the investigation's central artifact. Every question you will be asked later, by leadership, regulators, customers, or insurers, is answered from it: when did access begin, what was touched, when did we know, what did we do. Build it as you go, not afterwards, and keep two distinct streams in it: attacker activity (reconstructed from evidence) and response actions (what your team did and when). Mixing them makes it impossible to tell whether a log line shows the attacker or a responder poking around.

Normalize every timestamp to UTC at the moment you record it, and note the source's original timezone alongside. Most timeline errors come from a handful of predictable pitfalls:

SourceTypical pitfallMitigation
Web server access logsLogged in server local time; request time recorded at completion, not startConfigure UTC or ISO 8601 with offset; note request duration for long requests
Application logs / SentrySampling drops events; retention shorter than dwell timeRecord sampling rate; export before retention expires
Load balancer / CDN logsClient IP is real, but origin logs only show the proxy addressCorrelate by request ID; ensure the app logs a trusted forwarded-for value
Cloud audit logsDelivery delay of several minutes; per-region trailsQuery all regions; do not treat an empty recent window as proof of inactivity
Host logs (journald, auth.log)Clock drift on hosts without NTP; logs erasable by rootRecord measured drift per host; prefer copies already shipped off-host
Database logsStatement logging usually disabled; only errors and slow queries recordedRely on audit extensions if deployed; otherwise infer from application logs

A request ID propagated from the edge through the application into logs is the single most useful correlation key you can have during an investigation. If you do not have one, adding it is a cheap lessons-learned action item.

Also record evidence gaps explicitly. "No log data for 2 March to 9 March because retention is 14 days" is a finding. It changes what you can claim in notifications: you cannot state that no data was accessed during a window for which you have no records, and saying so honestly early is far better than retracting it later.

Credential rotation: order matters

Rotation is only effective after the entry vector is closed and persistence is removed; rotate earlier and the attacker simply reads the new secrets from the same foothold. Rotate in order of blast radius, starting with credentials that grant access to other credentials.

OrderCredentialWhy first / gotcha
1Cloud IAM keys, CI/CD tokens, secret-manager accessThese unlock everything else. Check for keys the attacker created, not just ones you know about.
2Deploy SSH keys and authorized_keys on every hostAudit files for extra entries rather than only replacing your own key.
3Database passwordsRequires coordinated app restart; plan a brief maintenance window.
4Django SECRET_KEYIf leaked, do not keep it in SECRET_KEY_FALLBACKS; accept that all sessions and reset links die.
5Third-party API keys (payment, email, storage)Review each provider's own logs for use of the old key before revoking it; that log is evidence.
6User passwords, sessions, issued API tokensLast, once attackers can no longer observe the reset flow.

Keep a rotation ledger: each secret, where it is used, who rotated it, when, and how the rotation was verified (for example, confirming the old key is rejected). Secrets that nobody can locate the owner or consumer of are themselves a finding.

Failure modes that derail investigations

Most failed responses fail in familiar ways. Knowing them in advance lets the Incident Commander recognize them in the moment.

  • Tipping off the attacker. Visibly blocking one IP or deleting one web shell tells an active attacker they are discovered, and they may switch to a backup foothold or destroy data. When you have reason to believe the attacker is still active, prepare containment steps in parallel and execute them together.
  • Restoring a backdoor. A backup taken after initial access contains the attacker's persistence. The timeline determines which backup is safe; the newest one usually is not.
  • Anchoring on the first finding. The first web shell found is rarely the only one. Keep asking what else the same access would have allowed, and search for every indicator across all hosts, not just the one that alerted.
  • Losing evidence to automation. Autoscaling groups terminate unhealthy instances, log rotation deletes old files, and containers are rescheduled. Suspend these for affected resources early in containment.

Closing the series

These ten tutorials cover the web application security landscape from threat models to incident response. The thread connecting them: defense in depth, applied with discipline, tested under realistic adversary pressure.

You will not prevent every incident. You will not detect every attack. Your goal is to make attacks expensive enough that most adversaries pick another target, detect the ones who persist before they finish, and respond effectively when something does succeed.

That outcome is achievable. The companies that achieve it are not the ones with the biggest security budget — they're the ones that prioritize, practice, and improve continuously. Tools matter. Discipline matters more.

Good luck out there.