File Storage at Scale in Django: S3, Signed URLs, Direct Uploads, and CDNs

Storing uploads on the application server's local disk works right up until you add a second server, deploy to an ephemeral container, or a user uploads a 2GB file that ties up a worker. This tutorial covers file storage that scales: moving to S3-compatible object storage with django-storages, serving private files with signed URLs, offloading uploads directly to the bucket, streaming large files, and the security and cost pitfalls that catch teams in production.

Why local disk fails at scale

Local disk breaks the moment your app is more than one machine. With two web servers behind a load balancer, a file uploaded to server A does not exist on server B, so downloads fail half the time. On modern platforms — containers, autoscaling groups, most PaaS — the filesystem is also ephemeral: it vanishes on every deploy or restart, taking user uploads with it. And local disk does not scale independently of compute, so storage growth forces you to over-provision servers. Object storage solves all three: it is shared across every server, durable across deploys, and scales without touching your app tier.

django-storages and S3

Django's storage backend is pluggable. django-storages provides an S3 backend (which also works with any S3-compatible service — AWS S3, Cloudflare R2, Backblaze B2, MinIO, DigitalOcean Spaces). You point Django at it and every FileField/ImageField transparently reads and writes to the bucket.

# settings.py (Django 4.2+ STORAGES API)
STORAGES = {
    "default": {
        "BACKEND": "storages.backends.s3.S3Storage",
        "OPTIONS": {
            "bucket_name": "myapp-media",
            "region_name": "eu-central-1",
            "querystring_auth": True,   # private files -> signed URLs
        },
    },
    "staticfiles": {"BACKEND": "storages.backends.s3.S3Storage",
                    "OPTIONS": {"bucket_name": "myapp-static", "querystring_auth": False}},
}

The application code does not change at all — model.file.save(), model.file.url, and forms all keep working. Only the destination moves. Keep media and static in separate buckets (or prefixes) with different access rules: static assets are public and cacheable, user media is often private.

Credentials: use roles, not keys

The most common security mistake is pasting long-lived access keys into settings. In a cloud environment, attach an IAM role to the instance or container instead; the SDK picks up temporary, rotating credentials automatically and there is no secret to leak. Scope the role's policy to exactly the buckets and actions the app needs — reading and writing its own media bucket, nothing more. A leaked key with s3:* on all buckets is a company-ending incident; a scoped role that rotates hourly is not.

Public files vs signed URLs

Public assets (logos, CSS) can be served directly from a public bucket URL. Private files (a user's purchased ebook, an invoice PDF) must never be world-readable. With querystring_auth=True, file.url returns a signed URL — a time-limited, cryptographically signed link that grants access for a short window.

# Generate a signed URL valid for 5 minutes
url = model.document.storage.url(model.document.name, expire=300)

This is how you serve paid downloads securely without proxying the bytes through Django: check the user's permission in a view, then hand them a short-lived signed URL and let S3 serve the file directly. The permission check stays in your app; the heavy lifting stays off it.

Direct browser uploads

Routing uploads through Django means the file travels browser → your server → S3, tying up a worker for the whole transfer and capping you at your server's bandwidth. For large files, use a presigned POST: Django generates a short-lived, constrained upload authorization, and the browser uploads straight to S3.

import boto3
def upload_authorization(request):
    s3 = boto3.client("s3")
    post = s3.generate_presigned_post(
        "myapp-media", f"uploads/{request.user.id}/{uuid4()}",
        Fields={"Content-Type": "image/jpeg"},
        Conditions=[{"Content-Type": "image/jpeg"}, ["content-length-range", 0, 10485760]],
        ExpiresIn=300,
    )
    return JsonResponse(post)

The Conditions are the security boundary: they pin the content type and cap the size (here 10MB), so the authorization cannot be reused to upload arbitrary or oversized content. Your server never touches the bytes, so a thousand concurrent uploads cost it almost nothing.

Streaming large files

Never load a large file fully into memory to serve it — that is how one download OOMs a worker. When you must serve through Django (for access logging, say), stream it, and prefer redirecting to a signed URL so S3 handles range requests, resumable downloads, and bandwidth for you.

from django.http import StreamingHttpResponse

def download(request, pk):
    doc = get_object_or_404(Document, pk=pk, owner=request.user)
    # Best: redirect to a signed URL and let S3 serve it
    return redirect(doc.file.storage.url(doc.file.name, expire=120))

Image processing pipelines

User images need derivatives — thumbnails, web-optimized sizes, format conversion. Do this asynchronously: accept the upload, enqueue a Celery task that generates the variants and writes them back to the bucket, and show a placeholder until they are ready. Generating a dozen thumbnails inline blocks the request and does not scale; a library like easy-thumbnails or django-imagekit paired with a task queue keeps uploads fast and processing off the request path.

Put a CDN in front

Serving media straight from the bucket works but is not fast globally and can be expensive in egress. Front the bucket with a CDN (CloudFront, Cloudflare) so files are cached at edge locations near users, and set long Cache-Control headers on immutable assets. Combine this with content-hashed filenames (Django's ManifestStaticFilesStorage for static) so you can cache forever and bust the cache by changing the URL.

Migrating existing local files to S3

Switching the storage backend does not move the files you already have. Plan a migration: a management command that walks existing FileFields and re-saves each object so its file lands in the bucket, ideally run while the app still reads from local disk as a fallback.

for doc in Document.objects.all():
    if doc.file and not doc.file.name.startswith("uploads/"):
        content = doc.file.read()
        doc.file.save(doc.file.name, ContentFile(content))  # writes to S3

Do it in batches, verify counts and a sample of downloads, and only cut over reads once every file is confirmed present. A big-bang switch that assumes the files "just moved" is how you lose a week of uploads.

Backups, versioning, and lifecycle

Object storage is durable but not a backup — a bad deploy or a bug can still delete or overwrite the wrong object. Enable bucket versioning so every overwrite keeps the prior version and an accidental delete is recoverable. Pair it with lifecycle policies to control cost: expire old versions after a window, and transition cold files to cheaper storage tiers automatically. Versioning plus lifecycle gives you undo without unbounded storage growth.

Multipart uploads for very large files

For files in the gigabytes, a single upload is fragile — one dropped connection and the whole transfer restarts. Multipart upload splits the file into parts uploaded independently and reassembled by S3, so a failed part retries alone and large uploads can run in parallel. The SDKs do this transparently above a size threshold, and presigned multipart lets the browser drive it directly. If your product accepts large media, video, or datasets, multipart is what makes those uploads reliable rather than a coin flip.

Testing code that touches storage

Tests should not hit a real bucket — it is slow, costs money, and pollutes shared state. Use an in-memory or local storage backend in tests, or mock the S3 layer with a library like moto that simulates the S3 API in-process. Override STORAGES in test settings so uploads go to a temp location, and assert on what your code does (a file saved, a signed URL issued) rather than on S3 itself. Fast, isolated storage tests keep the suite quick and deterministic.

Signed URL expiry, caching, and the CDN

Signed URL lifetime is a genuine tradeoff. Too short and links break mid-download or when a user pauses; too long and a leaked URL grants access for its whole window. Match the expiry to the use — minutes for a direct download, longer for a document a user keeps open — and never make a signed URL effectively permanent. Note the interaction with CDNs: because the signature is in the query string, each signed URL is a distinct cache key, which defeats caching for private files. Cache public assets aggressively, and keep private, signed content on a path where you accept it is not edge-cached.

Content-Type and download behaviour

How a browser treats a file is driven by headers you control at upload. Set the correct Content-Type so images render and PDFs open, and set Content-Disposition: attachment when you want a genuine download rather than inline display. Getting these wrong is a subtle bug — a PDF that downloads as a raw byte stream, or an SVG served as HTML that becomes a stored-XSS vector. Pin content types explicitly at write time rather than trusting whatever the client claimed.

Access logging and audit trails

For sensitive files — invoices, contracts, personal data — you often must prove who downloaded what and when. Object stores can emit access logs of every request, but those are coarse; for real audit you record the access in your own database at the moment you issue a signed URL, capturing the user, the object, the time, and the reason. That gives you a queryable trail tied to your identities rather than raw IPs, which is what compliance regimes and incident investigations actually need. Design the audit record when you build the download path, because reconstructing "who accessed this" after the fact from storage logs alone is painful and often impossible.

The cost model: storage, requests, and egress

Object storage bills on three axes and teams routinely misjudge which one hurts. Storage per gigabyte is usually the smallest line. Request charges (per GET/PUT) add up for workloads with millions of tiny operations. And egress — bytes served out to the internet — is where the surprise bills come from, often dwarfing storage itself. The levers are straightforward once you see them: a CDN with a high cache-hit rate slashes egress by serving from the edge, lifecycle policies move cold data to cheaper tiers, and batching or caching cuts request counts. Model these three before you scale, because a media-heavy product's storage bill is mostly egress, and egress is the one you control with caching rather than cleanup.

Security and cost pitfalls

Lock the bucket down: block all public access by default and grant it back only where genuinely needed. Validate uploads — check content type and size server-side, never trust the client, and for user-supplied files consider malware scanning before they are served to others. Watch egress cost, which is where object-storage bills surprise teams; a CDN with good cache hit rates is the main defense. Finally, remember collectstatic now uploads to S3, so give your deploy pipeline write access to the static bucket and expect the first deploy to take longer as it pushes every asset.