File Storage at Scale in Django: S3, Signed URLs, Direct Uploads, and CDNs

Storing uploads on the application server's local disk works right up until you add a second server, deploy to an ephemeral container, or a user uploads a 2GB file that ties up a worker. This tutorial covers file storage that scales: moving to S3-compatible object storage with django-storages, serving private files with signed URLs, offloading uploads directly to the bucket, streaming large files, and the security and cost pitfalls that catch teams in production.

Why local disk fails at scale

Local disk breaks the moment your app is more than one machine. With two web servers behind a load balancer, a file uploaded to server A does not exist on server B, so downloads fail half the time. On modern platforms — containers, autoscaling groups, most PaaS — the filesystem is also ephemeral: it vanishes on every deploy or restart, taking user uploads with it. And local disk does not scale independently of compute, so storage growth forces you to over-provision servers. Object storage solves all three: it is shared across every server, durable across deploys, and scales without touching your app tier.

django-storages and S3

Django's storage backend is pluggable. django-storages provides an S3 backend (which also works with any S3-compatible service — AWS S3, Cloudflare R2, Backblaze B2, MinIO, DigitalOcean Spaces). You point Django at it and every FileField/ImageField transparently reads and writes to the bucket.

# settings.py (Django 4.2+ STORAGES API)
STORAGES = {
    "default": {
        "BACKEND": "storages.backends.s3.S3Storage",
        "OPTIONS": {
            "bucket_name": "myapp-media",
            "region_name": "eu-central-1",
            "querystring_auth": True,   # private files -> signed URLs
        },
    },
    "staticfiles": {"BACKEND": "storages.backends.s3.S3Storage",
                    "OPTIONS": {"bucket_name": "myapp-static", "querystring_auth": False}},
}

The application code does not change at all — model.file.save(), model.file.url, and forms all keep working. Only the destination moves. Keep media and static in separate buckets (or prefixes) with different access rules: static assets are public and cacheable, user media is often private.

Credentials: use roles, not keys

The most common security mistake is pasting long-lived access keys into settings. In a cloud environment, attach an IAM role to the instance or container instead; the SDK picks up temporary, rotating credentials automatically and there is no secret to leak. Scope the role's policy to exactly the buckets and actions the app needs — reading and writing its own media bucket, nothing more. A leaked key with s3:* on all buckets is a company-ending incident; a scoped role that rotates hourly is not.

A minimal policy for the application role

"Scoped" is easier to agree with than to write. A good starting point for the web tier is object-level read, write and delete on the media bucket only, plus the ability to list it (which some storage operations need). Note that object actions apply to the bucket/* ARN while s3:ListBucket applies to the bucket itself — mixing those up is the usual reason a hand-written policy "does not work".

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "MediaObjects",
      "Effect": "Allow",
      "Action": ["s3:GetObject", "s3:PutObject", "s3:DeleteObject"],
      "Resource": "arn:aws:s3:::myproject-media/*"
    },
    {
      "Sid": "MediaList",
      "Effect": "Allow",
      "Action": "s3:ListBucket",
      "Resource": "arn:aws:s3:::myproject-media"
    }
  ]
}

Give the deploy pipeline a different role that can write the static bucket, and give nothing at runtime permission to change bucket policies, lifecycle rules or versioning. If you use SSE-KMS instead of the default S3-managed encryption, the role also needs kms:Decrypt and kms:GenerateDataKey on that key — a missing KMS grant surfaces as a confusing AccessDenied on objects the S3 policy clearly allows. On S3-compatible providers that lack IAM roles, create a dedicated key per environment with the narrowest scope the provider supports, load it from the environment rather than from settings files, and rotate it on a schedule.

Public files vs signed URLs

Public assets (logos, CSS) can be served directly from a public bucket URL. Private files (a user's purchased ebook, an invoice PDF) must never be world-readable. With querystring_auth=True, file.url returns a signed URL — a time-limited, cryptographically signed link that grants access for a short window.

# Generate a signed URL valid for 5 minutes
url = model.document.storage.url(model.document.name, expire=300)

This is how you serve paid downloads securely without proxying the bytes through Django: check the user's permission in a view, then hand them a short-lived signed URL and let S3 serve the file directly. The permission check stays in your app; the heavy lifting stays off it.

Direct browser uploads

Routing uploads through Django means the file travels browser → your server → S3, tying up a worker for the whole transfer and capping you at your server's bandwidth. For large files, use a presigned POST: Django generates a short-lived, constrained upload authorization, and the browser uploads straight to S3.

import boto3
def upload_authorization(request):
    s3 = boto3.client("s3")
    post = s3.generate_presigned_post(
        "myapp-media", f"uploads/{request.user.id}/{uuid4()}",
        Fields={"Content-Type": "image/jpeg"},
        Conditions=[{"Content-Type": "image/jpeg"}, ["content-length-range", 0, 10485760]],
        ExpiresIn=300,
    )
    return JsonResponse(post)

The Conditions are the security boundary: they pin the content type and cap the size (here 10MB), so the authorization cannot be reused to upload arbitrary or oversized content. Your server never touches the bytes, so a thousand concurrent uploads cost it almost nothing.

Bucket CORS for browser uploads

A browser will refuse to POST to the bucket from your app's origin unless the bucket returns the right CORS headers. Configure this once, as infrastructure, and allow only your real origins and only the methods the upload flow uses:

s3.put_bucket_cors(
    Bucket="myproject-media",
    CORSConfiguration={
        "CORSRules": [{
            "AllowedOrigins": ["https://app.example.com"],
            "AllowedMethods": ["POST", "PUT"],
            "AllowedHeaders": ["*"],
            "ExposeHeaders": ["ETag"],
            "MaxAgeSeconds": 3600,
        }]
    },
)

ExposeHeaders: ["ETag"] is only needed if the browser drives multipart uploads, where it must read each part's ETag. Resist "AllowedOrigins": ["*"]: CORS is not an authentication mechanism, but a narrow origin list is cheap defence in depth and makes the bucket's intended use explicit.

The complete flow: authorize, upload, confirm

The presigned POST is only the middle of the story. A robust direct-upload flow has three steps, and skipping the last one is the most common design flaw: if the browser can simply tell your app "my file is at key X", it can claim someone else's object or a file that was never uploaded. The server must choose the key, remember that it chose it, and verify the object before attaching it to a model. The example assumes Document has original_name, size and content_type columns alongside its FileField; recording that metadata at confirmation time means list pages never have to ask S3 about each object.

import boto3
from uuid import uuid4

from django.contrib.auth.decorators import login_required
from django.core import signing
from django.http import JsonResponse, HttpResponseBadRequest
from django.views.decorators.http import require_POST

ALLOWED_TYPES = {"application/pdf", "image/jpeg", "image/png"}
MAX_BYTES = 50 * 1024 * 1024
BUCKET = "myproject-media"


@login_required
@require_POST
def authorize_upload(request):
    content_type = request.POST.get("content_type", "")
    if content_type not in ALLOWED_TYPES:
        return HttpResponseBadRequest("unsupported type")
    key = f"incoming/{request.user.pk}/{uuid4().hex}"
    post = boto3.client("s3").generate_presigned_post(
        BUCKET, key,
        Fields={"Content-Type": content_type},
        Conditions=[{"Content-Type": content_type},
                    ["content-length-range", 1, MAX_BYTES]],
        ExpiresIn=300,
    )
    token = signing.dumps({"key": key, "uid": request.user.pk}, salt="upload")
    return JsonResponse({"upload": post, "token": token})


@login_required
@require_POST
def confirm_upload(request):
    try:
        data = signing.loads(request.POST["token"], salt="upload", max_age=3600)
    except (KeyError, signing.BadSignature):
        return HttpResponseBadRequest("invalid token")
    if data["uid"] != request.user.pk:
        return HttpResponseBadRequest("invalid token")

    s3 = boto3.client("s3")
    head = s3.head_object(Bucket=BUCKET, Key=data["key"])   # raises ClientError if absent
    if head["ContentLength"] > MAX_BYTES or head["ContentType"] not in ALLOWED_TYPES:
        s3.delete_object(Bucket=BUCKET, Key=data["key"])
        return HttpResponseBadRequest("rejected")

    final_key = data["key"].replace("incoming/", "documents/", 1)
    s3.copy_object(Bucket=BUCKET, Key=final_key,
                   CopySource={"Bucket": BUCKET, "Key": data["key"]})
    s3.delete_object(Bucket=BUCKET, Key=data["key"])

    doc = Document(owner=request.user, size=head["ContentLength"],
                   content_type=head["ContentType"],
                   original_name=request.POST.get("filename", "")[:255])
    doc.file.name = final_key          # attach the existing object, no re-upload
    doc.save()
    return JsonResponse({"id": doc.pk})

The signed token replaces a "pending uploads" table: it proves the key was issued by your server to this user, and max_age bounds how long a confirmation can arrive (signing.SignatureExpired is a subclass of BadSignature, so one except covers both). Assigning doc.file.name directly attaches the object without Django re-uploading it. Uploading to an incoming/ prefix and moving on confirmation means anything left under incoming/ is by definition abandoned, which a one-day expiration lifecycle rule scoped to that prefix can clean up automatically. A single copy_object call is limited to 5GB; for larger objects use the managed s3.copy() transfer method, which performs a multipart copy for you. Wrap the head_object call in a botocore.exceptions.ClientError handler in real code so a missing object returns a clean 400 instead of a 500, and make the confirm step idempotent if clients may retry it.

On the browser side the flow is three requests: POST to the authorize view, POST the multipart form to upload.url, then POST the token to the confirm view. The only gotcha is ordering: S3 requires the file field to be the last field in the multipart form, so append every field returned in upload.fields first, exactly as given.

If you need a progress bar, use XMLHttpRequest for the middle request, because its upload.onprogress event reports bytes sent while fetch does not. Remember that the content-type check in the confirm step only proves what the client declared; for files other users will open, run content sniffing or malware scanning asynchronously before the document becomes visible to anyone but the uploader.

Streaming large files

Never load a large file fully into memory to serve it — that is how one download OOMs a worker. When you must serve through Django (for access logging, say), stream it, and prefer redirecting to a signed URL so S3 handles range requests, resumable downloads, and bandwidth for you.

from django.http import StreamingHttpResponse

def download(request, pk):
    doc = get_object_or_404(Document, pk=pk, owner=request.user)
    # Best: redirect to a signed URL and let S3 serve it
    return redirect(doc.file.storage.url(doc.file.name, expire=120))

Image processing pipelines

User images need derivatives — thumbnails, web-optimized sizes, format conversion. Do this asynchronously: accept the upload, enqueue a Celery task that generates the variants and writes them back to the bucket, and show a placeholder until they are ready. Generating a dozen thumbnails inline blocks the request and does not scale; a library like easy-thumbnails or django-imagekit paired with a task queue keeps uploads fast and processing off the request path.

Put a CDN in front

Serving media straight from the bucket works but is not fast globally and can be expensive in egress. Front the bucket with a CDN (CloudFront, Cloudflare) so files are cached at edge locations near users, and set long Cache-Control headers on immutable assets. Combine this with content-hashed filenames (Django's ManifestStaticFilesStorage for static) so you can cache forever and bust the cache by changing the URL.

Migrating existing local files to S3

Switching the storage backend does not move the files you already have. Plan a migration: a management command that walks existing FileFields and re-saves each object so its file lands in the bucket, ideally run while the app still reads from local disk as a fallback.

for doc in Document.objects.all():
    if doc.file and not doc.file.name.startswith("uploads/"):
        content = doc.file.read()
        doc.file.save(doc.file.name, ContentFile(content))  # writes to S3

Do it in batches, verify counts and a sample of downloads, and only cut over reads once every file is confirmed present. A big-bang switch that assumes the files "just moved" is how you lose a week of uploads.

Backups, versioning, and lifecycle

Object storage is durable but not a backup — a bad deploy or a bug can still delete or overwrite the wrong object. Enable bucket versioning so every overwrite keeps the prior version and an accidental delete is recoverable. Pair it with lifecycle policies to control cost: expire old versions after a window, and transition cold files to cheaper storage tiers automatically. Versioning plus lifecycle gives you undo without unbounded storage growth.

Multipart uploads for very large files

For files in the gigabytes, a single upload is fragile — one dropped connection and the whole transfer restarts. Multipart upload splits the file into parts uploaded independently and reassembled by S3, so a failed part retries alone and large uploads can run in parallel. The SDKs do this transparently above a size threshold, and presigned multipart lets the browser drive it directly. If your product accepts large media, video, or datasets, multipart is what makes those uploads reliable rather than a coin flip.

Testing code that touches storage

Tests should not hit a real bucket — it is slow, costs money, and pollutes shared state. Use an in-memory or local storage backend in tests, or mock the S3 layer with a library like moto that simulates the S3 API in-process. Override STORAGES in test settings so uploads go to a temp location, and assert on what your code does (a file saved, a signed URL issued) rather than on S3 itself. Fast, isolated storage tests keep the suite quick and deterministic.

Signed URL expiry, caching, and the CDN

Signed URL lifetime is a genuine tradeoff. Too short and links break mid-download or when a user pauses; too long and a leaked URL grants access for its whole window. Match the expiry to the use — minutes for a direct download, longer for a document a user keeps open — and never make a signed URL effectively permanent. Note the interaction with CDNs: because the signature is in the query string, each signed URL is a distinct cache key, which defeats caching for private files. Cache public assets aggressively, and keep private, signed content on a path where you accept it is not edge-cached.

Content-Type and download behaviour

How a browser treats a file is driven by headers you control at upload. Set the correct Content-Type so images render and PDFs open, and set Content-Disposition: attachment when you want a genuine download rather than inline display. Getting these wrong is a subtle bug — a PDF that downloads as a raw byte stream, or an SVG served as HTML that becomes a stored-XSS vector. Pin content types explicitly at write time rather than trusting whatever the client claimed.

Access logging and audit trails

For sensitive files — invoices, contracts, personal data — you often must prove who downloaded what and when. Object stores can emit access logs of every request, but those are coarse; for real audit you record the access in your own database at the moment you issue a signed URL, capturing the user, the object, the time, and the reason. That gives you a queryable trail tied to your identities rather than raw IPs, which is what compliance regimes and incident investigations actually need. Design the audit record when you build the download path, because reconstructing "who accessed this" after the fact from storage logs alone is painful and often impossible.

The cost model: storage, requests, and egress

Object storage bills on three axes and teams routinely misjudge which one hurts. Storage per gigabyte is usually the smallest line. Request charges (per GET/PUT) add up for workloads with millions of tiny operations. And egress — bytes served out to the internet — is where the surprise bills come from, often dwarfing storage itself. The levers are straightforward once you see them: a CDN with a high cache-hit rate slashes egress by serving from the edge, lifecycle policies move cold data to cheaper tiers, and batching or caching cuts request counts. Model these three before you scale, because a media-heavy product's storage bill is mostly egress, and egress is the one you control with caching rather than cleanup.

Performance: the hidden network calls

The Storage API makes remote objects look like local files, which hides where the network round trips are. Knowing which operations are free and which cost a request to S3 is the difference between a file list that renders in 30ms and one that takes three seconds.

The classic incident is a template that displays {{ doc.file.size|filesizeformat }} for each row of a 50-item list: fifty sequential HEAD requests, each 20–80ms, on every page view. It behaves fine in development against local disk and falls over in production. The fix is to record metadata when the file is written — as the confirm view above does — and to populate width_field and height_field on image fields so dimensions come from the database.

boto3 clients are also not free to create: building one loads service models and resolves credentials. Create a client once per process (a module-level function decorated with functools.cache is enough) rather than in every view call, and raise the connection pool size via botocore.config.Config(max_pool_connections=...) if you run many threads per process. Clients are thread-safe; boto3.resource objects and sessions are not, so do not share those across threads.

Troubleshooting S3 errors

Most storage incidents come down to a small set of error codes whose messages are unhelpfully generic. When something fails, get the actual error code from the exception (err.response["Error"]["Code"] on a botocore.exceptions.ClientError) or from the XML body the browser receives, then work through this table:

Two habits shorten these investigations considerably. Enable botocore's debug logging temporarily (boto3.set_stream_logger("botocore", logging.DEBUG)) in a shell on the affected host to see the exact request and signing details — and turn it off again, because it logs headers you do not want in log aggregation. And reproduce with the AWS CLI using the same role (aws s3api head-object --bucket ... --key ...): if the CLI fails too, the problem is permissions or configuration, not your Django code.

Security and cost pitfalls

Lock the bucket down: block all public access by default and grant it back only where genuinely needed. Validate uploads — check content type and size server-side, never trust the client, and for user-supplied files consider malware scanning before they are served to others. Watch egress cost, which is where object-storage bills surprise teams; a CDN with good cache hit rates is the main defense. Finally, remember collectstatic now uploads to S3, so give your deploy pipeline write access to the static bucket and expect the first deploy to take longer as it pushes every asset.