How to Scale Django Media Uploads with Presigned URLs

How Did We Discover the Django Media Upload Bottleneck?

During a recent project for a fast-growing retail SaaS platform, we encountered a severe architectural bottleneck. The system allowed merchants to create digital storefronts, which required uploading high-resolution product images and demonstration videos. Initially, the development team built the feature using standard Django form handling, piping the media directly through the backend server before passing it on to a third-party object storage service.

As the platform scaled, this approach began to buckle. During peak onboarding hours, the web server (running Gunicorn) exhausted its worker threads. Gigabyte-sized video uploads were holding HTTP connections open for minutes, blocking other critical API requests and causing widespread timeouts. We realized that routing heavy media through the HTTP server was an anti-pattern for this scale, but moving away from it introduced a new set of data consistency issues that we had to carefully engineer our way out of. This challenge inspired this deep dive into handling decoupled media uploads, so other technical leaders can avoid the same infrastructure pitfalls.

What Causes the Disconnect Between File Storage and Database Records?

In a standard monolithic workflow, a file upload and a database insert happen synchronously. If the upload to storage fails, the database transaction rolls back. But when you distribute this process to prevent HTTP bottlenecks, you break that atomic guarantee.

In our business use case, the data layer was structured similarly to this generic representation:

class Product(models.Model):
    name = models.CharField(max_length=255)
    unit_price = models.DecimalField(max_digits=10, decimal_places=2)
    video = models.URLField(blank=True, null=True)
class ProductImage(models.Model):
    product = models.ForeignKey(Product, on_delete=models.CASCADE)
    image_url = models.URLField()

When decoupling the upload process, the business logic requires placing the file into a third-party bucket and saving its URL to the Django database. The architectural conflict arises at the intersection of frontend orchestration and backend reliability: if the frontend successfully uploads a 500MB video to the media server, but the user closes their laptop before the final API call linking that video to the Product record fires, you are left with an “orphan file.” Over time, these orphaned files silently inflate cloud storage costs while serving no business purpose.

Why Do Traditional Upload Architectures Fail at Scale?

Before implementing our final architecture, we investigated the symptoms dragging down the platform. The primary failures stemmed from two distinct areas: thread starvation and lack of state tracking.

First, when handling direct orchestration through the backend, the synchronous nature of Django (even with async views, I/O bound tasks tie up resources) meant that incoming media streams consumed memory and network bandwidth disproportionate to standard JSON API requests. The infrastructure logs showed huge spikes in Gunicorn queue times.

Second, when a previous team attempted to offload this by moving to a client-side direct-to-storage upload pattern, they faced orchestration failure. They used a two-step process: client uploads the file, then client sends a POST request with the new storage URL to create the product. Our audits revealed that nearly 15% of all uploaded videos were completely orphaned. Network drops, browser crashes and strict ad-blockers occasionally interrupted the final POST request. The storage bucket became a black hole of unlinked media.

What Architectural Patterns Were Evaluated for Seamless Media Handling?

To eliminate the bottleneck without creating a data consistency nightmare, we evaluated several architectural approaches. When companies scale, they often choose to hire python developers for scalable data systems specifically to navigate tradeoffs like these.

Approach 1: Direct Backend Proxying (The Original Bottleneck)

We considered scaling up the backend infrastructure, allocating dedicated upload-only servers. The client POSTs the file and product data to Django, which saves it to temporary storage, queues a Celery task to upload it to the media server and returns a response. While this guarantees no orphaned files, it still forces heavy network traffic through the backend infrastructure, requiring massive load balancer and server scaling.

Approach 2: Frontend-First Upload with Cleanup (High Orphan Risk)

The client uploads directly to the third-party media server, receives a URL and then POSTs the product payload to Django. This prevents server bottlenecking. To handle orphans, we considered writing a script to crawl the entire storage bucket weekly and cross-reference every file against the Django database. We discarded this because crawling millions of objects across network boundaries is intensely slow and costly via API calls.

Approach 3: Pre-allocation with Webhooks (Complex Orchestration)

The client POSTs a product without media to Django. Django creates an incomplete record and returns a presigned upload URL. The client uploads the file directly to storage. Once uploaded, the storage provider (like AWS S3) fires a webhook back to Django to confirm the file exists, triggering Django to mark the product as “complete.” This approach is highly resilient but introduces complex edge cases around webhook retries and temporary “pending” states in the UI.

How Did We Architect a Scalable Direct-to-Storage Upload Pipeline in Django?

We selected a hybrid strategy: Presigned URLs with Prefix-Based Lifecycle Policies and Eventual Consistency Cleanup. This approach entirely bypasses the HTTP server for heavy lifting while automating the deletion of orphan files without requiring expensive bucket crawls.

Step 1: The Intent to Upload

The frontend requests an upload URL from Django. Django generates a secure, time-limited presigned URL via the storage provider’s SDK. Crucially, Django enforces a specific storage path prefixed with tmp/.

import boto3
import uuid
from django.conf import settings
def generate_presigned_upload_url(file_extension):
    s3_client = boto3.client('s3')
    file_name = f"tmp/{uuid.uuid4()}.{file_extension}"
    
    url = s3_client.generate_presigned_url(
        'put_object',
        Params={
            'Bucket': settings.MEDIA_BUCKET_NAME,
            'Key': file_name,
            'ContentType': f"video/{file_extension}"
        },
        ExpiresIn=3600
    )
    return url, file_name

Step 2: Direct Client Upload

The frontend application takes this URL and uploads the massive video file directly to the storage bucket. The Django backend remains completely unaware and unburdened during this phase. If you plan to hire app developer to create a mobile app later, this exact same endpoint can be consumed by iOS/Android clients natively.

Step 3: Database Finalization and Object Promotion

Once the client finishes the upload, it sends a lightweight POST request (or PATCH, if the product exists) to Django containing the tmp/ file path. Django validates the request, updates the database and uses a lightweight backend SDK call to move (copy and delete) the object from tmp/ to its permanent directory (e.g., products/).

Step 4: The Automated Orphan Fix

Because unlinked files are always trapped in the tmp/ prefix, we didn’t need to write complex database reconciliation scripts. We simply configured a Cloud Storage Lifecycle Rule on the bucket to automatically delete any object in the tmp/ folder that is older than 24 hours. If the user closes their browser at Step 2, the file sits in tmp/ and is quietly purged by the cloud provider the next day at zero compute cost to our Django servers.

What Essential Lessons Can Engineering Teams Extract from This Challenge?

When re-architecting systems for high-throughput I/O, there are several foundational rules that engineering teams should adopt:

• Never block web workers with file transfers: Keep HTTP requests lightweight. Delegate file transfers to specialized infrastructure (CDNs, Object Storage) using presigned URLs.
• Embrace eventual consistency: Accept that distributed operations might fail midway. Design states (like our tmp/ prefix strategy) that expect and handle failure gracefully.
• Leverage cloud-native lifecycle rules: Do not write custom cron jobs to delete temporary files if your storage provider has built-in lifecycle management. It reduces codebase complexity and compute costs.
• Design for multi-client consumption: Offloading orchestration to the storage layer makes it easier for diverse clients to interface with your system.
• Assume frontend drop-offs: Users will close tabs, networks will drop and tokens will expire. Your backend must inherently clean up dangling states without relying on the client to send a “cancel” API call.

How Can We Summarize This Architecture Modernization?

By shifting media uploads away from the Django HTTP server and utilizing presigned URLs combined with prefix-based storage lifecycle rules, we successfully handled a massive increase in media volume. We completely eliminated Gunicorn worker timeouts and eradicated the silent accumulation of orphan files. Technical leaders prioritizing system stability must ensure their architecture cleanly separates data serialization from heavy file I/O.

If you are looking to scale your engineering capabilities or need to hire software developer talent capable of navigating distributed system challenges, contact us to discuss how our dedicated remote engineering teams can support your next platform modernization.

Social Hashtags

#Django #Python #DjangoDevelopment #PythonDevelopment #WebDevelopment #SoftwareArchitecture #SystemDesign #AWS #AmazonS3 #CloudComputing #BackendDevelopment #Scalability #PresignedURLs #DevOps #SoftwareEngineering