A photo-sharing application has two different storage problems. It needs to answer questions such as who owns an image and whether it is ready to display. It also needs to move large files reliably between users, processing workers and storage.
Putting both workloads through the same database or application server creates an avoidable bottleneck. As the library grows, upload recovery, media processing, delivery and deletion become just as important as the number of bytes the storage service can hold.
This article follows a media item through its complete lifecycle, from the first upload request to playback, access changes and eventual removal.
Introduction
The instruction to store files in object storage is a useful starting point, but it is not a complete media architecture. A production system must know which uploaded bytes were verified, which transformations belong to them and which users may retrieve the resulting files.
For our example, users can upload photographs and videos, attach them to posts and retrieve suitable renditions on different devices. Uploads may be interrupted. Processing may take time. Popular content can receive sudden traffic, while private content needs access decisions that remain meaningful after a delivery URL has been issued.
We will separate the control path from the transfer path. The application handles identity, quotas, upload sessions, metadata and lifecycle decisions. Object storage handles the large byte transfers, and background workers generate the versions that readers actually need.
This separation allows each workload to scale independently. It also creates boundaries where failures must be reconciled: a file can exist without a completed database update, and a database record can refer to processing work that has not yet finished.
Define the Workflows and Service Guarantees
The main workflows are creating an upload session, transferring bytes, verifying the uploaded object, processing it, serving a rendition and deleting the media item. Each stage should have a visible outcome rather than being hidden behind one generic upload status.
A successful upload response should mean something precise. It could mean that the bytes reached temporary storage, that they passed validation or that a displayable rendition is ready. These milestones are different, particularly for a video that needs several minutes of processing.
Define the smallest usable result. A photograph may be ready when its display image and thumbnail exist. A video may become playable once a baseline rendition and its manifest are available, even while higher resolutions are still being generated. The interface should describe that partial readiness honestly.
Availability requirements also differ. A short delay in generating a larger thumbnail may be acceptable. Returning another user's private original is not. The architecture should distinguish presentation quality from access control and durability.
Set practical limits for file size, duration, dimensions, upload concurrency and per-account storage. Without limits, one unusual input can monopolise bandwidth or processing memory even when ordinary traffic is modest.
Estimate Storage, Processing and Delivery Separately
At an illustrative two megabytes per original, one billion photographs require roughly two petabytes in decimal units before derivatives and backups. If several display sizes together add another megabyte per photo, the total grows by another petabyte.
Video changes the calculation. A five-minute video encoded at an average of four megabits per second represents roughly 150 megabytes for that rendition alone. Keeping several renditions, the source file and preview assets can multiply the stored bytes.
Stored capacity is only one part of the workload. Upload traffic is driven by newly created media. Processing capacity depends on the formats, durations and transformations requested. Delivery traffic depends on popularity and how much of each object users actually view.
For example, a small set of popular videos can dominate outgoing bandwidth while most stored photographs are rarely requested. A storage optimisation that saves a few percent of cold capacity may matter less than avoiding repeated delivery of unnecessarily large renditions.
Use these estimates to identify the likely constraint, then validate them with the product's actual distribution. Do not plan around an average file that hides a long tail of very large videos or high-resolution images.
Separate Media Files from Their Metadata
Store the bytes in object storage. Keep the records needed for application decisions in a database: media ID, owner, upload session, object location, verified object version, detected type, dimensions, duration, lifecycle state and visibility.
The object key identifies a stored object. The database record explains what that object represents. A random-looking key is useful for uniqueness but does not replace authorisation; knowing a key must not automatically grant access to private media.
Use application-generated identifiers rather than original filenames as storage identities. Two users can upload a file called holiday.jpg. A retry or replacement should not overwrite an unrelated object merely because its filename happens to match.
Keep the original filename as display metadata if the product needs it, and handle it as untrusted input. It should not become an unrestricted filesystem path, processing command argument or response header value.
A separate rendition table can describe generated outputs:
media_id
source_version
transformation_version
rendition_name
object_key
content_type
width, height, duration
byte_size
status
This avoids overloading one record with assumptions that every image and video has the same outputs. It also lets a later transformation version coexist with the currently published version until its replacement is ready.
Design an Explicit Media State Model
A practical lifecycle might include these states:
CREATED -> UPLOADING -> VERIFYING -> PROCESSING -> READY
| | |
v v v
EXPIRED REJECTED FAILED
READY -> DELETING -> DELETED
The state is not a substitute for detailed job records. A media item can be processing while one rendition is complete and another has failed. Preserve that detail so a retry can resume missing work instead of discarding successful outputs.
Transitions need guards. A worker finishing an old transformation must not change a media item from DELETING back to READY. A completion request must refer to the correct owner, upload session and verified object version.
Use a version check or equivalent transactional condition around important state changes. If two workers compete, one should observe that its expected state is no longer current and stop or reload deliberately.
Retain enough history to explain failures. A support view that shows only FAILED cannot distinguish an interrupted transfer, rejected format, corrupt input or unavailable processing service. Those failures have different recovery actions.
Create a Controlled Direct-Upload Session
The client begins by asking the application for an upload session. The application authenticates the account, checks its quota and allocates a specific temporary object key. It records the expected size and declared type while treating both as claims that require later verification.
The response can contain a temporary upload capability such as an Amazon S3 presigned URL. This lets the browser send bytes directly to storage while the application remains responsible for deciding what can be uploaded.
Limit the destination, operation and lifetime of the capability. Avoid giving the client a broad ability to choose arbitrary keys. Enforce size or request restrictions through mechanisms supported by the storage provider and upload method, then verify the final object independently.
A presigned URL is a bearer capability, not proof of the current user's identity on each request. Anyone who receives it may be able to use it within its allowed lifetime. Avoid placing it in analytics events, ordinary logs or pages where unrelated scripts can collect it unnecessarily.
Direct transfer also needs the correct browser cross-origin configuration. Allow the required application origins and methods without turning the storage bucket into a generally writable public endpoint. Upload permissions and read permissions should remain separate.
Verify the Exact Bytes Before Publication
The client's upload-complete notification is useful, but it is not authoritative. The application should inspect storage metadata and verify that the expected object exists, meets the agreed constraints and belongs to the allocated upload session.
Check the actual file format rather than trusting an extension or declared content type. Processing a file based only on the name image.jpg can send unexpected input into a decoder. Validate dimensions and duration as well as byte size, because a compact encoded input can expand substantially in memory.
There is also a time-of-check problem. An upload capability may remain reusable after the first transfer. If the application verifies an object and then a client replaces it, a worker could process bytes that were never validated.
Pin processing to an immutable object version, or promote verified content to a separate key that the client cannot overwrite. Store the chosen identity in the processing job. The worker should load that exact source, not whatever currently happens to exist under a reusable temporary key.
Integrity checksums can detect damaged transfers when the client and storage workflow support them. Do not assume every object identifier or ETag is a simple checksum of the complete file. Use the provider's documented integrity mechanism and preserve the expected value with the upload record.
Make Large Uploads Recoverable
Restarting a large video upload from zero after a brief connection failure wastes time and bandwidth. Multipart upload divides the object into independently transferable parts, allowing the client to retry a failed part without resending successful ones.
The upload session should retain its provider upload ID, part identifiers and completion information. A resumed client needs to know which parts belong to this upload and which have been acknowledged. The S3 multipart upload guide describes its initiation, part transfer and completion lifecycle.
Choose part size and concurrency together. Very small parts increase request overhead; very large parts make retries more expensive. Too many simultaneous transfers can compete for a mobile device's limited bandwidth rather than improve throughput.
Completion must be idempotent at the application level. If storage assembles the object but the response is lost, the client should query or retry the existing session rather than create a second logical media item. Reconcile the provider's state before declaring that the upload failed.
Abandoned uploads need cleanup. Give sessions a product-level expiry and arrange removal of incomplete parts after the recovery window. Completing a database cleanup alone does not necessarily stop the provider charging for partially uploaded data.
Process Media Outside the Request Path
Once an upload is verified, persist a processing instruction and enqueue work. The interactive request should not remain open while a worker transcodes a long video or generates a large set of image variants.
If the database state and processing intent must advance together, write an outbox record in the same transaction. A relay can then publish the job reliably. Storage notifications may also trigger work, but they should be correlated with an authorised upload record rather than treated as permission to publish arbitrary objects.
Give every job a stable identity based on media ID, verified source version and transformation version. A duplicate delivery should refer to the same intended output. The worker can reuse completed outputs or replace its own incomplete attempt without creating another media item.
Use separate capacity pools where workloads differ materially. Short image transformations should not sit behind hours of video encoding if the product promises quick photo publication. Within each pool, bound memory, runtime and concurrent operations.
Treat decoders as processors of untrusted input. Run them with restricted access and resource limits, keep them updated and record failures without exposing sensitive source data. A queue isolates timing; it does not by itself isolate security or resource consumption.
Build Renditions Around Reader Needs
Images
A gallery usually needs a small thumbnail, while a detail page needs a larger image. Sending the original for every display wastes bandwidth and may increase page latency without improving visible quality.
Define a small set of supported dimensions and formats. Generating every possible width on demand can create an unbounded transformation workload, especially when arbitrary URL parameters select the requested size. Normalise requests to approved variants or use a carefully controlled dynamic transformation service.
Consider orientation, colour handling, transparency and metadata. Stripping location metadata may be appropriate for a public photo product, but the policy should be explicit. The transformed file should be tested for the way clients actually display it.
Video
Prepare a set of renditions suitable for different bandwidth and device capabilities. A streaming manifest allows the player to select media segments and adapt quality as conditions change. The rendition ladder should reflect real source quality rather than generating large files that add no useful detail.
Store the manifest and segments as one versioned output set. Publishing a manifest before its referenced segments are available can produce intermittent playback failures even though the processing job reports partial success.
Decide what READY means for video. The product may permit a baseline quality first, with higher qualities added later through a new manifest version. That is a useful optimisation only when the player's behaviour and caching rules support the transition.
Publish Outputs Atomically at the Metadata Boundary
Generating several objects does not provide one atomic transaction across all of them. A worker may successfully upload thumbnails but fail before writing the final rendition record.
Write outputs under a versioned staging prefix, verify the required set and then update the database pointer to the completed version. Readers continue using the previous published version until the new set is ready.
For a newly uploaded item, the pointer becomes visible only after the minimum required outputs exist. For reprocessing, the old version can remain available while the new one is generated. This reduces the chance that a routine transformation change makes existing media temporarily disappear.
Use ownership checks when workers update progress. A slow worker from an older attempt must not overwrite the result of a newer transformation. Store the source and transformation versions in every completion condition.
Orphaned staging outputs can be collected later after checking that no active job or published pointer references them. Cleanup should be conservative enough to avoid racing a legitimate slow worker.
Deliver Through a CDN
A content delivery network caches media closer to readers and reduces repeated transfers from origin storage. The application returns references to suitable renditions, while the CDN handles the expensive byte delivery.
Use immutable, versioned paths for generated files. If a transformation changes, publish a new path instead of silently replacing bytes beneath a long-lived cache entry. This makes it clear which content a client or edge cache has received.
Set cache policy according to the object lifecycle. An immutable public thumbnail can have a long cache lifetime. A changing manifest or access-sensitive response may need a different policy. Avoid putting every response behind the same generic cache rule.
A popular cache miss can still overload the origin. Consider origin protection, request coalescing where supported and limits on expensive on-demand transformations. A CDN improves repeated delivery; it does not remove the first-request cost for every unique object.
Monitor origin fetches alongside CDN hit ratio. A high overall hit ratio can hide one region, rendition type or newly published collection generating a disproportionate origin workload.
Protect Private Content and Handle Revocation
For private media, the application authorises the viewer before issuing a short-lived delivery URL or cookie. The capability should cover only the resources needed for that access, such as a rendition or a bounded set of video segments.
Protect the origin against direct bypass. If an object remains publicly readable from storage, restricting only the CDN URL does not enforce the intended rule. CloudFront's private-content guidance describes the relationship between signed delivery access and protected origins.
Revocation has a timing problem. A previously issued capability may remain usable until it expires unless the chosen design supports an additional revocation check. Short lifetimes reduce that window but increase renewal traffic and can interrupt long playback sessions if handled poorly.
Define the requirement before choosing the mechanism. A private family photograph, a paid training video and a public post removed for moderation can have different acceptable revocation windows. The interface should not promise immediate disappearance if cached copies and active capabilities remain usable.
Encryption at rest protects stored data under the provider's access model, while transport encryption protects transfers. End-to-end encrypted media requires a different processing design because a server without decryption keys cannot freely generate readable thumbnails or transcode the content.
Make Deletion a Durable Workflow
Deleting a database record is not the same as removing the media. Originals, thumbnails, manifests, segments, temporary files and replicas may all have separate lifecycles.
First mark the item unavailable to normal readers and persist a deletion instruction. Workers then remove the owned objects and record progress. A repeated deletion request should resume the same operation rather than fail because one object has already been removed.
Stop or invalidate processing jobs before they can publish new outputs for the deleted item. Every processing completion should check that the current lifecycle still permits publication. Otherwise, a delayed worker can recreate media after the deletion service appears to have finished.
Track physical removal separately from access revocation. CDN invalidation, token expiry and storage deletion may complete at different times. Document the operational guarantee and monitor deletions that remain incomplete beyond the expected window.
Backups require a retention and restoration policy. Restoring an older database or object snapshot must not silently reactivate content deleted later. Preserve deletion history or replay the appropriate lifecycle events during recovery.
Reconcile Incomplete and Uncertain Work
Several failures cross system boundaries. The storage upload can succeed before the database records completion. A processing worker can upload all outputs before losing its acknowledgement. A deletion request can time out after the provider has already removed the object.
A reconciliation process looks for records stuck beyond their expected stage duration and checks the authoritative external state. It should know which action is safe to repeat and which uncertain outcome needs investigation.
For example, a VERIFYING item with a known object version can be checked again and re-enqueued using the same job identity. A PROCESSING item with all required outputs can have its metadata completed only after verifying that those outputs match the intended source and transformation version.
Use bounded scans and checkpoints. Reconciliation should not list billions of objects on every run or compete with ordinary traffic without limits. Record enough identifiers during the normal workflow to locate the relevant objects directly.
The goal is not to hide every failure from operators. It is to make unfinished work visible and repeatable so the system does not depend on someone manually guessing which files belong together.
Follow One Upload Through a Failure
Suppose a user begins uploading a two-gigabyte video from a laptop. The application creates media item 842 and an upload session bound to its temporary storage key. The browser transfers parts with bounded concurrency and retains the acknowledged part information.
The connection fails after most of the file has arrived. When the user reconnects, the client resumes the existing session rather than creating media item 843. It checks which work is still needed and retries the missing transfers. The original quota reservation and session expiry still apply.
Storage then completes the object, but the application's completion response is lost. Repeating the completion request identifies the same media item and provider upload. The application reconciles the completed object instead of telling the client to send the video again.
Verification records the exact source version and the pipeline creates a job for transformation version three. The worker successfully writes a baseline video, thumbnail and manifest, then crashes before updating the database pointer. Those output objects exist, but the item is not yet marked ready.
On retry, the worker checks the expected outputs for that source and transformation. It can reuse verified complete outputs and finish the missing metadata transition. It must not attach files generated for another source version just because their names resemble the expected paths.
Before a higher-quality rendition completes, the user deletes the video. The deletion transition makes the item unavailable and records removal work. The delayed worker's completion condition now fails because the media item no longer permits publication. Its newly written output becomes cleanup work instead of reviving the item.
This sequence contains several successful external actions whose acknowledgements were uncertain. Treating each timeout as a fresh upload or job would create duplicates and orphaned files. Stable identities and guarded transitions let recovery determine what already happened.
It also shows why a single upload status cannot explain the whole lifecycle. Support can distinguish transfer progress, verified source ownership, processing outputs and deletion state, while the user receives a clear description of the current stage.
Operate Cost and Capacity as Product Constraints
Measure stored bytes by originals, derivatives, staging uploads and retained versions. A growing staging category can reveal broken cleanup even while the user-visible library grows normally.
Track processing cost by duration, input type and transformation. An expensive format may need a quota or different scheduling policy. Increasing worker count without controlling admission can move the bottleneck into storage requests, memory or downstream delivery.
Delivery measurements should include bytes per view, rendition selection, cache misses and playback failures. A client that repeatedly downloads a full original for a small preview can create a larger cost problem than an inefficient database query.
Move cold content into cheaper storage only after considering retrieval time, retrieval charges and recovery requirements. A storage class suited to archival originals may be inappropriate for frequently requested thumbnails.
Use budgets and alerts that connect to the product. Operators should be able to explain whether a cost increase comes from more active users, a popular video, a transformation rollout or an accumulating backlog of abandoned uploads.
Test the Complete Lifecycle
Test an interrupted multipart upload, repeated completion requests, an object replaced after verification and a worker that crashes after uploading outputs. These cases exercise the boundaries that ordinary happy-path tests miss.
Test deletion while processing is active and permission removal during playback. Check that old workers cannot republish deleted content and that the observed revocation window matches the documented access policy.
Restore a representative sample from backup and verify both metadata and playable files. Replication helps with some infrastructure failures, but it does not automatically protect against every mistaken deletion or incorrect transformation.
Finally, test a burst of reads for newly published media. Observe origin traffic, processing queues and client behaviour when higher-quality renditions are still unavailable. The product should degrade in a controlled way rather than treating partial readiness as a mysterious broken upload.
Summary
A large media library requires a coordinated lifecycle, not simply a large storage bucket. Separate metadata from bytes, create controlled upload sessions and verify the exact source object before processing or publication.
Make uploads resumable, processing retry-safe and published renditions versioned. Use a CDN to deliver appropriate files efficiently, while keeping private access and origin protection consistent.
Deletion, reconciliation and recovery deserve the same attention as the first successful upload. When those workflows have durable state and clear ownership, the system can grow to billions of objects without losing the ability to explain what each media item is, who can access it and what should happen next.