Handling Large Files in Message-Driven Apps: The Claim Check Pattern

A customer uploads a large spreadsheet, and a background worker needs to import its rows. Putting the complete file inside a queue message looks convenient until the file exceeds a broker limit, makes every retry expensive or fills worker memory before processing begins.

The claim check pattern separates the file from the message. The file goes into storage designed for large objects, while the message carries a small reference and enough information for an authorised worker to retrieve the correct file.

That separation solves the oversized-message problem, but introduces a lifecycle to manage. The file must exist before work is advertised, remain available through retries and recovery, and be deleted only when the application's obligations allow it.

Introduction

We will use a product-import feature for a small shop. An administrator uploads a spreadsheet, the application stores it and a worker validates and imports its contents asynchronously.

The administrator does not need to keep a browser request open until every row has been processed. They need an accepted job identifier, visible progress and a final result that distinguishes successful rows from errors according to the import contract.

We will follow the bytes and the message separately, explain the failure windows between storage and publishing, and design retrieval, duplicate protection and cleanup. The useful result is a small message whose reference stays meaningful for the entire life of the job.

Understand the Basic Separation

A claim check is a reference to a payload stored elsewhere. The name comes from receiving a ticket for an item held in storage and later presenting that ticket to retrieve the item.

Microsoft's claim check pattern guidance describes storing a large payload externally and sending only its reference through the messaging system.

For the product import, the object store holds the spreadsheet bytes. The queue carries the import identifier, object reference, expected format and other small metadata needed to route and validate the job.

The worker receives the message, retrieves the object and performs the import. Intermediaries that only route the message do not need to copy or inspect the spreadsheet.

A reference is not automatically permission. The worker still needs authorisation to retrieve that object, and the application must verify that the object belongs to the intended job and customer scope.

Know Why Large Messages Are Awkward

Brokers have message-size limits that vary by product, tier and protocol. Large messages can also consume storage, network bandwidth and memory even when they fit within the formal limit.

Retries and multiple subscribers multiply those costs. A large payload may be copied through several delivery paths even when only one consumer needs the complete file.

Serialising binary data into a text envelope can add further overhead. The message's encoded size, headers and broker accounting may differ from the original file size on disk.

Raising a broker limit does not address every operational consequence. A worker may still buffer the entire message before it can reject an invalid job or wait for a processing slot.

Use claim check when payload size or repeated transfer makes it worthwhile. Small ordinary commands can remain inline; adding an object upload and download for every tiny message introduces latency and complexity without a clear benefit.

Choose Storage for the Required Lifetime

Object storage, such as a blob or object store, suits immutable files that workers retrieve by identifier. The application does not need to put the spreadsheet into a database row simply because its job record lives in a database.

Choose storage with availability, durability and access controls appropriate to the accepted job. A temporary local file on the uploading API instance is not enough if another worker on another machine must retrieve it after the API restarts.

Record where the object is stored without hard-coding credentials into the reference. An internal store identifier plus an object key can let the worker select a configured storage client.

Consider the whole recovery window, including queue delay, processing retries, dead-letter investigation and any permitted replay. Storage intended to expire after a few minutes may be unsuitable for a job that can remain unresolved for days.

Separate input retention from output retention. An imported spreadsheet, its validation report and the resulting product data can have different purposes and deletion rules.

Design a Small, Explicit Message

A message should identify both the logical job and the exact payload. The following illustrative envelope keeps those responsibilities visible:

{
  "messageType": "ProductImportRequested",
  "schemaVersion": 1,
  "operationId": "import-7421",
  "tenantId": "shop-17",
  "payload": {
    "store": "imports",
    "objectKey": "shop-17/import-7421/input-01.xlsx",
    "objectVersion": "recorded-storage-version",
    "lengthBytes": 12582912,
    "contentType": "application/vnd.openxmlformats-officedocument.spreadsheetml.sheet"
  }
}

These names are application choices, not a universal broker schema. A checksum can also be included with its algorithm when the producer has calculated a meaningful value for the exact stored bytes.

Keep the operation identifier stable across delivery attempts. Broker message identifiers can change during replay, while import-7421 still represents the same accepted import.

The object version is meaningful only if the selected store provides that versioning contract. Otherwise use an application policy that assigns a unique key and prevents overwriting the accepted input.

Store the Complete Object before Advertising Work

The simplest ordering is to upload the object, verify successful completion, then submit the reference message. Publishing first can allow a fast worker to request a file that does not yet exist.

An upload request starting successfully is not the same as an upload finishing. Use the storage client's completion contract, and record the final object identity and relevant metadata.

Large uploads may use multiple parts. In Amazon S3, the multipart upload lifecycle distinguishes uploading individual parts from completing the object. A message should refer to the completed object, not an unfinished collection of parts.

If a direct browser upload is used, the server should verify the object against the intended upload session before accepting the import. Do not trust a client message saying “upload complete” as the sole evidence of existence, expected size and ownership.

Once accepted, return the durable import identifier and status location. This makes the asynchronous responsibility clear without implying that storing the input file means its product rows have already been imported.

Handle Upload Success Followed by Publish Failure

Uploading and publishing are separate operations. The file can exist even when the broker is unavailable or the producer crashes before submitting the message.

That leaves an orphan candidate: an object with no confirmed processing request. Deleting it immediately may be unsafe if a later recovery process is still expected to publish the job.

Persist a job record that describes the upload and its intended processing state. A dispatcher can find accepted jobs that still need publication and retry with the same logical operation identity.

A transactional outbox can record the job and message intent together in the database transaction. It does not make the earlier object-store upload part of that database transaction, so abandoned uploads still require cleanup.

Track upload sessions and accepted jobs separately enough to distinguish an unfinished user upload from a committed import awaiting dispatch. The recovery action differs: one may eventually expire, while the other still represents promised business work.

Handle an Uncertain Publish Response

The producer may submit the message successfully but lose the broker's response. Retrying can create another delivery of the same import request.

Do not upload a new unrelated file and assign a new operation identifier merely because the publish response was lost. Reuse the accepted job identity and the immutable payload reference.

The worker should recognise whether that operation is already complete or currently owned by a valid processing attempt. The exact coordination depends on the job store, but a transport retry must not automatically become a second import.

Database uniqueness constraints and conditional state changes can help enforce the intended transition. A simple in-memory list of recently seen messages does not survive restarts or coordinate multiple workers.

Keep original input and processing attempts distinct. Several attempts may legitimately inspect the same file during recovery, while only one protected final import outcome should be recorded for the logical operation.

Make the Payload Stable across Retries

A reference to latest.xlsx can retrieve different bytes on different attempts if another upload overwrites the same key. That makes a retry a different operation in disguise.

Assign a unique object key for each accepted input, or retrieve a specific storage version. Azure Blob Storage's versioning overview explains identifying and reading earlier versions when versioning is enabled and supported.

Versioning alone is not a retention policy. A lifecycle rule can still remove versions that a replay requires, and the worker needs permission to retrieve the referenced version.

If an administrator corrects the spreadsheet, create a new input revision or replacement job with an explicit relationship to the earlier one. Preserve which input produced each import result.

This makes failures reproducible. A developer investigating a rejected row can inspect the same input the worker processed, rather than a newer file that happens to occupy the same name.

Validate the Reference before Retrieval

Treat message fields as data to validate, even when messages usually originate inside the application. A misconfigured producer or compromised credential can submit references outside the intended scope.

Resolve the store name through trusted configuration and verify that the key and version belong to the recorded job. Check tenant ownership against authoritative job state rather than relying only on a tenant identifier supplied in the message.

Avoid making unrestricted HTTP requests to arbitrary URLs from message payloads. Such behaviour can let a message direct a worker towards unintended internal services or external destinations.

If the contract supports URLs, constrain the allowed destinations and retrieval behaviour carefully, including redirects. Prefer a storage-specific reference and client when the workflow only needs a known object store.

Validate before downloading large bytes. Rejecting an invalid job after loading a huge payload consumes resources unnecessarily and can expose data to a worker that should never have retrieved it.

Keep Access Credentials Separate from Durable References

For internal workers, a workload identity with narrowly scoped storage permissions can retrieve the object using its stable key. The durable message need not contain a secret or a long-lived bearer token.

Microsoft describes identity-based access in its Azure Blob Storage authorisation guidance. The identity needs the appropriate data permissions, not merely the ability to view a resource in a management interface.

Presigned URLs can provide time-limited access when that model fits. However, a URL created during upload may expire before a delayed worker or replay reaches the file.

Amazon S3's presigned URL documentation explains that validity also depends on the signing credentials. Temporary credentials can expire before a longer requested URL lifetime.

When durable processing outlives a short access grant, retain a stable authorised reference and obtain suitable fresh access at retrieval time. Do not solve the mismatch by casually embedding permanent credentials in queue messages and logs.

Check That the Bytes Match the Intended Input

Verify the retrieved object's expected version, size and format before treating it as valid input. A storage key resolving successfully does not establish that the contents match the accepted upload.

A checksum is a value calculated from the bytes using a defined algorithm. Comparing the expected and retrieved values can detect accidental changes or corruption when the checksum contract covers the exact object representation.

Amazon S3 documents supported integrity mechanisms in its object integrity guidance. Do not assume every ETag is a simple hash of the complete file; multipart and encryption behaviour can affect its meaning.

A matching checksum does not prove the spreadsheet is safe or semantically correct. Validate its file structure, allowed size, sheet layout and row rules through the normal import pipeline.

Keep errors actionable. “Input checksum mismatch” calls for investigating storage identity or transfer, while “row 42 has an invalid price” belongs to content validation and can be explained in the import report.

Stream Large Inputs and Bound Processing

Moving the file out of the broker does not remove its memory cost if every worker downloads the whole file into a byte array. Several large concurrent imports can still exhaust the process.

Stream input where the parser and format support it, or use bounded temporary storage when random access is required. Spreadsheet libraries differ in whether they retain the full workbook, so check actual behaviour rather than assuming a streaming download guarantees streaming parsing.

Limit simultaneous downloads and imports according to memory, disk and downstream database capacity. A small reference message can represent a very large amount of work, so message count alone is a poor estimate of resource demand.

Apply limits to expanded content too. A compressed file can require substantially more storage or processing than its transferred size suggests.

Keep temporary files owned by a specific attempt and clean them after completion or through a safe recovery process. Do not let a worker mistake a partial local download from a previous crash for a complete verified input.

Separate File Retrieval from Import Completion

Downloading the spreadsheet is only the first stage. The worker still needs to validate rows, apply the chosen import contract and record the result durably.

Decide whether the import is all-or-nothing or permits partial success. An all-or-nothing import may validate into staging before committing the intended changes. A partial import needs clear row outcomes and safe restart behaviour.

Avoid acknowledging the queue message merely because the download succeeded. If the worker crashes before recording the accepted business result, the job still needs to be recoverable.

For long jobs, store checkpoints at meaningful boundaries. A replacement worker can then resume or restart safely without assuming that the presence of an output file proves completion.

Publish result references only after the output is complete and validated. An attempt-specific report file can become the official result through a controlled job-state transition, preventing incomplete output from appearing as a finished import report.

Keep Files through Retry and Dead-Letter Recovery

A failed message may be delivered again or moved to a dead-letter queue for investigation. Its payload must remain available for the recovery policy the application promises.

Deleting the object immediately after one handler downloads it can break a later retry if that handler fails during validation or database processing.

Deleting immediately after business completion can also surprise a duplicate delivery if the consumer fetches the file before checking durable completion state. Check the operation state first and define how completed duplicates are settled without unnecessary retrieval.

If several subscribers need the same file, one consumer finishing does not mean all consumers are done. Track the required consumers or use a retention rule that covers their supported processing and recovery windows.

Retain referenced objects while a job is unresolved, unless an explicit policy terminates that obligation. A storage lifecycle rule should not silently overrule a still-valid recovery contract.

Design Cleanup as a Controlled Workflow

Cleanup needs to distinguish incomplete uploads, abandoned upload sessions, accepted jobs, completed jobs and unresolved failures. A blanket age-based delete over the entire storage prefix cannot make those distinctions.

An abandoned upload can become eligible after its session expires and no accepted job refers to it. A completed input can become eligible after the required retention and replay windows end.

Protect cleanup decisions against concurrent state changes. A job being accepted or placed under investigation must not race with a cleaner that has already decided its object is unused.

One approach uses durable state transitions that mark an upload expired or an object eligible for deletion, with acceptance and cleanup checking the same authoritative state. The exact transaction covers the job database state; deleting the storage object remains a separate operation that can require retry.

Make cleanup idempotent. Repeating a confirmed deletion should be harmless, while an unexpected missing object for an active job should produce an investigation signal rather than being treated as ordinary housekeeping.

Observe the Message and File as One Job

Use the import identifier to connect upload logs, storage metadata, message deliveries, processing attempts and the final result. Operators should be able to follow one accepted import across those stages.

Measure time waiting for dispatch, queue age, download duration, validation duration, import duration and unresolved failures. A slow import can be stuck before it reaches any row-processing code.

Track object counts and bytes by lifecycle state. A growing set of completed inputs may indicate broken cleanup, while growing uploaded-but-unaccepted objects may reflect abandoned sessions or an upload completion bug.

Monitor missing payloads and access failures separately. A permission change and a deleted file can both prevent retrieval, but their recovery actions differ.

Keep progress honest. “Uploaded”, “queued”, “processing” and “completed” are distinct states, and each should reflect durable evidence rather than a hopeful assumption about the next stage.

Walk through an Import across Two Failures

An administrator uploads the spreadsheet for import-7421. The server verifies the completed object and records its exact reference. It then commits the accepted job and its dispatch intent together in the job database.

The broker is temporarily unavailable. The browser still receives the accepted job identifier because the application has durably taken responsibility through its job record and supported recovery process. The status shows waiting for processing; it does not claim the worker has started.

When the broker recovers, the dispatcher sends the message. Its first confirmation is lost, so it sends again. Both deliveries carry the same import identifier and input reference rather than creating two unrelated jobs.

A worker claims the job through the job store's controlled state transition, validates the reference and retrieves the exact object version. The other delivery follows the duplicate-handling policy instead of starting an independent import over the same products.

Suppose the active worker crashes after validating the file but before committing the product changes. The message and job recovery design makes the work available again. A replacement worker can inspect durable state and resume or restart according to the import contract.

The original file is still present because cleanup has not marked an unresolved accepted job eligible for deletion. A short-lived URL would not have been sufficient here; the replacement worker retrieves the stable reference using its current authorised access.

After the import result is committed, the worker completes its delivery. Any later duplicate sees the completed job before downloading and can return the recorded outcome. Only after the defined retention window does cleanup remove the input, while preserving the required result and audit information.

Each failure has a specific recovery path. The message does not need to contain the spreadsheet, but the system must preserve the relationship between job, input and final outcome throughout those failures.

Decide How to Handle a Missing Payload

A missing object should be interpreted in the context of the job. An already completed job receiving a late duplicate may no longer need its original input. An active job whose input disappeared has a different and more serious problem.

Check whether the reference points to the expected store, key and version. A deployment configuration error can make a valid reference resolve against the wrong account or environment.

Distinguish an access denial from confirmed absence. Some storage interfaces deliberately avoid revealing whether an inaccessible object exists, so the diagnostic process may need an authorised storage check rather than guessing from one response.

If an accepted input was deleted prematurely, recover its exact version from an available recovery mechanism when permitted and possible. Do not substitute a similarly named newer file, because it may change the meaning of the import.

If recovery is impossible, record the job as requiring a new input or another explicit business decision. Repeatedly retrying a permanently missing payload creates load without restoring the original obligation.

Use the incident to correct the underlying lifecycle rule. A DLQ can preserve the reference and evidence, but cannot recreate bytes that no longer exist in any retained copy.

Test the Boundaries between Systems

Test an upload that fails halfway through. No accepted import message should refer to the unfinished object, and incomplete upload resources should eventually follow their cleanup policy.

Then test a completed upload followed by a crash before job acceptance. The object should be recognised as an abandoned candidate only after the upload session and acceptance rules make that decision safe.

Interrupt the flow after job acceptance but before publish, and verify the dispatcher eventually submits the same job. Interrupt after publish but before confirmation, and verify duplicate delivery does not create duplicate product changes.

Attempt to process a reference belonging to a different tenant or an unexpected store. The worker should reject it before retrieving the bytes. Also test a valid reference whose content exceeds allowed processing limits despite a small message envelope.

Run cleanup while a supported retry or investigation remains active. The input must remain accessible until the lifecycle policy permits removal. This test is especially useful when storage lifecycle rules are configured separately from application code.

Finally, test a replay after an access token has expired but while the retained input is still valid. A design using stable references and current worker identity should distinguish expired access credentials from an expired business obligation.

Account for the Extra Storage Work

Claim check moves costs rather than making bytes disappear. The workflow adds object writes, reads, retained storage and possibly transfer between regions or services.

Repeated downloads can matter when a large input is replayed many times. Investigate permanent validation failures promptly and avoid downloading completed duplicates when the job record already supplies the answer.

Place workers and storage with the intended access pattern in mind. A design that repeatedly moves a large file across distant regions may add latency and transfer cost without improving the import outcome.

Measure file size, retrieval count and retained bytes alongside message counts. These measurements show whether the chosen threshold for using claim check remains sensible as the workload changes, and whether failed cleanup or repeated processing is creating avoidable expense.

Keep the input contract small enough for another developer to explain: which job owns the object, how the exact bytes are identified, who may read them, how long they remain available and what state allows deletion. Those answers should agree across the API, dispatcher, worker and cleaner. Reviewing that contract together catches mismatches that each component's successful unit tests can miss when considered in isolation.

Summary

The claim check pattern keeps large files in object storage and sends small references through the broker. It reduces oversized-message pressure while allowing authorised workers to retrieve only the payloads they need.

Store and verify the complete input before advertising work, preserve a stable operation and payload identity, and handle the failure window between upload and publishing. Validate access and bytes, bound processing resources and record the import outcome durably.

Retention is part of correctness: files must remain available through the promised retries and recovery, then be cleaned up through a policy that understands job state. The reference is useful only while it reliably identifies the right, accessible input for the original operation.