Sending one email is straightforward. Sending order updates, security alerts and campaign messages across email, text and push channels introduces queues, provider limits and uncertain delivery outcomes.
A notification service preserves business intent without requiring every application to handle those details. Its most difficult failure is often deceptively ordinary: a provider may accept a message while the worker sending it times out before receiving confirmation.
Introduction
A notification service turns a business event into one or more messages for a recipient. It decides which channels apply, respects preferences, renders the appropriate content and records what happened during delivery.
The service is an asynchronous workflow rather than a thin wrapper around an email API. Requests may wait for scheduled times, providers may throttle sends and delivery events may arrive long after the original application transaction has completed.
For this design, imagine a platform sending transactional order updates, short-lived security codes and bulk announcements. Email, text and mobile push are required. The example volumes are hypothetical planning assumptions, not claims about a particular company's infrastructure.
We will follow a notification from durable acceptance through orchestration, delivery, callbacks and recovery. That path provides a useful interview structure because each boundary introduces a specific reliability question.
Clarify Requirements and Delivery Guarantees
Separate message classes
An order confirmation may need to leave the queue within seconds. A newsletter can be distributed over several hours. A verification code is useful only before its expiry, even if the infrastructure can eventually deliver it.
Define priority, latency target, expiry and permitted channels by notification type. Do not allow every calling service to label its messages urgent without control. If everything receives the highest priority, the label stops protecting important work.
Clarify scheduling and local time. A campaign intended for 09:00 in a user's time zone cannot be represented by one global send timestamp without additional audience logic. Quiet hours and daylight-saving transitions need an agreed interpretation.
Also identify whether channels are alternatives or cumulative. An order confirmation may intentionally use both email and push. A security alert may try push first and use another channel only under specified conditions. These are different workflows and should create different delivery plans.
Define success precisely
Accepted by our API means the service has durably recorded the request. Accepted by a provider means the provider has acknowledged responsibility under its own contract. Delivered means a channel-specific delivery signal has been received.
Opening or reading is a separate observation and may not be available or reliable. A mobile device can be offline. An email recipient's server can accept a message without the person ever seeing it.
For email, Amazon SES defines a delivery event in terms of acceptance by the recipient's mail server and provides separate event types for send, bounce, open and other outcomes. A status page should preserve those distinctions. Amazon SES event definitions.
Agree on duplicate tolerance
The service should avoid creating duplicate logical notifications when callers retry. It should also reduce duplicate provider sends. Those are separate boundaries: internal deduplication does not prove exactly-once delivery through an external provider.
Discuss the user impact. A repeated newsletter is annoying. A repeated payment receipt can cause support queries. Repeating a verification code with different content may confuse the user about which code remains valid.
State the desired guarantees and the remaining uncertainty early. A defensible design records ambiguous outcomes and explains recovery instead of asserting that a queue makes every message exactly once.
Estimate Delivery Work and Storage
Suppose the platform receives one million personal notification requests per day. If each creates an average of two channel jobs, that is two million delivery jobs before retries. Dividing by the number of seconds in a day gives only an average, not the peak capacity requirement.
A single campaign to 100,000 recipients over two channels adds 200,000 jobs. If the campaign must complete in ten minutes, that alone requires roughly 333 initial channel submissions per second. The actual target needs headroom for retries, provider throttling and ordinary transactional traffic.
Estimate the fan-out before choosing queue and worker capacity. One incoming campaign record can represent far more work than thousands of individual events. Acceptance rate is therefore an incomplete measure of system load.
Storage includes notification records, delivery records, individual attempts and provider callbacks. If a job averages two attempts, attempt history can exceed the number of jobs. Retention should reflect troubleshooting and business needs while avoiding indefinite storage of rendered personal content.
Provider capacity can dominate internal capacity. If the account can submit fewer messages per second than the required campaign rate, adding workers only creates more throttling. The design needs scheduling, another approved route or a revised completion target.
Define the API and Durable Data Model
An authenticated request might look like this:
POST /notifications
Idempotency-Key: order-874-confirmed-recipient-42
Content-Type: application/json
{
"source": "orders",
"eventId": "order-874-confirmed",
"recipientId": "42",
"type": "order_confirmation",
"templateVersion": 3,
"data": { "orderId": "874" },
"expiresAt": "2026-09-13T12:00:00Z"
}
Derive tenant identity from the authenticated caller rather than trusting an arbitrary payload field. Validate that the source is allowed to send this notification type and use the requested template.
Return an acceptance response only after durable storage succeeds. A response can contain the notification identifier and a status URL. It should not claim delivery merely because the request passed validation.
Separate intent, delivery and attempt records
A notification record represents business intent. It can hold tenant, source event identity, type, recipient or audience reference, schedule, expiry, template version and orchestration status.
A delivery record represents one recipient-channel job. It holds its stable delivery identifier, selected route, current workflow state, next attempt time and relevant provider identifiers.
An attempt record represents one provider interaction. It records when the attempt began, which provider was used, the request identity, response classification and whether the outcome is known. Several attempts can belong to one delivery without becoming several notifications.
A separate callback record stores provider event identity and verified metadata before processing. This supports deduplication and investigation without allowing a late callback to overwrite history invisibly.
Enforce uniqueness in storage
For recipient-specific requests, a logical identity can combine tenant, source, event identifier, recipient and notification type. Enforce that combination with a unique constraint or an equivalent conditional write.
A campaign has a parent identity for its audience. Expanded jobs should have a unique combination of notification, recipient and channel. Device-specific push jobs may need a further device identity if the policy deliberately targets several devices.
Intentional repeated reminders need distinct occurrence identifiers. Reusing the same identity for tomorrow's reminder would cause the service to suppress a legitimate new notification.
If a caller repeats an idempotency key with different content, return a conflict or another documented error. Silently accepting the new payload under the old notification identifier makes it impossible to know which message was intended.
Accept Business Events Without Losing Intent
A business service can call the notification API after committing an order, but that creates a gap. The order commit can succeed and the process can stop before sending the notification request.
A transactional outbox closes that local gap by storing the order change and an outbound notification intent in the same database transaction. A relay later publishes the intent or calls the notification service.
The relay can still repeat a publication. It may send successfully, lose the response and retry before recording completion. That is why durable acceptance needs idempotency even when the producer uses an outbox.
Keep the originating business event identifier throughout the workflow. It links the order, notification, delivery attempts and provider callbacks. A support investigation should not require guessing which email corresponds to which order.
Decide how much data travels in the event. Capturing the relevant order summary preserves the intended content if the order changes later. Fetching current state at send time reduces payload duplication but can produce a message describing a later state.
For an order confirmation, a bounded snapshot of the confirmed values may be appropriate. For a reminder saying an item is still awaiting action, checking current state immediately before sending may prevent an obsolete message. Choose according to the notification's meaning.
Separate Orchestration from Channel Delivery
The high-level path is business producer, durable acceptance, orchestration, channel queues, delivery workers, providers and callback processing.
Orchestration workers
An orchestration worker resolves recipients, evaluates notification rules and creates delivery jobs. It should not hold a database transaction open while calling providers or expanding a huge audience.
Persist orchestration progress and create jobs in bounded transactions. If a worker stops after inserting a batch but before advancing its checkpoint, the next worker can repeat the batch safely because job uniqueness prevents duplicate creation.
Represent suppressed notifications explicitly when that outcome matters operationally. A job not created because the user opted out is different from a job lost because orchestration failed.
Use a durable retry path for transient preference-store or template-store failures. Falling back to permissive defaults during an outage can send messages the user had disabled.
Channel queues and workers
Use separate queues or clearly isolated capacity for email, text and push. A slow email provider should not consume every worker needed for security push notifications.
Separate urgent and bulk work too. Reserve capacity for transactional traffic while giving bulk work a defined share, so a constant stream of urgent messages cannot starve a campaign indefinitely.
The exact mechanism might be weighted scheduling, separate worker pools or quotas per queue. The important point is that priority affects available processing capacity, not merely a field on a message that every worker ignores.
Workers claim a delivery with a bounded ownership period. A crashed worker's claim must become recoverable, while a long-running worker must not keep sending after ownership has been reassigned without an explicit safety mechanism.
Expand Campaign Audiences Safely
A campaign can reference a saved segment rather than contain every recipient in its acceptance request. Expansion workers read that segment in bounded pages and checkpoint their position.
Define whether membership is captured at campaign creation or evaluated as expansion proceeds. A snapshot provides a reproducible audience. A live segment may include later joiners or exclude people whose attributes changed. Either can be valid, but reporting should use the same interpretation.
Store counts for selected, suppressed, scheduled, accepted and failed deliveries separately. A campaign with 100,000 selected recipients and 20,000 suppressed preferences should not appear to have lost 20,000 messages.
Do not place a huge recipient list inside one queue message. Keep the queue item small and point to durable campaign state. This makes retries and worker recovery more manageable.
Throttle expansion as well as sending. Creating millions of jobs immediately can flood the database and queues even when providers are carefully rate-limited. Maintain enough queued work to keep senders busy without making backlog growth unbounded.
Cancellation also needs a durable campaign state checked by expansion and delivery workers. Stopping further expansion is insufficient if hundreds of thousands of jobs have already been created.
Apply Preferences, Templates and Expiry at the Right Time
Recheck send eligibility
A user may opt out after a campaign is created but before their job reaches a worker. Recheck applicable preferences near the actual send, using the service's agreed consistency requirement.
Define the final eligibility checkpoint. A preference change racing with an external provider call cannot always stop a request already in flight. Persist the decision and make the product's cancellation semantics explicit.
Transactional types may have different preference rules from campaigns, but that classification belongs in controlled policy. A caller should not evade preferences simply by submitting a marketing message under an urgent label.
Invalid addresses, suppressed destinations and revoked device tokens should be filtered before repeated provider submissions. Keep the distinction between a user preference and a technical destination failure.
Version templates and validate data
Store a template version with the notification or delivery plan. A delayed message should not unexpectedly use a different template because someone edited the current version after scheduling.
Validate required variables before the job reaches a provider. Missing order totals or unsupported localisation should become visible preparation failures, not empty placeholders sent to customers.
Escape user-controlled values for the channel's rendering context. Restrict links and sender identities to approved configuration. The notification service should not become a general mechanism for an application caller to send arbitrary content from a trusted domain.
Rendering can happen during orchestration or close to sending. Early rendering makes the content reproducible but stores more sensitive payload. Late rendering reduces stored content but requires durable template and data references. Choose deliberately and retain enough information to explain the message.
Schedule against a defined clock
Store the user's time-zone identifier as well as the scheduling rule when delivery depends on local time. Convert each occurrence into an actual scheduled instant using an explicit policy for repeated or missing local times during clock changes.
A scheduler should claim due work durably and create each occurrence once, even if it restarts and scans the same interval again. Use an occurrence identifier rather than relying on the scheduler's last in-memory tick.
Scheduling determines eligibility, not guaranteed delivery time. A message becoming due at 09:00 still passes through capacity limits, preference checks and the provider. Report lateness separately so a campaign's intended schedule is not confused with its actual submission history.
Expire work that is no longer useful
Check expiry before attempting delivery, not only when accepting the notification. A verification code delayed in a queue should become expired rather than eventually sent after the user has moved on.
Include provider queueing behaviour where supported. Passing a time-to-live to a push provider can reduce delivery of obsolete messages, but the internal service must still stop submitting expired jobs.
If a newer notification supersedes an older one, define a collapse or replacement key. A current account-state update can sometimes replace an earlier update; separate order receipts normally cannot. Deduplication of retries and intentional supersession are different operations.
Respect Provider Capacity and Retry Carefully
Each channel adapter translates a delivery into the provider's API and normalises its response into a small set of internal outcomes.
Limit both request rate and concurrency. Rate controls how quickly work begins, while concurrency limits requests that remain active during slow responses. Several workers sharing one provider account must respect the account's aggregate allowance.
Classify outcomes into accepted, permanent failure, temporary failure and unknown. Provider-specific error details belong in the adapter, while the workflow responds to a consistent internal classification.
Firebase recommends controlling send traffic, respecting throttling responses and using exponential backoff with variation to reduce retry amplification. Its guidance also emphasises avoiding abrupt spikes and discarding messages that are no longer timely. Exact retry rules should follow the provider and API in use. Sending FCM messages at scale.
Keep retries bounded and durable
Store the next attempt time instead of having a worker sleep while holding a job indefinitely. A delayed queue or scheduler can make eligible work available later.
Use an increasing delay with random variation and a maximum total retry budget. Honour a provider's meaningful Retry-After guidance. Stop when the delivery expires or a permanent error is confirmed.
Do not independently retry at every layer. If an SDK, HTTP policy and queue worker each make several attempts, the actual send count can be much larger than the workflow records. Understand SDK behaviour and assign clear ownership of retry decisions.
After bounded attempts, move unresolved work to an investigation path such as a dead-letter queue. Preserve the reason, original identity and attempt history. Dead-lettering is a visible outcome requiring a recovery policy, not proof that delivery succeeded.
Handle the Ambiguous Send
Suppose a worker submits an email. The provider accepts it, but the connection breaks before the worker receives the provider message identifier.
Retrying might send the email twice. Recording failure without retrying might leave the recipient with nothing if the provider never accepted it. The database transaction around the delivery row cannot determine which external outcome occurred.
Use stable provider identities where available
If the provider supports idempotent submission, send the stable delivery identifier as its idempotency key according to the provider's contract. Reuse the identity for retries of the same logical delivery rather than generating a new one per attempt.
Check the provider's deduplication scope and retention window. A key remembered for a limited period does not prevent duplicates from a manual replay months later. Different provider accounts or regions may also have different scopes.
Internal attempt identifiers remain useful for diagnostics, but they should not replace the logical delivery identity where the provider expects repeated requests to refer to the same send.
Reconcile unknown outcomes
When idempotent submission is unavailable, retain an unknown outcome. Query provider status or correlate callbacks where the provider supports a suitable reference.
A timeout should not immediately become a confirmed permanent failure. Nor should a callback absence be treated as proof that nothing was sent. Delivery events themselves can be delayed or unavailable.
For some message classes, the agreed policy may prefer a possible duplicate over a possible omission. For others, an ambiguous send should wait for reconciliation. Make that choice by notification type and record it.
Fail over without creating avoidable duplicates
Switching providers after a confirmed rejection is easier to reason about than switching after an unknown outcome. If the first provider may have accepted the message, the second provider does not share its deduplication state automatically.
Failover also changes available templates, sender configuration, quotas and delivery reporting. Prepare and test the secondary route before an outage rather than assuming every provider adapter is interchangeable.
Keep all attempts under the same delivery record. That lets support staff see that one intended email was tried through two providers, rather than misinterpreting the history as two separate business requests.
Process Delivery Events as Durable Inputs
Provider callbacks can arrive more than once, out of order or before the sending worker has saved its response. Treat them as asynchronous events requiring validation and durable processing.
Verify authenticity using the provider's documented mechanism and preserve the original data needed for verification. Limit payload size and reject untrusted callback sources without allowing them to alter delivery state.
Persist a verified event before acknowledging it when the provider contract and endpoint design permit. A separate worker can apply the event so slow database transitions do not make the callback endpoint unnecessarily fragile.
Deduplicate using the provider's event identity or a carefully chosen stable composite. Store the provider message identifier separately from the originating business identifier, because they serve different lookup paths.
Apply explicit state transitions
A delayed accepted event should not downgrade an already delivered job to accepted. A bounce or complaint can carry a meaningful outcome after earlier acceptance, so a single simplistic numeric status ranking is also insufficient.
Keep submission state, delivery outcome and recipient health as separate concepts where necessary. The workflow may have completed submission while a later callback marks the destination invalid for future messages.
Record both the provider's event time and the time our service received it. Use event semantics and transition rules rather than blindly choosing whichever event has the latest timestamp.
If a callback arrives before the worker's provider response is recorded, retain it for correlation instead of discarding it as unknown. A small reconciliation process can join newly available identifiers with pending events.
Operate with Fairness, Security and Clear Metrics
Protect tenants and recipients
Rate-limit acceptance by tenant so one campaign cannot monopolise ingestion or storage. Apply channel and tenant budgets during delivery as well, since accepted work can otherwise dominate providers later.
Store only the personal data necessary for sending and support. Restrict access to rendered content, destination addresses and security message payloads. Ordinary logs should use notification and delivery identifiers rather than full message bodies.
Provider credentials belong in managed secret configuration with appropriate access controls and rotation. Template authors, application callers and operators may need different permissions; one broad administrative role need not control all three.
For short-lived codes, avoid storing reusable secrets in broad diagnostic systems. The notification service should transport the content under a controlled policy while the owning authentication system remains responsible for code validity.
Measure age and outcomes
Track acceptance-to-orchestration delay, oldest queued age, acceptance-to-send time, provider latency, throttling, retries, expiry and permanent failures by channel and priority.
Queue length alone is insufficient. A queue of fifty old security messages can be more urgent than a queue of 50,000 bulk messages progressing within its schedule.
Measure duplicate suppression and unknown attempts. A rising deduplication count may indicate callers timing out or an outbox relay struggling. A rising unknown-outcome count can expose a provider-network problem before confirmed delivery failures increase.
Maintain a traceable path from business event to notification, delivery, attempt and provider message. This is the evidence needed to answer whether a message was accepted, submitted, delivered or still uncertain.
Test Crashes, Outages and Backlog Recovery
Test a worker crash before sending, after provider acceptance and after saving success but before acknowledging the queue. These points exercise different recovery paths and should not all produce the same expected behaviour.
Test duplicate acceptance requests and payload conflicts under the same key. Test campaign expansion restarting from an old checkpoint, preference changes while work is queued and expiry during a provider outage.
Feed callbacks in reverse order and repeat them. Verify that state remains meaningful, history is retained and invalid destinations are handled according to policy.
During a provider outage, pause or reduce submissions through a circuit breaker and keep the backlog bounded. Apply expiry before retrying old work when service returns. A recovered provider should receive a controlled ramp rather than the entire accumulated queue at once.
An investigation interface should allow targeted replay with audit history. Operators need to know whether replay means continuing the same delivery or intentionally creating a new occurrence. A button that silently duplicates every selected message is not a safe recovery tool.
Further Improvements
Once the core workflow is reliable, add digests for notification types where several events can be combined without losing meaning. A daily activity summary can reduce user noise and provider cost, while a time-sensitive security alert should remain immediate.
Introduce preference-aware channel selection only when the rules are understandable and measurable. Choosing the cheapest channel is not useful if it consistently fails to reach the intended recipient.
For regional growth, assign clear ownership of each delivery or coordinate claims so failover does not create concurrent senders. Replicated data alone does not guarantee that two regions cannot both call a provider.
Evaluate improvements against delivery usefulness, not just throughput. Sending obsolete messages faster or producing more provider requests per second does not improve the service's actual outcome.
Summary
A scalable notification service accepts intent durably, expands it into identifiable delivery jobs and isolates channels and priorities. Preferences, templates, schedules and expiry determine whether a message should be sent at all.
Retries and provider limits require careful pacing, while ambiguous sends require idempotency or reconciliation where available. Internal uniqueness cannot by itself promise exactly-once delivery through an external service.
Meaningful states and traceable history make the design operable. The system should explain what it knows about each notification, preserve uncertainty where it exists and recover from failures without turning a backlog into another outage.