Email Webhook Retries: Handle Duplicates and Out-of-Order Events

TL;DR

• Expect webhook retries and deduplicate them using the provider's stable event or delivery identity.
• Verify signatures against the required raw request bytes before trusting the event.
• Acknowledge only after durable acceptance, then process business effects through a retryable worker.
• Preserve event history and explicit state rules so late events do not overwrite newer, more meaningful information.

A webhook is a delivery attempt, not a unique business action

An email provider may retry a webhook when your endpoint times out or returns an error. Your application can therefore receive the same event more than once. It may also receive different events in an order that does not match the order in which they occurred.

Build the receiver around those possibilities. A send idempotency key protects an outbound request; it does not automatically protect your inbound event handler. Each direction needs its own identity and durable record.

MailBlastr's webhook documentation describes its event envelope and delivery workflow. Use that provider-specific contract when implementing the receiver rather than copying another service's header names or signature format.

Verify authenticity before accepting the event

Read the raw request body required by the signature scheme, verify it with the correct signing secret and apply the provider's timestamp or replay checks. Parsing and reserializing JSON can change the bytes used for verification.

This is a common webhook integration boundary: for example, Stripe's webhook guidance explicitly requires the raw body for signature verification and discusses duplicate events and delivery ordering. The principle is transferable, but its exact signature protocol is not a substitute for MailBlastr's protocol.

Keep signing secrets in server-side secret storage. During rotation, follow the provider's supported procedure and avoid logging the secret or full sensitive payload when verification fails. Record a safe correlation identifier and the failure category instead.

Store the event before doing slow work

After verification, insert the event into a durable inbox table using a unique key for the provider, account and stable event identity. Store the event type, occurrence timestamp, receipt timestamp and the message reference needed for processing.

Commit that acceptance before returning success. If the database is unavailable, do not acknowledge an event that exists only in process memory. Once the event is durable, a separate worker can perform slower updates without holding the HTTP request open.

The exact queue or database design can vary. The invariant is that a successful acknowledgment means your system can recover the accepted event after a process restart.

Deduplicate the business effect as well as the input

An inbox uniqueness constraint prevents a duplicate delivery from creating another event row. The worker still needs to avoid applying the same business effect twice if it crashes after an update but before marking the event complete.

Where possible, commit the state update and the processed marker in one transaction. For another external effect, such as sending a follow-up message, create a separate logical operation with its own idempotency identity. Do not send a new email directly every time an email.delivered webhook is received.

MailBlastr's event-storage guide provides details for its delivery identifiers and stored event fields. Keep the deduplication boundary aligned with the actual stable identifier, not a timestamp rounded to the nearest second.

Do not reduce event handling to last arrival wins

Suppose a delivered event arrives, then an older sent event is retried. Updating the message status to whichever event arrived last would move the record backward. Compare event meaning and occurrence time using explicit rules instead.

Keep the event history even when you maintain a convenient current-state projection. Transport status, complaint state and suppression eligibility may be different dimensions. A later complaint should not disappear because another delivery-related event arrives afterward.

Use the provider's actual event model to define transitions. If a state cannot be resolved safely, retain the evidence and flag the record for reconciliation rather than guessing from arrival order.

Design retries and dead-letter handling

Give the processing worker bounded retries with backoff and a visible terminal state for unresolved failures. Record the attempt count, error category and next retry time. An event that repeatedly fails should not disappear into a successful HTTP response with no operational trace.

Test at least these cases:

• The identical event is delivered twice.
• Two workers claim the same accepted event.
• Processing crashes after the state update.
• An older event arrives after a newer one.
• A signature is invalid or the body was altered.
• A dependency remains unavailable beyond the retry policy.

Make replay a controlled operation

When replaying a stored event after a fix, preserve its identity and original occurrence time. Replaying should run the same deduplication and state rules as ordinary processing, not bypass them.

Connect each event to the original application notification and provider message ID. That makes investigations such as delivered but not received easier to explain. The result should be a system where retries improve reliability without creating duplicate actions or erasing the history needed to diagnose a problem.