Idempotency Design for Webhook Event Consumers

Every webhook provider on the planet makes the exact same promise: your event will occur at least once. Not exactly once. At least once. That single word difference is the entire ballgame, and it means idempotency is the foundation you build in from the start. It's the foundation. A webhook handler that assumes single delivery isn't software, it's a bug report with extra steps.

Exactly-once delivery is impossible to guarantee at the network level. Exactly-once delivery being impossible to guarantee at the network level is a matter of math, not a skill issue or a vendor being lazy. It's a matter of math, not a skill issue or a vendor being lazy. Two parties trying to confirm over an unreliable channel that a message arrived can never reach full certainty, no matter how many acknowledgments they send back and forth. The FLP impossibility theorem covers the related case of reaching consensus in an asynchronous network where any participant might fail. Nobody can promise exactly-once delivery. The best any provider can do is pick a side, and they all pick the same one.

That side is at-least-once. When a provider can't confirm you got the message, it has two choices: assume you got it and move on, or assume you didn't and send it again. The first option (at-most-once) risks silently losing a payment.succeeded event forever. The second risks you processing that same event twice. A missing payment notification is a support nightmare and possibly a compliance problem. A duplicate is just an inconvenience, assuming your handler is built to shrug it off. So every serious provider bets on duplication over data loss, and that bet becomes your problem to solve on the receiving end.

And the scariest duplicates aren't the ones caused by some visible network hiccup. They happen when the handler actually finishes its job correctly, but the response back to the provider lands a few hundred milliseconds too late. The provider's timeout clock doesn't care that the work succeeded; it just sees silence, assumes failure, and fires the event again. It just sees silence, assumes failure, and fires the event again. Congratulations, you've now processed a successful transaction twice, and nothing on your dashboard will tell you why.

How retry windows and failure thresholds differ across providers shapes your deduplication design

Every provider has its own retry personality, and none of them read the same rulebook.

Shopify gives a handler 5 seconds to return a 2xx response. If a handler misses that window, Shopify retries up to 8 times over the following 4 hours. Rack up 8 consecutive failures, and Shopify may quietly delete the webhook subscription altogether; your integration then goes dark without so much as a strongly worded email. Shopify tags each delivery with an X-Shopify-Webhook-Id header (unique per delivery attempt) and an X-Shopify-Event-Id header (unique per underlying event), which matters a lot once you get to deduplication logic.

Svix, the infrastructure a lot of platforms use to send their webhooks, runs a longer game: roughly 8 attempts spread across about 27 hours. The event identifier appears in the svix-id header, or webhook-id if the sending platform is on the Standard Webhooks spec.

AWS SQS Standard queues use a different approach, with no fixed retry count or countdown clock, just a visibility timeout and a maxReceiveCount setting that you configure yourself. There's no fixed retry count or countdown clock, just a visibility timeout and a maxReceiveCount setting that you configure yourself. And because SQS stores messages redundantly across multiple servers for durability, a message you already deleted can, in rare cases, reappear. That's the architecture doing what it's built to do. That's the architecture doing what it's built to do.

Your dedup window has to be built around the specific provider's retry math. A system tuned for Shopify's 4-hour retry ceiling will fall over the moment it meets a provider that retries for a day and a half.

The fast-ack pattern: why the endpoint should do almost nothing before returning 200

The rule is simple: your webhook endpoint should verify the signature, drop the payload onto a queue, and return 200. That's the whole job. That's the whole job. Actual business logic, the database writes, the emails, the LLM calls, none of that belongs inside the request-response cycle.

Why? Because provider timeout windows can be as short as 5 seconds, and plenty of legitimate operations take longer than that. Doing real work inline isn't a performance optimization gone wrong, it's a structural guarantee that you will eventually get double-processed events.

This bites AI-native products especially hard. Calling an LLM directly inside a webhook handler means 8 to 12 seconds of response time on a good day. Stripe, for instance, caps webhook delivery windows at 20 seconds flat. Do the math: a single slow model response can single-handedly trigger a retry storm, and now you've got the same event landing twice while your LLM is still mid-sentence on the first attempt.

This failure mode is common: inline processing means a slow downstream call can blow past the timeout window, prompting the provider to retry en masse, flooding the system with duplicates while genuine first-time deliveries get buried in the noise. Nobody planned it. Everybody built it, one inline function call at a time.

Accept fast, queue the work, process it async. It's the software equivalent of a bouncer taking your coat and telling you the show starts later. Nobody's mad about the wait once they know it's coming.

Building a deduplication store with the atomic insert pattern, correct under concurrency

The pattern here isn't fancy, it just has to be airtight. Keep a table (or a cache, if speed matters more than long-term storage) of every event ID that's already been processed. When a new event shows up, check if its ID is already in there. Found it? Skip the work, return success, move on with your life. Not found? Insert it and proceed.

The "check" and the "insert" must happen atomically. Doing the "check" and the "insert" as two separate steps builds a race condition instead of a dedup system. Two near-simultaneous deliveries of the same event can both check "not found" before either one finishes inserting, and now they both proceed, and now you've defeated the entire point of the exercise. Use an atomic insert (INSERT ... ON CONFLICT DO NOTHING one relational database, a conditional put in another database service, whatever your database's equivalent is) so the database itself enforces the uniqueness, rather than trusting your application code to check first and behave nicely.

Providers hand you the identifier to key this off of, so there's no guesswork involved:

• Shopify sends X-Shopify-Webhook-Id
• GitHub sends X-GitHub-Delivery
• Svix sends webhook-id

Grab that header, make it your primary key or unique index, and let the database do the heavy lifting on concurrency.

Setting the right TTL: matching deduplication key lifetime to the provider's full retry window

Deleting a dedup key too early is like taking down a "wet paint" sign before the paint's dry. Someone's going to touch it, and that someone is going to be a retried event that your handler no longer recognizes as a duplicate.

The TTL calculation has a few inputs: the provider's full retry window, whatever delay your own queue introduces, clock skew between systems, and your policy on manual replays. The largest of those, not the average, should set the floor for how long a key sticks around.

Concrete example: if a provider retries for up to 3 days, your dedup store needs to hold onto keys for at least 3 days, full stop. For something like Shopify's 4-hour retry window, padding out to something like 48 hours or beyond gives room for queue delays and clock drift without cutting it close.

Manual replay is the one everybody forgets about. Support teams and ops folks replay webhooks all the time to recover from some processing bug that happened days ago. If your dedup keys have already expired by the time that replay happens, the "fix" just triggers the exact same side effect a second time, and now you're debugging the debugging.

Write down how long keys are honored and whether they're globally unique across your system. It matters because the person debugging a replay issue at 2 a.m. six months from now needs to know the rules without reverse-engineering your codebase first. six months from now needs to know the rules without reverse-engineering your codebase first.

State-machine guards and the fetch-before-process pattern for high-stakes transitions

Deduplication by event ID handles the "did I already see this exact event" problem. It does nothing for the "did these events arrive in the wrong order" problem, and that's a completely different animal.

Picture a subscription.updated event carrying an old status that arrives after a newer event already moved the subscription forward. Both events have unique IDs. Your dedup store lets both through, no complaints, because as far as it's concerned they're two different events, which they are. But applying the older one last means you've just overwritten valid, current state with stale data. Nobody in your system logged an error, because nothing about that was technically wrong. Applying the older one last means you've just overwritten valid, current state with stale data, which is just backwards.

State-machine guards fix this by making certain transitions conditional on current state. An "Order Shipped" event should only apply if the order's current status is "Paid." Any other starting state makes that event a no-op or an explicit error.

Another layer: on every insert or update, compare the incoming event's timestamp against whatever timestamp is already stored, and only apply the change if the incoming one is actually newer. Simple comparison, big payoff.

For the genuinely high-stakes stuff, entitlement changes, access grants, anything where getting it wrong means someone either loses access they paid for or keeps access they shouldn't, treat the webhook as a tap on the shoulder rather than the truth itself. Before applying the change, make a single API GET call back to the provider and pull the current state of the object directly. The webhook told you something happened. The GET request tells you what's actually true right now. Trust the second one.

Forwarding idempotency keys downstream so the whole processing chain deduplicates consistently

Deduplicating the webhook handler itself solves exactly one link in the chain. The moment that handler turns around and calls a payment API or fires off a transactional email, a whole new failure surface opens up. If that downstream call times out and retries without its own idempotency protection, you can end up with a duplicated charge or a duplicated email even though the webhook handler itself ran exactly once, behaving perfectly the entire time.

The fix carries the same identifier forward. The provider's event ID (or something derived from it) gets passed as the idempotency key on every downstream call the handler makes. Many payment processors accept an idempotency key header specifically for this reason, and using the originating event ID means the entire chain, from the original webhook all the way down to the final database write, deduplicates against the same stable anchor point instead of each link inventing its own logic.

This matters most in payments, where correctness can't depend on someone remembering to double-check the dashboard. It has to be built into the code structure itself: idempotency keys and conditional writes absorb the retries, balance updates ride inside a single atomic transaction, and the webhook layer enforces a unique event-ID constraint with proper ordering guarantees underneath it. Get that structure right once, and the system stops depending on anyone's vigilance.

Signature verification and replay attack prevention as the security layer that makes deduplication trustworthy

None of the deduplication logic above means anything if an attacker can forge or replay events at will. Signature verification is what makes the whole system trustworthy in the first place, and it has to be done carefully, not just present.

Always compare signatures using a constant-time comparison function, hmac.compare_digest in Python, crypto.timingSafeEqual in Node.js. A standard string comparison bails out early the moment it hits a mismatched character, and that tiny timing difference is enough for a patient attacker to reconstruct a valid signature byte by byte. It sounds paranoid until it's the actual attack vector.

Layer the defenses instead of leaning on one. HMAC signature verification is the primary gate, but an IP allowlist at the network level rejects obviously illegitimate traffic before it even reaches your application code. JSON schema validation on every incoming payload catches malformed data before it gets anywhere near your business logic. None of these replace the others, they just each cover a gap the previous one leaves open.

A valid signature proves the payload came from the right sender. It says nothing about when. An attacker who captures a legitimately signed payload can resend it later, and the signature will check out every single time, because it was never a fake in the first place, just a stale one. Closing this gap means including a timestamp inside the signed payload and rejecting anything older than a short window, a few minutes at most.

The frontier here is ephemeral key rotation: instead of a single long-lived shared secret sitting in an environment variable for years, providers mint short-lived HMAC keys, valid for a short, bounded window, and publish the active set through a JWKS-style endpoint. Receivers pull the current keys, cache them, and roll over automatically as new ones get published. If a key leaks, the blast radius is measured in minutes instead of however long it takes someone to notice and rotate a secret manually. It's the security equivalent of changing the locks on a schedule instead of waiting for someone to lose a key.