Stripe Webhook Event Ordering and Out-of-Order Event Handling

Payment webhooks fail in silence, not in stack traces. This piece walks through why a naive webhook handler breaks in three distinct ways (deduplication, ordering guards, and state regression prevention), and the exact patterns that fix each one.

Why payment webhooks are hard to handle correctly

A webhook endpoint can go down, quietly, and nothing about that failure looks like a fire alarm going off. Orders stop shipping. Entitlements stop updating. Meanwhile the rest of the system hums along, blissfully unaware that a whole category of events just stopped arriving.

Most billing systems let you register more than one endpoint. Up to 16, in fact, and every one of them has to be a publicly reachable HTTPS URL. That's a lot of doors for events to walk through, and a lot of doors that can quietly get bolted shut without anyone noticing right away.

And when something does go wrong, it rarely announces itself as a bug. It appears in a support ticket. "Why was I charged twice?" "Why does my account still say active after I cancelled?" These aren't stack traces. They're customers, annoyed, and usually right.

The three distinct failure modes that a naive handler produces

Write the simplest possible webhook handler, the kind that just reads the event and updates a row, and it will fail in three separate ways: deduplication, ordering guards, and state regression prevention. Not one bug wearing three costumes, but three actual, distinct problems, each requiring its own mechanism.

Failure mode one is duplicate processing, otherwise known as the idempotency trap. A retry sends the same event twice, and the handler, having no memory of the first time, runs the business logic again. The result: a customer ends up with two active subscriptions, a welcome email fires a second time, or a credit gets issued twice for the same refund. None of this requires malice or a weird edge case. It just requires a retry, which is a completely normal, expected thing for the system to do.

Failure mode two is state regression from out-of-order delivery, and it's the sneaky one. Picture this: a subscription gets cancelled, access gets revoked, all good. Then a second event arrives, one that was actually generated earlier but got delayed in transit, carrying an older snapshot where the subscription still looked active. Because that event has a brand new event ID, deduplication waves it through, no questions asked. The handler, trusting the payload, restores access that should stay revoked. The customer keeps using a product they cancelled, and nobody finds out until finance notices the numbers don't add up.

Both failure modes point to the same underlying gap. A handler needs separate, independent answers to three questions: has this delivery already been accepted, should this particular piece of information change the current state, and has the resulting business action already happened before? Answer only one of those questions and you've solved one-third of the problem. Answer all three and you're most of the way to a handler that actually holds up under retries and disorder.

Signature verification as the non-negotiable first gate

Before any of that logic runs, there's a gate that comes first, and skipping it isn't an option. Every incoming webhook needs its signature verified, and the verification function needs the raw, unparsed request body. Not the object your framework helpfully deserialized for you on the way in. The raw bytes.

Let any upstream middleware touch that body first, even something as innocent-looking as a global JSON body parser, and the signature check breaks. Not intermittently. Every time. It's one of those bugs that looks like a networking issue, gets escalated to the infrastructure team, and turns out to be a single middleware line registered in the wrong order.

And under no circumstances should signature tolerance get set to zero, or verification get disabled, in any environment expecting real, live signatures. That's not a shortcut; it's just turning off the lock on the front door and hoping nobody walks through it. Once a request clears this gate and is confirmed authentic, it can move on to the next layer: getting accepted into the system before anything business-related happens to it.

Separating receipt from processing: the inbox pattern

The core rule is to return a 2xx status immediately after verifying the signature and atomically recording the event, and to do all business logic asynchronously.

Why the rush to respond? Because if the endpoint doesn't answer fast enough, the delivery gets marked as failed and gets queued for retry, which floods the system with the exact retry storm the whole design is trying to avoid. Slow handler, more retries, more duplicate events, more load, slower handler. It's a feedback loop, and not the fun kind.

That is the whole rule: no business rules, no touching the payload beyond what's needed to route it later.

The word "atomically" is doing a lot of work in that sentence, and it matters more than it sounds like it should. Deduplication has to happen at the database level, using something like INSERT IGNORE or ON CONFLICT DO NOTHING against a unique key. A pattern where the code first checks "does this event exist?" and then inserts it if not, sounds reasonable, but it loses badly under concurrency. Congratulations, deduplication just failed at the one job it had.

One reference implementation states that the inbox layer handles transport only. Across the whole pipeline, the payload itself never gets read for state beyond its type and the ID of the object it refers to. Everything else about "what actually happened" gets fetched fresh, later, from a source of truth, not trusted from whatever arrived in the wire. In the inbox pattern, the inbox layer's only jobs are to verify the signature, attempt an atomic insert of event.id into a webhook_events table with a uniqueness constraint, and return 200, with zero business logic. According to the Zenn1t reference implementation, the inbox layer handles transport only, with no state, no business rules, and no payload inspection beyond evt_id.

Choosing an ordering strategy that matches the payload contract

Not every event type behaves the same way, or can be ordered the same way. There are, broadly, four kinds of payload contracts, and each one calls for its own strategy.

A snapshot event, one carrying a monotonic revision number, only needs one rule: apply it if the revision is newer than what's stored, ignore it otherwise. An ordered delta, the kind that says "increment by one" rather than "here's the new total," needs the next sequence number in line, and any gap needs to be noticed and recovered, not shrugged off. A notification with no reliable revision at all can't be trusted on its own; the only safe move is to go fetch the current state directly from the source and reconcile. And an independent, immutable business fact, something that happened once and stays true forever, just needs to be applied once, keyed to its own identity.

The mistake that trips people up is treating snapshots and deltas as if they follow the same logic. A snapshot event can skip over intermediate versions just fine, because each new snapshot is a complete replacement of the state that came before it. A delta event absolutely cannot skip an earlier step, even if the number attached to it is technically higher, because "add one to inventory" depends on every earlier addition having already landed. Neither failure is visible right away. Both are visible eventually, usually in a support ticket about inventory that doesn't match reality.

The lock-and-refetch pattern: how to make any ordering converge

There's a pattern that sidesteps the whole ordering puzzle by refusing to trust the payload's version of events at all, and it goes like this: lock, fetch, write, commit, all in one transaction.

While holding that lock, go fetch the current, authoritative state, mirror it locally, derive whatever downstream entitlements need deriving (a pure function, no side effects, no I/O), and commit. One transaction, start to finish.

The detail that makes or breaks this pattern is where the fetch happens relative to the lock. It has to happen inside the lock, never before it. Fetch before locking, and here's what goes wrong: worker A fetches, worker B fetches, worker B writes, worker A writes its now-stale data on top, and the older state wins. That's the exact bug this whole pattern exists to prevent, just reborn inside the fix meant to solve it.

Get the ordering right, though, and the handler becomes commutative, so any arrival order produces the same final state, and idempotent, so the same event twice has the same effect as once. One reference test suite actually proves this out by running all 120 possible orderings of a four-event subscription lifecycle through the handler and checking that every single one converges to the same result.

With v2 thin events the handler already must call the API for object data, so the lock-and-refetch pattern maps naturally. That said, thin events don't eliminate the need for version-skew handling: the object can still change again in the gap between the event firing and the fetch actually landing, so the lock-and-refetch discipline still matters just as much. The lock-and-refetch pattern uses SELECT … FOR UPDATE on a sync_locks table keyed by object_id.

Timestamp-guarded upserts for handlers that use payload state directly

Not every team wants to rebuild their handler around a full fetch-on-every-event pattern, and there's a lighter option for handlers that work directly off the payload's own state. A guarded upsert updates the row only if the incoming event's timestamp is newer than what's already stored.

In practice that looks like an ON CONFLICT DO UPDATE clause with a WHERE condition comparing the stored timestamp against the incoming one. An older event arrives late? The update runs, finds the condition false, and quietly does nothing. A newer event arrives? Same clause, condition true, row updates normally. One SQL statement, no application-level branching required.

This only works if the timestamp is actually being stored somewhere for future comparisons; the schema needs a dedicated column for it sitting right alongside the rest of the subscription data. Pairing that with a separate events table, one that tracks every event ID for both audit purposes and idempotency, plus optimistic locking on the subscription row itself, covers the race conditions that come from two workers touching the same row at once.

Fair to say up front: this pattern is easier to build than lock-and-refetch, no question. But it doesn't buy the same guarantee of commutativity across different event types, and any team reaching for it should go in knowing exactly which failure modes it doesn't cover, rather than assuming it's a drop-in replacement for the heavier pattern.

The generation-counter pattern for reconciliation under persistent uncertainty

For objects that get hit by a constant stream of updates, arriving in unpredictable order, forever, there's a pattern built specifically for that kind of chaos: a generation counter.

Every time a webhook arrives referencing the object, bump the requested number. Workers scan for anything where the applied number lags behind the requested one, fetch fresh state, and try to mark it applied, using an update along the lines of WHERE requested_generation = $3 AND applied_generation < $3 RETURNING object_id. If another webhook snuck in and bumped requested_generation again while that fetch was mid-flight, the update matches zero rows, the object stays marked dirty, and some worker, maybe the same one, maybe a different one, picks it back up and tries again.

If two workers fetched for the same generation, only one can mark it applied (the conditional update is the race guard). The other one's update just does nothing, no harm done.

A duplicate delivery of an event already accepted doesn't need to touch the generation counter at all, because deduplication at the inbox layer already caught it before it got anywhere near this logic. And when a fetch fails, the right response is to treat that object as still owing work, not to assume it vanished or was deleted. Failure means "try again," not "give up and move on." oneuptime.com describes the generation-counter pattern's schema as consisting of an object_refresh table with requested_generation and applied_generation columns per object.

Protecting business effects separately from state projections

Refreshing a subscription's state is cheap to get wrong and easy to fix. Just run the projection repeatedly and it converges to the correct answer. Sending an email, though, or issuing a credit, is a one-way door. Do it twice and there's no clean undo.

For business effects, the pattern is a dedicated actions table with its own unique identity, something like a combination of source scope, invoice ID, and the specific fulfillment action being taken. The action record and whatever downstream command it triggers get created together, atomically, the moment the business condition is actually met.

That separation pays off cleanly later. A reconciliation pass can update the subscription's projected state as many times as it needs to without ever risking a duplicate action for the same invoice, because the two tables don't know or care about each other. State can churn. Actions stay fixed once they've fired.

Resist modeling every subscription lifecycle as a simple ladder, active, then canceled, then done, ranked in order. Plenty of real billing setups don't work that way. A subscription can return to active from a paused state, and a cancellation that's merely scheduled for the future is a genuinely different thing from one that's already taken effect. Build the actual lifecycle the business runs, rather than a generic rank ordering borrowed from a tutorial.

On the subject of which events actually matter for product logic: invoice.paid is the one that fires for every subscription renewal and is, for most handlers, the event actually worth acting on. checkout.session.completed marks the exact instant a customer finishes checkout, which makes it the natural hook for analytics and welcome emails, rather than for anything tied to ongoing billing state.

Snapshot events versus thin events and ordering logic

Some event formats hand the handler a full snapshot of the object as it looked at that moment, complete, versioned, ready to use. That version of the payload gives the handler everything it needs immediately, no extra network call required, but it comes with a cost attached: the handler now has to manage version skew and payload drift against whatever the API considers current.

The alternative, a thinner event that just says "something changed on this object, go look," pushes the fetch responsibility onto the handler but sidesteps a lot of payload trust issues, since the handler never works from stale embedded data. Neither format eliminates the need for ordering discipline. It just moves where that discipline has to live, either inside the comparison logic guarding the snapshot's version number, or inside the fetch-and-lock sequence that guards against a stale refetch winning the race. Pick the pattern that matches the payload contract actually in front of the handler, rather than the one that happens to be easiest to write first.