Idempotent Webhook Handler Design for Payment and Billing Events

Payment providers guarantee at-least-once delivery for webhooks. Every handler you write will eventually get the same event twice. That's not a bug in their system; it's a design decision, and it quietly hands you a job you didn't apply for: making sure "twice" never turns into "double-charged." It's a design decision, and it quietly hands you a job you didn't apply for: making sure "twice" never turns into "double-charged."

Why at-least-once delivery makes idempotency the handler's problem

Deliver each event at most once, and risk quietly losing a payment.succeeded notification forever. Or deliver at least once, and accept that duplicate events appear in the stream. Deliver each event at most once, and risk quietly losing a payment.succeeded notification forever. Or deliver at least once, and accept that duplicate events appear in the stream. Every serious provider takes door number two, because a duplicate is annoying, but a dropped payment event is a support ticket, a refund, and possibly a very unhappy customer complaining on social media.

"Failed" from the provider's point of view doesn't mean your handler actually failed. Three things can happen that all look identical to the provider:

Your handler runs correctly but returns a 500 because a downstream call timed out. Or it runs correctly but takes too long, and the provider gives up waiting before you respond. Or it runs correctly, responds with a 200, and the response packet gets lost somewhere on the way back. In every one of these cases, the provider assumes failure and retries. And your handler, if it's not built for this, runs its side effect a second time.

The stakes aren't abstract. Double-charge a customer, and you're issuing a refund and an apology. Grant entitlements after a payment actually failed, and you've given away a product for free. Missing a churn signal because a duplicate event masked the real one means finding out about the cancellation a billing cycle late. The network layer leaves deduplication as homework you still have to do. It's homework, and this piece walks through how to actually do it.

A real incident: how a keying mistake let duplicate entitlements slip through a correct-looking handler

A billing service once started showing duplicate "Pro plan" entitlements after an invoice.paid storm hit during a regional outage. Users logged in to find two active seats attached to a single invoice, a bug that looks funny in a postmortem and infuriating in a support queue.

The root cause wasn't a missing deduplication check. The worker was doing the right thing, on paper: commit the event to a processed_webhook_events table, then call out to the entitlement service. When that entitlement call timed out, the worker returned a 500, and the provider (behaving exactly as designed) retried the event.

The second attempt found the event ID already recorded and returned a 200. Correct, so far. But a parallel worker, still running an older deploy, had the "not yet processed" code path open for a second event ID, one that represented the same invoice under a different event envelope. Two event IDs, one invoice, two grants.

The diagnosis lands on a single sentence: idempotency was keyed on the transport-level event ID, when the actual business rule needed was "one entitlement grant per invoice." Event IDs are a proxy for the thing you care about. They are not the thing itself.

Layer 1: signature verification before any business logic runs

A webhook endpoint has no logged-in user, no session cookie, nothing you'd normally call authentication. Instead, identity gets proven through an HMAC signature computed over the raw request body, and that signature has to check out before a single line of billing logic runs.

Verify the signature against the raw body, not the parsed JSON object. Body parsers reorder keys, add or strip whitespace, and generally reformat things in ways that look harmless but completely invalidate a signature computed on the original bytes. Parse first, verify second, and you'll spend an afternoon debugging signature failures that were never really failures.

Paystack sends its signature in the x-paystack-signature header, computed with HMAC-SHA512, and it should be checked using a constant-time comparison function like hmac.compare_digest rather than a plain ==, since a naive string comparison leaks timing information an attacker could exploit.

Another vendor's agent authentication scheme takes a different approach entirely, using RFC 9421 HTTP Message Signatures instead of a simple HMAC. It signs multiple components of the request and verifies them against the provider's public-key infrastructure. Replay protection comes from two angles at once: nonce rejection, plus a timestamp window. The open spec's guidance is to discard nonces seen within the last 5 to 8 minutes, which closes the door on someone capturing a valid signed request and firing it again later.

Layer 2: the idempotency table, schema, keying strategy, and race-condition handling

The workhorse of all this is a boring-looking database table, and boring is what you want here. A canonical version in PostgreSQL looks like this:

CREATE TABLE processed_webhooks (
  event_id TEXT UNIQUE NOT NULL,
  event_type TEXT NOT NULL,
  processed_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
  result JSONB,
  error TEXT
);

The UNIQUE constraint on event_id is doing the actual enforcement work here, not some if statement buried in application code. Application-layer checks race against themselves under concurrency. A database constraint doesn't.

When a duplicate event hits that constraint (Postgres error code 23505, or P2002 if it's coming through Prisma), the correct response is to catch that error and return a 200 immediately. The event was handled already. Telling the provider otherwise just earns you another retry.

But the incident above shows why keying on event ID alone isn't enough. The fix needs a business key too: a stable identifier tied to the actual object being changed, like an invoice_id or subscription_id, so the same underlying fact can't sneak through twice under two different event envelopes. Paystack's own recommended pattern reflects this directly: key as f"{event_type}:{id_or_reference_or_subscription_code}", combining the event type with whatever stable identifier is available. Two ingredients, not one, and the combination is what actually maps to the business fact you care about.

Layer 3: acknowledge fast, process asynchronously

The provider's timeout clock keeps running even if your handler eventually succeeds. A handler that does everything right but takes 11 seconds to respond is going head to head with a provider that gave up at 10, marked the delivery a failure, and already queued a retry. Correct and slow is functionally the same as wrong, as far as the provider's retry logic is concerned.

The fix is a pattern that shows up across just about every mature webhook system: verify the signature, drop the event onto a durable queue, and return a 2xx immediately. The actual business logic, the charge processing, the entitlement grant, the status update, runs later, pulled off the queue by a separate worker. Amazon SQS and RabbitMQ are the usual tools for this job, and both do the same core thing: they decouple how fast you acknowledge an event from how long it takes to actually process it, which also keeps a sudden spike in webhook traffic from slamming into your primary database all at once.

Moving the payload onto a queue does not solve deduplication. It relocates the problem. Now the worker pulling events off that queue is the one responsible for checking the idempotency table, and that check needs to happen at the moment the worker actually executes the job, not just when the event first landed at the front door.

Making handler logic itself idempotent, not just the delivery

The dedup table stops a known event ID from being processed twice. It says nothing about whether the code running inside that handler behaves itself when it's accidentally triggered more than once, and that's a separate problem.

Test this directly, rather than assuming it. Replay the exact same request twice and check that only one payment record gets created, with the second call simply returning whatever response got stored the first time. Then go a step further: simulate a lost response by forcing a timeout right after the server finishes processing, and retry with the same idempotency key. Does the second call create a second payment, or does it correctly return the first result?

Status-update handlers deserve the same scrutiny. Updating a subscription_status field to a value it already holds should be a complete no-op, producing no unintended side effects. And entitlement grants follow the same rule. Granting access to a user who already has that access shouldn't throw an error, and it definitely shouldn't create a second grant sitting next to the first one.

Handling out-of-order event delivery safely

Webhooks don't arrive in the order the underlying events actually happened, and that's not a provider messing up. It's just what distributed systems do when messages travel across networks with their own latency, retries, and queuing delays.

A few ways this appears in practice, and each one can quietly corrupt state if the handler assumes ordering that isn't guaranteed:

A subscription.updated event lands before the subscription.created event for the same subscription. A payment_intent.succeeded event for a retried charge arrives after an earlier payment_intent.payment_failed, out of sequence with how the retry actually unfolded. Or a refund.created shows up before the payment_intent.succeeded it's supposed to be refunding, which is a genuinely strange thing to process if your code assumes refunds always follow payments.

One fix is a conditional upsert: check the incoming event's timestamp against whatever timestamp is already stored, and only apply the update if the new one is actually newer. This stops an old, late-arriving retry from stomping on state that's already moved on.

The other approach sidesteps ordering. Treat the webhook as nothing more than a signal that something changed, then make a separate API call back to the provider to fetch the current, authoritative state, and process that instead of trusting the event payload's contents. The tradeoff is an extra API call per event, which adds overhead at high volume. It's the right call, though, when your business logic cares about where things stand right now, rather than the specific sequence of events that got there.

Failure recovery: exponential backoff, dead-letter queues, and safe bulk replay

Not every failure deserves the same response, and treating them identically is how retry storms happen. Split them into two buckets before deciding what to do.

Transient failures, things like an ERP timing out, a brief service outage, or a rate-limit response, should be retried with backoff, since the same request will probably succeed a few seconds or minutes later. Permanent failures are a different animal: a schema mismatch, an authorization error, or a malformed payload. None of those get fixed by trying again. Retrying a permanently broken request just wastes cycles and delays the point where a human finds out something's actually wrong, so route those straight to a dead-letter queue instead.

A standard backoff sequence doubles the wait time with each attempt, and after a set number of failures the event moves to the dead-letter queue rather than retrying indefinitely.

The dead-letter queue itself is a second queue that holds onto messages that couldn't be processed successfully, and its job is to stop two things from happening: infinite retry loops, and silently losing events that actually mattered. It's a monitored holding area, not a trash can. It's a monitored holding area, and anything that lands there should trigger an alert, wired into Slack, PagerDuty, or email, so a person actually looks at it instead of letting failed billing events pile up quietly in a table nobody checks until a customer complains first.