How to Diagnose Silent Billing Failures From Dropped Webhooks

Silent billing failures from dropped webhooks are usually a reliability problem. A billing provider can report successful delivery while your application leaves access, invoices, or subscription state untouched. Diagnosing that gap takes a sequence of checks, from signature verification through processing and reconciliation. A green dashboard is a receipt.

Why webhook failures stay invisible until they become a revenue problem

The billing dashboard says “200 OK” and “delivered.” Your application did nothing with the event. Those statements can both be true because an HTTP acknowledgment tells the sender only that its request arrived, leaving your application responsible for recording the payment in your database or provisioning access. The sender cannot see that last mile. Once your endpoint acknowledges receipt, you are responsible for making sure the work gets done.

That separation is a systems property. Webhooks push billing events to your application with at-least-once delivery. If an acknowledgment fails because of a slow response, network drop, or server restart, the billing system retries. That retry is useful, but it also means the same event can arrive more than once. Different events can arrive in an unexpected order, while downtime can keep them from arriving at all.

These conditions can cause billing drift even when no clear error is visible in the application. A payment can exist in the billing system while the application has no corresponding update. Processing the same event again can also grant access twice or create duplicate records. The HTTP exchange looks simple because it hides the hard work of keeping business state correct when delivery is uncertain.

The five structural failure modes that produce silent billing drift

Failure mode 1 is out-of-order delivery. A subscription lifecycle can produce customer.subscription.created, invoice.created, invoice.paid, and charge.created without guaranteeing their arrival sequence. Different servers and network paths can change the order. A handler that expects creation before payment can therefore apply valid events to the wrong starting state. The request succeeds, but the database becomes inconsistent.

Failure mode 2 is signature mismatch from body parsing. Signatures are computed over raw bytes. Express-style body parsers, framework-level JSON parsing, or platform-level base64 encoding can change what reaches verification. The meaning may survive while the signature fails. A handler can reject a genuine event because it is comparing the wrong representation.

Failure mode 3 is timeout-induced partial processing. A handler might write to the database, call an external API, or send email before returning 200. If that work exceeds the sender’s timeout, another attempt arrives even though some work has already happened. The retry meets a half-finished job. Without safe duplicate handling, it can repeat side effects or compound inconsistent state.

Failure mode 4 is a deployment-window gap. Depending on hosting, an endpoint can be unreachable for seconds to a minute while a new version replaces the old one. Events sent during that interval fail delivery. If retries run out before availability returns, those events are lost to the application. A routine deployment becomes a billing interruption.

Failure mode 5 is retry exhaustion without an alert. Once the retry budget is spent, the billing system may abandon the event without a notification or dashboard flag. Silence is not recovery in this case. Without its own way to track delivery, the application has no clear sign that work is still waiting to be done. Each failure begins at a different layer, so one fix cannot cover the whole route.

Step 1 — Confirm the raw body reaches signature verification untouched

Start at signature verification, because events must get in correctly before you can debug what happens next. Raw body middleware must run first, followed by verification against the raw string using HMAC-SHA256, followed by JSON parsing. Order is the whole point of this check. If the framework parses or transforms the body first, verification no longer receives the bytes that were signed.

To inspect that boundary in staging, log the raw body bytes and computed HMAC before comparison. If they never match, check if parsing runs before verification. For Lambda and Vercel functions, also log typeof req.body and its length before verification because the platform may base64-encode the body. A Buffer needs handling that matches what the verification function expects.

A brief code-comment sketch is // webhook raw body middleware → route signature verification → JSON parsing → processing. In a web framework, mount raw({ type: 'application/json' }) specifically on the webhook route before global body-parser middleware. Also check for constant-time comparison and, where the provider timestamps signatures, validation that rejects events older than a reasonable threshold. Matching a signature and accepting a replay are different questions.

Step 2 — Check whether your handler can receive the same event twice safely

Once verification works, look for duplicate processing. Look for user records, subscriptions, or invoices that were created or changed twice in a short time without a business event to account for it. That is the fingerprint to investigate first. If event history exists, also query logs or processing records for duplicate event.id values. Their presence confirms a gap in idempotency, but their absence does not prove the handler is safe.

The standard protection stores each incoming event.id in a table with a unique constraint. Before processing, the handler checks whether the event already exists and skips completed work with a 200 response. An existence check alone is not enough. Two concurrent deliveries can both pass it before either inserts the ID. The unique constraint catches that race, with the duplicate constraint violation handled by returning 200.

The transaction is what makes this protection trustworthy. The event ID insert and business update must commit atomically, so a crash cannot leave an event marked complete when its work is still undone. Seen does not mean processed. If the handler crashes between those operations, the next retry must redo the work.

Idempotency also belongs in the business logic itself. If an event grants an entitlement that the user already has, repeating the grant should be a safe no-op. Inspect both layers during diagnosis. A unique event ID protects against repeated delivery of that event, while safe business operations keep repeated actions from corrupting state.

Step 3 — Map the event types your handler is ignoring or misrouting

Even with correct verification and duplicate handling, an event needs a handler that does the work. Enumerate the event types your billing provider can send and compare them with the types your code explicitly switches on. A handler built for successful payments may accept failure events just fine and then do nothing at all.

Pay particular attention to payment failure, subscription cancellation, invoice past-due, and dispute opened events. Missing the first two can leave access in place when it should have ended. An ignored past-due event can mean no dunning, while an ignored dispute can mean no response begins. For each type the code does not cover, follow the branch it takes. Does it return 200, throw an error, or log a warning?

A default branch that returns 200 without changing the database needs a close look. It tells the sender to stop trying but gives no sign that the application understood the event. Replace silent acceptance with a warning naming the unhandled event type, then route the event to a dead-letter queue for manual review. “Other” is a category that still needs a processing strategy.

Under a Merchant of Record model, this map changes because retries and subscription management can be handled internally. The integration may expose an entitlement signal rather than requiring your application to act on individual payment or subscription-creation events. Map the surface your integration actually exposes. Otherwise, the audit may look for billing handlers the application does not need and miss the access signal it does need.

Step 4 — Test whether your handler survives out-of-order event sequences

Replay the event sequences in reverse order in staging. Send invoice.paid before invoice.created, and subscription.updated before subscription.created, then inspect the database after each sequence. Start with the resulting state. A handler that returns success throughout the test can still leave the subscription wrong.

An event can be valid even when the record its handler expects has not arrived yet. An older event can also arrive after a newer update. Diagnosing ordering failures requires checking whether the handler uses the current state to establish business order before applying a change.

Draw the allowed subscription transitions from created to active to past_due to cancelled. Compare that state machine with the handler’s checks. Where the provider supplies a reliable timestamp or sequence number per resource, compare it on write and apply only events newer than the last one applied. That pattern depends on provider support. It cannot turn an unreliable ordering field into a reliable one.

Some ordering bugs only appear on a particular day of the month.

Month-end invoice events can arrive in a different sequence from mid-cycle events. Daily error-count alerts can miss the resulting spike when it is already part of the baseline. Use success rates for each event type with absolute thresholds to bring that failure to light. The calendar should schedule invoices.

Step 5 — Verify that your endpoint responds before the billing system's timeout window

Instrument total response time in the webhook handler and log it per event type. Compare those times with the timeout threshold in the billing provider’s documentation. A handler can do the right work too slowly, prompting a retry before its first attempt finishes.

Inspect synchronous work before the response, including writes across multiple tables, external calls to email providers, analytics or CRMs, and file operations. Each adds variable latency that compounds under load. Idempotency still needs those overlapping attempts to be handled safely. Keeping all processing inside the request preserves the conditions that produce races, even after duplicate protection has been added.

The remedy is to verify, enqueue durably, and acknowledge receipt, leaving business processing to a worker. The endpoint confirms reliable receipt. The worker handles correct processing. This is more than a speed improvement. It prevents heavy processing from delaying acknowledgment and lets accepted work remain separate from the request that brought it in.

Inspect the hosting platform’s zero-downtime configuration for a gap between the old instance stopping and the new one accepting traffic. Verify that the provider’s retry timing exceeds any deployment window. Durable enqueueing protects work already accepted, but an unreachable endpoint cannot enqueue a new request.

Step 6 — Build visibility into events that were delivered but never processed

Finish by measuring processing independently of delivery. The fraction of each event type that produces its expected database state change is the useful signal, while broader checks of total error rate or endpoint reachability answer different questions. They cannot establish that a successfully acknowledged payment event actually updated the application.

At the event-generation layer, the outbox pattern closes a related gap. Write the event in the same database transaction as the business state change, so a committed change has a durable event record and a rollback leaves neither. This protects the link between state and notification. It does not replace checks on what a receiving handler eventually does.

For failed processing, inspect the dead-letter queue and its alerts. After N retries, with 5 a common choice, failed events should move there and trigger an immediate alert. A healthy queue should be empty. Any depth above zero shows that processing has broken, rather than just slowed down, and the queue gives abandoned events a place where they can be found and reviewed.

Keeping customer records, billing events, and session data together makes reconciliation possible without cross-tool queries. Wherever the data is stored, the comparison must link delivered events to what happened in the application. Make that comparison daily or weekly. Match the events the billing provider logged as delivered against the state changes recorded in the application database, then investigate any delivered event without its corresponding change.