What Webhooks Are and Why They Fail in Production

Webhooks look like plumbing, an HTTP POST that moves news from one system to another. In production, that simple handoff can lose events, do the same work twice, or lock out a customer who has paid. Reliability comes from understanding how these failures connect, then designing for retries, durable events, and visible recovery.

What webhooks actually do and the delivery guarantee they make

When an event happens on the sender’s side, a webhook pushes an HTTP POST to a URL the receiver configured. There is no polling for changes. The difficulty is confirming delivery across a distributed system, because the network conditions that can lose an event can also lose its acknowledgment. A sender that cannot confirm receipt must risk either a missing event or a duplicate.

That uncertainty sets the practical contract of at-least-once delivery. Events are retried after failure, and receivers must be prepared to handle the same event more than once. Exactly-once delivery is not the foundation to build on. Idempotency is the receiving side of this bargain, meaning repeated processing of the same event produces the same result as processing it once. This is the operating model.

Why the naive implementation looks correct and fails under load

A common implementation puts an outbound HTTP POST in the same request thread that processed the business operation. The local demo looks perfectly respectable. It usually runs without the transient network errors, slow consumer endpoints, timing of database transactions, and multi-tenant load that will test the design later. A successful request proves that the route works.

In production, the receiving handler often adds database writes, external service calls, PDF generation, or outbound email before responding. Each task uses up some of the sender’s response window. When the window closes, the sender can treat delivery as failed even while the receiver is still working. A retry can then overlap the original execution, turning one slow request into duplicate processing.

Business work and delivery are tied to the same request path, while committing a change and publishing its event remain separate operations. Fixing a slow call leaves the publication gap intact.

The five failure families and what actually breaks in each one

Webhook failures in production have five kinds of root cause. They break at different points along the delivery path, so they leave different evidence. An application error log cannot explain a request that never reached the application. Recognition starts with locating the break, rather than assuming every failed delivery needs another attempt.

DNS and network failures prevent the sender from reaching the endpoint. The handler has nothing to log. A registered URL may still point to localhost or an internal IP range, contain a domain typo, or resolve through stale cached DNS records after a migration. Slack’s Events API reports connection_failed for DNS failures and unreachable hosts.

Timeouts produce intermittent failures when acknowledgment takes too long. Synchronous work makes this family especially common. Shopify enforces a one-second connection limit and a five-second total limit, while serverless cold starts alone can consume two to three seconds before application code runs. Slack allows three seconds for a response, and GitHub imposes a ten-second hard limit with connection termination. Diagnosing timeouts requires measuring how much of that window disappears into startup, database writes, LLM calls, CRM updates, or email dispatch.

SSL/TLS certificate failures break the trusted connection needed for delivery. The endpoint can exist and still be unusable. Webhook senders generally require a valid, fully trusted certificate, and most providers reject self-signed certificates.

Signature verification failures mean the request arrives but cannot be authenticated. A frequent cause is computing the HMAC over re-serialized JSON instead of the raw, unparsed bytes. Whitespace and key ordering can change during that round trip, so JSON with the same meaning can yield a different signature. The bytes are the evidence. A separate diagnostic path is secret configuration, because a billing webhook’s per-endpoint signing secret is distinct from the API key, and updating only the API key leaves verification broken.

WAF and security-layer blocks reject the request before the handler sees it, typically returning a 403 or 401. Webhooks are automated server-to-server requests that neither act like a browser nor run JavaScript, so strict bot protection can see them as suspicious. To be fair, the bot detector has found a bot. Global CSRF middleware creates a similar rejection because the sender cannot supply a CSRF token. The diagnostic path runs through CDN rules, hosting-provider firewalls, and middleware.

Retry behavior cuts across these families but is provider-specific. GitHub does not automatically retry a 500 response, leaving manual redelivery from delivery history as the recovery route. Nor is every failure transient. A 400 indicates a malformed or invalid payload, while a 410 says the endpoint is permanently gone. Repeating either request does not repair its cause.

How failures compound: state corruption, duplicate grants, and the multi-tenant blast radius

A payment can be captured and committed, followed by a crash before the downstream webhook is published. The customer is charged but receives no access. Because the receiving system never learns that delivery should have happened, it has no failed request to investigate. Without monitoring to catch this, the customer becomes the alerting system, usually with less polite wording.

Retries can make a different failure worse. When a timeout triggers another delivery while the first is still executing, both attempts can process the same state transition. Without idempotency, that can grant two credit packs, two subscription activations, or two role assignments. Ordering can also corrupt state when a customer-update event arrives before the customer-creation event and tries to change a record that does not yet exist. Arrival order is not a safe substitute for state management.

In a shared FIFO delivery queue, a slow or unavailable tenant endpoint can hold up every tenant behind it. Those other endpoints are healthy, but their events are not moving.

These failures damage both state and confidence in the integration. Enterprise buyers treat event loss as a reliability disqualifier because business-critical workflows depend on changes arriving. For builders, the concrete problem comes first. A system must be able to tell lost, delayed, and repeated events apart to reliably link a payment to the access it should grant.

The transactional outbox pattern and async queue architecture that prevent the failures above

The transactional outbox pattern closes the most important structural gap. The application writes the event into an outbox table inside the same database transaction as the business change, so both succeed or fail together. A separate process reads the outbox and hands events to the delivery queue. A crash no longer leaves a committed change without its matching event.

More sophisticated implementations use change data capture to stream the outbox table’s write-ahead log into a message broker through Postgres logical replication or an equivalent mechanism. That removes polling latency and scales with event volume. The underlying guarantee stays the same.

On the receiving side, the corresponding design is acknowledgment followed by background processing. The handler verifies the signature against the raw bytes, enqueues the raw payload, and returns 200 immediately. Workers perform the actual business work. This removes long-running processing from the response window, though it does not make DNS, connection, or security failures disappear. For AI workloads, where an LLM response can take many seconds, waiting for the model before acknowledging is not a viable design.

The delivery queue then needs a recovery policy. Immediate retries amplify an outage by sending more work to an already struggling endpoint, so exponential backoff with jitter spreads attempts apart. A representative schedule starts at 30 seconds and grows over several hours, then moves exhausted events to a dead letter queue for inspection. HTTP 5xx responses call for retries, but permanent 4xx failures, especially 400 and 410, should be left out of retries.

Because recovery can repeat delivery, handlers must deduplicate using a stable event ID supplied in the payload header. The sender must provide that key. Processing the same ID more than once must give the same result as processing it once, with no extra grant each time. Meanwhile, a thin-event, rich-fetch approach limits payload exposure by sending only what the consumer needs and allowing additional detail to be fetched through authenticated API calls with scoped permissions.

Teams often overlook secret rotation when they design the system. During a configurable grace window, verification accepts signatures from either the old or the new secret. Afterward, it rejects the old secret. Receivers can make the change without every system having to switch at exactly the same moment. Otherwise, routine credential maintenance becomes a scheduled delivery outage.

Observability as a first-class requirement, not a debugging afterthought

Without dedicated monitoring, even failures the architecture preserves for recovery can remain unnoticed. Most teams building webhook systems do not have that monitoring in place. An outbox and a dead letter queue provide somewhere for events to survive, but survival alone does not tell an operator what needs attention.

A synchronous call surfaces failure in real time and produces a clean audit record, while an async acknowledgment confirms acceptance of the business work. That distinction deserves more than a reassuring dashboard. Visibility needs to be restored through logging and alerting across delivery and recovery without putting slow processing back into the request path.

Every delivery attempt needs its HTTP status code, response body, latency, and classification recorded as delivered, retrying, permanent error, or dead-lettered. Aggregates cannot replace the individual delivery record. Success-rate alerts need to operate per endpoint, so degradation generates a signal before a customer reports it. Operators must still be able to inspect and replay exhausted events, and to redeliver a given event from a given time window without doing the original business operation again.

Status codes, response bodies, a “send test webhook” button, and on-demand dashboard retries help separate integration mistakes from delivery problems without a support-ticket relay. The Standard Webhooks specification reduces further friction by defining HMAC-SHA256 signing over an ID, a timestamp, and the raw payload. Its standardized headers include webhook-id, webhook-timestamp, and webhook-signature, giving receivers and observability tooling a consistent format to work with.

What a webhook system that holds in production actually looks like from the outside

In a production-ready system, failure is a known event. It has a classification, visible delivery history, and a recovery path rather than becoming an unexplained gap in customer state. An operator can investigate and replay an event without charging the customer again or repeating the original operation. A receiver can distinguish a rejected signature from a delayed delivery.

For SaaS and AI products, billing makes that standard especially concrete. Subscription updates, payment failures, and cancellations control access and subscription status and can leave someone charged without receiving the service. Keeping authentication, billing events, and customer records in the same system can reduce the inter-system synchronization surface where ordering and consistency failures arise.

From the outside, reliability therefore looks less like uninterrupted success and more like controlled recovery. A failed delivery does not become a second credit grant, a missing cancellation, or a mystery left for support. The network may still have a bad day. The customer should not have to explain it to the engineering team.