Monitoring and Replaying Webhooks in Platform-Scale SaaS

Webhooks at platform scale need more than a working endpoint. Monitoring and replay form the operational backbone of event-driven SaaS, but they work only when durable storage, idempotency, and failure recovery are designed together. Webhooks need to be treated as a distinct infrastructure layer.

Why webhooks stopped being simple HTTP calls

A subscription change, payment confirmation, or provisioning request represents business state that another system depends on. Its delivery can be late, duplicated, or out of order. HTTP gives those distributed-systems problems a familiar address, which helps explain why a handler that works in staging can fail structurally in production.

External producers make that gap harder to close because the receiving team cannot pause them during a deployment or slowdown. Providers also differ in signature algorithms, timeout windows, retries, and delivery guarantees. The system must handle those differences even when events stray from the happy path. Receiving third-party events and sending events to customers also need different controls, though both are called webhooks. Autonomous agents sharpen the risk further because weak authentication, schemas, or audit trails can let failures compound without a human interrupting the chain.

The four-layer architecture that separates reliable webhook systems from fragile ones

The first layer is an event store that records what happened independently of delivery success. Its concrete model separates four tables, namely webhook_events, webhook_endpoints, webhook_deliveries, and webhook_attempts. These describe the business event, its destinations, each event-to-endpoint assignment, and each network operation. Their identifiers have different lifetimes by design. The Event ID stays the same across destinations and retries, the Delivery ID stays the same across retries, and the Attempt ID changes with every network call.

The second layer is a durable message queue between ingestion and processing. It absorbs spikes and holds work while services deploy or recover, then drains at a rate downstream systems can handle. The third layer, the delivery engine, controls signing, rate limits, retries, and circuit breaking per endpoint. Exponential backoff with jitter spaces out repeat attempts. Per-endpoint circuit breaking keeps one slow consumer from using up the retry capacity that other customers need.

The fourth layer makes delivery state queryable. Its operational test is plain enough to ask during an incident. How many events were not delivered successfully in the last hour? An answer should take under five seconds. Together, these layers support the move from a single monolithic handler toward an edge receiver that verifies signatures and publishes onto an internal bus, separating acceptance from processing.

Why idempotency is the load-bearing element, not an optional defensive measure

At-least-once delivery means duplicate receipt is an expected condition. Without idempotency, retries can create duplicate orders, double provisioning, or double charges. A stable event ID lets the consumer record processed events and discard repeats, so the same order reaches the ERP only once regardless of how often the webhook arrives.

The right processing pattern depends on what the event means. Fetch-before-process treats the webhook as a notification and retrieves current state from the provider API, which suits latest-state logic but costs API calls under rate limits. Conditional upserts compare incoming and stored timestamps so that only newer events mutate state. Idempotency key storage instead persists the event ID before processing and checks it before acting, requiring a fast, consistent key store. These patterns can be combined because they address different failure modes.

Traceability in agent-driven systems must do more than detect duplicates. Correlation IDs need to span receipt, signature verification, queue ingestion, worker execution, and downstream agent actions. When an account is provisioned twice, a trace is more useful than a confident theory delivered over a very long call.

What replay actually covers that retries do not, and why the distinction determines outage recovery

A deployment lasts longer than the retry window. By the time the service recovers, a cohort of events has exhausted its attempts and remains undelivered. The endpoint is healthy again, but the work is missing. Another automatic retry policy cannot bring back events it has already given up on.

Replay covers that gap through deliberate, event-scoped redelivery. Retries handle transient failures automatically within a bounded window, while replay lets an engineer or customer recover events across hours of systematic failure. Reconciliation checks for events that never arrived by polling the provider from time to time. None replaces the others. Replay also depends on writing the original payload, event ID, and delivery state to durable storage before attempting delivery.

Per-message replay restores one identified event to one endpoint. Bulk replay handles an outage by selecting events within a time window or matching a filter, rather than requiring hundreds or thousands of manual selections. Customers also need access to replay through a consumer portal. That changes recovery from a support request into a product capability, giving customers a practical reason to trust the integration after something goes wrong.

Observability requirements that turn webhook infrastructure into a managed system

Replay is useful only when the affected events can be found. Webhook observability therefore needs a queryable, per-customer view of delivery state. Aggregate dashboards can look healthy while one customer’s endpoints fail. Views for each customer show recent errors and which events can be recovered.

Rolling delivery success rates provide the primary health signal, while p99 latency from event creation to first successful delivery reveals systemic slowdown. Dead-letter queue depth shows how many events have run out of retries and need someone to step in. Retry counts per subscription expose endpoints consuming disproportionate capacity. A concrete SLO can target the vast majority of events receiving a 2xx response within six hours of creation, excluding endpoints customers have explicitly paused. That turns “delivery seems fine” into a measurable commitment.

These signals also need a route into incident response. OpenTelemetry export connects webhook telemetry to the existing operational stack, while alerting integrations reach tools such as Slack, PagerDuty, and OpsGenie. Visibility alone cannot stop a noisy neighbor. Per-endpoint throttling supplies the control, using that visibility to prevent one unhealthy destination from degrading delivery for other customers.

How the inbound and outbound webhook problems differ architecturally, and why one tool rarely solves both

Inbound infrastructure absorbs external behavior the receiving team does not control. Google Calendar push-notification channels have a default TTL of seven days and require proactive replacement, while HubSpot configures subscriptions at the app level, so one misconfiguration can affect every connected customer. Subscription management is therefore part of receiving events. So is attribution, because each event must reach the correct customer connection before processing.

Verification adds another provider-specific obligation through different signature algorithms, headers, timestamp rules, and challenge-response handshakes. Some providers offer no cryptographic signature at all. A single verification library cannot cover that range. Polling fallback completes the inbound picture by reconciling missed events that low-latency webhook delivery alone cannot account for.

Outbound infrastructure serves customers who depend on how the platform delivers events. It must fan events out to registered endpoints while isolating slow consumers. Its consumer portal is a product surface where customers register destinations, rotate signing secrets, inspect attempts, and trigger replay. Destinations also extend beyond HTTP to SQS queues, Pub/Sub topics, Kafka topics, and EventBridge.

What the current platform landscape covers and where the gaps remain

The 2026 landscape reflects this directional split. Selection turns on inbound depth, outbound management, deployment model, compliance posture, and cost. The tradeoffs are architectural before they are commercial. A good delivery engine, for example, does not by itself manage provider subscriptions or reconcile missing inbound events.

Svix Dispatch focuses on managed outbound delivery. It combines an embeddable consumer portal through svix-react or an iframe with per-message and bulk replay, granular retries, JavaScript transformations, FIFO ordering, and per-endpoint throttling. Its broader surface includes OpenTelemetry streaming, Standard Webhooks compatibility, and non-HTTP destinations such as Kafka, SQS, Pub/Sub, and EventBridge. That suits teams needing customer-facing management without building it. Its data residency options include the US, EU, Australia, Canada, India, and custom private regions, but it is not designed for inbound subscription lifecycle management or customer attribution.

Hookdeck addresses both directions with deliberately separate components. Event Gateway handles inbound durable queueing, backpressure, and filtering, while Outpost handles outbound retries, replay, telemetry, and destinations. A shared console and CLI connect the two. This suits engineering teams seeking lifecycle visibility across both directions, although Outpost has a narrower destination matrix and outbound feature set than Dispatch.

Nango concentrates on provider-aware inbound infrastructure within a catalog of over 1,000 APIs. URLs set up for each integration link incoming events to customer connections, while subscription management, verification, and polling fallback handle the ways providers differ. It does not manage outbound delivery to customers. Its fit is strongest when the difficult work is maintaining many third-party integrations.

Convoy offers self-hosted inbound and outbound infrastructure with retries, replay, circuit breaking, JavaScript transformations, and embedded dashboards. Its company has essentially wound down, however, leaving maintenance as a side project. That creates a substantial support constraint for production evaluation. Hook0 takes a narrower approach, offering France-hosted managed outbound HTTPS webhooks with subscriptions, retries, signatures, and replay. It is source-available under a license not approved as open-source, and its managed form is limited to a few thousand events per day. Transformations, FIFO, non-HTTP destinations, and SOC 2, HIPAA, and PCI-DSS certifications are absent.

Building internally remains reasonable for genuinely custom requirements and teams with maintenance capacity. A minimum delivery core combines in-memory datastore-backed BullMQ or SQS, a background worker, HMAC signing, and a dead-letter queue. The queue is not the expensive surprise. Customer dashboards, isolation, throttling, transformations, and bulk replay consume the longer-term engineering effort. A unified backend offers another architectural route by connecting events, authentication, payments, and customer data within one system, reducing synchronization work.

How webhook architecture intersects with billing events, usage metering, and the Merchant of Record model

A dropped payment event can grant access without payment, a duplicate can trigger a double charge, and a delayed event can leave provisioning ahead of confirmation or blocked behind it. Idempotency and replay become revenue integrity controls. In the common checkout flow, access follows server-side verification of payment state.

Usage-based billing expands this dependency beyond checkout. Every API call, document processed, and message sent must be captured in near real-time and routed to billing. For AI products with heavy usage, the event pipeline becomes the main source of data for the billing system. Missing or duplicated usage therefore affects the inputs from which invoices are built.

The Merchant of Record model addresses the tax and billing complexity around those transactions. A legal entity acts as the seller of record, handling obligations made more involved by dynamic invoice frequency, currency conversion, transaction amounts, and customer location. Without that model, the team must still build the compliance logic for each jurisdiction. When billing uses the product’s shared event infrastructure, its audit trail follows the same chain from when an event is received and verified through processing and invoice creation. Observability then helps teams both recover from incidents and account for the money.

Designing webhook infrastructure for GDPR and EU data residency from the start

Durable recovery creates a data-storage obligation as well as an engineering capability. Payloads containing user identifiers, transaction details, or behavioral events remain personal data when stored for retries. Debug logs that contain request bodies also store data. Replay archives therefore sit inside the GDPR compliance boundary, with retention and storage decisions that cannot sensibly wait until deployment.

Zero-storage pass-through integrations offer a different tradeoff by retrieving data directly from the source. They cannot replay from stored payloads they do not retain. Teams must explicitly decide what to store, where to store it, and for how long. “Temporary” is not a magic privacy setting.

Hosting jurisdiction adds a separate consideration. A US infrastructure provider serving an EU region can still be compelled under the CLOUD Act to produce data, so an EU region and EU-resident infrastructure do not describe identical compliance postures. Residency also does not replace the certification requirements discussed above. For European SaaS founders, these constraints belong in the event store, logging layer, and replay design from the beginning, because the webhook infrastructure shares the product’s compliance boundary.