Fan-Out Patterns for Internal Webhook Event Distribution

Fan-out is the unglamorous work of copying one inbound event and handing a version to every system that needs it. This piece walks through what changes when you do that work correctly, and what breaks when you don't. Fan-out and routing are different jobs that often get conflated. Routing sends one event to one destination chosen by a rule: a billing.created event goes to the billing service, a user.deleted event goes to the audit service. Most production systems need both, and treating them as the same mechanism produces implementations that half-work in ways that are hard to diagnose.

In a monolith, the fan-out problem barely exists. One endpoint receives the webhook, internal logic dispatches to the right module, and everything lives in the same process. You call a method, life goes on.

Decompose into microservices and the picture changes. Each service owns a bounded context and its own endpoint, but the webhook provider sends the event exactly once, to one URL. Something external now has to copy and distribute that event, because the provider isn't going to send it three times just because you now have three services. The same requirement appears even without microservices: analytics pipelines, audit logs, and backup stores need copies of events even when a single service handles the primary business logic.

A correct fan-out layer delivers three properties. Each of those properties is cheap to claim and expensive to build. The sections that follow track what that cost actually looks like, stage by stage.

Why naive in-process forwarding breaks under real conditions

The simplest fan-out approach is to forward copies of the event to downstream services inside the same HTTP handler that receives the webhook. It is also the most common approach, because it requires no new infrastructure and no architectural decisions. Then production happens.

The handler's response time becomes the sum of all downstream call latencies. Webhook providers enforce strict response windows, and a slow or unavailable destination causes the provider to mark delivery as failed, which triggers a retry, which hits the same slow destination again. You have built a small machine for generating duplicate traffic and timeout noise.

Crash behavior is worse. Not retrying means some destinations permanently miss the event. Both options are wrong in different ways.

There is also no independent retry per destination. A single forwarding failure with no explicit state tracking means the entire fan-out must be retried as a unit. One destination being down for an hour means every other destination receives the event again every time you retry, multiplying load across the whole system for the sake of recovering one failure.

Formbricks ran into exactly this failure mode. Their codebase delivered webhooks synchronously to multiple target URLs on the responseFinished event path, and the result was performance bottlenecks, timeouts, and a lack of proper retry mechanisms if a recipient server was down. They filed and addressed it as a bug, not an optimization, because it was causing real failures in real deployments.

The core flaw is that in-process forwarding conflates event ingestion and event delivery into one blocking operation. Separating those concerns is the structural fix every subsequent pattern in this piece builds around. The first real attempt at that separation is the queue.

How queue-based fan-out separates ingestion from delivery and what it costs

Queue-based fan-out moves the delivery work out of the HTTP handler. Downstream consumers subscribe independently and process the event without coupling to the ingest path. The handler gets fast again, and a slow destination no longer blocks the acknowledgment your provider is waiting for.

The cloud implementations differ in how much machinery you are asked to assemble. GCP Pub/Sub achieves the same fan-out with a single service: a topic plus subscriptions replaces the SNS-plus-SQS combination. Same result, fewer primitives to configure and monitor.

Streams handle fan-out to multiple consumers particularly well because they are durable append-only logs. Multiple independent consumers can read from the same stream at their own pace, which queues alone cannot offer. You don't need a separate queue per consumer. You need a stream they can all tail.

The operational cost is real and easy to underestimate. You are now managing a broker, multiple queues or subscriptions, consumer workers, dead-letter queues for each consumer, and monitoring across all of them. This is an infrastructure surface, not just an integration, and every component on that surface has a failure mode you now own.

Truto's production topology makes the cost concrete. They arrived at that split through operational pain, not foresight, and the lesson is that queue topology is an architecture decision rather than a default. You can get it wrong, and when you do, one customer's broken webhook endpoint will slow down event intake for every other customer.

Queue-based fan-out also leaves the deduplication problem entirely unsolved. The queue made delivery faster. It did not make it correct.

Why atomicity at ingestion is the property that makes fan-out correct

Reliable fan-out requires that persisting the event and creating one delivery job per destination happen in a single atomic transaction. Any architecture that separates those two steps has a gap where data can be permanently lost, and the gap is silent. Consider the failure scenario. The provider's retry re-sends the event, but only if the endpoint returned an error, which it may not have. The event that was just lost is now unrecoverable, and no alert fires because nothing observable went wrong.

The correct implementation wraps three operations in one transaction: persist the event, resolve all matching destinations, and insert one delivery job per destination. If it rolls back, nothing is persisted and the provider's retry will re-send cleanly. There is no intermediate state where some destinations have jobs and others don't.

A single database transaction writing all per-hook sequences, log rows, and delivery jobs atomically prevents partial duplicate delivery records. This is the database-level guard against the race that queue-based systems leave to application code, and it is what turns fan-out from a best-effort activity into a correctness property. But the partial-fan-out failure, the one that loses events silently, is closed.

Parallel delivery without a broker is possible using FOR UPDATE SKIP LOCKED, which lets multiple workers process delivery jobs at once without contention. Each worker claims a batch of rows no other worker is currently holding, runs them, and returns for more. You get concurrent fan-out with the database as the coordination point, which is a much smaller operational surface than a message queue per consumer.

The reviewer of the Stoop PR identified a residual risk that atomicity alone does not eliminate. That is not a flaw in the design. It is a boundary marker for the next problem, which is idempotency at the delivery layer rather than atomicity at ingestion.

Conditional fan-out and payload strategy as first-class design decisions

Raw fan-out is sending every event to every destination, and raw fan-out is rarely what production systems want. Most destinations only care about a subset of events from a given source. Filter rules at the fan-out layer, based on event type, payload content, or metadata, control exactly which destinations receive which events without requiring each destination to discard irrelevant traffic after receiving it.

Filtering and fan-out combine naturally. Some events route to a single service, others fan out to several, and the decision is based on payload content. The destinations stay dumb and the fan-out layer gets smarter.

Payload strategy matters as much as routing rules. The principle is "store once, reference many." Ten destinations that each receive a small job record and fetch the shared payload means the heavy object moves once.

The fan-out multiplier applies to every component downstream of the branching point. Queues, workers, database connections, and network bandwidth all multiply by the number of destinations. Conditional filtering and lean payloads are the two levers that control that multiplier before it reaches infrastructure, and they are design decisions that should be made before you know how many destinations you have, not after.

Shopify's compliance webhook topics make the filtering obligation concrete from the other direction. When a redaction request arrives, the fan-out layer must route it only to the components that hold the data to be erased, not broadcast it to every destination. Broadcasting a deletion event to systems that don't need it is at best wasteful and at worst a compliance failure waiting to be discovered in an audit.

Even a well-filtered, lean-payload fan-out creates a new problem. One incoming event becomes many independent delivery traces, each with its own status, timing, and failure modes. That is the observability problem, and it multiplies at the same rate as the deliveries.

How fan-out multiplies observability complexity and what to instrument

When one event fans out to multiple destinations, the monitoring problem multiplies by the same factor as the deliveries. A system that looks healthy at the event level can be silently failing at the destination level. The ingestion dashboard shows a clean green line while one of five destinations has been dead-lettering events for three days.

One incoming event becomes as many delivery traces as there are destinations. A dashboard that shows successful ingestion without per-destination delivery status is not observability. It is aliasing: sampling a signal so coarsely that the failure pattern becomes invisible.

The minimum useful instrumentation surface has four parts. Track retry counts and backoff state per destination. And measure end-to-end latency from ingestion to successful delivery for each destination independently, because a destination that succeeds on the eighth retry after an hour is a very different operational situation from one that succeeds in 200 milliseconds.

Dead-letter behavior needs explicit alerting. A webhook endpoint returning a server error because a deploy was mid-rollout is transient. A webhook endpoint returning a "gone" status because the customer deleted it is permanent, and retrying it ten more times only deepens the log noise.

The ability to replay failed events is what separates observability that enables recovery from observability that merely records failure. Logs without replay capability are an audit trail, not an operational tool. You can see what went wrong, which is nice, but you cannot do anything about it except write code to resend manually.

At AI-agent scale, the problem intensifies. Without per-destination attribution, retries that fire duplicate agent invocations accumulate costs that may not surface until a billing cycle ends. You find out you spent money on duplicate work weeks after the spending happened, and the bill has already been paid.

Deduplication and idempotency as the boundary between "works in testing" and "correct in production"

Webhook providers operate on at-least-once delivery semantics. That is a polite way of saying they will send you the same event twice and expect you to handle it. A fan-out layer that is atomic at ingestion and fully observable at delivery still produces duplicate events unless idempotency is enforced at every layer where duplication can occur.

Duplication enters at two points. And the fan-out layer itself can repeat a committed transaction when a worker crashes before recording success, which the Stoop PR reviewer identified. Both paths are inevitable over a long enough runtime. The question is only whether the system treats them as recoverable conditions or as correctness bugs.

Deduplication before fan-out means tracking event IDs at the ingestion boundary and refusing to create delivery jobs for an event already processed. But it requires a shared state store, and that store becomes a contention point at high throughput. Every event now has to check in at the gate, and the gate becomes the bottleneck.

Idempotency at each destination means having each consumer track received event IDs and skip already-processed events. But every destination has to implement it correctly, and a single non-idempotent destination in the fan-out graph is a correctness gap. One destination that doesn't deduplicate means duplicate processing for that destination, which can mean duplicate charges, duplicate writes, or duplicate customer-facing side effects.

The unique constraint on (event_id, destination_id, attempt_number) handles the concurrent-worker case at the job layer. That requires the delivery job itself to be idempotent: it produces the same outcome regardless of how many times it runs. Idempotency at the job level is a property of the work being delivered, not of the machinery that delivers it.

At AI-agent scale, the stakes are financial as well as logical. A missing idempotency check means duplicate agent invocations across multiple providers, which accumulate billable calls that are difficult to attribute and impossible to refund after the fact. You re-run one agent workflow ten times and pay for ten executions, with no way to un-pay for nine of them.

The practical implementation decision is where idempotency keys live. Passing through the provider's original event ID to every delivery job and to every destination's processing logic gives each layer the information it needs to deduplicate without additional coordination. That key is a first-class schema decision made at the moment the event is ingested, not an afterthought bolted on when duplicates start appearing.

Where the fan-out layer sits in a backend that also needs to understand its own event data

A fan-out layer that is only a delivery mechanism throws away the most complete view of what is happening in the system. Every event passes through a single, durable log at the moment of ingestion. That log, accumulated over time and across all event types, is a unified record of every user action, transaction, and state change in the product.

Usage-based billing is the clearest downstream application. A backend that can count, attribute, and bill on those events from the same system that delivers them eliminates the reconciliation gap between what was delivered and what was charged. The number of jobs delivered and the number of units billed come from the same source, so they cannot disagree.

For products serving European customers or operating under GDPR, the fan-out log is also the data residency surface. A backend with explicit European hosting and enforced retention makes the fan-out log an auditable artifact rather than a liability. Where the fan-out layer runs is a decision about compliance, not just about latency.

Tiun's unified backend in a single system combines authentication, payments, customer database, and AI analytics. The fan-out layer and the business intelligence layer share the same data substrate, so the full picture of who is signing up, who is paying, and how they use the product is immediately visible without synchronization logic. The event that proves a delivery is the same event that proves a billable action, and both views read from the same log.

The fan-out layer is infrastructure. But where you put it, what you build around it, and whether it understands its own events are product decisions. Get them right and the same code that delivers webhooks answers questions about who your users are and what they do.