Webhook Delivery for AI Agent Event Pipelines

Picture an agent that receives a task.completed event, marks a pipeline step done, and moves on. Now picture it receiving that same event twice, marking the step done twice, and skipping the human review gate that was supposed to catch a mistake before a customer saw it. That's the failure this article is about: AI agent event pipelines break conventional webhook patterns, because the sender and the receiver are operating on two different clocks.

A webhook sender expects an answer in a few seconds. An AI agent, running an LLM to decide what a message actually means and what to do about it, can easily take longer than that. From the sender's point of view, a slow-but-working agent looks identical to a dead one, so it does the only sensible thing: it retries. That retry is correct behavior, not a glitch. The sender is honoring at-least-once delivery, the standard contract for webhook systems. The gap is that conventional SaaS receivers were built to log an event and move on, not to absorb a duplicate delivery that might trigger a second round of autonomous action. A second notification sent to a customer, or a review gate skipped because the agent thinks the work already happened, is a different category of problem.

There's a second failure mode layered on top of the first, and it's specific to agents talking to other agents. Conventional SaaS webhooks never had to think about this, because conventional SaaS systems don't typically subscribe to each other's outputs and act on them autonomously. Agent pipelines do exactly that. Agent-to-agent pipelines can form circular chains, and that needs its own fix, which this article gets to later. For now, the point is simple: time mismatch and circular choreography are the two cracks in the foundation, and everything else in this piece is about patching them.

Exactly-once delivery and the transport layer

The patterns in this article exist because of one unmovable fact: no system can guarantee exactly-once delivery across a network boundary. It's a structural limit on what asynchronous communication can promise.

The deadlock is that there's no third option that avoids both risks at once, and no amount of better code changes that.

Distributed transaction protocols like two-phase commit exist to paper over exactly this deadlock, forcing sender and receiver to agree before either commits to anything. But they add latency and coordination overhead that a high-throughput agent pipeline, processing a constant stream of events, can't afford to carry on every message.

The engineering response is to accept the impossibility and design around it: at-least-once delivery on the sender's side, paired with idempotent processing on the receiver's side, so a duplicate delivery produces no duplicate effect. Anything that drives agent decision-making needs the at-least-once-plus-idempotency combination. Everything from here forward is that combination, broken into its working parts.

The async acknowledgment pattern: returning 200 before the agent has done anything

Stop making the sender wait for the work to finish. Verify the payload, hand it off to a queue, and return 200 OK immediately. The agent processes the event on its own schedule, somewhere downstream, with no sender watching the clock.

The receiver's job, in that first pass, should be exactly three steps: verify the signature, enqueue the message, return 200. No database writes, no downstream API calls, no parsing the content to figure out what it means. The queue is what actually solves the time mismatch. The sender gets its acknowledgment fast enough to stop worrying. The agent gets as much time as the LLM needs to reason through the task, with nobody interpreting that thinking time as a crash.

It's showing up as a formal requirement in the platforms building agentic infrastructure right now. Google's Gemini API, in its event-driven webhook mode launched in May 2026, states that an endpoint must respond with a 2xx status within a few seconds to avoid a retry, and that failed requests get retried automatically for 24 hours using exponential backoff. MCP Events is still a proposed specification rather than a locked standard, but the direction is clear: the major platforms are building toward async acknowledgment as the baseline.

The choice of message broker, whether that's one managed queueing service or another, matters far less than the architectural decision itself. What the system needs is durable buffering between acknowledgment and processing. Which broker provides that buffering is a detail, not the point.

Signature verification and timestamp validation as preconditions, not afterthoughts

Async acknowledgment only works if what goes into the queue has already been checked. Enqueue an unverified payload and the agent pipeline becomes an open door: anyone who can guess or discover the receiver's URL can push fabricated events straight into an autonomous decision-making system.

Each step depends on the one before it, and skipping an early step undermines the ones that come after.

MCP Events, announced September 29, 2026, writes this requirement directly into the spec: HTTPS is mandatory for every callback, and private or local addresses are explicitly blocked from receiving them. That's a platform saying that security checks are part of the architecture, not a nice-to-have layered on top of it.

Timestamp validation earns its own line item because it closes a hole signature verification doesn't touch. A signed payload is still valid months after it was issued, by itself. For teams operating under data protection regulations, the deduplication store downstream (covered next) should hold event IDs, not full payload contents, partly for storage reasons and partly because a payload containing personal data sitting in a dedup log is its own compliance exposure.

Idempotent receivers: making duplicate delivery harmless

Once at-least-once delivery is accepted as the baseline, duplicates aren't an edge case, they're a guarantee. So the agent consumer has to be built so that processing the same event twice produces the exact same outcome as processing it once.

The sender's event ID is the right one, because it's stable across every retry. Using the sender's ID as the idempotency key is what makes the check catch duplicates.

Two implementation patterns handle this in practice, and they fail differently when something goes wrong. One is a Redis atomic SET NX command with a TTL attached: the first instance to see the event ID wins the lock and proceeds, every other instance in a horizontally scaled fleet sees the key already set and skips the work, and the TTL keeps the dedup store from growing forever. The other is a unique database constraint on the event ID column, where a duplicate insert simply fails and that failure is the signal to skip processing.

Either way, the idempotency check has to wrap the actual business logic, not just a record of having seen the event. One missing check like that can rack up real duplicate API costs before anyone notices the pattern in the billing report.

Deduplication logic and the 24-hour window, why TTL choice matters

Idempotency keys only protect the system for as long as they're kept around. Expire them too late and the dedup store grows without bound. Expire them too early and a legitimate late retry slips past the guard entirely, landing as if it were a brand-new event.

The TTL on the dedup store needs to match or exceed the sender's own retry window, which depends on the specifics of the sending platform, not some arbitrary round number that felt safe at the time.

The dedup store itself should stay lean. Storing full payload contents at volume adds storage cost for no benefit, and if those payloads carry personal data, it adds a compliance surface that nobody asked for.

Deduplication has to run at every hop in a multi-agent chain, not just at the first receiver. Each hop needs its own check, because each hop is its own opportunity for a duplicate to slip through.

Dead letter queues: preserving events that fail after all retries

An event that exhausts every retry attempt and still fails has to go somewhere. Without a dead letter queue, it disappears. In an agent pipeline, that disappearance might be a missed customer action, a broken step in a larger workflow, or a usage event that never gets billed.

A DLQ exists to catch exactly that: events that burn through their retry budget land there instead of vanishing, held for manual review or automated reprocessing later. The standard shape of an event pipeline runs Source to Receiver to Message Broker to Consumers, and that chain only counts as production-grade once the consumer tier has a DLQ sitting underneath it. Without one, failure in that chain is silent, and silent failure in an agent pipeline surfaces weeks later as a confused customer or a billing discrepancy nobody can trace.

SitecoreAI's webhook system is a useful real-world illustration of what happens without a retry safety net. Failures do get logged in the platform's lastRuns property, and a webhook auto-disables after 10 consecutive failures, so the failure isn't invisible. Platforms that handle retry on the sender's side, like Gemini's 24-hour window, take some of that burden off the receiver. Platforms that don't, like SitecoreAI, hand it straight back.

Preventing infinite loops in agent-to-agent event chains

Idempotency keys make duplicate delivery harmless. They do nothing for a different hazard that occurs specifically in agent-to-agent choreography: the infinite loop. If Agent A emits task.completed and Agent B reacts by emitting its own event, and that event eventually reaches Agent A again, each of those events carries a fresh ID. The idempotency check sees a new event every time and waves it through, because technically, it is a new event. The guard was never built to catch this.

The fix is call-chain tracking: assign a correlation ID to the event that started the whole chain, and carry that same ID through every downstream event the chain produces. As a backstop for cases where the correlation ID doesn't get propagated cleanly, a maximum hop count or chain-depth limit works as a blunt but reliable circuit breaker: once a chain crosses some fixed depth, it stops, no questions asked.

The choreography pattern itself, where Agent A's output triggers Agent B, whose output triggers Agent C, and so on down the line, is genuinely useful for building decoupled pipelines where agents don't need to know about each other directly. But it doesn't come with loop protection built in. That has to be engineered deliberately, every time. MCP Events, announced September 29, 2026, adds a relevant primitive here: its events/subscribe and events/unsubscribe methods give an agent explicit control over its own subscription state. An agent can unsubscribe from a topic right before emitting a downstream event that would otherwise loop straight back to it.

Recovering the observability cost of async acknowledgment

Async acknowledgment solves the time mismatch, but it doesn't come free. A synchronous call fails loudly and immediately, right in front of whoever's watching. An async system, with events sitting in a queue waiting to be picked up, creates a gap between the moment an event arrives and the moment anyone knows what happened to it.

Lorikeet, an AI support platform, officially discourages webhook-based integrations compared to synchronous API calls for exactly this reason: synchronous calls stay observable in real time and surface failures the moment they happen, where async processing does not.

Instrumenting the queue and the consumer brings the visibility back without giving up the reliability benefits already covered by retreating to synchronous calls everywhere. Every event should carry structured log fields: event ID, enqueue time, dequeue time, processing outcome, retry count. Logged consistently, that turns the async gap into a logging problem that's fully traceable after the fact.

A hybrid design handles the cases where real-time feedback genuinely can't wait: webhooks with async acknowledgment for reliable delivery of discrete events from external systems, synchronous API calls or WebSockets for the interactions where a user or system needs an answer immediately. And the dead letter queue from earlier does double duty here as the observability backstop. A DLQ event that fires an alert the moment it lands gets noticed about as fast as a synchronous error would, closing most of the gap Lorikeet's objection points to.

Usage-based billing as a direct beneficiary of reliable event pipelines

Everything built in this article, async acknowledgment, idempotent receivers, dedup windows tuned to match retry policy, dead letter queues, loop detection, has a direct line to revenue for any AI product charging by usage. A payment_intent.succeeded event processed twice isn't just a logging annoyance, it's a double charge or a double provisioning that someone in finance has to untangle later. A usage event lost silently because there was no DLQ to catch it is revenue that never gets billed.

Usage-based pricing depends entirely on the event pipeline counting things correctly, once, every time. This requirement rests on everything the article has built toward: at-least-once delivery from the sender, idempotent processing on the receiver, and a dead letter queue backing the whole chain. None of it is optional hygiene for a team running agents that bill by the task, the token, or the API call. It's the infrastructure the billing model is actually standing on.