Retry No. 185 Can Still Ring Twice: Designing the Call-End Webhook Delivery & Idempotency Gate for Callbot Loops

Why "At Least Once" Is the Default Contract

When a callbot platform fires webhooks for events like call completion, agent handoff, or DTMF input, most providers guarantee "at-least-once" delivery rather than "exactly once." Twilio resends the same status callback after a connection failure or a response timeout, and the call events Amazon Connect streams through EventBridge get re-queued for retry whenever the target invocation fails. If the receiving side is built on the assumption that each event arrives exactly once, a single retry turns straight into duplicate processing and duplicate billing.

The Two Lines Where EventBridge Retries Stop

Amazon EventBridge's retry policy stops at whichever of two limits is hit first: MaxRetryAttempts (0 to 185, default 5) and MaxEventAgeInSeconds (60 to 86,400, default 300). Raise the attempt count to 185 but leave the event age at its 300-second default, and retries only fire for failures that repeat within five minutes before the event drops straight to the dead-letter queue. Stretch the age window to 24 hours instead, and the same event gets a longer, more frequent window to re-arrive — widening the retry window is the same move as widening the duplicate-delivery window.

Field Guide: A Call-End Webhook Idempotency Checklist

Declare the idempotency targets as numbers before writing any code: zero duplicate-billing incidents from repeated events, a 100% detection rate for idempotency-key collisions, and a p99 response time under 300ms for the webhook handler's 2xx reply. Set the idempotency-key store's TTL longer than the MaxEventAgeInSeconds you configured for retries — if you're running a 24-hour window, give the TTL at least 48 hours of headroom.

A common failure pattern: the handler finishes transcription and CRM updates synchronously before replying, the processing time creeps past the response timeout, and the provider treats that as a failure and resends the same event. Returning a 202 immediately and pushing the actual work onto a queue cuts down on this class of retry on its own.

A second failure pattern: deduplicating on call_sid alone collapses genuinely different events from the same call — completion, hold, and transfer — into one. Build the idempotency key from call_sid plus event_type plus a timestamp bucket so each event type gets deduplicated independently.

Before shipping, reproduce a resend scenario where the same event is sent three times in a row. Log four fields on every delivery — receipt time, attempt count, the idempotency key, and the dedup verdict — so a retry pattern can be reconstructed after the fact. Before manually reprocessing anything sitting in the dead-letter queue, check why it wasn't handled within the 185-attempt budget first — a downstream outage and an idempotency-key collision call for different fixes.

When an idempotency-key collision is detected, skip processing the second request and return the first request's stored result as-is, so the caller still sees success. If processing genuinely failed and the event landed in the dead-letter queue, fire an alert immediately and hold it in a reprocessing queue so a human approval re-injects it right away.

Track the duplicate-delivery rate (the share of events with dedup=true) and dead-letter counts weekly. A rising duplicate rate points at the provider's timeout settings or the network path; a growing dead-letter queue points at a retry window or downstream capacity that needs retuning.

Takeaways at a Glance

Call-event webhooks in a callbot loop have to start from the assumption that at-least-once delivery is the baseline, not the exception. An idempotency key built from call_sid and event_type, a TTL longer than the retry window, an immediate-202 response structure, and a tested resend scenario together mean retries can run all the way to attempt 185 without duplicate billing or state corruption.

References

Event retry policy and using dead-letter queues — Amazon EventBridge

Webhooks (HTTP callbacks): Connection Overrides — Twilio