Why an LLM Failure Hits Differently in a Callbot Loop
When an LLM call fails in a text chatbot, the user sees an error message and a retry button. In a callbot loop, the same failure turns into silence on the line — the caller can't tell if the call dropped or the bot is still thinking. A phone call only holds together if each turn gets a response back within a few hundred milliseconds, so an LLM provider outage has to be detected and handled on a much tighter clock than in a text interface. The callbot pipeline runs STT→LLM→TTS, and when the LLM stage stalls, audio output stops while input keeps streaming in — an asymmetric failure.
Retrying the Same Provider Is Not Recovery
If the root cause sits with the provider itself — model serving, rate limits, a regional outage — sending the request to the same endpoint again just repeats the delay until the outage ends. A design that puts a fallback chain in front of the LLM stage, like LiveKit Agents' Fallback Adapter, switches to the next model or provider on failure and signals recovery through an availability-change event. Mixing retry and failover into one mechanism means the first failure already burns a retry against a provider that isn't coming back soon, piling delay onto the turn.
What Gets Decided in the First 10 Seconds: An LLM Failover Operations Guide
Per LiveKit's own docs, the LLM adapter in the Agent Fallback Adapter runs with an attempt_timeout default of 10.0 seconds and a max_retry_per_llm default of 1 — if a provider doesn't respond within 10 seconds, it gets exactly one retry before the loop moves to the next provider. Whether you keep those defaults or tune your own, the callbot loop has to fit that decision time inside its overall turn-latency budget. Reasonable operating targets: fallback completion within 15 seconds, no user-perceptible change in speaking pace or tone after a switch, and 99.9%+ monthly availability for the LLM stage.
Latency degradation is more common than outright failure, and harder to catch. VideoSDK's Fallback Adapter lets you set a latency_threshold_ms that measures time-to-first-token on the LLM stage, switching providers once consecutive_latency_hits (default 3) in a row exceed the threshold. A design that only watches for hard failures misses the case where a response still arrives but late enough to break barge-in timing — in a callbot, that's the more common cause of perceived quality loss.
The first failure pattern is not separating retry from failover, so requests keep piling up against the same struggling provider. If the cause is a rate limit or a regional outage, that same endpoint keeps failing until it recovers, so exhausting the retry count should itself be the trigger to move to the next provider.
The second is freezing the fallback chain's order without regard to latency characteristics. Providers differ in model, tone, and response speed, so the chain order should weigh latency, cost, and tone similarity together, so a mid-call switch doesn't feel jarring to the caller. Another version of this pattern is a successful switch that never gets undone — staying on the secondary provider forever with no recovery check. Periodically health-check the primary, and once an availability-recovered event fires, shift traffic back after a cooldown window.
Before deploying, list the fallback order explicitly for each stage — STT, LLM, TTS — and run a scenario test that forces the primary provider into a delayed or error state to confirm the switch actually happens inside the 10-to-15-second budget. Logs need provider ID, attempt count, the reason for the switch (timeout, latency threshold, explicit error), and time-to-switch as required fields, so you can trace afterward which provider triggered switches, and how often. Since call content lives in those logs, PII-masking rules apply the same way whether or not a fallback fired.
Weekly, aggregate switch frequency by provider and by reason to track the leading causes, and treat thresholds like attempt_timeout and consecutive_latency_hits as tunable against observed switch patterns rather than fixed constants. Providers that trigger frequent switches should drop in chain priority; a provider that almost never triggers a switch can be a sign the threshold is too loose and is missing real degradation.
What to Check Right Now
LLM provider failover in a callbot loop starts with separating retry from failover. Pair hard-failure criteria like attempt_timeout and max_retry with degradation criteria like latency_threshold_ms and consecutive_latency_hits, log the switch reason and recovery time, and adjust chain order weekly — and whichever provider goes down, the callbot can move to the next path within the same turn, before the caller notices.
References
Fallback strategies — LiveKit Docs
Fallback Adapter — VideoSDK Docs
Ask AI about this article
The assistant has read this article. Ask anything — it answers from the text and says so when something isn't in it.
Loading the chat…