Why a Single End-to-End Latency Number Can't Localize the Fault
A callbot loop runs at least three API calls per turn, sequentially or streamed: STT turns audio into text, the LLM generates a reply, and TTS synthesizes that reply back into audio. Those three services come from different vendors, carry different latency profiles, and fail in different ways. Logging one turn-latency number tells you nothing about which of the three actually slowed down — callbot observability needs to be one level more granular than a text agent's, where a single LLM call is usually the whole chain.
The Standard Doesn't Define Voice Stages Yet
OpenTelemetry's GenAI semantic conventions define spans for inference, embeddings, retrieval, and tool calls, but every one of those span types is still marked Development status. Audio isn't a dedicated span at all — it only shows up as input/output token counts (audio input tokens, audio output tokens) inside the inference span. There's no standard span or attribute for STT, TTS, VAD, or barge-in. Any team that wants to observe a callbot loop today has to design its own span schema.
From Design to Operations: A Roadmap for Stage-Level Attribution
Observability design starts by freezing a span-naming scheme in a document before any code gets written. Treat one call as the root span, open a child span per turn, and nest STT, LLM, tool-call, and TTS as children under that — a four-level structure that stays manageable even without a standard to lean on. A reasonable starting target: 100% span coverage (every turn leaves all four levels behind) and an unattributed-failure rate (failures you can't pin to a stage) under 5%.
Without a standard span to reach for, the common mistake is logging only turn-level latency and never splitting it by stage. An LLM regression and a backed-up TTS synthesis queue then show up as the same "turn latency went up" line on the dashboard, and the on-call engineer ends up retrying the wrong thing. Restart STT when the LLM was the one that slowed down, and the root cause stays untouched while latency doubles.
Once stages are split apart, recovery branches need to differ by stage too. An STT delay should reconnect while keeping the partial-transcript cache; an LLM timeout should get one short retry against the cached context before handing off to a human; a TTS failure is usually better handled by resynthesizing the whole turn than by splicing new audio onto what already played. Apply one retry policy across all three stages and at least one of them is always over-reacting.
Before shipping, run scenario tests across a full call — start to end — and confirm the four-level span structure is actually populated at every stage, not just some of them. If transcript text or synthesized copy ends up sitting in span attributes verbatim, PII flows straight into your logging backend, so mask or route the attributes that carry STT output and LLM input to storage with a short retention window before you ship.
Instrumentation itself can add latency, and that belongs on the pre-launch checklist too. Push span creation and attribute writes through an async exporter instead of the synchronous request path, or turning observability on becomes the thing that makes turns slower.
Once in production, aggregate latency distributions weekly by stage pair (STT→LLM, LLM→TTS, and so on) to find which transition is actually dragging p95 up. Since there's no standard schema to anchor to, attach a version field to your own attribute names and span structure — change the schema without one and old traces mix into new aggregates in ways that quietly break your numbers.
Once stage attribution is in place, the next step isn't chasing the unattributed-failure rate lower forever — it's triggering an automatic rollback or a fallback provider switch off a single span tag on a specific transition. Until the standard stabilizes, this homegrown schema is the real observability spec your callbot loop runs on.
A Checklist You Can Use Today
Freeze a call (root) → turn → stage four-level span scheme first, and set 100% span coverage and under-5% unattributed-failure rate as deploy gates. Apply a different recovery branch per stage, mask or short-retain any attribute that carries transcript text, and confirm before shipping that instrumentation itself runs on an async path. Until the standard settles, a versioned, homegrown span structure is what actually defines your callbot loop's observability spec.