The Last 200 Milliseconds: Engineering Latency in Real-Time Voice AI
A natural phone conversation lives or dies on response time. Here's how we engineer the latency budget for a real-time voice AI system — where the milliseconds go, and how to win the ones that matter.
When you build a voice AI system that answers real phone calls, you learn quickly that the technology people notice is not the intelligence. It is the timing. A model can produce a perfect answer, but if it arrives a beat too late, the caller has already started talking, or worse, assumed the line went dead. Natural conversation runs on a response rhythm of roughly 200 to 300 milliseconds between turns. Miss it, and the interaction feels wrong in a way callers can't articulate but always sense.
This is a field note on where those milliseconds go, and how we spend a latency budget when the target is "sounds like a person, not a system."
The pipeline is a chain, and the chain is the enemy
A real-time voice agent is a pipeline of at least three stages. Speech has to be transcribed as it arrives (speech-to-text). The transcribed intent has to be reasoned over and a response generated (the language model). The response has to be turned back into audio (text-to-speech). Each stage adds latency, and naively they add in series — the caller waits for the sum.
The first structural decision is to stop treating these as discrete steps. If you wait for a complete transcription before you start thinking, then wait for a complete thought before you start speaking, you have already lost. The winning architecture streams: transcription emits partial results as the caller is still talking, the language model begins generating as soon as intent is clear, and speech synthesis starts vocalising the first clause while the rest of the sentence is still being generated. The pipeline overlaps instead of queues. That single shift — from serial to streaming — is worth more than any individual component optimisation.
Where the budget actually goes
Assume a target of around 500 to 800 milliseconds from the moment the caller stops speaking to the moment they hear the first syllable of a response. That sounds generous until you itemise it.
Endpointing — detecting that the caller has actually finished, not just paused mid-sentence — is deceptively expensive. Wait too long and every response feels sluggish. Cut it too short and you interrupt people, which is worse. This is not a fixed timeout; it is a judgement about speech rhythm, and getting it right reclaims more perceived latency than almost anything downstream, because it sits at the very front of the turn.
Time-to-first-token from the language model is the number that dominates the felt experience. The caller does not need the whole answer computed before hearing anything; they need the first words fast. Optimising for first-token latency — through model choice, prompt structure, and keeping the context lean — matters more than optimising total generation time, because synthesis can begin the moment the first clause exists.
Synthesis start time is the last gate. A text-to-speech engine that needs the full sentence before producing audio reintroduces the serial delay you worked to eliminate upstream. Streaming synthesis, chunked at clause boundaries, keeps the overlap intact all the way to the caller's ear.
Add network round-trips between these services and the budget is gone faster than intuition suggests. Every hop between systems is latency you are paying for repeatedly, on every single turn, thousands of times a day.
Interruptions are a first-class feature, not an edge case
Real conversations are not turn-based in the clean way software wants them to be. People interrupt. They say "no, the other one" halfway through your sentence. A voice system that cannot be interrupted feels robotic no matter how good its answers are, because it violates the most basic rule of talking to someone.
Handling this — barge-in, in the jargon — means the system has to listen while it speaks, detect genuine interruption (distinct from a backchannel "mm-hm"), stop synthesis mid-word, discard the response in flight, and re-plan around what the caller just said. Every one of those steps is a latency-sensitive operation happening while another latency-sensitive operation is already running. It is the hardest part of the system to get right, and it is non-negotiable, because it is the difference between a conversation and a voicemail menu.
What this work teaches
The broader lesson generalises past voice. Real-time AI is a systems problem before it is a model problem. The quality of the underlying model sets a ceiling on how good the answers can be, but the architecture around it — how you stream, where you place compute, how you budget and reclaim milliseconds, how you handle the messy human behaviours that don't fit the happy path — determines whether users experience that quality at all.
You do not feel a model's benchmark score on a phone call. You feel the pause before it speaks. Engineering that pause down to something human is where the real work lives, and it is almost entirely invisible when it's done well. That invisibility is the point.