Why an 800 ms voice response is a systems problem

A pause in a voice conversation feels different from a loading indicator on a website. The caller cannot see whether the system is searching, disconnected or waiting for them to continue. After an unexplained silence, people often repeat themselves. The repetition then changes the input while the first answer is still being generated.

In our early-2025 client implementation, response latency was approximately 1.1 seconds, with 40 calls delivered concurrently and a maximum concurrency test of 100. We were aiming for sub-second responses, with 800 ms as the preferred budget.

The systems have advanced significantly since early 2025. Zoice's Pipecat architecture gives us more control over the conversation pipeline. Separately, the local TTS work added in 2026 lets us investigate synthesis quality, pacing and the path to self-hosted serving. I use the early numbers here to explain the decisions, not as a current performance ceiling.

An 800 ms response budget remains a useful worked example. A quick acknowledgment can help during a complicated lookup, but an immediate incorrect answer does not help anyone. Let's follow the time through the whole system.

Put the clock at the caller

Start with four timestamps. These are the ones I want on the trace:

t0  Last speech sample from the caller
t1  System commits the caller's turn
t2  First response audio leaves the agent
t3  First response audio reaches the listening endpoint

The most important interval is t3 − t0. If the dashboard starts at t1, it hides endpointing delay. If it stops at t2, it hides output transport and playback buffering.

On a telephone connection, t3 is usually unavailable from the application server. A playback acknowledgment can improve the estimate, but its exact meaning depends on the carrier. An instrumented receiving endpoint or a loopback recording is needed to measure acoustic end-to-end delay. Label server-side proxies accordingly.

Pipecat provides service metrics and a user-to-bot latency observer. These are useful instrumentation points, but a framework speaking event should not automatically be described as audio heard at the far end. Pipecat metrics, latency observer.

A concrete budget

For a warm turn without a tool lookup, I would split an 800 ms budget like this:

“Residual” matters. Recognition already runs while the user is speaking. Counting the entire utterance-processing time again would double-count work that overlaps with the call.

The model budget ends at the first speakable phrase, not the first token. A token such as “The” rarely produces a sensible independent speech segment. Measure the delay introduced by text aggregation as part of the LLM-to-speech path.

Nor can independently measured component p95s simply be added and called an end-to-end p95. The components can be correlated, and some overlap. Record per-turn traces and compute the total distribution directly.

Figure 2. Proposed budget. Eight serial budget components total 800 milliseconds; the largest allocations are endpointing at 220 ms and first speakable phrase at 180 ms. Scalable SVG.

Silence is evidence, not a complete turn detector

Think about a caller saying:

I need the policy for my mother … she is sixty-two … and has diabetes.

A fixed silence threshold can terminate the turn after “mother,” causing a premature question or recommendation. Increasing that threshold avoids some false endpoints but makes every short answer slower.

A better controller combines acoustic speech detection with conversational evidence:

• Has speech actually ended, or is there low-energy speech?
• Does the partial transcript end in a completed answer?
• Was the system asking for a short confirmation or a long explanation?
• Is the caller spelling a number, an address or an unfamiliar name?
• Has a maximum waiting time been reached?

The controller needs to separate a quick backchannel from a committed answer. A brief “mm-hm” may be appropriate while listening, but it must not mark the customer's unfinished statement as complete or trigger a business action.

Speculation needs a cancellation boundary

It is possible to begin generating from a stable partial transcript. That reduces delay when the final words do not change the meaning. It also creates wasted generation and the risk of answering the wrong question.

Speculative work should be tagged with a transcript revision and generation ID. It can prepare a read-only lookup, but should not send a message, place an order or issue a refund. Only committed input may cross the action boundary.

For example:

Partial: "I want to cancel..."
Speculation: retrieve cancellation policy
Final: "I want to cancel the callback, not the policy."
Action: cancel callback only after confirmation and validation

The retrieval can be discarded cheaply. An incorrectly cancelled policy cannot.

Barge-in is a distributed cancellation problem

When the caller begins speaking, there may already be:

1. An LLM request generating more tokens.
2. A TTS request turning previous tokens into audio.
3. Locally queued audio.
4. Audio buffered at the telephony provider.
5. A business operation that has already committed.

Cancelling the LLM solves only the first item. The cancellation path needs to increment a generation epoch, stop generation and synthesis, clear local playback, send the provider's clear command, and reject late packets carrying the old epoch.

Here is the stale-audio guard in a compact form:

from dataclasses import dataclass

@dataclass(frozen=True)
class AudioChunk:
    epoch: int
    pcm: bytes

class PlaybackEpoch:
    def __init__(self):
        self.current = 0

    def interrupt(self) -> int:
        self.current += 1
        return self.current

    def accepts(self, chunk: AudioChunk) -> bool:
        return chunk.epoch == self.current

gate = PlaybackEpoch()
old = AudioChunk(gate.current, b"previous answer")
gate.interrupt()
assert not gate.accepts(old)
assert gate.accepts(AudioChunk(gate.current, b"new answer"))

Give one transport writer ownership of clear commands and audio writes. Otherwise, a packet can pass this check, race with an interruption and still enter the provider's buffer. The epoch check and the serialized writer solve different parts of the problem.

In the Bolna fork we modified for the client, this work involved sequence IDs, task cancellation and Plivo audio clearing. In Zoice, the same concern runs through Pipecat's frame and transport lifecycle: an interruption has to reach every stage that can still produce or play the old answer.

What was said is different from what was generated

Suppose the assistant generates:

Your refund is approved and will arrive in three working days.

The caller interrupts after “Your refund…”

The conversation record must preserve two facts: whether the backend actually approved the refund, and how much of the explanation was played. Truncating the entire business result because speech was interrupted is incorrect. Keeping the full sentence as if it was heard is also incorrect.

Store generated text, submitted audio and acknowledged playback separately. Where reliable word-to-audio alignment is unavailable, mark delivery as partial or uncertain. A subsequent response can then say, “The refund was approved. Shall I explain the timing?” without assuming the caller heard the earlier sentence.

Optimize the tail

The first implementation task is instrumentation, followed by a latency distribution split by language, provider, tool use, call duration and concurrency.

The most useful experiments change one variable:

Cold connections, GPU warm-up and first-use kernels should be measured separately from warm turns. They still matter: a campaign's first hundred calls should not all become involuntary warm-up requests.

The eventual acceptance criterion should combine latency and correctness. “800 ms while preserving turn-taking and task success” is a useful specification. “800 ms” by itself is an incomplete one.

Read a latency trace without fooling yourself

Here is a worked warm-turn trace. All times use one monotonic clock in a test harness:

0 ms     final caller speech sample
40 ms    final relevant media packet arrives
260 ms   turn committed
320 ms   recognition finalized
340 ms   context ready
520 ms   first speakable phrase
660 ms   first synthesized audio chunk
740 ms   first response audio heard in loopback receiver

The trace totals 740 ms, leaving 60 ms of the proposed 800 ms budget. An LLM-only dashboard would display 180 ms and omit most of the customer's wait.

This small helper makes the timing boundary explicit:

def summarize_turn(trace):
    required = (
        "speech_end_ms", "turn_commit_ms", "phrase_ready_ms",
        "first_audio_ms", "playback_start_ms",
    )
    values = [trace[key] for key in required]
    if values != sorted(values):
        raise ValueError("Invalid timeline or incompatible clocks")
    return {
        "endpoint_delay_ms": values[1] - values[0],
        "phrase_delay_ms": values[2] - values[1],
        "synthesis_delay_ms": values[3] - values[2],
        "delivery_delay_ms": values[4] - values[3],
        "caller_response_ms": values[4] - values[0],
    }

example = dict(
    speech_end_ms=0, turn_commit_ms=260, phrase_ready_ms=520,
    first_audio_ms=660, playback_start_ms=740,
)
assert summarize_turn(example)["caller_response_ms"] == 740

The monotonic ordering is appropriate for this specific serial trace. In a speculative pipeline, phrase generation can start before turn commitment; that requires a different trace model. Do not “fix” such a trace by sorting timestamps.

Across machines, monotonic clocks are not shared. Record local durations and parent/child span relationships, or use a synchronized measurement endpoint with a stated error bound. Subtracting unrelated host clocks can create negative latency or apparent improvements.

Echo, noise and overlapping speech

Suppose the assistant asks a question through the caller's loudspeaker. That sound re-enters the microphone, and the recognizer hears the assistant's own words.

The resulting failure can look like an LLM problem: repeated questions, self-interruption or answers to its own prompt. The first diagnostic is to compare input audio with the recent output waveform, not rewrite the prompt.

Acoustic echo cancellation uses a reference of the played signal to reduce its return into the input. Noise suppression targets other unwanted sound. Voice activity detection decides whether speech is present. They solve related but different problems.

Test at least headset, handset and speakerphone conditions. If provider-side processing is already active, evaluate the combined path before stacking another aggressive filter. A filter that removes quiet speech can damage names and short negations.

An illustrative barge-in case:

Agent: Your next instalment is—

Caller: Nahi, amount nahi. Due date bataiye.

Expected behavior: Stop the amount explanation, interpret “not the amount; tell me the due date,” and answer the revised question.

A detector that requires a long uninterrupted phrase may miss the initial “Nahi.” A detector that fires on every noise may repeatedly cancel valid answers.

Tune a detector with an explicit cost

For a proposed speech detector, use separate start and stop conditions—hysteresis—rather than toggling on a single threshold every frame. For example, a development experiment could require 80 ms of strong speech evidence to start and 200 ms of low speech evidence to stop, while retaining 120 ms of pre-roll.

Treat these as tuning inputs and compare them on your own recordings. Pre-roll protects the beginning of a word; it also adds buffered context. Semantic turn completion can still delay commitment after acoustic speech stops.

Choose the configuration using false starts, missed short interruptions and truncated-word errors. Average response time alone rewards an over-eager detector.

Two latency numbers for a slow tool

Suppose a support lookup takes 2.4 seconds. The agent says “I'm checking the delivery record” after 500 ms and gives the answer after 2.9 seconds.

Record both:

acknowledgment latency: 500 ms
useful-answer latency: 2,900 ms

That acknowledgment can make the wait understandable. It cannot honestly turn the lookup into a 500 ms answer.

The same distinction applies to hold music, cached greetings and repeated fillers. They are experience decisions, not substitutes for measuring useful progress.

Next: the Bolna architecture.