Evaluating a voice agent: listen to the call, inspect the action

The difficult part of voice evaluation is that a correct-looking transcript can conceal a bad conversation.

The assistant may have spoken over the customer. It may have taken four seconds to respond. It may have generated the right answer but played only half of it. Or it may have said that a refund was processed when the backend contains no refund.

Hamel Husain's evaluation writing offers a useful starting principle: inspect real failures and turn them into concrete, domain-specific checks. For voice systems, that inspection must include the audio timeline and the resulting system state. Hamel's evaluation guide.

Let me start with what we built in Zoice, then work through how the call record becomes a useful evaluation.

What we developed in Zoice

The evaluation foundation has three parts: instrumentation during the conversation, a durable record after the call, and workflow-specific extraction from that record. I keep those separate because a fast model response, a useful summary and a correctly completed task are different things.

The extraction questions are configurable by workflow. A qualification call and a complaint call should not be forced into the same generic summary. For a tax enquiry, useful fields include the income categories discussed, whether a specialist was requested and the agreed next step. For an appointment enquiry, the important fields are different.

This is the practical connection between scoping and evaluation: the fields agreed with the customer at the start become the fields reviewed after the call.

Instrumentation has to be connected to the pipeline

Pipecat can emit service timing and usage metrics when the corresponding options are enabled. A processor can then observe those frames and update the application's metrics. Metrics collection and metrics export are separate responsibilities. Pipecat metrics.

The Kuzushi Labs metric publisher labels histograms by processor and model. Here is the core update, condensed from the Zoice processor:

def publish_ttfb(metric, histogram):
    labels = {
        "processor": metric.processor,
        "model": metric.model or "unknown",
    }
    histogram.labels(**labels).observe(metric.value)

The original timing buckets are 0.1, 0.5, 1, 2, 5, 10, 30 and 60 seconds. That range exposes severe stalls. For distinguishing a 700 ms response from a 900 ms response, add finer buckets around the operating target.

Keep call IDs in traces and event records rather than as a histogram label. The fleet dashboard needs bounded labels; a particular failure investigation needs the call identity.

There is one integration check I consider essential: exercise the cascaded and native-audio branches separately and verify that each emits the expected events. Importing a processor class or enabling a metrics flag is not enough. Attach the observer/processor at the intended boundary, send a known fixture and assert that the event reached the sink.

Also test the event contract itself. Every emitted event type must exist in the schema; unknown metric types should be skipped or handled explicitly. The observation path should not be able to crash the audio path merely because a provider adds a new metric.

The post-call path we use for structured results

Zoice's recording processor loads the call and pipeline configuration, stores the recording, derives a text representation, and applies the configured extraction questions. It then persists the recording, transcript and extraction records and prepares the configured webhook result.

There are two text views worth keeping distinct. The live conversation messages describe what the conversational system had available. A fresh pass over the recording provides another view for post-call analysis. If the two disagree, that disagreement is useful evidence rather than something to silently overwrite.

One important implementation detail is that the existing post-call Whisper path uses the audio translation endpoint. That produces English text, which is useful for a common-language extraction workflow. It is not a same-language reference transcript for Hindi or Hinglish WER. The endpoint's behavior is documented in the OpenAI audio translation reference.

For recognition evaluation, use the original audio and a separately annotated reference in the declared script. For post-call classification, an English translation can be useful. For judging whether the agent followed its actual input, retain the live model transcript. These three views answer three different questions.

The same applies to speaker labels. If a recording-derived segment has only a generic speaker label, do not infer that it establishes who interrupted whom. Use channel-separated audio or aligned speaker/turn events for that judgment.

Extraction becomes evaluation only when it has a target

An LLM returning a plausible JSON object does not prove the values are correct. Start with a field contract:

{
  "workflow": "tax-enquiry",
  "fields": {
    "income_categories": {
      "type": "array",
      "allowed": ["salary", "freelance", "investment", "other"]
    },
    "requested_specialist": {"type": "boolean"},
    "next_step": {
      "type": "string",
      "allowed": ["callback", "information_only", "no_follow_up"]
    }
  },
  "missing_value": null
}

Then compare extraction against a reviewed call record. For critical fields, retain the supporting turn or audio span so a reviewer can see why the value was accepted.

The original extraction path has both missing-field handling and fallback behavior. At the evaluation boundary, normalize missing values consistently. A string such as INFO_NA must not accidentally be interpreted as a positive answer merely because a nonempty string is truthy.

Here is a small validator for a boolean extraction field:

def evaluate_boolean_field(raw_value, expected):
    if expected is not None and type(expected) is not bool:
        raise ValueError("Expected must be a boolean or None")
    missing = raw_value is None or raw_value == "INFO_NA"
    valid = missing or type(raw_value) is bool
    normalized = None if missing else raw_value
    return {
        "valid_type": valid,
        "missing": missing,
        "matches_reference": valid and normalized is expected,
    }

assert evaluate_boolean_field("false", False)["valid_type"] is False
assert evaluate_boolean_field(False, False)["matches_reference"] is True
assert evaluate_boolean_field("INFO_NA", True)["matches_reference"] is False

This separates invalid output, missing information and an incorrect value. A single “extraction succeeded” status would conceal all three.

How I use the framework to investigate a call

Suppose a caller says, “Salary income hai, freelance bhi, lekin abhi plan mat book karo.” The meaning is salary plus freelance income, with an explicit instruction not to book a plan.

The review follows the causal chain:

1. Listen to the audio. Was the final restriction audible, or was it clipped by the transport?
2. Read the committed transcript. Did the recognizer preserve “mat book karo”?
3. Inspect the model input. Did context assembly retain the restriction after any summarization?
4. Inspect the decision and tool events. Was a booking attempted, or only a callback proposed?
5. Check business state. Did any operation actually commit?
6. Listen to the delivered response. Did the caller hear a confirmation that matches the backend?
7. Check the extracted result. Did the summary preserve the restriction and the requested next step?

If recognition lost the restriction, changing a TTS voice will not fix it. If the transcript was correct but a forbidden tool ran, improving WER will not fix it. The point of joining these records is to identify the first boundary where the behavior diverged.

For a slow call, the same method separates endpoint wait, recognition finalization, model generation, synthesis and playback. A service-level TTFB histogram helps locate a bottleneck; the aligned turn trace explains the customer's wait.

Turn reviewed failures into regression cases

For each reviewed failure, save the minimal reproducer: permitted audio or a synthetic replacement, expected transcript/intent, relevant context, backend fixture and the required outcome. Preserve the failure category and the configuration revision that produced it.

Use three replay modes:

An evaluation run needs an ID, configuration hash, fixture revision and grader revision. Persist a result even when a component times out; otherwise the slowest failures disappear from the denominator. A missing recording, failed extraction and failed business action should be separate statuses.

Use the representative review set to estimate normal performance. Use the accumulated failure set to test whether known problems return. The second set should get harder over time, so its raw pass rate is not directly comparable to a randomly sampled production success rate.

Connect a release decision to an explanation

The release view should answer four questions: did task success improve, did critical errors increase, did the conversation remain responsive, and what happened to cost?

Keep a call-by-call diff for changed outcomes. “Candidate passed 92%” is less useful than knowing which scenarios moved from failure to success, which regressed, and whether those regressions are concentrated in a language or workflow.

For TTS, bring the separate 2026 local experiments into this framework through pronunciation and audio-timing fixtures. For a new recognizer, bring in the public-data adaptation record and the same call-level tests. The framework then gives each component experiment a path back to the customer conversation.

Sample the calls to review

Start a review with a sample of 100 calls across languages, outcomes and traffic periods. Add every known serious failure even if it is rare. Keep the representative sample separate from the enriched failure sample when estimating rates.

For each call, align:

caller audio
recognition revisions and committed transcript
turn/interrupt events
model input and output events
tool request, validation and backend result
generated audio
playback evidence
final business state

Reviewers should identify the earliest causal failure, not simply the last visible symptom. An incorrect recommendation may have begun with a lost negation in STT, a stale product document, a policy error or an unauthorized tool response.

Figure 11. Reference evaluation design. Audio, event and business evidence combine into an aligned timeline; graders then assess task correctness, critical errors and conversation quality independently. Scalable SVG.

Use a failure taxonomy

Use the categories to locate the first failure in the chain before assigning a fix.

Build evaluation in layers

Deterministic checks cover codec conversion, transcript assembly, argument parsing, authorization and idempotency. They are ordinary software tests.

Component datasets evaluate STT, TTS, retrieval and model decisions under controlled inputs.

Conversation simulations test multi-turn behavior, corrections, interruptions and tools. Run both transcript-level and audio-level versions to identify where the voice interface introduces failures.

Real-call review checks whether simulation represents actual speakers, acoustic conditions and customer behavior.

Production monitoring detects changes after release. A passing offline suite is not a permanent certificate.

The unit tests matter, but they cannot prove that an agent understands a naturally interrupted Hinglish request. Conversely, an LLM judge should not be used to decide whether an idempotency key was duplicated when a database query can establish that directly.

A worked refund scenario

Use a sandbox business environment with a known order, policy and expected result:

id: wrong-item-refund-001
language_slice: hinglish
initial_state:
  customer_authenticated: true
  delivered_item_matches_order: false
  within_refund_window: true
  refund_count: 0
caller_goal: request the permitted refund
perturbations:
  - pause before the order number is complete
  - interrupt an explanation to correct one digit
expected:
  correct_order_selected: true
  refund_count: 1
  unauthorized_mutation_count: 0
  unsupported_denial: false
  spoken_result_matches_backend: true

The policy fixture determines eligibility. Keep that decision outside the model.

A deterministic outcome grader can then inspect the sandbox:

def grade_refund(final_state, expected_order):
    refunds = final_state["refunds"]
    exact = [
        x for x in refunds
        if x["order_id"] == expected_order and x["status"] == "succeeded"
    ]
    return {
        "correct_refund_once": len(exact) == 1 and len(refunds) == 1,
        "no_unauthorized_writes":
            final_state["unauthorized_mutation_count"] == 0,
    }

A separate audio review decides whether the explanation was understandable and whether the interruption was handled naturally. Do not merge these checks so early that a pleasant voice compensates for a wrong refund.

WER is necessary and incomplete

Corpus WER is:

WER = (substitutions + deletions + insertions) / reference words

It can exceed 100% when insertions are numerous. Compute total errors divided by total reference words, not the unweighted mean of per-sentence percentages.

For Hinglish, publish the normalization and script convention. Also track critical entities and intent-changing terms. A single missing “नहीं” can matter more than several punctuation or transliteration differences.

The completed Whisper experiment in article 7 demonstrates why both are needed: its WER improved substantially while remaining unsuitable for customer deployment. A relative improvement without the absolute result is misleading.

Measure the experience from the timeline

Recommended metrics include:

For a worked release gate, set ordinary warm turns to a median first response at or below 800 ms and p95 at or below 1.5 seconds, then tune the thresholds to the workflow. Tool-heavy turns need separate budgets and an honest acknowledgment policy.

Keep all attempts in the accounting. Report latency among completed turns together with timeouts and failures; otherwise dropping the slowest calls can make the chart look better.

Simulators can be wrong too

A user simulator may accept a bad answer, forget its own goal or pronounce a test identifier incorrectly. That is a simulator failure, not necessarily an agent failure.

Validate generated conversations against the scenario. Retain rejected simulations and the reason for rejection. Use native-speaker recordings alongside synthetic speech so the evaluation does not become a test of one synthesizer's accent.

Two recent research references help frame this:

• τ-Voice combines grounded task completion with full-duplex interactions and realistic audio. Its reported results show a substantial gap between text and voice performance in its test conditions; those numbers should not be generalized to every deployment. τ-Voice.
• EVA-Bench includes simulation validation and separates task accuracy from conversational experience, with multiple trials. This is useful when designing a benchmark that must distinguish a lucky success from reliable behavior. EVA-Bench.

I find both useful for separating task completion from conversational experience and for checking whether a simulator is behaving like a credible caller.

Calibrate model judges

Use an LLM or audio model for subjective dimensions only after comparing it with human labels. Create a rubric with positive and negative examples, including polite failures and abrupt but correct handoffs.

Measure false passes on critical errors. A judge with good overall agreement can still be unsuitable if it overlooks unauthorized actions. Blind the judge to which model or version produced the call when practical.

Keep source evidence with each judgment. “Empathy: 4/5” without the relevant turn and audio segment is difficult to audit.

A practical first evaluation release

An initial suite can contain 240 scenarios: 40 each for ordinary qualification, information lookup, booking, complaints, corrections/interruptions and failure recovery. Apply selected language and acoustic variants rather than multiplying every combination indiscriminately.

Run three trials per selected configuration and report pass@1 plus repeatability. Do not report best-of-three success as if it were the customer's probability of success on their one call.

Bootstrap comparisons at the scenario or speaker group level, depending on the experiment. Related variants are not independent observations.

Zero critical failures in 300 independent trials still gives an approximate one-sided 95% upper bound of 1% using the rule of three. It does not prove a zero failure rate. Correlated tests weaken the inference further.

The inspectable evaluation package

Save scenario versions, fixtures, model and prompt revisions, audio references, raw events, backend snapshots, grader versions, per-case judgments, aggregate results and known limitations. Public examples should use synthetic identities and permitted recordings.

The first dashboard should make three things impossible to miss: whether the task was completed, whether a material error occurred, and how the conversation felt to the caller. Business impact comes after those measurements have a dependable meaning.

One missing word can reverse the instruction

Take an example six-token example:

Reference: I do not want this renewed
Hypothesis: I do want this renewed

There is one deletion and no other word error: WER is 1/6, or 16.7%. Yet the business meaning has reversed. A second recognizer that substitutes a harmless product-name spelling can obtain the same WER with a different operational risk.

Keep the ordinary WER calculation because it is reproducible. Add an intent-changing-error label and entity-specific checks. Do not invent a weighted score whose favorable average hides a lost negation.

The following dynamic-programming example calculates edit distance for token sequences. It returns the total minimum edit count, not a full substitution/deletion/insertion alignment:

def word_edit_distance(reference, hypothesis):
    previous = list(range(len(hypothesis) + 1))
    for i, ref_word in enumerate(reference, start=1):
        current = [i]
        for j, hyp_word in enumerate(hypothesis, start=1):
            current.append(min(
                previous[j] + 1,       # delete reference token
                current[j - 1] + 1,     # insert hypothesis token
                previous[j - 1] + (ref_word != hyp_word),
            ))
        previous = current
    return previous[-1]

reference = "I do not want this renewed".split()
hypothesis = "I do want this renewed".split()
assert word_edit_distance(reference, hypothesis) == 1

For a corpus, sum these counts and divide by the total reference-token count. If that denominator is zero, report the condition separately rather than quietly assigning zero WER. For multilingual results, version the normalization that precedes tokenization; punctuation removal and transliteration can materially change the score.

A useful fixture contains more than a prompt

An audio fixture should bind the recording to expected events and business state:

{
  "fixture_id": "illustrative-callback-correction-01",
  "recording_kind": "synthetic-example-not-yet-recorded",
  "language_slice": "hi-en",
  "caller_text": "नहीं, Thursday नहीं, Friday को call कीजिए।",
  "expected_final_preference": "Friday",
  "forbidden_effect": "unreconciled_duplicate_booking",
  "audio_path": null,
  "audio_sha256": null,
  "speech_end_ms": null,
  "interrupt_on_agent_phrase": "Thursday at",
  "expected_operation_count": 1
}

The phrase means “No, not Thursday; please call on Friday.” The null fields deliberately identify an unrecorded fixture. A recorded release case must fill them with permitted audio and actual timestamps.

Test both a clear studio recording and a telephone-channel version. Add background noise and overlap as separate variants so the source of degradation can be identified. A synthetic accent does not establish performance on the corresponding population.

An event-level replay can deterministically test operation ordering. Audio replay tests recognition and turn-taking as well. Neither substitutes for a real full-duplex session when evaluating echo, carrier buffering or network variation.

Calibrate the judge on the failure that matters

Imagine a synthetic calibration set of 100 calls. Human reviewers identify ten critical failures. A model judge catches six, misses four, and incorrectly flags five of the ninety acceptable calls.

Overall agreement is 91%, which sounds strong. Critical-failure recall is only 60%, and four of ten dangerous failures passed the judge. The judge's flag precision is 6/11, or 54.5%. This example explains why a headline agreement score is insufficient.

Review missed failures and update the rubric before increasing reliance on the judge. Keep a held-out calibration set so rubric development does not become memorization. Where the failure is a wrong database mutation, prefer a deterministic backend check to subjective grading.

Compare candidates with paired evidence

Run the same scenarios, speaker recordings and controlled perturbations against both candidates. Compute each scenario's change in success or delay, then aggregate. If a scenario has three stochastic runs and five acoustic variants, do not pretend that all fifteen are unrelated customers.

For uncertainty estimates, resample independent scenario or speaker groups with their associated trials intact. Report effect size and interval, along with the number of groups. A small average gain accompanied by more critical failures should fail a gate even if a composite score rises.

Language slices need enough examples to support their own conclusions. A pooled improvement can hide deterioration on Tamil if most test calls are English. Keep per-slice counts visible and mark sparse slices inconclusive.

Release gates are ordered

A proposed release proceeds through:

1. Deterministic correctness: codecs, state transitions, authorization and idempotency.
2. Component quality: recognition, synthesis and retrieval on fixed datasets.
3. Conversation behavior: corrections, interruptions, task completion and failure recovery.
4. Load behavior: the same important cases at realistic concurrency.
5. Limited live operation: reviewed calls, measured outcomes and a rollback path.

Do not replace the fourth step with a test that merely opens 100 silent connections. Include overlapping speech, long turns, tool delays and natural call endings. Conversely, a heavy failure-injection suite should not be used to estimate ordinary customer failure prevalence.

The release report should say what changed, what worsened, which slices remain uncertain and why the candidate passed. That is the evaluation layer a recruiter or another engineer can inspect and challenge.

Next: cost and business impact.