Choosing the speech and reasoning models for a voice agent

Model selection becomes confusing when every provider reports low latency and high accuracy, but each benchmark measures something different.

A voice system needs a recognizer that understands its callers, a reasoning model that follows its workflow, and a voice that can deliver the response clearly. The best component in isolation may produce a worse complete call.

The provider comparison is current to September 2026. I use it to decide what to benchmark: language support, tool behavior, streaming and cost have to be checked on the same customer task.

The cascaded design

In an STT–LLM–TTS system, each stage can be selected independently. The transcript becomes an explicit boundary for inspection, retrieval, policy checks and evaluation.

That modularity has costs. Recognition errors propagate into reasoning. Prosody and some non-verbal information disappear at the text boundary. Separate providers introduce separate network paths, quotas and failure modes.

For a workflow requiring precise entity capture and inspectable business actions, those explicit boundaries can be valuable. For a highly expressive, open-ended conversation, they may feel restrictive.

The native-audio design

OpenAI Realtime and Gemini Live provide native-audio conversation paths. They can process audio without requiring the application to serialize every turn through a separate recognizer and synthesizer. OpenAI documents both browser WebRTC and server WebSocket integration. OpenAI Realtime guide, Gemini live sessions.

This can improve conversational continuity and preserve information lost in plain text. The trade-off is less independent control over recognition and voice generation, provider-specific event semantics, and a need to inspect the audio directly during evaluation.

An auxiliary transcript should not be assumed to reveal exactly what the native model internally understood. Grade the audio behavior and the business result as well.

“Native audio is expensive and loses context” is too broad. Cost depends on the model and usage pattern. Context loss can result from finite windows, truncation, session renewal or faulty application state. Cascaded systems have context limits too.

Recognition candidates

Deepgram's current documentation lists Hindi in its multilingual Nova-3 and Flux configurations. Check the specific model and streaming mode when selecting a language route. Google documents Chirp 3 recognition separately from Chirp 3 synthesis. Matching product names do not mean matching APIs. Deepgram model/language reference, Google Chirp 3 STT.

Whisper and Qwen are examined in the next two articles. In particular, a checkpoint supporting English is not automatically a validated Singlish recognizer.

Reasoning candidates

For OpenAI, Gemini and Anthropic, the useful comparison is between pinned model configurations on the same voice workload, not between brand names.

OpenAI is a candidate where an application already uses its tool interfaces or wants to compare a text cascade with native audio. Gemini is a candidate for a similar cascade/native comparison within Google's stack. Anthropic is a candidate for the text reasoning and tool stage, including through Bedrock; do not describe that as an interchangeable native speech service.

Streaming implementations must preserve structured events. For example, Anthropic's streaming interface distinguishes content blocks and deltas, including incremental tool arguments. Concatenating every incoming string into spoken text would be an incorrect adapter. Anthropic streaming reference.

An open-weight text model is also worth evaluating when data locality, predictable serving or a narrow domain makes self-hosting attractive. It adds GPU scheduling, KV-cache management, model-update and incident-response responsibilities. A smaller parameter count is not sufficient evidence of a lower cost per successful call.

Evaluate the intelligence–latency frontier

A generic reasoning score rarely tests whether a model can:

• Ask one useful question instead of five.
• Preserve a caller's correction.
• Use the correct tool with exact arguments.
• Avoid speaking an approval before the tool confirms it.
• Recover from a failed lookup without restarting the conversation.
• Stay concise without skipping a required confirmation.

Define a local frontier with task success to maximize, and latency/cost to minimize. A candidate is dominated if another is at least as good on every chosen dimension and strictly better on one.

Let's work through a synthetic benchmark so the calculation is clear:

def dominates(a, b):
    no_worse = (
        a["success"] >= b["success"]
        and a["p95_ms"] <= b["p95_ms"]
        and a["cost"] <= b["cost"]
    )
    better = (
        a["success"] > b["success"]
        or a["p95_ms"] < b["p95_ms"]
        or a["cost"] < b["cost"]
    )
    return no_worse and better

candidates = [
    dict(name="A", success=.92, p95_ms=850, cost=.04),
    dict(name="B", success=.90, p95_ms=950, cost=.05),
    dict(name="C", success=.96, p95_ms=1200, cost=.08),
]
frontier = [
    x["name"] for x in candidates
    if not any(dominates(y, x) for y in candidates)
]
assert frontier == ["A", "C"]

Here B is dominated. Choosing A or C still requires a business decision about the extra correctness relative to latency and cost. In a real comparison, confidence intervals and minimum safety criteria come before declaring small differences meaningful.

Figure 5. Synthetic comparison. Invented model A achieves 92 percent success at 850 ms and four cents; B is dominated at 90 percent, 950 ms and five cents; C offers 96 percent at 1200 ms and eight cents. Scalable SVG.

Use separate frontiers for speech and the complete call

For STT, compare WER, critical-entity accuracy, finalization delay and inference cost on identical audio. Then run the shortlisted recognizers through the complete agent. A reduction in filler-word errors may improve WER without changing the business outcome; one lost negation can have the opposite effect.

For TTS, compare first playable audio, real-time generation rate, intelligibility and pronunciation. A pleasant voice can still read an amount or date incorrectly.

For the final system, compare task completion, useful-response delay, interruptions and total cost. Component scores help explain a result; they do not replace it.

Languages are evaluation slices

The model routing policy may use different recognizers or voices for these slices. Qwen3-ASR's published language list includes English and Hindi, but not Tamil or Telugu. It should therefore not silently become the universal replacement for all languages in this portfolio.

Keep the cost comparison at the right boundary

A text cascade may charge by recognition duration, LLM tokens and synthesized characters or audio duration. Native audio may charge for audio/text tokens or session duration, depending on the API. Repeated context and caching can materially affect the result. OpenAI voice cost accounting.

Use recorded usage per call, and include failures, retries and idle capacity. For a self-hosted service, allocate the warm GPU fleet across the actual delivered minutes, not the theoretical maximum.

The final article develops a worked cost model. The model-selection decision should feed that model with measurements rather than a vendor's best-case demo.

Build a comparison that answers a decision

A useful benchmark begins with a question such as: “Can a smaller reasoning model handle callback qualification without losing corrections, while reducing useful-response latency?”

Freeze the prompt, tools, scenario fixtures, recognition output and voice for that experiment. Change the reasoning model alone. Otherwise a better recognizer or shorter prompt may receive credit as an LLM improvement.

Then repeat the shortlisted configuration through audio. This second experiment captures interactions that a transcript-only test cannot: partial recognition, interruptions and phrase timing.

A proposed comparison record is:

{
  "configuration": "candidate-A",
  "model_revision": "pin-before-run",
  "prompt_revision": "qualification-v7",
  "scenario_set": "voice-scenarios-v2",
  "decoding_settings": {"maximum_output_tokens": 160},
  "trials_per_scenario": 3,
  "results": {
    "verified_success_rate": null,
    "critical_error_count": null,
    "p95_useful_response_ms": null,
    "usd_per_success": null
  }
}

Null means unmeasured, not zero. Store per-case results before calculating aggregate rankings.

Test instruction-following under speech constraints

The following constructed exchange separates fluent wording from correct behavior:

Caller: Compare the basic and premium plans, but don't book anything.

Weak response: The premium plan is a great choice. I'll reserve a consultation for you.

Better response: I can compare them. What matters most to you: price, coverage or support?

The weak response violates the action constraint before any complicated reasoning is required.

Other useful probes include an answer that must fit in two short sentences, a caller who changes language, and a tool result that contradicts an earlier assumption. Score whether the required behavior occurs, not whether the answer uses the same wording as a reference.

Express provider capability explicitly

Interoperability needs a capability check before a configuration is admitted:

from dataclasses import dataclass

@dataclass(frozen=True)
class SpeechCapabilities:
    languages: frozenset[str]
    streaming: bool
    supports_cancellation: bool

def validate_speech_choice(capabilities, language, needs_streaming):
    if language not in capabilities.languages:
        raise ValueError("Language not declared for this service")
    if needs_streaming and not capabilities.streaming:
        raise ValueError("A streaming service is required")
    return True

example = SpeechCapabilities(frozenset({"en", "hi"}), True, True)
assert validate_speech_choice(example, "hi", True)

This is an original simplified contract. A production capability record also distinguishes language detection, code-switching, timestamps, output codec, available voices and the scope of cancellation.

Do not silently map an unsupported language to English. An explicit configuration error is easier to fix than a fluent conversation in the wrong language.

Cascades preserve control; native audio preserves other signals

In a cascade, the application can inspect the exact recognized text before acting. It can also choose a specialist recognizer and a separate regional voice.

In native audio, intonation and non-textual cues may remain available to the model. That advantage is workload-dependent. It should be tested with examples such as hesitation, correction and overlapping speech, rather than assumed from the architecture name.

A fair experiment uses the same business fixture and outcome grader for both. Record differences in what is observable: a transcript emitted alongside native audio is not a guaranteed transcript of internal perception.

Hybrid systems are also possible. A native-audio front end can delegate a difficult business decision to a separate backend model or deterministic workflow. That adds another latency and consistency boundary. It should appear in the architecture and cost model rather than disappear behind the phrase “voice-to-voice.”

Budget reasoning, not just output tokens

A model can generate a concise answer after substantial internal computation. A limit on final spoken words does not necessarily limit time to first useful speech.

Where the API exposes reasoning-effort controls, compare settings on the real task. For narrow classification or a known workflow, a smaller model or deterministic branch may work better than a general high-reasoning configuration. For a complex ambiguous request, spending more time can reduce costly mistakes.

A routing policy should be evaluated as a system: include routing errors, extra model calls and fallback latency. An oracle that sends every difficult case to the right model is a useful upper bound, not a deployable result.

Next: making Whisper behave like a streaming service.