Speech synthesis that can survive a real customer call

A voice can sound natural in a demonstration and still fail when it has to read a premium amount, a customer name or a mixed-language sentence.

In 2026, we added a separate local TTS workstream at Kuzushi Labs under Solo & Silicon. That work runs Kokoro, Qwen3-TTS and Chatterbox Turbo through MLX Audio on Apple Silicon. It sits alongside the earlier Zoice calling system, which already used managed speech providers.

I will keep those two contexts clear: Zoice's provider and telephony integration, and the 2026 local synthesis experiments. Then I will work through the additional serving and adaptation decisions needed to turn a local voice into an interactive service.

The local TTS comparison added in 2026

The Solo & Silicon comparison uses the same English dashboard narration across three models. The local environment uses Python 3.12.9, MLX Audio 0.4.8 and MLX 0.32.0.

All three saved outputs are mono WAV at 24 kHz. These are audio durations, which help compare pacing for the same narration. Generation speed needs the wall-clock measurement as well: a shorter output does not automatically mean a faster model.

The model identifiers and waveform metadata are kept in the local comparison record. The underlying packages are MLX Audio and the corresponding Kokoro, Qwen3-TTS and Chatterbox Turbo conversions.

The launch commands in that environment are:

python -m mlx_audio.tts.generate \
  --model mlx-community/Kokoro-82M-bf16 \
  --voice af_heart --text "$TEXT" --file_prefix samples/kokoro

python -m mlx_audio.tts.generate \
  --model mlx-community/Qwen3-TTS-12Hz-0.6B-Base-4bit \
  --ref_audio reference_voice.wav --text "$TEXT" \
  --file_prefix samples/qwen3tts

python -m mlx_audio.tts.generate \
  --model mlx-community/chatterbox-turbo-fp16 \
  --ref_audio reference_voice.wav --text "$TEXT" \
  --file_prefix samples/chatterbox

Here, TEXT is the fixed narration string, and the reference recording is the voice sample supplied to the two reference-conditioned models. Kokoro uses its built-in voice. That distinction matters when listening: this comparison answers “which output works best for this narration?” more directly than “which model reproduces the same speaker best?”

What changes between these models?

Kokoro's small parameter count makes it a useful compact baseline. The selected af_heart voice is an English voice; the upstream project also provides other language/voice combinations, including Hindi. Choose the actual language configuration rather than sending every script to one English voice. Kokoro model card.

Qwen3-TTS Base conditions generation on reference audio. The 0.6B model in the local comparison uses a four-bit MLX conversion. Quantization reduces weight storage, but the service still needs room for activations, codec state and output buffers. Its upstream language list includes English, but not Hindi, Tamil or Telugu; those languages need their own synthesis route. Qwen3-TTS and Qwen3-ASR are separate models with different coverage. Qwen3-TTS repository.

Chatterbox Turbo is the English-focused model in the Chatterbox family. Keep it distinct from Chatterbox Multilingual when deciding language coverage. The Turbo experiment is useful for expressive English narration and reference-conditioned voice generation. Chatterbox repository.

For Singlish, the question is more specific than whether a model supports English: can it speak the intended local rhythm and vocabulary clearly, without exaggerated imitation? Use a listening set with the intended speakers and terms. For Hinglish, test the switch inside a sentence; separate English and Hindi demonstrations do not answer that question.

From a local comparison to an interactive benchmark

I separate four clocks:

1. Cold initialization: loading weights, initializing the runtime and warming relevant shapes.
2. First playable audio: accepting the phrase to making the first valid audio chunk available.
3. Complete generation: accepting the phrase to producing its final sample.
4. Delivered playback: the time the caller actually hears each chunk.

The local comparison script measures process wall time and records the output duration. That is a useful smoke test. For an interactive comparison, keep the model resident and measure the generation iterator directly.

In MLX, make sure the output has actually been evaluated before stopping a compute timer; scheduling lazy work is not the same as completing it. Measure device work and end-to-end service time separately when profiling. MLX lazy evaluation.

Use a run directory per model and trial. Record process exit status, every generated segment and total waveform duration. Otherwise an older WAV file or only the first segment can contaminate the result. Keep cold and warm trials separate, and repeat the same set of short confirmations, full sentences, numbers and mixed-language phrases.

For a streaming service, record the largest gap between chunks as well as aggregate RTF. A model can finish a ten-second waveform in three seconds and still cause a noticeable gap if one required chunk arrives after playback has caught up.

A concrete Zoice adapter: getting the language code right

The Zoice Sarvam adapter handles regional Pipecat language codes before creating the provider request. This is an easy detail to overlook: “Telugu is supported” is not useful if a regional enum is converted incorrectly.

Here is the language-mapping logic from the Kuzushi Labs adapter, condensed to a standalone function:

def sarvam_language_code(language):
    raw = str(getattr(language, "value", language))
    regional = {
        "bn-IN": "bn-IN", "en-IN": "en-IN", "gu-IN": "gu-IN",
        "hi-IN": "hi-IN", "kn-IN": "kn-IN", "ml-IN": "ml-IN",
        "mr-IN": "mr-IN", "od-IN": "od-IN", "pa-IN": "pa-IN",
        "ta-IN": "ta-IN", "te-IN": "te-IN",
    }
    if raw in regional:
        return regional[raw]
    base = {
        "bn": "bn-IN", "en": "en-IN", "gu": "gu-IN",
        "hi": "hi-IN", "kn": "kn-IN", "ml": "ml-IN",
        "mr": "mr-IN", "or": "od-IN", "pa": "pa-IN",
        "ta": "ta-IN", "te": "te-IN",
    }
    return base.get(raw.split("-")[0].lower())

assert sarvam_language_code("te-IN") == "te-IN"
assert sarvam_language_code("ta") == "ta-IN"
assert sarvam_language_code("unsupported") is None

The caller of this function should reject an unsupported language before synthesis. Silently falling back to English makes configuration appear successful while changing the customer's experience.

The same boundary applies to voice IDs, sample rates and provider options. A configuration-driven product needs typed translation between its saved settings and each provider's constructor.

Start with what must be pronounced correctly

For a financial or service call, a useful synthesis test set contains:

Text normalization should be deterministic for business-critical values. Asking the LLM to turn every number into speech can work, but it introduces another opportunity to change the number.

Store structured values alongside their spoken rendering. A date such as 03/04 should not reach synthesis until the application knows which calendar date it means.

Managed options

ElevenLabs' low-latency models are candidates when conversational voice quality and fast first audio matter. Its documentation also discusses normalization differences and limitations for numbers. Provider-reported model latency is not the complete telephone response time. ElevenLabs model documentation.

Google Chirp 3 HD supports streaming synthesis and lists English, Hindi, Tamil and Telugu among its languages. The choice still needs a tested voice, output codec and endpoint region. Google Chirp 3 HD.

Sarvam's Bulbul v3 documentation describes an Indian-language-focused TTS service with HTTP and WebSocket streaming. Treat it as a managed API here; an integration with Sarvam is not evidence that its weights were self-hosted. Sarvam Bulbul.

These observations support a shortlist. They do not establish which service has the best pronunciation or p95 latency on the intended customers.

An additional self-hosted path for Indic languages

Indic Parler-TTS is a roughly 0.9B-parameter open-weight candidate with Hindi, Tamil, Telugu and English in its published coverage. It conditions speech on both the text and a description of the voice. Its model card describes distinct text and description tokenizers. Access conditions and the published licence should be checked before retrieving weights. Indic Parler-TTS model card.

This is a separate Indic-language serving recipe from the three-model local comparison above. It is useful when Hindi, Tamil and Telugu coverage drives the decision. Singlish quality still needs its own listening evaluation.

The first serving experiment should use an existing published voice. Add custom adaptation only after measuring what actually fails: pronunciation, speaking style, speaker consistency or language switching.

Load the model and its two tokenizers once, then put a small boundary around full-phrase synthesis:

import torch

def synthesize_phrase(model, text_tokenizer, description_tokenizer,
                      text, description, device):
    prompt = text_tokenizer(text, return_tensors="pt").to(device)
    style = description_tokenizer(
        description, return_tensors="pt"
    ).to(device)
    with torch.inference_mode():
        waveform = model.generate(
            input_ids=style.input_ids,
            attention_mask=style.attention_mask,
            prompt_input_ids=prompt.input_ids,
            prompt_attention_mask=prompt.attention_mask,
        )
    return model.config.sampling_rate, waveform.cpu().numpy().squeeze()

The worker must impose request length, timeout and output-duration limits around this boundary. In particular, do not run this blocking generation on the media event loop.

Adaptation data and training

For a proposed small pilot, collect 5–10 hours of clean, consented audio per intended voice/language combination. That is a planning target rather than an empirically established minimum. Use consistent speaker identity, varied sentence forms and the actual domain vocabulary.

Each example needs:

{
  "audio": "licensed-recordings/voice-a/0001.wav",
  "transcript": "आपकी अगली payment की तारीख पाँच अक्टूबर है।",
  "description": "A clear conversational voice at a moderate pace.",
  "speaker_id": "voice-a",
  "language_mix": "hi-en",
  "session_id": "recording-day-03",
  "rights_record": "consent-and-licence-reference"
}

This is an illustrative record. Training material should preserve clean full-band speech; telephone degradation belongs in output-channel evaluation. Training only on narrowband audio can bake degraded sound into the generated voice.

Split by recording session and held-out text patterns. Reserve names, numbers and code-switched combinations that the model has not memorized. Precompute audio codec targets using the checkpoint's matching codec, then adapt using the upstream training implementation. The Parler project publishes a training workflow and a single-speaker fine-tuning example. Parler training guide.

An initial full-adaptation experiment can start at learning rate 1e-5, microbatch one and accumulation 16, with frequent held-out synthesis. These are proposed tuning values. The correct loss masks, codec delays and tokenizer configuration must match the selected checkpoint; an ASR trainer cannot be reused by changing the model name.

For hardware, budget one 48 GB GPU for the first full-model training experiment, 64 GB host RAM and space for decoded audio and codec caches. A 24 GB GPU is a candidate for serving tests and perhaps a smaller training configuration, not a guaranteed minimum. At 0.9B parameters, a rough 16-byte training-state estimate already approaches 14.4 GB before activations.

Two different meanings of streaming

Text streaming means the LLM sends text incrementally. Audio streaming means the synthesizer emits playable waveform chunks before the utterance is complete.

Supporting one does not guarantee the other. A system can accept small text fragments and still generate each fragment offline. Conversely, a synthesizer can stream audio after receiving the entire sentence.

Parler provides an audio streamer and optimization guidance including attention implementations and compilation. Its general performance examples should not be transferred automatically to the Indic checkpoint or the target hardware. Parler inference guide.

For a proposed Indic service:

1. Load weights and both tokenizers once per worker.
2. Warm the exact generation shapes used by the service.
3. Receive a complete short phrase and its generation ID.
4. Generate audio chunks through the supported streamer.
5. Convert continuously to the telephony codec and sample rate.
6. Stop generation and suppress stale output on interruption.

A background generation thread is not sufficient cancellation. The generator needs a supported stopping mechanism; otherwise it continues consuming GPU resources after the caller interrupts.

Phrase segmentation is a latency-quality trade-off

Splitting on every token produces poor rhythm. Waiting for a paragraph delays speech.

A proposed policy is to flush at sentence boundaries, or at a safe clause boundary after a bounded wait. Keep currency expressions, dates and identifiers together. For example, never synthesize “Your premium is one…” while the rest of the amount is still being generated.

The service should also reject incomplete structured values. A deliberate pause before an amount is better than an incorrect amount spoken confidently.

Measure:

first-audio delay = first playable chunk − accepted phrase
generation RTF = generation wall time / produced audio duration
underrun = playback needs audio before the next chunk is ready
interruption residual = audible old speech after interruption

For smooth continuous playback, average generation must keep up with consumption, and the tail of chunk-generation time must fit the available buffer. An RTF below one is necessary for sustained single-stream playback but does not establish acceptable first-audio delay or many-stream capacity.

Figure 8. Synthetic timing. Synthetic synthesis generates six seconds of audio in 1.8 seconds with first audio at 250 ms. Playback lasts until 6.25 seconds, illustrating why throughput and first-audio delay differ. Scalable SVG.

Evaluate synthesis as speech

ASR round-trip transcription is useful for finding obvious pronunciation mistakes, but the recognizer can hide or introduce errors. Combine it with blinded human listening by native speakers.

Rate intelligibility, numeric correctness, name pronunciation, naturalness, language transitions and speaker consistency. Run the audio through the intended telephone codec before rating the deployed experience. Report the number of utterances, speakers and raters, and keep the raw rating distribution.

For the self-hosted adaptation comparison, keep the base voice and adapted voice side by side. Report pronunciation wins and regressions together with first-audio delay, compute cost and the concurrency configuration.

A language fixture should sound like the intended call

Here are reading prompts for the evaluation set. Have native speakers check the intended regional variety and delivery before recording the release fixtures.

The Singlish line is primarily an understanding fixture. Automatically adding “lah” to every generated sentence would be caricature rather than localization. The agent can respond plainly: “Sure. Do you mean after six this evening, Singapore time?”

Create paired prompts differing in only one material value: thirteen/thirty, fifteen/fifty, ₹1,250/₹12,500, and a policy identifier with B/D or M/N. Include a caller correction followed by the updated value. Rate the audio delivered through the telephone codec, not a full-band sample from the vendor console.

Keep numerical rendering outside free-form generation

Represent money as decimal values and retain currency separately. This original example prepares an English rendering for rupees and paise using digit strings; a production language renderer supplies localized number words:

from decimal import Decimal

def inr_parts(amount: str):
    value = Decimal(amount)
    if not value.is_finite() or value < 0:
        raise ValueError("Expected a finite non-negative amount")
    if value != value.quantize(Decimal("0.01")):
        raise ValueError("More than two decimal places")
    rupees = int(value)
    paise = int((value - rupees) * 100)
    return {
        "currency": "INR",
        "rupees": rupees,
        "paise": paise,
        "text_for_renderer": f"{rupees} rupees and {paise} paise",
    }

assert inr_parts("1250.50")["paise"] == 50

The important boundary is that the LLM selects a permitted message with a typed amount; it does not recalculate the amount while speaking it. Preserve the original value and rendered phrase in the operation record. If a provider supports a pronunciation dictionary or phonetic markup, keep that provider-specific representation behind its adapter and test it on the actual voice. Do not send one provider's markup blindly to another.

Ambiguous dates need clarification before rendering. “03/04” is not solved by choosing whichever interpretation sounds more natural. Similarly, a phone number is a sequence of digits, whereas a currency amount is a quantity. A generic number-to-words function must know which one it is handling.

There are two queues after the language model

One queue contains text waiting to be synthesized. Another contains generated audio waiting to be played. Both can grow even while the caller hears a smooth response.

Suppose, in a synthetic timing example, six seconds of speech take 1.8 seconds to generate and the first chunk appears after 250 ms. The aggregate RTF is 0.3. That is encouraging for one stream, but it says little about eight simultaneous streams sharing a GPU, or a single delayed chunk that empties the playback buffer.

Track audio debt in seconds:

ready-to-play seconds = generated audio duration − played duration
pending-text seconds = estimated duration of accepted unsynthesized text

A proposed policy might pause acceptance of further nonessential text above 1.5 seconds of ready audio and resume below 0.5 seconds. Those values require experiment. Never drop arbitrary PCM chunks to reduce the queue: that removes parts of words. Stop at a defined phrase/generation boundary and discard the superseded response consistently.

The provider and carrier may maintain additional buffers. Local queue length is therefore a useful control signal, not proof of how much speech remains audible after a barge-in.

Avoid discontinuities at phrase boundaries

Independent phrase generation can introduce changes in pace, volume or voice identity. Continuous resampling needs filter state across chunks; resetting it for each tiny packet can introduce boundary artifacts. Output chunks should maintain the expected sample format, sequence and timeline.

For the first implementation, keep voice configuration stable for the call, segment at complete clauses, and measure pauses between clauses. If the upstream generator exposes a continuous stream, preserve that continuity through codec conversion. Do not splice generated waveforms merely because their text strings concatenate correctly.

A useful failure fixture is: “Your payment is…” followed by a corrected amount before the first amount is spoken. The expected result is one coherent corrected sentence, not overlapping versions or a dangling “one thousand” from the cancelled response.

What would justify self-hosting this component?

Self-hosting is justified by a measured requirement: a missing pronunciation capability, control over a licensed voice, data locality, or a better cost-quality point at real utilization. It also adds model warm-up, audio-codec dependencies, serving operations and an additional failure domain.

The decision record should contain telephone-band listening results, first-audio and inter-chunk latency, cancellation cost, peak memory, maximum safe measured concurrency and the fully allocated idle cost. A good offline voice sample is the start of that record, not its conclusion.

Next: context, retrieval and business tools.