Zoice

The tax-filing pilot gives a concrete example of Zoice’s purpose. An AI phone agent asked about the caller’s income and circumstances, classified the enquiry and arranged an appropriate expert handoff. This took on the preliminary information gathering that a staff member would otherwise need to complete before passing the case on.

Period: July–September 2025 · Team: three people
Scope: Pipecat voice runtime, model interoperability, operator portal and platform delivery

One product, two speech architectures

Zoice was developed at Kuzushi Labs as a configurable conversational AI product for telephone-based calling. The system combined a Pipecat streaming backend with a web portal for defining agents, selecting providers and managing call workflows.

ClearTax was piloting with Zoice during this period. Tax-filing enquiries required the agent to classify the customer's profile, gather the relevant income context and arrange an appropriate expert handoff. The resulting call record needed to be useful to the next person or system in the workflow.

Requirements became configuration

Agent definitions, provider settings, pipeline records and call context were represented as reusable product configuration. The backend resolved that configuration for each live call. A persona or voice choice could therefore be changed without editing the call's transport implementation.

Both cascaded and native-audio pipelines were implemented:

• STT → LLM → TTS: recognition, reasoning and speech generation were independently configured and constructed.
• Native audio: a Gemini realtime multimodal service handled speech interaction within one model session, with transport and context managed by the platform.

The cascade exposed separate control over recognition, reasoning and synthesis. Native audio consolidated the speech interaction into one service. Language quality, interruption behavior, cost and conversation control shaped the choice between the two.

Provider interoperability

Provider settings were translated into the constructor arguments expected by each service. Top-level options, nested parameters, voice IDs and language representations were handled at that boundary.

The Sarvam adapter explicitly mapped regional language enums, including Hindi-India and Telugu-India, into provider language codes. A supported language needed a working configuration path all the way to the speech service.

Conversation state and lifecycle

In the cascade, the opening greeting was sent directly to TTS and appended to assistant history, avoiding an extra LLM request for the first greeting. Context aggregation, STT muting, idle behavior and interruption handling were coordinated around the pipeline.

On disconnect, conversation messages were submitted to Redis, the pipeline task was ended and the per-call HTTP session was closed. Uvicorn owned process signals rather than allowing each pipeline runner to install independent signal handlers.

These details connected conversation behavior with resource ownership: a completed call should leave neither an orphaned provider stream nor a stale response waiting to play.

The platform around the audio

The telephony service, bot runtime and post-call processing were separated around common configuration and data models. PostgreSQL stored pipeline definitions; Redis supplied call-associated configuration and context. Call identifiers linked the telephony request, chosen agent and resulting records.

The portal used Next.js, React and TypeScript, with React Query for server data and Zustand for client state. It included pipeline creation, provider configuration forms, a pipeline visualizer and realtime-agent controls. Production delivery used containers and Kubernetes.

Evaluation begins with the call record

The post-call path stored recordings, produced a text representation, applied workflow-specific extraction questions and persisted the result for downstream delivery. This gave customer teams a structured result rather than only a recording to listen to.

The existing post-call Whisper translation path produces English text for extraction. Recognition evaluation separately needs source-language references; a translated transcript cannot be used as a Hindi or Hinglish WER reference. Live model history, recording-derived text and backend outcomes therefore serve different review purposes.

The detailed voice evaluation article develops the metric publisher, event contracts, extraction checks, causal review and regression workflow.

Hinglish recognition and later speech experiments

Hinglish adaptation addressed Hindi-English switching within a conversation, particularly business terms, names and numbers. Earlier speech work included fine-tuning Whisper with internal voice samples in late 2024.

In September 2026, a separate public-data reproduction used Whisper Tiny and MUCS Hindi-English lecture audio. On 128 held-out clips, normalized WER changed from 215.62% to 77.29%. The absolute error rate remained too high for customer deployment. The experiment demonstrates the training and evaluation process, with its improvements and regressions available for inspection. Read the measured experiment.

The 2026 local TTS work remains separate from Zoice's earlier managed integrations. Solo & Silicon compares Kokoro, Qwen3-TTS and Chatterbox Turbo through MLX Audio. Read the synthesis article.

Product outcome

Zoice brought multiple speech architectures, configurable providers, agent management and post-call results into one product. It supported the ClearTax pilot and made agent and provider choices available through a common operational workflow.

The engineering value lies in that complete path: customer requirement, agent configuration, live conversation, lifecycle handling and a usable result for the business.

Contribution

My contribution covered commercial scoping, requirements definition, architecture and hands-on development within the three-person team. The work spanned the Pipecat voice backend, provider interoperability and operator-portal integration, connecting the customer's workflow to product delivery.