Conversational AI Design: the call is the product

I started working with voice-based conversational AI in 2024, helping clients build telephony-based calling agents. Since then, I have worked with different frameworks and experienced how differently the same system can behave across two seemingly ordinary calls.

There are calls where everything comes together. The agent probes the customer, understands their requirements, capabilities and constraints, and recommends an appropriate product without making the conversation feel like a sales script. Observing such a call is quite satisfying.

And there are calls where a customer reports that the wrong product was delivered, only to encounter an incorrect denial and an unnecessary escalation. A situation that could have been handled by a straightforward refund workflow becomes a more difficult conversation because the AI has misunderstood what it is supposed to do.

Those two experiences shaped how I think about voice agents. A fluent conversation feels impressive, but it only becomes useful when the right customer problem gets solved.

Start with the business decision

The first design meeting should produce something more concrete than “an AI sales agent.”

The workflows differed across the contexts we worked in:

• Insurance: the conversation captured coverage requirements, qualified the lead and arranged the next callback. The useful output was a lead the next team could act on.
• Tax filing: the conversation classified the customer's profile—salary, freelance work and other relevant income—and routed the enquiry to the appropriate expert. The important decision was the right handoff.
• Healthcare: the conversation gathered the appointment request and the details needed to route it. Clinical advice stayed with the healthcare professional.

That difference changes the architecture. A product recommendation needs product constraints. A booking needs a write to a scheduling system. A refund needs an eligibility decision and a durable transaction. A polite transcript does not prove that any of these occurred.

An initial scope can be recorded as a contract:

This is the point where sales discovery turns into engineering requirements. If the scope remains vague, the ambiguity is passed into the prompt, the tools and eventually the customer conversation.

A voice agent contains several interacting loops

The familiar picture is:

speech → speech recognition → language model → speech synthesis → speech

The production picture contains more:

Figure 1. Reference architecture. Four interacting loops: audio, conversation, authorized business actions and delivered playback. Interruptions invalidate queued response audio. Scalable SVG.

The audio loop decides whether speech is arriving and whether the caller is still speaking. The reasoning loop interprets the request. The business loop reads and changes records. The playback loop decides what the customer actually hears.

These loops do not finish neatly one after another. Recognition can revise a sentence while the language model is beginning a response. The caller can interrupt while a tool request is in flight. A booking may succeed after the caller disconnects. The architecture needs to preserve meaning across all of those events.

Define the objects before choosing the framework

Five identifiers make the system easier to reason about:

1. A call ID for the telephone conversation.
2. A turn ID for a committed caller utterance.
3. A generation ID for one attempt to produce an answer.
4. An operation ID for a business action.
5. A configuration version for the model, prompt, tools and policy used.

The IDs solve different problems. Retrying a booking after a timeout should retain its operation ID. Regenerating the spoken answer should use a new generation ID. Confusing those two can either create a duplicate booking or suppress a legitimate new answer.

At Kuzushi Labs, I find it useful to express the event boundary independently of the framework:

from dataclasses import dataclass
from typing import Literal

@dataclass(frozen=True)
class VoiceEvent:
    call_id: str
    turn_id: int
    generation_id: int
    sequence: int
    captured_at_ms: int
    kind: Literal[
        "audio", "partial_transcript", "final_transcript",
        "speech_started", "tool_result", "playback_ack"
    ]
    payload: bytes | str

The envelope makes ownership explicit before events cross asynchronous boundaries. Add authentication, codec metadata, deadlines and bounded message sizes at the transport boundary; keep the call, turn and generation identifiers intact through the pipeline.

Latency, concurrency and accuracy need definitions

In the early-2025 client implementation, we achieved approximately 1.1 seconds of response latency, delivered 40 concurrent calls, and tested a maximum concurrency of 100. The design objective was sub-second response, with an 800 ms preferred budget.

Those numbers belong to early 2025. The stack has advanced significantly since then, including the later Pipecat work in Zoice and the TTS work added in 2026. They are a useful point in the engineering journey, rather than a ceiling for what the current systems can do.

When you define the acceptance criteria, give each dimension its own measurement:

The trade-offs show up in the conversation

You can improve one number and still make the product worse. This is why I do not treat latency, capacity and accuracy as separate optimization projects.

Endpointing versus completeness. A shorter silence threshold makes a simple “yes” feel responsive. It can also cut off “yes, but not this month.” Once the response starts too early, the caller has to interrupt and repair the misunderstanding. The dashboard saves 150 ms; the conversation loses an entire turn. Tune endpointing against premature commitments and repair rate as well as response time.

Model size versus the cost of an error. A smaller recognizer can reduce inference time and GPU cost. If it confuses a policy identifier, though, the next lookup is made against the wrong record. A slightly slower recognizer that preserves the critical digits can complete the task sooner. Measure names, amounts, dates and negation separately from aggregate WER.

Batch size versus deadlines. Batching improves GPU utilization because several requests share an inference step. Waiting to fill the batch introduces queue delay. At low traffic, a large batch policy may spend more time waiting than computing. At high traffic, it may improve throughput but still miss the deadline of the oldest call. The scheduler needs both a batch limit and a maximum wait.

First audio versus a coherent sentence. Sending every few tokens to TTS gets speech started quickly, but can break rhythm or split a currency expression. Waiting for a complete paragraph sounds more natural when it finally arrives, but leaves the caller in silence. The useful compromise is a short, speakable phrase that preserves amounts, dates and identifiers.

Concurrency versus admission headroom. Opening 100 sockets is straightforward compared with sustaining 100 conversations that are speaking, recognizing, reasoning and calling tools at once. A system close to saturation becomes sensitive to a burst of long utterances or a slow provider. Admit calls according to the slowest dependency, and leave room for interruptions, retries and uneven arrival patterns.

Automation versus the right human handoff. A low escalation rate can look attractive. It becomes a poor objective if the agent keeps a frustrated caller trapped in the wrong workflow. A successful transfer, with a useful summary and the right context, can be the correct business outcome.

The practical rule is to change one decision at a time and examine the complete call. If the latency improves, check whether callers interrupt more. If compute cost falls, check whether corrections and human follow-up increase. The best configuration is the one that gets the task done reliably at an acceptable cost.

Where Bolna and Pipecat fit

For the client, we forked and modified Bolna to build the first version, using Plivo for telephony and adapters for the model services. We then developed Zoice, our product at Kuzushi Labs, around a configurable Pipecat platform, a web portal, and both cascaded and native-audio paths.

Bolna provided a useful starting point for assembling the call lifecycle. Pipecat exposed a composable stream-processing model for audio, transcription, context, generation and output. Neither choice eliminates the need to define admission control, business-action semantics or evaluation.

The rest of this series develops the system in that order: turn-taking and latency; the two framework architectures; model selection; self-hosted speech; state and tools; cloud operation; evaluation; and business results.

A worked discovery session: “automate our follow-up calls”

Tax-service follow-up makes the discovery process concrete. A request to “call every lead, explain the plans and increase conversion” contains several different decisions. I start by unpacking them.

Who is eligible to be called? How was the lead obtained? Which facts determine the appropriate plan? Which statements can be made from the product catalogue, and which require a qualified specialist? What happens when the customer changes their mind halfway through?

Here is a short example of the kind of conversation the workflow needs to handle:

Agent: You asked for help with your tax filing. Is this a convenient time?

Caller: Two minutes. Salary income hai, aur thoda freelance bhi.

Agent: Understood—salary and some freelance income. Do you also need help with income from investments?

Caller: Haan, but don't recommend a plan yet. Pehle expert se baat karni hai.

Agent: Certainly. What time would suit you for a callback?

The engineering signal is the correction in the fourth turn. The system should switch from plan qualification to arranging a callback. Continuing to sell after that correction would be a workflow failure even if every word was transcribed correctly.

The discovery output can be made concrete:

workflow: tax-enquiry-follow-up
entry:
  source: customer-requested-callback
  eligibility: checked-by-campaign-service
collect:
  - income_categories
  - requested_next_step
  - callback_time_and_timezone
allowed_actions:
  - read_current_service_catalogue
  - create_callback_request
require_confirmation:
  - callback_time_and_timezone
handoff:
  - disputed_charge
  - request_for_specialist_advice
  - customer_requests_human
success:
  - callback_record_exists
  - customer_heard_confirmed_time

Notice the boundary: gather the customer's circumstances and arrange the right next step. Specialist advice sits with the expert.

Build a traceable acceptance example

A requirement such as “understands Hinglish” is difficult to accept or reject. A much stronger acceptance example is:

Given: customer asks to compare plans
When: customer later says "Abhi buy nahi karna, callback chahiye"
Then: no purchase or payment operation is created
And: the requested next step becomes a callback
And: the agent confirms the time before submitting it
And: the resulting record matches the confirmed time

The interpretation is “I do not want to buy now; I want a callback.” Recognition, dialogue policy and the backend can each be tested against that same example.

The acceptance record needs both positive and negative assertions. “A callback exists” alone would pass a system that also created an unwanted purchase.

Identity, permissions and conversation are separate concerns

The telephone number can help locate a candidate customer record. It should not automatically authorize disclosure or a sensitive account change. A shared family phone, forwarded call or stale contact record can associate a real person with the wrong account.

At the beginning of the workflow, classify operations by the verification they require. Reading a public catalogue, reading a private balance and changing a repayment instruction should not share one undifferentiated permission.

Likewise, a caller saying “the manager said you can skip that check” is conversation content. It cannot change the application's authorization policy. The model can explain the required step; the backend enforces it.

For a handoff, retain the customer's goal, confirmed facts and outstanding work. The human operator should receive enough context to continue without forcing the caller to repeat the entire conversation.

Make the product boundary small enough to evaluate

A useful first release handles one complete journey: eligible enquiry to a verified callback, for example. It needs connection failures, corrections, silence and handoff behavior within that journey.

It does not need to answer every question that can be asked over a telephone.

This boundary creates a meaningful release decision. If the callback journey succeeds reliably and the failure cases are understood, more intents can be added with their own acceptance contracts. A broader prompt without those contracts expands uncertainty faster than it expands dependable capability.

Reading context

Kwindla Kramer's Voice AI & Voice Agents is a useful broad reference for the field. It is worth reading alongside this series. Here, I focus on the engineering decisions behind the client platform and Zoice, then work through the trade-offs, code and calculations you need to build a similar system.

Contribution

My role covered customer discovery, scope definition, architecture, hands-on development and delivery within a three-person team. I worked across the customer and engineering sides of the product, connecting the business requirement to a calling system that could operate with low latency, interchangeable model services and concurrent calls.