Designing a configurable voice platform with Pipecat

When we built Zoice at Kuzushi Labs in July–September 2025, the problem extended beyond producing a convincing call. The platform needed configurable agents, multiple speech and language-model providers, telephony integration and a portal through which those capabilities could be managed.

We built Zoice with FastAPI, Pipecat, Plivo WebSockets, PostgreSQL and Redis, alongside a separate web application. The version discussed here used Pipecat 0.0.91. I will keep the original pipeline visible while explaining how to strengthen the lifecycle, resource and deployment boundaries around it.

The cascaded voice pipeline

The main cascaded path has this structure:

transport input
  → STT mute filter
  → speech recognition
  → idle handling
  → user context aggregator
  → LLM
  → TTS
  → transport output
  → assistant context aggregator

The order matters. The user aggregator assembles the input for reasoning. The assistant aggregator appears after the output stage in this branch, linking generated output to conversation history. Its placement alone does not guarantee that the final telephone endpoint heard every word.

The service factory converts saved provider configuration into concrete services. It handles differences such as language values, parameter classes and provider-specific options. This is where a configurable product becomes more demanding than a one-off demo: the configuration has to preserve behavior across different APIs.

The native-audio branch uses Gemini rather than separately calling STT, a text LLM and TTS. It also has conversation-context handling. Zoice supports multiple pipeline types and configurable provider integration. The pipeline is selected when configuring the call. Switching services during a live conversation is a further design problem: the replacement needs the right context, media position and operation state.

Frames make the system inspectable

Pipecat represents activity through frames processed along a pipeline. Its frame categories include ordinary data, ordered control and urgent system-level behavior. The distinction matters when an interruption must overtake buffered speech. Pipecat frame overview.

The topology becomes easier to inspect when each service is passed into a small composition function:

def cascade_stages(
    transport, mute_filter, stt, idle_handler,
    context, llm, tts
):
    return [
        transport.input(),
        mute_filter,
        stt,
        idle_handler,
        context.user(),
        llm,
        tts,
        transport.output(),
        context.assistant(),
    ]

Keeping stage construction separate from provider selection enables useful tests. One can replace the recognizer with a recorded event stream, delay the synthesizer, or inject a disconnect without changing the business logic.

What SEDA contributes

The original SEDA architecture describes stages connected by explicit queues, with resource management and load conditioning. Its relevance is the ability to reason about overload at each stage. It is not merely a synonym for asynchronous Python. Welsh, Culler and Brewer, SEDA, SOSP 2001.

Pipecat's processor graph makes a staged design possible. It should not be described as automatically providing every controller or isolation mechanism in classic SEDA.

For a production voice platform, apply the idea explicitly:

A critical operational metric is the age of the oldest item in each queue. Another is event-loop lag. An async service can have low aggregate CPU and still miss audio deadlines if a synchronous operation blocks its event loop.

Do not turn every stage into a microservice

At tens of calls, transport, turn control, context aggregation and provider adapters can remain inside a session worker. That keeps cancellation and ordering easier to reason about.

GPU recognition and synthesis deserve independent serving pools because their resource lifecycle differs from the call worker. They benefit from shared loaded weights, warm-up, specialized scheduling and independent scaling.

Slow post-call work should be a separate consumer. Live audio should not travel through a durable job queue designed for retries measured in seconds.

Figure 4. Reference architecture. A CPU session worker selects a cascade or native audio path, with shared business controls and generation-aware playback. Model serving and post-call work have separate resource lifecycles. Scalable SVG.

This is the deployment boundary I use to reason about the platform: a call worker owns the conversation, model pools own expensive inference, and background workers own post-call work.

A greeting is a useful special case

The Zoice cascade sends the welcome message directly to synthesis and appends it to the assistant context. A fixed greeting does not need an LLM round trip.

That can improve time to first greeting. It must be reported separately from ordinary response latency. Otherwise a fast cached introduction can make a system appear faster than its actual conversations.

There is also a state issue: appending a greeting when it is queued does not prove it was fully delivered. A hardened version records playback completion or marks the greeting as partially delivered if the caller interrupts.

Mute filters need a reason

Temporarily muting recognition can prevent echo or unwanted triggers during specific call phases. Muting it throughout every assistant response also prevents natural interruption.

The mute policy should be explicit: connection setup, known echo-prone phase, a brief non-interruptible legal message if the workflow requires it, or another clearly defined state. For ordinary conversation, full-duplex behavior requires continuing to listen while output is playing.

Test background speech, speakerphone echo and the caller saying a short “no.” A detector that responds only to a long interruption misses precisely the corrections that matter.

Native audio uses the same business boundary

The cascade and native-audio pipeline can share:

• Call admission and authentication.
• The durable business state and operation ledger.
• Tool authorization and validation.
• Audio-delivery accounting.
• Evaluation scenarios and outcome definitions.

They cannot blindly share the same conversation serializer. A text transcript, native audio items, provider-specific function results and context-compression events carry different semantics.

Define a provider-neutral state object containing confirmed facts, unresolved questions, committed operations and played-response information. Convert it into each provider's input format at the boundary. Keep the authoritative state outside the model session.

Configuration is an engineering surface

Each call should retain a configuration snapshot:

{
  "schema_version": 1,
  "agent_version": "qualification-v4",
  "pipeline": "cascade",
  "language_policy": "preserve-code-switching",
  "prompt_version": "prompt-v7",
  "tool_schema_version": "tools-v3",
  "speech_model_revision": "resolved-at-release",
  "turn_policy_version": "turns-v2"
}

These are illustrative values. A release replaces the model revision with a resolved immutable identifier and records actual service settings. Secrets belong in a secret store, not in this snapshot or logs.

Changing a language code, endpointing threshold or TTS normalization setting can alter customer behavior as much as changing the prompt. All of them need regression evaluation.

Give every processor a contract

A pipeline diagram does not specify who owns a frame or whether a processor may emit events after cancellation. Those details determine whether the composition is reliable.

For a proposed recognizer adapter, define:

A synthesizer has a different contract: it emits audio with an output format, supports bounded text input, identifies phrase completion and exposes whether cancellation actually stops compute.

Adapters should be tested against recorded provider event fixtures. The fixture can contain a partial transcript, a revised partial, a final, a duplicate final and a late disconnect. The expected pipeline output should contain one committed utterance, not three concatenated versions.

Supervise the lifetime of the whole call

The input reader and output writer should not become orphaned tasks after a disconnect. Python's structured-concurrency primitives help express that ownership, although they do not make blocking GPU calls interruptible. Python task groups and cancellation.

This original Python 3.11+ example accepts two asynchronous functions and an externally controlled stop event:

import asyncio

async def supervise_call(read_input, write_output, stop):
    async def run_until_end(operation):
        try:
            await operation()
        finally:
            stop.set()

    async with asyncio.TaskGroup() as group:
        reader = group.create_task(run_until_end(read_input))
        writer = group.create_task(run_until_end(write_output))
        await stop.wait()
        reader.cancel()
        writer.cancel()

When either side ends, the stop signal cancels the other. An exception remains visible to the caller through the task group. Each operation still needs a finally block to close its own connections.

Cancelling a coroutine waiting on a thread does not necessarily stop the work in that thread. GPU or SDK adapters need their own supported abort mechanism, or must discard obsolete results while accounting for the remaining compute.

Observe queues without blocking them

A metrics processor should not make a network request for every audio frame. At fifty 20 ms frames per second and forty calls, that would create 2,000 metric submissions per second before considering output audio.

Update local counters and histograms, then export them asynchronously. Use call IDs in sampled traces or structured logs, rather than making every call ID a persistent metric label with unbounded cardinality.

The important decomposition is:

stage time = queue wait + service work + downstream blocking

If recognition becomes slow because it cannot enqueue its result, attributing the entire interval to the speech provider produces the wrong diagnosis.

A worked pipeline incident

Consider an example incident: after a deployment, callers hear the beginning of an old answer immediately after interrupting.

The LLM cancellation log looks correct. The TTS cancellation log also looks correct. The remaining possibilities include audio already queued locally, audio already accepted by the carrier, and a late chunk emitted after cancellation.

A useful investigation follows the generation ID:

generation 18: text accepted
generation 18: first TTS chunk produced
interrupt: active generation becomes 19
generation 18: provider clear requested
generation 18: late chunk arrives
generation 18: output writer rejects stale chunk
generation 19: first new chunk accepted

If the stale chunk reaches the transport, the missing boundary is in the output path. Adding another instruction to the LLM cannot repair it.

This is where pipeline instrumentation becomes more useful than a single transcript viewer.

Replaying an event trace

Keep a small library of synthetic event traces covering disconnects, interruptions, tool completion and provider errors. Replaying them should be deterministic at the controller layer even when real model generation is probabilistic.

The trace should include its configuration version and expected terminal state. Use fake provider adapters to test lifecycle logic without paying for model calls or handling customer audio.

A second integration suite then checks the real adapters against their provider contracts. The two suites answer different questions: whether the controller is correct, and whether the external API behaves as the adapter assumes.

The delivery lesson

The framework provides a way to express the conversation. The platform work is in making that expression configurable, observable and bounded under load.

My contribution to Zoice covered requirements, architecture and hands-on implementation across the voice pipeline and platform integration. The work brought cascaded and native-audio paths, configurable model services, telephony serialization and conversation lifecycle handling into one product.

Next: choosing models and pipeline types.