Text and voice on top of the avatar runtime

A conversational avatar is a distributed interaction. The backend generates text and audio; the browser plays speech, renders the character and accepts interruption. Memory and tools may add more asynchronous work. The implementation needs a coherent turn even when those components finish in a different order.

Avashi’s surrounding services contain streaming text with emotion events, bounded tool calling, voice orchestration and memory clients. The inspected Pipecat path uses LiveKit transport, Silero voice-activity detection and explicit pipeline lifecycle handling. An older README also describes a LiveKit Agents path, so the code history should not be compressed into one timeless architecture.

Start with the text channel

The text-streaming service defines token, emotion, final, error, stream-end, tool-call and tool-result events. It also maintains cancellation events for active message streams. These are useful primitives because presentation can react incrementally while still receiving a terminal outcome.

The basic mode extracts emotion from structured model output. The tool-aware mode supports a multi-turn loop with a configured ceiling of ten iterations in the inspected code. That ceiling bounds one dimension of work; it does not by itself cap wall-clock time, total tokens or tool cost.

A robust client treats a stream as a state machine. It accumulates content for one message identity, applies compatible emotion cues and handles either completion or failure. Losing a connection halfway through should not cause a partial message to be silently reclassified as complete.

A shared turn identity

The publication’s reference event shape is intentionally small:

{
  "event": "speech.started",
  "session_id": "session-example",
  "message_id": "assistant-turn-7",
  "generation": 3,
  "sequence": 12,
  "audio_offset_ms": 0
}

The session provides continuity. The message identifies the response. The generation distinguishes a new attempt from abandoned output. The sequence orders events inside that generation. The audio offset is relative to the response’s audio clock, not a server wall-clock timestamp.

This is an authored reference contract, rather than a verbatim production event. Its purpose is to expose what a renderer needs to reject late cues and keep text, speech and expressions aligned.

The voice path

The Pipecat orchestrator creates a LiveKit transport, builds the pipeline, configures a pipeline task, registers event handlers and owns shutdown. The task configuration includes interruption support and optional timing and usage metrics.

The corresponding service factory resolves provider configuration. Recognition, reasoning and synthesis can therefore be constructed behind a common pipeline without hardcoding every provider choice into the avatar view.

The avatar browser participates in a different contract. It owns microphone permission, playback, the visible transcript and animation. A voice session ending should release its transport resources and return the character to a stable state, even if post-session processing continues.

Three clocks that must agree

The server’s generation clock measures when text or audio is produced. The transport clock measures when packets reach the client. The playback clock measures when audio is heard. Lip sync belongs primarily to the third.

A useful latency trace records speech end, recognizer finalization, context availability, first model output, first synthesis output and playback start. If memory retrieval delays context, a faster synthesizer will not remove that delay. If the browser blocks during a texture upload, server timings can look healthy while the caller experiences a gap.

The project contains timing and metric hooks, but this series does not convert them into an invented end-to-end percentile. An instrumented code path and a measured service-level result are different evidence.

Interruption is a coordinated cancellation

Suppose the user interrupts after half a sentence. The model may still produce tokens, TTS may still generate chunks and the browser may already have buffered audio. Clearing only one layer lets obsolete output leak into the next turn.

The cancellation protocol should stop or invalidate generation, synthesis, playback and facial cues. It should also preserve the conversation accurately: the assistant should not later assume the user heard the whole planned answer.

A transport reconnect adds another edge case. A fresh connection needs to know which generation is current; replaying all buffered events can revive an interrupted answer. The same identity rule used for visual cues should govern replay and acknowledgment.

Tools and memory should not freeze the face

A tool call can take longer than ordinary text generation. The renderer can remain in a restrained thinking or listening state while the backend reports progress. It should not mime speech merely because a model loop is active.

Memory prefetch can reduce the work required at the beginning of a voice turn. A bounded fallback can allow a response with less historical context if that is appropriate, while making the omission visible to the backend. Missing authorization context is different and must not be treated as optional memory.

Text and voice should consume the same identity and memory semantics even when their time budgets differ. Otherwise switching channels creates two assistants with inconsistent knowledge of the user.

A useful integration test

Exercise the complete path with a fixed scenario: text greeting, voice response, interruption, a tool timeout, reconnection and a follow-up referring to earlier context. Inspect the persisted messages as well as the visible output.

Pass conditions include one terminal state per admitted response, no stale audio or visemes after cancellation, no duplicate tool mutation on retry, and a coherent follow-up after channel switching. These tests turn a convincing avatar demo into an engineering contract that can survive ordinary network and service failures.