A Bolna-based telephony agent, from connection to hangup

The first client engagement ran from October 2024 to January 2025. We forked and modified Bolna, connected it to Plivo, and built calling workflows for Indian financial services, healthcare and education. The work included insurance qualification, healthcare enquiries and education follow-up.

I will walk through that system from configuration to hangup, then explain the deployment decisions that matter when you operate this type of workload today.

Separate the control plane from the call

The control plane stores agent configuration, campaigns, credentials, customer metadata and call results. Its work is ordinary request-response application work.

The media plane has a much stricter responsibility: move audio and events through an active call without accumulating delay.

Figure 3. Reference architecture. Bolna reference architecture separates portal configuration and admission from Plivo media, per-call workers and model adapters; lifecycle proceeds through connect, active, drain or transfer, and close. Scalable SVG.

The important boundary is between configuring a call and serving its audio. Our fork connected telephony handlers, model adapters and the task manager; deployment then has to give each live call enough warm capacity to meet its deadlines.

Walk through one call

Before dialling, the system resolves an immutable agent configuration: prompt version, selected services, language policy, allowed tools and business workflow. It reserves an available call slot. Making the call first and discovering there is no inference capacity afterwards creates an avoidable bad customer experience.

When the telephony provider establishes the media connection, the worker binds the provider's stream and call IDs to the internal call record. Incoming messages are validated, decoded and converted into the expected audio format.

For a configured 8 kHz μ-law stream, decoding produces linear samples at the same sample rate. Resampling to 16 kHz satisfies a recognizer's input contract; it does not restore frequencies that the telephone channel removed. The adapter must know the actual configured codec instead of treating every incoming payload as the same format.

Recognition events enter the conversational controller. Stable partials may support speculative reads. A final committed turn becomes the input to the language model. Its text is segmented into speakable units, synthesized, then encoded for the outbound telephony stream.

A disconnect cancels active streaming work, releases the admission lease and records a terminal call event. Summarization, quality analysis and downstream reporting happen outside the audio path.

The pieces we changed around the fork

The task manager contains separate queues for audio, LLM work, synthesis, transcriber output and buffered output. It tracks sequence IDs and several latency components. The Plivo output handler sends a clear-audio event on interruption.

The Whisper transcriber connects over WebSocket to the recognition service. Keeping that boundary separate let the call worker stream audio while the recognition service owned the model runtime. We also fine-tuned Whisper in late 2024; the serving and adaptation details are covered in the speech articles.

An adapter needs contract tests: language settings must propagate all the way to the remote recognizer. A configuration field in a portal is not evidence that the corresponding model request used that language.

Queues are latency storage

An asynchronous queue makes stages easier to compose. It does not make a slow stage faster.

At 20 ms per frame, 25 queued frames represent 500 ms of audio. A process can appear healthy, keep its sockets connected and still be half a second behind the customer.

Bound both queue capacity and the age of its oldest item. A queue that can grow indefinitely turns overload into delayed audio and rising memory use. Queue count alone is insufficient when chunks vary in duration.

There are different overload policies:

Blindly applying a “drop oldest” queue to every stage is dangerous. Losing 200 ms from an account number is materially different from discarding an obsolete interim transcript.

Model interoperability needs semantic adapters

A common STT interface should describe:

Transcript:
  text
  final_or_partial
  revision
  audio_start_ms / audio_end_ms
  language_hint
  confidence_if_available
  endpoint_reason

Confidence is optional. Providers do not necessarily produce calibrated or comparable confidence values.

A TTS adapter should expose the actual output format, first-audio timestamp, cancellation behavior and completion event. An LLM adapter must preserve tool IDs and arguments, not only text deltas.

This makes configuration-level substitution practical. It does not make a live mid-call swap free. A replacement recognizer needs buffered audio and a clear transcript boundary. A replacement native-audio model needs reconstructed conversation state. In the Bolna fork, we selected providers through configuration. A seamless mid-call handover needs an additional state-transfer and recovery protocol.

An admission controller protects the conversation

The following example models a single-process call limit. No network requests or telephone calls are made:

import asyncio
from contextlib import asynccontextmanager

class CallAdmission:
    def __init__(self, limit: int):
        self.limit = limit
        self.active = 0
        self.lock = asyncio.Lock()

    @asynccontextmanager
    async def reserve(self):
        async with self.lock:
            if self.active >= self.limit:
                raise RuntimeError("No warm call capacity")
            self.active += 1
        try:
            yield
        finally:
            async with self.lock:
                self.active -= 1

The distributed version needs atomic leases, expiration, renewal and fencing. A Redis read followed by an unrelated increment is insufficient: two dialers can observe the same free slot. The lease also has to account for calls still ringing, depending on which downstream resources they reserve.

A useful capacity policy is to admit based on the tightest dependency: media workers, STT sessions, LLM request/token quotas, TTS capacity or tool-service limits. A spare CPU core does not help when the speech provider refuses another stream.

Failure handling belongs in the design

The recognizer disconnects. Retain a bounded recovery buffer if the provider supports continuation. Mark recognition as interrupted. Do not fabricate a transcript across an unobserved gap.

The LLM stalls. Use an honest brief acknowledgment when appropriate, enforce a deadline and offer a fallback. Repeated filler speech can hide latency on a dashboard while worsening the call.

The synthesis service stalls. Cancel the pending answer after its deadline. A second provider should begin at a safe phrase boundary, with the same business state and a new output generation.

The call disconnects after a booking request. Reconcile the operation using its idempotency key. Disconnecting the audio stream does not undo a committed booking.

A pod is terminated. Stop accepting new calls, allow existing sessions to drain, and keep a bounded shutdown deadline. A durable transcript helps reconstruct context; it does not migrate a live TCP connection.

Telephony signaling and audio are different paths

In a SIP deployment, signaling establishes and changes the call while a media transport carries the audio. A hosted telephony provider can abstract those details and expose a WebSocket media interface to the application.

The application still has to respect the media contract. RTP, where present, supplies sequence numbers and timestamps for real-time media; it does not guarantee delivery or reserve network capacity. An application behind a provider WebSocket may receive a different event envelope and should not pretend it directly controls the underlying RTP stream. RTP specification.

For 20 ms of 8 kHz mono audio, there are 160 samples. In 8-bit μ-law that is 160 bytes before framing; in signed 16-bit PCM it is 320 bytes. A sample-rate label error can make a perfectly valid payload sound twice as fast or slow.

An original length check is:

def pcm16_duration_ms(payload: bytes, sample_rate: int, channels=1):
    if sample_rate <= 0 or channels <= 0:
        raise ValueError("Invalid audio format")
    bytes_per_frame = 2 * channels
    if len(payload) % bytes_per_frame:
        raise ValueError("Incomplete PCM frame")
    sample_frames = len(payload) // bytes_per_frame
    return 1000 * sample_frames / sample_rate

assert pcm16_duration_ms(bytes(320), 8000) == 20
assert pcm16_duration_ms(bytes(640), 16000) == 20

This validates shape, not sound quality. It cannot detect a μ-law payload incorrectly labeled PCM. Integration tests should include a known tone, known speech and expected duration through every codec boundary.

DTMF is a second input channel

An agent may need “press one” input or entry of an identifier through the keypad. Telephone-event signaling is standardized separately from ordinary speech payloads; the provider may expose decoded DTMF events instead of audible tones. DTMF telephone-event specification.

Treat keypad input as a structured event with its own collection timeout, maximum length and terminator. If digits also appear in the recognizer's transcript, avoid counting them twice.

For sensitive data, design an approved collection path that keeps it out of unnecessary transcripts and logs. This is a data-flow decision, not a request for the LLM to remember not to repeat a secret.

A transfer is a small workflow

Consider this constructed conversation:

Caller: I already explained this yesterday. Please connect me to someone.

Agent: I'll try to connect you to the support team and pass on the issue.

The agent should not announce that a person is connected merely because a transfer request was submitted.

transfer requested
  → target ringing
  → human accepts
  → context delivered
  → media bridge confirmed
  → AI relinquishes the conversation

If nobody accepts, the workflow returns to a defined fallback: offer a callback, another queue or an accurate statement that support is unavailable. Dropping the caller into silence is not a completed transfer.

A warm handoff can carry the verified customer identity, issue summary, attempted actions and unresolved question. Keep the summary concise enough for an operator to use, and distinguish caller allegations from backend-confirmed facts.

For example: “Caller reports receiving the wrong item; order A123 was located; eligibility lookup timed out; no refund was submitted.” That is more useful than “Customer is unhappy.”

Webhooks are authenticated events

Plivo documents V3 signature validation and provides SDK helpers. Validate the original external URL, method and relevant request parameters using the documented scheme; reverse-proxy URL rewriting can otherwise invalidate a legitimate signature. Plivo V3 signature validation.

Authentication establishes the event's origin. It does not make delivery exactly once or guarantee that callbacks arrive in business order. Persist provider event identifiers where available, deduplicate repeated callbacks and reject invalid state transitions.

For instance, a delayed “ringing” event should not reopen a call already marked ended. Preserve it in the event log while keeping the derived state terminal.

A realistic connection-failure drill

The proposed drill starts twenty calls, deliberately delays one recognizer connection and drops another call's media connection. Expected observations are specific:

• The delayed recognizer consumes a bounded setup deadline.
• No personalized answer is generated from an empty transcript.
• A failed call releases its admission lease exactly once.
• The dropped connection's partial recording is marked incomplete.
• Background reporting remains retryable without restarting the call.
• The other eighteen conversations continue within their operating limits.

Use the drill to test what “reliable under concurrent calling” means in practice: correct session ownership, bounded recovery and a clear outcome for every affected call.

What the architecture demonstrates

The substantive work is in connecting a customer's workflow to a real-time system with explicit lifecycle and failure semantics. The reusable capabilities include telephony integration, asynchronous orchestration, model adapters, interruption handling and operational instrumentation.

The early-2025 system delivered 40 concurrent calls, and we tested a maximum concurrency of 100. When changing the model, runtime or hosting arrangement, repeat the load test against the new configuration: its sustainable capacity depends on its own latency, queueing and failure behavior.

Next: Pipecat as a staged conversation pipeline.