Self-hosted Whisper: streaming, scheduling and the concurrency trap

We fine-tuned Whisper in late 2024 and connected the client platform to a Whisper service through a WebSocket adapter in our Bolna fork. That separates two responsibilities: the call worker streams audio and manages the conversation; the recognition service owns inference.

Here is how I reason about that service today—rolling audio windows, stable transcript updates, deadline-aware scheduling and the capacity needed for concurrent calls.

Whisper is an encoder–decoder speech-recognition model. Its ability to transcribe an audio segment does not by itself make it a continuously streaming recognizer. Whisper paper.

The serving problem

Telephone audio arrives in small packets. A model generally benefits from more context than a single packet contains.

If every 20 ms packet triggers an independent decode, the system wastes compute and lacks linguistic context. If it waits for the whole utterance, the final transcript arrives too late. The streaming layer must decide when to decode, what context to retain and which words are stable enough to commit.

media packets
  → stateful codec conversion and resampling
  → per-call rolling audio buffer
  → speech detector
  → decode scheduler
  → GPU inference
  → hypothesis alignment and stable-prefix decision
  → partial/final transcript events

Use a streaming resampler that retains filter state between packets. Resampling each packet independently can introduce edge artifacts. Keep a mapping between sample indices and call timestamps so trimming the buffer does not reset time.

Figure 6. Illustrative schedule. Successive four-second recognition windows overlap by three and a half seconds. The received-audio, decoded-audio and committed-transcript cursors are distinct. Scalable SVG.

Repeated decoding must not repeat the transcript

A rolling recognizer may produce:

pass 1: mujhe policy
pass 2: mujhe policy renew
pass 3: mujhe policy renew nahi karni

The application must not concatenate the three strings. They are revisions of overlapping audio.

A simple local-agreement policy commits a prefix that agrees across consecutive hypotheses. This idea is demonstrated in Whisper-Streaming, whose published reference reports a latency regime of seconds; it is not evidence of an 800 ms complete agent. Its repository also points to newer streaming work. Whisper-Streaming.

The following example demonstrates the prefix calculation for the same untrimmed window:

def stable_prefix(previous: list[str], current: list[str]) -> list[str]:
    end = 0
    for left, right in zip(previous, current):
        if left != right:
            break
        end += 1
    return current[:end]

assert stable_prefix(
    ["mujhe", "policy", "renew"],
    ["mujhe", "policy", "renew", "nahi", "karni"],
) == ["mujhe", "policy", "renew"]

This does not mean a renewal should be performed. The stable words are still part of an unfinished sentence whose final meaning is negative.

A real implementation aligns hypotheses across trimmed windows, tracks committed offsets, handles punctuation revisions and finalizes the remaining suffix at end of turn. Prefix matching alone is not a complete streaming algorithm.

Choose a decoding schedule

An initial experiment can use:

These are tuning candidates, not measured optimal settings.

Retaining a longer window improves context but increases repeated work. A 4-second window decoded every 0.5 seconds processes roughly eight times the new audio duration before accounting for caching, trimming and implementation details. A recognizer that is fast offline can therefore become expensive when wrapped naïvely for streaming.

Shortening the window may damage code-switching, names or numbers. Compare both transcript stability and final WER when tuning it.

Use an optimized runtime without confusing throughput and latency

Faster-whisper uses CTranslate2 and supports quantized inference. Its published large-v2 benchmark on an RTX 3070 Ti 8 GB reports 63 seconds for 13 minutes of audio with FP16, beam size 5, versus 17 seconds with batch size 8; reported VRAM rises from 4,525 MB to 6,090 MB. That benchmark measures offline transcription. For a voice agent, we also need to measure finalization delay and queueing under concurrent calls. Faster-whisper benchmark.

Batching improves utilization because one GPU can process compatible work together. Waiting for the batch introduces delay. Padding short clips to a long clip's shape can waste work. The scheduler should group similar lengths while respecting per-request deadlines.

For live speech, optimize the maximum safe queue age and p95 finalization delay. “Largest batch that fits in memory” is usually the wrong objective.

A capacity calculation

Let:

N = simultaneous calls
d = fraction of wall time with caller speech
r = single-stream compute seconds per audio second
w = streaming repeated-work multiplier
u = target usable GPU fraction

approximate GPUs = ceil(N × d × r × w / u)

This is a planning approximation. It assumes roughly linear work and excludes batching gains, shared overhead, memory constraints and correlated speech bursts.

With illustrative assumptions N=40, d=0.45, r=0.06, w=3 and u=0.65:

40 × 0.45 × 0.06 × 3 / 0.65 = 4.98 → 5 GPUs

The point is not that forty calls require five GPUs. None of these values has been measured for a target deployment. The point is that ignoring repeated decoding and headroom can produce a very different estimate. Replacing r with the actual loaded streaming service measurement is essential.

At 100 calls, the same assumptions produce 12.46, rounded to 13 GPUs before redundancy. A new model, batching strategy or streaming algorithm may change this substantially.

Separate session state from loaded weights

One model copy per phone call wastes memory. A serving process should load weights once, then multiplex bounded recognition requests while retaining separate buffers and hypothesis state for each call.

The scheduler needs fairness. A caller speaking continuously should not prevent nineteen short replies from being recognized. Earliest-deadline scheduling with a per-call pending-work cap is a useful starting point.

Cancellation also needs to release capacity. If an utterance has been superseded, discard its pending decode. If GPU work is already running and cannot be preempted, suppress its obsolete result and account for the wasted work.

Training format and serving format are separate artifacts

A fine-tuned Transformers checkpoint may need conversion before CTranslate2 can serve it. Preserve the tokenizer, preprocessing configuration, model revision and decoding policy with the converted weights.

Compare converted and original outputs on a frozen validation sample. Quantization or a changed decoder can alter recognition. Export success alone is not a quality check.

For deployment, store a release manifest with:

• Base model and fine-tuned checkpoint hashes.
• Runtime, CUDA and numerical precision.
• Audio conversion and language settings.
• Decoder settings and generation limits.
• Streaming-window and commit policy.
• Measured latency, quality and concurrency envelope.

What makes the service production-capable

The live service needs tests beyond a transcription notebook: silence, noise, packet gaps, long speech, rapid code-switching, abrupt hangups and multiple synchronized speakers across different calls.

Monitor audio-buffer age, inference-queue age, decode duration, transcript revision rate, finalization delay, GPU memory and capacity rejections. A rising revision rate can indicate a quality problem even while the GPU remains healthy.

The right outcome is a stated envelope: a particular model and runtime, on particular hardware, sustains a defined traffic mix within quality and latency limits. The forty-call delivery gives us a useful workload to reproduce when benchmarking a replacement recognition service.

Follow one buffer through overlapping windows

Suppose a caller speaks for eight seconds. A proposed wrapper decodes a maximum four-second window every half-second. After the initial warm-up, the window ending at 4.0 seconds covers 0.0–4.0; the next covers 0.5–4.5; the next covers 1.0–5.0.

Most audio is processed repeatedly. That repetition supplies context, but every retained token and audio offset has to be reconciled with earlier output.

The wrapper needs three cursors:

received_audio_end: newest sample available
decoded_audio_end: newest sample included in a finished inference
committed_audio_end: last stable region already emitted downstream

They are related but not interchangeable. Advancing the committed cursor when an inference merely starts can permanently lose words if that inference fails.

Trim audio only after preserving enough overlap for alignment. Retain absolute sample offsets so a word at 3.2 seconds does not become a word at 0.2 seconds simply because the buffer was shortened.

Schedule for deadlines and fairness

Batching is useful when requests can share an execution shape. State and ordering complicate it. Triton's documentation distinguishes dynamic batching for stateless requests from sequence-oriented scheduling for stateful models. A serving wrapper must decide which state belongs in the request and which belongs to the worker. Triton scheduling and batching.

The following helper selects a compatible batch from pending jobs. It is a deterministic policy illustration, not a complete GPU scheduler:

from dataclasses import dataclass

@dataclass(frozen=True)
class DecodeJob:
    call_id: str
    deadline_ms: int
    shape_bucket: int

def choose_batch(jobs, now_ms, limit):
    if limit < 1:
        raise ValueError("Batch limit must be positive")
    ready = sorted(
        (j for j in jobs if j.deadline_ms > now_ms),
        key=lambda j: j.deadline_ms,
    )
    if not ready:
        return []
    bucket = ready[0].shape_bucket
    selected, seen = [], set()
    for job in ready:
        if job.shape_bucket == bucket and job.call_id not in seen:
            selected.append(job)
            seen.add(job.call_id)
        if len(selected) == limit:
            break
    return selected

The scheduler selects at most one job per call and preserves shape compatibility. Production code must separately account for expired jobs, reinsert deferred work, coordinate running jobs and release reservations after cancellation. It should never silently lose caller audio because a scheduling deadline was missed.

Silence can still produce text

A recognition model is not a reliable speech-presence detector merely because its output is text. Noise, long silence or music can produce plausible words, especially with an autoregressive decoder and strong prior context.

A proposed service combines independent speech detection with recognition diagnostics, such as repeated tokens, output length and low-confidence regions where the runtime exposes them. These diagnostics are reasons to investigate or ask for clarification; they are not universal rejection thresholds.

Include silent and non-speech recordings in the test set. Their expected output can be empty, but they still need a defined false-transcription metric.

A worked code-switching repair

Constructed caller audio:

“Policy number AB seven two hai—sorry, seven three.”

A naïve concatenation can produce “AB72AB73.” The desired final entity is AB73, with the correction retained in the conversation.

Recognition can preserve the utterance verbatim. A separate entity-resolution step interprets the correction, and the agent confirms the result:

“AB73—A, B, seven, three. Is that correct?”

Do not train the recognizer to silently rewrite every correction into a single normalized value. A verbatim transcript and a normalized business entity serve different purposes, and the latter needs its own evaluation.

Quantization is another experiment

An INT8 deployment may reduce memory and improve throughput, but quality equivalence should be demonstrated for the target language slices. Keep the fine-tuned checkpoint and the exported runtime artifact separate in the record.

Compare baseline precision and quantized inference on the same clips, with the same decoder and normalization. Then test live scheduling, because a memory saving may allow a larger batch while increasing per-request delay.

A useful result reports both the quality difference and the capacity envelope. A single statement such as “INT8 is faster” leaves out the decision the system designer actually needs to make.

Next: a 2026 multilingual adaptation design.