Adapting Whisper Tiny for Hindi-English speech

This is a new local ASR experiment. It uses public lecture recordings in place of Zoice's internal proprietary dataset. Its model choice, training method and results are not historical Zoice production claims.

Held-out results

Absolute WER reduction: 138.34 percentage points. Relative WER reduction: 64.16%. A paired bootstrap over the 30 dataset group IDs gives a conditional 95% interval of 89.49 to 193.96 percentage points for the absolute reduction. This interval does not cover training-seed variability or a different speech domain.

Median inference-time change: +132.12%; a positive value means slower. The model architecture and parameter count are unchanged. Differences in output length and device scheduling affect this comparison, so it is not an isolated kernel-speed benchmark.

Interpretation and deployment decision

This run demonstrates a functioning adaptation and evaluation pipeline. It does not establish a production-ready Hinglish recognizer. The fine-tuned model still makes 1133 word errors against 1466 reference words, with substitutions, missing words and repetitive outputs remaining. Keep this checkpoint as an educational reference; do not replace a customer STT service with it on this evidence.

The baseline is particularly weak under this fixed Hindi-prompt, greedy-decoding configuration. Some outputs are Romanized, translated or repetitive instead of matching the mixed-script reference. WER penalizes script and spelling differences as well as misrecognition. No transliteration-tolerant metric, alternative prompt/decoder search or stronger-model baseline was evaluated. The relative improvement therefore describes this exact comparison, not superiority to contemporary ASR systems.

Across the 128 paired clips, 104 improved, 15 regressed and 9 were unchanged in word-error count. The saved error analysis includes both directions. Reducing pathological repetition contributes to the WER and tail-time improvement; median transcription time rises as the typical output becomes longer. No runtime-speed optimization was implemented.

What was actually run

• Multilingual openai/whisper-tiny, pinned revision in data/assets.json; 37,760,640 parameters as counted by PyTorch.
• 768 training clips (78.53 minutes), 96 validation clips (10.13 minutes), and 128 test clips (12.85 minutes).
• Three complete epochs; 288 optimizer steps; best fine-tuned epoch 2, chosen solely by validation normalized corpus WER.
• Full-model fine-tuning, FP32, microbatch 4, accumulation 2, effective batch 8; AdamW, learning rate 3e-5, linear decay, 5% warmup and gradient clipping at 1.0.
• Actual hardware: Apple M2 Max, 32 GiB unified memory, PyTorch 2.8.0, device mps.
• Timed training loops: 1.81 minutes. Training command including feature preparation and validation: 4.67 minutes. Downloads and final paired test are outside those times.
• Validation selected fine_tuned over the baseline. Test results were generated only after that selection.

The selection field in the machine-readable record means the better experimental candidate, not approval for deployment. Validation quality and training loss were:

Validation WER worsened in epoch 3 despite lower training loss, so epoch 2 was retained. This is consistent with overfitting on the small training subset; it does not prove that additional data or a different model will deliver a particular gain.

Memory was sampled immediately after optimizer updates:

These are sampled observations, not a true peak during forward/backward execution. The process and GPU counters can overlap in unified memory and must not be added together. The measured sufficient configuration is the 32 GiB M2 Max machine above; a lower minimum was not established. Allow 8–10 GB free disk for the isolated environment, assets, features and checkpoint. CUDA/CPU support is present but unbenchmarked. No external compute or training API was purchased.

Evaluation method

The fixed test subset contains Hindi-English mixed-script references, 2–10 second audio clips and at most 160 reference tokens. It is a filtered subset of the official test source, not the full MUCS test benchmark. Selected training data come from three of 23 training shards. Sampling is balanced across dataset group IDs.

Training, validation and test manifests have disjoint speaker_id, source session, normalized text, exact audio hash and segment ID sets. Dataset group IDs are used conservatively; actual speaker identities are not independently verified. Exact matching does not establish freedom from all near-duplicate audio or model pretraining overlap.

Primary WER is the sum of substitutions, deletions and insertions across the corpus divided by reference words. It is not the mean of sentence WERs and is not clamped to 100%. Text is NFKC-normalized, casefolded, punctuation/symbols are replaced with spaces and whitespace is collapsed. Indic marks and digits remain; no transliteration or translation is applied. Raw WER is provided to expose normalization effects.

Source transcripts and segment alignment were inherited from the public corpus and were not exhaustively checked by a human listener. Corpus quality, spelling variation and speaker-group uncertainty remain part of the evaluation limits.

Inference uses batch size one, greedy decoding, the same Hindi transcription prompt and a 192-new-token cap for both models. There are two validation warm-ups per model, then test model order alternates per clip. Timing includes host-to-device feature transfer and synchronized generation from cached log-Mel features. It excludes model loading, audio decoding, feature extraction, endpoint detection, telephony/network time, LLM response generation and TTS.

Consequently, these timings do not establish sub-one-second conversational latency or thirty-call production concurrency. No telephone-channel simulation or proprietary call data was used. A claim about live voice-agent quality would require a separate end-to-end evaluation.

A final token-diagnostic audit corrected an initial double-subtraction of the decoder prompt length. All 256 test hypotheses were regenerated and matched exactly; original WER and per-clip inference timings were retained. The original files and correction record remain under audit/ and reports/token_metadata_audit.json.

Next controlled experiment

Freeze this record. For a new experiment, first audit audio/transcript alignment on a development sample and define the intended transcription script. Compare a stronger multilingual checkpoint using the same held-out policy, then expand training coverage and assess new speakers and a separate telephone-domain test. Add code-switch, entity/number and ordinary Hindi/English regression slices. Choose decoder and normalization settings on development data only. Report new results separately rather than revising this completed test to improve its score.

Inspect the record

Download the measured record. The public record includes the training configuration, held-out results, error analysis and measurement boundaries.

Sources and attribution

The implementation follows the general sequence-to-sequence ASR workflow described in Hugging Face's Whisper fine-tuning tutorial, implemented here as an explicit PyTorch loop. The model is Whisper Tiny. Data are from the pinned MUCS mirror, sourced from OpenSLR SLR104. The original speech is technical-tutorial content, with a different distribution from customer calls. See the source and license notices for the upstream dataset license discrepancy and attribution.