Voice, multilingual input and human handoff

Our command bar includes browser speech recognition. In the inspected component, it uses interim transcripts and an en-US language setting, then places the transcript into the command input. That is a useful voice entry point. It is not evidence of a deployed telephone agent or a validated Hindi/Hinglish service.

To extend it, I would keep the same administrative workflow after transcription: resolve the patient, show the interpreted fields, prepare a proposal and confirm it. Voice should change how intent enters the application, not remove the review boundary.

Treat the transcript as a revisable observation

An interim transcript can change. If the recognizer initially hears “cancel” and later corrects it to “can we move,” the system must not already have executed a cancellation. I would allow interim text to improve responsiveness while restricting action preparation to a final utterance or an explicit staff submission.

{
  "utterance_id": "utt-001",
  "sequence": 4,
  "is_final": true,
  "language_hint": "hi-IN",
  "text": "Kal afternoon follow-up shift karna hai",
  "context_version": 12
}

This is a reference event, using a worked Hinglish phrase. It expresses a rescheduling request but leaves the exact date, time, appointment and location unresolved. The backend should interpret it under the clinic's time zone and selected context, then ask a precise follow-up or show candidate slots.

Preserve names and numbers carefully

Code-switching introduces several independent errors: language identification, transliteration, name recognition and number interpretation. A recognizer can transcribe the sentence fluently while changing a patient name or time.

I would keep both the original transcript and a normalized interpretation in the short-lived review context. Patient selection should still use verified records. A spoken name narrows candidates; it should not silently establish identity. For dates and times, show the resolved value in the user's preferred display language before preparing the write.

The evaluation should therefore score task fields, not only word error rate. A missing filler word is less consequential than changing “fifteen” to “fifty.” For scheduling, exact appointment identity, local date, time, practitioner and cancellation intent are the important fields.

A telephone agent needs a different runtime

A live call introduces audio transport, voice activity detection, endpointing, speech recognition, model latency, synthesis and interruption handling. The browser component does not provide those pieces automatically.

For a future telephone connector, I would represent each call as a session linked to an administrative workflow. Audio frames and model tokens are ephemeral; confirmed action receipts and handoff tasks are durable. The call can end while a staff task remains open.

def may_prepare_action(utterance, interpretation):
    return (
        utterance["is_final"]
        and interpretation["required_fields_complete"]
        and not interpretation["identity_ambiguous"]
        and interpretation["operation"] in {
            "appointment.prepare", "communication.prepare"
        }
    )

This policy fragment is a reference example, not an ASR implementation. Permission checks and proposal confirmation still happen after it. A numerical ASR confidence value should be calibrated for the chosen recognizer and language before it is used as a threshold.

Handle interruptions at the action boundary

Suppose the assistant begins reading a 2:30 PM proposal and the caller interrupts with “No, tomorrow.” The audio system should stop playback and revise the interpretation. If confirmation has not occurred, the old proposal can be superseded. If a booking call is already in flight, the workflow needs to establish its outcome before preparing a change.

I would keep separate flags for speech_playing, proposal_pending and action_in_flight. A single busy flag cannot express what may be safely interrupted. The transcript should also distinguish the caller's actual confirmation from a word the assistant itself just spoke; echo or overlapping audio must not become consent.

Make a handoff useful to the next person

For an administrative exception, I would create a handoff packet containing the unresolved task, verified identifiers, completed operations and the exact reason human help is needed:

{
  "handoff_id": "handoff-001",
  "workflow_id": "wf-001",
  "queue": "appointment-coordination",
  "reason": "identity_ambiguous",
  "verified_patient_id": null,
  "completed_effects": [],
  "pending_question": "Confirm which patient record to use",
  "preferred_language": "Hindi"
}

The receiving team should acknowledge ownership. If transfer fails or no one is available, the interface needs a visible fallback state and a durable callback task according to the organization's policy. An assistant saying “I am transferring you” does not prove anyone received the case.

If a conversation moves into clinical concerns, the administrative workflow should follow the organization's clinician-approved escalation procedure. This series does not define symptom triage or medical advice. The relevant engineering work is detecting an out-of-scope request, preserving necessary context and making the handoff observable.

Measure a complete turn

I would measure end-of-utterance to usable response, interruption stop time, field correction rate, confirmed action accuracy and successful handoff acknowledgement. A fast first audio packet is not useful if the appointment time is wrong.

For the first multilingual evaluation, I would use consented or purpose-created recordings across English, Hindi and Hinglish, with varied accents, clinic names, telephone quality and background noise. Keep speakers separated between development and held-out evaluation. Include negation, self-correction, repeated dates and a caller who changes the requested action mid-sentence.

The implementation kit contains the proposed voice test matrix and handoff contract. It makes no claim of measured speech accuracy. That is the next evidence needed before presenting multilingual voice as a completed capability.