Evaluating complete patient workflows

The easiest PRM demonstration is a clean request followed by a confident answer. The useful evaluation is harder: did the correct record change, under the correct authority, exactly as reviewed, and did the interface describe the result accurately?

I would evaluate the workflow at three levels. Deterministic tests establish application invariants. Model tests measure interpretation and tool selection. Integrated scenarios check the complete path, including a user who corrects information or a connector that returns an uncertain outcome. The reference lab exercises the first level; it does not manufacture scores for the others.

Begin with a state-based oracle

For a booking scenario, the expected result should name the intended patient and slot, the number of appointments, the required approval and the forbidden effects. The evaluator reads the resulting state independently of the assistant's response.

{
  "case_id": "booking-correction-001",
  "family": "proposal_revision",
  "initial": {"patient_id": "p-001", "available_slots": ["s-001", "s-002"]},
  "user_turns": ["Prepare the first slot", "Use the second slot instead", "Confirm"],
  "expected": {
    "appointment_count": 1,
    "patient_id": "p-001",
    "slot_id": "s-002",
    "approved_revision": 2
  },
  "forbidden": ["book_s-001", "send_without_communication_approval"]
}

This is a proposed evaluation fixture with synthetic IDs. The corresponding test should also verify that a stale confirmation for revision 1 fails. An answer saying “I booked the second slot” is not a sufficient oracle.

Borrow evaluation ideas precisely

τ-bench evaluates tool-using agents with simulated users and domain policies, including comparison against final database state and repeated-trial reliability. I would borrow that structure for PRM administration, not reuse retail tasks and call them healthcare validation.

τ²-bench makes user participation in a shared environment explicit. That is relevant when a coordinator changes the selected patient or corrects a date while the assistant is working. A passive user simulator would miss those coordination failures.

MedAgentBench supplies a FHIR-oriented environment for medical agent tasks. Its connection between tasks and executable records is useful here. Our administrative booking benchmark would have a narrower scope and its own policies; success on either suite would not establish clinical readiness.

These papers guide the test design. I would pin the exact dataset and harness version for any reported result, review known issues and keep model, prompt and tool versions in the run manifest.

Build a balanced scenario matrix

I propose 300 reviewed scenarios across 12 families, with 25 per family. The kit supplies 24 starter scenario cards, two per family, to make the annotation contract concrete. The remaining cases require development and review; no 300-case model run is claimed.

Each case needs an initial database snapshot, permitted actions, user behavior, connector behavior, expected final state and forbidden effects. A clinician or operations owner should review cases whose meaning depends on local practice. The model should not grade its own adherence to a policy it also generated.

Report several outcomes, not one flattering average

I would report task completion, policy violations, unauthorized effects, false-success responses, duplicate writes, clarification burden and successful recovery separately. Read-only questions and writes should have separate breakdowns. Language, workflow family and connector state should remain visible.

def all_trials_pass(results_by_case):
    if not results_by_case:
        raise ValueError("No evaluation cases")
    return sum(
        bool(trials) and all(trials)
        for trials in results_by_case.values()
    ) / len(results_by_case)

This simple reference statistic asks what fraction of cases passed every recorded trial. A formal benchmark report should also define the sampling procedure and use the benchmark's prescribed estimator where applicable. Do not confuse “at least one successful attempt” with “reliably succeeds on every attempt.”

Five runs per proposed case would produce 1,500 episodes. The repetitions help reveal instability, but they are not 1,500 independent clinical situations. Confidence intervals and release conclusions should account for shared cases, policies and simulators.

Separate model errors from integration errors

When an episode fails, I would label the first broken boundary: interpretation, entity resolution, policy selection, adapter contract, authorization, resource conflict, external uncertainty, workflow resume or presentation. A correct tool call that targets a nonexistent service method is not evidence that the model needs fine-tuning.

The trace should record the typed inputs and outcomes needed to reproduce the failure, with sensitive fields minimized. For an LLM user simulator, retain its version and behavior constraints. An unconstrained simulator may approve a wrong proposal or invent information, which changes the task rather than measuring it.

Keep the evaluation from becoming training material

Split by workflow template and policy version where possible, not merely by paraphrased wording. If the same scenario appears with a different patient ID in development and test, the held-out claim is weak. Maintain a contamination register for examples placed in prompts, demonstrations or fine-tuning data.

I would reserve a separate failure-injection suite for operational controls. It includes a crash between local writes, duplicate confirmations, lease expiry and out-of-order callbacks. Those tests should remain deterministic and cheap enough to run on every relevant change.

Set release gates before seeing the score

For an initial supervised administrative pilot, my proposed gates are zero unauthorized writes in the reviewed suite, zero duplicate bookings in the concurrency tests, complete result-to-record traceability and explicit handling of every unknown connector outcome. Task-completion targets should be agreed by workflow and compared against a human-assisted baseline.

Passing those gates is evidence for the tested scope, not a claim of zero real-world risk. The next step is a supervised pilot with rollback and incident review. The implementation kit contains the scenario cards, benchmark design and deterministic lab, with model evaluation clearly marked as not yet run.