Judges, human review and regression evidence

An evaluation judge can confidently mark an answer wrong because it never saw the tool result that justified it. It can also reward a clear, friendly explanation that contains an invented operational claim. Adding an LLM judge does not remove the need to design the evaluation carefully.

The client platform exposes judge configuration, results, reasons, human review and baseline comparison in the console. The objective is to make the grading process inspectable, not to replace engineering judgement with a single score.

Score different qualities separately

Groundedness, helpfulness, escalation and format answer different questions. A configurable evaluation layer can vary the rubric, judge model, repeat count and aggregation rule while keeping those dimensions visible to reviewers.

These dimensions should remain distinct. An answer can be helpful but ungrounded, grounded but incomplete, or correct in content but unsuitable for a messaging channel. Collapsing them too early hides the reason a case failed.

A rubric should define the boundary between adjacent scores with domain examples. “Five means excellent” is not a reliable grading contract.

Give the judge the evidence it needs

For operational agents, a judge needs the request, relevant history, final response, selected tool arguments, tool outcomes and applicable context. Otherwise it is judging plausibility rather than support.

Build judge evidence from the correlated run record: the request, selected context, tool arguments, tool outcomes and final reply. Mark omitted or truncated information explicitly. A missing field must not be interpreted as proof that the underlying event did not occur.

A useful evidence packet makes omissions explicit:

{
  "request": "When will someone arrive?",
  "task_result": {"assigned": true, "slot": null},
  "reply": "A team member has been assigned; a time is not yet confirmed.",
  "evidence_complete": true,
  "rubric": "Do not reward an arrival time unless a booked slot supports it."
}

If the slot field is missing because capture was truncated, that is different from an authoritative slot: null. The grader should be allowed to express insufficient evidence rather than inventing certainty.

Groundedness is about claims, not word overlap

An answer can paraphrase a tool result correctly without sharing many words. Conversely, copying an irrelevant property paragraph can have high lexical overlap and still be wrong for the guest.

Groundedness review should identify material claims and their supporting evidence. For this domain, claims about price, dates, availability, assignment, notification and confirmation deserve particular attention. A small adjective about tone is not as consequential as a false booking confirmation.

Deterministic checks can verify many structured claims before the judge is invoked. The judge is most useful for interpretation and completeness where a simple assertion would be brittle.

Compare cases, not just averages

The comparison layer keys results by case slug. It identifies regressions, fixes, unchanged cases and cases present in only one run. Judge-score deltas are displayed alongside pass/fail changes.

Consider a hypothetical ten-case comparison: two regressions, two fixes and six unchanged. The pass rate might remain identical, but the release could be unacceptable if the regressions concern authority and the fixes concern wording.

def regressions(baseline, candidate):
    common = baseline.keys() & candidate.keys()
    return sorted(
        case for case in common
        if baseline[case] is True and candidate[case] is False
    )

assert regressions(
    {"booking": True, "faq": False},
    {"booking": False, "faq": True},
) == ["booking"]

The example is deliberately small. The console comparison additionally retains result IDs, tool information and judge deltas so a reviewer can open the evidence behind the change.

Pairwise judging and position bias

The implemented pairwise path compares candidate and baseline replies, then repeats the comparison with their order swapped. It maps the second verdict back to the original ordering and records whether the judgments agree.

This is useful because judges can favour whichever answer appears first or second. Swapping does not eliminate every bias, but an inconsistent result should not be treated as a confident win.

The comparison should use the same scenario evidence and keep the answer labels neutral. Candidate identity, expected winner and an author’s preferred outcome should not be part of the grading prompt.

A tie is a legitimate result. So is an undecided or inconsistent comparison. Forcing every pair into a winner produces precision the evidence does not support.

Human calibration measures the evaluator

The console supports human labels of pass, fail and unsure. The calibration view compares those labels with the automated verdict and reports agreement, false passes and false fails. In this implementation the automated verdict includes assertions and judges, so that comparison is broader than judge-only calibration.

The most important error depends on the use case. A false pass on an invented confirmation is costly. A false fail on a correct but terse answer wastes review effort. Report both rather than relying on agreement alone.

Suppose, as a worked example, reviewers label 80 outputs. The automated decision agrees on 68, falsely passes 5 and falsely fails 7. Agreement is 85%, but the five false passes need inspection before the evaluator is trusted as a release gate. The example is not a client benchmark.

Review sampling should include random outputs as well as known failures. Reviewing only the worst-looking cases produces a useful debugging queue but a biased estimate of overall agreement.

Keep evaluation spend visible

Judges consume tokens too. The system separates agent cost from judge cost in reports and exposes both in spend views. Multiple judges, repeated samples and order-swapped pairwise comparisons can multiply the bill even when the underlying agent task is cheap.

Start with deterministic assertions and one calibrated judge for the dimensions that need it. Add a panel when disagreement analysis justifies the extra cost. More judges do not automatically create independence if they share similar blind spots or incomplete evidence.

Pin the judge template and model setting with the report. Changing the grader at the same time as the agent makes a baseline comparison difficult to interpret.

Use disagreement to improve the system

When a human and judge disagree, inspect whether the issue is the agent, the rubric, missing evidence or an incorrect fixture. A strict judge is not necessarily a bad judge; a permissive one is not necessarily realistic.

The useful loop is: inspect the trace, correct the case or implementation, update the rubric if needed, and rerun a stable comparison. This keeps the evaluation system connected to actual operational failures rather than turning it into a leaderboard of unexplained numbers.

Judges are most valuable as review tools with evidence and calibration. The release decision remains a combination of hard constraints, scenario coverage, observed regressions and accountable human interpretation.