Evaluation starts with the operational scenario

An agent can produce a convincing answer while acting on the wrong stay. It can call the expected tool with the wrong dates. It can create a task and then tell the guest a visit was scheduled when no slot exists.

Evaluating only the final sentence misses those failures. The client platform built evaluation around cases with context, conversation turns, expected behaviour and captured runtime evidence.

A case is more than a question

A useful case includes the agent, channel, starting scope, prior history or user turns, deterministic expectations and a qualitative rubric. The same sentence can require a different action for a prospect, a guest with one stay and a guest with several stays.

The database-backed case model supports required and forbidden tools, no-tool expectations, required or forbidden phrases, regular expressions, expected status, minimum response length and limits on iterations, latency and cost. It also carries tags and workflow categories so results can be examined by scenario family.

{
  "title": "A repeated issue should not create another task",
  "audience": "guest",
  "channel": "whatsapp",
  "turns": [
    "The shower is leaking.",
    "It is still leaking. Has anyone been assigned?"
  ],
  "fixture": {"stay": "example-active-stay"},
  "expectations": {
    "minimum_response_characters": 20,
    "forbidden_phrases": ["your booking is confirmed"],
    "rubric": "Use the existing issue and report only evidenced assignment or slot details."
  },
  "tags": ["issues", "multi-turn", "duplicate-work"]
}

This is a teaching case skeleton. A runnable integration case also needs a controlled tenant, stay and backend fixture. “Has a reservation” without an actual test reservation is not enough to establish the intended starting state.

Replay the same runtime path

The live evaluation runner invokes the application’s runtime, while the offline harness can drive the synchronous chat path in bulk and persist reports. The console exposes both cases and results.

Multi-turn cases retain a stable session and accumulate replies as history. This is essential for booking references and issue follow-ups. Replaying each user message as a fresh session tests a different product.

However, replaying text alone does not recreate the world. Calendar availability, task state, configuration and model versions may have changed. A reproducible test records or controls those dependencies. Use fixtures for deterministic contract tests and a dedicated environment for end-to-end integration tests.

Separate three testing layers

A dry-run agent eval cannot prove a real maintenance task was assigned. A backend unit test cannot prove the model explained the outcome correctly. Both are valuable, but their claims should remain separate.

Give every evaluation suite an explicit execution policy: fixture-only, read-only integration, or controlled mutation in a test environment. Classify tools by side effect and verify that classification against their behaviour. A “dry run” label alone does not prove that reads are isolated or that a misclassified write cannot occur.

Put deterministic checks first

The runner combines applicable hard assertions with optional judge results. A failed hard assertion cannot be rescued by a flattering language score.

def passes(assertions, judge_pass=None):
    hard_pass = all(value for value in assertions if value is not None)
    return hard_pass and (judge_pass is None or judge_pass)

assert not passes([True, False], True)
assert passes([True, True], None)

This small example expresses the decision rule, but the selection of assertions is the real work. A required tool name checks intent, not necessarily success. For high-value workflows, add argument, result and backend-state assertions.

For a booking case, compare the selected property and dates. For an issue case, inspect the task outcome and assignee. For authority tests, prove that the forbidden handler was not invoked. A phrase check alone can be gamed accidentally by an answer that repeats the expected words while doing the wrong thing.

Build a failure taxonomy

The useful categories follow the pipeline: wrong route, missing context, wrong scope, stale evidence, invalid tool arguments, failed backend action, unsupported final claim and failed delivery. Each points to a different owner.

The case tags can also capture domain slices: inquiry, existing stay, multiple stays, issue dispatch, host analytics, multilingual phrasing and channel formatting. A global pass rate hides a regression concentrated in a small but consequential slice.

For example, improving FAQ answers can raise the overall score while booking selection gets worse. A release gate should protect critical workflows separately from the aggregate.

Turn real failures into durable cases

The control portal connects a persisted run to case authoring. A reviewer selects the relevant history and evidence, replaces private records with fixtures, and defines the expected behaviour. This shortens the path from an observed failure to a repeatable regression test.

The original response is not automatically the reference answer. The reviewer must decide what should have happened and preserve enough starting state to reproduce the failure. Validate case completeness before accepting it into a release suite: missing history, unavailable dependencies or truncated evidence can produce a misleading pass.

Before sharing a dataset beyond the authorised environment, replace guest identities, property access details and client-specific records with fixtures. Preserve the failure mechanism while removing private data.

Test variance, not only one lucky run

Model behaviour can vary across repeated calls. A scenario that passes once and fails four times is not fixed. Repeat the difficult cases under the same configuration and track failure frequency as well as examples.

A small repeated battery is particularly useful for continuation routing, ambiguous selection and unsupported-action claims. Use a fixed dataset and record model settings, configuration and dependency state. Otherwise a change in the test environment can be mistaken for a prompt improvement.

Repeated evaluation has a cost. Budget it explicitly and use deterministic checks to eliminate simple defects before paying for judge calls.

Release gates need coverage and completeness

An absolute pass-rate floor and a comparison to the baseline answer different questions. The first protects minimum quality; the second detects regression from the current release.

Also check that the run completed the intended case set. A candidate that evaluated only the easy half of the dataset can appear better. Missing cases, infrastructure errors and cancelled cases must be visible in the comparison.

A sensible gate combines critical-case assertions, required dataset coverage and acceptable regression bounds. It should not turn every minor wording difference into a release blocker, nor allow a severe authorization failure to disappear inside a high average.

The next article looks at the subjective layer: how to use judges without allowing a model-generated score to become the only evidence of quality.