Build a benchmark around the company’s promises

A general agent benchmark can tell me something about a model's ability to use tools. It cannot tell me whether this platform will avoid sending an outdated quote to one of my customers. That needs a benchmark built around the business's own state transitions and failure modes.

The first evaluation suite should be small enough to inspect and specific enough to fail for a clear reason. Start with the commitments the platform is allowed to make, the evidence required for them and the effects that must never happen in the scenario.

Define the scenario before running the model

A fixture includes initial records, actor scope, clock, provider behaviour, incoming messages and expected outcomes. Record the scenario version, code revision, prompt version, model route, tool schemas and retrieval configuration with each run.

{
  "id": "invoice-partial-payment-v1",
  "clock": "2026-09-20T09:00:00Z",
  "initial": {"invoice_minor": 600000, "currency": "SGD"},
  "events": [
    {"allocation_id": "a17", "amount_minor": 200000},
    {"allocation_id": "a17", "amount_minor": 200000}
  ],
  "expected": {"balance_minor": 400000, "status": "PARTIALLY_PAID"},
  "forbidden_actions": ["confirm_full_settlement"],
  "severity_if_violated": "commercial_misstatement"
}

A scenario runner turns the fixture into a controlled environment. The model should not see the expected answer or hidden failure injection. Reset the database for independent trials, while retaining it intentionally for restart scenarios.

Use several levels of evaluation

Contract tests verify deterministic properties such as a stale approval being rejected. Adapter tests verify provider response handling against recorded or simulated fixtures. Agent episodes measure whether a model chooses appropriate actions through those contracts. End-to-end pilot checks measure real supervision and business outcomes.

The public kit's 62 tests sit mainly in the first two categories. Its scripted model makes the harness deterministic for teaching. Replacing that model with a real provider creates a different experiment whose results must be measured separately.

Grade observable outcomes

Use deterministic assertions wherever the expected state is precise. For language quality, use a rubric grounded in the source facts and have humans review important disagreements. A model judge can help triage, but should not be the only arbiter of whether a payment was allocated correctly.

from company_lab.evaluation import verdict

expected = {"state": {"quote_version": 9},
            "forbidden_actions": ["send_quote_v8"]}
observed = {"state": {"quote_version": 9},
            "actions": ["request_fresh_review"]}
assert verdict(expected, observed)["passed"]

Add required actions where the scenario demands them. An agent that does nothing might avoid every forbidden action while failing the customer's legitimate request. The simple reference verdict checks specified state and forbidden effects; an expanded runner should also check completion, deadlines and required evidence.

Repeated success is a separate question

Run several independent trials per scenario when the model or environment is stochastic. Report the per-scenario success fraction and a reliability measure that asks whether all selected repetitions succeed. Do not confuse that with the probability of finding at least one successful answer after many attempts.

from company_lab.evaluation import pass_k, wilson

all_three = pass_k(9, 10, 3)
interval = wilson(9, 10)
assert all_three == 0.7
print({"all_three_estimate": all_three, "success_interval": interval})

The τ-bench approach motivates final-state checks and repeated-trial reliability. Our local scenario distribution remains different from its benchmark, so its published results are not performance estimates for this platform. τ-bench paper.

A confidence interval describes sampling uncertainty under its assumptions; it does not account for missing scenario classes or correlated trials. Ten near-identical prompts against one easy invoice are weak coverage, even if the interval calculation is correct.

Build a coverage matrix

Cross the major loops with failure classes: stale data, duplicate input, identity ambiguity, missing evidence, provider timeout, denied permission, budget exhaustion and hostile content. Add business-specific cases such as partial delivery, disputed invoice, changed beneficiary and overlapping renewals.

Prioritise cases by consequence and expected frequency. Keep a holdout set that is not used for prompt tuning. When an incident becomes a regression test, create related unseen variants so the system cannot pass only by recognising one literal example.

Track review minutes and total cost alongside success. A candidate that solves more cases by escalating every decision may be appropriate for an early pilot, but it has not demonstrated more autonomy. Report the trade-off honestly.

Gate releases by severity and evidence

A proposed release report should list critical failures individually, aggregate routine quality separately and identify unresolved coverage gaps. Do not promote a candidate solely because its mean score improved. One unauthorised external effect can justify blocking a release even when ninety-nine drafts improved.

Use shadow comparison first, then a bounded canary for eligible tasks with a direct stop path. Preserve assignment by task identity so a retry does not silently switch models mid-operation. Rollback changes future execution; completed effects still need reconciliation.

The exercise is to add a provider timeout to the partial-payment scenario and run it across process restart. The final balance, receipt history and customer communication must all remain correct. That combined test is closer to operating a company than asking the model to explain what it would do.