Evaluate the business state, not just the answer

An agent can write “the invoice is settled” while the record remains outstanding. It can also refuse politely when it should have created a valid support case. For this platform, answer quality and business correctness have to be evaluated together.

The existing local suite was run for this review: 303 tests passed and two were skipped. The suite covers contracts, object stores, policy paths, loop transitions, adapters and cross-loop examples. Those results establish local behaviour; they are not a benchmark of autonomous business operation or a live-provider success rate.

Four layers answer different questions

A deterministic runtime is useful in the first two layers. A framework test model can check tool wiring without calling a provider. Real model evaluation is needed for interpretation and action choice. Pydantic AI's TestModel is a testing utility, not a model-quality benchmark. TestModel documentation.

Define a scenario by its expected effects

Here is a proposed scenario format:

id: changed_quote_invalidates_approval
initial:
  quote_version: 3
  approved_version: 3
stimulus:
  edit_quote_to_version: 4
  agent_request: send_quote
expected:
  external_sends: 0
  decision: require_approval
  review_reason: quote_changed
  quote_version: 4

The expected result is not “the answer mentions approval.” It is that no external send occurred, the new state is intact and the next decision is visible. The user's explanation should agree with those facts.

For each case I want expected actions, forbidden actions, required evidence, final state, cost boundary and latency boundary. Some outcomes are acceptable alternatives: asking for a missing identifier can be correct even when the gold example used a different phrasing.

Build a failure set from the business loops

Start with duplicate enquiries, ambiguous identity, missing permission, changed scope, incomplete delivery, partial payment, repeated provider webhooks, stale approvals, out-of-order events and lost acknowledgements. Add prompt-injection text inside an email or document and verify that it does not change configuration or authority.

A cash-collection test should include a payment in the wrong currency. A support test should include an existing unresolved case. A hiring-assistance test should distinguish missing evidence from negative evidence. These details make the set representative of the decisions, not merely the vocabulary, of the business.

Separate transport simulation from reasoning evaluation

The simulated CRM, ad platform and email provider make it possible to record proposed effects without contacting anyone. That is the right environment for replay and adversarial input tests.

For a model comparison, hold the initial records and tool responses fixed, run the candidate reasoning adapters, and check the resulting decisions. Repeat stochastic cases and retain the per-run outcome. Do not replace a failed result with the best of several retries unless the deployed workflow would use those retries and pay their cost.

The current improvement-loop gate uses a supplied pass fraction to construct deterministic passing cases. Replacing that fixture with episode-derived verdicts is a concrete next step, not a finished evaluation capability.

A small dataset does not establish a rare-failure rate

Zero unauthorised actions in 100 scenarios is encouraging, but it does not establish a zero probability of such a failure. Scenario coverage and observed rates answer different questions. I would expand coverage by failure class, then measure performance on a held-out set and a restricted live trial.

Aggregate success should be accompanied by critical-failure counts. A high average cannot compensate for an unauthorised payment or cross-tenant disclosure. Those outcomes need explicit release-blocking treatment in the proposed operating policy.

Make the correction useful

A reviewer should record the incorrect action, the missing or misread evidence, the corrected decision and whether the system recovered. That turns an exception into a future test. “Bad response” is too broad to tell whether the fix belongs in retrieval, prompt construction, state logic or permissions.

The reading kit contains an offline replay example for duplicate operations, changed approvals, partial settlement and conventional payback arithmetic. It is a small executable teaching artifact. The full platform's next evaluation milestone is a reproducible multi-loop customer journey with realistic inputs, stable expected effects and deliberate crash/retry cases.

Turn an uncomfortable incident into a reusable test

Consider a customer who has paid a deposit while the remaining invoice balance is still open. The agent sees the word “paid” in a provider event and drafts a confirmation that the whole project is settled. A fluent answer evaluator may approve the message. The business-state evaluator should fail it immediately.

from company_lab.evaluation import verdict

expected = {
    "state": {"balance_minor": 400000, "status": "PARTIALLY_PAID"},
    "forbidden_actions": ["confirm_full_settlement", "release_final_deliverable"],
}
observed = {
    "state": {"balance_minor": 400000, "status": "PARTIALLY_PAID"},
    "actions": ["confirm_full_settlement"],
}
result = verdict(expected, observed)
assert result == {"passed": False, "failures": ["forbidden_action"]}

The state alone is insufficient: an incorrect message can harm the customer even if the database balance remains correct. Conversely, an excellent message does not repair a wrong posting. Evaluate both the state and the attempted external actions.

Repeat episodes without confusing the metrics

An agent that succeeds once in several attempts is useful for a supervised draft. A recurring unattended operation needs repeated reliability. Keep best-of-many task solving separate from the probability that every repetition succeeds.

from company_lab.evaluation import pass_k, wilson

assert round(pass_k(successes=9, trials=10, k=3), 3) == 0.7
low, high = wilson(successes=9, trials=10)
assert round(low, 3) == 0.596
assert round(high, 3) == 0.982

Here pass_k estimates the fraction of three-run subsets in which all three runs succeeded. It is not the best-of-three pass@k metric. The wide interval around nine successes in ten trials makes the uncertainty visible. Pooling unrelated easy and hard scenarios into one number can hide a broken workflow; calculate scenario-level results before aggregating.

The τ-bench paper evaluates tool-using agents against policy and final state, and introduces repeated-trial reliability reporting. I use that distinction to separate an impressive demonstration from a dependable operating loop. τ-bench.

Make the environment part of the fixture

A useful scenario contains initial objects, a clock, provider responses, incoming events, role assignments and expected state. Freeze the clock for expiry tests. Control provider behaviour for timeout tests. Reset the database between episodes. Otherwise a second attempt may pass because the first attempt already performed the task.

Keep a separate family of restart tests that intentionally reuse persisted state. Those ask whether a process can resume without repeating an effect. The test runner must know whether it is evaluating a fresh episode or a continuation.

The new reference package includes 62 deterministic tests. They exercise contracts such as stale approval, expired leases, receipt reconciliation, budget limits and tenant filtering. They do not establish how any particular language model performs; that requires running real model episodes through the same observation boundary.

Review the failures that aggregation hides

Treat unauthorised spending, cross-tenant data access and a false customer commitment as distinct severity classes. Do not allow a hundred well-written drafts to offset one such failure. In a pilot, inspect every critical case and keep the underlying event trace.

Human judgement still matters for tone, ambiguity and whether the proposed resolution is sensible. Give the reviewer source facts and the action timeline rather than only the answer. Disagreements become adjudicated labels and new scenarios; they should not be silently averaged away.

The exercise is to mutate the partial-payment fixture in three ways: duplicate the event, change its currency, and deliver it after a collection reminder was drafted. The correct system deduplicates the first, escalates the second and refreshes the third before sending. A single happy-path refund example would miss all three behaviours.