Typed scoring for financial answers and agent traces

A financial answer is more than a number embedded in a sentence. It has a unit, a subject, a date, assumptions and a source. An agent answer also has an execution history: which tool was called, with what arguments, under whose authority and with what effect.

The scorer treats those fields as part of correctness. This makes evaluation stricter and more informative than searching for a familiar number in generated text. It also makes failures localizable: the result can be numerically right and still fail because it refers to the wrong snapshot.

Define the answer before collecting predictions

The authored future-value case expects ₹3,345,563.94 for ₹25 lakh over five years at an assumed annual 6% return. Its claim contract looks like this:

{
  "future_value": {
    "value": "3345563.94",
    "unit": "INR",
    "subject": "scenario-A",
    "as_of": "2026-09-01",
    "source": "calc-fv-v1"
  }
}

The expected tool call carries the principal, decimal annual rate and whole-year horizon. The expected policy identifies the result as illustrative rather than guaranteed. The scorer also requires zero portfolio writes, because this workflow is read-only.

The calculation is deterministic. The language model is being tested on whether it extracts the request, uses the contract and returns the supported result. The evaluator does not ask a second model to decide whether the arithmetic “looks reasonable.”

Ten examples of what passes and fails

The following are authored scorer probes against a 60% equity-concentration contract. They are deliberately small enough to inspect without loading any model.

The exact equality here is intentional because the fixture specifies deterministic rounding. A different benchmark may require a tolerance. If it does, define an absolute and relative tolerance appropriate to the unit and record the normalization rule. Do not apply a generous percentage tolerance to account identifiers, dates or policy labels.

Use decimal arithmetic and explicit fields

The scalar comparison should reject non-finite values. The structured comparison should require the same field set, so a missing subject or an extra unsupported claim does not disappear into a permissive parser.

from decimal import Decimal, InvalidOperation

def decimal_equal(actual, expected):
    try:
        a, b = Decimal(str(actual)), Decimal(str(expected))
        return a.is_finite() and b.is_finite() and a == b
    except (InvalidOperation, ValueError, TypeError):
        return False

def same_claim(actual, expected):
    if set(actual) != set(expected):
        return False
    return all(
        decimal_equal(actual[key], value) if key == "value"
        else actual[key] == value
        for key, value in expected.items()
    )

Normalization belongs before this comparison and must be tied to the task. For public numerical datasets, a ratio such as 0.935 can correspond to a display answer of 93.5%. That does not mean a scorer should divide any inconvenient prediction by 100 until it matches. The gold representation defines the convention.

For evidence-grounded explanations, evaluate claim support separately from citation formatting. A syntactically valid citation can point to a passage that does not support the claim. The FinanceBench paper is a relevant research reference for financial questions grounded in document evidence.

Score attempted execution as well as the final state

The reference runner supplies a trusted list of attempted tool calls. The scorer checks available tools, household scope, required argument values and the final write count. A model cannot pass by declaring that it made no unsafe calls after the runtime recorded one.

The 24 public fixtures use a specific expected trace to make their contracts easy to inspect. That is deliberately narrow. A broader task can have several valid plans; in that case, score permitted actions, required evidence and final state rather than requiring one exact sequence.

For example, a system might reuse an already authorised snapshot instead of fetching it again. That can be valid if the task contract permits cached evidence of the correct date. The benchmark must define this before comparing candidates; otherwise a harmless implementation difference can masquerade as a capability difference.

Keep missing predictions in the denominator

An evaluation has a fixed set of attempted cases. A crashed request remains a failed attempt. Duplicate IDs and unknown IDs are rejected because they make joining predictions to cases ambiguous.

def join_predictions(cases, predictions):
    known = {case["id"] for case in cases}
    by_id = {}
    for prediction in predictions:
        case_id = prediction["id"]
        if case_id not in known or case_id in by_id:
            raise ValueError("unknown or duplicate prediction ID")
        by_id[case_id] = prediction
    return [(case, by_id.get(case["id"])) for case in cases]

This rule prevents a timeout-heavy candidate from looking better simply because only successful responses reached the report. Record attempted, submitted, completed, passed and critical-failure counts separately. Include parser and runtime failures as their own categories.

The reference scorer treats a runtime error as a failure even if a partial final object contains the right number. A late or incomplete response is not the same delivered product outcome as a valid response within the task budget.

Group uncertainty around independent tasks

The public kit has 24 rows but eight scenario families. Translations within a family are correlated. A row-level percentage is descriptive; it should not be presented as 24 independent trials of broad financial competence.

The scorer reports per-language counts and how many families pass every language variant. It also includes a descriptive cluster bootstrap that resamples families. With only eight families, that interval is a diagnostic aid, not a strong population estimate.

For a larger paired model comparison, compute each family’s change in success rate, then resample families and summarize the mean change. Keep both candidates’ results paired within each resampled family. Independent resampling of model A and model B throws away useful task pairing.

An apparently perfect small safety set also has uncertainty. Zero failures among a few examples establishes what happened on those examples, not that the true failure probability is zero. Expand the scenario coverage and report the count before making a deployment claim.

Test the instrument without claiming a model score

The kit includes gold contract records used to test the scorer itself. Running them should produce 24 passing contracts. Mutating the unit, date, source, tool arguments or household scope should produce a failure. The runtime tests additionally check trace forgery, deadlines and step limits.

python -m unittest discover -v
python scorer.py oracle-contracts.jsonl --output /tmp/oracle-report.json

The oracle-contracts command is an instrument self-check. It does not call a model and must never be shown as “100% model accuracy.” The candidate run in Part 11 is the separate operation that produces model predictions.

Make failures useful to engineering

A good report identifies which contract failed, retains the case and links to the trusted trace. It does not need to publish a private conversation or an internal debugging transcript. Public examples can use authored scenarios with the same failure class.

Aggregate failures into numerical extraction, tool choice, permission, evidence, freshness, clarification and runtime categories. Then inspect representative cases before changing training data or prompts. A retrieval failure should not be “fixed” by teaching the model to confidently answer without evidence.

This is the purpose of evaluating the complete system: the score tells you whether the task was completed, and the trace tells you which component to improve. The next article applies that discipline to public financial datasets without mixing incompatible benchmark protocols.