WealthOps: an executable financial-agent benchmark

A useful agent benchmark should let another engineer inspect the task, run a candidate and explain why the result passed or failed. WealthOps does that with a deliberately small public set: eight financial workflow families, each expressed in English, Hindi and Hinglish.

The package contains the case definitions, candidate-only inputs, a read-only tool sandbox, a local model adapter, a structured scorer and tests for the evaluation machinery. It makes numerical accuracy, missing-input handling, access scope, evidence freshness and tool failure part of the same executable workflow.

The eight families cover different kinds of work

The numerical values and households are authored fixtures. They expose the contract without disclosing client records. Every family has three language variants, so a run attempts 24 cases. It is a development and integration benchmark, not a broad leaderboard of financial intelligence.

Keep three artifacts separate

candidate-inputs.jsonl contains the request, language, fixture and available tool scope. cases.json contains the expected outcomes and is read by the scorer. oracle-contracts.jsonl contains the known-correct result objects used to test the instrument.

These names describe files in the downloadable public kit. They are not paths into a client application. The benchmark runner never opens the gold case file. It also removes case IDs and family labels from the actual model request, so the candidate is asked to solve the task rather than classify a conveniently named fixture.

The test suite checks that separation. The corpus is public, so it is not a secret held-out benchmark; the architecture still establishes the same candidate/evaluator boundary needed for a private evaluation set.

Install nothing for the offline checks

The scorer, calculators, fixture runtime and tests use the Python standard library. From the extracted kit directory:

python -m unittest discover -v
python scorer.py oracle-contracts.jsonl --output /tmp/oracle-report.json

The oracle report should pass all 24 contracts. That is a verification of the scorer against known records, not a model result. The tests then mutate fields and exercise denied calls, deadlines, missing predictions and arithmetic parsing to ensure that incorrect records fail.

The public FinQA sample check executes eight arithmetic programs and compares them with their stored execution targets. It validates the small interpreter on those selected operations. It is likewise an instrument check rather than an evaluation of a language model.

Connect an actual local candidate

Install a model in a compatible local Ollama runtime, give it a stable local identifier and keep the server on loopback. The benchmark adapter sends JSON-mode chat requests with a fixed decision protocol. It does not assume native function-calling support.

python run_candidate.py \
  --model your-local-candidate \
  --endpoint http://127.0.0.1:11434/api/chat \
  --output /tmp/candidate-predictions.jsonl

python scorer.py /tmp/candidate-predictions.jsonl \
  --output /tmp/candidate-report.json

python summarise_latency.py /tmp/candidate-predictions.jsonl

your-local-candidate is replaced with the identifier of the model you have actually imported. Record the corresponding model digest and template configuration. A mutable friendly name is insufficient for a reproducible comparison.

The adapter uses the documented Ollama chat endpoint, requests one JSON object per decision and caps output at 512 tokens. Temperature is zero for the reference comparison. That reduces one source of sampling variation; it does not establish bitwise deterministic execution across hardware and runtimes.

Walk through a complete candidate attempt

For the future-value request, the candidate proposes future_value with principal 2500000, annual rate 0.06 and years 5. The sandbox validates those arguments, performs Decimal arithmetic and returns a typed result with source calc-fv-v1.

The candidate then emits a final outcome containing a future_value claim and the policy code illustrative_not_guaranteed. The runtime appends the actual tool call and its zero-write count. The scorer compares the collected result with the fixed contract.

{
  "kind": "final",
  "result": {
    "outcome": "answer",
    "claims": {
      "future_value": {
        "value": "3345563.94",
        "unit": "INR",
        "subject": "scenario-A",
        "as_of": "2026-09-01",
        "source": "calc-fv-v1"
      }
    },
    "evidence_ids": ["calc-fv-v1"],
    "policy_code": "illustrative_not_guaranteed"
  }
}

This is the specified correct contract, not a captured candidate prediction. The actual prediction file records what the chosen model returns, including invalid outputs and runtime errors.

The runner owns the execution evidence

A candidate can generate a field called tool_calls, but the runner ignores it. It copies only the allowed final-result fields, then inserts the trusted sandbox trace. This prevents an answer from gaining credit by claiming that a calculator was called when it was not.

The sandbox logs rejected attempts too. If the model requests H-B’s portfolio, the read is denied and the attempted call remains visible. The scorer can then distinguish the model’s unsafe proposal from the runtime’s successful enforcement.

The public adapter limits each case to four decisions and 30 seconds. Network failures, invalid JSON, invalid decisions and exceeded budgets produce failed records rather than disappearing from the file. The output is flushed after each attempt, so a stopped run still has inspectable completed records and an honest missing-case denominator.

Read the report at more than one level

The scorer reports attempted and submitted cases, passed contracts, critical cases, per-language counts and the number of families that pass all languages. It retains per-case failure labels so the aggregate can be explained.

A model can fail the future-value case through a wrong amount, wrong rate, missing calculator call, wrong source or inappropriate policy. Those failures call for different changes. Keep the original prediction and trusted trace with the score.

The language slices measure structured task success on prompts in each language. Because the final protocol is structured JSON, they do not establish the quality of a customer-facing Hindi or Hinglish explanation. Add that as a separate generation track when evaluating the full UI experience.

The exact-trace requirement is appropriate for these compact reference cases. For a larger workflow with multiple valid plans, extend the scorer to check authorized actions, required evidence and final state without insisting on one sequence. Research such as τ-bench is useful when designing richer interactive task environments.

Extend the benchmark without weakening it

Add a family when it introduces a new capability or boundary, not merely another wording of an existing question. Examples include a corrected amount in a follow-up turn, a stale cached snapshot, an explicitly permitted historical query, a zero-rate savings scenario and an ambiguous rate convention.

Each addition needs a candidate request, fixture state, expected outcome, accepted evidence, tool policy and failure probes. Add language variants only after the underlying scenario is well specified. Keep related variants grouped in training and evaluation partitions.

For a private holdout, use independently authored cases and keep gold contracts outside the candidate runtime. The public kit’s cases are useful for development and regression checks; repeated tuning on them turns them into training feedback. That does not reduce their value as integration tests, but it changes what a score means.

Produce a run bundle that another engineer can inspect

A run bundle contains the candidate configuration, input digest, raw predictions, scorer version, structured report, latency summary and environment metadata. It also states whether the run used a model, an oracle self-check or a deterministic test driver. That single field prevents several easy forms of accidental overclaiming.

The downloadable reading and benchmark kit includes everything needed for the offline checks and the local-candidate adapter. The final chapter shows how those artifacts become a clear base-versus-adapted release comparison.