Benchmark protocols for FinQA, FinanceBench and IndiaFinBench

Public financial benchmarks give us useful common tasks, but they do not all measure the same capability. A numerical program over a supplied table, an answer grounded in a company report and an interpretation of a regulatory passage require different inputs and different scoring.

The evaluation framework keeps those tracks separate. Each track declares its source revision, split, candidate input, permitted tools and scorer. This makes a result reproducible and prevents a convenient aggregate score from hiding a change in the task itself.

Give each dataset a clear role

The first three are external resources. WealthOps is the small authored reference benchmark shipped with this series. Report their results separately. They have different units of analysis and different difficulty distributions.

The FinQA paper, FinanceBench paper and IndiaFinBench paper provide the research context for their respective tasks. This implementation uses them as complementary evaluation tracks, not interchangeable leaderboards.

Pin the data and name the split’s purpose

The public registry records the canonical FinQA revision and the Hugging Face revisions for the other datasets. FinQA’s pinned files contain 6,251 training, 883 development and 1,147 test examples. FinanceBench exposes 150 public examples. IndiaFinBench contains 82 development and 324 test records. These are resource sizes, not results from the local model.

IndiaFinBench can be loaded with an explicit revision:

from datasets import load_dataset

REPO = "Rajveer-code/IndiaFinBench"
REVISION = "716228cefa5d3282c01cc8f282f70d9325065751"

def load_development_cases():
    dataset = load_dataset(REPO, revision=REVISION)
    dev = dataset["dev"]
    return [
        {
            "id": row["id"],
            "candidate": {
                "question": row["question"],
                "context": row["context"],
            },
            "gold": {
                "reference_answer": row["reference_answer"],
                "task_type": row["task_type"],
            },
        }
        for row in dev
    ]

This is a public-data preparation step, performed outside the restricted household environment. The Hugging Face loading documentation describes revision-pinned loading. Save the resulting artifact digest as well as the repository revision in your run manifest.

The gold object is never passed to the candidate. Keeping it in a clearly separate structure makes that boundary easier to inspect than removing a few fields from a large object at the last moment.

Separate three evidence conditions

A supplied-evidence experiment asks whether the model can reason over material already selected for it. A retrieval experiment asks the system to locate that material and then answer. A closed-book experiment asks the model to answer without the external evidence.

These are different experiments even when the question text is identical. Do not label all three “RAG accuracy.” A supplied gold passage removes a major source of retrieval error; a complete report introduces a much larger context-selection problem.

For a retrieval track, retain the retrieved document and passage IDs before generation. Measure whether the necessary evidence was present, then whether the final claims were supported. This separates “could not find the evidence” from “found it but answered incorrectly.”

Construct FinQA inputs without annotations

A full-context numerical task uses the question, table and surrounding text. The gold program and answer stay with the evaluator.

def finqa_candidate(record):
    return {
        "question": record["qa"]["question"],
        "table": record["table"],
        "pre_text": record["pre_text"],
        "post_text": record["post_text"],
    }

def finqa_gold(record):
    return {
        "id": record["id"],
        "program": record["qa"]["program"],
        "execution_target": record["qa"]["exe_ans"],
        "display_answer": record["qa"]["answer"],
    }

Fields containing selected gold sentences or a prepared gold-evidence input belong to a different protocol. Do not accidentally include them by serializing the complete record.

The reference kit’s small arithmetic interpreter supports addition, subtraction, multiplication and division with backward result references. It is sufficient for the eight selected sample annotations. It is not the official complete FinQA executor and should not be used to claim full benchmark coverage. For a full FinQA result, use the benchmark’s intended execution protocol and account for every unsupported or failed case.

Match the scorer to the task

Numerical tasks need a defined representation, rounding rule and execution comparison. Contradiction tasks need normalized labels and an account of invalid outputs. Temporal tasks need the answer to respect the supplied effective dates. Long explanations need claim-level support and completeness checks.

A generic “similarity to reference answer” score cannot establish all of these at once. It can reward shared wording while missing a reversed condition or wrong period. Start with deterministic checks wherever the answer structure permits them, and use reviewed semantic evaluation for the remaining content.

For regulatory passages, evaluate against the supplied versioned context. The benchmark is a language-and-reasoning task; its historical excerpts do not establish current legal advice. The deployed product uses its own approved, date-aware evidence source.

Make the experiment matrix explicit

A compact initial comparison has three candidates and three evidence conditions. Not every cell is relevant to every dataset, but the matrix makes changes visible.

Keep decoding settings, output budget and prompt constant within a comparison. If the adapted model requires a different template, record that difference as part of the candidate. If you also change the retriever, it becomes a system comparison rather than an isolated fine-tuning comparison.

Use development data to establish parsing and scoring. Freeze those choices before the held-out run. A useful improvement can then be attributed to a named change rather than to repeated tuning on the final test answers.

Report eight fields before presenting a score

Every result should state dataset revision, split, attempted count, evidence protocol, model identity, decoding settings, scorer version and failure accounting. Add task-level and language-level slices where applicable.

Keep missing outputs and parser failures in the denominator. Publish both attempted and submitted counts. For numerical tasks, retain the raw prediction alongside the normalized value so a reviewer can inspect the transformation.

The reference registry links to the FinQA Hugging Face mirror for convenient discovery while using the canonical repository for its pinned sample data. The exact source used in a run is part of its identity.

Respect the resource’s intended use

Dataset availability and permitted use are separate fields. The source registry records the licences and intended roles. In particular, the FinanceBench public card’s non-commercial condition means it is not silently mixed into a commercial fine-tuning corpus. This reference package links to that resource rather than redistributing its records.

Keep public benchmark examples separate from private operational evaluation. A public benchmark is valuable for comparison, but it may already be familiar to a widely trained model. Private holdouts test the organisation’s actual request patterns, access rules and evidence sources. The next chapter supplies a small public workflow benchmark that makes those mechanics inspectable without publishing household data.