PART 12 OF 14 · 7 MIN READ · ARUNANK SHARAN · 26 OCTOBER 2025

Evaluate the evidence chain, not just the final answer

A RAG answer can fail even when every service returns HTTP 200. The parser may have attached the wrong header to a number. Retrieval may have selected the wrong year. The answer may cite a relevant document but an irrelevant paragraph. The viewer may highlight the wrong region.

For the client, the parser migration made evaluation especially consequential: a 60% cost reduction was valuable only if the resulting quality remained acceptable for the initial release. The broader framework developed here makes that decision inspectable at every layer of the evidence chain.

Evaluate the chain in stages

An end-to-end score tells you whether the answer worked. It rarely explains what to fix next.

The layers share test cases but need different labels. A parser test needs source fields and regions. A retrieval test needs relevant passages. A synthesis test needs acceptable claims and calculation results.

Build a task set from analyst work

Start with representative tasks rather than only generated questions about arbitrary chunks. Include exact lookups, comparisons, explanations, multi-document synthesis, missing information and contradictions.

A proposed first evaluation suite could contain 120 tasks, allocated as follows:

These are starting allocations, not a historical client benchmark result. Increase coverage where observed failures concentrate. Keep a private customer task set separate from any public demonstration set.

For each case, store the question, corpus snapshot, issuer/time constraints, required source spans, expected calculations, acceptable answer variants and explicit failure conditions.

Public benchmarks are useful, but different

FinanceBench provides financial question-answering tasks with evidence, including a public evaluation sample. TAT-QA focuses on reasoning across tables and text. They help construct public tests, but do not replace the client’s task distribution. FinanceBench, TAT-QA

Check dataset licences, document availability and split definitions before packaging a public benchmark. Do not present a public benchmark’s scores as results from the client system. For model comparisons, consider whether the model or its training data may already include the benchmark.

The strongest public portfolio artifact is a reproducible evaluation specification, small permitted dataset and honest result record. A blank result field is preferable to a fabricated improvement.

Keep tuning and acceptance separate

Repeatedly changing prompts against the same twenty questions can produce an impressive demonstration and a weak system. Split tasks into development, validation and held-out acceptance sets. Group by issuer or source document where appropriate so near-identical passages do not leak across splits.

For temporal tasks, include documents published after the requested as-of date as deliberate distractors. The correct behaviour is to exclude them, even if they contain an easier answer.

When comparing parser configurations, hold the rest of the pipeline fixed. When comparing retrieval, hold parser and chunker outputs fixed. An end-to-end improvement with five simultaneous changes is difficult to attribute and risky to reproduce.

Use oracle experiments to locate the bottleneck

An oracle experiment replaces one stage with known-correct input:

• Give retrieval the gold parsed text. If quality improves, parsing is limiting the task.
• Give synthesis the gold evidence passages. If the answer still fails, retrieval is not the main problem.
• Give the viewer a known-correct source span. If navigation fails, the citation geometry or UI is responsible.
• Give the agent a complete evidence ledger. If it continues searching, the completion policy is wrong.

These experiments prevent endless prompt tuning when the real issue is a missing table header or a malformed service contract.

Separate recall from complete support

For a task requiring three evidence units, finding two gives recall of two-thirds. It may still leave the task impossible to answer correctly.

def evidence_metrics(required: set[str], retrieved: list[str], k: int):
    if not required or k <= 0:
        raise ValueError("Need required evidence and a positive cutoff")
    selected = set(retrieved[:k])
    found = len(required & selected)
    return {
        "recall_at_k": found / len(required),
        "complete_at_k": required <= selected,
    }

assert evidence_metrics({"A", "B", "C"}, ["A", "X", "B"], 3) == {
    "recall_at_k": 2/3, "complete_at_k": False
}

Also measure incorrect-scope retrieval. Finding the correct metric in the wrong issuer’s report is not partially correct financial evidence.

For ranking, nDCG can incorporate graded relevance, while MRR focuses on the first relevant result. Neither alone captures whether all evidence needed for a multi-step calculation is available.

Judge claims with a rubric

An LLM judge can help classify failures, but it should be calibrated against human review. The rubric should ask concrete questions:

Does the cited passage support the claim as written?
Are issuer, period, units and reporting basis consistent?
Is a management statement attributed to the correct speaker?
Is a derived value backed by a valid calculation?
Does the answer express a stronger conclusion than the evidence supports?

Include the source passage and claim, not just the final answer. Require a category and short evidence-based explanation. Blind the judge to system names during paired comparisons and review disagreements manually.

A judge’s fluent explanation can also be wrong. Maintain a calibration set containing subtle unit mismatches, unsupported causality and nearly correct citations. Track agreement by failure category rather than only overall agreement.

Citation evaluation needs its own labels

Citation precision asks whether attached citations support the claims they accompany. Citation completeness asks whether claims needing support actually have it. Location accuracy asks whether the highlighted region corresponds to the intended evidence.

For multi-region citations, compare coverage of the relevant source tokens or cells. Rectangle overlap can be useful, but a large rectangle covering half a page should not receive full credit simply because it contains the correct sentence.

Include UI checks at several zoom levels, rotated pages, repeated text, split tables and mobile widths. These are deterministic tests that complement the semantic evaluation.

Release gates should reflect the failure cost

Set explicit gates with the customer. Wrong issuer, wrong period, unauthorised evidence and fabricated citations deserve stronger treatment than awkward wording. A proposed gate can require zero such failures in the acceptance set while still reporting the sample size and uncertainty.

Zero observed failures does not establish zero production risk. As a rough statistical reminder, with zero failures in 100 independent trials, the one-sided 95% upper bound is about 3%, using 1 - 0.05**(1/100). Correlated questions provide less information than independent ones.

For quality changes, report paired task outcomes: improved, unchanged and regressed. Bootstrap uncertainty at a sensible unit such as task or issuer, rather than treating every claim from one document as independent.

Connect offline evaluation to operations

After release, sample real failures and add them to the development set. Keep the held-out acceptance set protected. Record parser, chunker, embedding, retriever, reranker, prompt and corpus versions with each trace.

Monitor answer completion, citation resolution, freshness, latency and cost. A provider update can change output structure or token usage even when application code is unchanged. Replay a fixed evaluation suite before promoting a new configuration.

The result is a practical improvement loop: observe a failure, locate the responsible stage, change one controlled part, run the relevant regression tests and review the business trade-off. That is how a cost reduction remains useful after the first successful release.