Generate financial QA with executable answer checks

Once the source material is structured, the next task is to turn it into examples of useful assistant behaviour. The goal is not to produce the largest possible pile of questions. It is to teach the model how to complete the specific tasks the product exposes: extract a scenario, call the right tool, ask for a missing input and explain a supported result.

A teacher model can help write natural questions and explanations. It does not get to certify its own numbers. The generator supplies a question; a deterministic component establishes the expected calculation; a validator checks that the final example is grounded in its evidence.

Plan coverage before generating language

For each source family, create a coverage plan with the task type, required inputs, expected outcome and evidence scope. A future-value topic should not produce only “What will ₹10 lakh become?” It should include missing horizons, alternative compounding conventions, ambiguous rates and requests that confuse an assumed return with a guaranteed one.

These dimensions become dataset labels, evaluation slices and eventually product metrics. That is more useful than a topic label such as “wealth management” attached to every row.

Calculate first, then ask the teacher to explain

Consider ₹10 lakh over five years at an assumed annual 6% return with annual compounding. The arithmetic is 1,000,000 × 1.06^5, yielding ₹1,338,225.58 after rounding to paise. The teacher receives the verified inputs, formula, result and allowed wording. It can vary the phrasing without changing the numbers.

from decimal import Decimal, ROUND_HALF_UP

def projection(principal, annual_rate, years):
    if isinstance(years, bool) or not isinstance(years, int):
        raise ValueError("whole-year horizon required")
    if years < 0 or years > 100:
        raise ValueError("horizon outside this calculator's range")
    p, r = Decimal(principal), Decimal(annual_rate)
    if not p.is_finite() or not r.is_finite() or p < 0 or r <= -1:
        raise ValueError("invalid scenario inputs")
    result = p * (1 + r) ** years
    return result.quantize(Decimal("0.01"), rounding=ROUND_HALF_UP)

assert projection("1000000", "0.06", 5) == Decimal("1338225.58")

The synthetic example remains a mathematical scenario. It does not become an investment recommendation. The output states the assumption directly rather than suggesting that the amount is promised.

For a monthly savings question, write a different calculator contract. With end-of-month deposits, target future value F, monthly rate r and n deposits, the payment is F × r / ((1+r)^n − 1). When r is zero, it becomes F/n. If the product starts from an effective annual return, convert it with (1+annual_rate)^(1/12)−1; dividing by 12 defines a different nominal-rate convention. The dataset must say which it uses.

Eight inspectable training-format examples

The reference kit includes the following authored examples as JSONL. They show the schema and language style; they are not a sufficient corpus for a useful fine-tune. Their source identifiers point to the authored scenario family rather than pretending to cite a regulatory document.

A complete row looks like this:

{
  "id": "fv-hinglish",
  "family_id": "fv-demo",
  "language": "hinglish",
  "prompt": "10 lakh ko 5 saal ke liye 6% annual compounding pe project karo.",
  "completion": "Given assumptions par future value INR 1338225.58 hai; guaranteed return nahi.",
  "evidence_ids": ["synthetic-reference:fv-demo"]
}

The three future-value variants share a family. They remain together during splitting. Adding translations increases language coverage; it does not create three independent financial problems.

Supervise explanations that can be checked

An explanation is useful when it connects the answer to evidence: the principal, rate, horizon, compounding rule and calculator result. Long speculative reasoning adds tokens without necessarily adding supervision value.

The Distilling Step-by-Step paper studies learning from teacher-generated rationales alongside labels. The application here is narrower: use concise teaching explanations whose inputs and conclusions can be checked. A teacher’s narrative is an annotation, not proof that its calculation is correct.

For source-based questions, require each factual statement to map to an evidence identifier. A citation to a whole report is weaker than a citation to the specific passage or table. For calculations, retain both the supporting inputs and the calculator version. For refusals and clarifications, retain the policy condition that makes that response correct.

A useful target can be as short as: “Equity is ₹30 lakh out of ₹50 lakh of eligible assets, so its share is 60%. This uses the supplied snapshot.” It teaches the computation, denominator and scope in three sentences.

Validate at several levels

A schema check rejects missing fields. A numerical check recomputes the result. An evidence check verifies that referenced source IDs exist and belong to the permitted family. A semantic review checks whether the question is answerable from those inputs. These checks address different failure classes.

REQUIRED = {
    "id", "family_id", "language", "prompt", "completion", "evidence_ids"
}

def validate_training_row(row, known_evidence):
    if not REQUIRED <= row.keys():
        raise ValueError("required field missing")
    if row["language"] not in {"en", "hi", "hinglish"}:
        raise ValueError("unsupported language")
    if not row["prompt"].strip() or not row["completion"].strip():
        raise ValueError("empty text")
    evidence = set(row["evidence_ids"])
    if not evidence or not evidence <= known_evidence:
        raise ValueError("unresolved evidence")
    return row

Do not use a second model as the only acceptance check. It can help identify an unnatural Hindi sentence or an unsupported inference, but it shares the general failure mode of producing plausible text. Deterministic invariants should remain deterministic.

Keep rejection counts by reason: unsupported fact, invalid unit, missing evidence, duplicate question, unresolved ambiguity or language-quality issue. These are pipeline measurements. They help improve generation without turning the evaluation set into a prompt-development workspace.

Use negative examples with precise outcomes

“Refuse when uncertain” is too vague for supervised data. The model must learn when to ask, deny, escalate or report unavailability. A missing horizon calls for a question. A denied household calls for an access boundary. A tool timeout calls for an operational response. None requires a fabricated financial explanation.

Balance these outcomes against answerable work. A model that denies every request can look safe under a narrow metric while failing the product’s purpose. Track both accepted-task success and unnecessary abstention. Keep the same task families across languages so that one language is not rewarded mainly for refusing.

The final dataset has two complementary forms. Short prompt/completion examples teach language and bounded explanations. Multi-step trajectories teach tool selection, observations and final contracts. Part 6 develops those trajectories; Part 4 shows how the model is adapted without training on the prompt text itself.