From benchmark runs to a reproducible release

The final output of this engineering process is a deployable artifact with an evidence package. The package identifies what was trained, what was evaluated, how the runtime was configured and which tasks the system completed. Another engineer should be able to reproduce the comparison without access to an internal discussion thread.

I organize that evidence around the actual release candidates: the local base model, the selected adapter and the exported serving artifact. Each candidate runs against the same frozen task set and execution contract. The release report then connects task quality, resource use and operational behaviour.

Define the comparison before collecting results

Start with a base model using the same tools and evidence pipeline as the adapted model. Otherwise a better tool catalogue or a different retriever can be mistaken for a fine-tuning gain. Add the serving export as its own candidate because conversion and runtime differences can change behaviour.

The comparison should answer specific questions: Are more calculations correct? Are units and dates preserved? Does Hindi task success improve? Are there fewer unnecessary clarifications? Is the model still within the serving budget? These are separate results, even if a dashboard later summarizes them.

The LoRA and QLoRA methods described in the LoRA paper and the QLoRA paper provide the adaptation mechanism. The release decision comes from the workflow evaluation, not from adopting the method’s name.

Make the run manifest a real artifact

Record immutable identifiers, not only friendly names. A model tag can move; a dataset can be updated; a template can change while retaining the same filename. Digests make the exact evaluated combination explicit.

{
  "run_id": "local-financial-comparison-001",
  "run_kind": "candidate_evaluation",
  "candidate": {
    "base_model": "google/gemma-4-E4B-it",
    "base_revision": "ee0ef6023621cff504d758262d4e04895a5af4a2",
    "adapter_sha256": null,
    "serving_artifact_sha256": null,
    "template_sha256": null
  },
  "protocol": {
    "suite": "WealthOps-public-v2",
    "cases": 24,
    "families": 8,
    "max_decisions": 4,
    "deadline_seconds": 30,
    "temperature": 0,
    "max_output_tokens": 512
  },
  "measurements": {
    "task_success": null,
    "critical_cases": null,
    "p95_task_latency_ms": null,
    "peak_device_memory_bytes": null
  }
}

This is a report schema. Null fields are filled from an actual run; they are not zeros and not estimates. Keep unmeasured fields explicit rather than publishing a plausible-looking comparison table.

The model revision identifies the public reference base. An adapter, export and template each need their own digest after they are produced. The training manifest additionally records target modules, trainable parameter count, package versions, source partitions and token-length settings.

Summarize improvement with paired cases

Compare candidates on the same case IDs. For each family, calculate the change in mean success across its language variants. Report the distribution of improvements and regressions instead of only the global average.

from collections import defaultdict

def family_deltas(base_records, adapted_records):
    base = {r["id"]: r for r in base_records}
    adapted = {r["id"]: r for r in adapted_records}
    if set(base) != set(adapted):
        raise ValueError("candidate reports must cover the same cases")
    groups = defaultdict(list)
    for case_id, before in base.items():
        after = adapted[case_id]
        if before["family_id"] != after["family_id"]:
            raise ValueError("family assignment changed")
        groups[before["family_id"]].append(
            int(after["pass"]) - int(before["pass"])
        )
    return {family: sum(values) / len(values) for family, values in groups.items()}

On a sufficiently broad benchmark, resample families to estimate uncertainty in the paired change. On the small public kit, use the deltas as a readable diagnostic. Eight families cannot establish universal financial competence.

For safety-relevant outcomes, retain the raw counts. One unauthorised attempt should not be obscured by a small increase in arithmetic accuracy. For usability, retain unnecessary refusals and clarification turns. A model can become more conservative while becoming less useful.

Evaluate the full serving path

The adapter checkpoint is not necessarily the artifact users will call. Quantization, merging, conversion, tokenizer packaging, prompt templates and stop settings all form part of serving. Evaluate the final imported bundle through the same gateway and tool adapter that the deployment uses.

Keep deterministic smoke tests for arithmetic and access controls. Add multilingual behavioural tests for tool arguments and outcome selection. Run a load test separately to establish capacity; functional benchmark latency is useful but does not establish concurrent throughput.

A release that passes quality checks but exceeds the memory budget is not ready for that hardware. A fast release that changes numerical units is not an acceptable substitute. The report needs both dimensions, with the workload and sample count stated.

Use gates that map to actual product risks

A release gate is a predeclared condition, not a score chosen after inspecting the winner. The following structure keeps the decision understandable without inventing universal thresholds.

The product owner sets the quality and capacity targets before the final comparison. The engineering report supplies the measurements and exceptions. This keeps an operational decision tied to the work the assistant is expected to complete.

Roll out with reviewable outputs

Begin with shadow execution on approved test or replay traffic. Compare the candidate’s result with the current workflow without exposing unreviewed outputs to users. Capture the case category and outcome, with private evidence retained only in the approved environment.

For an assisted rollout, keep the result’s source and assumptions visible. Let the adviser correct a value or reject an explanation without reconstructing the whole request. Corrections become reviewed evaluation candidates, not automatic training rows. A user correction can itself be mistaken or contain information that is unsuitable for training.

Track accepted answers, corrections, escalations and time to a reviewable result. These connect model engineering to the actual work saved. A claim about reduced adviser effort requires a measured baseline and a defined comparison period; model benchmark accuracy alone cannot establish it.

Keep a compact research shelf

The following primary references support distinct engineering choices. Each has a role in the series; none is treated as evidence that this particular deployment achieved the paper’s results.

Model and dataset cards remain necessary alongside papers. They identify the available artifacts, revisions and usage conditions. The reference kit’s source lock file records the versions actually used for its public samples and configuration.

The deliverable is inspectable end to end

The series now has a connected technical path: source documents become evidence-linked examples; examples become a masked adaptation run; the model operates through a bounded runtime; trusted traces become typed scores; and scores, latency and artifact identities become a release report.

The download includes the articles, diagrams, public sample annotations, training recipe, local candidate adapter and offline test suite. It does not include model weights or a prefilled model-quality result. Running the candidate and training stages produces those artifacts in the chosen environment.

That is the standard I want from a private financial AI system: a clear explanation of what the assistant does, a concrete implementation of its boundaries and enough reproducible evidence to judge the deployed result.