Audit the financial model before another training run

This worked application uses explicit planning assumptions to demonstrate FIELD. Quantities below are inputs, calculated implications or proposed targets; they are not reported customer results. The engagement decision remains conditional on the evidence described.

Planning basis and engagement boundary

F — Frame the outcome

Decide whether retraining is warranted and whether the finance SLM can support a bounded task with credible evidence.

Value measure: Verified task improvement per training/evaluation cost, with provenance-qualified scores.

I — Investigate the workflow

Inspect the exact checkpoint/template/labels and training/benchmark provenance. Separate formatting, reasoning behavior, tool execution and source overlap hypotheses.

First evidence task: Inspect serialized examples and run a tiny controlled baseline before another training job.

E — Engineer the solution

Use canonical serialization checks, controlled adaptation and contamination audit before spending GPU budget; benchmark model, retrieval and agent behavior separately.

The implementation assessment below makes the authority, workload, resource and failure assumptions concrete. Its numbers are calculations from the stated scenario, not a vendor performance guarantee.

Freeze and inspect

Record checkpoint/tokenizer/processor, trainer and template revisions, special-token IDs, dataset format, generation settings and stop rules. Render and tokenize a few actual training examples, inspect which positions receive labels, and compare with inference serialization. Repeated markers could reflect malformed examples, double templating, wrong masks/stops or learned repetition; the symptom alone does not prove one cause.

Template correction

Use the exact checkpoint's supported serialization. Current Gemma guidance documents thinking activation through its processor/template configuration, but that inference switch is not a training-supervision recipe. Do not assume manually inserting literal markers into assistant text creates the intended control structure. Also do not claim literal tags are universally invalid across all model families. Gemma thinking reference.

Data/evaluation architecture

Group documents with amendments, near duplicates and generated derivatives, then split before generation. Build distinct tests for closed-book domain knowledge, retrieved-evidence QA, numerical reasoning, tool choice/arguments and full agent outcomes. SEBI overlap is a provenance warning requiring document-level investigation; RBI exclusion from fine-tuning says nothing conclusive about base pretraining. An external benchmark remains complementary to workflow-specific evaluation.

Experiment sequence

First compare the base checkpoint under correct inference formatting with the current fine-tuned model. Verify the corrected training path by overfitting a tiny clean batch. Then compare a corrected-format SFT run against a no-SFT prompting/RAG baseline on untouched tasks. Add tool trajectories only if tool behavior is an intended objective and baseline failures justify them. Keep data amount, decoding and evaluation conditions controlled.

Resources

Estimate actual trainable state, activations, sequence-length distribution and optimizer before reserving hardware. As a rough example, full 8B training at 16 persistent bytes/parameter is 128GB before activations, whereas adapter training has a different accounting. A small trainable adapter does not remove base weights or activation memory.

Decision

Deploy only if domain/task gains survive general-instruction, format and tool regressions within predeclared limits. Reused development examples cannot support an independent quality claim. Retain a run manifest, overlap report and failure taxonomy.

L — Launch with evidence

Require protected evaluation and explicit overlap/unknown-pretraining limits. Compare a repaired baseline before attributing gains to training.

Critical decision condition: Invalid template supervision or contaminated evaluation cannot justify deployment claims.

Attach raw artifacts, the tested configuration, population, denominator, date and limitations to the release record. Missing or stale evidence holds the dependent decision. A favorable value forecast cannot offset a failed critical condition.

D — Develop ownership and reuse

The model/data owner maintains versioned lineage and experiment records; the harness owner maintains execution checks independently.

Candidate reusable asset: Template/token audit and provenance manifest.

Commercial decision and next commitment

Accountable decision owner: Model lead with domain evaluator/data owner.

The initial commitment is the bounded journey above, with the stated exclusions. Confirm access, customer staffing and acceptance ownership during discovery; price and schedule unresolved dependencies explicitly. A new source, tenant class, language, action or operating envelope requires an impact review rather than silent scope growth.

Approve only the next exposure level supported by the evidence. The next useful investment is: inspect serialized examples and run a tiny controlled baseline before another training job.

Method and references

Apply the FIELD method, economic model and evidence rules. Methodological and technical influences are listed in the source register.