Fine-tuning a local financial assistant with QLoRA

Fine-tuning has a specific job in this system. It teaches the assistant to interpret financial requests consistently, preserve units, choose a bounded tool and explain the returned evidence. It does not replace the portfolio database or turn a language model into a calculator.

The reproducible recipe here uses Gemma 4 E4B Instruct, a quantized frozen base and trainable low-rank adapters on the language stack. The download includes a single-GPU training entry point, configuration, example row format and token-mask checks. The CPU reference tests exercise data and runtime contracts; running the training recipe itself requires a compatible GPU environment and a reviewed corpus.

Fix the experimental unit before training

A run is the combination of a base snapshot, tokenizer, chat template, training and validation partitions, adaptation settings and software environment. Give that combination an identifier before launching the optimizer. Otherwise “the fine-tuned model” becomes an ambiguous collection of weights and prompt changes.

The public recipe pins google/gemma-4-E4B-it on Hugging Face to revision ee0ef6023621cff504d758262d4e04895a5af4a2. It uses the text path of the multimodal model and adapts language projections only. The official Transformers Gemma 4 documentation provides the model and processor classes used to load that architecture.

Prepare three partitions before any tuning. Training updates the adapters. Validation selects a checkpoint and development settings. The held-out test set is opened for the final comparison. Related documents and translated variants share a source family and therefore a partition.

The eight format examples in the kit are there to explain the interface. Supply a larger reviewed dataset through --train and --validation; the training entry point rejects overlap in their family identifiers. A good partition is about independence and coverage, not just row count.

The reference recipe, field by field

These values define an inspectable starting recipe, not a universal optimum. Changing the rank, sequence length and example mix changes memory use and behaviour. Change one controlled dimension at a time and keep the baseline run available.

LoRA represents a weight update with low-rank matrices. QLoRA combines low-rank adaptation with a quantized base and memory-saving training techniques. In this build, the base remains frozen while the adapters learn the workflow-specific behaviour.

Quantize the base and select targets explicitly

The training entry point uses AutoModelForImageTextToText, AutoProcessor and PEFT. It loads an already imported local snapshot. The environment has no reason to download weights in the middle of a restricted training run.

import torch
from transformers import AutoModelForImageTextToText, BitsAndBytesConfig
from peft import LoraConfig, get_peft_model, prepare_model_for_kbit_training

def make_adapter(snapshot):
    quant = BitsAndBytesConfig(
        load_in_4bit=True,
        bnb_4bit_quant_type="nf4",
        bnb_4bit_use_double_quant=True,
        bnb_4bit_compute_dtype=torch.bfloat16,
    )
    model = AutoModelForImageTextToText.from_pretrained(
        snapshot,
        local_files_only=True,
        quantization_config=quant,
        device_map={"": 0},
        dtype=torch.bfloat16,
        attn_implementation="sdpa",
    )
    model.config.use_cache = False
    model = prepare_model_for_kbit_training(
        model, use_gradient_checkpointing=True
    )
    leaves = {
        "q_proj", "k_proj", "v_proj", "o_proj",
        "gate_proj", "up_proj", "down_proj",
    }
    targets = [
        name for name, _ in model.named_modules()
        if "language_model" in name
        and name.rsplit(".", 1)[-1] in leaves
    ]
    if not targets:
        raise ValueError("no language projection targets matched")
    return get_peft_model(model, LoraConfig(
        r=32, lora_alpha=32, lora_dropout=0.05,
        target_modules=targets, bias="none", task_type="CAUSAL_LM",
    ))

The explicit module list prevents a generic suffix match from accidentally adapting similarly named vision or audio projections. Save the resolved names in the run manifest. A library or model revision can change module structure; failing on an empty match is preferable to quietly running a different experiment.

The PEFT quantization guide documents the preparation step for training adapters on a quantized base. The reference script also records installed package versions at run time, so an environment that succeeds can be recreated.

Know what the adapter size means

For a linear map with input width d-in and output width d-out, rank r adds r × (d-in + d-out) trainable parameters. A 4,096-by-4,096 projection at rank 32 adds 262,144 parameters. That is 0.5 MiB for those adapter weights in BF16, before gradients and optimizer state.

That worked example is deliberately one matrix. The complete adapter count depends on the actual projection shapes and target list. Read the count from the instantiated model instead of multiplying a guessed layer count by a guessed hidden size.

def parameter_counts(model):
    return {
        "trainable": sum(p.numel() for p in model.parameters() if p.requires_grad),
        "frozen": sum(p.numel() for p in model.parameters() if not p.requires_grad),
    }

Activation memory grows with sequence length and microbatch size. Gradient accumulation increases the effective batch without keeping eight microbatches of activations resident at once. It does not remove the memory required for a single long example. Gradient checkpointing trades extra computation for reduced stored activations.

Train only on the intended answer tokens

The prompt contains the question and evidence; the completion contains the response to learn. Prompt tokens receive label -100, which the loss ignores. Completion tokens remain supervised. Padding also receives -100.

The supplied recipe constructs those labels explicitly rather than assuming that a template supports an automatic assistant mask. Part 5 walks through the exact prefix and mask checks. A malformed mask changes the training objective even when the optimizer runs without errors.

Examples longer than the configured context are rejected for evidence shortening or segmentation. Silent right truncation can remove the answer while leaving a perfectly shaped input tensor. Record the rejected count and length distribution as part of corpus preparation.

Select a checkpoint, then evaluate a deployment artifact

During training, retain validation loss, supervised token count, gradient norm, memory peaks and step timing. Save and evaluate at the same cadence so the “best checkpoint” points to a persisted artifact. The recipe retains two checkpoints and reloads the one with the lowest validation loss before exporting the selected adapter.

Loss is a useful optimization signal, not the release criterion. A lower loss can coexist with wrong household access, bad numerical units or excessive refusals. After selection, run the same behavioural suite on the base model and the adapted model. If the adapter is merged or converted for serving, run it again on that exact export.

python training/train_qlora.py \
  --snapshot /models/gemma-4-e4b-pinned \
  --train /data/approved/train.jsonl \
  --validation /data/approved/validation.jsonl \
  --output /runs/financial-adapter-r32

Store the base revision, adapter digest, tokenizer digest, template configuration, dataset manifests and dependency versions beside the result. A successful release comparison can then answer a precise question: with the same inputs and runtime, did this artifact complete more tasks correctly?

Compare the dimensions that users notice

The main evaluation slices are correct calculations, canonical units, tool arguments, evidence grounding, missing-input clarification and English/Hindi/Hinglish task parity. Track general instruction following as well, so narrow adaptation does not damage ordinary interaction.

For each slice, retain both successes and failure classes. A model that improves number extraction but doubles unnecessary clarifications has changed the product experience. That trade-off belongs in the report alongside latency and serving memory.

The deliverable from fine-tuning is a versioned adapter plus an evaluation record. The following chapters make that record executable: templates establish the training contract, the runtime establishes execution evidence and the benchmark establishes the comparison protocol.