Improve tool use with a small, inspected dataset

This worked application uses explicit planning assumptions to demonstrate FIELD. Quantities below are inputs, calculated implications or proposed targets; they are not reported customer results. The engagement decision remains conditional on the evidence described.

Planning basis and engagement boundary

F — Frame the outcome

Improve semantic tool execution with 500 reviewed trajectories without mistaking synthetic volume for independent evidence.

Value measure: Correct authorized task completion and critical argument-error rate per labeling/training cost.

I — Investigate the workflow

Decompose wrong selection, invalid syntax, incorrect arguments and missing clarification. Test tool contract improvements before deciding training is the bottleneck.

First evidence task: Measure residual argument errors after a deterministic schema/validation repair.

E — Engineer the solution

Keep protected grouped splits, versioned schemas, bounded augmentation and independent executor policy; retain token/training resource estimates.

The implementation assessment below makes the authority, workload, resource and failure assumptions concrete. Its numbers are calculations from the stated scenario, not a vendor performance guarantee.

Diagnose first

Split the 20% wrong-tool behavior from invalid arguments among correct tool choices. Inspect descriptions, schemas, ambiguity and context. Try clearer bounded tools, canonical argument construction and deterministic validation before attributing every failure to model capability.

Dataset

Keep the 500 expert trajectories grouped by underlying task/source; reserve a protected evaluation portion before augmentation. Include successful calls, no-tool requests, clarifications, permission denial, unavailable tools, invalid observations and recovery. Synthetic variants come from training groups only, with expert checks on semantics and source grounding. Preserve tool/schema versions and outcome labels.

Training design

Compare base+improved tool contract with one controlled SFT/adapter intervention. Inspect serialized messages, tool results and supervised tokens. Do not teach the model to fabricate tool outputs as if it had executed them. Vary one major intervention at a time; a clearer schema and new training data changed together make attribution difficult.

Sizing

If 500 trajectories average 2,000 tokens, one pass is 1M tokens. Expanding to 5,000 similar trajectories gives 10M tokens/pass, but not ten times as much independent evidence. Padding/truncation, action frequency and supervised-token count determine useful learning and memory. Expert validation and evaluation calls belong in the budget.

Schema change

Keep adapters/tool versions compatible through deployment. Add contract tests for renamed fields, new enum values and changed semantics. A model trained on an old schema must not silently execute a near-matching new action. Prefer clarification/validation failure over guessed repair of critical parameters.

Evaluation

Report tool selection, argument semantics, schema validity, authorization and verified task success separately. Test unseen task families and, only if claiming transfer, unseen tools. Compare critical-error rate and general capabilities as well as average success; run repeated trials where sampling matters.

L — Launch with evidence

Compare base plus improved schema with one controlled adaptation, using unseen task families and actual operation outcomes.

Critical decision condition: Schema validity cannot substitute for correct resource/argument semantics.

Attach raw artifacts, the tested configuration, population, denominator, date and limitations to the release record. Missing or stale evidence holds the dependent decision. A favorable value forecast cannot offset a failed critical condition.

D — Develop ownership and reuse

The tool owner governs schema migration; the model owner maintains held-out evaluations and expert labeling quality.

Candidate reusable asset: Tool-trajectory specification and semantic outcome evaluator.

Commercial decision and next commitment

Accountable decision owner: Model lead with tool/domain owner.

The initial commitment is the bounded journey above, with the stated exclusions. Confirm access, customer staffing and acceptance ownership during discovery; price and schedule unresolved dependencies explicitly. A new source, tenant class, language, action or operating envelope requires an impact review rather than silent scope growth.

Approve only the next exposure level supported by the evidence. The next useful investment is: measure residual argument errors after a deterministic schema/validation repair.

Method and references

Apply the FIELD method, economic model and evidence rules. Methodological and technical influences are listed in the source register.