Launch with evidence that matches the promise

A release decision is a claim about a specific configuration, population, workload and operating context. It is not a permanent declaration that an application is safe or correct. FIELD requires evidence to remain attached to the conditions under which it was collected.

Specify the evidence before collecting it

Each record contains the requirement ID, artifact reference, source/origin, collection date, configuration hash or version, population, environment, measurement method, owner and limitations. A cryptographic digest can detect an altered artifact; it cannot prove that the artifact represents a real measurement.

Distinguish assumption, calculated implication, target, controlled test result and observed production outcome. A manually entered latency value is not telemetry. A generated test set can reveal a mechanism but does not automatically represent customer traffic. Unknown evidence stays unknown.

This structure builds on NASA’s requirements verification matrices and SEI’s work on claims, arguments and evidence. FIELD applies those established ideas to commercial workflow commitments and versioned release decisions.

Build an acceptance set that represents the work

For a worked 240-task pilot evaluation, allocate 100 routine tasks, 50 common exceptions, 30 missing/conflicting-evidence tasks, 30 access and action-control tasks, and 30 recovery/cancellation tasks. The counts are proposed coverage allocations. They are not a basis for extrapolating a rare-event risk without a sampling model.

Freeze acceptance inputs before final tuning. Separate development and acceptance sets by entity, source, time or workflow family where near-duplicates could leak across them. Include languages, tenant sizes and user categories that matter commercially. Record disagreement among human reviewers and calibrate automated judges against adjudicated examples.

For stochastic outputs, repeat selected cases and show variability. Report both task correctness and operational cost. A supported explanation that takes too much review time can still be a poor workflow. The evaluation principles here are informed by OpenAI’s evaluation guidance; the actual tasks and release thresholds are owned by the engagement.

Decide using independent critical requirements

An aggregate score cannot offset a failed critical control. Risk acceptance must identify what is being accepted, for how long, by whom and within whose authority. Some obligations cannot be waived by the project sponsor.

Understand statistical limits

Suppose 300 suitably sampled independent attempts show zero failures. The approximate “rule of three” gives a one-sided 95% upper bound near 1% on the failure rate under a simple Bernoulli model, not proof of zero risk. Correlated tasks, selected examples and repeated variants of the same source weaken that interpretation. For rare, severe events, combine explicit control testing, threat analysis, monitoring and containment rather than relying on a small average score.

Define confidence intervals and sample requirements for ordinary outcome comparisons with the actual unit of independence. A hundred tasks from one operator may not provide a hundred independent observations. Clustered workflows and marketplace interference need an appropriate experimental unit.

Release by exposure, not just by date

A rollout ladder can move from offline evaluation to read-only shadowing, assisted pilot, bounded write access and broader adoption. Each step names the population, traffic, action limits, duration, monitoring and rollback trigger. Shadowing must suppress external effects; replaying a write in “shadow mode” can still harm the customer if the integration is live.

For a proposed invoice pilot, start with five operators and at most 50 eligible attempts/day, then expand to 20 operators after the agreed review. A permission leak, duplicate committed correction or unresolved reconciliation backlog stops affected actions immediately. A worsening effort or cost signal triggers review without necessarily taking down safe read-only access.

Make rollback include business state

Application rollback does not undo external effects or restore compatibility with old task records. Specify schema compatibility, in-flight run handling, approval versions, receipts, data restoration and reconciliation. Test the operator’s ability to perform the rollback and explain what remains irreversible.

A model, prompt, parser, embedding index, permission policy, source schema or runtime change may invalidate evidence. Use the requirement-to-evidence links to identify affected claims and rerun the relevant tests. Schedule a periodic review as well as change-triggered reassessment. End the release meeting with a signed decision record: scope, evidence, limitations, operating owner, next review and triggers for reversal.