FIELD · Models and data · 4 min read
Fraud decisions with delayed evidence
One transaction family, mature-label evaluation and explicit fallback policy.
Arunank Sharan · Kuzushi Labs ·
In this article
This worked application uses explicit planning assumptions to demonstrate FIELD. Quantities below are inputs, calculated implications or proposed targets; they are not reported customer results. The engagement decision remains conditional on the evidence described.
Planning basis and engagement boundary
| Item | Basis for this application |
|---|---|
| Demand and context | 5,000 transactions/s; proposed 100 ms server deadline; 500 in flight if mean time at this boundary is 100 ms |
| First delivery boundary | One transaction family, mature-label evaluation and explicit fallback policy |
| Acceptance to agree and test | Evaluate loss, false blocks and subgroup impact with delayed-label corrections |
| Exclusion or constraint | Missing labels are not treated as proof that transactions were safe. |
F — Frame the outcome
Reduce fraud losses while controlling false blocks and reviewer workload; define the business action policy with risk operations.
Value measure: Matured loss and legitimate-customer friction per transaction, constrained by review capacity.
I — Investigate the workflow
Map delayed labels, appeal outcomes, attack adaptation and review capacity. Separate immature outcome data from true recent performance.
First evidence task: Stress a feature outage while preserving an actionable review/fallback policy.
E — Engineer the solution
Use point-in-time features, calibrated scoring/rules and immutable decisions; size inference and review queues together.
The implementation assessment below makes the authority, workload, resource and failure assumptions concrete. Its numbers are calculations from the stated scenario, not a vendor performance guarantee.
Contract
Define scoring, allow/block/review actions, loss categories, latency and the label maturation horizon. The decision policy accounts for amount, false blocks and reviewer capacity; a single global accuracy metric is inadequate.
Architecture
Versioned event ingestion feeds point-in-time offline training data and fresh online aggregates. The scoring service retrieves scoped features, runs a calibrated model plus explicit rules and produces a logged decision. Review outcomes and delayed confirmed labels join immutable predictions by transaction/model version. Data quality and model deployment have separate gates.
Sizing
At 5,000 transactions/s and a 100ms hypothetical server deadline, allocate illustrative 40ms to feature access, 10ms model/rules, 20ms other work/network and 30ms reserve; validate the actual distribution. At measured 1ms CPU/inference, scoring alone needs 5 CPU-s/s, before headroom and features. If 1% of all 5,000/s were reviewed continuously, that would be 50 reviews/s, exposing the need to model sustained traffic and staffing rather than casually sending every uncertain case to humans.
Delayed evidence
Recent confirmed fraud rates look artificially low before labels mature. Compare cohorts at equal outcome age and track missing-label mechanisms. Feature distribution drift is an alarm, not proof that accuracy declined; a broken feature source can be diagnosed immediately without waiting weeks.
Failure/attack
When a critical feature fails, choose a pre-evaluated conservative rule/review/refusal policy consistent with business risk. Attack adaptation needs updated slices, adversarial analysis and controlled retraining, not automatic feedback ingestion that attackers can poison.
Validation
Time/group splits, calibration, PR curves and expected cost at operational thresholds; report false blocks and delayed loss by cohort. Shadow/canary releases with protected rollback and investigate online-offline feature parity.
L — Launch with evidence
Compare equally matured cohorts and evaluate expected cost, false blocks and operational thresholds. Shadow first where permitted and preserve a tested fallback.
Critical decision condition: A threshold policy exceeding reviewer capacity or relying on immature success evidence cannot expand.
Attach raw artifacts, the tested configuration, population, denominator, date and limitations to the release record. Missing or stale evidence holds the dependent decision. A favorable value forecast cannot offset a failed critical condition.
D — Develop ownership and reuse
Risk operations owns thresholds and appeals; ML/data owners maintain labels, feature quality and release evidence.
Candidate reusable asset: Delayed-label evaluation and threshold-capacity sheet.
Commercial decision and next commitment
Accountable decision owner: Risk policy owner with scoring service owner.
The initial commitment is the bounded journey above, with the stated exclusions. Confirm access, customer staffing and acceptance ownership during discovery; price and schedule unresolved dependencies explicitly. A new source, tenant class, language, action or operating envelope requires an impact review rather than silent scope growth.
Approve only the next exposure level supported by the evidence. The next useful investment is: stress a feature outage while preserving an actionable review/fallback policy.
Method and references
Apply the FIELD method, economic model and evidence rules. Methodological and technical influences are listed in the source register.