Plan model capacity from real token demand

This worked application uses explicit planning assumptions to demonstrate FIELD. Quantities below are inputs, calculated implications or proposed targets; they are not reported customer results. The engagement decision remains conditional on the evidence described.

Planning basis and engagement boundary

F — Frame the outcome

Meet useful first-response and completion deadlines at a sustainable cost; the product owner decides which task/context classes deserve priority.

Value measure: SLO-compliant successful tasks per cost, separated by context class.

I — Investigate the workflow

Observe input/output length distributions, long-context cohorts and what users consider useful progress. Confirm provider or device limits.

First evidence task: Sweep offered load with a realistic long-input tail and output limits.

E — Engineer the solution

Use separate prefill/decode load, KV memory and measured serving curves; derive fleet/quota from the full workload and failure target.

The implementation assessment below makes the authority, workload, resource and failure assumptions concrete. Its numbers are calculations from the stated scenario, not a vendor performance guarantee.

Contract

Define TTFT, inter-token gap and completion targets by request class, plus quality, burst shape, cancellation and tenant fairness. Record input/output length distributions, especially the 20k-token tail. Averages cannot determine peak KV residency or long-request tails.

Load

50 requests/s×2,000 input tokens=100k input tokens/s; output demand is 15k tokens/s. If mean completion time were 8s, mean in-flight requests would be 400, but this is an illustrative added latency assumption. Each request's growing cache and scheduling state must fit the admitted fleet.

Architecture

Gateway estimates/tokenizes work→per-tenant/class admission→bounded scheduler→model replicas→streaming response. Keep a long-input class or chunked-prefill policy, cancellation cleanup, usage accounting and a durable path only for tasks that need it. Scale on queued work/age and token capacity, with cold-load delay included.

Memory

For an illustrative 8B BF16 model, weights are 16GB≈14.9GiB. A cache with 32 layers, eight KV heads, head dimension 128, two-byte cache elements, 8,192 tokens and 16 sequences adds 16GiB, exceeding 24GiB before runtime. Actual architecture, sequence lengths and precision determine the answer; provider APIs instead expose external quotas that must be modeled.

Fleet arithmetic

Suppose a real mixed-workload test later establishes 10 requests/s per replica at the required quality/SLO, and you choose a 70% operating target: usable capacity is 7/s. Normal demand needs ceil(50/7)=8 replicas; one-replica-loss reserve needs 9. Three equally sized zones need enough surviving capacity: 12 replicas, four/zone, leave eight after one zone fails. These counts are conditional on that hypothetical measured capacity and no other bottleneck.

Validation

Use a validated arrival generator, real length/tenant mix, warm/cold conditions and quota limits. Sweep offered load; report accepted/completed goodput, rejections, TTFT/gaps/completion tails, HBM and cost/success. Compare long-prefill scheduling changes and verify no starvation.

L — Launch with evidence

Load-test representative mixtures and report goodput, tails, quality and cost per successful task. Do not use peak token throughput as readiness evidence.

Critical decision condition: A capacity claim without representative quality and tail-latency evidence cannot justify rollout.

Attach raw artifacts, the tested configuration, population, denominator, date and limitations to the release record. Missing or stale evidence holds the dependent decision. A favorable value forecast cannot offset a failed critical condition.

D — Develop ownership and reuse

Serving operators own admission/scale alerts and the product owner owns context limits and degraded modes.

Candidate reusable asset: Workload manifest and serving capacity report.

Commercial decision and next commitment

Accountable decision owner: Serving service owner with product outcome owner.

The initial commitment is the bounded journey above, with the stated exclusions. Confirm access, customer staffing and acceptance ownership during discovery; price and schedule unresolved dependencies explicitly. A new source, tenant class, language, action or operating envelope requires an impact review rather than silent scope growth.

Approve only the next exposure level supported by the evidence. The next useful investment is: sweep offered load with a realistic long-input tail and output limits.

Method and references

Apply the FIELD method, economic model and evidence rules. Methodological and technical influences are listed in the source register.