Fit the workload, not just the model weights

This worked application uses explicit planning assumptions to demonstrate FIELD. Quantities below are inputs, calculated implications or proposed targets; they are not reported customer results. The engagement decision remains conditional on the evidence described.

Planning basis and engagement boundary

F — Frame the outcome

Choose a model/context/concurrency offer that fits actual memory and meets task quality at an acceptable cost.

Value measure: Successful task cost and admitted concurrency at the promised context/quality bar.

I — Investigate the workflow

Inspect usable device bytes, architecture/KV dimensions, runtime overhead and real context demand; ask whether long contexts are necessary for the workflow.

First evidence task: Measure peak allocation and quality at the proposed maximum context/concurrency.

E — Engineer the solution

Use weight-plus-KV-plus-workspace arithmetic and compare quantization, retrieval, smaller models or lower admission under measured quality.

The implementation assessment below makes the authority, workload, resource and failure assumptions concrete. Its numbers are calculations from the stated scenario, not a vendor performance guarantee.

Inspect

Read actual usable device memory, model architecture, weight dtype, KV heads/layers/head dimension, cache dtype, runtime workspace and output limits. A nominal 24GB-class label may not mean 24GiB available; use bytes from the runtime/device.

Worked memory

Eight billion two-byte weights are 16GB≈14.90GiB. With 32 layers, 8 KV heads, dimension 128 and two-byte KV elements, each cached token/sequence is 128KiB. At 8,192 tokens, one sequence uses 1GiB; 16 sequences use 16GiB. Weights plus cache are 30.90GiB, so this configuration cannot fit a 24GiB device even before workspace.

Options

Four-bit raw weights are approximately 4GB≈3.73GiB plus quantization metadata/unquantized tensors. With the same 16GiB cache and an illustrative 3GiB workspace/reserve, total already reaches 22.73GiB plus metadata, which may exceed a nominal 24GB device's usable memory. Reducing to four 8k sequences with BF16 weights gives roughly 14.90+4+3=21.90GiB under the illustrative reserve, still requiring actual measurement and output-growth limits.

Decision

Compare smaller model, supported quantization, shorter context/retrieval, fewer admitted sequences or a larger/sharded deployment. Quantization must pass task-slice checks and have efficient kernels on the actual hardware. A successful initial load proves little about concurrent peak memory or latency.

Serving controls

Admit by token/cache budget with safety margin, cap input/output lengths, free cancelled sequences and expose overflow behavior. Distinguish allocated/reserved HBM and account for prefix sharing only when observed and safely scoped.

Evaluation

Sweep sequence length and concurrency, include long outputs, record peak memory/OOM, TTFT/gaps/goodput and quality. Compare equal workloads and precision policies; do not present theoretical compressed size as a speed benchmark.

L — Launch with evidence

Sweep context and concurrency with generated-token growth; require memory, tail-latency and task-slice evidence for the advertised limits.

Critical decision condition: Loading weights alone cannot justify a concurrent-serving capacity claim.

Attach raw artifacts, the tested configuration, population, denominator, date and limitations to the release record. Missing or stale evidence holds the dependent decision. A favorable value forecast cannot offset a failed critical condition.

D — Develop ownership and reuse

The serving owner publishes supported limits and overload behavior; the model owner maintains quantization regressions.

Candidate reusable asset: Memory accounting and context-admission fixture.

Commercial decision and next commitment

Accountable decision owner: Serving release owner with model/product owner.

The initial commitment is the bounded journey above, with the stated exclusions. Confirm access, customer staffing and acceptance ownership during discovery; price and schedule unresolved dependencies explicitly. A new source, tenant class, language, action or operating envelope requires an impact review rather than silent scope growth.

Approve only the next exposure level supported by the evidence. The next useful investment is: measure peak allocation and quality at the proposed maximum context/concurrency.

Method and references

Apply the FIELD method, economic model and evidence rules. Methodological and technical influences are listed in the source register.