FIELD · Capacity and platforms · 4 min read
Find the bottleneck behind an apparently idle CPU
Diagnose one service with request tracing, profiling and dependency timing.
Arunank Sharan · Kuzushi Labs ·
In this article
This worked application uses explicit planning assumptions to demonstrate FIELD. Quantities below are inputs, calculated implications or proposed targets; they are not reported customer results. The engagement decision remains conditional on the evidence described.
Planning basis and engagement boundary
| Item | Basis for this application |
|---|---|
| Demand and context | Eight vCPUs; 35% aggregate CPU utilization in the scenario |
| First delivery boundary | Diagnose one service with request tracing, profiling and dependency timing |
| Acceptance to agree and test | Demonstrate p99 improvement under unchanged load and correctness checks |
| Exclusion or constraint | Additional machines are deferred until the limiting resource is identified. |
F — Frame the outcome
Restore user-visible tail latency without spending on machines that do not address the bottleneck.
Value measure: p99 goodput and cost per request at matched workload.
I — Investigate the workflow
Trace CPU service time, database/queue waits, throttling, event-loop delay and native thread counts; aggregate 35% CPU is insufficient diagnosis.
First evidence task: Capture per-stage time and thread/pool behavior during a p99 spike.
E — Engineer the solution
Use bounded concurrency and whole-path profiling; tune processes/native libraries/database pools as one resource budget.
The implementation assessment below makes the authority, workload, resource and failure assumptions concrete. Its numbers are calculations from the stated scenario, not a vendor performance guarantee.
Hypothesis tree
Thirty-five percent average CPU on eight vCPUs does not imply spare capacity on the critical path. One event-loop thread can be saturated while others idle; cgroup quota can throttle despite an idle host; requests can wait for locks, DB pools, network or the GIL in the relevant runtime. Native libraries can also oversubscribe threads.
Investigation
Reproduce a realistic p99 regression and segment by request/host/input size. Capture traces separating queue, tokenization, DB wait and response work; inspect per-core CPU, runnable delay, event-loop lag, throttling, pool wait, native thread count, allocation/GC and memory growth. Compare CPU-seconds/request with wall time. Name the actual runtime/build before claiming threading behavior.
Remediation plan
Move blocking CPU work off the event loop into a bounded suitable executor or native/vectorized path, batch/cap tokenizer work where justified, and align process/native thread pools with measured hardware. Tune the database pool to useful DB concurrency; more application workers cannot create DB capacity. Preserve a total concurrency/deadline budget.
Worked example
If a stage takes 4ms of serial CPU/request on one execution thread, its ideal ceiling is 250 requests/s, before overhead. An eight-vCPU host does not give that single stage 2,000/s unless the work can execute in parallel. If 500 requests/s each wait 80ms remotely, 40 average requests wait, requiring occupancy management but not necessarily 40 OS threads.
Changed conditions
More workers may duplicate model/tokenizer memory, expand DB pools and worsen locks. Evaluate process count and native threads jointly, and account for RSS/cgroup peaks. A lower median with worse p99 and rejection rate is not an unconditional improvement.
Evidence
State one causal hypothesis, change one factor and retest the same load. Report throughput at SLO, p99, CPU demand, memory, DB waits and effects on adjacent tenants. Keep unresolved alternatives explicit.
L — Launch with evidence
Change one limiting mechanism and compare the same offered load, p99, errors and downstream pressure against baseline.
Critical decision condition: A fix that meets local latency by overloading the database is not acceptable.
Attach raw artifacts, the tested configuration, population, denominator, date and limitations to the release record. Missing or stale evidence holds the dependent decision. A favorable value forecast cannot offset a failed critical condition.
D — Develop ownership and reuse
The API owner maintains the performance regression fixture and operators monitor the identified saturation signal.
Candidate reusable asset: Critical-path profiling and concurrency budget.
Commercial decision and next commitment
Accountable decision owner: API service owner with dependency owner.
The initial commitment is the bounded journey above, with the stated exclusions. Confirm access, customer staffing and acceptance ownership during discovery; price and schedule unresolved dependencies explicitly. A new source, tenant class, language, action or operating envelope requires an impact review rather than silent scope growth.
Approve only the next exposure level supported by the evidence. The next useful investment is: capture per-stage time and thread/pool behavior during a p99 spike.
Method and references
Apply the FIELD method, economic model and evidence rules. Methodological and technical influences are listed in the source register.