The Benchmark Says We Won—Customers Say Otherwise
P1 — FALSE BENCHMARK WIN
Inference release: The candidate produces more tokens per second, but customers wait longer for answers. The benchmark counts empty heartbeats as first tokens and hides failed and rejected requests.
Your job: Fix only `summarize_run(events, policy)` in `benchmark.py`. Measure first content and streaming gaps, report all request outcomes, and calculate SLO-compliant goodput over the fixed observation window.
Investigate: Run python benchmark_monitor.py. Four requests reach each configuration; the candidate emits more tokens but gives only one customer a timely complete answer. Use the supplied CPU traces and measurement policy—no GPU or model download required.
Proof: The baseline delivers 0.3 good requests/s versus the candidate's 0.1. Failed, rejected and unfinished requests stay visible, so dropping difficult traffic cannot manufacture a qualified win. Run python -m pytest -v, then open a PR against main.
Book: Higher throughput means better inference. Prod: Better for which users, at which latency and failure rate?
The fixtures are authored traces, not hardware measurements. The exercise teaches measurement correctness and honest release comparison, not real-GPU performance certification.
Skills you'll prove
Interview Tip
I distinguish first content from transport activity and throughput from goodput. I retain failures, rejections and unfinished work against offered demand, then count successful requests satisfying first-token and streaming-gap targets. A percentile over survivors can look healthy while users are rejected. I compare deployments under the same workload, observation window and SLO policy.