The Cache Is Warm—Why Is Every Request Cold?
P1 — PREFIX ROUTING MISS
On-call: Requests repeat a long system prompt, but round-robin keeps missing the worker holding its prefix. Pinning all traffic to that warm worker then creates a queue hotspot.
Your job: Fix only `choose_worker(request, workers)` in `router.py`. Balance reusable prefill against queue delay, require healthy compatible workers, and reject stale cache credit.
Investigate: Run python routing_monitor.py. Compare a warm idle worker, a warm overloaded worker and a repeated-prefix burst using supplied CPU timing profiles.
Proof: Warm idle routing estimates 40 ms to first content versus 120 ms cold, but 500 ms of queue reverses that choice. The fixed burst improves prefix reuse while spreading work. Run python -m pytest -v, then open a PR against main.
Book: Cache repeated work. Prod: Route to compatible reusable state without overwhelming its worker.
These are synthetic profiles, not GPU measurements. Cache fingerprints and compatibility metadata are supplied; the learner repairs routing rather than implementing a tokenizer or cache.
Skills you'll prove
Interview Tip
I route toward reusable prefix state only when the worker is compatible and its metadata is still valid. I compare saved prefill time against queued work rather than pinning traffic blindly. A cold idle worker can beat a warm overloaded one. I validate cache hits and first-token waits together, because cache-hit rate alone can hide a routing hotspot.