The Deployment Passed—Half the Answers Stopped
P1 — STREAMS CUT DURING ROLLOUT
Support: The replacement passed health checks, but 38 of 80 in-flight answers stopped mid-sentence when old workers received SIGTERM. HTTP success stayed green because responses had already started.
Your job: Fix only `advance_shutdown()` in `lifecycle.py`. Stop new admissions while preserving active generations, then apply an explicit timeout policy at a fixed deadline. Repeated shutdown signals must be safe.
Investigate: Run python drain_monitor.py to compare a normal rollout with a stuck generation. The same deterministic CPU replay and fake clock drive the tests—no GPU, model download, or cloud account required.
Proof: Normal drain completes both answers; a stuck generation is interrupted only at the deadline; late traffic is refused by the old worker. Run python -m pytest -v, then open a PR against main.
Book: A healthy replacement means the old worker can stop. Prod: New-request routing and in-flight stream completion have separate lifecycles.
This exercise repairs a custom lifecycle adapter. It teaches readiness, admission, graceful shutdown, and engine-owned retirement using an explicit simulation contract, not real-GPU performance certification.
Skills you'll prove
Interview Tip
A healthy replacement does not mean the old worker is idle. I stop discovery and local admission while keeping existing streams alive, then exit after they retire. A fixed deadline bounds the rollout, and unfinished requests receive an explicit abort policy after the grace period. I verify stream completion, timeout cleanup, and repeated-signal safety rather than relying only on HTTP health.