The User Left—The GPU Didn’t
P1 — GHOST GENERATIONS
On-call: Only 12 browsers are connected, but 48 generations still hold serving capacity. Users close long answers; the custom gateway closes their streams while backend work keeps running.
Your job: Fix only `handle_disconnect()` in `gateway.py`. Propagate cancellation once for queued and running requests, preserve terminal states, and leave resource retirement to the engine.
Investigate: Run python cancellation_monitor.py. The abandoned request generates six unwanted tokens and keeps its allocation, while a second user still wants their answer. The supplied fake engine demonstrates delayed acknowledgment and completion races without a GPU or model download.
Proof: Cancellation stops future scheduling; engine-owned acknowledgment releases capacity exactly once. Duplicate disconnects and inline callbacks must be safe, and unrelated users must finish. Run python -m pytest -v, then open a PR against main.
Book: Closing the socket stops generation. Prod: Downstream work needs an explicit lifecycle signal, and cancellation requested is not cancellation completed.
This exercise repairs a custom adapter using deterministic CPU events. It does not claim real-GPU performance or that a particular serving engine inherently leaks disconnected requests.
Skills you'll prove
Interview Tip
Closing a client stream does not necessarily cancel downstream inference. I propagate cancellation once and distinguish requested cancellation from confirmed retirement. The engine retains resource ownership until it is safe to release, including completion races and callbacks that happen immediately. I verify that abandoned work stops without interrupting other users or double-freeing capacity.