The Continuous Batching Bottleneck
LOW_SCHEDULER_OCCUPANCY — requests waiting, decode slots idle
ML Platform: A 32-slot generation batch starts full, but short responses finish while long responses keep the batch open. Waiting requests cannot enter until the longest response completes.
Measured workload: Static request-level batching uses only 20.3% of available decode token-slots. The queue still has work, but completed sequences leave holes behind the batch barrier.
Your task
Implement `ContinuousBatchScheduler.serve()` in `continuous_batching.py`. After every decode iteration, evict completed requests and immediately fill their slots from the waiting queue.
Book: Build a batch and wait for it. Prod: Change batch composition after every generated token.
Skills you'll prove
Interview Tip
Static request-level batching creates a barrier at the longest output. With iteration-level scheduling, the server runs one decode step, returns finished requests, and admits waiting work into the newly free slots. This raises useful tokens per iteration and reduces queue delay under mixed-length traffic. Production schedulers must also account for KV-cache capacity, prefill cost, latency objectives, and fairness.