Maximum Batched Tokens: When a Bigger Batch Makes Chat Slower
STREAMING_LATENCY_REGRESSION — GPU utilization is not user experience
Inference Platform: Raising max_num_batched_tokens to 32,768 increased throughput to 610 tokens/s and GPU utilization to 94%, but a long prefill pushed p95 inter-token latency to 186ms against a 70ms SLO.
The trap: Large token budgets improve prefill efficiency and TTFT while extending scheduling iterations. Active decode requests wait longer for their next token, so a healthy throughput graph can hide frozen streams.
Your task
Implement `choose_max_batched_tokens()` in `token_budget_scheduler.py`. Benchmark every allowed budget, reject candidates that violate any workload SLO, and choose the highest-throughput passing configuration. The policy must adapt across interactive, premium-streaming, and offline workloads.
Book: Maximize batch size and utilization. Prod: Maximize goodput under explicit TTFT, ITL, and throughput SLOs.
Skills you'll prove
Interview Tip
Maximum batched tokens caps token work per scheduler iteration, not request count. Larger values can improve prefill throughput and TTFT but lengthen iterations and hurt tail ITL for active streams. Tune against representative prompt and output distributions using TTFT, tail ITL, throughput, goodput, and KV-cache pressure; separate latency classes or serving pools when one global policy cannot satisfy both interactive and offline traffic.