Same RPS, Triple the Queue
P1 — TOKEN WORKLOAD BLIND SPOT
On-call: Larger documents launched. Request rate remains 15 RPS, but queued token work triples and first-token latency climbs. The request-count autoscaler sees no change.
Your job: Fix only `desired_replicas(snapshot, policy)` in `autoscaler.py`. Use supplied incoming and queued token-work estimates, sustained observations, ready versus starting capacity, cooldown and budget limits.
Investigate: Run python autoscaling_monitor.py. The authored CPU replay follows a controller through workload growth, replica warmup and a quieter period. Return desired total capacity, not replicas to add.
Proof: Long-document demand raises the target from two to four; a repeated decision counts the two replicas already starting. Scale-down preserves busy replicas and respects cooldown. Run python -m pytest -v, then open a PR against main.
Book: Scale with requests per second. Prod: Requests contain different amounts of work, and new capacity takes time.
The supplied workload profiles are synthetic. This tests scaling decisions and reconciliation, not real GPU throughput or measured latency recovery.
Skills you'll prove
Interview Tip
I scale on estimated incoming and queued token work rather than request count alone. I count ready and starting replicas separately so slow startup does not trigger duplicate capacity requests. Sustained observations justify growth, while cooldown and active-work protection govern shrinkage. The budget bounds capacity, so hitting the limit may still require overload control.