Four GPUs, One Expensive Bottleneck
P1 — SHARDING WITHOUT CAPACITY
Inference platform: The team spread one model across four GPUs without comparing independent replicas. The model fits, but serving capacity disappoints.
Your job: Fix only `choose_deployment(candidates, workload, budget)` in `deployment_selector.py`. Qualify memory, GPU count, topology and workload compatibility before ranking latency-compliant fleet goodput.
Investigate: Run python deployment_monitor.py. Supplied CPU profiles compare replicas and sharding under the same four-GPU budget. A small model favors replicas, a larger model requires sharding, and a fast-link profile makes sharding win on performance.
Proof: Include per-GPU KV, workspace and reserved memory; use TP times replicas for total GPU count; never multiply fleet goodput twice. Return no choice if every candidate is infeasible. Run python -m pytest -v, then open a PR against main.
Book: More GPUs make the model faster. Prod: Sharding one instance and running more instances solve different problems.
Profiles are authored evidence, not actual GPU benchmarks or universal scaling formulas. No GPU or model download is required.
Skills you'll prove
Interview Tip
I compare sharding and replication using measured fleet outcomes under the same GPU budget. Memory qualification includes KV, workspace and headroom per device, not just weights. I match the profile's workload and interconnect before ranking goodput under latency constraints. Replicas may win when the model fits, while sharding can be necessary for memory or preferable on the right topology.