The Cold Start Latency Spike
COLD_START_LATENCY_SPIKE — Kubernetes says Ready before the model is ready
Platform Team: A new LLM pod entered Service endpoints after 2 seconds, but weights, CUDA, and kernel warmup needed another 45 seconds. Its first routed request took 45,200ms.
The trap: An HTTP health check proves the process can answer a request. It does not prove the model can run inference. Kubernetes startup and readiness probes have different jobs, and readiness is what gates Service traffic.
Your task
Implement `OptimizedPod.startup()` in `cold_start.py`. Preserve the model-loading dependency order, complete warmup, pass the startup probe, and publish readiness only when the pod can serve a 200ms inference immediately.
Book: Mark the pod ready when /health responds. Prod: Keep warming replicas out of Service endpoints until model-aware readiness succeeds.
Skills you'll prove
Interview Tip
For slow-starting inference pods, use a startup probe with a failure budget above measured cold-start tails, then make readiness reflect the model being loaded and warmed. Readiness controls Service endpoints; startup delays liveness and readiness evaluation while initialization proceeds. Warm representative shapes before accepting traffic, and keep existing ready capacity during scale-out.