The Quantization Accuracy Cliff
HALLUCINATION_SPIKE_DETECTED — the model fits, the financial canary fails
ML Platform: A 70B deployment moved from FP16 to groupwise INT4. The ideal packed weight payload fell from 140 GB to 35 GB, but task-specific decision agreement fell to 60.7%.
The trap: Weight-only RTN sees only the weight matrix. Modest weights on channels with unusually large production activations can dominate the layer output, so their rounding error is amplified.
Your task
Implement `activation_aware_quantize()` in `quantization.py`. Measure per-channel activation magnitude on representative calibration traffic, select salient channels, and apply equivalent channel scaling before groupwise INT4 quantization.
Book: Quantize the weights and check model size. Prod: Calibrate on representative inputs and gate the exact product task.
Skills you'll prove
Interview Tip
Low-bit quantization must be evaluated on representative product data. AWQ uses activation statistics to identify salient weight channels and searches channel scaling that reduces their quantization error while retaining a hardware-friendly low-bit format. GPTQ instead uses approximate second-order information, while SmoothQuant targets weight-and-activation quantization by migrating activation difficulty into weights.