Speculative Decoding Made Us Slower
P1 — SPECULATION REGRESSION
On-call: Four-token drafts accelerated repetitive completions. After traffic changed, the policy still selects them for high acceptance, but drafting and verification erase the benefit.
Your job: Fix only `choose_decode_policy(profile)` in `decode_policy.py`. Compare full cost per committed token, validate profile compatibility and memory, and keep ordinary decoding as a fallback.
Investigate: Run python speculation_monitor.py. Supplied CPU profiles show draft-4 winning at 3 ms per committed token on one workload, but losing at 12 ms versus the baseline's 10 on changed traffic—even with 100% acceptance.
Proof: Choose supported, target-verified policies from matching workload, concurrency and revision profiles. Count drafting, verification and overhead; never bypass target verification. Run python -m pytest -v, then open a PR against main.
Book: Generate several draft tokens at once. Prod: Only useful if committed tokens justify the extra work.
These are authored timing records and a toy target-authority boundary, not real GPU benchmarks or a speculative sampling implementation.
Skills you'll prove
Interview Tip
I enable speculation only when end-to-end time per committed token beats ordinary decoding for the workload. Acceptance rate alone can hide an expensive drafter or verifier. I match the measured concurrency and target revision, account for extra memory, and retain ordinary decoding as a fallback. Target verification remains mandatory regardless of which draft length wins.