The Semaphore That Didn't Stop the OOM
INGESTION_WORKER_OOM — 32 active jobs, thousands of retained payloads
Systems Platform: The Rust ingestion worker is limited to 32 concurrent jobs, but burst traffic grows memory from 420 MiB to 7.8 GiB until Kubernetes OOM-kills the pod. Active-worker metrics never exceed 32.
The trap: Each Tokio task captures its job payload before waiting for a semaphore permit. The semaphore bounds active processing while thousands of suspended tasks retain payload memory. Draining a bounded channel into unbounded tasks also defeats upstream backpressure.
Your task
Fix `run_dispatcher()` in `src/lib.rs`. Acquire an owned permit before spawning, move the permit into the worker task, and let the bounded input queue push back on producers when capacity is full. CI freezes workers at deterministic checkpoints and checks task ownership, retained bytes, error paths, capacity reuse, and eventual completion.
Book: A semaphore bounds concurrency. Prod: Resource safety depends on where work and memory become owned.
Skills you'll prove
Interview Tip
A semaphore only bounds code after permit acquisition. If I spawn a task containing a large payload and acquire inside it, every suspended waiter still owns memory. I acquire an OwnedSemaphorePermit before spawning and move it into the task, so the number of payload-owning tasks is bounded and RAII returns capacity on success or failure. A bounded channel then propagates backpressure to producers.