Why we stopped treating Kafka DLQ as a dumping ground

A dead-letter queue exists to catch messages a consumer couldn't process. In practice, it often becomes the place where problems go to be quietly ignored — nobody owns triage, and the queue grows until someone notices it during an incident or quietly removed by retention policy.

The fix wasn't a smarter retry policy. It was treating the DLQ as a first-class queue with an owner and a deliberate process, the same rigor as the primary pipeline. Three changes made that stick.

1. A DLQ consumer that exists, but doesn't run by default.

We keep a dedicated DLQ consumer deployed, replicas set to zero. Retrying isn't the default reaction to a failed message — it's a decision made after understanding why the message failed in the first place. Blindly retrying until exhaustion just replays the same failure N times and burns the retry budget for nothing. The consumer only gets scaled up when retrying actually makes sense — the downstream dependency recovered, the bug that caused the failure got fixed, and so on.

2. Two alerts, not one.

The first fires when a consumer failure sends a message to the DLQ — that's the signal that starts root-cause triage. The second fires if the DLQ consumer itself gets exhausted — working through the backlog without resolving it, or failing again on the reprocessed messages. That second alert is what tells us scaling the consumer up didn't actually fix anything.

3. Manual root cause before manual retry.

Turning the DLQ consumer on is a human decision, not an automated response to backlog size. Someone confirms root cause first. That single gate is what turned the DLQ from a place where failures got quietly replayed (and often failed again) into a queue with an actual owner.

---