1. Store-First Durability
No job is ever processed in-memory only. Upon HTTP POST / enqueue, the job is persisted to PostgreSQL/Supabase in pending status before any worker thread is notified.
2. Mutual Exclusion via FOR UPDATE SKIP LOCKED
Multiple concurrent workers continuously poll available jobs using atomic row locks:
SELECT id FROM jobs WHERE status = 'pending' AND next_run_at <= now() ORDER BY priority_weight DESC, created_at ASC LIMIT 1 FOR UPDATE SKIP LOCKED;
This guarantees that two workers can never race or claim the exact same task simultaneously.
3. Heartbeats & Crash Recovery Watchdog
If a worker abruptly crashes or loses network connectivity mid-execution, its heartbeat ceases. The watchdog reaper detects the expired heartbeat, safely logs an abandoned attempt in telemetry, and re-queues the job without risking triple execution.
4. Dead Letter Queue (DLQ)
Jobs that repeatedly fail and exhaust their max retry ceiling transition to failed and are quarantined in the Dead Letter Queue for operator root-cause diagnosis and one-click manual retry.