Runbook: Queue backlog and DLQ
Alerts: QueueDepthHigh (waiting + prioritized > 50k, 10 min; prioritized jobs live in BullMQ's prioritized set), DeadLetterQueueHigh (failed > 100), WorkerJobsStalled (jobs present, none completing), WorkerDown. Dashboard:…
Alerts: QueueDepthHigh (waiting + prioritized > 50k, 10 min; prioritized jobs live in BullMQ's prioritized set), DeadLetterQueueHigh (failed > 100), WorkerJobsStalled (jobs present, none completing),
WorkerDown. Dashboard: Grafana Queues. Queues and priorities: apps/worker/src/queues.ts.
1. Which queue?
# compose
docker compose --env-file .env.prod -f infra/compose/docker-compose.prod.yml exec redis \
redis-cli -a "$REDIS_PASSWORD" --no-auth-warning eval "local r={} for _,q in ipairs(ARGV) do r[#r+1]=q..'='..redis.call('llen','hs:'..q..':wait')..'+'..redis.call('zcard','hs:'..q..':prioritized')..'/'..redis.call('zcard','hs:'..q..':delayed')..'/'..redis.call('zcard','hs:'..q..':failed') end return r" 0 \
scheduler audience dispatch-ios dispatch-android dispatch-hms dispatch-web dispatch-email dispatch-sms events webhooks journeys maintenance
# platform admin: GET /admin/v1/system/health (queue depths + worker heartbeats)| Queue | Normal cause | Typical fix |
|---|---|---|
audience | campaign to 1M+ subscriptions, concurrency 2 | wait; add worker-dispatch replicas |
dispatch-* | large campaign (200 deliveries/batch), slow provider/rate limit (delayed grows) | provider-outage.md; increase WORKER_CONCURRENCY and replicas |
events | SDK/API event burst (target of 10k events/s) | add worker-system replicas; move the events queue to a separate service |
webhooks | slow customer endpoint (10 s timeout, 1 concurrent) | endpoint health; WEBHOOK_TIMEOUT_MS |
journeys | wait nodes expiring at the same time | add replicas |
maintenance | retention/rollup running long | single concurrency; check the job duration (queue_job_duration_seconds{queue="maintenance"}) |
2. Quick capacity
# compose: a second dispatch worker (same queues, Redis distributes fairly)
docker compose --env-file .env.prod -f infra/compose/docker-compose.prod.yml up -d --scale worker-dispatch=2
# K8s: HPA/KEDA (infra/k8s/pushizo values: worker.groups[].hpa / keda)
kubectl -n pushizo scale deploy/pushizo-worker-dispatch --replicas=4Limits: Postgres connections (min(DATABASE_POOL_MAX,5) per worker), provider rate limit (5000/s per app), APP_DISPATCH_CONCURRENCY=4.
A single tenant cannot starve the others (fair scheduling semaphore); still, if a huge campaign is waiting, that is expected behavior.
3. Stalled / lost jobs
stalledlogs: lock 60 s,maxStalledCount=2→ the job becomesfailed. If you see this often, the handler is blocking (DB lock, provider timeout).- Redis loss/flush:
pushizo_deliveries_pendingis high but the queue is empty →npm run queue:recover(restore.md §5). - A message stuck in
queued(audience job lost):reconcile-deliveriespicks it up after 15 min; if urgent,queue:recoveralso re-adds the audience job.
4. Inspecting and retrying the DLQ (failed set)
Preferred path: Admin panel → System health → click the "Failed" count in the queue table. The last 100 failed jobs (name, error, attempts, time; the payload is not shown) are listed; you can "Retry" them one by one or in bulk (up to 1000). API: GET /admin/v1/queues/{kuyruk}/failed, POST /admin/v1/queues/{kuyruk}/failed/retry { job_id? } (platform admin + MFA; admin.queue_retry is written to the audit log).
From the command line:
# last 20 failed jobs (reason + data)
docker compose … exec redis redis-cli -a "$REDIS_PASSWORD" --no-auth-warning zrevrange hs:dispatch-ios:failed 0 19
docker compose … exec redis redis-cli -a "$REDIS_PASSWORD" --no-auth-warning hgetall "hs:dispatch-ios:<jobId>"- Delivery jobs are idempotent:
queue:recoveris enough to requeue them (dispatch only sendspendingrows). - For permanent errors (
RenderError, invalid content) the message is closed asfailed; follow up with the customer. - The
failedset keeps 5000 entries (removeOnFail); to clean up old entries:zremrangebyrank hs:<q>:failed 0 -1001.
5. Prevention
- For the campaign start latency SLO (send_at → first send < 60 s) the scheduler runs with concurrency 1; do not schedule large campaigns for the same minute.
- Quotas: plan limits (
HS_QUOTA_EXCEEDED) curb abuse; new orgs get 1000 messages per 24 h. - Load test targets and capacity plan:
tests/load/README.md.
Runbook: Provider outage (APNs / FCM / Web Push / SES / Resend / Twilio / Netgsm)
If you need to pause a single channel temporarily (e.g. the provider is returning false positives/resending): remove the relevant queue from the worker-dispatch service's WORKER_QUEUES value…
Runbook: Data key (DEK) rotation
Provider credentials (APNs p8, FCM private_key, VAPID private key) are stored with envelope encryption (ADR-0008): MASTER_KEY only wraps the data keys (DEKs); each record is encrypted with the active DEK as v2.<kid>.<iv>.<tag>.<ct>…