PushizoDocs
OperationsRunbooks

Runbook: Queue backlog and DLQ

Alerts: QueueDepthHigh (waiting + prioritized > 50k, 10 min; prioritized jobs live in BullMQ's prioritized set), DeadLetterQueueHigh (failed > 100), WorkerJobsStalled (jobs present, none completing), WorkerDown. Dashboard:…

Alerts: QueueDepthHigh (waiting + prioritized > 50k, 10 min; prioritized jobs live in BullMQ's prioritized set), DeadLetterQueueHigh (failed > 100), WorkerJobsStalled (jobs present, none completing), WorkerDown. Dashboard: Grafana Queues. Queues and priorities: apps/worker/src/queues.ts.

1. Which queue?

# compose
docker compose --env-file .env.prod -f infra/compose/docker-compose.prod.yml exec redis \
  redis-cli -a "$REDIS_PASSWORD" --no-auth-warning eval "local r={} for _,q in ipairs(ARGV) do r[#r+1]=q..'='..redis.call('llen','hs:'..q..':wait')..'+'..redis.call('zcard','hs:'..q..':prioritized')..'/'..redis.call('zcard','hs:'..q..':delayed')..'/'..redis.call('zcard','hs:'..q..':failed') end return r" 0 \
  scheduler audience dispatch-ios dispatch-android dispatch-hms dispatch-web dispatch-email dispatch-sms events webhooks journeys maintenance
# platform admin: GET /admin/v1/system/health (queue depths + worker heartbeats)
QueueNormal causeTypical fix
audiencecampaign to 1M+ subscriptions, concurrency 2wait; add worker-dispatch replicas
dispatch-*large campaign (200 deliveries/batch), slow provider/rate limit (delayed grows)provider-outage.md; increase WORKER_CONCURRENCY and replicas
eventsSDK/API event burst (target of 10k events/s)add worker-system replicas; move the events queue to a separate service
webhooksslow customer endpoint (10 s timeout, 1 concurrent)endpoint health; WEBHOOK_TIMEOUT_MS
journeyswait nodes expiring at the same timeadd replicas
maintenanceretention/rollup running longsingle concurrency; check the job duration (queue_job_duration_seconds{queue="maintenance"})

2. Quick capacity

# compose: a second dispatch worker (same queues, Redis distributes fairly)
docker compose --env-file .env.prod -f infra/compose/docker-compose.prod.yml up -d --scale worker-dispatch=2
# K8s: HPA/KEDA (infra/k8s/pushizo values: worker.groups[].hpa / keda)
kubectl -n pushizo scale deploy/pushizo-worker-dispatch --replicas=4

Limits: Postgres connections (min(DATABASE_POOL_MAX,5) per worker), provider rate limit (5000/s per app), APP_DISPATCH_CONCURRENCY=4. A single tenant cannot starve the others (fair scheduling semaphore); still, if a huge campaign is waiting, that is expected behavior.

3. Stalled / lost jobs

  • stalled logs: lock 60 s, maxStalledCount=2 → the job becomes failed. If you see this often, the handler is blocking (DB lock, provider timeout).
  • Redis loss/flush: pushizo_deliveries_pending is high but the queue is empty → npm run queue:recover (restore.md §5).
  • A message stuck in queued (audience job lost): reconcile-deliveries picks it up after 15 min; if urgent, queue:recover also re-adds the audience job.

4. Inspecting and retrying the DLQ (failed set)

Preferred path: Admin panel → System health → click the "Failed" count in the queue table. The last 100 failed jobs (name, error, attempts, time; the payload is not shown) are listed; you can "Retry" them one by one or in bulk (up to 1000). API: GET /admin/v1/queues/{kuyruk}/failed, POST /admin/v1/queues/{kuyruk}/failed/retry { job_id? } (platform admin + MFA; admin.queue_retry is written to the audit log).

From the command line:

# last 20 failed jobs (reason + data)
docker compose … exec redis redis-cli -a "$REDIS_PASSWORD" --no-auth-warning zrevrange hs:dispatch-ios:failed 0 19
docker compose … exec redis redis-cli -a "$REDIS_PASSWORD" --no-auth-warning hgetall "hs:dispatch-ios:<jobId>"
  • Delivery jobs are idempotent: queue:recover is enough to requeue them (dispatch only sends pending rows).
  • For permanent errors (RenderError, invalid content) the message is closed as failed; follow up with the customer.
  • The failed set keeps 5000 entries (removeOnFail); to clean up old entries: zremrangebyrank hs:<q>:failed 0 -1001.

5. Prevention

  • For the campaign start latency SLO (send_at → first send < 60 s) the scheduler runs with concurrency 1; do not schedule large campaigns for the same minute.
  • Quotas: plan limits (HS_QUOTA_EXCEEDED) curb abuse; new orgs get 1000 messages per 24 h.
  • Load test targets and capacity plan: tests/load/README.md.

On this page