Runbook: Provider outage (APNs / FCM / Web Push / SES / Resend / Twilio / Netgsm)
If you need to pause a single channel temporarily (e.g. the provider is returning false positives/resending): remove the relevant queue from the worker-dispatch service's WORKER_QUEUES value…
Symptoms
- Alert
DispatchFailureRateHigh(dispatch job failures > 5%) orDeliveryFailureRateHigh(delivery failed/(sent+failed) > 5%). - Alert
ProviderCredentialsInvalid/ProviderCredentialFailuresSpike: channelinvalid(auth rejected), deliveryerror_code = CREDENTIALS. - Grafana Providers and delivery dashboard: failures on a single channel (
dispatch-ios,dispatch-email…); risingqueue_depth{state="delayed"}(rate_limited deferrals). providererror codes in the logs (apns:ServiceUnavailable,fcm:UNAVAILABLE,ses:Throttling,twilio:30003).
1. Diagnosis (5 min)
- Provider status pages: APNs https://developer.apple.com/system-status/ · FCM https://status.firebase.google.com/ · SES https://health.aws.amazon.com/ · Resend https://resend-status.com · Twilio https://status.twilio.com · Netgsm support line.
- A single tenant or widespread? Loki:
{service=~"worker-.*"} | json | queue="dispatch-ios" | level="error"→ distribution byapp_id.- A single app → credential problem: dashboard Channels → Validate; the org owner has already received a
channel invalidemail (dispatchnotifyChannelInvalid). - All apps → provider outage or a network/IP block (from the server:
curl -v https://api.push.apple.com/https://fcm.googleapis.com).
- A single app → credential problem: dashboard Channels → Validate; the org owner has already received a
- Is the retry schedule working? Transient errors (5xx, timeout, rate_limited) are retried at 30 s → 2 min → 10 min → 30 min → 2 h intervals,
up to 6 attempts (docs/plan/05 §5.6);
rate_limiteddoes not count as an attempt (at most 10 deferrals). The queue'sdelayedcount shows this.
2. Response
| Situation | Action |
|---|---|
| Short provider outage (< 2 h) | Nothing; the retry schedule covers it. Open a degraded incident on the status page (component delivery) and name the channel. |
| Long outage (> 2 h) | The last attempt also fails and deliveries become failed. Notify campaign owners; once the outage ends, the decision to requeue failed deliveries for resending belongs to the product team (the same message may be delivered twice). |
Provider rate limit (rate_limited, 429) | Do not lower PROVIDER_RATE_PER_SECOND (apps/worker/src/delivery/types.ts) or per-app concurrency (APP_DISPATCH_CONCURRENCY); request a quota increase from the provider. Spread large campaigns out with send_at. |
| Credential rejection (auth) | Channel invalid: the customer enters a new p8/service account/VAPID key → Validate. If the problem is MASTER_KEY/DEK on our side, see key-rotation.md. |
| Webhooks/feedback not arriving (SES/Twilio) | Check the webhook URL configured at the provider (/hooks/...) and the signature verification logs (HS_WEBHOOK_SIGNATURE_INVALID). The Netgsm polling job (netgsm-poll) runs once a minute. |
If you need to pause a single channel temporarily (e.g. the provider is returning false positives/resending): remove the relevant queue from the worker-dispatch service's
WORKER_QUEUES value (scheduler,audience,dispatch-android,dispatch-hms,dispatch-web,dispatch-email,dispatch-sms)
and run docker compose up -d worker-dispatch; iOS deliveries wait in the queue and no data is lost.
3. After the outage
- Monitor until
queue_depth{state="delayed"}andfailedreturn to normal; ifDeadLetterQueueHighfires, inspect the Bull failed set (queue-backlog.md). - Close the status page incident; record the number of affected tenants and the duration in the post-mortem.
- If a provider change is required (e.g. Resend → SES), change the channel configuration from the dashboard; the template/sender domain DNS
(SPF/DKIM/DMARC) is re-verified (the
dns-checkjob runs every 6 hours).
Runbook: Incident management and status page
Sources: Alertmanager (#pushizo-alerts, #pushizo-oncall, email), Grafana dashboards (infra/monitoring/grafana/dashboards), Loki logs (tracing by request_id), Sentry. Status page: https://status.<domain> →…
Runbook: Queue backlog and DLQ
Alerts: QueueDepthHigh (waiting + prioritized > 50k, 10 min; prioritized jobs live in BullMQ's prioritized set), DeadLetterQueueHigh (failed > 100), WorkerJobsStalled (jobs present, none completing), WorkerDown. Dashboard:…