PushizoDocs
OperationsRunbooks

Runbook: Provider outage (APNs / FCM / Web Push / SES / Resend / Twilio / Netgsm)

If you need to pause a single channel temporarily (e.g. the provider is returning false positives/resending): remove the relevant queue from the worker-dispatch service's WORKER_QUEUES value…

Symptoms

  • Alert DispatchFailureRateHigh (dispatch job failures > 5%) or DeliveryFailureRateHigh (delivery failed/(sent+failed) > 5%).
  • Alert ProviderCredentialsInvalid / ProviderCredentialFailuresSpike: channel invalid (auth rejected), delivery error_code = CREDENTIALS.
  • Grafana Providers and delivery dashboard: failures on a single channel (dispatch-ios, dispatch-email…); rising queue_depth{state="delayed"} (rate_limited deferrals).
  • provider error codes in the logs (apns:ServiceUnavailable, fcm:UNAVAILABLE, ses:Throttling, twilio:30003).

1. Diagnosis (5 min)

  1. Provider status pages: APNs https://developer.apple.com/system-status/ · FCM https://status.firebase.google.com/ · SES https://health.aws.amazon.com/ · Resend https://resend-status.com · Twilio https://status.twilio.com · Netgsm support line.
  2. A single tenant or widespread? Loki: {service=~"worker-.*"} | json | queue="dispatch-ios" | level="error" → distribution by app_id.
    • A single app → credential problem: dashboard Channels → Validate; the org owner has already received a channel invalid email (dispatch notifyChannelInvalid).
    • All apps → provider outage or a network/IP block (from the server: curl -v https://api.push.apple.com / https://fcm.googleapis.com).
  3. Is the retry schedule working? Transient errors (5xx, timeout, rate_limited) are retried at 30 s → 2 min → 10 min → 30 min → 2 h intervals, up to 6 attempts (docs/plan/05 §5.6); rate_limited does not count as an attempt (at most 10 deferrals). The queue's delayed count shows this.

2. Response

SituationAction
Short provider outage (< 2 h)Nothing; the retry schedule covers it. Open a degraded incident on the status page (component delivery) and name the channel.
Long outage (> 2 h)The last attempt also fails and deliveries become failed. Notify campaign owners; once the outage ends, the decision to requeue failed deliveries for resending belongs to the product team (the same message may be delivered twice).
Provider rate limit (rate_limited, 429)Do not lower PROVIDER_RATE_PER_SECOND (apps/worker/src/delivery/types.ts) or per-app concurrency (APP_DISPATCH_CONCURRENCY); request a quota increase from the provider. Spread large campaigns out with send_at.
Credential rejection (auth)Channel invalid: the customer enters a new p8/service account/VAPID key → Validate. If the problem is MASTER_KEY/DEK on our side, see key-rotation.md.
Webhooks/feedback not arriving (SES/Twilio)Check the webhook URL configured at the provider (/hooks/...) and the signature verification logs (HS_WEBHOOK_SIGNATURE_INVALID). The Netgsm polling job (netgsm-poll) runs once a minute.

If you need to pause a single channel temporarily (e.g. the provider is returning false positives/resending): remove the relevant queue from the worker-dispatch service's WORKER_QUEUES value (scheduler,audience,dispatch-android,dispatch-hms,dispatch-web,dispatch-email,dispatch-sms) and run docker compose up -d worker-dispatch; iOS deliveries wait in the queue and no data is lost.

3. After the outage

  • Monitor until queue_depth{state="delayed"} and failed return to normal; if DeadLetterQueueHigh fires, inspect the Bull failed set (queue-backlog.md).
  • Close the status page incident; record the number of affected tenants and the duration in the post-mortem.
  • If a provider change is required (e.g. Resend → SES), change the channel configuration from the dashboard; the template/sender domain DNS (SPF/DKIM/DMARC) is re-verified (the dns-check job runs every 6 hours).

On this page