PushizoDocs
OperationsRunbooks

Runbook: Chaos drill (monthly, staging)

Test matrix (docs/plan/09): worker kill, Redis restart, provider 5xx injection — monthly, manual. Script: infra/scripts/chaos.sh.

Test matrix (docs/plan/09): worker kill, Redis restart, provider 5xx injection — monthly, manual. Script: infra/scripts/chaos.sh.

Prerequisites

  • A staging server (.env.staging, PROVIDER_MODE=mock — the script refuses to run in an environment with real providers).
  • The synthetic monitoring app and key (docs/runbooks/synthetic-monitoring.md): SYNTHETIC_API_URL, SYNTHETIC_API_KEY, SYNTHETIC_APP_ID, SYNTHETIC_SUBSCRIPTION_ID.
  • jq, curl, Docker Compose v2.24+.

Running

./infra/scripts/chaos.sh all

Single scenario: worker-kill, redis-restart, provider-5xx.

ScenarioInjectionExpectedObservation
worker-killSIGKILL worker-dispatch, restart after 5 sThe interrupted job is completed by the BullMQ stalled check or reconcile; a new send goes out in < 120 sBrief WorkerDown alert; queue_jobs_total{state="stalled"}
redis-restartdocker compose restart redis (AOF)Queues resume where they left off; API rate limiting and sessions are uninterruptedBrief RedisDown, no API 5xx
provider-5xxSubscription with a mock:transient tokenRetries with exponential backoff, eventually failed (MAX_RETRIES); other sends are unaffectedprovider_send_total{outcome="transient"} increases, DispatchFailureRateHigh stays below threshold

Log

For every drill, add the date, scenario, result and timings below; open an incident for any failed scenario (docs/runbooks/incident.md).

DateScenarioResultNote

On this page