Runbook: Chaos drill (monthly, staging)
Test matrix (docs/plan/09): worker kill, Redis restart, provider 5xx injection — monthly, manual. Script: infra/scripts/chaos.sh.
Test matrix (docs/plan/09): worker kill, Redis restart, provider 5xx injection — monthly, manual. Script: infra/scripts/chaos.sh.
Prerequisites
- A staging server (
.env.staging,PROVIDER_MODE=mock— the script refuses to run in an environment with real providers). - The synthetic monitoring app and key (
docs/runbooks/synthetic-monitoring.md):SYNTHETIC_API_URL,SYNTHETIC_API_KEY,SYNTHETIC_APP_ID,SYNTHETIC_SUBSCRIPTION_ID. jq,curl, Docker Compose v2.24+.
Running
./infra/scripts/chaos.sh allSingle scenario: worker-kill, redis-restart, provider-5xx.
| Scenario | Injection | Expected | Observation |
|---|---|---|---|
| worker-kill | SIGKILL worker-dispatch, restart after 5 s | The interrupted job is completed by the BullMQ stalled check or reconcile; a new send goes out in < 120 s | Brief WorkerDown alert; queue_jobs_total{state="stalled"} |
| redis-restart | docker compose restart redis (AOF) | Queues resume where they left off; API rate limiting and sessions are uninterrupted | Brief RedisDown, no API 5xx |
| provider-5xx | Subscription with a mock:transient token | Retries with exponential backoff, eventually failed (MAX_RETRIES); other sends are unaffected | provider_send_total{outcome="transient"} increases, DispatchFailureRateHigh stays below threshold |
Log
For every drill, add the date, scenario, result and timings below; open an incident for any failed scenario (docs/runbooks/incident.md).
| Date | Scenario | Result | Note |
|---|
Load testing (k6-operator, distributed)
v2 Phase 16 item 5 (docs/plan/10). Manifests in infra/load/, scripts in tests/load/. The single-machine Phase 14 procedure (10k events/s, 1M fan-out) is in tests/load/README.md; this runbook covers the distributed run on Kubernetes…
Postgres role separation (keeping RLS effective in production)
The second line of defense for tenant isolation is Postgres Row Level Security (db/migrations/0005_rls.up.sql, FORCE ROW LEVEL SECURITY). Superusers and BYPASSRLS roles are never subject to RLS; if the application…