Runbook: Transactional synthetic monitoring
The Phase 14 acceptance criterion "7 days of error-free synthetic monitoring on staging" is measured with this check. .github/workflows/synthetic.yml runs scripts/synthetic.ts every 15 minutes:
The Phase 14 acceptance criterion "7 days of error-free synthetic monitoring on staging" is measured with this check. .github/workflows/synthetic.yml runs scripts/synthetic.ts every 15 minutes:
GET /healthzandGET /status.json— fails if any of theapi,sdk,worker,deliverycomponents ismajor_outage.POST /v1/notifications(with an Idempotency-Key) — a transactional notification to a single synthetic subscription; it must be accepted withinSYNTHETIC_ACCEPT_BUDGET_MS(default 1000 ms).- Polling
GET /v1/notifications/{id}— untilsent + delivered >= 1; fails ifSYNTHETIC_DELIVERY_TIMEOUT_MS(default 60 s) is exceeded or the delivery becomesfailed/errored/suppressed.
Setup (staging)
- Create a separate org/app ("Synthetic") on staging; use push with a mock provider (staging
PROVIDER_MODE=mock) or a real test device as the channel. - Register a subscription (
POST /sdk/v1/apps/{app}/subscriptions, tokenmock:ok-synthetic) and note its ID. - Create an API key scoped to this app only, with
notifications:write+analytics:readpermissions. - Secrets in the GitHub
stagingenvironment:SYNTHETIC_API_KEY,SYNTHETIC_APP_ID,SYNTHETIC_SUBSCRIPTION_ID; repository variables:STAGING_API_URL,SYNTHETIC_ENABLED=true.
Manual run: SYNTHETIC_API_URL=… SYNTHETIC_API_KEY=… SYNTHETIC_APP_ID=… SYNTHETIC_SUBSCRIPTION_ID=… npx tsx scripts/synthetic.ts (single-line JSON output; exit code 1 on failure).
Production (server cron)
While GitHub Actions is disabled, the production check runs on the server: the infra/scripts/synthetic.sh script runs scripts/synthetic.ts in a one-off container using the running API image's Node (no Node is installed on the server) and writes the result to the node-exporter textfile directory. Metrics: pushizo_synthetic_success{step}, pushizo_synthetic_last_run_timestamp_seconds, pushizo_synthetic_accept_seconds, pushizo_synthetic_delivery_seconds. Alerts: PushizoSyntheticFailing (two consecutive runs failed, critical), PushizoSyntheticStale (no run for 1 hour).
- Create a separate app ("Synthetic") in the dashboard. The mock provider is disabled in production, so the target must be a real subscription. Recommended: set up the email channel and add an email subscription for a catch-all mailbox (e.g.
synthetic@pushizo.com, with an auto-delete rule on the mail server); this way no device receives a notification every 15 minutes. The script sends both push and email content; the channel is chosen from the subscription type. - Create an API key scoped to this app only, with
notifications:write+analytics:readpermissions. - Add
SYNTHETIC_API_KEY,SYNTHETIC_APP_ID,SYNTHETIC_SUBSCRIPTION_IDto.env.prod(optionallySYNTHETIC_API_URL; if empty,https://$API_HOST). - Run
sudo ./infra/scripts/install-cron.sh(adds the cron job), then once by hand:sudo ./infra/scripts/synthetic.sh(single-line JSON, exit code 1 on failure). Log:/var/log/pushizo-synthetic.log.
Synthetic notifications count against the plan's monthly message quota (about 2,900 messages per month); keep the synthetic app in an org with enough quota.
On failure
step: health|status→docs/runbooks/incident.md(API/worker unreachable, component outage).step: accept→ API latency/5xx: Grafana API dashboard,ApiTransactionalLatencyHighalert.step: delivery→ queue backlog or provider:docs/runbooks/queue-backlog.md,docs/runbooks/provider-outage.md.
7-day acceptance
In the Actions → synthetic workflow history, there must be no failed runs in the last 7 days (672 runs). Results appear as JSON in each run's summary (accept_ms, delivery_ms).
Runbook: Data key (DEK) rotation
Provider credentials (APNs p8, FCM private_key, VAPID private key) are stored with envelope encryption (ADR-0008): MASTER_KEY only wraps the data keys (DEKs); each record is encrypted with the active DEK as v2.<kid>.<iv>.<tag>.<ct>…
Cells (EU/TR data residency): setup, region moves, cleanup, DR
ADR-0032. A cell is a complete Pushizo installation with its own Postgres/Redis/ClickHouse/S3 and API/worker deployment. One cell (CONTROL_PLANE_MODE=true) is also the control plane: the org → cell mapping (org_regions),…