PushizoDocs
OperationsRunbooks

Runbook: Transactional synthetic monitoring

The Phase 14 acceptance criterion "7 days of error-free synthetic monitoring on staging" is measured with this check. .github/workflows/synthetic.yml runs scripts/synthetic.ts every 15 minutes:

The Phase 14 acceptance criterion "7 days of error-free synthetic monitoring on staging" is measured with this check. .github/workflows/synthetic.yml runs scripts/synthetic.ts every 15 minutes:

  1. GET /healthz and GET /status.json — fails if any of the api, sdk, worker, delivery components is major_outage.
  2. POST /v1/notifications (with an Idempotency-Key) — a transactional notification to a single synthetic subscription; it must be accepted within SYNTHETIC_ACCEPT_BUDGET_MS (default 1000 ms).
  3. Polling GET /v1/notifications/{id} — until sent + delivered >= 1; fails if SYNTHETIC_DELIVERY_TIMEOUT_MS (default 60 s) is exceeded or the delivery becomes failed/errored/suppressed.

Setup (staging)

  1. Create a separate org/app ("Synthetic") on staging; use push with a mock provider (staging PROVIDER_MODE=mock) or a real test device as the channel.
  2. Register a subscription (POST /sdk/v1/apps/{app}/subscriptions, token mock:ok-synthetic) and note its ID.
  3. Create an API key scoped to this app only, with notifications:write + analytics:read permissions.
  4. Secrets in the GitHub staging environment: SYNTHETIC_API_KEY, SYNTHETIC_APP_ID, SYNTHETIC_SUBSCRIPTION_ID; repository variables: STAGING_API_URL, SYNTHETIC_ENABLED=true.

Manual run: SYNTHETIC_API_URL=… SYNTHETIC_API_KEY=… SYNTHETIC_APP_ID=… SYNTHETIC_SUBSCRIPTION_ID=… npx tsx scripts/synthetic.ts (single-line JSON output; exit code 1 on failure).

Production (server cron)

While GitHub Actions is disabled, the production check runs on the server: the infra/scripts/synthetic.sh script runs scripts/synthetic.ts in a one-off container using the running API image's Node (no Node is installed on the server) and writes the result to the node-exporter textfile directory. Metrics: pushizo_synthetic_success{step}, pushizo_synthetic_last_run_timestamp_seconds, pushizo_synthetic_accept_seconds, pushizo_synthetic_delivery_seconds. Alerts: PushizoSyntheticFailing (two consecutive runs failed, critical), PushizoSyntheticStale (no run for 1 hour).

  1. Create a separate app ("Synthetic") in the dashboard. The mock provider is disabled in production, so the target must be a real subscription. Recommended: set up the email channel and add an email subscription for a catch-all mailbox (e.g. synthetic@pushizo.com, with an auto-delete rule on the mail server); this way no device receives a notification every 15 minutes. The script sends both push and email content; the channel is chosen from the subscription type.
  2. Create an API key scoped to this app only, with notifications:write + analytics:read permissions.
  3. Add SYNTHETIC_API_KEY, SYNTHETIC_APP_ID, SYNTHETIC_SUBSCRIPTION_ID to .env.prod (optionally SYNTHETIC_API_URL; if empty, https://$API_HOST).
  4. Run sudo ./infra/scripts/install-cron.sh (adds the cron job), then once by hand: sudo ./infra/scripts/synthetic.sh (single-line JSON, exit code 1 on failure). Log: /var/log/pushizo-synthetic.log.

Synthetic notifications count against the plan's monthly message quota (about 2,900 messages per month); keep the synthetic app in an org with enough quota.

On failure

  • step: health|status → docs/runbooks/incident.md (API/worker unreachable, component outage).
  • step: accept → API latency/5xx: Grafana API dashboard, ApiTransactionalLatencyHigh alert.
  • step: delivery → queue backlog or provider: docs/runbooks/queue-backlog.md, docs/runbooks/provider-outage.md.

7-day acceptance

In the Actions → synthetic workflow history, there must be no failed runs in the last 7 days (672 runs). Results appear as JSON in each run's summary (accept_ms, delivery_ms).

On this page