Status page and monitoring
status.json and the public status page, incident management, health probes, Prometheus metrics, alerts, Grafana dashboards, Loki logs and synthetic transactional monitoring.
Status page
- Public:
https://status.pushizo.com(thestatussubdomain on your own installation) shows component states and open incidents; visitors can subscribe by email. - Data endpoint:
GET /status.json(no authentication) →
{
"status": "operational",
"checked_at": "2026-10-17T09:00:00Z",
"components": [
{ "name": "api", "status": "operational" },
{ "name": "dashboard", "status": "operational" },
{ "name": "worker", "status": "operational" },
{ "name": "delivery", "status": "operational" },
{ "name": "sdk", "status": "operational" }
],
"incidents": [{ "id": "inc_…", "title": "…", "status": "investigating", "severity": "minor", "components": ["delivery"], "started_at": "…", "updates": [] }]
}A component's status can be operational, degraded (affected by an open incident) or major_outage (DB/Redis unreachable, no worker heartbeat); the overall status is determined by the worst component. Incidents are opened, updated and closed from the platform admin panel and stored in the platform_incidents table; for the process, see the incident runbook.
Health probes
| Endpoint | Purpose |
|---|---|
GET /healthz | Liveness: the process is up |
GET /readyz | Readiness: DB + Redis (+ storage) reachable; not exposed externally (nginx 404) |
GET / | Service info, version, documentation links |
Workers write heartbeats to Redis; the worker/delivery components in status.json are derived from these records.
Metrics (Prometheus /metrics)
| Metric | Description |
|---|---|
http_request_duration_seconds{route,status} | API latency |
queue_depth{queue}, queue_jobs_total{queue,state} | Queue backlog (including priority jobs) |
provider_send_total{provider,outcome}, provider_latency_seconds{provider} | Provider outcome and latency |
deliveries_total{channel,status} | Delivery outcomes |
webhook_deliveries_total{status}, webhook_delivery_latency_seconds, webhook_request_duration_seconds{outcome} | Outgoing webhooks |
messaging_feedback_total{source,type}, suppression_entries_total{channel,reason} | Email/SMS feedback |
active_subscriptions{app} | Hourly gauge |
db_pool_in_use | Connection pool |
dispatcher_go_requests_total{outcome} | If the Go dispatcher is used |
pushizo_backup_last_success_timestamp_seconds{kind} | Backup freshness (node-exporter textfile) |
analytics_store_mismatch_total{method,reason} | Differences in ClickHouse dual mode |
The Postgres exporter produces pushizo_* application metrics (channel status, pending/failed deliveries) through queries.
Alerts and dashboards
The Compose stack includes Prometheus + Alertmanager (email first, Slack optional) + Grafana (API, queue, provider and DB dashboards) + Loki/Promtail (pino JSON, secret masking) + blackbox exporter (HTTPS probes, certificate expiry; including www, grafana, storage). Key alerts: WorkerDown, RedisDown, ApiTransactionalLatencyHigh, queue backlog, provider error rate, PushizoBackupStale (> 26 hours), PushizoBackupMetricMissing, certificate expiry. For alert emails, fill in infra/compose/secrets/smtp_password.
Synthetic transactional monitoring
scripts/synthetic.ts runs an end-to-end check every 15 minutes (GitHub Actions synthetic.yml or your own scheduler): GET /healthz + GET /status.json → POST /v1/notifications (with an Idempotency-Key, to a single synthetic subscription) → acceptance time budget (default 1000 ms) → GET /v1/notifications/{id} checking sent + delivered ≥ 1 (default 60 s). Setup and the runbooks to follow on failure: synthetic monitoring. Go-live acceptance: 7 days (672 runs) without failure.
Error tracking
If SENTRY_DSN is set, API/worker errors are sent to Sentry (with body scrubbing in beforeSend); dashboard browser errors are forwarded through the same-origin tunnel /monitoring/client-error.
Related runbooks
Queue backlog · Provider outage · Load testing · Chaos drill · Real device testing
Runbook: Restoring from backup and disaster recovery (DR)
Targets (docs/plan/08 §8.10): RPO 1 hour, RTO 2 hours. Backup sources:
DMARC and TLS-RPT monitoring
Receiving DMARC aggregate reports and TLS-RPT reports for the platform domain (Postal HTTP endpoint or IMAP), parsing, unauthorized source alerts, policy recommendations and one-click p= changes with Cloudflare.