PushizoDocs
Operations

Status page and monitoring

status.json and the public status page, incident management, health probes, Prometheus metrics, alerts, Grafana dashboards, Loki logs and synthetic transactional monitoring.

Status page

  • Public: https://status.pushizo.com (the status subdomain on your own installation) shows component states and open incidents; visitors can subscribe by email.
  • Data endpoint: GET /status.json (no authentication) →
{
  "status": "operational",
  "checked_at": "2026-10-17T09:00:00Z",
  "components": [
    { "name": "api", "status": "operational" },
    { "name": "dashboard", "status": "operational" },
    { "name": "worker", "status": "operational" },
    { "name": "delivery", "status": "operational" },
    { "name": "sdk", "status": "operational" }
  ],
  "incidents": [{ "id": "inc_…", "title": "…", "status": "investigating", "severity": "minor", "components": ["delivery"], "started_at": "…", "updates": [] }]
}

A component's status can be operational, degraded (affected by an open incident) or major_outage (DB/Redis unreachable, no worker heartbeat); the overall status is determined by the worst component. Incidents are opened, updated and closed from the platform admin panel and stored in the platform_incidents table; for the process, see the incident runbook.

Health probes

EndpointPurpose
GET /healthzLiveness: the process is up
GET /readyzReadiness: DB + Redis (+ storage) reachable; not exposed externally (nginx 404)
GET /Service info, version, documentation links

Workers write heartbeats to Redis; the worker/delivery components in status.json are derived from these records.

Metrics (Prometheus /metrics)

MetricDescription
http_request_duration_seconds{route,status}API latency
queue_depth{queue}, queue_jobs_total{queue,state}Queue backlog (including priority jobs)
provider_send_total{provider,outcome}, provider_latency_seconds{provider}Provider outcome and latency
deliveries_total{channel,status}Delivery outcomes
webhook_deliveries_total{status}, webhook_delivery_latency_seconds, webhook_request_duration_seconds{outcome}Outgoing webhooks
messaging_feedback_total{source,type}, suppression_entries_total{channel,reason}Email/SMS feedback
active_subscriptions{app}Hourly gauge
db_pool_in_useConnection pool
dispatcher_go_requests_total{outcome}If the Go dispatcher is used
pushizo_backup_last_success_timestamp_seconds{kind}Backup freshness (node-exporter textfile)
analytics_store_mismatch_total{method,reason}Differences in ClickHouse dual mode

The Postgres exporter produces pushizo_* application metrics (channel status, pending/failed deliveries) through queries.

Alerts and dashboards

The Compose stack includes Prometheus + Alertmanager (email first, Slack optional) + Grafana (API, queue, provider and DB dashboards) + Loki/Promtail (pino JSON, secret masking) + blackbox exporter (HTTPS probes, certificate expiry; including www, grafana, storage). Key alerts: WorkerDown, RedisDown, ApiTransactionalLatencyHigh, queue backlog, provider error rate, PushizoBackupStale (> 26 hours), PushizoBackupMetricMissing, certificate expiry. For alert emails, fill in infra/compose/secrets/smtp_password.

Synthetic transactional monitoring

scripts/synthetic.ts runs an end-to-end check every 15 minutes (GitHub Actions synthetic.yml or your own scheduler): GET /healthz + GET /status.json → POST /v1/notifications (with an Idempotency-Key, to a single synthetic subscription) → acceptance time budget (default 1000 ms) → GET /v1/notifications/{id} checking sent + delivered ≥ 1 (default 60 s). Setup and the runbooks to follow on failure: synthetic monitoring. Go-live acceptance: 7 days (672 runs) without failure.

Error tracking

If SENTRY_DSN is set, API/worker errors are sent to Sentry (with body scrubbing in beforeSend); dashboard browser errors are forwarded through the same-origin tunnel /monitoring/client-error.

Related runbooks

Queue backlog · Provider outage · Load testing · Chaos drill · Real device testing

On this page