PushizoDocs
OperationsRunbooks

Runbook: Incident management and status page

Sources: Alertmanager (#pushizo-alerts, #pushizo-oncall, email), Grafana dashboards (infra/monitoring/grafana/dashboards), Loki logs (tracing by request_id), Sentry. Status page: https://status.<domain> →…

Sources: Alertmanager (#pushizo-alerts, #pushizo-oncall, email), Grafana dashboards (infra/monitoring/grafana/dashboards), Loki logs (tracing by request_id), Sentry. Status page: https://status.<alan adı> → dashboard /status, data GET /status.json (API; the platform_incidents table + component health: DB/Redis ping, worker heartbeats). Management: platform admin → /admin/incidents.

Severity

LevelDefinitionResponseStatus page
SEV1API/dashboard unreachable, risk of data loss, deliveries stoppedon-call engaged within 15 min, first update within 30 minmajor_outage
SEV2One channel/feature broken (e.g. email sending), severe delays30 mindegraded
SEV3Partial/cosmetic, single tenantbusiness hoursno incident (support ticket)

Flow

  1. Detection — an alert or a customer report. The on-call engineer takes ownership of the incident and opens a thread in Slack #pushizo-oncall ([SEV1] <title> — <time UTC>).
  2. Status page incident (SEV1/SEV2, within 10 min) — /admin/incidents → New incident: title, severity (minor|major|critical), affected components (api, dashboard, worker, delivery, sdk), first update (investigating). GET /status.json immediately returns degraded/major_outage; alerts such as ApiTransactionalLatencyHigh are inhibited (Alertmanager inhibit rules).
  3. Diagnosis — the relevant runbook: provider-outage.md, queue-backlog.md, restore.md, key-rotation.md. Standard questions:
    • When was the last deployment? (docker compose ps image tags / GitHub release) → if it is suspect, roll back: IMAGE_TAG=<önceki> ./infra/scripts/deploy.sh prod (migrations are backward compatible, so the previous image works with the new schema).
    • Which component? GET /readyz (DB/Redis), worker /readyz (heartbeat), the Grafana API and Database dashboards.
    • A single tenant? Loki | json | org_id="org_…".
  4. Update cadence — SEV1 every 30 min, SEV2 every 60 min: status page update (identified → monitoring) + Slack. Customer email (status page subscription) only for a SEV1 lasting more than 1 hour.
  5. Resolution — root cause fixed, metrics normal for 15 min → status page resolved (components operational). Alerts report resolved.
  6. Post-mortem (SEV1, within 5 business days) — docs/postmortems/YYYY-MM-DD-<slug>.md: timeline (UTC), impact (number of tenants, lost/delayed deliveries), root cause, what went well/what didn't, action items (owner + date). Blameless.

Ready-made commands

# overall status
curl -s https://api.<alan>/readyz | jq . ; curl -s https://api.<alan>/status.json | jq .components
docker compose --env-file .env.prod -f infra/compose/docker-compose.prod.yml ps
# logs (last 10 min, errors)
docker compose … logs --since 10m api-1 api-2 worker-dispatch worker-system | grep '"level":50'
# tracing a single request
# Loki: {project="pushizo"} | json | request_id="req_…"
# emergency capacity
docker compose … up -d --scale worker-dispatch=2
# rollback
IMAGE_TAG=<önceki sürüm> ./infra/scripts/deploy.sh prod

Communication templates

  • investigating: "We are investigating delays/errors in notification sending. Affected component: <channel>. Next update within 30 min."
  • identified: "The cause has been identified: <provider/infrastructure>. A fix is being applied; pending sends are queued and no data has been lost."
  • monitoring: "A fix has been applied and sending is back to normal; we are monitoring for 30 min."
  • resolved: "The incident is resolved. For <duration>, <impact>. A post-mortem will be shared."

Responsibilities

RoleDuties
On-callfirst response, status page, technical intervention
Incident manager (SEV1)communication cadence, decisions (rollback / DR), post-mortem owner
Platform admin/admin screens: incidents, tenant suspension, support session (read-only), feature flags (turning off a risky feature)

On this page