Runbook: Incident management and status page
Sources: Alertmanager (#pushizo-alerts, #pushizo-oncall, email), Grafana dashboards (infra/monitoring/grafana/dashboards), Loki logs (tracing by request_id), Sentry. Status page: https://status.<domain> →…
Sources: Alertmanager (#pushizo-alerts, #pushizo-oncall, email), Grafana dashboards (infra/monitoring/grafana/dashboards),
Loki logs (tracing by request_id), Sentry. Status page: https://status.<alan adı> → dashboard /status, data GET /status.json
(API; the platform_incidents table + component health: DB/Redis ping, worker heartbeats). Management: platform admin → /admin/incidents.
Severity
| Level | Definition | Response | Status page |
|---|---|---|---|
| SEV1 | API/dashboard unreachable, risk of data loss, deliveries stopped | on-call engaged within 15 min, first update within 30 min | major_outage |
| SEV2 | One channel/feature broken (e.g. email sending), severe delays | 30 min | degraded |
| SEV3 | Partial/cosmetic, single tenant | business hours | no incident (support ticket) |
Flow
- Detection — an alert or a customer report. The on-call engineer takes ownership of the incident and opens a thread in Slack
#pushizo-oncall([SEV1] <title> — <time UTC>). - Status page incident (SEV1/SEV2, within 10 min) —
/admin/incidents→ New incident: title, severity (minor|major|critical), affected components (api,dashboard,worker,delivery,sdk), first update (investigating).GET /status.jsonimmediately returnsdegraded/major_outage; alerts such asApiTransactionalLatencyHighare inhibited (Alertmanager inhibit rules). - Diagnosis — the relevant runbook:
provider-outage.md,queue-backlog.md,restore.md,key-rotation.md. Standard questions:- When was the last deployment? (
docker compose psimage tags / GitHub release) → if it is suspect, roll back:IMAGE_TAG=<önceki> ./infra/scripts/deploy.sh prod(migrations are backward compatible, so the previous image works with the new schema). - Which component?
GET /readyz(DB/Redis), worker/readyz(heartbeat), the Grafana API and Database dashboards. - A single tenant? Loki
| json | org_id="org_…".
- When was the last deployment? (
- Update cadence — SEV1 every 30 min, SEV2 every 60 min: status page update (
identified→monitoring) + Slack. Customer email (status page subscription) only for a SEV1 lasting more than 1 hour. - Resolution — root cause fixed, metrics normal for 15 min → status page
resolved(componentsoperational). Alerts reportresolved. - Post-mortem (SEV1, within 5 business days) —
docs/postmortems/YYYY-MM-DD-<slug>.md: timeline (UTC), impact (number of tenants, lost/delayed deliveries), root cause, what went well/what didn't, action items (owner + date). Blameless.
Ready-made commands
# overall status
curl -s https://api.<alan>/readyz | jq . ; curl -s https://api.<alan>/status.json | jq .components
docker compose --env-file .env.prod -f infra/compose/docker-compose.prod.yml ps
# logs (last 10 min, errors)
docker compose … logs --since 10m api-1 api-2 worker-dispatch worker-system | grep '"level":50'
# tracing a single request
# Loki: {project="pushizo"} | json | request_id="req_…"
# emergency capacity
docker compose … up -d --scale worker-dispatch=2
# rollback
IMAGE_TAG=<önceki sürüm> ./infra/scripts/deploy.sh prodCommunication templates
- investigating: "We are investigating delays/errors in notification sending. Affected component: <channel>. Next update within 30 min."
- identified: "The cause has been identified: <provider/infrastructure>. A fix is being applied; pending sends are queued and no data has been lost."
- monitoring: "A fix has been applied and sending is back to normal; we are monitoring for 30 min."
- resolved: "The incident is resolved. For <duration>, <impact>. A post-mortem will be shared."
Responsibilities
| Role | Duties |
|---|---|
| On-call | first response, status page, technical intervention |
| Incident manager (SEV1) | communication cadence, decisions (rollback / DR), post-mortem owner |
| Platform admin | /admin screens: incidents, tenant suspension, support session (read-only), feature flags (turning off a risky feature) |
Marketing site, SEO and AI SEO: pushizo.com launch steps
Pushizo's marketing site (apps/web, https://pushizo.com), developer documentation (apps/docs-site, https://docs.pushizo.com) and customer dashboard (apps/dashboard, https://app.pushizo.com; API https://api.pushizo.com)…
Runbook: Provider outage (APNs / FCM / Web Push / SES / Resend / Twilio / Netgsm)
If you need to pause a single channel temporarily (e.g. the provider is returning false positives/resending): remove the relevant queue from the worker-dispatch service's WORKER_QUEUES value…