PushizoDocs
Operations

Runbook: Restoring from backup and disaster recovery (DR)

Targets (docs/plan/08 §8.10): RPO 1 hour, RTO 2 hours. Backup sources:

Targets (docs/plan/08 §8.10): RPO 1 hour, RTO 2 hours. Backup sources:

DataMethodFrequencyRetentionTool
PostgreSQL (full)pg_dump -Fc → S3 postgres/daily 03:3030 daysinfra/scripts/backup.sh (install-cron.sh → /etc/cron.d/pushizo)
PostgreSQL (WAL)wal-g wal-push → S3 wal/continuous30 daysinfra/scripts/wal-g.env.example → .wal-g.env at the repo root, image pushizo-postgres-walg (release)
PostgreSQL (wal-g base backup)backup-push + delete retain FULL 30daily 03:0030 full backupsinfra/scripts/walg-backup.sh (cron)
RedisAOF (appendfsync everysec) + hourly RDBcontinuousvolumecompose redis service
RustFS/S3object versioning (local)continuous30 days (lifecycle)storage-init
RustFS (off-server)aws s3 sync bucket → local mirror (BACKUP_STORAGE_MIRROR_DIR) → S3 storage/<S3_BUCKET>/daily 03:30mirror (versioning in the off-server bucket)infra/scripts/backup.sh
Configuration.env.prod, infra/compose/secrets/*, TLS (certs volume)on changeSOPS/password vaultmanual

An RPO of 1 hour is achieved only when WAL archiving is enabled; with daily pg_dump alone, the RPO is 24 hours. In production, POSTGRES_IMAGE must be the wal-g image (ghcr.io/exec0de/pushizo-postgres-walg:<version>, Debian based — same libc as the default postgres:16-bookworm) and PG_ARCHIVE_COMMAND="wal-g-env wal-push %p" must be set; PITR additionally requires a daily base backup (walg-backup.sh) (GO_LIVE_CHECKLIST §5).

backup.sh and restore.sh access the off-server backup bucket with separate credentials: in .env.prod, BACKUP_S3_ACCESS_KEY / BACKUP_S3_SECRET_KEY (optionally BACKUP_S3_REGION). This user must only have list/read/write/delete permissions on the BACKUP_S3_BUCKET bucket; the application's S3_ACCESS_KEY service user is not used (so backups can't be deleted if the application is compromised). If these are not defined, backup.sh stops with an explicit error.

Scheduling and monitoring: sudo bash infra/scripts/install-cron.sh (idempotent) writes /etc/cron.d/pushizo (03:30 backup.sh, 03:00 walg-backup.sh if wal-g is enabled, with flock overlap protection) and /etc/logrotate.d/pushizo; the log is /var/log/pushizo-backup.log. Each successful step writes pushizo_backup_last_success_timestamp_seconds{kind="postgres"|"storage"} under BACKUP_METRICS_DIR (default /var/lib/node_exporter/textfile); it is read by the node-exporter textfile collector, and PushizoBackupStale (critical) fires after 26 hours, PushizoBackupMetricMissing if the metric is absent. The scripts don't source .env.prod (safe line reading, infra/scripts/env-file.sh) and run the AWS CLI via the pinned amazon/aws-cli image without installing it on the host (AWS_CLI_IMAGE). When you need a manual AWS CLI command, use the same image (credentials are passed through the environment without writing values):

export AWS_ACCESS_KEY_ID="$(grep -E '^BACKUP_S3_ACCESS_KEY=' .env.prod | tail -n1 | cut -d= -f2-)"
export AWS_SECRET_ACCESS_KEY="$(grep -E '^BACKUP_S3_SECRET_KEY=' .env.prod | tail -n1 | cut -d= -f2-)"
export AWS_DEFAULT_REGION=eu-central-1
docker run --rm -e AWS_ACCESS_KEY_ID -e AWS_SECRET_ACCESS_KEY -e AWS_DEFAULT_REGION amazon/aws-cli:2.36.40 \
  --endpoint-url https://s3.eu-central-1.amazonaws.com s3 ls s3://pushizo-backups/postgres/

1. When?

  • Data loss / corruption (a bad migration, an erroneous bulk delete, disk failure).
  • Server loss (moving to a new server).
  • Weekly automated drill (infra/scripts/verify-restore.sh, staging, Sunday 04:00) and a full manual drill every quarter.

2. Incident start (T+0)

  1. Start the docs/runbooks/incident.md process: open an incident on the status page (/admin/incidents, components api, delivery), Slack #pushizo-oncall.
  2. Stop write traffic: docker compose … stop nginx worker-dispatch worker-system (API down → SDKs queue offline, server SDKs get 5xx and retry).
  3. Note the outage time (UTC): T_outage. RPO = T_outage − time of last backup/WAL.

3. Option A — Restore from a full backup (pg_dump)

On a new or the same server, with the repo at /opt/pushizo and .env.prod ready:

# 1) Data layer (volumes are empty on a new server)
docker compose --env-file .env.prod -f infra/compose/docker-compose.prod.yml up -d postgres redis storage storage-init

# 2) Load the newest backup into a separate database and verify it (does NOT overwrite the production DB)
./infra/scripts/restore.sh --latest --target pushizo_restore --jobs 4
#    report: migration count, org/app/subscription counts, last delivery/event time (RPO), durations (RTO)

# 3) If verification passed, swap the production database (the broken DB is renamed, not dropped)
BROKEN="pushizo_broken_$(date +%Y%m%d)"
docker compose --env-file .env.prod -f infra/compose/docker-compose.prod.yml exec -T postgres \
  psql -U pushizo -d postgres -v ON_ERROR_STOP=1 <<SQL
select pg_terminate_backend(pid) from pg_stat_activity where datname = 'pushizo' and pid <> pg_backend_pid();
alter database pushizo rename to $BROKEN;
alter database pushizo_restore rename to pushizo;
SQL

# 4) Apply any missing migrations, start the application
docker compose --env-file .env.prod -f infra/compose/docker-compose.prod.yml run --rm migrate up
IMAGE_TAG=<çalışan sürüm> ./infra/scripts/deploy.sh prod

4. Option B — Point-in-time recovery with WAL (PITR, wal-g)

In cases such as an erroneous bulk delete, the last full backup + WAL return you to the moment just before the error:

docker compose … stop api-1 api-2 worker-dispatch worker-system
# Fetch the newest full backup into an empty data directory
docker compose … run --rm --entrypoint sh -v postgres-restore:/restore postgres -c \
  'wal-g-env backup-fetch /restore LATEST'
# recovery settings: target time (UTC) and WAL fetch command
cat > /tmp/recovery.conf <<'CONF'
restore_command = 'wal-g-env wal-fetch %f %p'
recovery_target_time = '2026-10-13 09:41:00+00'
recovery_target_action = 'promote'
CONF
# Append to postgresql.auto.conf, create recovery.signal, attach the volume to the postgres service and start it; the log
# should show "database system is ready" + "recovery stopping before commit". Then run the verification queries from step 3.

Detailed wal-g PITR steps: https://github.com/wal-g/wal-g/blob/master/docs/PostgreSQL.md#backup-fetch

5. Redis loss (queues)

If Redis could not recover from AOF (volume loss), deliveries remain in the database as pending:

# from the worker image (DATABASE_URL/REDIS_URL are ready in the migrate/worker container)
docker compose --env-file .env.prod -f infra/compose/docker-compose.prod.yml run --rm worker-dispatch \
  node apps/worker/dist/cli/recover.js --dry-run       # how many deliveries/jobs will be queued
docker compose --env-file .env.prod -f infra/compose/docker-compose.prod.yml run --rm worker-dispatch \
  node apps/worker/dist/cli/recover.js                 # idempotent; jobId = the batch's first delivery ID
# development environment: npm run queue:recover [-- --dry-run --app app_…]

The reconcile-deliveries maintenance job (every 10 minutes) already picks up anything missed; queue:recover re-queues everything without waiting. Scheduler (BullMQ job scheduler) entries are recreated with upsertJobScheduler when the worker restarts.

6. Object storage (RustFS/S3)

Versioning is enabled: accidentally deleted export/import files are recovered from the previous version (S3 ListObjectVersions + GetObject?versionId=, e.g. aws s3api list-object-versions / get-object --version-id; with the root credentials). Export files are kept for 24 hours and import files for 7 days (cleanup-data-files); they are not persistent data — they can be regenerated.

Server loss / storage-data volume loss: backup.sh keeps an exact mirror of the bucket in the off-server bucket under storage/<S3_BUCKET>/ (with --delete; to protect against deleted objects, enable versioning + a 30-day noncurrent version retention rule on the off-server bucket). To restore (on the new server, with storage and storage-init running, as root):

cd /opt/pushizo && MIRROR=/var/backups/pushizo/storage-mirror && install -d -m 700 "$MIRROR"
# 1) From the off-server bucket to the mirror (backup credentials; passed through the environment as above)
docker run --rm -e AWS_ACCESS_KEY_ID -e AWS_SECRET_ACCESS_KEY -e AWS_DEFAULT_REGION -v "$MIRROR:/mirror" \
  amazon/aws-cli:2.36.40 --endpoint-url https://s3.eu-central-1.amazonaws.com \
  s3 sync s3://pushizo-backups/storage/pushizo/ /mirror --only-show-errors
# 2) From the mirror to local RustFS (application service user S3_ACCESS_KEY/S3_SECRET_KEY from the environment; compose `app` network)
docker run --rm --network pushizo_app -e AWS_ACCESS_KEY_ID -e AWS_SECRET_ACCESS_KEY -e AWS_DEFAULT_REGION \
  -e AWS_REQUEST_CHECKSUM_CALCULATION=when_required -e AWS_RESPONSE_CHECKSUM_VALIDATION=when_required \
  -v "$MIRROR:/mirror:ro" amazon/aws-cli:2.36.40 --endpoint-url http://storage:9000 \
  s3 sync /mirror s3://pushizo --only-show-errors

In the second step, replace the AWS_ACCESS_KEY_ID/AWS_SECRET_ACCESS_KEY variables with S3_ACCESS_KEY/S3_SECRET_KEY from .env.prod. If the mirror is still on the server (volume loss only), skip step 1.

6a. v3 data (ads, attribution, web growth, live chat) and ClickHouse ad_insights

v3 tables (migrations 0060–0068) live in Postgres; they come back with the §3/§4 restore and need no separate backup. After the restore, apply the following checks and rebuild steps (ADR-0041):

DataSourceAfter restore
ad_accounts (encrypted tokens), ad_audiences, ad_audience_members_state, ad_audience_syncsPostgresMASTER_KEY must be the same (tokens are envelope encrypted). In the lost interval, some chunks were sent to the platform but not written to the DB: request a full reload for affected audiences (dashboard "Full reload" or full_reload_requested = true) — the platform deduplicates, so no missing removals remain.
ad_campaigns_cache, ad_insights (Postgres), fx_ratesPostgres; can be re-fetched from the platformPer account, "Refresh" (POST …/ads/accounts/{id}/refresh) or insights_next_sync_at = now(); the daily 28-day correction window fills in missing days. For older gaps, set insights_synced_at = null, insights_next_sync_at = now() (goes back ADS_INSIGHTS_BACKFILL_DAYS days, like the initial import). Exchange rates are filled by ads-fx-refresh (on startup).
attribution_touches, conversion_api_deliveries, conversion_api_configsPostgresTouches in the lost interval don't come back (the SDK resends touches still pending locally). CAPI: pending rows are resent after the lease period (10 minutes); platforms deduplicate by event_id.
forms, form_submissions, surveys, survey_responses, web_campaigns, web_campaign_events, landing_pages, landing_page_versionsPostgresThe landing HTML cache lives in Redis; if Redis is empty, it is regenerated from the published version on first visit (no extra action).
conversations, conversation_messages, chat_visitors, inbox_*, canned_responses, assignment_rules, agent_presencePostgres; attachments in object storage under chat/<app>/…Restore object storage with §6. An attachment that exists in the DB but not in storage returns 404 (the message remains). Presence fills in within 90 seconds from dashboard heartbeats.
Ad creativesObject storage ads/<app>/creatives/…Comes back with §6; otherwise re-upload in the wizard (media on the platform is unaffected).

ClickHouse ad_insights (rebuild; ClickHouse is not backed up, Postgres is the source):

# 1) Schema (including ad_insights; idempotent)
npm run ch:migrate --workspace @pushizo/db
# 2) Are the outbox triggers enabled (ch:outbox:enable is idempotent)
npm run ch:outbox:enable --workspace @pushizo/db
# 3) History: rows in Postgres are added to the outbox day by day and transferred (by the ad_insights `day` column; safe to repeat,
#    ReplacingMergeTree keeps the latest version). Start = the oldest spend day (the initial import goes back 30 days).
npm run ch:backfill --workspace @pushizo/db -- --from=2026-01-01
# 4) Verification: row counts must match in both stores
psql "$DATABASE_URL" -c "select count(*) from ad_insights"
curl -s "$CLICKHOUSE_URL/?query=SELECT%20count()%20FROM%20ad_insights%20FINAL" --user "$CLICKHOUSE_USER:$CLICKHOUSE_PASSWORD"

If Postgres is also lost and only an old backup exists: first restore Postgres, then re-fetch ad metrics from the platform ("Refresh" above), and finally run the ClickHouse backfill. During the transfer, reports can be read with ANALYTICS_STORE=postgres (instant fallback). When moving regions (npm run region:move), ad_insights and the chat/ and ads/ prefixes are also moved and deleted at the source with --purge-source.

7. Closing

  • Set the status page incident to resolved and summarize the customer impact (the lost time interval).
  • Record the RPO/RTO measurement in the table below; for an RTO over 2 hours, run a post-mortem (docs/runbooks/incident.md §Post-mortem).
  • If MASTER_KEY hasn't changed, the encrypted provider credentials remain valid; if you moved to a new server, .env.prod must be identical (key loss = all provider credentials must be re-entered, see key-rotation.md).

8. Drill log

verify-restore.sh appends a row to /var/log/pushizo-restore-drills.tsv every week; quarterly manual drills are recorded here.

DateEnvironmentBackup (key)MethodRPO (measured)RTO (measured)ResultNotes / action
stagingpg_dump / PITR

On this page