An RPO of 1 hour is achieved only when WAL archiving is enabled; with daily pg_dump alone, the RPO is 24 hours. In production, POSTGRES_IMAGE
must be the wal-g image (ghcr.io/exec0de/pushizo-postgres-walg:<version>, Debian based — same libc as the default postgres:16-bookworm)
and PG_ARCHIVE_COMMAND="wal-g-env wal-push %p" must be set; PITR additionally requires a daily base backup (walg-backup.sh) (GO_LIVE_CHECKLIST §5).
backup.sh and restore.sh access the off-server backup bucket with separate credentials: in .env.prod,
BACKUP_S3_ACCESS_KEY / BACKUP_S3_SECRET_KEY (optionally BACKUP_S3_REGION). This user must only have list/read/write/delete
permissions on the BACKUP_S3_BUCKET bucket; the application's S3_ACCESS_KEY service user is not used
(so backups can't be deleted if the application is compromised). If these are not defined, backup.sh stops with an explicit error.
Scheduling and monitoring:sudo bash infra/scripts/install-cron.sh (idempotent) writes /etc/cron.d/pushizo (03:30 backup.sh,
03:00 walg-backup.sh if wal-g is enabled, with flock overlap protection) and /etc/logrotate.d/pushizo; the log is
/var/log/pushizo-backup.log. Each successful step writes pushizo_backup_last_success_timestamp_seconds{kind="postgres"|"storage"}
under BACKUP_METRICS_DIR (default /var/lib/node_exporter/textfile); it is read by the node-exporter textfile
collector, and PushizoBackupStale (critical) fires after 26 hours, PushizoBackupMetricMissing if the metric is absent.
The scripts don't source .env.prod (safe line reading, infra/scripts/env-file.sh) and run the AWS CLI via the pinned
amazon/aws-cli image without installing it on the host (AWS_CLI_IMAGE). When you need a manual AWS CLI command,
use the same image (credentials are passed through the environment without writing values):
On a new or the same server, with the repo at /opt/pushizo and .env.prod ready:
# 1) Data layer (volumes are empty on a new server)docker compose --env-file .env.prod -f infra/compose/docker-compose.prod.yml up -d postgres redis storage storage-init# 2) Load the newest backup into a separate database and verify it (does NOT overwrite the production DB)./infra/scripts/restore.sh --latest --target pushizo_restore --jobs 4# report: migration count, org/app/subscription counts, last delivery/event time (RPO), durations (RTO)# 3) If verification passed, swap the production database (the broken DB is renamed, not dropped)BROKEN="pushizo_broken_$(date +%Y%m%d)"docker compose --env-file .env.prod -f infra/compose/docker-compose.prod.yml exec -T postgres \ psql -U pushizo -d postgres -v ON_ERROR_STOP=1 <<SQLselect pg_terminate_backend(pid) from pg_stat_activity where datname = 'pushizo' and pid <> pg_backend_pid();alter database pushizo rename to $BROKEN;alter database pushizo_restore rename to pushizo;SQL# 4) Apply any missing migrations, start the applicationdocker compose --env-file .env.prod -f infra/compose/docker-compose.prod.yml run --rm migrate upIMAGE_TAG=<çalışan sürüm> ./infra/scripts/deploy.sh prod
In cases such as an erroneous bulk delete, the last full backup + WAL return you to the moment just before the error:
docker compose … stop api-1 api-2 worker-dispatch worker-system# Fetch the newest full backup into an empty data directorydocker compose … run --rm --entrypoint sh -v postgres-restore:/restore postgres -c \ 'wal-g-env backup-fetch /restore LATEST'# recovery settings: target time (UTC) and WAL fetch commandcat > /tmp/recovery.conf <<'CONF'restore_command = 'wal-g-env wal-fetch %f %p'recovery_target_time = '2026-10-13 09:41:00+00'recovery_target_action = 'promote'CONF# Append to postgresql.auto.conf, create recovery.signal, attach the volume to the postgres service and start it; the log# should show "database system is ready" + "recovery stopping before commit". Then run the verification queries from step 3.
If Redis could not recover from AOF (volume loss), deliveries remain in the database as pending:
# from the worker image (DATABASE_URL/REDIS_URL are ready in the migrate/worker container)docker compose --env-file .env.prod -f infra/compose/docker-compose.prod.yml run --rm worker-dispatch \ node apps/worker/dist/cli/recover.js --dry-run # how many deliveries/jobs will be queueddocker compose --env-file .env.prod -f infra/compose/docker-compose.prod.yml run --rm worker-dispatch \ node apps/worker/dist/cli/recover.js # idempotent; jobId = the batch's first delivery ID# development environment: npm run queue:recover [-- --dry-run --app app_…]
The reconcile-deliveries maintenance job (every 10 minutes) already picks up anything missed; queue:recover re-queues everything without waiting.
Scheduler (BullMQ job scheduler) entries are recreated with upsertJobScheduler when the worker restarts.
Versioning is enabled: accidentally deleted export/import files are recovered from the previous version (S3 ListObjectVersions +
GetObject?versionId=, e.g. aws s3api list-object-versions / get-object --version-id; with the root credentials). Export files are kept
for 24 hours and import files for 7 days (cleanup-data-files); they are not persistent data — they can be regenerated.
Server loss / storage-data volume loss:backup.sh keeps an exact mirror of the bucket in the off-server bucket
under storage/<S3_BUCKET>/ (with --delete; to protect against deleted objects, enable versioning + a 30-day noncurrent
version retention rule on the off-server bucket). To restore (on the new server, with storage and storage-init running, as root):
cd /opt/pushizo && MIRROR=/var/backups/pushizo/storage-mirror && install -d -m 700 "$MIRROR"# 1) From the off-server bucket to the mirror (backup credentials; passed through the environment as above)docker run --rm -e AWS_ACCESS_KEY_ID -e AWS_SECRET_ACCESS_KEY -e AWS_DEFAULT_REGION -v "$MIRROR:/mirror" \ amazon/aws-cli:2.36.40 --endpoint-url https://s3.eu-central-1.amazonaws.com \ s3 sync s3://pushizo-backups/storage/pushizo/ /mirror --only-show-errors# 2) From the mirror to local RustFS (application service user S3_ACCESS_KEY/S3_SECRET_KEY from the environment; compose `app` network)docker run --rm --network pushizo_app -e AWS_ACCESS_KEY_ID -e AWS_SECRET_ACCESS_KEY -e AWS_DEFAULT_REGION \ -e AWS_REQUEST_CHECKSUM_CALCULATION=when_required -e AWS_RESPONSE_CHECKSUM_VALIDATION=when_required \ -v "$MIRROR:/mirror:ro" amazon/aws-cli:2.36.40 --endpoint-url http://storage:9000 \ s3 sync /mirror s3://pushizo --only-show-errors
In the second step, replace the AWS_ACCESS_KEY_ID/AWS_SECRET_ACCESS_KEY variables with S3_ACCESS_KEY/S3_SECRET_KEY
from .env.prod. If the mirror is still on the server (volume loss only), skip step 1.
v3 tables (migrations 0060–0068) live in Postgres; they come back with the §3/§4 restore and need no separate backup. After
the restore, apply the following checks and rebuild steps (ADR-0041):
MASTER_KEY must be the same (tokens are envelope encrypted). In the lost interval, some chunks were sent to the platform but not written to the DB: request a full reload for affected audiences (dashboard "Full reload" or full_reload_requested = true) — the platform deduplicates, so no missing removals remain.
Per account, "Refresh" (POST …/ads/accounts/{id}/refresh) or insights_next_sync_at = now(); the daily 28-day correction window fills in missing days. For older gaps, set insights_synced_at = null, insights_next_sync_at = now() (goes back ADS_INSIGHTS_BACKFILL_DAYS days, like the initial import). Exchange rates are filled by ads-fx-refresh (on startup).
Touches in the lost interval don't come back (the SDK resends touches still pending locally). CAPI: pending rows are resent after the lease period (10 minutes); platforms deduplicate by event_id.
Postgres; attachments in object storage under chat/<app>/…
Restore object storage with §6. An attachment that exists in the DB but not in storage returns 404 (the message remains). Presence fills in within 90 seconds from dashboard heartbeats.
Ad creatives
Object storage ads/<app>/creatives/…
Comes back with §6; otherwise re-upload in the wizard (media on the platform is unaffected).
ClickHouse ad_insights (rebuild; ClickHouse is not backed up, Postgres is the source):
# 1) Schema (including ad_insights; idempotent)npm run ch:migrate --workspace @pushizo/db# 2) Are the outbox triggers enabled (ch:outbox:enable is idempotent)npm run ch:outbox:enable --workspace @pushizo/db# 3) History: rows in Postgres are added to the outbox day by day and transferred (by the ad_insights `day` column; safe to repeat,# ReplacingMergeTree keeps the latest version). Start = the oldest spend day (the initial import goes back 30 days).npm run ch:backfill --workspace @pushizo/db -- --from=2026-01-01# 4) Verification: row counts must match in both storespsql "$DATABASE_URL" -c "select count(*) from ad_insights"curl -s "$CLICKHOUSE_URL/?query=SELECT%20count()%20FROM%20ad_insights%20FINAL" --user "$CLICKHOUSE_USER:$CLICKHOUSE_PASSWORD"
If Postgres is also lost and only an old backup exists: first restore Postgres, then re-fetch ad metrics from the platform
("Refresh" above), and finally run the ClickHouse backfill. During the transfer, reports can be read with ANALYTICS_STORE=postgres
(instant fallback). When moving regions (npm run region:move), ad_insights and the chat/ and ads/ prefixes are also
moved and deleted at the source with --purge-source.
Set the status page incident to resolved and summarize the customer impact (the lost time interval).
Record the RPO/RTO measurement in the table below; for an RTO over 2 hours, run a post-mortem (docs/runbooks/incident.md §Post-mortem).
If MASTER_KEY hasn't changed, the encrypted provider credentials remain valid; if you moved to a new server, .env.prod must be identical
(key loss = all provider credentials must be re-entered, see key-rotation.md).