PushizoDocs
OperationsRunbooks

Cells (EU/TR data residency): setup, region moves, cleanup, DR

ADR-0032. A cell is a complete Pushizo installation with its own Postgres/Redis/ClickHouse/S3 and API/worker deployment. One cell (CONTROL_PLANE_MODE=true) is also the control plane: the org → cell mapping (org_regions),…

docs/adr/0032-hucre-mimarisi-white-label-bayi.md. A cell is a complete Pushizo installation with its own Postgres/Redis/ClickHouse/S3 and API/worker deployment. One cell (CONTROL_PLANE_MODE=true) is also the control plane: the org → cell mapping (org_regions), login routing and the global registry of custom domains (custom_domain_registry) are kept there. If CELL_ID is empty, the installation is single-cell and none of the steps in this runbook are needed.

Configuration summary

VariableWhereNote
CELL_IDevery celleu, tr … (lowercase letters/digits/hyphens)
CELLSevery cell (same value)JSON registry: id, name, api_url, dashboard_url, internal_url
CONTROL_PLANE_MODEtrue in exactly one cell
CONTROL_PLANE_URL, CONTROL_PLANE_DASHBOARD_URLother cellscontrol plane internal API address / dashboard address
CELL_INTERNAL_SECRETevery cell (same value, ≥ 32 characters)secret: Secret / ExternalSecret; never logged
CUSTOM_DOMAIN_CNAME_TARGETevery cell (cell-specific)customer CNAME target, e.g. edge.eu.pushizo.com
CUSTOM_DOMAIN_ASK_SECRETevery cellsecret; the edge layer's (Caddy) ask call

Helm: each cell is a separate release in its own namespace — infra/k8s/pushizo/values-cell-eu.yaml (control plane) and values-cell-tr.yaml. The cell.* values are written to the ConfigMap; CELL_INTERNAL_SECRET and CUSTOM_DOMAIN_ASK_SECRET are read from a shared vault document (pushizo/cells) via externalSecrets.data. validate.yaml rejects inconsistent configuration (a cell.id not in the registry, a missing control plane address, no secret mapping in a multi-cell setup, or, with the network policy enabled, no cell.networkPolicy.peers entry for another cell).

Network: egress from API pods is opened only to the other cells' internal API address (cell.networkPolicy.peers, 443); there are no rules to another cell's Postgres/Redis/ClickHouse/S3. This way, EU data cannot be written to the TR data layer at the network layer either. At the application layer, CellGuard rejects a request for an org that belongs to another cell with 421 HS_WRONG_CELL (proof test: apps/api/src/__tests__/phase21.int.test.ts, "EU verisi TR veritabanına yazılmaz").

Adding a new cell

  1. Set up the data layer in the region (managed Postgres 16, Redis 7, ClickHouse, S3 bucket); the application role must be NOSUPERUSER NOBYPASSRLS (docs/runbooks/postgres-roles.md).
  2. Vault document pushizo/production-<cell>: DATABASE_URL, REDIS_URL, MASTER_KEY (cell-specific, generate a new one), SESSION_SECRET, S3_*, CLICKHOUSE_PASSWORD …
  3. values-cell-<cell>.yaml: cell.id, controlPlaneUrl, the registry (including the new cell), networkPolicy.peers, customDomains.cnameTarget, ingress hostnames.
  4. In the values file of all existing cells, add the new cell to the cell.registry list and its internal address to networkPolicy.peers; deploy the existing cells first, then the new cell (the registry must be identical everywhere, otherwise signed calls get 401 with bilinmeyen hücre (unknown cell)).
  5. helm upgrade --install pushizo-<hücre> … → the migration hook sets up the schema. ClickHouse: npm run ch:migrate.
  6. Verification: on the control plane, GET /internal/v1/cells lists the new region; the region can be selected when creating a new org in the dashboard; an org created in the new cell appears in org_regions on the control plane.

Region move (platform admin operation)

Copies an org from one cell to another and verifies it; the source is cleaned up in a separate step. An org in a reseller hierarchy (a reseller or a sub-org) cannot be moved on its own.

Prerequisites: both cells are on the same migration version; workers are running (the ClickHouse outbox drains); a maintenance window has been announced (the org is suspended during the move, requests get HS_ORG_SUSPENDED).

# With the source cell's environment (DATABASE_URL, MASTER_KEY, S3_*, CLICKHOUSE_* point to the source cell):
export CELL_DATABASE_URLS='{"tr":"<hedef Postgres bağlantısı>"}'     # or REGION_MOVE_TARGET_DATABASE_URL
export CONTROL_PLANE_DATABASE_URL='<kontrol düzlemi Postgres>'        # not needed if the source is the control plane
export REGION_MOVE_TARGET_MASTER_KEY='<hedef hücre MASTER_KEY>'       # if different, encrypted values are rewrapped
export REGION_MOVE_TARGET_S3_ENDPOINT=… REGION_MOVE_TARGET_S3_BUCKET=… REGION_MOVE_TARGET_S3_REGION=… \
       REGION_MOVE_TARGET_S3_ACCESS_KEY=… REGION_MOVE_TARGET_S3_SECRET_KEY=…
export REGION_MOVE_TARGET_CLICKHOUSE_URL=… REGION_MOVE_TARGET_CLICKHOUSE_DATABASE=… \
       REGION_MOVE_TARGET_CLICKHOUSE_USER=… REGION_MOVE_TARGET_CLICKHOUSE_PASSWORD=…

npm run region:move -- --org=org_… --to=tr --report=tasima-org_….json

Flow and scope:

  1. The source org is suspended (suspended_reason = region_move), then the tool waits --settle-seconds (default 5).
  2. If ClickHouse is enabled, it waits until the analytics_outbox rows of the org's apps are drained (up to 60 s; if they do not drain, ship them with npm run ch:drain --workspace @pushizo/db and try again).
  3. Postgres: all org-scoped tables are copied to the target in a single transaction from one REPEATABLE READ snapshot; each table is verified by row count + an order-independent md5. v1. encrypted values are re-encrypted with the target master key, and the data keys of v2. envelope values are rewrapped on the target (retired).
  4. ClickHouse: deliveries, delivery_events, events, conversions rows (FINAL, month by month, paginated) are copied to the target; row count + cityHash64 sum are compared. If the target already has rows for these apps (an earlier move left half-finished), the operation is aborted.
  5. Object storage: org/<org>/ (exports/imports, audit NDJSON), branding/<org>/ (logo) and legacy-format export keys are copied, then read back on the target and verified with sha256. If source and target are the same store/bucket, the copy is skipped. Because the logo address is stored in the database as a cell-independent relative path, the target cell serves it from its own API root.
  6. On any verification failure, the Postgres transaction is rolled back, the ClickHouse rows and objects written to the target are deleted, and the source org returns to its previous state.
  7. On success, the target org takes the previous state and cell_id = <hedef>; on the control plane, org_regions and custom_domain_registry point to the new cell. The source org stays suspended and requests are redirected to the target region with 421.

If no target store is configured, --skip-objects must be passed explicitly; if ClickHouse is not being moved, --skip-clickhouse (nothing is skipped silently).

After the move:

  • Redis is not moved (quota counters, caches, rate limits and nonces are ephemeral): these counters start from zero on the target. Persistent usage records (usage_daily etc.) are moved along with Postgres.
  • Account sessions are tied to the account and are not moved: users switch to the target region in the dashboard via session handoff (/handoff).
  • Custom domains: the customer's CNAME must be updated to the new cell's CUSTOM_DOMAIN_CNAME_TARGET; until it is updated, the old cell's edge layer will not issue a certificate (the domain is no longer there) — notify the customer before the move.
  • SDKs: the regional API address is the api_url in the GET /sdk/v1/apps/:appId/config response; a request to the old address gets 421 + details.api_url. The Web, Android and iOS SDKs adopt this address (unless a custom apiUrl was explicitly given in init); for other bridges and server SDKs, the regional address must be set in the configuration.

Cleaning up the source (--purge-source)

Only after the move report has been reviewed and the org has run without problems in the target region for a few days:

npm run region:move -- --org=org_… --to=tr --purge-source

The primary key of every source row must exist on the target, every object must exist on the target with the same content (sha256), and the target must have at least as many ClickHouse rows as the source; if anything is missing, nothing is deleted. Deletion runs in reverse FK order in a single transaction; then the source ClickHouse rows (lightweight DELETE) and objects are deleted. Org data remains in the source database backups for the backup retention period (note this in your KVKK (Turkish data protection law)/GDPR records).

DR (per cell)

  • Each cell's backups are kept in its own region (docs/runbooks/restore.md); backups are not copied to another region (data residency). RTO/RPO targets are per cell (plan 10: Postgres RPO 1 s, ClickHouse 15 min).
  • Losing one cell does not affect the others; only that region's orgs are down. If the control plane cell is lost, the other cells keep serving their existing orgs (location cache + local cell_id), but new org creation, login routing and adding custom domains return 503 until the control plane is back.
  • org_regions consistency after the control plane is restored: in each cell, compare the output of select id, cell_id from organizations where deleted_at is null with org_regions on the control plane; rewrite missing rows into the control plane database by hand with insert into org_regions (org_id, cell_id, name, slug, parent_org_id) … on conflict (org_id) do update.
  • Drills: at least twice a year, for each cell, a restore + a region move of a test org (keep the report file).

Troubleshooting

SymptomCause / fix
HS_WRONG_CELL (421)The org is in another cell; details.api_url / dashboard_url is the correct address. Expected after a move
/cell/v1/* 401Signature: clock difference > 60 s (NTP), different CELL_INTERNAL_SECRET, caller not in the registry, reused nonce
Regional org creation 503No access to the target cell's internal address (NetworkPolicy peers, DNS, internal load balancer)
Custom domain stays pendingTXT _pushizo-challenge.<alan> or CNAME; the worker verify-custom-domains job retries every 5 min and marks it failed after 72 hours
Custom domain 409The domain is registered to another org (in any cell) (custom_domain_registry)

On this page