Cells (EU/TR data residency): setup, region moves, cleanup, DR
ADR-0032. A cell is a complete Pushizo installation with its own Postgres/Redis/ClickHouse/S3 and API/worker deployment. One cell (CONTROL_PLANE_MODE=true) is also the control plane: the org → cell mapping (org_regions),…
docs/adr/0032-hucre-mimarisi-white-label-bayi.md. A cell is a complete Pushizo installation with its own Postgres/Redis/ClickHouse/S3 and API/worker
deployment. One cell (CONTROL_PLANE_MODE=true) is also the control plane:
the org → cell mapping (org_regions), login routing and the global registry of custom domains (custom_domain_registry)
are kept there. If CELL_ID is empty, the installation is single-cell and none of the steps in this runbook are needed.
Configuration summary
| Variable | Where | Note |
|---|---|---|
CELL_ID | every cell | eu, tr … (lowercase letters/digits/hyphens) |
CELLS | every cell (same value) | JSON registry: id, name, api_url, dashboard_url, internal_url |
CONTROL_PLANE_MODE | true in exactly one cell | |
CONTROL_PLANE_URL, CONTROL_PLANE_DASHBOARD_URL | other cells | control plane internal API address / dashboard address |
CELL_INTERNAL_SECRET | every cell (same value, ≥ 32 characters) | secret: Secret / ExternalSecret; never logged |
CUSTOM_DOMAIN_CNAME_TARGET | every cell (cell-specific) | customer CNAME target, e.g. edge.eu.pushizo.com |
CUSTOM_DOMAIN_ASK_SECRET | every cell | secret; the edge layer's (Caddy) ask call |
Helm: each cell is a separate release in its own namespace — infra/k8s/pushizo/values-cell-eu.yaml (control plane) and
values-cell-tr.yaml. The cell.* values are written to the ConfigMap; CELL_INTERNAL_SECRET and CUSTOM_DOMAIN_ASK_SECRET
are read from a shared vault document (pushizo/cells) via externalSecrets.data. validate.yaml rejects inconsistent configuration
(a cell.id not in the registry, a missing control plane address, no secret mapping in a multi-cell setup, or, with the network
policy enabled, no cell.networkPolicy.peers entry for another cell).
Network: egress from API pods is opened only to the other cells' internal API address (cell.networkPolicy.peers, 443); there are no rules
to another cell's Postgres/Redis/ClickHouse/S3. This way, EU data cannot be written to the TR data layer at the network layer
either. At the application layer, CellGuard rejects a request for an org that belongs to another cell with 421 HS_WRONG_CELL
(proof test: apps/api/src/__tests__/phase21.int.test.ts, "EU verisi TR veritabanına yazılmaz").
Adding a new cell
- Set up the data layer in the region (managed Postgres 16, Redis 7, ClickHouse, S3 bucket); the application role must be
NOSUPERUSER NOBYPASSRLS(docs/runbooks/postgres-roles.md). - Vault document
pushizo/production-<cell>:DATABASE_URL,REDIS_URL,MASTER_KEY(cell-specific, generate a new one),SESSION_SECRET,S3_*,CLICKHOUSE_PASSWORD… values-cell-<cell>.yaml:cell.id,controlPlaneUrl, the registry (including the new cell),networkPolicy.peers,customDomains.cnameTarget, ingress hostnames.- In the values file of all existing cells, add the new cell to the
cell.registrylist and its internal address tonetworkPolicy.peers; deploy the existing cells first, then the new cell (the registry must be identical everywhere, otherwise signed calls get 401 withbilinmeyen hücre(unknown cell)). helm upgrade --install pushizo-<hücre> …→ the migration hook sets up the schema. ClickHouse:npm run ch:migrate.- Verification: on the control plane,
GET /internal/v1/cellslists the new region; the region can be selected when creating a new org in the dashboard; an org created in the new cell appears inorg_regionson the control plane.
Region move (platform admin operation)
Copies an org from one cell to another and verifies it; the source is cleaned up in a separate step. An org in a reseller hierarchy (a reseller or a sub-org) cannot be moved on its own.
Prerequisites: both cells are on the same migration version; workers are running (the ClickHouse outbox drains); a maintenance window has been
announced (the org is suspended during the move, requests get HS_ORG_SUSPENDED).
# With the source cell's environment (DATABASE_URL, MASTER_KEY, S3_*, CLICKHOUSE_* point to the source cell):
export CELL_DATABASE_URLS='{"tr":"<hedef Postgres bağlantısı>"}' # or REGION_MOVE_TARGET_DATABASE_URL
export CONTROL_PLANE_DATABASE_URL='<kontrol düzlemi Postgres>' # not needed if the source is the control plane
export REGION_MOVE_TARGET_MASTER_KEY='<hedef hücre MASTER_KEY>' # if different, encrypted values are rewrapped
export REGION_MOVE_TARGET_S3_ENDPOINT=… REGION_MOVE_TARGET_S3_BUCKET=… REGION_MOVE_TARGET_S3_REGION=… \
REGION_MOVE_TARGET_S3_ACCESS_KEY=… REGION_MOVE_TARGET_S3_SECRET_KEY=…
export REGION_MOVE_TARGET_CLICKHOUSE_URL=… REGION_MOVE_TARGET_CLICKHOUSE_DATABASE=… \
REGION_MOVE_TARGET_CLICKHOUSE_USER=… REGION_MOVE_TARGET_CLICKHOUSE_PASSWORD=…
npm run region:move -- --org=org_… --to=tr --report=tasima-org_….jsonFlow and scope:
- The source org is suspended (
suspended_reason = region_move), then the tool waits--settle-seconds(default 5). - If ClickHouse is enabled, it waits until the
analytics_outboxrows of the org's apps are drained (up to 60 s; if they do not drain, ship them withnpm run ch:drain --workspace @pushizo/dband try again). - Postgres: all org-scoped tables are copied to the target in a single transaction from one
REPEATABLE READsnapshot; each table is verified by row count + an order-independent md5.v1.encrypted values are re-encrypted with the target master key, and the data keys ofv2.envelope values are rewrapped on the target (retired). - ClickHouse:
deliveries,delivery_events,events,conversionsrows (FINAL, month by month, paginated) are copied to the target; row count +cityHash64sum are compared. If the target already has rows for these apps (an earlier move left half-finished), the operation is aborted. - Object storage:
org/<org>/(exports/imports, audit NDJSON),branding/<org>/(logo) and legacy-format export keys are copied, then read back on the target and verified with sha256. If source and target are the same store/bucket, the copy is skipped. Because the logo address is stored in the database as a cell-independent relative path, the target cell serves it from its own API root. - On any verification failure, the Postgres transaction is rolled back, the ClickHouse rows and objects written to the target are deleted, and the source org returns to its previous state.
- On success, the target org takes the previous state and
cell_id = <hedef>; on the control plane,org_regionsandcustom_domain_registrypoint to the new cell. The source org stays suspended and requests are redirected to the target region with 421.
If no target store is configured, --skip-objects must be passed explicitly; if ClickHouse is not being moved, --skip-clickhouse
(nothing is skipped silently).
After the move:
- Redis is not moved (quota counters, caches, rate limits and nonces are ephemeral): these counters start from zero
on the target. Persistent usage records (
usage_dailyetc.) are moved along with Postgres. - Account sessions are tied to the account and are not moved: users switch to the target region in the dashboard via session handoff (
/handoff). - Custom domains: the customer's CNAME must be updated to the new cell's
CUSTOM_DOMAIN_CNAME_TARGET; until it is updated, the old cell's edge layer will not issue a certificate (the domain is no longer there) — notify the customer before the move. - SDKs: the regional API address is the
api_urlin theGET /sdk/v1/apps/:appId/configresponse; a request to the old address gets 421 +details.api_url. The Web, Android and iOS SDKs adopt this address (unless a customapiUrlwas explicitly given ininit); for other bridges and server SDKs, the regional address must be set in the configuration.
Cleaning up the source (--purge-source)
Only after the move report has been reviewed and the org has run without problems in the target region for a few days:
npm run region:move -- --org=org_… --to=tr --purge-sourceThe primary key of every source row must exist on the target, every object must exist on the target with the same content (sha256), and the target must have at least as many ClickHouse rows as the source; if anything is missing, nothing is deleted. Deletion runs in reverse FK order in a single transaction; then the source ClickHouse rows (lightweight DELETE) and objects are deleted. Org data remains in the source database backups for the backup retention period (note this in your KVKK (Turkish data protection law)/GDPR records).
DR (per cell)
- Each cell's backups are kept in its own region (
docs/runbooks/restore.md); backups are not copied to another region (data residency). RTO/RPO targets are per cell (plan 10: Postgres RPO 1 s, ClickHouse 15 min). - Losing one cell does not affect the others; only that region's orgs are down. If the control plane cell
is lost, the other cells keep serving their existing orgs (location cache + local
cell_id), but new org creation, login routing and adding custom domains return 503 until the control plane is back. org_regionsconsistency after the control plane is restored: in each cell, compare the output ofselect id, cell_id from organizations where deleted_at is nullwithorg_regionson the control plane; rewrite missing rows into the control plane database by hand withinsert into org_regions (org_id, cell_id, name, slug, parent_org_id) … on conflict (org_id) do update.- Drills: at least twice a year, for each cell, a restore + a region move of a test org (keep the report file).
Troubleshooting
| Symptom | Cause / fix |
|---|---|
HS_WRONG_CELL (421) | The org is in another cell; details.api_url / dashboard_url is the correct address. Expected after a move |
/cell/v1/* 401 | Signature: clock difference > 60 s (NTP), different CELL_INTERNAL_SECRET, caller not in the registry, reused nonce |
| Regional org creation 503 | No access to the target cell's internal address (NetworkPolicy peers, DNS, internal load balancer) |
Custom domain stays pending | TXT _pushizo-challenge.<alan> or CNAME; the worker verify-custom-domains job retries every 5 min and marks it failed after 72 hours |
| Custom domain 409 | The domain is registered to another org (in any cell) (custom_domain_registry) |
Runbook: Transactional synthetic monitoring
The Phase 14 acceptance criterion "7 days of error-free synthetic monitoring on staging" is measured with this check. .github/workflows/synthetic.yml runs scripts/synthetic.ts every 15 minutes:
ClickHouse analytics store and Kafka ingest
ADR-0026. Both are optional: when they are not configured, the system keeps working as before with Postgres + BullMQ.