Production deployment with Helm (Kubernetes)
v2 Phase 16 item 4 (docs/plan/10). Chart: infra/k8s/pushizo. Example values: values-production.yaml, values-staging.yaml. Validation: values.schema.json (single field) + templates/validate.yaml (cross-field rules,…
v2 Phase 16 item 4 (docs/plan/10). Chart: infra/k8s/pushizo. Example values: values-production.yaml, values-staging.yaml.
Validation: values.schema.json (single field) + templates/validate.yaml (cross-field rules, meaningful fail messages).
1. Prerequisites
| Component | Why | Notes |
|---|---|---|
| Kubernetes ≥ 1.27 | matchLabelKeys (topology spread), policy/v1 PDB, autoscaling/v2 HPA | Chart.yaml kubeVersion |
| ingress-nginx + cert-manager | Ingress, TLS (ingress.clusterIssuer) | server-snippet is required for /metrics and /readyz (snippets must be allowed), or ingress.blockInternalPaths=false + some other blocking |
| metrics-server | API HPA (CPU/memory) | |
| External Secrets Operator | externalSecrets.enabled | external-secrets.io/v1 (ESO ≥ 0.17); on older installations externalSecrets.apiVersion: external-secrets.io/v1beta1 |
| KEDA ≥ 2.12 | keda.enabled (worker scaling) | fallback and advanced.horizontalPodAutoscalerConfig are used |
| Prometheus Operator | podMonitor / serviceMonitor, KEDA query | kube-prometheus-stack recommended |
| A CNI that enforces NetworkPolicy | networkPolicy.enabled | Calico, Cilium, AWS VPC CNI (with network policy enabled), etc. |
| Managed Postgres 16, Redis 7, S3 | the data layer is outside the chart | application role NOSUPERUSER NOBYPASSRLS (docs/runbooks/postgres-roles.md) |
The namespace is compatible with the Pod Security Admission restricted profile (runAsNonRoot, seccomp RuntimeDefault, drop: [ALL],
allowPrivilegeEscalation: false, read-only root filesystem; writable directories are emptyDir only):
kubectl create ns pushizo
kubectl label ns pushizo pod-security.kubernetes.io/enforce=restricted pod-security.kubernetes.io/enforce-version=latest2. First installation
NS=pushizo
# (a) Managed Postgres/Redis CA bundle (e.g. AWS RDS: global-bundle.pem). Before installing, since the migration hook mounts it too.
kubectl -n $NS create configmap pushizo-ca-bundle --from-file=ca.pem=./global-bundle.pem
# (b) Secrets: ExternalSecrets (§3) or a manual Secret (keys use the packages/config names)
kubectl -n $NS create secret generic pushizo-secrets \
--from-literal=DATABASE_URL="$DATABASE_URL" --from-literal=REDIS_URL="$REDIS_URL" \
--from-literal=MASTER_KEY="$MASTER_KEY" --from-literal=SESSION_SECRET="$SESSION_SECRET" \
--from-literal=S3_ACCESS_KEY="$S3_ACCESS_KEY" --from-literal=S3_SECRET_KEY="$S3_SECRET_KEY" --from-literal=SMTP_URL="$SMTP_URL"
# (c) Environment values: copy values-production.yaml and edit the domains, CIDRs and store name (don't write secret values)
helm lint infra/k8s/pushizo -f infra/k8s/pushizo/values-production.yaml -f prod.values.yaml
helm template pushizo infra/k8s/pushizo -n $NS -f infra/k8s/pushizo/values-production.yaml -f prod.values.yaml \
--set image.tag=0.1.0 | kubectl apply --dry-run=server -f -
helm upgrade --install pushizo infra/k8s/pushizo -n $NS \
-f infra/k8s/pushizo/values-production.yaml -f prod.values.yaml --set image.tag=0.1.0 --wait --timeout 15mInstallation order (Helm hooks): ExternalSecret/Secret hook (weight −10) → migrate Job (weight 0, pre-install,pre-upgrade)
→ release resources. The migrate Job relies only on resources that exist before it: the Secret, the CA ConfigMap and the namespace's
default ServiceAccount (no token is mounted). The Job doesn't read the ConfigMap; the migration CLI only uses DATABASE_URL.
Checks:
kubectl -n $NS get pods,pdb,hpa,scaledobject,networkpolicy -l app.kubernetes.io/instance=pushizo
kubectl -n $NS logs job/pushizo-migrate # before it is deleted by hook-succeeded; on failure the Job remains
kubectl -n $NS port-forward svc/pushizo-api 4000:4000 & curl -fsS localhost:4000/readyz3. ExternalSecrets
- The chart doesn't create a SecretStore/ClusterSecretStore; the platform team defines the store with its identity (IRSA, GKE Workload Identity,
Vault Kubernetes auth), and the chart references it via
externalSecrets.secretStoreRef. - The document in the vault uses the packages/config key names: required
DATABASE_URL,REDIS_URL,MASTER_KEY,SESSION_SECRET,S3_ACCESS_KEY,S3_SECRET_KEY,SMTP_URL; optionalSTRIPE_SECRET_KEY,STRIPE_WEBHOOK_SECRET,SENTRY_DSN,GOOGLE_OAUTH_CLIENT_SECRET,GITHUB_OAUTH_CLIENT_SECRET,IYZICO_*;DATABASE_POOLED_URLif PgBouncer is enabled. - Bulk mapping:
dataFrom: [{ extract: { key: pushizo/production } }]; individual mapping:data: [{ secretKey, remoteRef }]. - The ExternalSecret is a
pre-install,pre-upgradehook (so the Secret exists before the migrate Job). Thanks tocreationPolicy: Orphan, the target Secret isn't deleted when the hook is recreated on every upgrade;helm uninstallleaves the Secret in place (deliberately: data layer credentials outlive the release). To delete it manually:kubectl -n $NS delete secret pushizo-secrets. - The chart rejects keys containing secret values in
config(ConfigMap) (config.MASTER_KEY gizli bir değerdir ...). - Rotation: change the value in the vault → ESO updates the Secret within
refreshInterval(default 1 h) → runkubectl -n $NS rollout restart deploy -l app.kubernetes.io/instance=pushizoso pods re-read their environment variables.MASTER_KEYrotation is a separate procedure: docs/runbooks/key-rotation.md. secrets.create=trueis for development only (values via--set); it cannot be enabled together withexternalSecrets.enabled.
4. Blue/green API deployment
When enabled, there are two Deployments: pushizo-api-blue and pushizo-api-green. The pushizo-api Service (and the api alias
used by the dashboard) selects only the api.blueGreen.activeColor color; pushizo-api-preview points to the passive color.
The HPA is attached only to the active color. The worker and dashboard use the image.tag tag with rolling updates.
Rules
- In blue/green mode, always set both colors' tags explicitly (
api.blueGreen.blue.imageTag,green.imageTag). An empty tag followsimage.tag; whenimage.tagchanges, the passive color also moves to the new version and you lose the rollback option. - If a color's
replicasfield isnull, the existing replica count in the cluster is kept (lookup); on first install the active color getsapi.replicasand the passive color 0. With tools wherelookupdoesn't work (helm template,--dry-run=client, Argo CD), setreplicasexplicitly for both colors; otherwise the passive color is pulled to 0 and the active color toapi.replicas. - Migrations must always be backward compatible: in the preparation step, the migrate hook runs with the new version's image (
migrate.imageTag), and the old color keeps running against the new schema (docs/plan/08 §8.8: add first, then use, then drop). On rollback,migrate.imageTagalso stays on the new version (the old migrator ignores applied versions it doesn't know and doesn't roll back the schema).
Switchover (blue active; old version $OLD, new version $NEW)
OLD=0.1.0 NEW=0.2.0
H="helm upgrade pushizo infra/k8s/pushizo -n $NS -f infra/k8s/pushizo/values-production.yaml -f prod.values.yaml --wait --timeout 15m"
BLUE_N=$(kubectl -n $NS get deploy pushizo-api-blue -o jsonpath='{.spec.replicas}')
# 1) Preparation: bring up green with $NEW at active capacity (the migrate hook runs in this step). Traffic is still on blue.
$H --set image.tag=$OLD --set api.blueGreen.activeColor=blue \
--set api.blueGreen.blue.imageTag=$OLD --set api.blueGreen.green.imageTag=$NEW --set migrate.imageTag=$NEW --set api.blueGreen.green.replicas=$BLUE_N
kubectl -n $NS rollout status deploy/pushizo-api-green
# 2) Smoke test against the passive color (port-forward is not affected by NetworkPolicy)
kubectl -n $NS port-forward svc/pushizo-api-preview 4100:4000 &
curl -fsS localhost:4100/healthz && curl -fsS localhost:4100/readyz
curl -fsS -H "Authorization: Bearer $SMOKE_API_KEY" "localhost:4100/v1/apps/$SMOKE_APP_ID/channels"
# 3) Switch: the Service selector moves to green (instantly); blue replicas are kept (replicas=null → lookup). Worker/dashboard move to $NEW.
$H --set image.tag=$NEW --set api.blueGreen.activeColor=green \
--set api.blueGreen.blue.imageTag=$OLD --set api.blueGreen.green.imageTag=$NEW --set migrate.imageTag=$NEW
kubectl -n $NS get svc pushizo-api -o jsonpath='{.spec.selector}'; echo
# 4) Observation (at least 30 min): 5xx rate, p95, queue depth (Grafana API/Queues dashboards), Sentry.
# 5a) Rollback (if needed): traffic returns to blue instantly; blue is still up and on $OLD.
$H --set image.tag=$OLD --set api.blueGreen.activeColor=blue \
--set api.blueGreen.blue.imageTag=$OLD --set api.blueGreen.green.imageTag=$NEW --set migrate.imageTag=$NEW
# 5b) Finalize: if observation is clean, scale down the old color (the next release is prepared on blue).
$H --set image.tag=$NEW --set api.blueGreen.activeColor=green \
--set api.blueGreen.blue.imageTag=$OLD --set api.blueGreen.green.imageTag=$NEW --set migrate.imageTag=$NEW --set api.blueGreen.blue.replicas=0On rollback the schema is not rolled back (the new schema is compatible with the old code). If the migration itself is faulty, first check the backup as described in docs/runbooks/restore.md, then roll back the last migration (the connection string is read from the Secret, not written on the command line):
kubectl -n $NS run migrate-down --rm -i --restart=Never --image=ghcr.io/exec0de/pushizo-migrate:$NEW --overrides='{
"spec": {
"securityContext": { "runAsNonRoot": true, "runAsUser": 1001, "seccompProfile": { "type": "RuntimeDefault" } },
"containers": [{
"name": "migrate-down", "image": "ghcr.io/exec0de/pushizo-migrate:'"$NEW"'", "args": ["down", "--steps=1"],
"env": [{ "name": "DATABASE_URL", "valueFrom": { "secretKeyRef": { "name": "pushizo-secrets", "key": "DATABASE_URL" } } }],
"securityContext": { "allowPrivilegeEscalation": false, "capabilities": { "drop": ["ALL"] } }
}]
}
}'Zero-downtime migration from rolling updates to blue/green
api.blueGreen.enabled=true,transitionFromRolling=true,activeColor=blue,blue.imageTag=<current>,blue.replicas=<current replicas>: the oldpushizo-apiDeployment stays andpushizo-api-blueis added; the Service selects all API pods.kubectl rollout status deploy/pushizo-api-blue.transitionFromRolling=false: the Service selects only blue; the old Deployment, PDB and HPA are deleted. The reverse direction (from blue/green back to rolling updates) uses the same two steps: firsttransitionFromRolling=true, thenenabled=false.
Note: dashboard.apiHostAlias (default api) is a single Service name per namespace; if two releases run in the same namespace,
leave it empty in one of them.
5. KEDA (worker scaling)
- One
ScaledObjectper group:sum(max by (queue, state) (queue_depth{queue=~"<regex>",state=~"waiting|prioritized",namespace="<ns>"})).queue_depth{queue,state}is written by the worker every 15 seconds (apps/worker/src/registry.ts); since priority jobs live in BullMQ'sprioritizedset, both states are summed.max byremoves duplicates from multiple pods publishing the same queue. thresholdis the target number of waiting jobs per replica: desired replicas = ⌈depth / threshold⌉ (within the min/max bounds). Starting value: for dispatch, 1 minute of processable work givenWORKER_CONCURRENCY × job duration; tune it after measuring.minReplicaCount ≥ 1is required (schema): the metric comes only from the worker consuming the queue, so at 0 replicas the metric disappears.keda.namespaceLabel: Prometheus Operator adds thenamespacelabel. If the label differs on a Prometheus with manually configured pod discovery (e.g.kubernetes_namespace), change it; an empty value removes the restriction.fallback: if Prometheus fails to respond to 3 consecutive queries, the group falls back to a fixed replica count (keda.fallbackReplicasorhpa.minReplicas). No fallback is generated whilekeda.cpuTrigger.enabled=true.- When KEDA is enabled, the chart doesn't generate a worker HPA; the HPA KEDA generates has the same name as the Deployment. During an HPA → KEDA migration, KEDA logs a conflict error once and retries until Helm deletes the old HPA (transient).
- Check:
kubectl -n $NS get scaledobject
kubectl -n $NS describe scaledobject pushizo-worker-dispatch | sed -n '/Conditions/,$p'
kubectl -n $NS get hpa pushizo-worker-dispatch -o jsonpath='{.status.currentMetrics}'; echoAPI scaling is a CPU + memory HPA (api.hpa.behavior: fast scale-up — 100% or 4 pods in 60 seconds; slow scale-down — 300-second
window, at most 20% per minute).
6. Data layer connection profiles
Postgres (managed)
- The connection string is
DATABASE_URLin the Secret. The application role cannot be a superuser or haveBYPASSRLS; the worker checks this at startup in production (docs/runbooks/postgres-roles.md). database.sslMode→PGSSLMODE(node-postgres reads it only if the connection string has nosslmode):verify-full: TLS + certificate and hostname verification (production). Because node-postgres treatsrequire/verify-calikeverify-full, the schema only accepts"",disableandverify-full.- If the provider's CA isn't in Node's trust store (RDS, Cloud SQL server CA, Azure), mount it with
caBundle.configMapName/secretName;NODE_EXTRA_CA_CERTSis passed to the API, worker and migrate Job. The same bundle also applies torediss://.
- Pool:
DATABASE_POOL_MAXper pod (the worker uses at most 5). Total connections ≈ API replicas × pool + worker replicas × 5 + migrate 2; calculate with the HPA/KEDA upper bounds and keep it at 80% of the DB'smax_connections.
PgBouncer (database.pgbouncer.enabled)
The API and worker take DATABASE_URL from the database.pgbouncer.urlSecretKey key in the Secret (default DATABASE_POOLED_URL);
the migrate Job uses DATABASE_URL directly (the migrator takes a session-level pg_advisory_lock, and in a transaction pool
the lock is lost along with the connection).
What the application expects from the pooler:
pool_mode = transactionis compatible: the tenant RLS context is transaction-local viaset_config(..., true)(apps/api/src/common/tenant-rls.interceptor.ts), the automation lock ispg_advisory_xact_lock, there are no session-levelSET/LISTEN, and node-postgres uses unnamed prepared statements.- Application connections are opened with the
options=-c app.rls_bypass=onstartup parameter (packages/db/src/client.ts). PgBouncer doesn't forward this parameter to the server; you must setignore_startup_parameters = optionsand provide the same default at the role level:
-- Over a direct connection (admin role). Behavior doesn't change: connections still start with bypass on, and API requests turn it off with SET LOCAL.
alter role pushizo set app.rls_bypass = 'on';; pgbouncer.ini
pool_mode = transaction
ignore_startup_parameters = optionsVerification: psql "$DATABASE_POOLED_URL" -c "show app.rls_bypass" → on; a tenant request in the API (the RLS-off path) and worker
/readyz 200. Without the role setting, worker queries get caught by RLS (empty results).
Redis
- The only supported profile is
redis.mode: standalone:REDIS_URL(redis://, orrediss://for TLS). The API and worker create ioredis withnew Redis(REDIS_URL); there is no Sentinel/Cluster client and no{hash}-tagged prefix for BullMQ Cluster. Ifsentinelorclusteris selected, the chart stops with a descriptive error. - High availability: a managed service exposing a single primary endpoint (ElastiCache replication group primary endpoint +
Multi-AZ automatic failover, Memorystore Standard, Azure Cache Standard/Premium). On failover, ioredis reconnects;
BullMQ jobs are protected by Redis persistence, and on loss use
queue:recover(docs/runbooks/restore.md). - Required setting:
maxmemory-policy noeviction(BullMQ keys must not be evicted). AOF/RDB persistence enabled.
7. Adapting NetworkPolicy
When enabled: default deny (ingress + egress) for release pods, then only the required flows.
| Flow | Value |
|---|---|
| ingress controller → api, dashboard | networkPolicy.ingressController.{namespaceSelector,podSelector} |
Prometheus → api :4000/metrics, worker :4001/metrics | networkPolicy.monitoring.* |
| dashboard → api | automatic |
| all pods → DNS | networkPolicy.dns (kube-dns; if NodeLocal DNSCache is used, add 169.254.20.10/32:53 as an ipBlock to egress.extra) |
| api/worker → Postgres, Redis | egress.postgres / egress.redis: cidrs (the managed service's private subnets) or to (in-cluster selector) — required when enabled |
| api/worker → S3, SMTP | egress.s3 / egress.smtp (VPC endpoint CIDR); if empty, the internet rule covers it |
| api/worker → providers, webhook targets, IdPs | egress.internet: 0.0.0.0/0 minus private networks and 169.254.0.0/16 (cloud metadata — second line of SSRF defense), ports 443, 2197 (APNs), 465/587 |
| migrate → Postgres | automatic (egress.postgres) |
| additional flows | egress.extra (e.g. OTel collector), api.extraIngress (e.g. load testing namespace), dashboard.allowInternetEgress (Sentry) |
- Finding the Postgres/Redis CIDR: AWS
aws rds describe-db-subnet-groups,aws elasticache describe-cache-subnet-groups; GCP private services access range; Azure delegated subnet. Provide the subnet CIDR rather than a direct IP (the IP changes on failover). - Kubelet probes come from the node and aren't affected by policy on most CNIs; if your CNI also subjects node traffic to policy
(e.g. Cilium
allow-localhost=policy), add an ingress allow for the node source. - IPv6 dual stack: add
{ cidr: "::/0", except: ["fc00::/7", "fe80::/10", "::1/128"] }to theegress.internet.blockslist. - Verification:
kubectl -n $NS exec deploy/pushizo-worker-general -- wget -qO- -T 5 http://169.254.169.254/ || echo "metadata engelli (beklenen)"
kubectl -n $NS exec deploy/pushizo-worker-general -- wget -qO- -T 5 https://fcm.googleapis.com/ >/dev/null; echo "internet çıkışı: $?"
kubectl -n $NS run np-probe --rm -i --restart=Never --image=busybox:1.36 --overrides='{"spec":{"securityContext":{"runAsNonRoot":true,"runAsUser":65534,"seccompProfile":{"type":"RuntimeDefault"}},"containers":[{"name":"np-probe","image":"busybox:1.36","command":["wget","-qO-","-T","5","http://pushizo-api:4000/healthz"],"securityContext":{"allowPrivilegeEscalation":false,"capabilities":{"drop":["ALL"]}}}]}}' || echo "ad alanı içinden etiketsiz pod API'ye ulaşamıyor (beklenen)"8. Pod security, resources and probes
podSecurityContext:runAsNonRoot, UID/GID 1001 (thepushizouser in the images),seccompProfile: RuntimeDefault.containerSecurityContext:readOnlyRootFilesystem,allowPrivilegeEscalation: false,capabilities.drop: [ALL]; the schema rejects loosening these guarantees. Writable directories:/tmp(all) and dashboard.next/cache(emptyDir, size-limited).- The ServiceAccount token is not mounted (the application doesn't access the Kubernetes API).
- Probes hit real endpoints: API
/healthz(process) and/readyz(Postgres + Redis) — apps/api/src/health/health.controller.ts; worker/healthzand/readyz(Postgres + Redis) on:4001— apps/worker/src/http.ts; the dashboard has no separate health endpoint, so it uses the unauthenticated/login(same as the image HEALTHCHECK).startupProbeallows up to 2 minutes of startup time. - Shutdown: the API and dashboard
preStopwaits 5 seconds (so removal from the endpoint list propagates to the ingress), then SIGTERM; after SIGTERM the worker finishes active jobs within 30 seconds,terminationGracePeriodSeconds45. - Resource recommendations are in
values-production.yaml; after the first load test (docs/runbooks/load-testing.md), update the request/limit values according to the observed p95 CPU/memory. - Distribution:
topologySpreadgenerates zone (topology.kubernetes.io/zone) and node (kubernetes.io/hostname) constraints for each component (withmatchLabelKeys: [pod-template-hash], only pods of the same version are counted). The default isScheduleAnyway; on a cluster with at least 3 zones, setting the zone constraint toDoNotSchedulecauses new pods to wait when a zone is lost.
9. Observability
podMonitor.enabled(recommended): scrapes API (both colors) and worker pods with thepushizo.com/metrics=truelabel.serviceMonitor.enabled: the API Service (active color only) and the worker headless Services. The two cannot be enabled together.- The
labels.releaselabel must match kube-prometheus-stack's selector. Alert rules:infra/monitoring/prometheus/alerts.yml.
10. Zero-downtime deployment checklist
-
helm lintandhelm template … | kubectl apply --dry-run=server -f -are clean (schema +validate.yaml). - Migrations are backward compatible (old code works with the new schema); a
downfile exists; large table changes are a separate maintenance job. - Secret keys exist (
kubectl get externalsecret→SecretSynced); the CA ConfigMap exists. - API:
maxUnavailable: 0,minReadySeconds,preStop, readiness/readyz; PDBminAvailable< HPAminReplicas(if equal, node draining gets stuck). - Worker:
terminationGracePeriodSeconds(45) > application shutdown time (30); PDBmaxUnavailable: 1. - Blue/green: both colors'
imageTagexplicit; passive colorrollout statusready; preview smoke test passed; after the switch, the old color stays up during the observation period. - HPA/KEDA min replicas at least 2 (API, dispatch) and spread across zones (
kubectl get pods -o wide). - The
ApiHigh5xxRate,QueueDepthHighandPostgresConnectionsHighalerts did not fire during the deploy. - The same steps were rehearsed under load in staging (docs/runbooks/load-testing.md §5): k6 error rate < 0.1%.
11. Chart validation (developer)
helm lint infra/k8s/pushizo
for f in "" "-f infra/k8s/pushizo/values-production.yaml" "-f infra/k8s/pushizo/values-staging.yaml" \
"-f infra/k8s/pushizo/values-production.yaml -f infra/k8s/pushizo/values-cell-eu.yaml" \
"-f infra/k8s/pushizo/values-production.yaml -f infra/k8s/pushizo/values-cell-tr.yaml"; do
helm template hs infra/k8s/pushizo -n pushizo $f | kubeconform -strict -summary -kubernetes-version 1.30.0 \
-schema-location default \
-schema-location 'https://raw.githubusercontent.com/datreeio/CRDs-catalog/main/{{.Group}}/{{.ResourceKind}}_{{.ResourceAPIVersion}}.json'
done
helm template hs infra/k8s/pushizo --set redis.mode=sentinel # expected: descriptive errorCombinations (validated on 2026-10-15 with helm v3.17.3 + kubeconform v0.6.7): default, production, staging,
ExternalSecrets, blue/green, blue/green transition mode, KEDA (± CPU trigger), NetworkPolicy, PgBouncer + CA bundle, secrets.create
- ServiceMonitor, without HPA, all together; plus the 13 expected error scenarios (Redis sentinel/cluster, secret key in config, missing store, both monitors, no DB egress in NetworkPolicy, invalid color/queue/sslmode, root user, unknown key). Installation on a live cluster (end to end with ESO/KEDA/CNI) is not part of this validation; it is done with the §2 and §4 steps during the first staging installation.
Pushizo — Going live (step by step)
This document covers the first production installation on a single server (Docker Compose). The detailed checklist is GO_LIVE_CHECKLIST.md, infrastructure details are in infra/README.md. For Kubernetes, see docs/runbooks/helm-production.md.
Runbook: Restoring from backup and disaster recovery (DR)
Targets (docs/plan/08 §8.10): RPO 1 hour, RTO 2 hours. Backup sources: