Go dispatcher
ADR-0026 §3. Moves push sending (APNs, FCM, HMS, Web Push) to a Go service (services/dispatcher); rendering, claiming and applying results stay in the worker. It is optional and can be rolled back instantly: DISPATCHER_GO_CHANNELS…
docs/adr/0026-olcek-mimarisi.md. Moves push sending (APNs, FCM, HMS, Web Push) to a Go service
(services/dispatcher); rendering, claiming and applying results stay in the worker. It is optional and can be rolled back
instantly: while DISPATCHER_GO_CHANNELS is empty, all sending is done with the TS providers. Technical details and the bridge contract:
services/dispatcher/README.md.
Enabling
- Prerequisite: the Go service uses the same
MASTER_KEY,DATABASE_URLandREDIS_URLvalues as the worker. Server clocks must be synced with NTP (deadline comparison happens on both sides). - Start the service:
- Compose:
docker compose --env-file .env.prod -f infra/compose/docker-compose.prod.yml --profile go-dispatcher up -d dispatcher - Kubernetes: until a dispatcher Deployment is added to the chart, run the image
ghcr.io/exec0de/pushizo-dispatcheras a separate Deployment (values below).
- Compose:
- Verify it is ready: the container is healthy (
/dispatcher healthcheck),GET :4002/readyz→ready; in Redis,ZRANGE hs:dispatch-go:alive:ios_push 0 -1 WITHSCORESshows the instance name. - Enable channels one at a time: for worker-dispatch,
DISPATCHER_GO_CHANNELS=web_push(lowest risk) → monitor → add more (ios_push,android_push,hms_push,web_push). Restart the worker (the variable is read at startup). - Monitor (at least one campaign cycle): the metrics below; delivery rate and error codes should match the previous day.
While DISPATCHER_GO_FALLBACK=true (the default), if there is no live Go instance the batch is sent with the TS provider
(dispatcher_go_requests_total{outcome="fallback"}); sending does not stop even if the Go service is briefly down during a deployment.
If set to false, batches wait in the stream while Go is unavailable, and on timeout they get a transient error and are retried.
Rollback
Set DISPATCHER_GO_CHANNELS= (empty) and restart worker-dispatch: the TS providers are used from the next batch
onward. Requests left in the stream are deleted by the Go service without being sent once their deadline passes; if the worker shuts down
while waiting for a response, the affected batch is retried as a transient error (no duplicate notifications are produced). The Go service can also
be stopped separately; channels can also be rolled back one by one.
Metrics
| Metric | Source | Meaning / threshold |
|---|---|---|
provider_send_total{provider,outcome} | Go (channels in Go mode) / worker (TS mode) | Same name and labels; in Go mode the worker does not write it (no double counting). Compare the auth and invalid_token rates with the previous period |
provider_latency_seconds{provider} | Go / worker | Batch send duration; p95 should be close to or below the TS period |
dispatcher_go_requests_total{channel,outcome} | worker | ok; timeout > 0 means the Go service cannot keep up/is not running; rejected means a contract/version mismatch; fallback means no live instance |
dispatcher_go_roundtrip_seconds{channel} | worker | XADD → response time; scale out if it approaches DISPATCHER_GO_TIMEOUT_MS |
dispatcher_go_batches_total{channel,outcome} | Go | ok; stale (request past its deadline, not sent), invalid, error (credentials/DB) |
dispatcher_go_inflight_batches{channel} | Go | If it hits the DISPATCHER_MAX_BATCHES limit, add replicas or raise the limit |
In compose, Prometheus scrapes it with the pushizo-dispatcher job (DNS discovery; if the service is absent, no target is created).
Troubleshooting
| Symptom | Likely cause | What to do |
|---|---|---|
aktif veri anahtarı MASTER_KEY ile çözülemedi at startup | The Go service was given a different MASTER_KEY | Provide the same value as the worker/API (from the secret manager) |
GO_DISPATCHER_TIMEOUT on deliveries | Go service down (fallback disabled), overload, Redis latency | readyz, dispatcher_go_inflight_batches; add replicas or raise DISPATCHER_MAX_BATCHES; roll back if urgent |
GO_CREDENTIALS / GO_DATABASE error codes | The Go service cannot read/decrypt the row (DB access, RLS role, key) | Service log (kimlik bilgisi çözülemedi); the channel is not marked invalid, batches are retried |
GO_CONFIG_CHANGED | The channel configuration changed during sending | Resolves itself (the retry uses the new configuration) |
GO_DISPATCHER_REJECTED | The worker and the Go service versions speak different protocols | Bring both components to the same version |
stale increasing | Workers are abandoning timed-out requests; clock skew | NTP, DISPATCHER_GO_TIMEOUT_MS, Go capacity |
hs:dispatch-go:* stream growing | The Go service is not reading the channel stream (DISPATCHER_CHANNELS missing) | Fix the channel list; the stream is trimmed at ~100k entries |
Credentials are never written to Redis; stream entries contain only the rendered content, the device token and the config_id.
The Go service reads the channel_configs row in an RLS-enabled transaction (app.current_app_id) with an app_id filter.
Kubernetes values (to be added to the chart)
For the dispatcher Deployment: image pushizo-dispatcher, port 4002 (/healthz liveness, /readyz readiness,
/metrics ServiceMonitor), environment DATABASE_URL, REDIS_URL, MASTER_KEY (ExternalSecret), NODE_ENV=production,
DISPATCHER_CHANNELS, DISPATCHER_MAX_BATCHES, WEB_PUSH_EXTRA_HOSTS; terminationGracePeriodSeconds: 45,
readOnlyRootFilesystem: true, runAsNonRoot (UID 65532); NetworkPolicy egress: Postgres, Redis and providers
(443). For worker-dispatch: DISPATCHER_GO_CHANNELS, DISPATCHER_GO_TIMEOUT_MS, DISPATCHER_GO_FALLBACK.
ClickHouse analytics store and Kafka ingest
ADR-0026. Both are optional: when they are not configured, the system keeps working as before with Postgres + BullMQ.
Load testing (k6-operator, distributed)
v2 Phase 16 item 5 (docs/plan/10). Manifests in infra/load/, scripts in tests/load/. The single-machine Phase 14 procedure (10k events/s, 1M fan-out) is in tests/load/README.md; this runbook covers the distributed run on Kubernetes…