PushizoDocs
OperationsRunbooks

Go dispatcher

ADR-0026 §3. Moves push sending (APNs, FCM, HMS, Web Push) to a Go service (services/dispatcher); rendering, claiming and applying results stay in the worker. It is optional and can be rolled back instantly: DISPATCHER_GO_CHANNELS…

docs/adr/0026-olcek-mimarisi.md. Moves push sending (APNs, FCM, HMS, Web Push) to a Go service (services/dispatcher); rendering, claiming and applying results stay in the worker. It is optional and can be rolled back instantly: while DISPATCHER_GO_CHANNELS is empty, all sending is done with the TS providers. Technical details and the bridge contract: services/dispatcher/README.md.

Enabling

  1. Prerequisite: the Go service uses the same MASTER_KEY, DATABASE_URL and REDIS_URL values as the worker. Server clocks must be synced with NTP (deadline comparison happens on both sides).
  2. Start the service:
    • Compose: docker compose --env-file .env.prod -f infra/compose/docker-compose.prod.yml --profile go-dispatcher up -d dispatcher
    • Kubernetes: until a dispatcher Deployment is added to the chart, run the image ghcr.io/exec0de/pushizo-dispatcher as a separate Deployment (values below).
  3. Verify it is ready: the container is healthy (/dispatcher healthcheck), GET :4002/readyz → ready; in Redis, ZRANGE hs:dispatch-go:alive:ios_push 0 -1 WITHSCORES shows the instance name.
  4. Enable channels one at a time: for worker-dispatch, DISPATCHER_GO_CHANNELS=web_push (lowest risk) → monitor → add more (ios_push,android_push,hms_push,web_push). Restart the worker (the variable is read at startup).
  5. Monitor (at least one campaign cycle): the metrics below; delivery rate and error codes should match the previous day.

While DISPATCHER_GO_FALLBACK=true (the default), if there is no live Go instance the batch is sent with the TS provider (dispatcher_go_requests_total{outcome="fallback"}); sending does not stop even if the Go service is briefly down during a deployment. If set to false, batches wait in the stream while Go is unavailable, and on timeout they get a transient error and are retried.

Rollback

Set DISPATCHER_GO_CHANNELS= (empty) and restart worker-dispatch: the TS providers are used from the next batch onward. Requests left in the stream are deleted by the Go service without being sent once their deadline passes; if the worker shuts down while waiting for a response, the affected batch is retried as a transient error (no duplicate notifications are produced). The Go service can also be stopped separately; channels can also be rolled back one by one.

Metrics

MetricSourceMeaning / threshold
provider_send_total{provider,outcome}Go (channels in Go mode) / worker (TS mode)Same name and labels; in Go mode the worker does not write it (no double counting). Compare the auth and invalid_token rates with the previous period
provider_latency_seconds{provider}Go / workerBatch send duration; p95 should be close to or below the TS period
dispatcher_go_requests_total{channel,outcome}workerok; timeout > 0 means the Go service cannot keep up/is not running; rejected means a contract/version mismatch; fallback means no live instance
dispatcher_go_roundtrip_seconds{channel}workerXADD → response time; scale out if it approaches DISPATCHER_GO_TIMEOUT_MS
dispatcher_go_batches_total{channel,outcome}Gook; stale (request past its deadline, not sent), invalid, error (credentials/DB)
dispatcher_go_inflight_batches{channel}GoIf it hits the DISPATCHER_MAX_BATCHES limit, add replicas or raise the limit

In compose, Prometheus scrapes it with the pushizo-dispatcher job (DNS discovery; if the service is absent, no target is created).

Troubleshooting

SymptomLikely causeWhat to do
aktif veri anahtarı MASTER_KEY ile çözülemedi at startupThe Go service was given a different MASTER_KEYProvide the same value as the worker/API (from the secret manager)
GO_DISPATCHER_TIMEOUT on deliveriesGo service down (fallback disabled), overload, Redis latencyreadyz, dispatcher_go_inflight_batches; add replicas or raise DISPATCHER_MAX_BATCHES; roll back if urgent
GO_CREDENTIALS / GO_DATABASE error codesThe Go service cannot read/decrypt the row (DB access, RLS role, key)Service log (kimlik bilgisi çözülemedi); the channel is not marked invalid, batches are retried
GO_CONFIG_CHANGEDThe channel configuration changed during sendingResolves itself (the retry uses the new configuration)
GO_DISPATCHER_REJECTEDThe worker and the Go service versions speak different protocolsBring both components to the same version
stale increasingWorkers are abandoning timed-out requests; clock skewNTP, DISPATCHER_GO_TIMEOUT_MS, Go capacity
hs:dispatch-go:* stream growingThe Go service is not reading the channel stream (DISPATCHER_CHANNELS missing)Fix the channel list; the stream is trimmed at ~100k entries

Credentials are never written to Redis; stream entries contain only the rendered content, the device token and the config_id. The Go service reads the channel_configs row in an RLS-enabled transaction (app.current_app_id) with an app_id filter.

Kubernetes values (to be added to the chart)

For the dispatcher Deployment: image pushizo-dispatcher, port 4002 (/healthz liveness, /readyz readiness, /metrics ServiceMonitor), environment DATABASE_URL, REDIS_URL, MASTER_KEY (ExternalSecret), NODE_ENV=production, DISPATCHER_CHANNELS, DISPATCHER_MAX_BATCHES, WEB_PUSH_EXTRA_HOSTS; terminationGracePeriodSeconds: 45, readOnlyRootFilesystem: true, runAsNonRoot (UID 65532); NetworkPolicy egress: Postgres, Redis and providers (443). For worker-dispatch: DISPATCHER_GO_CHANNELS, DISPATCHER_GO_TIMEOUT_MS, DISPATCHER_GO_FALLBACK.

On this page