Load testing (k6-operator, distributed)
v2 Phase 16 item 5 (docs/plan/10). Manifests in infra/load/, scripts in tests/load/. The single-machine Phase 14 procedure (10k events/s, 1M fan-out) is in tests/load/README.md; this runbook covers the distributed run on Kubernetes…
v2 Phase 16 item 5 (docs/plan/10). Manifests in infra/load/, scripts in tests/load/. The single-machine Phase 14 procedure
(10k events/s, 1M fan-out) is in tests/load/README.md; this runbook covers the distributed run on Kubernetes.
1. Scenarios and acceptance thresholds
| Scenario | Manifest / script | Acceptance (docs/plan/10 Phase 16) | k6 threshold |
|---|---|---|---|
| A. Event ingest 50k events/s, 15 min | infra/load/testrun-events-ingest-50k.yaml / tests/load/events-ingest.js | 50k events/s with no loss; errors < 0.1% | events_accepted rate ≥ 50000/RUNNERS (per runner), events_batch_failed and http_req_failed rate < 0.001, http_req_duration p(95) < 500 ms, dropped_iterations count < 1 |
| B. 10M fan-out (mock provider) | infra/load/testrun-fanout-10m.yaml / tests/load/fanout-campaign.js | start < 60 s; sending ≥ 50k/s | fanout_start_seconds max < 60, fanout_send_rate min ≥ 50000, fanout_missing_ratio max ≤ 0.001, fanout_completed rate == 1 |
Additional acceptance criteria (outside k6, §4.4 and §5.4): in A, the number of events written to Postgres = the number of accepted events, and the events queue
drains after the load ends; in B, there are no failed deliveries. The plan also requires "ingest → ClickHouse latency p95 < 30 s": once ClickHouse
shipping (Phase 16 item 1) is enabled, this is measured on the ClickHouse side with the same RUN_ID; today this setup measures the Postgres path.
2. Installing k6-operator
helm repo add grafana https://grafana.github.io/helm-charts && helm repo update
K6_OPERATOR_VERSION=$(helm search repo grafana/k6-operator -o json | jq -r '.[0].version') # record it and pin it
helm upgrade --install k6-operator grafana/k6-operator -n k6-operator-system --create-namespace --version "$K6_OPERATOR_VERSION"
kubectl get crd testruns.k6.io
kubectl apply -f infra/load/namespace.yaml # `load` namespace (Pod Security: restricted)Runners use the grafana/k6:1.0.0 image (TestRun runner.image). 50k events/s requires 10 runners × (1–2 vCPU, 1–2 GiB);
separate: true places each runner on its own node — you need at least 10 suitable nodes (or a separate load node pool), otherwise
the runners stay Pending. Runners should be in the same region as the target (so network latency does not inflate p95).
3. Common preparation
- Target environment: staging,
infra/k8s/pushizo/values-staging.yaml(PROVIDER_MODE=mock,ALLOW_MOCK_PROVIDERS=true;mock:ok-*tokens count as delivered, no real APNs/FCM calls are made). Watch the scaling limits throughout the test: API HPAmaxReplicas, the workereventsanddispatchgroups (KEDA or HPA), Postgresmax_connections. - Tenant: a temporary org + app (
ios_pushmock channel) + admin API key. From a machine that can reach the staging API and Mailpit:
E2E_API_URL=https://api.staging.pushizo.example E2E_MAILPIT_URL=https://mailpit.staging.pushizo.example \
npx tsx tests/load/bench-tenant.ts > tenant.json # { appId, apiKey, orgId, subId }
export APP_ID=$(jq -r .appId tenant.json) API_KEY=$(jq -r .apiKey tenant.json) ORG_ID=$(jq -r .orgId tenant.json)- Plan and quota: a new org is on the Free plan (API key 20 requests/s, 100k messages/month). A platform admin assigns
enterprisevia the dashboard Admin → Organizations → Plan (orPOST /admin/v1/orgs/:orgId/plan {"plan_code":"enterprise"}): 2000 requests/s, unlimited messages. Scenario A needs ~102 requests/s; on a lower plan you can supply multiple keys withAPI_KEYS. A new org can send at most 1000 messages per day for its first 24 hours (abuse protection, docs/platform-admin.md §3): for Scenario B, the tenant must be created at least 24 hours in advance, or use a permanent load test org. - Target Secret and scripts (
loadnamespace; never committed to the repo):
NS_APP=pushizo-staging
kubectl -n load create secret generic pushizo-load-target \
--from-literal=API_URL=http://pushizo-api.$NS_APP.svc:4000 --from-literal=APP_ID="$APP_ID" --from-literal=API_KEY="$API_KEY"
kubectl -n load create configmap pushizo-k6-scripts \
--from-file=tests/load/events-ingest.js --from-file=tests/load/fanout-campaign.jsAPI_URL is the in-cluster Service (it measures application capacity; for an end-to-end measurement including ingress/TLS you can supply
the public address). If NetworkPolicy is enabled in the target namespace, the API port must be opened to the load namespace: see the
networkPolicy.api.extraIngress example in values-staging.yaml.
4. Scenario A — 50k events/s ingest (15 min)
4.1 Running
RUN_ID=ingest-$(date +%Y%m%d%H%M)
kubectl -n load create configmap pushizo-load-params --from-literal=RUN_ID=$RUN_ID --dry-run=client -o yaml | kubectl apply -f -
kubectl apply -f infra/load/testrun-events-ingest-50k.yaml
kubectl -n load get testrun pushizo-events-ingest-50k -w # stage: initialization → created → started → finishedparallelism: 10 runners share the constant arrival rate (102 requests/s × 500 events = 51k events/s) via execution segments. Because k6 divides the rate
across the whole test (including ramp-up and ramp-down), a 2% margin is left: generation is 51k, the threshold is 50k. Thresholds are evaluated per runner
(k6-operator does not aggregate results): the rate threshold is MIN_EPS/RUNNERS; if parallelism changes, RUNNERS must change too.
4.2 Results
kubectl -n load get pods -l k6_cr=pushizo-events-ingest-50k
for p in $(kubectl -n load get pods -l k6_cr=pushizo-events-ingest-50k,runner=true -o name); do
kubectl -n load logs $p | grep -E "✓|✗|events_accepted|http_req_duration|dropped_iterations"; doneEach runner pod exits with code 0 if the thresholds passed and 99 if a threshold was breached (Completed / Error). For acceptance, all 10 runners
must pass. If remote-write is enabled (§6), the aggregate values are in Prometheus:
sum(rate(k6_events_accepted_total{testid="events-ingest-50k"}[1m])) # events/s (total)
sum(k6_events_accepted_total{testid="events-ingest-50k"}) # total accepted events
max(k6_http_req_duration_p95{testid="events-ingest-50k"}) # request p95 (s)
max(k6_http_req_failed_rate{testid="events-ingest-50k"}) # failed request rate (worst runner)4.3 Application-side observation (during load)
- Grafana API dashboard: 5xx rate,
http_request_duration_secondsp95, pod count (HPA). - Queues dashboard:
queue_depth{queue="events",state=~"waiting|prioritized"}must stay bounded;state="failed"must not grow. - Database: connection utilization < 90% (
PostgresConnectionsHigh), WAL/IO.
4.4 Loss check
Events are tagged with properties.run = RUN_ID. After the load ends and the events queue has drained, the count in Postgres must equal the total
k6 accepted (sum(k6_events_accepted_total) or the sum of events_accepted in the runner logs):
PGOPTIONS='-c app.rls_bypass=on' psql "$DATABASE_URL" -tAc \
"select count(*) from events where app_id = '$APP_ID' and properties->>'run' = '$RUN_ID' and occurred_at > now() - interval '3 hours'"5. Scenario B — 10M fan-out
5.1 Seeding recipients
10M users + mock ios_push test subscriptions are written directly to the DB (tests/load/fanout-bench.mjs --phase seed). This takes
a few GB in Postgres and 20–60 min; check the staging DB storage in advance.
TAG=fanout-$(date +%Y%m%d%H%M) IMG_TAG=0.1.0 # IMG_TAG: the version installed on the target
kubectl -n $NS_APP create configmap pushizo-load-seed-script --from-file=tests/load/fanout-bench.mjs \
--dry-run=client -o yaml | kubectl apply -f -
kubectl -n $NS_APP create configmap pushizo-load-seed-params --from-literal=APP_ID="$APP_ID" \
--from-literal=FANOUT_PHASE=seed --from-literal=FANOUT_TAG=$TAG --from-literal=FANOUT_COUNT=10000000 \
--from-literal=FANOUT_PARALLEL=6 --from-literal=FANOUT_MESSAGE= --from-literal=PGSSLMODE=verify-full \
--dry-run=client -o yaml | kubectl apply -f -
sed "s#pushizo-migrate:0.1.0#pushizo-migrate:${IMG_TAG}#" infra/load/fanout-seed-job.yaml | kubectl -n $NS_APP apply -f -
kubectl -n $NS_APP wait --for=condition=complete job/pushizo-fanout-seed --timeout=3h
kubectl -n $NS_APP logs job/pushizo-fanout-seed | tail -1 # {"phase":"seed","tag":"...","count":10000000,...}The Job connects directly to the managed DB (not PgBouncer), and since it does not carry the chart labels, the release NetworkPolicies do not
restrict it; the DB security group must be open to the cluster subnet. Alternative: from a management machine with DB access,
node tests/load/fanout-bench.mjs --db "$DATABASE_URL" --app "$APP_ID" --phase seed --tag $TAG --count 10000000 --parallel 6.
5.2 Running
kubectl -n load create configmap pushizo-load-params --from-literal=RUN_ID=$TAG --dry-run=client -o yaml | kubectl apply -f -
kubectl apply -f infra/load/testrun-fanout-10m.yaml
kubectl -n load logs -f -l k6_cr=pushizo-fanout-10m,runner=truesetup() sends a single notification (test_devices_only, ios_push, idempotency key k6-<RUN_ID>; the same RUN_ID does not send
a second time), and a single VU reads the counters once per second until the message is completed. The JSON line in the log contains the message ID,
the counters, the start time and the total duration.
5.3 Interpretation
fanout_start_seconds: acceptance → firstsent(scheduler + audience resolution + first dispatch job). If > 60 s, inspect the audience pagination rate and thescheduler/audiencequeue depth.fanout_send_rate: average rate from first send to completion. If < 50k/s, check in order: dispatch worker replicas (KEDAmaxReplicaCount) andconcurrency, Redis CPU (BullMQ), Postgres write latency (deliveriesinserts/updates), worker CPU throttling.fanout_missing_ratio:COUNT − sent; expected to be 0 with the mock provider. If it is greater than zero, check thefailed/suppressedcounters andquota_exceeded(new org limit, plan).
5.4 Cleanup
MSG_ID=$(kubectl -n load logs -l k6_cr=pushizo-fanout-10m,runner=true | grep -o '"message_id":"[^"]*"' | head -1 | cut -d'"' -f4)
kubectl -n $NS_APP delete job pushizo-fanout-seed
kubectl -n $NS_APP create configmap pushizo-load-seed-params --from-literal=APP_ID="$APP_ID" \
--from-literal=FANOUT_PHASE=cleanup --from-literal=FANOUT_TAG=$TAG --from-literal=FANOUT_COUNT=10000000 \
--from-literal=FANOUT_PARALLEL=1 --from-literal=FANOUT_MESSAGE=$MSG_ID --from-literal=PGSSLMODE=verify-full \
--dry-run=client -o yaml | kubectl apply -f -
sed "s#pushizo-migrate:0.1.0#pushizo-migrate:${IMG_TAG}#" infra/load/fanout-seed-job.yaml | kubectl -n $NS_APP apply -f -
kubectl -n $NS_APP wait --for=condition=complete job/pushizo-fanout-seed --timeout=3h6. Sending results to Prometheus (optional)
kubectl apply -f infra/load/k6-prometheus-rw.yaml # ConfigMap pushizo-k6-output (K6_OUT=experimental-prometheus-rw)- TestRuns read this ConfigMap via
envFrom(optional); if the ConfigMap is absent, output goes only to the runner logs. - Prometheus must have the remote-write receiver enabled (kube-prometheus-stack:
prometheus.prometheusSpec.enableRemoteWriteReceiver=true), or a Mimir/Thanos Receive endpoint must be provided. If authentication is required, addK6_PROMETHEUS_RW_USERNAME/PASSWORDfrom a separate Secret to the TestRun'srunner.envFrom. - Metric names are prefixed with
k6_; counters have the_totalsuffix, and trends have_p95,_p99,_min,_max,_avgsuffixes viaK6_PROMETHEUS_RW_TREND_STATS. Each run carries thetestidandscenariolabels (TestRunarguments).
7. Interpreting results and common problems
| Symptom | Likely cause | Where to look |
|---|---|---|
dropped_iterations > 0 | Not enough VUs (latency increased) or the runner CPU is saturated | MAX_VUS, runner CPU; API p95 |
http_req_failed 429 | plan api_rps (per key) | org plan, API_KEYS |
http_req_failed 5xx/timeout | API CPU, DB pool, Redis | API dashboard, HPA events, PostgresConnectionsHigh |
| High p95, no errors | HPA scales up too late | api.hpa.behavior.scaleUp, raise minReplicas before the test |
events queue keeps growing | Not enough events workers | worker.groups[events] KEDA threshold/maxReplicaCount |
| Missing events in the loss check | Write error after a 2xx response | API logs (request_id), events table partitions |
| Fan-out does not start | new org limit of 1000/day, plan message quota | API response HS_QUOTA_EXCEEDED, Admin → Abuse |
Acceptance: all thresholds green in two consecutive runs, and the ApiHigh5xxRate, QueueDepthHigh, PostgresConnectionsHigh
alerts must not have fired during the load. Record the results in the results table of tests/load/README.md and in the verification
section of docs/IMPLEMENTATION_STATUS.md (date, version, environment size, number of runners, threshold values).
8. Cleanup
kubectl -n load delete testrun pushizo-events-ingest-50k pushizo-fanout-10m --ignore-not-found
kubectl -n load delete secret pushizo-load-target
kubectl -n load delete configmap pushizo-k6-scripts pushizo-load-params --ignore-not-found
kubectl -n $NS_APP delete configmap pushizo-load-seed-script pushizo-load-seed-params --ignore-not-foundDelete the tenant via Admin → Organizations; purge-deleted-tenants clears its data after 30 days. Scenario A events
are deleted at the end of the event retention period (RETENTION_EVENTS_DAYS); if you need them gone sooner on staging, delete the tenant.
Go dispatcher
ADR-0026 §3. Moves push sending (APNs, FCM, HMS, Web Push) to a Go service (services/dispatcher); rendering, claiming and applying results stay in the worker. It is optional and can be rolled back instantly: DISPATCHER_GO_CHANNELS…
Runbook: Chaos drill (monthly, staging)
Test matrix (docs/plan/09): worker kill, Redis restart, provider 5xx injection — monthly, manual. Script: infra/scripts/chaos.sh.