PushizoDocs
A/B testing

A/B testing and holdout (Phase 12)

Compare content variants in campaign and journey sends, measure incremental impact with a control (holdout) group, and pick the winner automatically or manually. Design decisions: ADR-0019. Shared code:…

Compare content variants in campaign and journey sends, measure incremental impact with a control (holdout) group, and pick the winner automatically or manually. Design decisions: docs/adr/0019-ab-test-ve-tercihler.md. Shared code: packages/db/src/ab-testing.ts.

1. Campaign variants

When creating/updating a campaign, pass a variants array (push, email, SMS; at most 10):

{
  "name": "Spring",
  "variants": [
    { "name": "A", "weight_percent": 45, "content": { "contents": { "tr": "Bahar indirimi" } } },
    { "name": "B", "weight_percent": 45, "content": { "contents": { "tr": "%20 indirim" } } },
    { "name": "Control", "weight_percent": 10, "is_control": true }
  ],
  "ab_test": {
    "enabled": true,
    "test_percentage": 30,
    "winner": { "mode": "auto", "metric": "click_rate", "evaluate_after_minutes": 120 }
  }
}

Rules: at least 2 variants, weights summing to 100, at most one is_control. content: null uses the campaign content (practical when only the subject/heading differs). Variants are stored in the campaign_variants table; on each run they are frozen (snapshot) into the messages.variants field, so later edits do not change past reports.

2. Deterministic assignment

During audience resolution (apps/worker/src/delivery/audience.ts), a 0–99 bucket is computed for each recipient with abBucket(subject, campaign_id) (FNV-1a + murmur3 mixing). The subject is user_id, or subscription_id if there is none. The bucket is mapped to a variant by weight; the same user always gets the same variant in the same campaign (a multi-device user gets the same content on all devices). deliveries.variant_id is the assigned variant.

Acceptance criterion (plan 09): distribution within ±2% for two 50/50 variants — apps/worker/src/__tests__/ab-testing.int.test.ts verifies this on 1,000 users.

3. Holdout (control group)

Recipients that fall into the is_control variant are not sent the message: the delivery is written with status = suppressed, suppression_reason = holdout. Conversion is still measured: if the user has a conversion event (event_definitions.is_conversion) within 24 hours of the suppression (campaignVariantStats(..., windowHours)), it counts toward the control group's converted counter. The difference between a variant's conversion rate and the control conversion rate is therefore the incremental impact.

4. Test percentage and winner send

  • If ab_test.test_percentage < 100, the first run only goes to recipients with abBucket(subject, "t:"+campaign_id) < test_percentage; the campaign becomes ab_test.phase = "testing", and refreshCampaignStats does not consider the campaign complete.
  • Manual selection: POST /v1/apps/{app_id}/campaigns/{id}/winner { "variant_id": "cvr_…" } (internal: same path). recordWinner writes winner_variant_id, decided_at, decided_by and the significance at that moment to the campaign; a second run is created with createCampaignRun(db, c, now, { winnerVariantId }) — the first run's recipients (test group + holdout) are excluded, and only the winning variant goes to the remaining audience; phase = "winner_sent".
  • Automatic selection: every 60 seconds, the worker's ab-evaluate job evaluates tests with winner.mode = auto whose last_run_at + evaluate_after_minutes has passed (evaluateAbTests, apps/worker/src/processors/scheduler.ts). pickWinner selects the best non-control variant by the metric; decided_by = "auto".

5. Significance

twoProportionZTest(a, b) is a two-proportion, two-tailed z-test computed between the best and second-best variants; p_value < 0.05 → significant: true. A single variant or an insufficient sample (n = 0, pooled rate 0/1) → p_value = 1. The metric denominator is sent for variants and the holdout count for the control group.

GET /v1/apps/{app_id}/campaigns/{id}/variants/stats (campaigns:read):

{
  "ab_test": { "enabled": true, "phase": "testing", "winner": { "mode": "auto", "metric": "click_rate" } },
  "variants": [
    { "variant_id": "cvr_…", "name": "A", "weight_percent": 45, "is_control": false, "targeted": 450, "sent": 448, "delivered": 440, "opened": 120, "clicked": 40, "converted": 12, "holdout": 0, "open_rate": 0.268, "click_rate": 0.089, "conversion_rate": 0.027 },
    { "variant_id": "cvr_…", "name": "Control", "is_control": true, "holdout": 100, "converted": 1, "conversion_rate": 0.01, "…": "…" }
  ],
  "leader": "cvr_…",
  "significance": { "p_value": 0.03, "significant": true, "metric": "click_rate" }
}

The dashboard's Campaigns → Campaign → Variants shows the same data with a comparison table and a "Pick winner" button.

6. A/B in a journey message step

Add ab_test to the send_push | send_email | send_sms node data (packages/shared/src/journeys.ts):

{
  "ab_test": {
    "variants": [
      { "id": "a", "label": "Short", "percent": 50, "template_id": "tpl_1" },
      { "id": "b", "label": "Long", "percent": 50, "content": { "contents": { "tr": "…" } } }
    ],
    "winner": { "metric": "click_rate", "min_sends": 200, "confidence": 0.95 }
  }
}

The engine picks the variant deterministically per run (journeySplitBranch seed ${node.id}:ab); the message is written with journey_node_id + variant_id. evaluateJourneyNodeWinners (inside the segment scan lock, every minute) applies the z-test once the node's total sends reach min_sends; if p_value ≤ 1 − confidence, it writes journeys.node_winners[node_id] = variant_id, and subsequent runs receive only the winner. GET …/journeys/{id}/stats returns a variants breakdown per node.

7. In-app variants

In-app messages define variants: [{ id, name, weight_percent, display }]; the SDK receives the message with an assigned variant (variant_id) from /sdk/v1/apps/{id}/config, and impression/click/dismiss events are reported with variant_id. The GET …/in-app-messages/{id} response includes variant_stats (impressions, clicks, ctr).

8. SDKs

SDKCall
Nodehs.campaigns.variantStats(id), hs.campaigns.selectWinner(id, { variant_id })
PHP$hs->campaigns->variantStats($id), $hs->campaigns->selectWinner($id, $variantId)
Python / Go / Java / .NET / RubyGenerated from OpenAPI (campaigns.variant_stats, campaigns.select_winner; names in sdk/scripts/server-sdks/ir.mjs)

9. Tests

  • apps/worker/src/__tests__/ab-testing.int.test.ts: 50/50 distribution, holdout conversion + winner + z-test, test percentage + winner run, automatic evaluation.
  • apps/api/src/__tests__/phase12.int.test.ts: variant CRUD/validation, stats and manual winner, journey A/B validation, tenant isolation.

On this page