A/B testing and holdout (Phase 12)
Compare content variants in campaign and journey sends, measure incremental impact with a control (holdout) group, and pick the winner automatically or manually. Design decisions: ADR-0019. Shared code:…
Compare content variants in campaign and journey sends, measure incremental impact with a control (holdout) group, and pick the winner automatically or manually. Design decisions: docs/adr/0019-ab-test-ve-tercihler.md. Shared code: packages/db/src/ab-testing.ts.
1. Campaign variants
When creating/updating a campaign, pass a variants array (push, email, SMS; at most 10):
{
"name": "Spring",
"variants": [
{ "name": "A", "weight_percent": 45, "content": { "contents": { "tr": "Bahar indirimi" } } },
{ "name": "B", "weight_percent": 45, "content": { "contents": { "tr": "%20 indirim" } } },
{ "name": "Control", "weight_percent": 10, "is_control": true }
],
"ab_test": {
"enabled": true,
"test_percentage": 30,
"winner": { "mode": "auto", "metric": "click_rate", "evaluate_after_minutes": 120 }
}
}Rules: at least 2 variants, weights summing to 100, at most one is_control. content: null uses the campaign content (practical when only the subject/heading differs). Variants are stored in the campaign_variants table; on each run they are frozen (snapshot) into the messages.variants field, so later edits do not change past reports.
2. Deterministic assignment
During audience resolution (apps/worker/src/delivery/audience.ts), a 0–99 bucket is computed for each recipient with abBucket(subject, campaign_id) (FNV-1a + murmur3 mixing). The subject is user_id, or subscription_id if there is none. The bucket is mapped to a variant by weight; the same user always gets the same variant in the same campaign (a multi-device user gets the same content on all devices). deliveries.variant_id is the assigned variant.
Acceptance criterion (plan 09): distribution within ±2% for two 50/50 variants — apps/worker/src/__tests__/ab-testing.int.test.ts verifies this on 1,000 users.
3. Holdout (control group)
Recipients that fall into the is_control variant are not sent the message: the delivery is written with status = suppressed, suppression_reason = holdout. Conversion is still measured: if the user has a conversion event (event_definitions.is_conversion) within 24 hours of the suppression (campaignVariantStats(..., windowHours)), it counts toward the control group's converted counter. The difference between a variant's conversion rate and the control conversion rate is therefore the incremental impact.
4. Test percentage and winner send
- If
ab_test.test_percentage < 100, the first run only goes to recipients withabBucket(subject, "t:"+campaign_id) < test_percentage; the campaign becomesab_test.phase = "testing", andrefreshCampaignStatsdoes not consider the campaign complete. - Manual selection:
POST /v1/apps/{app_id}/campaigns/{id}/winner { "variant_id": "cvr_…" }(internal: same path).recordWinnerwriteswinner_variant_id,decided_at,decided_byand the significance at that moment to the campaign; a second run is created withcreateCampaignRun(db, c, now, { winnerVariantId })— the first run's recipients (test group + holdout) are excluded, and only the winning variant goes to the remaining audience;phase = "winner_sent". - Automatic selection: every 60 seconds, the worker's
ab-evaluatejob evaluates tests withwinner.mode = autowhoselast_run_at + evaluate_after_minuteshas passed (evaluateAbTests,apps/worker/src/processors/scheduler.ts).pickWinnerselects the best non-control variant by the metric;decided_by = "auto".
5. Significance
twoProportionZTest(a, b) is a two-proportion, two-tailed z-test computed between the best and second-best variants; p_value < 0.05 → significant: true. A single variant or an insufficient sample (n = 0, pooled rate 0/1) → p_value = 1. The metric denominator is sent for variants and the holdout count for the control group.
GET /v1/apps/{app_id}/campaigns/{id}/variants/stats (campaigns:read):
{
"ab_test": { "enabled": true, "phase": "testing", "winner": { "mode": "auto", "metric": "click_rate" } },
"variants": [
{ "variant_id": "cvr_…", "name": "A", "weight_percent": 45, "is_control": false, "targeted": 450, "sent": 448, "delivered": 440, "opened": 120, "clicked": 40, "converted": 12, "holdout": 0, "open_rate": 0.268, "click_rate": 0.089, "conversion_rate": 0.027 },
{ "variant_id": "cvr_…", "name": "Control", "is_control": true, "holdout": 100, "converted": 1, "conversion_rate": 0.01, "…": "…" }
],
"leader": "cvr_…",
"significance": { "p_value": 0.03, "significant": true, "metric": "click_rate" }
}The dashboard's Campaigns → Campaign → Variants shows the same data with a comparison table and a "Pick winner" button.
6. A/B in a journey message step
Add ab_test to the send_push | send_email | send_sms node data (packages/shared/src/journeys.ts):
{
"ab_test": {
"variants": [
{ "id": "a", "label": "Short", "percent": 50, "template_id": "tpl_1" },
{ "id": "b", "label": "Long", "percent": 50, "content": { "contents": { "tr": "…" } } }
],
"winner": { "metric": "click_rate", "min_sends": 200, "confidence": 0.95 }
}
}The engine picks the variant deterministically per run (journeySplitBranch seed ${node.id}:ab); the message is written with journey_node_id + variant_id. evaluateJourneyNodeWinners (inside the segment scan lock, every minute) applies the z-test once the node's total sends reach min_sends; if p_value ≤ 1 − confidence, it writes journeys.node_winners[node_id] = variant_id, and subsequent runs receive only the winner. GET …/journeys/{id}/stats returns a variants breakdown per node.
7. In-app variants
In-app messages define variants: [{ id, name, weight_percent, display }]; the SDK receives the message with an assigned variant (variant_id) from /sdk/v1/apps/{id}/config, and impression/click/dismiss events are reported with variant_id. The GET …/in-app-messages/{id} response includes variant_stats (impressions, clicks, ctr).
8. SDKs
| SDK | Call |
|---|---|
| Node | hs.campaigns.variantStats(id), hs.campaigns.selectWinner(id, { variant_id }) |
| PHP | $hs->campaigns->variantStats($id), $hs->campaigns->selectWinner($id, $variantId) |
| Python / Go / Java / .NET / Ruby | Generated from OpenAPI (campaigns.variant_stats, campaigns.select_winner; names in sdk/scripts/server-sdks/ir.mjs) |
9. Tests
apps/worker/src/__tests__/ab-testing.int.test.ts: 50/50 distribution, holdout conversion + winner + z-test, test percentage + winner run, automatic evaluation.apps/api/src/__tests__/phase12.int.test.ts: variant CRUD/validation, stats and manual winner, journey A/B validation, tenant isolation.
Decision engine
Picks the best option for each user by learning from behavior. An A/B test picks a single winner; the decision engine sees the differences between user groups and keeps learning.
Global control group
The global control group receives no marketing messages for a fixed percentage of users. Comparing these users' conversions with everyone else's measures the total impact of your messaging. Decision record:…