Runbook: Data key (DEK) rotation
Provider credentials (APNs p8, FCM private_key, VAPID private key) are stored with envelope encryption (ADR-0008): MASTER_KEY only wraps the data keys (DEKs); each record is encrypted with the active DEK as v2.<kid>.<iv>.<tag>.<ct>…
Provider credentials (APNs p8, FCM private_key, VAPID private key) are stored with envelope encryption (ADR-0008):
MASTER_KEY only wraps the data keys (DEKs); each record is encrypted with the active DEK in the format v2.<kid>.<iv>.<tag>.<ct>.
When?
- Routine: every 90 days.
- Incident: when you suspect a DEK has leaked, when an authorized employee leaves, or when a database backup has left your control.
Command
npm run keys:rotateWhat it does (in a single transaction):
- A new DEK is generated, wrapped with
MASTER_KEY, and written to thedata_keystable as active. - The previously active key becomes
retired. - All envelope (
v2.) values in every*_enccolumn in the schema are decrypted with the old DEK and re-encrypted with the new DEK; the matching*_kidcolumn, if any, is updated. Because the list is read from the schema, new tables are covered automatically (channel_configs, webhooks, integrations, warehouse and reverse ETL connections, data stream headers, deliverability, Wallet).
Example output:
Veri anahtarı döndürüldü: dek_01… → dek_02…; yeniden şifrelenen kayıt: 3To delete unused old keys (at least 2 minutes after the rotation; API/worker processes cache the active key for 60 s and may write with the old key during that window):
npm run keys:rotate -- --purge-onlyThe cleanup never deletes a key that appears in any *_enc value or *_kid column in the schema. --purge runs together with a rotation, but it does not delete the newly retired key because of the waiting period.
Changing MASTER_KEY
MASTER_KEY only protects the wrapped DEKs and a few directly encrypted fields (MFA secrets, SDK identity secrets, SSO client secrets, Shopify webhook secrets).
- Generate a new 32-byte key:
node -e "console.log(require('crypto').randomBytes(32).toString('base64'))". - Replace the
MASTER_KEYvalue in.envwith the new key; pass the old value to the command:
npm run keys:rewrap -- --old=<eski-base64>(or provide the old key in the MASTER_KEY_PREVIOUS environment variable). In a single transaction, the command re-encrypts data_keys.wrapped_dek, mfa_factors.secret_enc (TOTP only; recovery codes are one-way hashes and are left untouched) and apps.sdk_secret_enc with the new key.
3. Restart the API and worker processes; use POST /internal/v1/apps/{appId}/channels/{channel}/validate to verify that a channel can be decrypted, and confirm that an MFA login works.
Verification
- The "key rotation" and "MASTER_KEY rewrap" tests in
npm run test:integration: after rotation/rewrap, credentials decrypt and validation succeeds. - The API can keep running during the rotation; since decryption loads the DEK by
kid, old and new records can be read side by side.
Runbook: Queue backlog and DLQ
Alerts: QueueDepthHigh (waiting + prioritized > 50k, 10 min; prioritized jobs live in BullMQ's prioritized set), DeadLetterQueueHigh (failed > 100), WorkerJobsStalled (jobs present, none completing), WorkerDown. Dashboard:…
Runbook: Transactional synthetic monitoring
The Phase 14 acceptance criterion "7 days of error-free synthetic monitoring on staging" is measured with this check. .github/workflows/synthetic.yml runs scripts/synthetic.ts every 15 minutes: