PushizoDocs
OperationsRunbooks

Runbook: Data key (DEK) rotation

Provider credentials (APNs p8, FCM private_key, VAPID private key) are stored with envelope encryption (ADR-0008): MASTER_KEY only wraps the data keys (DEKs); each record is encrypted with the active DEK as v2.<kid>.<iv>.<tag>.<ct>…

Provider credentials (APNs p8, FCM private_key, VAPID private key) are stored with envelope encryption (ADR-0008): MASTER_KEY only wraps the data keys (DEKs); each record is encrypted with the active DEK in the format v2.<kid>.<iv>.<tag>.<ct>.

When?

  • Routine: every 90 days.
  • Incident: when you suspect a DEK has leaked, when an authorized employee leaves, or when a database backup has left your control.

Command

npm run keys:rotate

What it does (in a single transaction):

  1. A new DEK is generated, wrapped with MASTER_KEY, and written to the data_keys table as active.
  2. The previously active key becomes retired.
  3. All envelope (v2.) values in every *_enc column in the schema are decrypted with the old DEK and re-encrypted with the new DEK; the matching *_kid column, if any, is updated. Because the list is read from the schema, new tables are covered automatically (channel_configs, webhooks, integrations, warehouse and reverse ETL connections, data stream headers, deliverability, Wallet).

Example output:

Veri anahtarı döndürüldü: dek_01… → dek_02…; yeniden şifrelenen kayıt: 3

To delete unused old keys (at least 2 minutes after the rotation; API/worker processes cache the active key for 60 s and may write with the old key during that window):

npm run keys:rotate -- --purge-only

The cleanup never deletes a key that appears in any *_enc value or *_kid column in the schema. --purge runs together with a rotation, but it does not delete the newly retired key because of the waiting period.

Changing MASTER_KEY

MASTER_KEY only protects the wrapped DEKs and a few directly encrypted fields (MFA secrets, SDK identity secrets, SSO client secrets, Shopify webhook secrets).

  1. Generate a new 32-byte key: node -e "console.log(require('crypto').randomBytes(32).toString('base64'))".
  2. Replace the MASTER_KEY value in .env with the new key; pass the old value to the command:
npm run keys:rewrap -- --old=<eski-base64>

(or provide the old key in the MASTER_KEY_PREVIOUS environment variable). In a single transaction, the command re-encrypts data_keys.wrapped_dek, mfa_factors.secret_enc (TOTP only; recovery codes are one-way hashes and are left untouched) and apps.sdk_secret_enc with the new key. 3. Restart the API and worker processes; use POST /internal/v1/apps/{appId}/channels/{channel}/validate to verify that a channel can be decrypted, and confirm that an MFA login works.

Verification

  • The "key rotation" and "MASTER_KEY rewrap" tests in npm run test:integration: after rotation/rewrap, credentials decrypt and validation succeeds.
  • The API can keep running during the rotation; since decryption loads the DEK by kid, old and new records can be read side by side.

On this page