Data transformation and source pipeline
Fix, filter and enrich incoming event and person data before it is written, and bring in data from your own systems with webhooks or scheduled HTTP pulls. A faulty rule never drops data: the record is quarantined…
Fix, filter and enrich incoming event and person data before it is written, and bring in data from your own systems with webhooks or scheduled HTTP pulls. A faulty rule never drops data: the record is quarantined and reprocessed once the rule is fixed. Available on Growth and above (ADR-0088).
How it works
- An event arrives from the SDK, the public API, an import, Segment or a source connector.
- The app's enabled transformations run in order (
position). A transformation can be limited by target (eventorperson), source (sdk,api,import,system,webhook_source,pull_source) and event name; an empty list means all. - The result is validated and written. A record caught by a filter rule is dropped on purpose. A record that fails a rule, the JavaScript code, validation or the write is written to the quarantine in its original form.
Event transformations can only change name, properties, value and occurred_at; the person/subscription link
and event_id cannot be changed. The public API's accepted count does not change (filtered and quarantined events
were accepted too). Person transformations apply to person records from source connectors and to Segment identify
calls.
Rule transformation
Dashboard → Audience → Data pipeline → Transformations → New transformation. Rule steps run in order; field paths
use dots (properties.cart.0.sku).
| Operation | Description |
|---|---|
map | Copies the value to another field (field mapping). |
rename | Moves the value and deletes the old field. |
remove | Deletes the field (for example a secret field). |
set | Writes a constant value (enrichment). |
template | Builds text with {{path}} placeholders. |
compute | lower, upper, trim, length, round, abs, add, multiply, now. |
cast | string, number, integer, boolean, date; if conversion fails, on_error: fail (quarantine, default), null, keep. |
filter | Drops the record when the conditions match (drop) or keeps only matching records (keep); all / any. |
Condition operators: eq, neq, gt, gte, lt, lte, contains, starts_with, ends_with, in, exists,
not_exists.
{
"name": "Price fix",
"target": "event",
"kind": "rules",
"event_names": ["checkout"],
"rules": [
{ "op": "filter", "action": "drop", "conditions": [{ "path": "properties.test", "operator": "eq", "value": true }] },
{ "op": "rename", "from": "properties.fiyat", "to": "properties.price" },
{ "op": "cast", "path": "properties.price", "to": "number" },
{ "op": "set", "path": "properties.channel", "value": "web" }
]
}With the API: POST /v1/apps/{app_id}/data-pipelines/transforms (admin scope); read endpoints work with users:read.
JavaScript transformation
Define a transform(record, context) function. Returning an object updates the record, returning null filters it
out on purpose; any other return value or an exception sends the record to the quarantine. context: app_id,
target, source, now.
function transform(record, context) {
if (record.properties.internal) return null;
record.properties.email = (record.properties.email || '').toLowerCase();
record.properties.total = Number(record.properties.price || 0) * Number(record.properties.qty || 1);
return record;
}The code runs in a QuickJS interpreter inside WebAssembly, in a fresh, isolated instance on every run: there is no
require, process, file system, network or timers. Limits: 50 ms per record, 2 s per batch, 16 MB memory, 20 KB of
code. A record that exceeds the time or memory limit is quarantined with js_timeout / js_memory. console.log
output is shown only in the preview. The code is compiled when saved; it is rejected if transform is not defined.
Preview
The editor's Preview button in the dashboard, or POST /v1/apps/{app_id}/data-pipelines/transforms/preview, does a
dry run with sample records; nothing is written or quarantined. You can run an unsaved definition (transform), a
single saved transformation (transform_id) or all of the app's enabled transformations. For each record the result
(kept, dropped, failed), the output, the error code/path and console lines are returned.
Source connectors
A webhook source gives you an endpoint: POST /hooks/data-sources/{id} (body up to 1 MB, up to 1000 records per
request). Authentication uses one of two modes:
signature(default): thePushizo-Signatureheader in the same format as our outgoing webhooks (t=<unix>,v1=<HMAC-SHA256>, signed text<t>.<body>, 5-minute tolerance).token:Authorization: Bearer <key>orX-Pushizo-Token: <key>.
The key is shown only once, when the source is created and when it is rotated; it is stored encrypted.
An HTTP pull source fetches the https URL you provide every 5–1440 minutes. An optional credential is written to a
header (default Authorization) and stored encrypted. Private network addresses are rejected, redirects are not
followed, and a request is limited to 10 s and 5 MB. For pagination, set next_cursor_path (the next cursor in the
response) and cursor_param (the query parameter); at most 10 pages / 5000 records per run. With resume_cursor the
next run continues from the last cursor. You can run it right away from the dashboard or with POST …/sources/{id}/run.
The mapping turns a raw record into an event or a person:
{
"records_path": "data.items",
"event_name_path": "type",
"external_id_path": "customer.id",
"event_id_path": "id",
"occurred_at_path": "created_at",
"value_path": "total",
"properties_path": "attributes"
}If properties_path is empty, the whole record becomes the properties. A person source uses external_id_path
(required), properties_path and tags_path. The event's person is found by external_id and created if missing. To
avoid duplicates when the same record is pulled again, map both event_id_path and occurred_at_path (the event ID
is derived from both). Source records are transformed with the webhook_source / pull_source source and written as
import.
Quarantine and error log
Dashboard → Data pipeline → Quarantine lists failing records with their code, field path and message. Each transformation shows its error count and last error code.
| Code | Meaning |
|---|---|
mapping_failed | The source record is not an object or could not be mapped. |
invalid_path | A field path in the rule is invalid. |
cast_failed | The type could not be converted (on_error: fail). |
rule_failed | A rule step could not be applied (for example add on a non-number). |
invalid_output | The result cannot be written (event name, person ID, properties size). |
js_error, js_invalid_return | JavaScript exception or invalid return value. |
js_timeout, js_memory | The time or memory limit was exceeded. |
write_failed | The write was rejected (for example the tag quota). |
After fixing the rule, Reprocess (or POST …/quarantine/reprocess; without IDs, the oldest 500 pending records)
runs the records again with the current transformations: successful ones are written, ones that still fail stay
pending. Discard closes a record without reprocessing. The content of processed and discarded records is deleted.
The quarantine holds personal data: related records are deleted when a user is deleted, and records are purged after
30 days.
Notes
- Transformation changes take effect on every API server within 15 seconds.
- If the plan drops below Growth, definitions are kept but not applied; a webhook source returns
403and a pull source reportsskipped. - The HTTP pull scheduler runs in the API process; set
DATA_PIPELINE_CONSUMER=falseto turn it off on a replica.
MCP server: Pushizo with AI assistants
With its Model Context Protocol (MCP) server, Pushizo lets Claude Desktop, Claude.ai, Cursor and other MCP clients read your segment, campaign, message, template, journey, user, analytics and web growth data and…
CLI, additional SDKs and Postman
packages/cli provides the pushizo command (npm @pushizo/cli, public repo pushizo-cli). PUSHIZO_API_KEY, PUSHIZO_APP_ID and the optional PUSHIZO_API_URL are environment variables. The key is never printed to output. Node.js 20+.