Uplift report

The uplift report shows what a campaign or a journey achieved on its own. It compares the users who received it with the holdout, a group you set aside on purpose so they don’t get the message, and shows how much more often the first group reached the goal.

A campaign is in the report only if you sent it with a holdout, a share of recipients you set aside. Your developers set it with the holdout_percentage parameter, from 1 to 20, in the Messaging API. A journey is in the report once for every Holdout Split point it contains.

How to get the report

• Journey: open the Holdout Split point of the journey. It shows Goal uplift vs holdout in the Control Panel.
• Campaign: you can get the report only as a CSV file. The export runs through the API, so ask your developers to run the export. Give them the dates, up to 90 days, and, if you need only one, the campaign or journey. Without one, the file covers every campaign and journey of the application. What’s in the file

Control groups have their own Impact report in Settings. It measures the application’s control groups, not a single campaign or journey.

What counts as treatment and control

The report compares two groups: treatment, the users who got the message, and control, the holdout, the users who didn’t.

	Treatment	Control
Campaign	Every recipient who was sent the campaign.	Every recipient the holdout kept from getting it, as in the application’s control groups.
Journey	Every user who went through the Holdout Split point onto the In journey branch.	Every user routed to the Holdout branch of that point.

Users and recipients with no User ID are left out of both groups, because Pushwoosh can’t tell which group they belong to or match them to a conversion without it.

Goal and attribution window

Campaign: The goal and the attribution window (how long after a send a conversion still counts) come from the application’s Conversion Goals. When more than one is configured, the report uses the shortest attribution window among them, so a send is never credited with a distant, unrelated conversion.

Journey: The goal is the journey’s own Goal points, and the window is the journey’s Conversion period setting (7 days if the journey doesn’t set one). The window starts when each user passes the Holdout Split point, because holdout users never get a send to start it from.

Both exclude message-engagement events (opens, clicks, and similar) from the goal. The holdout group never receives a message to react to, so counting engagement would only show that a message went out, not what it achieved. Control group analytics excludes them for the same reason.

Metrics in the report

Metric	What it means
Uplift %	The percentage difference between the treatment and control conversion rates, relative to the control rate. Same formula as the Uplift of the application’s control groups.
Incremental events	An estimate of how many more events the treatment group produced than it would have at the control group’s rate per user, counted for all treatment users. Same formula as Conversions driven by messaging.
% of treatment	Incremental events as a share of all events the treatment group generated.
Z-score, P-value, Confidence %	Confidence % shows how sure the report is that the gap between the groups is not chance: the higher, the more you can trust the result. Z-score and P-value are the statistics behind it, from the same test the application’s control groups use.
Significance	Significant, Not significant, or Not enough data. Significant means Confidence % is above 95%. It works both ways: a significant negative Uplift % means the message lowered the conversion rate. It reads as not enough data when either group has fewer than 10 conversions or fewer than 10 non-conversions.
Treatment revenue / Control revenue	Money each group brought in on the report’s goal event, kept separately per currency.
Incremental revenue	The money version of Incremental events: the difference in revenue per user between treatment and control, counted for all treatment users, per currency.
Revenue tracked	Whether the revenue source for this event was actually configured, not guessed. See Revenue in the report.

For example, a journey sends a welcome message to 90,000 users and holds back 10,000. In the In journey group, 4,500 users reach the goal (5%). In the Holdout group, 400 do (4%). The report shows:

Metric	Value
Uplift %	25%, because 5% is a quarter more than 4%
Significance	Significant, because with groups this large the gap is unlikely to be chance

Note

Uplift % counts each user once, so one unusually active user can’t skew it. Incremental events counts every event, without capping per user, as the application’s control groups do, so a single very active treatment user can raise that number without moving Uplift %. When the two disagree, trust Uplift %.

What’s in the exported file

Each row of the file is one campaign, or one Holdout Split point of a journey, for one goal event. A row contains:

• the campaign or journey and the goal event;
• the number of users and conversions in each group;
• the metrics from the table above;
• the revenue for each currency.

For each campaign, Holdout Split point, and goal event, the file holds the latest daily calculation within the dates you chose. See How far back the report can look.

For the journey from the example above, the main columns look like this:

Column	Example	What it means
owner_type	TYPE_JOURNEY	The row is about a journey, not a campaign.
event_name	purchase	The goal event.
treatment_users	90000	Users in the In journey group.
control_users	10000	Users in the Holdout group.
treatment_conversions	4500	Users in the In journey group who reached the goal.
control_conversions	400	Users in the Holdout group who reached the goal.
uplift_pct	25.00	Uplift %.
significance	TYPE_SIGNIFICANT	Significant.

Revenue in the report

Treatment, control, and incremental revenue are counted for the same goal event, the same users, and the same attribution window as the conversion numbers in the report.

Revenue is kept per currency, never converted

Revenue isn’t converted between currencies, because Pushwoosh keeps only today’s exchange rates, and converting an older window at today’s rate would be a guess. Treatment revenue, Control revenue, and Incremental revenue each hold one amount per currency instead of one combined total. Revenue from an event that doesn’t name a currency is listed under unknown.

Revenue tracked tells you whether the source was configured

Revenue tracked shows whether revenue was set up for the goal event, so an application that never set it up reads as unmeasured, not as zero revenue.

Pushwoosh reads an event’s revenue from three places, in this order:

1. The built-in PW_Conversion event’s own value and currency fields.
2. An event revenue mapping you configured for that event.
3. The legacy __amount and __currency attributes on the event, if neither of the above applies.

Revenue tracked is on only for the first two sources. With the third, the report still shows the amount but leaves the flag off, because nobody configured revenue for that event.

When the report updates

The report isn’t real time. It’s recalculated once a day, at 01:30 UTC, so a campaign sent or a journey point passed today shows up in the report tomorrow.

Only running journeys with at least one Goal point are recalculated. When a journey stops, the Holdout Split point keeps showing its last numbers.

How far back the report can look

Each daily calculation counts campaign sends from about the last 45 days and Holdout Split passes from the last 180 days. Older sends and passes drop out of the next calculation, so a campaign is no longer recalculated about 45 days after it was sent.

Pushwoosh keeps every daily calculation for 90 days. The Holdout Split point shows the latest one. An export shows the latest one within the dates you give, so an export for dates two months ago shows the numbers as they were then, for sends and passes before that.