Control groups FAQ

Can I hold out users for just one journey instead of my whole app?

Yes. Use the Holdout Split element inside Customer Journey Builder to hold out a share of users at one point in a single journey, instead of holding out users everywhere with this account-wide control group. It reports its own goal uplift, separate from this page.

Can an application have more than one control group?

In the Control Panel, no. It creates one control group per application, applied automatically to every marketing send. This is an interface limit, not a platform one. The panel doesn’t offer a way to point a specific message at a different group.

Can I delete the control group?

Yes, from the three-dot menu on the group’s page. This cannot be undone, and it leaves the application without a fallback: marketing messages go to everyone until you create a new control group. To pause the group without losing it, disable it instead of deleting it.

What users are in the control group?

Pushwoosh doesn’t keep a stored list of control group members. It uses a formula, a calculation that always gives the same answer for the same inputs, based on your app, the user’s ID, and the group’s current settings. This applies to your entire existing user base the moment you enable the group, not only to new sign-ups, and the result stays the same for a given user until you reshuffle the group or change its size, the user’s ID changes, or an Auto-refresh cycle ends or an Experiment reaches its end date.

Do control group users receive transactional messages?

Yes. Transactional messages (e.g. password resets, order confirmations) are delivered to all users, including the control group. Only marketing messages are blocked for control group users.

Where do I set the marketing or transactional message type?

You choose it wherever you send messages. For each channel, see Where to set the message type.

What happens to campaigns created before I enabled the control group?

Campaigns created before you enabled the control group work exactly like new ones: Pushwoosh checks a campaign’s message type at send time, not at creation time.

• Marketing message (or no message type at all, which can happen for campaigns created via the API): the control group excludes those users starting immediately.
• Transactional message: control group users keep receiving it as before.

Edit the campaign and set the message type to Transactional message if you want it to reach everyone, including the control group.

Can I change the control group size after enabling it?

Yes. Open the group’s page from Settings > Control groups, click Control group settings, choose a new percentage (1% to 20%), and click Apply. Pushwoosh recomputes membership for your whole user base against the new percentage: growing the group adds users without removing anyone already in it, and shrinking it releases some existing members. This can affect comparison with earlier campaigns, so change the size only when needed.

What control group modes are available?

Three. Permanent keeps the same group until you turn it off. Auto-refresh replaces it with a new group automatically at the end of each cycle (1, 2, or 3 months), returning the previous group to messaging. Experiment with an end date runs the group until a date you set (at least 30 days out), then returns everyone to messaging automatically. Set the mode when you create the group, or later in Control group settings, the same dialog used to edit the group’s size and countries. Changing the mode closes the group’s current cycle, the same as resizing or rescoping it.

How do I see a control group’s past cycles?

1. Open the group’s page.
2. Click the date-range selector in the top right. It’s labeled Current cycle for an Auto-refresh group, Current experiment for an Experiment with an end date group, and has no label for a Permanent group.
3. Pick a cycle under Previous. Ended cycles are listed by number, as Cycle or Experiment.

A Permanent group can have past cycles too, because resizing, rescoping, or changing the mode starts a new one.

Why did my control group results start over?

The measurement window restarts whenever the group’s members or audience change:

• you resize the group or change its countries or tag;
• you reshuffle the group;
• you change the mode;
• an Auto-refresh cycle ends or an Experiment reaches its end date.

Data from before that point no longer matches the current group, so it isn’t included. Ended cycles stay available under Previous in the date-range selector on the group’s page.

What happens when an Experiment with an end date ends?

On the To date you set, everyone in the group automatically returns to marketing messages. The experiment then appears under Previous in the date-range selector on the group’s page, named Experiment with its own number, so you can still review its results.

Can I scope the control group to specific countries?

Yes. Set Countries when you create the group, or later: open the group’s page from Settings > Control groups and click Control group settings. Leave it empty to hold out users across your whole audience, or select one or more countries to hold out only devices there. Country scope is a per-device check on top of the percentage formula: it doesn’t change who the formula selects, only which of their devices get excluded from marketing. Changing Countries restarts the uplift measurement, since the compared audience changes.

What happens to a device with no country data?

It’s never held out, even when Countries is set and its user is otherwise selected. Pushwoosh checks a device’s system Country tag against the scope, and a device without that tag has nothing to check against, so it keeps receiving marketing messages.

Can I scope the control group by tag?

Yes. Set Tag when you create the group, or later: open the group’s page from Settings > Control groups and click Control group settings. Choose any string or boolean tag except Country, then choose one or more values. It works like country scope: a per-device check on top of the percentage formula, combined with Countries if you set both. A device with no value for the tag, or a value you didn’t include, keeps receiving marketing messages even if its user is otherwise selected.

How does tag scope interact with device and user tags?

A tag you set per device only ever describes that one device, so scoping by it holds out devices one at a time. A tag you set per user is copied onto every device of that user, so scoping by it holds out or spares all of that user’s devices together. Either way, a device with no value for the scope tag keeps receiving marketing messages, and which devices are held out can change over time as a user’s tag value changes, without anyone editing the control group.

What happens if I disable the control group?

Pushwoosh stops holding out any users and everyone starts receiving marketing messages again. If you enable the control group again later without reshuffling, you get back the exact same group, because membership is computed by the same formula. To get a different group after re-enabling, reshuffle it.

What does Reshuffle do?

Reshuffle control group draws a brand-new random sample at the same percentage, replacing the current control group with a different set of users. It’s available from the three-dot menu on the group’s page. Use it when the same users have been excluded from marketing for a long time and you want fresh holdout coverage. Reshuffling also restarts the measurement window for control group analytics.

How do I see which users are in the control group?

Pushwoosh doesn’t tag control group users. Open the group’s page from Settings > Control groups, click the three-dot menu, and select Export users to download a CSV of current members. To check membership programmatically instead, call CheckControlGroupMembership from the Control Groups API with a batch of user IDs.

Why might a user in the control group still receive marketing messages?

Pushwoosh computes control group membership by User ID. If the same person has several contact methods (e.g. email, push device) that are not linked under one User ID, Pushwoosh treats them as different users and computes membership for each separately: one may land in the control group while another does not. A contact method with no User ID at all is never held out, since Pushwoosh needs a User ID to compute membership. To avoid this, register all contact methods with the same User ID.

What control group percentage should I choose?

The right percentage depends on your goals, app size, and how you plan to use the results. A smaller percentage (e.g. 3%) keeps more users in the audience and may suit teams that prioritize reach. A larger percentage (e.g. 10%) can give stronger statistical confidence when comparing groups. You can set a Custom value between 1% and 20% to match your needs.

Why isn’t my control group exactly the percentage I set?

The group size shown on the page is the actual number of users the hash formula assigned to the group, not the percentage multiplied by your user count. On smaller apps this count can visibly differ from the percentage, since the formula distributes users individually rather than sampling an exact share. The count is cached and refreshed about once a day. You can trigger an immediate recalculation from the page, which shows the previous count with a loading indicator while it runs.

What does Not enough data mean in the events table?

Not enough data means that either the treatment or control group had fewer than 10 conversions or fewer than 10 non-conversions for that event during the selected time window. Pushwoosh cannot determine whether the difference between the groups is statistically significant until more data is available.

How is Uplift calculated?

Uplift shows the percentage difference between the treatment and control conversion rates, relative to the control rate. A positive value means the treatment group converted better. Pushwoosh calculates it as (conv. treatment − conv. control) / conv. control × 100%.

What does Conversions driven by messaging mean?

Conversions driven by messaging is the estimated number of additional events generated by your messages beyond the number expected without messaging. Pushwoosh calculates it as (events per user in treatment − events per user in control) × treatment group size. The percentage shows these additional events as a share of all treatment events.

Which events can I compare and how many?

Events set as Conversion Goals in Customer Journey appear in the table automatically. Click Add events to compare to add any other event, up to 10 events in total. Message-engagement events (e.g. opens, clicks, sends, bounces) are excluded even if set as a goal, since control group users never receive marketing messages to react to.