A/B test helps you decide which push and email configurations work better for your audience. Create and send different versions of push or email messages to understand which one is more relevant and leads to more conversions. Specify different content, actions performed after the push is opened, segmentation or scheduling rules, and track the results!
To start with A/B Testing, go to Messaging -> Push notifications -> A/B Experiments section of your Control Panel.
Here, you'll see the list of already launched experiments.
To set up a new A/B Test, press +Create Experiment.
Step 1. Enter the name of your experiment.
Step 2. Select an app to carry out an experiment on.
Step 3. Define the audience to which you'd like to send messages in the course of the experiment.
First, select a Filter to apply. The experiment will be held on only for those users who meet Filter's conditions.
Then, specify a segment size — the percentage of users meeting the Filter's conditions who will take part in your experiment.
If you use a precompiled Filter, you’ll see the number of users complying with its conditions and the number of users in a segment specified.
Step 4. Split the segment into groups. Each group will receive its own message.
Step 5. Specify content that will be sent to each of the groups.
First, select a content type — Push preset or Email preset. The selected content type will be used for both groups.
Then, select a preset that will be sent to each of the groups.
The experiment will be held for all platforms enabled in all presets accumulatively. For example, if a preset for a Group A has the iOS platform enabled, and the preset for a Group B has the Android and Chrome platforms, the A/B experiment will include users with iOS, Android, or Chrome.
All that users will be split into groups according to the rate you set on step 4. In case a user's platform does not match the platform of preset specified for the user's testing group, that user will receive the preset with no platform-specific parameters.
When everything's set up, press Send. The experiment will start immediately, that means all messages will be sent to all users taking part in the experiment with no delay.
To explore the results of your A/B Test, go to A/B Tests section and press Statistics button next to the experiment.
Statistics for the experiment contains the summary of conditions applied and the number of users involved. For each of the groups, here you'll see the number of messages sent and opened, as well as the CTR. To see the detailed stats for each message, press the Statistics button placed in the corresponding table row.