Running A/B/n tests

Learn how to set up and analyze A/B/n tests in Pushwoosh Customer Journey Builder

Use the A/B/n Split element in customer journeys to test which sequence of messages works best for your Conversion Goals. This will help you adjust your communications to your audience's needs and preferences.

Setting up a test

1. If you want to set up A/B/n testing in an active journey, pause it first. To learn more about editing an active journey, read this article.

2. Set at least one Conversion Goal that will be used for calculating test results. For example, if you want to test which of two push notifications is the most effective, you can set the PW_InAppPurchase Event as the Conversion Goal.

If you specify several Conversion Goals, the results will be calculated for each Goal separately.

3. Place the A/B/n Split element to the canvas after any other element and double-click it to configure the test.

4. Select the number of branches. You can add up to four branches to your test. The first branch is always considered a control group against which results in other branches are compared. Keep following this guide to learn how to set up this branch depending on your needs.

5. Set the percentage of users going to each branch. The total of all branches should be 100%. According to the percentage you set, journey travelers who reach this journey step will be split randomly into the specified number of branches.

Please note that users assigned to Branch A will be considered a control group against which results in other branches will be compared. The next step describes how to set up Branch A for different cases.

You may want to set a higher percentage for Branch A (the original version) to carefully test a new campaign version without risking lower conversions.

However, we recommend setting up an even distribution. In this case, you will get statistically significant results faster.

6. Now you can set up communication flows for each branch. Since Branch A is a control group, you need to configure it in a certain way depending on your case:

• Test the existing communication flow against new versions. In this case, the existing version should be in Branch A. Also, ensure the Conversion Goal can be completed in any branch, including Branch A.

• Test only new versions against each other. Since there is no old communication flow, one of the new versions will be marked as a control group (the one presented in Branch A). Make sure the Conversion Goal can be completed in all branches.

• Test an existing version without any communication against new communication flows. For example, you want to add one more push at the end of a journey to see if the campaign performance increases compared to the original version without this push. In this case, the ‘empty’ version should be presented in Branch A. To get reliable results, add an equal Time Delay to each branch:

7. If everything is ready, launch the journey.

Analyzing the results

As soon as the test is launched, you can see real-time statistics on it.

On the journey canvas, you can see the number of users who pass through each branch within the journey:

To view the test results, double-click on the A/B/n Split element. You will see the statistics for each branch, including the conversion to the selected goal and the metrics used to calculate statistical significance.

If you have set several Conversion Goals, you can view the results for each of them by selecting the required Goal from the list:

If your test has a statistically significant winner, you can instantly turn off less-performing branches by clicking Keep only winner. New journey travelers will be directed to the winning branch. Users who already went to deactivated branches will continue going through them until they complete the journey.

You can also deactivate branches manually, but this will require you to pause the journey.

You can view test results for a specific period of time. To do this, select the desired period on the left panel and double-click on the A/B/n Split element. You will see the statistics for the specified time.

The test results are calculated as follows:

• The conversion is calculated based on the number of users in the branch and the Conversion Goals completed. Branch A will be marked as the Baseline, as it represents the control group.

• When calculating statistical significance, Z-Score and P-Value are taken into account. If the result is valid, it will be marked as Significant.

• A branch is considered the winner if it has the highest conversion, and this result is marked as Significant.

If the result is marked as Significant, it means that its confidence level is 90%. If this percentage is not enough for you, you can calculate the result by specifying a different confidence level. For example, you can use this calculator.

Deactivating branches

If Pushwoosh has identified a statistically significant winner, you can instantly turn off less-performing branches by clicking Keep only winner in the test statistics window. In this case, you don't need to pause the journey.

If some branch performs poorly, you can deactivate it. New users will no longer get into the deactivated branch.

To deactivate a branch, pause the journey first:

After that, double-click the A/B/n Split element. Set 0% for this branch and redistribute the percentage of users going to the remaining branches: