Knowledge Base

A/B Testing

Each time you create a campaign, you may wonder whether your subject line will work or which sender's name should be used. You can use an A/B testing campaign to test how small changes to your campaigns impact your results. 

Soundest provides you with an ability to test your campaigns in two different ways:

  • Testing subject line
  • Testing sender's name and sender's email address

This article explains how A/B testing works and how you can setup and launch an A/B test campaign on Soundest.

Before you start your A/B test

  • A/B testing allows you to choose the best subject line, sender's name or sender's email address for your campaign. You can choose the percentage of recipients A and B versions will be sent to and the winner can either be decided automatically or you can do it manually as well. 
  • Althought A/B testing can only be performed with lists containing 10 contacts or more, we suggest having at least 2000 contacts and splitting it to 25% for A, 25% for B and 50% for the winner. This way you would be sure about the accuracy of your results.
  • As the clicks and opens do not happen immediately, we suggest postponing your winner campaign for 24 hours. Your data will then be accurate enough to correctly choose the winner and your campaign will go out at the same time the A and B versions went out, without making any impact on the results. If that is too long of a wait, postponing for at least couple of hours is suggested.
  • It is up to you to choose, which metric decides the winner: opens or clicks. We suggest using opens.
  • If you change your mind about the test, you can either pause it or select the winner manually and let it through. Once paused, resume option is available for up to 7 days from the start of campaign.
  • A/B test campaigns cannot be boosted.

How A/B test campaigns work


Setting up an A/B test campaign

To start setting up your A/B test campaign, click on A/B testing button on Campaigns tab.

Once you click it, you are brought to the setup wizard. This is where you can change all settings for your A/B campaign. 

Settings

To start the setup, choose what you would like to test: email subject line or sender's name (along with sender's email address).

If you choose to test your subject lines, you are provided with a form with two subject lines for each of the versions. These fields should not be matching and the better performing one will be chosen as a winner. You will be able to set up the preferred portions of A/B and winner campaign later on. See the image below for the Settings step when the subject line testing is chosen.

If you choose to test your sender's name, you are provided with a form with two sender's name and two sender's email fields for each of the versions. These fields should not be matching and the better performing one will be chosen as a winner. You will be able to set up the preferred portions of A/B and winner campaign later on. See the image below for the Settings step when the sender's name testing is chosen.

Once you are done with the Settings tab, click NEXT STEP to move forward.

Template

Template step allows you to choose the template you are going to be working with on your campaign. This is the same step you encounter with regular campaigns, so feel free to choose your favourite template or one of your saved ones. You can change the template chosen before you send your campaign.

Content

This step allows you to edit all the content on your newsletter. As both A and B versions will be using the same content, you only need to add it once. This step does not differ from the regular campaign and you can use all the same settings you find there. Once you are done with the content setup, click Next to move forward.

Recipients

Once you have your content ready, selection of your recipients follows. You can choose to send to all subscribers, particular lists or particular segments. Regardless of which option you choose, A and B version will be sent to randomly chosen contacts from the lists (or segments) chosen. Make sure your chosen list or segment has 10 contacts or more, otherwise A/B testing is not available. Once you're done with it, click NEXT STEP.

A/B settings

A/B settings step allows you to choose your preferences for your testing campaign. First, you can choose the size of your test group. We suggest going with 25/25/50, but you can go as low as 10/10/80 or as high as 50/50. In such case (50/50) there will be no winner campaign and both versions will be sent to an equal amount of recipients. To change these options, click and drag the handle between B and Winner (see image below).

Once you have decided the sizes of test group, you can choose how the winning combination should be determined: by open rate or by click rate. We suggest using open rates for more accurate outcome. You can also choose how long should we wait to decide the winner. We suggest waiting for 24 hours, but you can go as low as 1 hour or as high as 7 days.  

Once you are done with the A/B settings, click NEXT STEP to move forward.

Send

This is the last step of your campaign setup. You can now either schedule this campaign for later or send it immediately. Once you click the send button, the process will be initiated and A and B version will start going out to your test groups. 

If you are testing a campaign with a sender's email address that has not been yet verified, you are noted about that as well. You need to verify it before you can start an A/B testing campaign. Verification process is as simple as clicking on VERIFY button and checking your inbox for the email. Click a verification link there and you are good to go.

Once you're good to go with your sender's email address, feel free to double check all your setup: subject lines, sender's name, chosen lists, and A/B testing preferences. 

You can now either schedule the campaign for later or send it immediately.




Was this article helpful?
Yes No
Thank you for your feedback!