Do you ever have a problem deciding between a few different subject lines? Or maybe you were sure that one was better, but a co-worker disagreed? Well, maybe you're both wrong. With subject line A/B splits, you can take the guess work out of fine tuning your messages. This Brontoversity Blog topic will walk you through the basics of setting up a subject line A/B split test and help you understand how this powerful feature can help improve your email marketing campaign.

In its most simple form, a subject line A/B split test starts with two groups chosen randomly from all of your targeted contacts for a delivery. You then send a message to those groups with a variable subject line in each. After all the responses have come in, review the results and identify which version performed better. You can then use that information to drive future marketing. A/B split tests within the application allow you to automate the tedious steps of testing different message types and provides access to valuable testing data with a minimal investment of your time.

There are two main types of subject line A/B split tests that you can use:


A/B-Winner split tests involve testing messages with two or more groups to see which is most effective. After a set period of time, a winner is chosen based off of preselected criteria, such as highest open rate or highest number of clicks. After the winner is selected, the winning message is sent to the remainder of your targeted contacts.


Champion/Challenger is quite similar to A/B-Winner, the difference being that you don't have a winner group. In this type of test you have an established way of doing something, called the Champion, and you want to test if it is still the best by introducing a Challenger. When creating a Champion/Challenger split test, it is common to send most of your contacts the champion, while setting aside a smaller portion for the challenger. Once the test is run, there is no winner group who receives a message, but you will have a better idea about whether or not your champion is as good as you thought it was.

The Setup

1. Click the Send A/B Split button on the Message Overview page for particular message in Bronto. 

2. Choose Subject Line from the Split Content pull-down menu in the Perform Test On section.

3.  Choose whether you want to perform an A/B - Winner or Champion Challenger split test. 

4.  Next you need to specify your groups.  Remember that the groups are randomly chosen from your targeted group of contacts, which you'll be selecting later. What you are setting here is just the size of those groups, which is done in percentages of the whole group. For A/B-Winner, it's a good idea to try and keep the total for your test groups to under 50% of your list. If you are just using two groups, then 20% and 20% is good; you should reduce the size of the groups as you add more. As you create groups and specify their sizes, you will see that the Winner group is automatically updated to contain the remainder of the percentage. 

For Champion/Challenger there is no winner group, so you need to allocate a full 100% amongst the groups you create. You can create a maximum of 5 test groups. 

5.  On the next page you will assign the content for each group. 

6.  After you enter the content for each test group or select an alternate message version, you will be brought into the familiar delivery scheduling page. In delivery scheduling, you will need to specify the Sender Options, select the lists and segments that you would like to have receive this message, and pick a time for the test groups to be sent out. For an A/B-Winner split test you will also need to pick a time for the winner group to be sent. You should give enough time for your message to get out and for people to open, click, and even convert. Remember that if you throttle the message, it may cause it to take more time for the message to go out. Even with a throttle of 2-4 hours, sending the winner 4-6 hours later should be fine.

7.  Once you have selected the send times, you will be brought to a verification screen where you should double check all of your selections. Click the Schedule Delivery button and your A/B split test is ready to go out.

Please feel free to share any experiences with Subject Line A/B split tests and what results you noticed in your campaigns.

Rob Slade
Client Support Specialist at Bronto