Are your email subject lines not getting the attention they deserve? Are you struggling to improve open rates and engagement with your email campaigns? Look no further than the power of A/B testing. 

Under the guidance of an expert digital agency like Schaefer, A/B testing subject lines is a game-changer, allowing you to optimize your email performance to determine the highest converting keywords and phrases, in order to drive up open and click rates and increase conversions. 

What is A/B Testing?

A/B testing, also known as split testing, is an invaluable technique in marketing that involves testing two different versions of content on an audience. These versions, labeled “version A” and “version B,” are nearly identical except for one key detail: a subject line, a call-to-action, an image, the sender’s name, an element of the email content, or even the time of sending. 

Start with Subject Lines

“If people are unwilling to open the emails you send them, the rest of your marketing campaign is simply going to fall flat.”

Mailchimp

Subject lines are a great starting point in implementing A/B testing in your marketing efforts, as it provides an opportunity to test different theories about what encourages your contacts to open your emails. Additionally, since you’re testing on your actual audience, your expert agency can then utilize the personalized insights to optimize your future campaigns. 

Here are some ideas of what can be tested in a subject line split test:

• Length
  • Does a shorter or longer subject line length perform better?
• Personalization
  • Does including the contact’s first name resulted in improved metrics? Does it have no discernible effect?
• Specific Word Inclusions
  • How does including your company name, core values, specific products or services, or other key words affect the results of the email?
• Point of View
  • Does your audience respond to first-person, second-person, or third-person subject lines better?
•  Sentence case vs. Title Case
  • How does capitalization affect your email metrics? 
• Offer
  • What happens if you tease the inclusion of a discount code within the email, in the subject line?

You may even lean on the assistance of AI like Klaviyo’s Subject Line Assistant or Jasper.ai for subject line generation. When used under the guidance of an experienced marketing partner, this tool can increase efficiency and speed in conducting an A/B test on subject lines.

Analyzing the Results

After an A/B test concludes, it is important to analyze the data and provide recommendations. A skilled agency partner possesses the expertise to identify changes in the essential metrics and apply the results to future emails, for improved contact engagement and customer growth. 

Your expert partner will focus on the following key points:

1. Focus on a couple of key metrics.  Trying to analyze too many metrics can lead to many different conclusions. Hone in on the results that were likely to be impacted by the A/B subject line test – open and clickrates:
   1. Open rate – the number of emails opened divided by the total number of emails delivered, excluding bounces or other delivery failures.
   2. Click rate – the number of unique clicks on a link divided by the total number of delivered emails, excluding bounces or other delivery failures.
      1. Since recent privacy law changes have affected the accuracy of open rate, click rate is another way to see the people that actually opened an email – by looking at the click rate.
2. Retest, and retest again. Even a statistically significant result could be a false positive. Test it again to see if the results can be mimicked. 
3. Patience. Email marketing is composed of longer results cycles than other types of digital advertising (like display ads). Consider running the test multiple times, and waiting up to seven days after an email is sent before compiling results.

A/B testing subject lines offers powerful potential benefits, like decreasing bounce rate, increasing open/click rate, and increasing user engagement. However, this process takes time – from setting up the test, to running it, to reviewing the results, and finally to implementing those insights into future tests. To fully leverage the advantages possible from A/B testing subject lines, it is crucial to collaborate with an expert marketing agency. 

At Schaefer, we strategically analyze the results of our A/B test variations to gain crucial insights into what resonates best with each individual client’s specific subscribers, and then use that knowledge to improve the overall effectiveness of future messaging, in order to drive business growth.

It’s been just over a year since Apple released iOS 15 with its intensified consumer privacy settings that had a huge effect on email marketing open rate reporting. Over the past year, email management platforms have continued to make updates to account for these changes, and we can expect to see more in the future, as consumer privacy continues to be a high priority.

Although the most significant impact is a veritable loss of trust in the open rate metric, a standard to measure email effectiveness by marketers everywhere, it’s not a complete setback. Open rates were never 100% accurate anyways – autoresponders, anti-spam filters, and message clipping impacted the accuracy even before iOS 15 hit consumer phones. Additionally, this change pushes marketers to rely more heavily on more dependable metrics, such as click rates, click-through rates, delivered rates, unsubscribe rates, and overall audience growth and website traffic. Email management platforms have also released additional metrics or reporting capabilities to either filter out Apple Mail users, or account for machine-opens.

First, a refresher on iOS 15:

The iOS 15 update came with two main additions to consumer privacy options:

• Mail Privacy Protection: Apple Mail users can opt-in to mail privacy settings to hide their IP addresses and block third parties from tracking email opens.
• iCloud+: Paid iCloud+ users can choose to monitor websites with whom they’re sharing information, block third parties from tracking their website visits on Safari, and hide their email addresses.

The Mail Privacy Protection has been the most relevant update, as the iCloud+ increased privacy is only available to paid users – though both have and will continue to affect marketing reporting.

Who is affected by this update?

This only applies to Apple Mail Users – whether someone actively checks their mail in Apple Mail, or even if they just set it up initially.

How are email management platforms responding?

Hubspot:

• Hubspot created a metric called “adjusted open rate” that is calculated by dividing unique reliable opens, by the result of unique unreliable opens subtracted from unique delivered.
  
  • Adjusted open rate = Unique reliable opens / (Unique delivered – Unique unreliable opens)
• Adjusted open rate is an estimation, as Hubspot calculates unreliable opens by their estimation of the number of times an email was opened by a bot, but they cannot know for sure whether it was a bot or a human. Adjusted open rate will be equal to or higher than the standard open rate, and can be seen within the performance tab of individual emails. It cannot be used as a filter in lists or within workflows.
• Hubspot also offers the ability to view percentages of opens broken down by email client in the performance tab of individual emails.

Constant Contact:

• Constant Contact changed how they calculate clickthrough rate, now dividing the number of clicks by the number of delivered emails – instead of their previous calculation of dividing by the number of opened emails.
  • Clickthrough rate = Number of clicks / Number of delivered emails
• They do not currently offer the ability to view a breakdown of opens by email client within their reporting, nor any other adjustments for this change.

Mailchimp:

• Mailchimp updated how they calculate contact ratings (though no specifics on how that was done have been provided), so “email marketing engagement” metrics can still be used for segmentation and automation triggers.
• They do not currently offer the ability to view a breakdown of opens by email client within their reporting, nor any other adjustments for this change.

Pardot:

• Pardot has an “Open Rules Audit” within Automations for marketers to review where email opens are used in automations throughout the account, and update to another metric.
• Pardot also offers a report tab for “Email clients” within individual list emails and the overall list email report. While marketers can not see the specific contacts that use Apple Mail for example, it can be used to gauge a general overview of the portion of the audience that uses Apple Mail.

Klaviyo:

• Klaviyo was one of the first email management platforms to release updates to their system in response to iOS 15, and since then, they’ve added additional options to assist marketers with this change.
• Within the “Opened Email event” in Klaviyo, there is a connected property of “Apple Privacy open.” Klaviyo calls this a ‘flag.’ Essentially, this flag allows users to segment their customers based on whether or not their opens came from a bot with Apple Mail’s Privacy Protection.
  • Marketers can use this flag within segments and workflows to adjust the open rate filter to only be for non-Apple privacy opens, or any way desired. However, similar to Hubspot, this flag is an estimation of bot opens, based off of all opens on the Apple Mail app on a device that has at least iOS 15 installed and the mail privacy protections enabled, but they cannot know for sure if it was opened by a bot or human.
• Marketers also have the ability to set their revenue attribution from emails to be based on: opened or clicked, opened or clicked (excluding Apple Privacy Opens), or just clicked.
• Additionally, Klaviyo offers the metrics of “Total Apple Privacy Opens” and “Unique Apple Privacy Opens” as metrics in their custom reports and single-metric reports, and the report “Email Opens by Email Client” to view how email opens compare across different email clients.

How should marketers change their reporting, if it all?

Regarding this change to the open rate metric, marketers should take time to re-examine how they report on email success. Look to other metrics that can be relied upon to account for actions taken within an email, as well as other tracking options that can be tied to email:

• Click rate
• Click-through rate
• Delivered rate
• Unsubscribed rate
• Audience list growth
• Website traffic from email campaigns

Instead of focusing on how many contacts open an email, judge an email’s success by:

1. Tracking how many emails are successfully delivered
2. Reviewing how the audience is engaging and clicking through emails
3. Identifying how many users are removing themselves from the audience after an email send.
4. Identifying how the audience is growing or decreasing over time
5. Comparing web traffic from email campaigns across deployments

Understanding the measurement in these key metrics can give insight into what specific aspects of your email are most successful in driving the most desirable engagement.  Noting the differences in subject lines, creative, and calls-to-action across email deployments alongside email metrics will help identify your high performing content.

Additionally, through proper analytics tracking using UTM links, traffic driven to websites from email deployments can be tracked from the first click to the final conversion on whichever analytics platform you use, from Google Analytics to Adobe Analytics. Benchmarking email traffic performance against other paid channels can allow for insight into how different types of emails drive meaningful traffic and more importantly, meaningful engagement, to your site.

However, it’s also important to not fully ignore the open rate metric.

While this metric should be taken with a grain of salt, it is still recommended to monitor open rate and how it changes over time and over email campaigns.

• Use an adjusted benchmark, or filter out Apple Mail opens – it is still possible to gauge whether a significant portion of the audience has opened an email and/or if the subject line needs work, with the open rate metric.
• The importance of the open rate metric can also be intertwined with the goal of your email deployment. Do you want to drive conversions from your email? Or do you simply want to raise awareness? If it’s the latter, open rate can still be a useful metric to consider, whereas with the former, it might not matter quite as much.

What other steps should marketers take?

Take a look at current marketing efforts. Are there lists and segments, workflows, or reports that are set up to rely on open rate? Consider updating those to use click rate or click-through rate. Be aware of the fact that any specific reporting metrics within individual email clients that use open rate in their calculation will be skewed, and tools such as re-sends to non-openers, send time optimization, and location targeting, for example, will treat Apple Mail bot opens as human opens.

Focus on improving email deliverability and sender reputation. Frequently clean email lists, provide unsubscribe and preference management links within emails, rely on personalization and segmentation wherever possible, and make sure emails are aligned with any and all legal requirements, such as ADA and GDPR.

Get ahead of updates, and make the shift to rely on first-party data as much as possible now. Consider building additional tools into your organization’s wheelhouse, such as SMS, paid social media marketing, developing organic SEO, and more, all of which offer different ways to engage with contacts outside of email.

What now?

Schaefer has already been reporting on the other relevant metrics to email besides open rate, as well as making an effort to prioritize email deliverability and sender reputation for our clients. With the additional available updates, Schaefer will continue to report on all metrics, as well as those additional metrics where available – like adjusted open rate in Hubspot. Schaefer will also adjust to use click rates with segments, workflows and reports, and keep an eye on any email-client-specific tools that use open rate, in order to ensure data is as clean as possible. As data privacy continues to call for new standards in marketing reporting, Schaefer will continue to adjust. 

Frequent new developments are expected in marketing, and the iOS 15 update hasn’t been the only change to user privacy over the last year or so. Along with Google’s plan to go cookieless in 2023 and the increased consumer privacy seen with the sunset of Universal Analytics and subsequent shift to GA4, marketers can expect to have to consider higher expectations for data privacy across many instances in their organizations as new developments occur and updates are announced. Schaefer recommends being fully transparent with your clients, and educating yourself on this and other privacy changes as they come, so you can offer support and relevant information on how reporting and metrics will or could change.