Try Your Hand at A/B Testing for a Chance to Win the Email Subject Line Contest

Posted by danielburstein

This blog post ends with an opportunity for you to win a stay at the ARIA in Vegas and a ticket to
Email Summit, but it begins with an essential question for marketers…

How can you improve already successful marketing, advertising, websites and copywriting?

Today’s Moz blog post is unique. Not only are we going to teach you how to address this challenge, we’re going to offer an example that you can dig into to help drive home the lesson.

Give the people what they want

Some copy and design is so bad, the fixes are obvious. Maybe you shouldn’t insult the customer in the headline. Maybe you should update the website that still uses a dot matrix font.

But when you’re already doing well, how can you continue to improve?

I don’t have the answer for you, but I’ll tell you who does – your customers.

There are many tricks, gimmicks and technology you can use in marketing, but when you strip away all the hype and rhetoric, successful marketing is pretty straightforward –
clearly communicate the value your offer provides to people who will pay you for that value.

Easier said than done, of course.

So how do you determine what customers want? And the best way to deliver it to them?

Well, there are many ways to learn from customers, such as focus groups, surveys and social listening. While there is value in asking people what they want, there is also a major challenge in it. “People’s ability to understand the factors that affect their behavior is surprisingly poor,” according to research from Dr. Noah J. Goldstein, Associate Professor of Management and Organizations, UCLA Anderson School of Management.

Or, as Malcolm Gladwell more glibly puts it when referring to coffee choices, “The mind knows not what the tongue wants.”

Not to say that opinion-based customer preference research is bad. It can be helpful. However, it should be the beginning and not the end of your quest.

…by seeing what they actually do

You can use what you learn from opinion-based research to create a hypothesis about what customers want, and then
run an experiment to see how they actually behave in real-world customer interactions with your product, marketing messages, and website.

The technique that powers this kind of research is often known as A/B testing, split testing, landing page optimization, and/or website optimization. If you are testing more than one thing at a time, it may also be referred to as multi-variate testing.

To offer a simple example, you might assume that customers buy your product because it tastes great. Or because it’s less filling. So you could create two landing pages – one with a headline that promotes that taste (treatment A) and another that mentions the low carbs (treatment B). You then send half the traffic that visits that URL to each version and see which performs better.

Here is a simple visual that Joey Taravella, Content Writer, MECLABS create to illustrate the concept…

That’s just one test. To really learn about your customers, you must continue the process and create a testing-optimization cycle in your organization – continue to run A/B tests, record the findings, learn from them, create more hypotheses, and test again based on these hypotheses.

This is true marketing experimentation, and helps you build your theory of the customer.

But you probably know all that already. So here’s your chance to practice while helping us shape an A/B test. You might even win a prize in the process.

The email subject line contest

The Moz Blog and MarketingExperiments Blog have joined forces to run a unique marketing experimentation contest. We’re presenting you with a real challenge from a real organization (VolunteerMatch) and
asking you to write a subject line to test (it’s simple, just leave your subject line as a comment in this blog post).

We’re going to pick three subject lines suggested by readers of The Moz Blog and three from the MarketingExperiments Blog and run a test with this organization’s customers. Whoever writes the best performing subject line will
win a stay at the ARIA Resort in Las Vegas as well as a two-day ticket to MarketingSherpa Email Summit 2015 to help them gain lessons to further improve their marketing.

Sound good? OK, let’s dive in and tell you more about your “client”…

Craft the best-performing subject line to win the prize

Every year at Email Summit, we run a live A/B test where the audience helps craft the experiment. We then run, validate, close the experiment, and share the results during Summit as a way to teach about marketing experimentation. We have typically run the experiment using MarketingSherpa as the “client” website to test (MarketingExperiments and MarketingSherpa are sister publications, both owned by MECLABS Institute).

However, this year we wanted to try something different and interviewed three national non-profits to find a new “client” for our tests.

We chose
VolunteerMatch – a nonprofit organization that uses the power of technology to make it easier for good people and good causes to connect. One of the key reasons we chose VolunteerMatch is because it is an already successful organization looking to further improve. (Here is a case study explaining one of its successful implementations – Lead Management: How a B2B SaaS nonprofit decreased its sales cycle 99%).

Another reason we chose VolunteerMatch for this opportunity is that it has three types of customers, so the lessons from the content we create can help marketers across a wide range of sales models. VolunteerMatch’s customers are:

  • People who want to volunteer (B2C)
  • Non-profit organizations looking for volunteers (non-profit)
  • Businesses looking for corporate volunteering solutions (B2B) to which it offers a Software-as-a-Service product through VolunteerMatch Solutions

Designing the experiment

After we took VolunteerMatch on as the Research Partner “client,” Jon Powell, Senior Executive Research and Development Manager, MECLABS, worked with Shari Tishman, Director of Engagement and Lauren Wagner, Senior Manager of Engagement, VolunteerMatch, to understand their challenges, take a look at their current assets and performance, and craft a design of experiments to determine what further knowledge about its customers would help VolunteerMatch improve performance.

That design of experiments includes a series of split tests – including the live test we’re going to run at Email Summit, as well as the one you have an opportunity to take part in by writing a subject line in the comments section of this blog post. Let’s take a look at that experiment…

The challenge

VolunteerMatch wants to increase the response rate of the corporate email list (B2B) by discovering the best possible messaging to use. In order to find out, MarketingExperiments wants to run an A/B split test to determine the
best messaging.

However the B2B list is relatively smaller than the volunteer/cause list (B2C) which makes it harder to test in (and gain
statistical significance) and determine which messaging is most effective.

So we’re going to run a messaging test to the B2C list. This isn’t without its challenges though, because most individuals on the B2C list are not likely to immediately connect with B2B corporate solutions messaging.

So the question is…

How do we create an email that is relevant (to the B2C list), which doesn’t ask too much, that simultaneously helps us discover the most relevant aspect of the solutions (B2B) product (if any)?

The approach – Here’s where you come in

This is where the Moz and MarketingExperiments community comes in to help.

We would like you to craft subject lines relevant to the B2C list, which highlight various benefits of the corporate solutions tool.

We have broken down the corporate solutions tool into three main categories of benefit for the SaaS product.
In the comments section below, include which category you are writing a subject line for along with what you think is an effective subject line.

The crew at Moz and MarketingExperiments will then choose the top subject line in each category to test. Below you will find the emails that will be sent as part of the test. They are identical, except for the subject lines (which you will write) and the bolded line in the third paragraph (that ties into that category of value).

Category #1: Proof, recognition, credibility


Category #2: Better, more opportunities to choose from


Category #3: Ease-of-use

About VolunteerMatch’s brand

Since we’re asking you to try your hand at crafting messaging for this example “client,” here is some more information about the brand to inform your messaging…


VolunteerMatch’s brand identity


VolunteerMatch’s core values

Ten things VolunteerMatch believes:

  1. People want to do good
  2. Every great cause should be able to find the help it needs
  3. People want to improve their lives and communities through volunteering
  4. You can’t make a difference without making a connection
  5. In putting the power of technology to good use
  6. Businesses are serious about making a difference
  7. In building relationships based on trust and excellent service
  8. In partnering with like-minded organizations to create systems that result in even greater impact
  9. The passion of our employees drives the success of our products, services and mission
  10. In being great at what we do

And now, we test…

To participate, you must leave your comment with your idea for a subject line before midnight on Tuesday, January 13, 2015. The contest is open to all residents of the 50 US states, the District of Columbia, and Canada (excluding Quebec), 18 or older. If you want more info, here are the
official rules.

When you enter your subject line in the comments section, also include which category you’re entering for (and if you have an idea outside these categories, let us know…we just might drop it in the test).

Next, the Moz marketing team will pick the subject lines they think will perform best in each category from all the comments on The Moz Blog, and the MarketingExperiments team will pick the subject lines we think will perform the best in each category from all the comments on the MarketingExperiments Blog.

We’ll give the VolunteerMatch team a chance to approve the subject lines based on their brand standards, then test all six to eight subject lines and report back to you through the Moz and MarketingExperiments blogs which subject lines won and why they won to help you improve your already successful marketing.

So, what have you got? Write your best subject lines in the comments section below. I look forward to seeing what you come up with.

Related resources

If you’re interested in learning more about marketing experimentation and A/B testing, you might find these links helpful…

And here’s a look at a previous subject line writing contest we’ve run to give you some ideas for your entry…


Sign up for The Moz Top 10, a semimonthly mailer updating you on the top ten hottest pieces of SEO news, tips, and rad links uncovered by the Moz team. Think of it as your exclusive digest of stuff you don’t have time to hunt down but want to read!

Reblogged 4 years ago from moz.com

Is that Mind-Blowing Title Blowing Your Credibility? You Decide

Posted by Isla_McKetta


Image of Tantalus courtesy of Clayton Cusak

What if I told you I could teach you to write the perfect headline? One that is so irresistible every person who sees it will click on it. You’d sign up immediately and maybe even promise me your firstborn.

But what if I then told you not one single person out of all the millions who will click on that headline will convert? And that you might lose all your credibility in the process. Would all the traffic generated by that “perfect” headline be worth it?

Help us solve a dispute

It isn’t really that bad, but with all the emphasis lately on
headline science and the curiosity gap, Trevor (your faithful editor) and I (a recovering copywriter) started talking about the importance of headlines and what their role should be in regards to content. I’m for clickability (as long as there is strong content to back the headline) and, if he has to choose, Trevor is for credibility (with an equal emphasis on quality of the eventual content).

credible vs clickable headlines

What’s the purpose of a headline?

Back in the good ol’ days, headlines were created to sell newspapers. Newsboys stood on street corners shouting the headlines in an attempt to hawk those newspapers. Headlines had to be enough of a tease to get readers interested but they had to be trustworthy enough to get a reader to buy again tomorrow. Competition for eyeballs was less fierce because a town only had so many newspapers, but paper cost money and editors were always happy to get a repeat customer.

Nowadays the competition for eyeballs feels even stiffer because it’s hard to get noticed in the vast sea of the internet. It’s easy to feel a little desperate. And it seems like the opportunity cost of turning away a customer is much lower than it was before. But aren’t we doing content as a product? Does the quality of that product matter?

The forbidden secrets of clickable headlines

There’s no arguing that headlines are important. In fact, at MozCon this year,
Nathalie Nahai reminded us that many copywriters recommend an 80:20 ratio of energy spent on headline to copy. That might be taking things a bit far, but a bad (or even just boring) headline will tank your traffic. Here is some expert advice on writing headlines that convert: 

  • Nahai advises that you take advantage of psychological trigger words like, “weird,” “free,” “incredible,” and “secret” to create a sense of urgency in the reader. Can you possibly wait to read “Secret Ways Butter can Save Your Life”?
  • Use question headlines like “Can You Increase Your Sales by 45% in Only 5 Minutes a Day?” that get a reader asking themselves, “I dunno, can I?” and clicking to read more.
  • Key into the curiosity gap with a headline like “What Mother Should Have Told You about Banking. (And How Not Knowing is Costing You Friends.)” Ridiculous claim? Maybe, but this kind of headline gets a reader hooked on narrative and they have to click through to see how the story comes together.
  • And if you’re looking for a formula for the best headlines ever, Nahai proposes the following:
    Number/Trigger word + Adjective + Keyword + Promise = Killer Headline.

Many readers still (consciously or not) consider headlines a promise. So remember, as you fill the headline with hyperbole and only write eleven of the twelve tips you set out to write, there is a reader on the other end hoping butter really is good for them.

The headline danger zone

This is where headline science can get ugly. Because a lot of “perfect” titles simply do not have the quality or depth of content to back them.

Those types of headlines remind me of the Greek myth of Tantalus. For sharing the secrets of the gods with the common folk, Tantalus was condemned to spend eternity surrounded by food and drink that were forever out of his reach. Now, content is hardly the secrets of the gods, but are we tantalizing our customers with teasing headlines that will never satisfy?

buzzfeed headlines

For me, reading headlines on
BuzzFeed and Upworthy and their ilk is like talking to the guy at the party with all those super wild anecdotes. He’s entertaining, but I don’t believe a word he says, soon wish he would shut up, and can’t remember his name five seconds later. Maybe I don’t believe in clickability as much as I thought…

So I turn to credible news sources for credible headlines.

washington post headlines

I’m having trouble deciding at this point if I’m more bothered by the headline at
The Washington Post, the fact that they’re covering that topic at all, or that they didn’t really go for true clickbait with something like “You Won’t Believe the Bizarre Reasons Girls Scream at Boy Band Concerts.” But one (or all) of those things makes me very sad. 

Are we developing an immunity to clickbait headlines?

Even
Upworthy is shifting their headline creation tactics a little. But that doesn’t mean they are switching from clickbait, it just means they’ve seen their audience get tired of the same old tactics. So they’re looking for new and better tactics to keep you engaged and clicking.

The importance of traffic

I think many of us would sell a little of our soul if it would increase our traffic, and of course those clickbaity curiosity gap headlines are designed to do that (and are mostly working, for now).

But we also want good traffic. The kind of people who are going to engage with our brand and build relationships with us over the long haul, right? Back to what we were discussing in the intro, we want the kind of traffic that’s likely to convert. Don’t we?

As much as I advocate for clickable headlines, the riskier the headline I write, the more closely I compare overall traffic (especially returning visitors) to click-throughs, time on page, and bounce rate to see if I’ve pushed it too far and am alienating our most loyal fans. Because new visitors are awesome, but loyal customers are priceless.

Headline science at Moz

At Moz, we’re trying to find the delicate balance between attracting all the customers and attracting the right customers. In my first week here when Trevor and Cyrus were polling readers on what headline they’d prefer to read, I advocated for a more clickable version. See if you can pick out which is mine…

headline poll

Yep, you guessed it. I suggested “Your Google Algorithm Cheat Sheet: Panda, Penguin, and Hummingbird” because it contained a trigger word and a keyword, plus it was punchy. I actually liked “A Layman’s Explanation of the Panda Algorithm, the Penguin Algorithm, and Hummingbird,” but I was pretty sure no one would click on it.

Last time I checked, that has more traffic than any other post for the month of June. I won’t say that’s all because of the headline—it’s a really strong and useful post—but I think the headline helped a lot.

But that’s just one data point. I’ve also been spicing up the subject lines on the Moz Top 10 newsletter to see what gets the most traffic.

most-read subject lines

And the results here are more mixed. Titles I felt like were much more clickbaity like “Did Google Kill Spam?…” and “Are You Using Robots.txt the Right Way?…” underperformed compared to the straight up “Moz Top 10.”

While the most clickbaity “Groupon Did What?…” and the two about Google selling domains (which was accurate but suggested that Google was selling it’s own domains, which worried me a bit) have the most opens overall.

Help us resolve the dispute

As you can tell, I have some unresolved feelings about this whole clickbait versus credibility thing. While Trevor and I have strong opinions, we also have a lot of questions that we hope you can help us with. Blow my mind with your headline logic in the comments by sharing your opinion on any of the following:

  • Do clickbait titles erode trust? If yes, do you ever worry about that affecting your bottom line?
  • Would you sacrifice credibility for clickability? Does it have to be a choice?
  • Is there such thing as a formula for a perfect headline? What standards do you use when writing headlines?
  • Does a clickbait title affect how likely you are to read an article? What about sharing one? Do you ever feel duped by the content? Does that affect your behavior the next time?  
  • How much of your soul would you sell for more traffic?

Sign up for The Moz Top 10, a semimonthly mailer updating you on the top ten hottest pieces of SEO news, tips, and rad links uncovered by the Moz team. Think of it as your exclusive digest of stuff you don’t have time to hunt down but want to read!

Reblogged 5 years ago from feedproxy.google.com