“We Have a 57% Failure Rate” – A/B Testing at Ladder

A/B testing in real life.

Every marketer, product manager, or ‘growth hacker’ you’ve ever met has preached to you about the importance of it… but it’s ridiculously hard to find any real information about its effectiveness.

Well, I have the information you’re looking for.

I analyzed the results of 330 marketing tests we ran at Ladder for 16 clients – from Y Combinator-backed startups to Global Corporations – to check our failure rate.1

The results will shock you.

Flip a coin.

Heads? You lose.

Now imagine you spent hours looking at historical coin flip data, carefully planning your coin flipping strategy, and you (plus the rest of your team) were sure that you’d come up tails.

You were ready to bet your job on the result… and yet… it came up heads.

Damn.

Welcome to modern marketing.

Except the odds are even worse than that coin flip.

57% – That’s how many of our marketing tests at Ladder have failed.

Why do more than 50% of tests fail at moving the needle?

It’s not the people…

We have a grueling interview process involving an Excel test, a copywriting test, a logical reasoning test, a video submission, rounds of interviews, and a (paid) day working on actual client work in our office before we hire someone. (plus we throw away half the applications at random to ensure we don’t hire anyone who’s unlucky…)

It’s not the process…

We’ve spent a lot of time designing our process. Our strategists crunch the numbers before deciding on what growth tactics they want to test. They fill in a test script complete with the hypothesis they’re testing and how they’ll report on performance. They even ‘stress-test’ their ideas in front of a panel of their peers and senior Ladder leadership before they pitch it to the client, who gives their final approval before launch.

It’s not the products…

Our clients are some of the fastest growing startups and innovative corporations out there. Their users love them, retention is high, and their team executes to a world class level. Despite all the failed experiments, the vast majority of our relationships still drive positive ROI growth.

So what on Earth is going on?

The truth is… growing a business is hard.

That’s why it’s such a valuable skill. You don’t think Silicon Valley startups pay people with silly titles like ‘Growth Hacker’ over $139,000 a year out of kindness, do you? That’s why Facebook is worth $341b (growth) and Yahoo is selling for $5b (no growth) despite once being worth $125b (when it still had growth).

Even the experts struggle with growth. Noah Kagan, an early employee at Facebook and founder of AppSumo, only sees a 1 in 8 success rate. An industry study by VWO shows a 1 in 7 success rate. A Study by Harvard Business Review pegs the failure rate at 80-90%.

So, how can you grow a successful business when you’re failing more than you’re winning?

The trick? Lose small, win BIG.

How do you lose small? Through A/B testing.

Make a small bet by A/B testing something you think will work to grow your business, and only double down on it when you see it working.

Everything is better when you hedge your bets with A/B Testing.

  • You aren’t ‘betting the house’ on every decision. You’re making lots of small bets that won’t hurt you if they fail.
  • You’re tracking the results of your actions so you know which tactics to double down on (and which to abandon).
  • You’re actually learning what works (or not) rather than hoping and guessing, and you just get better, faster.

The fact that you even KNOW your failure rate puts you miles ahead of the average marketer, who just does stuff without testing it. They live their lives in blissful ignorance of how many decisions they’re getting wrong, and they never improve.

If their companies grow at all, it’s down to luck.

Smart Marketers make their own luck, by A/B testing.


A/B testing examples.

Don’t blow the whole budget on one ad campaign: start with a few hundred dollars a week and only scale up if it works.

Don’t roll out a new feature to every user: choose a small subsection of users and see if it even moves the needle before rolling it out.

Don’t redesign the homepage ‘just because’: run the numbers to see if it’s a worthwhile exercise vs. all the other actions you could take.

Don’t try every tactic under the sun: model your marketing funnel and pick tactics that can realistically plug the gaps where people drop off.

Don’t make 15 changes at once: systematically test one element at a time to truly learn its relative importance to performance.

Don’t argue about what ad looks the best: take all the different versions and test them in market to let the data decide what’s best.


A Growth Mindset.

When you approach growth with this mindset, every failure is still a big win because you’ve learned another way that doesn’t work.

“I have not failed. I’ve just found 10,000 ways that won’t work.” – Thomas Edison

A person who never made a mistake, never tried anything new. – Albert Einstein

Sometimes, by losing the battle, you find a new way to win the war. – Donald Trump

When you adopt this mindset, growth is inevitable. It’s just a matter of time before you find the winning combination.


The Sad Truth: We’re Lazy.

The truth is that most people never make the effort to figure out if they’re making the right decisions. Most people just want you to do thing X without ever thinking about whether it’s the right thing to do (“Of course it is! I thought of it and I’m the CEO”) or that maybe thing Y might be a better use of your time.

It ain’t easy: to do this right you have to meticulously keep track of every variable you’re testing and run the numbers to justify each action before you take it. It’s a lot more work and it takes discipline. It’s like losing weight: knowing what you should do isn’t the same as actually doing it.

It’s slow. It’s hard. It’s no fun.

…but it gets results.


You’re Not Born With It

Nobody is born with the knowledge of how to grow a business. If your business is a startup, by definition what you’re attempting hasn’t been done before. You just can’t be right 100% of the time… you’d be lucky to get it right 50% of the time.

If you can’t KNOW the right action, HOW can you justify just doing it without testing it first?

Test, Learn, Repeat.

Get smarter about what you test, based on what you learn, every time you repeat the process.

Compounding insights.

If you can do that, and keep doing it, you’ll win.


Solve your problems by A/B Testing

  • Nobody using your product? Good. Start A/B Testing.
  • Competitor stealing customers? Good. Start A/B Testing.
  • Is your growth stagnant? Good. Start A/B Testing.
  • Are your targets too high? Good. Start A/B Testing.
  • Want that promotion? Good. Start A/B Testing.
  • Chasing a funding round? Good. Start A/B Testing.

You get the idea. Whatever your problem, A/B Testing can help.


*BONUS: Elon Musk even A/B tests his personal relationships. #MaybeTooFar


At Ladder, we test and scale strategic experiments to help high-potential businesses accelerate their growth.
Is that something you need?


  1. I used a sample of 330 marketing tests from 6 months in 2016, across 16 clients. 144 had a negative impact, 42 were inconclusive (treated as a failure) and 144 had a positive impact (statistically significant to a 95% confidence level).