Split testing: What it is and how it transforms your online presence

“Every day without a test is a wasted day. Testing is learning — learning about your audience, learning what works, and why.”

~Peep Laja, conversion optimization expert

Nobody’s perfect, and neither is the stuff we make. There’s always room for improvement.

This is true everywhere, including in your digital marketing efforts. It’s why we keep learning — and why we test.

Split testing is one extremely reliable way to keep improving results for your clients. Here’s what you need to know about this important discipline and how you can use it to transform your clients’ online presence.

What is split testing?

Split testing is an analytics-driven marketing strategy that compares two or more versions of an asset, often a web page or email. It helps you determine which of the two does a better job of achieving a specific outcome. 

Usually, that outcome is a conversion or improvement of some kind: higher click-through rate, signing up for a newsletter, signing up for a trial, buying an ecommerce item, and so on.

How does split testing work?

In split testing, you create two (or more) versions of an asset. Both versions have the same goal but different approaches. Those differing versions are then distributed simultaneously, with website visitors or email subscribers receiving only one of the variations.

Marketing and analytics software (or purpose-built split testing software) then analyzes how well the different versions perform based on the defined criteria or desired action. If version A performs 20% better than version B, then version B gets dumped, and version A becomes the new standard. 

…at least until the next split test! 

Split testing vs. A/B testing vs. multivariate testing

Split testing and A/B testing are similar concepts — in fact, some experts use the terms interchangeably. Others see a difference in focus: A/B testing focuses on the two versions being tested (A and B), while split testing focuses on the people receiving those tests.

Others specify split testing as URL-based, with different groups being directed to distinctly different pages. While A/B testing can be a simpler test, such as email subject lines in an A/B tested email or minor elements like fonts or button color.

For multivariate testing, think about how some PPC advertising works, like Google text ads or responsive display ads. With these, you create multiple headings and short and long text strings, and Google’s AI decides which to combine based on where the ad appears. 

Multivariate testing is a little like that. Multiple pieces of the content or destination have two or more variations, and these get mixed together into random combinations. When you’re doing this kind of testing, you can see which combinations perform better.

The benefits of split testing

So, why do split testing in the first place? Because it delivers deep insights into your audience’s behavior, needs, and preferences.

Here’s a sampling of what split testing can accomplish.

Enhancing user experience

Split testing gives you objective data on the user experience of a website or other piece of digital content. No matter what does or doesn’t happen during the test, you’re guaranteed to learn which version of the split performed better.

By regularly split testing, you’ll identify better- and worse-performing elements in your or your clients’ digital assets. As you get rid of the low performers and continue adding higher-performing elements, the user experience gets better.

Boosting conversion rates and revenue

One of the performance elements you’ll definitely be testing is how well the versions of a web page (or other conversion-oriented asset) convert. One version will always generate more website traffic, conversions, and revenue than the other.

So, just like we saw with the user experience, gradual changes based on split testing should continually boost conversions and sales. In other words, split testing leads to conversion optimization.

Increased user engagement

You already have defined metrics to measure engagement, and now you’ve got two (or more) versions of an asset. By testing multiple versions, you’ll see which ones perform the best according to your engagement metrics and KPIs.

Reducing risks in decision-making

As an agency, you’re well-versed in risk. Every client who comes to you needs your help and expertise, but time and again, they freeze up when you suggest a change that seems too radical to them.

It’s frustrating because you’re the expert. You’ve done this before, and you know it will work, but the client isn’t ready to commit.

Split testing massively reduces the risk of a big change. Yes, the customer has to pay for whatever work goes into the test. But through a series of relatively quick and easy split tests, you can confidently prove that your suggestion will perform. (You can also identify when those risks aren’t panning out before rolling them out at scale.)

Project management software for agencies

Do you need project management software designed especially for agencies?

Check out Teamwork

How to set up a split test

With the right tools in place, split testing is relatively easy to do. Here’s a step-by-step guide on how to set up a split test.

1. Identify the goal of your test 

First, and we can’t stress the importance of this enough, figure out why you’re testing before you start testing.

We know how obvious that sounds — but it’s still too easy to skip. And the confusion and conflict that ensues afterward is never a good time.

The goal could be to solve a big-picture problem (we need more visitors to convert!) or to answer a “why” question (why aren’t visitors converting?). It could also be more tactical, like determining which of two designs is better received.

Make sure everyone involved — agency team members and client stakeholders —  agrees that the stated goal is the right one.

And for streamlined communication and collaboration between teams and clients, try Teamwork.com — the project and client management solution designed for agencies and client work.

2. Choose the elements you want to test

Once you’ve aligned on a goal, it’s time to figure out how you’ll achieve that goal.

The elements involved in a split test can be as simple as the placement of a call to action (CTA) or button. Or they could be as complex as two entirely different website experiences.

There could be some healthy debate on which elements will actually help you achieve the goal you settled on, and that’s okay. 

If the goal is a broad one like “increase website conversions,” both of our examples (CTA button and entire website redesign) could be ways to achieve that goal. The redesign will likely deliver more dramatic results but is also way, way more expensive to set up.

Because there’s no single right answer on what to test for which goal, make sure everyone’s on the same page about what is and isn’t being tested before you proceed.

3. Design your variations

Whatever elements you picked in step two, now it’s time to make the variations. Build those two versions of the landing page or email or other piece of your conversion funnel and get them ready for testing.

Website project plan template

Create websites your clients love. Whether you’re working on a brand new website or a redesign, use our website project plan template to streamline the process from beginning to end.

Try our website project template

4. Determine your sample size and test duration

Next up is determining two technical details: your sample size (how many people need to be included in the test) and your test duration (how long you’ll run the test).

Let’s start with the sample size. You want to reach a specific level of statistical significance, usually 95% (which means there’s only a 5% chance that randomness affected the results). For calculations, you can lean on resources like Optimizely’s calculator or something similar for the best (and most accurate) results.

Usually, your sample size defines your test duration, so run it until you hit the number. But if there are mitigating factors, make sure you define a maximum length for the test.

5. Implement the test using tools or platforms 

Even if you’ve got the coding chops, you don’t want to have to attempt to run these experiments manually. The better option is to use tools made for split testing, like Google Optimize or Optimizely, to create the test. Some marketing automation platforms also include related functionality that could help with testing.

If you’d like to explore more options for A/B testing and split testing tools, check out HubSpot’s roundup of the best testing tools.

6. Monitor the results

The last step is to sit back, relax, and watch the data roll in — sort of.

One mistake we’ve seen here is data camping: sitting there watching the data and then panicking or making a decision too early. 

Remember, you established a sample size and test duration for a reason. Not every item that can be split tested is guaranteed to perform normally in the first hour, day, or even week. 

Certain testing methods, like SEO or organic search, take time to get rolling. If you’re reliant on a page getting indexed by Google, for example, you can’t directly control how quickly Google gets around to doing so.

Another caution here: Make sure you pick the right group to test with. Testing is faster via an email list than organic search, sure. But depending on how the email list was built, it might be a skewed representation of the entire pool of site visits, thereby skewing your test results.

The asset you’re trying to optimize for should have an audience in mind. So make sure you’re testing this audience (or testing from a group that roughly correlates to it).

Once you collect a reasonable amount of data, it’s time to analyze it and start making adjustments to the main page or final product.

Best practices in split testing

Running split tests is a powerful part of marketing campaign management, but it’s easy to get off track. Follow these five best practices as you design split testing for your agency or clients.

Start with a clear objective

Make sure all stakeholders agree on what you’re trying to determine with the split testing process. Designing an effective test is only possible when you’re testing for the right thing (and you know what that thing is).

Sometimes, it’s helpful to frame your objective as a hypothesis. Here are a couple of examples:

• “We think the low landing page conversion rate is due to a confusing call to action.”

• “People are falling out of the funnel because we take too long to introduce the product.”

The objective for each is to figure out why the given metric is too low or how to raise it. The hypothesis points to the suspected reason why, which should then be the focus of the split test.

Test one thing at a time

Next, be sure you’re testing only one thing at a time. That could be a button, or an image, or a section of copy, or an entire landing page. Just make sure you’re staying focused on a specific test that achieves a specific objective, and stick with just one of those at a time.

Here’s an example of why it’s tough to get good results from multiple simultaneous split tests.

Imagine you’re running a split test trying to increase conversion rate optimization on a specific site’s homepage. One theory is that the CTA is weak and not graphically supported (no big clickable call-to-action button). Another is that the header image is uninteresting, and another is that the page’s H1 isn’t compelling enough.

You could run a split test where Version A has one CTA, header image, and H1, and Version B has a different version of each page element. 

You’d eventually learn whether the A or B version performs better. However, the problem is that you wouldn’t learn why. 

You wouldn’t know which of the three changes had the most significant impact on user behavior. So, instead of running three tests in one, test just one of these things at a time.

To be clear, there’s absolutely still a place for testing an entirely new page vs. the existing one. Just stay focused on your goal and make sure the test you’re running actually shows you what you need to know. A whole-page test gives you insight into whole-page performance — it doesn’t say anything specific about that one image or that one button.

Run the test simultaneously

You want your split tests to show comparable results, so run the two examples at the same time. It’s not enough to put up Version A for a week and then sneakily switch it over to Version B. Too many factors change over time for those results to be valid.

For one, it’s nearly impossible to drive traffic to the test at identical levels this way. Subtle economic or calendar shifts can affect audience behavior, too.

But when you send both versions of a website to separate representative groups at the same time, you control for those sorts of factors. 

Don’t stop the test too early

Split testing isn’t free, and it takes time. Many people are tempted to kill the split test after a number of hours or days, thinking they have all the information they need. But doing so squashes their sample size, stopping them from reaching statistically significant levels. 

It could even lead them to make conclusions that are opposite reality. Peep Laja (CXL) explains:

“Any seasoned tester has had plenty of experiences in which a “winning” variation at 80% confidence ends up losing badly after getting pushed live to a site and exposed to more traffic.”

Peep recommends keeping the test going until you’ve reached greater than 95% confidence.

Ensure your results are statistically significant

There’s a mathematical way to determine how confident you can be in the results of your tests. Usually, leaving the test open until you hit the right sample size is all it takes. 

Your clients could get antsy if they feel like this is taking too long, so it’s up to you to educate them on why you need that sample size and the risks of not reaching statistical significance.

Organize your split testing workflows with Teamwork.com

Split testing is a powerful tactic, but it involves some detail-oriented work. Doing split testing right as an agency, scaled across multiple clients, creates a lot of moving parts, and it’s easy to lose track of who’s doing what and when.

Teamwork.com can help. Teamwork.com is a project management platform built to scale agency work. With it, you can organize your split testing workflows, assign tasks, track progress, and (alongside your split testing software) analyze data to produce data-driven results.

Getting started with Teamwork.com is easy — and free. Ready to experience a better approach to project management? Sign up now!