Complete Guide to A/B Testing

January 10, 2023 | APage Studio | Try it free!

What is A/B Testing?

A/B testing (also known as split testing) is a process of showing two variants of the same web page to different segments of website visitors at the same time and comparing which variant drives more conversions.

In the online world, the number of visitors on your website equals the number of opportunities you have to expand your business by acquiring new customers and build relationships by catering to existing ones.

And it is your conversion funnel that decides whether your website gets good traffic and if it converts more visitors.

Businesses want visitors to take action (also called a conversion) on their website, and the rate at which a site can drive this is called its “conversion rate.” The more optimized your funnel, the higher is the conversion rate.

And one of the most important ways to optimize your website’s funnel in digital marketing is A/B testing.

Typically in A/B testing, the variant that gives higher conversions is the winning one, and that variant can help you optimize your site for better results.

The metrics for conversion are unique to each website. For eCommerce, it may be the sale of the products, while for B2B, it may be the generation of qualified leads. A/B testing is one of the components of the overarching process of Conversion Rate Optimization (CRO) using which you can gather both qualitative and quantitative user insights and use them to understand your potential customers and to optimize your conversion funnel based on that data.

Why should you A/B Test?

If B2B businesses today are unhappy with all the unqualified leads they get per month, eCommerce stores, on the other hand, are struggling with a high cart abandonment rate. Meanwhile, media and publishing houses are also dealing with low viewer engagement. These core conversion metrics are affected by some common problems like leaks in the conversion funnel, drop-offs on the payment page, etc.

Let’s see why you should do A/B testing to deal with all these problems:

Solve Visitor Pain Points

Visitors on your website come to achieve a specific goal that they have in mind. It may be to understand more about your product or service, to buy a product, to read/learn more about a particular topic, or simply to browse. Whatever the user’s goal, they may face some common pain points while achieving their goal: it can be a confusing copy or hard to find the CTA button like buy now, request a demo, etc. Not being able to achieve their goals lead to bad user experience. This increases friction and eventually impacts your conversion rates. Use data gathered through visitor behavior analysis tools such as heatmaps, Google Analytics, and website surveys to solve your visitors’ pain points. This stands true for all businesses, be it eCommerce, travel, SaaS, education, or media, and publishing.

Get Better ROI from Existing Traffic

As most marketers have come to realize, the cost of acquiring any quality traffic can be huge. A/B testing lets you make the most out of your existing traffic and helps you increase conversion without having to spend on acquiring new traffic. A/B testing can give you high ROI as sometimes, even the most minor changes can result in a significant increase in conversions.

Reduce Bounce Rate

One of the most important metrics to be tracked to judge your website’s performance is its bounce rate. There may be many reasons for your website’s high bounce rate, such as too many options, expectations mismatch, and so on. As different websites serve different goals and cater to different audiences, there is no sure-shot way of fixing the bounce rate. One way to do it is through A/B testing. With A/B testing, you can test multiple variations of an element of your website till you find the best possible version. This improves your user experience, making visitors spend more time on your site and reduce bounce rates.

Make Low-risk Modifications

Make minor, incremental changes to your web page with A/B testing instead of getting the entire page redesigned. This can reduce the risk of jeopardizing your current conversion rate. A/B testing lets you target your resources for maximum output with minimal modifications, resulting in increased ROI. An example of that could be product descriptions changes. You can perform an A/B test when you plan to remove or update your product descriptions. You do not know how your visitors are going to react to the change, and A/B testing is one way to ascertain which side the weighing scale will be tilted. Another example of low-risk modification can be the introduction of a new feature change. Before introducing a new feature, launching that new feature as an A/B test in the web page’s copy can make the outcome that much more predictable. It can be very useful beneficial if the changes affect customer data or purchase funnel. Changes without testing may or may not pay off. Testing and then making changes can make the outcome certain.

Achieve Statistically Significant Improvements

Since A/B testing is completely data-driven with no room for guesswork, gut feelings, or instincts, you can easily determine a “winner” and a “loser” based on statistically significant improvements on metrics like time spent on the page, number of demo requests, cart abandonment rate, click-through rate, and so on.

Profitably Redesign your Website

Redesigning can range from a minor CTA text or color tweak to particular web pages to a complete revamping of the website. The decision to implement one version or the other should always be based on data-driven A/B testing. Do not quit testing with the design being finalized. As the new version goes live, test other elements of your webpage to make sure that the most engaging version is being served to the visitors.

How do you Perform an A/B Test?

A/B testing offers a very systematic way of finding out what works and what doesn’t work in any given marketing campaign. Most marketing efforts are geared toward driving more traffic. But, as traffic acquisition becomes more difficult and expensive, it becomes paramount to offer the best experience to your users who come to your website. This will help them achieve their goals and allow them to convert in the fastest and most efficient manner possible. A/B testing in marketing allows you to make the most out of your existing traffic.

A structured A/B testing program can make marketing efforts more profitable by pinpointing the most crucial problem areas that need optimization. A/B testing is now moving away from being a standalone activity that is conducted once in a blue moon to a more structured and continuous activity, which should always be done through a well-defined CRO process. Broadly, it includes the following steps:

Step 1: Research

Before building an A/B testing plan, one needs to conduct thorough research on how the website is currently performing. You will have to collect data on everything related to how many users are coming onto the site, which pages drive the most traffic, what are the various conversion goals of different pages etc. The A/B testing tools used here can include quantitative website analytics tools such as Google Analytics, Omniture, Mixpanel, etc., which can help you figure out your most visited pages, pages with most time spent or pages with the highest bounce rate. For example, you may want to start by shortlisting pages which have the highest revenue potential or the highest daily traffic. Following this, you may want to dive deeper into the qualitative aspects of this traffic.

Heatmap tools are the leading technology used to determine where users are spending the most time on, their scrolling behavior, etc. This can help you identify problem areas on your website. Another popular tool used to do more insightful research is website user surveys. Surveys can act as a direct conduit between your website team and the end-user and can often highlight issues that may be missed in aggregate data. Further qualitative insights can be derived from session recording tools that collect data on visitor behavior, which helps in identifying gaps in the user journey. In fact, session recording tools combined with form analysis surveys can uncover insights on why users may not be filling your form. It may be due to some fields which ask for personal information or users, maybe abandoning your forms for being too long.

As we can see, both quantitative and qualitative research can help us prepare for the next step in the process, which will be to make actionable observations for the next steps.

Step 2: Observe and Formulate Hypothesis

Get closer to your business goals by logging research observations and creating data-backed hypotheses aimed at increasing conversions. Without these, your test campaign is like a directionless compass. The qualitative and quantitative research tools can only help you with gathering visitor behavior data. It is now your responsibility to analyze and make sense of that data. The best way to utilize every bit of data collated is to analyze it, to make keen observations on them, and then to draw website as well as user insights to formulate data-backed hypotheses. Once you have a hypothesis ready, test it against various parameters like how much confidence you have of it winning, its impact on macro goals, and how easy it is to set up and so on.

Step 3: Create Variations

The next step in your testing program should be to create a variation based on your hypothesis, and A/B test it against the existing version (control). A variation is another version of your current version with changes that you want to test. You can test multiple variations against the control to see which one works best. Create a variation based on your hypothesis of what might work from a UX perspective. For example, enough people not filling forms? Does your form have too many fields? Does it ask for personal information? Maybe you can try a variation with a shorter form or another variation by omitting fields that ask for personal information.

Step 4: Run Test

Before we get to this step, let’s first explore how many kinds of testing methods are there and when to use which method. A/B Testing, Multivariate Testing, Split URL Testing, and Multipage Testing are 4 different types of testing. We have already discussed the first kind namely A/B testing. Let us now move on to the other types of testing.

Split URL Testing

Many people confuse Split URL Testing with A/B testing, but the two are fundamentally different. Split URL Testing is testing multiple versions of your webpage hosted on different URLs. Your website traffic is split between the control, and variations and each of their conversion rates are measured to decide the winning version. The main difference between a Split URL test and an A/B test is that in case of a Split test, the variations are hosted on different URLs.

A/B testing is preferred when only front-end changes are required, but split URL testing is preferred when significant design changes are necessary, and you don’t want to touch existing website design.

Creating a Split URL test broadly consists of the following steps:

• Setting up pages for the Split URL test
• Adding conversion goals and estimating test duration
• Finalizing the test
• Previewing and starting the test

Multivariate Testing (MVT)

It is a method in which changes are made to multiple sections of a webpage, and variations are created for all the possible combinations. In multivariate testing, you can test all the combinations within a single test. The multivariate test helps you figure out which element on a web page makes the most impact on its conversion rate. It is more complicated than A/B testing and is best suited for advanced marketing professionals.

For instance, you decide to test two versions of each of the hero image and the CTA button color on a webpage. By using MVT, you can create one variation for the hero image and one for the CTA button color. To test all the versions create combinations of all the variations, as shown here:

After running a multivariate test on all combinations, you can use the data procured to determine which combination affects your page’s conversion rate most and deploy that combination.

Multipage Testing

Multipage Testing is a form of experimentation where you can test changes to particular elements across multiple pages.

There are two ways of doing multipage testing. One, you can either take all the pages of your sales funnel and create new versions of each, which makes your challenger sales funnel, and you then test it against the control. This is called Funnel Multipage Testing. And two, you can test how the addition or removal of the recurring element(s), like security badges, testimonials, and so on, can impact conversions across an entire funnel. This is called Classic or Conventional Multipage Testing. In addition to the 4 types just discussed, there are also two different statistical approaches to testing: Frequentist and Bayesian.

Frequentist approach of probability defines the probability of an event with relation to how frequently (hence the name) a particular event occurs in a large number of trials/data points. When applied to the world of A/B testing, one can see that anyone going with the frequentist approach would need more data (a function of more number of visitors tested and over longer durations) to come to the right conclusions. This is something that limits you in scaling up any A/B testing effort. According to the Frequentist approach, it is essential to define your A/B test’s duration based on sample size to reach the right test conclusions. The tests are based on the fact that every experiment can be repeated infinite times.

Following this approach calls for a lot of attention to detail for every test that you run because for the same set of visitors, you’ll be forced to run longer duration tests compared to the Bayesian approach. Hence, each test needs to be treated with extreme care because there are only a few tests that you can run in a given timeframe. Unlike Bayesian statistics, it is less intuitive and often proves difficult to understand.

On the other hand, Bayesian statistics is a theory based on the Bayesian interpretation of probability, where probability is expressed as a degree of belief in an event. In simple words, the more you know about an event, the better and faster you can predict the end outcomes. Rather than being a fixed value, probability under Bayesian statistics can change as new information is gathered. This belief may be based on past information such as the results of previous tests or other information about the event. Unlike the frequentist approach, the Bayesian approach provides actionable results almost 50% faster as compared to the older Frequentist method while focusing on statistical significance. At any given point, provided you have enough data at hand, the Bayesian approach tells you the probability of variation A having a lower conversion rate than variation B or the control. It does not have a defined time limit attached to it, nor does it require you to have an in-depth knowledge of statistics.

In the simplest of terms, the Bayesian approach is akin to the way we approach things in everyday life. For instance, you misplaced your mobile phone in your house. As a frequentist, you would only use a GPS tracker to track it and only check the area the tracker is pointing to. While as a Bayesian, you will not only use a GPS tracker but also check all the places in the house you earlier found your misplaced phone. In the former, the event is considered to be a fixed value, while in the latter, all past and future knowledge are utilized to locate the phone. To get a clearer understanding of the two, take a look at the comparison below:

Once you have locked down on either one of these types and approaches based on your website’s needs and business goals, kick off the test and wait for the stipulated time for achieving statistically significant results. Keep one thing in mind – no matter which method you choose, your testing method and statistical accuracy will determine the end results. For example, one such condition is the timing of the test campaign. The timing and duration of the test have to be on point. Calculate the test duration keeping in mind your average daily and monthly visitors, estimated existing conversion rate, minimum improvement in conversion rate you expect, number of variations (including control), percentage of visitors included in the test, and so on.

Use our Bayesian Calculator to calculate the duration for which you should run your A/B tests for achieving statistically significant results.

Step 5: Result Analysis and Deployment

Even though this is the last step where you find your campaign winner, analysis of the results is extremely important. Because A/B testing calls for continuous data gathering and analysis, it is in this step that your entire journey unravels. Once your test concludes, analyze the test results by considering metrics like percentage increase, confidence level, direct and indirect impact on other metrics, etc. After you have considered these numbers, if the test succeeds, deploy the winning variation. If the test remains inconclusive, draw insights from it, and implement these in your subsequent tests.

A/B testing lets you systematically work through each part of your website to improve conversions.

What Can you A/B Test?

Your website’s conversion funnel determines the fate of your business. Therefore every piece of content that reaches your users via your website must be optimized to its maximum potential. This is especially true for elements that can influence visitor behavior and conversion rate. When undertaking an optimization program, the following key elements should be A/B tested (the list, however, is not exhaustive):

Copy

1. Headlines and Sub-headlines

Your headline is the first thing that visitors see on your page. The headline is what defines your first impression in a visitor’s eyes. And the first impression is what determines if the visitor will go further down your conversion funnel. This is why it becomes imperative to be extra cautious about your website’s copy, its writing style, and formatting. Make sure your headline catches the visitors’ attention as soon as they land on your website. Keep it short and to the point, ensuring it talks clearly about what your product or service is and its benefits. Try A/B testing various fonts, sizes, copy, and messaging.

2. Body

The body of your website should clearly state what the visitor is getting – what’s in store for them. It should also resonate with your page’s headline. A well-written headline and body can increase the chances of turning your website into a conversion magnet. While writing content for your page’s body, keep in mind these two parameters:

• Writing style: Use the right tonality based on the target audience. Your copy should directly address the end-user and answer all their questions. It should consist of key phrases that improve usability and stylistic elements that highlight important points.
• Formatting: Use relevant headlines and subheadlines, break the copy into small and easy paragraphs, and format it for skimmers using bullet points or lists.

Design and Layout

Because everything seems so essential, businesses sometimes struggle with finding only the most essential elements to keep on their website. With A/B testing, this problem can be solved once and for all. For example, as an eCommerce store, your product page is extremely important from a conversion perspective. One thing for sure is that with technological progress in its current stage, customers like to see everything in high definition before buying it. Therefore, your product page must be at its most optimized form in terms of design and layout. Along with the copy, the design and layout of a page include images (product images, offer images, etc.) and videos (product videos, demo videos, advertisements, etc.). Your product page should answer all of your users’ questions without confusing them and without getting cluttered:

• Provide clear information: Based on the products you sell, find creative ways to provide all necessary context and accurate product descriptions, so that prospective buyers do not get overwhelmed with an unorganized copy while looking for answers to their queries. Write clear copies and provide easily noticeable size charts, color options, etc.
• Highlight customer reviews: Add both good and bad reviews for your products. Negative reviews add credibility to your store.
• Write simple content: Avoid confusing potential buyers with complicated language in the quest to decorate your content. Keep it simple and fun to read.
• Create a sense of urgency: Add tags like ‘only 2 left in stock’, countdowns like ‘offer ends in 2hours 15 minutes’, or highlight exclusive discounts and festive offers, etc. to nudge the prospective buyer to purchase immediately.

Other important pages whose design needs to be on point are pages like home page and landing page. Use A/B testing to discover the most optimized version of these critical pages. Test many ideas like adding plenty of white space and high definition images, featuring product videos instead of images or testing out different layouts. Declutter your pages using insights from click maps to analyze dead clicks and identify distractions. The less cluttered your home page and landing pages, the more likely that visitors can easily and quickly find what they are looking for.

To Be Continued...