This article presents a framework for Split Testing. From an overview of the most common techniques (AB and Multivariate Testing) to the resources needed for testing (People, Process, Technology) and the value it brings to the organization. Most companies are already acquainted with these techniques, but very often they are less prepared to get it up and running in their websites. Here are three critical things to understand about testing (especially if you are in one of these companies):
- The most valuable insight from a test is not the winning version, but the learning experience about the customer. Website Testing teaches us what our customers like and helps us not only to increase conversions but also to create a better experience in future designs.
- Get the first test running very quickly, it will help showing management how important it is and help the tester getting support and budget for testing.
- Testing is not limited to landing pages or campaigns, it should be implemented across the website, wherever visitors are abandoning the website and wherever the website is leaving money on the table.
- Quick Wins – testing is an excellent way to produce quick wins. By changing a few images, buttons and text we can significantly improve conversion rates.
- Money Generating Machine – since the outcomes are very measurable and we can see how much money we earned as a result of testing, it is very easy to prove how testing can pay for its expenses.
- Improved Customer Experience – as mentioned above, one of the most important outcomes of testing is to understand customers and create better experiences on the website, not always related to the page tested or to the ultimate conversion. It just helps us make our customers at home.
- Granularity of results: since it is a full factorial experiement, Multivariate tests show which elements are the best performing separately and the correlation between the elements. This can be very useful when projecting the results to other parts of the website.
- No redirects: since all elements tested are inside the page, there is no need to redirect from the original page to the tested pages. Although redirects can be performed smoothly, I believe it is better not to use them whenever possible, they can slow the flow and affect user experience.
- Less design resources: since we will be testing different designs to existing elements in the page, this will not require too much design, which might be a scarce resource in some organizations.
- Design Freedom: AB tests are more flexible in terms of testing elements. Since the testing page is unrelated to the original, the designer can experiment more radical ideas, which can lead to disruptive innovations, as mentioned by previous Google CIO, Douglas Merril (see Innovation at Google)
- Faster Results: since AB tests usually experiment with a few pages, and not a combination of many elements and their interactions, it give results much faster. In addition, as mentioned above, if a disruptive innovation is found, the conversion rates will rapidly increase and the test will be significant faster. Multivariate Testing is more about incremental innovation.
- Optimization Analyst is the person responsible to get the testing up there. S/he will think through ideas for testing, get ideas from other people and decide what, how, when should be tested.
- Creative Specialist is the person who will design the elements and be sure usabilty guidelines are in place.
- Web Developer is the beloved person that will implement the codes and make sure the website does not break with it.
- Strategy: What is the business objective behind the test? Where can I create the highest impact by testing?
- Test Selection: Which page should I test?
- Test Planning: Which technique should I choose for this specific test? Which page elements should be tested?
- Test Setup: Which codes need to be implemented and where?
- Test Activation: Blast off!
- Reporting & Analysis: How did it affect my site? How can I create a follow-up test to improve my page based on the new insights I have? And, most important, what is the bonus I should ask from my boss?
Multivariate Testing Overview
Multivariate tests experiment with elements inside a specific page. Basically, we define elements inside a page (e.g. a picture, a text, and a button) and provide different alternatives to each element. The testing tool will show each element combined with all other elements to visitors. The resulting combinations are derived from the number of elements multiplicated by the number of element variations. Below is a representation of how this technique works.
AB Testing Overview
An A/B test consists of creating alternative pages for a specific page and showing each of them to a certain percentage of visitors. For example, if you create 4 different variations of a landing page, 20% of visitors to the website will see each version (4 variations + original). Since this technique experiments with separate pages, it is very useful when testing layout or radical changes to a page. Below is a representation of how this technique works.
Split Testing Value
The main benefits of testing can be summed into three values:
In the past, I have written about the differences between A/B and Multivariate Testing. I think it is always important to understand the strengths and weakness of both methods before choosing the right technique for a specific test. Here they are:
Multivariate Testing Advantages
AB Testing Advantages
Requirements for Success
This section was inspired by slides created by Bob Garcia and Frans Keylard from Webtrends (see Slideshare embedded in the end of the article). The presentation provides an encompassing overview of the Multivariate Testing process. However, the same framework also applies to all Split Testing efforts. Note: I have not been on the presentation myself, but since the slides are very comprehensive, the context could be understood.
As with all other website optimization efforts, in the end of the day the success will depend on three main factors: the people running it, the process used to run it, and the technology used.
People
Process
The process will change from company to company, but below are a few questions that will certainly help anyone that wants to have split tests up and running.
Note that the chart above uses the colors of the people in order to facilitate the visualization of who is involved in each step of the process.
Technology
Technology plays is important, but I believe that the people behind it are the ones that will make the difference on the test success. Below are the biggest solutions, each has its own advantages and disadvantages. Check their websites to find the most suitable one for your needs.
Test Twice Before You Think
Basically, the most important thing about testing is to test fast and continuosly. Each test provides insights into the customers’ tastes and gives us the chance to improve our conversion rates. Just do it.
What do you think?
