“The interesting thing to do here is to here is to actually also look at the cumulative conversion rate over time. By doing that, you can actually get a sense for the statistical confidence without actually having to understand the math.”
Dan Siroker of Optimizely talks about how to beat the back button and optimize for engagement. He talks about things he learned about best practices from his work at Optimizely and from his work at the Director of Analytics of the Obama presidential campaign.
He worked at Google as a Product Manager, and left to join the Obama campaign, because he got a chance to see Obama speak about how he could take Google’s methods and apply it to government.
He formed the New Media Analytics desk of the Obama campaign, and tried to bring all the information together to bring the campaign together across the board and make decisions. Eventually he left the campaign and formed Optimizely to help customers make better optimization tests.
Below we provide a brief summary of the lessons learned during the presentation.
Lesson 1: Define Success.
Define quantifiable success metrics. This drives everything else you do. There is an engagement link on Google Analytics. One place to start is the Visitors Flow section. This is a way to identify the bottleneck in a given tunnel. For example, on the Obama campaign, many people came to the website, few signed up for the email list, and of those, a larger percentage made donations. This meant that the bottleneck was with getting people to sign up with their email addresses. Using Google Website Optimizer, they tried several splash pages with words like Sign Up, Learn More, Join Us Now, and Sign Up Now. The also changed the images/videos in various tests. There was an 18% improvement, and images performed better than videos. With the best combination, they got a 40.6% improvement, and it turned out that the impact was 2.8 million additional signups, 288,000 additional volunteers, and an additional $57 million raised.
Lesson 2: Less is More. Reduce Choices
If you go to One Kings Lane, they want you to sign up immediately. To begin improving their site, they removed the request for zip code and for email address confirmation, and conversions increased by 8.4%. (It should be noted that now, the site only asks for an email address.)
Another example: The Clinton Bush Haiti Fund asked for optimization. They removed the Title and Phone number fields, and there was an 11% lift in dollars per visitor.
A more complex example: A startup that allows you to report street issues (such as potholes). The site was designed to be interactive and dynamic. By moving from a lot of forms to a form that says only: report neighborhood issues and see them get fixed, engagement increased by 8%.
Lesson 3: Words Matter. Focus on Your Call to Action
On the Obama campaign site, they tested various calls for donation. For people who had never signed up, “Sign up and get a gift” worked best. For people who’d signed up and never donated, “Please Donate” worked best. For those who’d previously donated, “Contribute” performed best.
ABC Family ran an experiment on the Videos page. They realized that people were often coming in and searching for episode titles, so they thought that if they helped people catch up on previous episodes, they would get better engagement. They improved engagement by 600%. They ran an experiment for Jane by Design. They used the words “full episodes” versus “watch episodes.” Full episodes increased clickthrough rate by 10%. This is counter to most wisdom, which tells people to “tell people what to do.”
LiveChat compared the call to action “Free Trial” with “Try it Free.” “Try it Free” improved clicks by 14.6%. Generally, if you want someone to do something, tell them to do it.
From the Clinton Bush Haiti Fund, they tested the donate button text of “Submit Here” versus “Support Haiti,” and it added 15.75% to the dollars per visitor. Using Optimizely, the fund was able to raise an additional $1 million.
Lesson 4: Fail Fast
Measure in real time, and if something does not work well. For example, IGN moved their Video link further left to attract attention, reducing their video click through rate by 92.3%. They were able to find out quickly, and revert immediately. It might be a reaction of returning visitors to the link moving, and it might have been wise to try the experiment with new users only.
Lesson 5: Start Today
Google Website Optimizer, Omniture Test & Target, and Optimizely are good tools for A/B testing. Optimizely has a visual editor, You can select different areas that you want to change. To implement Optimizely, you put a single snippet on your site and the experiments are run automatically from a WYSIWYG editor
Dan Siroker – Founder & CEO, Optimizely
Dan Siroker is Founder and CEO of Optimizely, a revolutionary website testing optimization platform that is tightly integrated with Google Analytics. Prior to founding Optimizely, Dan was Director of Analytics for the 2008 Obama presidential campaign and worked at Google as a Product Manager.