“Statistics are like a bikini. What they reveal is suggestive, but what they conceal is vital.” ~ Aaron Levenstein
Understanding how visitors behave on our sites is no trivial matter, but all too often, we’re distracted by trivia in the torrent of data we collect.
Here’s an example: If you want to know whether you and your date have long-term potential, find out if you both like horror movies. A correlation between those two variables popped up in data from 34,260 couples visiting former dating site OkCupid. The statistic is valid, but you’d hardly want to base a lifetime decision on it. The truth is we can pretty much trick numbers into becoming fool’s gold at will. As the old saying goes: “Figures don’t lie, but liars figure.”
Any statistic that doesn’t help you make better decisions is trivia. The trap opens when we discover some interesting correlation or pattern and, in our excitement, think it has real meaning, when in fact it’s just a curiosity. We’re elated at finding a needle in the haystack, but it turns out to just be a needle.
These statistical curiosities don’t help you evaluate how your business is performing or tell you what the next step should be. If you did base business decisions on these numbers, you’d probably drive your company into the ground.
The Two Types of Trivia
Typically, trivia results from two common analytical processes. See if you recognize yourself in these.
- You drill down so far that the statistics you’re looking at are based on a minuscule fraction of your data. For example, you may determine that users coming from Facebook who also do an onsite search for women’s clothing spend 10 percent more per order than the average user.
- You use poor, overly broad estimators of what you really care about. Looking at session-based conversion rates when you really care about the lifetime value of a group of users doesn’t provide your company with any actionable knowledge.
Is It Trivia?
Because needles in the data haystack are so shiny and attractive, we need some way to recognize them for what they are. Here are three clues:
Clue No. 1
Most marketers already have a set of expectations before they ever see statistics. The results can reinforce their expectations, challenge them, or simply fail to be directly relevant. Trivia falls into the last category.
Clue No. 2
Trivia isn’t actionable, while insights inspire action. For instance, if you were to discover that branded search traffic is up 4 percent this month, what would you do with this information? For the most part, this is trivia, unless you’re running a campaign specifically designed to target branded search traffic. But if you discovered that revenue from visitors who have used a branded search is up 4 percent this month, that’s suddenly very interesting.
The first question you might be tempted to ask is, “Why?” That’s a great question, but it’s often very difficult to answer. An even better question is, “If branded search revenue is trending up, what changes can we make to the branded search experience to gain even more revenue?” If it was down 4 percent, you might ask yourself how to reverse the trend.
The key here is that branded search traffic is little more than a pretty statistic you might include in a presentation to impress the board, while branded search revenue can lead directly to new ideas and give you the feedback you need to test those ideas.
Clue No. 3
Trivia forces you to group several statistics together to get a vague understanding of the truth. Insights deliver them directly.
Let’s say you have an expectation about the total lifetime value of an organic searcher compared to that of a paid searcher. You could use channel conversion rates, unique visitors, and some sort of attribution model to get a rough estimate of the value of an organic versus paid searcher. Or you could just partition your visitors into paid and organic searchers and value each individual directly, possibly even including how much you paid for each of those paid searches. Now, you have an apples-to-apples comparison of the real value, and the decisions you make based on this information are grounded in reality rather than rough estimates.
In general, trivia is much easier to generate than insights. Trivia is readily available from any number of tools, platforms, and products. And those products tend to be cheap — if not free. Insights require you to first understand your expectations and then develop the right metrics to isolate each of the contributing factors that could reinforce or challenge your expectations. This requires an analytics platform that can be tailored to your own needs and can change as your needs change.
Why We Get Caught Up in Trivia?
You shouldn’t feel too bad about falling for trivia. It’s almost built into the system. Understanding how we’re distracted by it is the first step in avoiding it. There are two reasons:
- We are slaves of the tools. Most of the Web Analytics packages available grew out of tools useful to IT departments. Things like clicks, page load latency, sessions, and requests matter to people trying to optimize servers and systems. But in general, marketers care about people, time, and money. That’s not to say there aren’t useful metrics available from these tools, but out of the millions of charts and metrics you can potentially pull out of these systems, only a very small percentage are actually useful.
- Trivia is surprising. People tend to only look for the unexpected. If you’re a competent marketer, you probably have pretty good intuition about what’s driving profits on your site, and a lot of the metrics you look at simply reinforce what you already knew. So you keep searching and eventually stumble upon something you didn’t expect. That seems more important than it actually is, just because it looks different.
The Trivia Trap
You and your team need not follow the siren song of pretty but useless data. Here are four rules to keep you on track:
- Analyze the value. If it’s not something that can inspire you to action, then stop looking at it and stop reporting it. If it’s not reinforcing or challenging your expectations over the things you ultimately care about, then it’s a distraction at best and misleading at worst.
- Outline your expectations. Spend some time defining what it means for each of your vendors and marketing programs to be successful and valuable. Until you understand what you want to know, you won’t be able to measure it.
- Find the right tools. The right tools depend on your environment, your site, and your needs. There is no single best tool in the marketplace. If you really want the insights you need to reinforce and challenge your expectations, be prepared to invest some time and money in the project. The right tool can shift your feedback cycle from months to days.
- Experiment. You know your business better than anyone. Find an idea that you think will have an impact on your bottom line, develop a way to measure your expectations about the new program or site change, and see if it works. Lather, rinse, repeat.
At best, trivial discoveries in the data flow are time-wasters. At worst, they’re distractions from the genuine insights that will strengthen your business. Acting on them can do serious harm. It would be like setting off on a journey with a faulty GPS: the information it provides looks valid, but after you’ve committed to your direction, you suddenly realize you are completely lost.
