Documentation Is the Backbone of Analytics

Analytics documentation

So, you can't find enough time to do the stuff you want to do at work because you are very busy? Take heart. It's possible to arrange your work to spend more time on the big questions, but it'll take some work to get there. How? Read on...

Everybody wants to do cool stuff at work! We, the digital analytics folks, love doing analyses, presenting data-driven insights, exploring new tools, convincing the business to launch a new feature / product derived from our analyses, etc. I will categorize these as the "sexy" parts of the job, the parts we wish we had more time for. Now, just like any other job, there are also the not-so-sexy parts of our jobs. That's the focus of this post - the mundane (but don't go away! it ties back into the sexy part of the job at the end).

Whenever I go to an analytics event or talk to someone from the industry, the conversation is always about the aforementioned sexy parts of the job. Now that I am not at an event, I can talk more about one of the the less sexy but probably most important aspects of our jobs: documentation, the backbone of the analytics implementation process, IMHO.

Analytics Process From Beginning To End

If you think about the analytics process at your company, it would probably look something like this:

  1. Business Problem: define the business problem you are trying to solve.Let's assume that there is a product/feature that is being built to solve this problem for the sake of clarity.
  2. KPI Design: work closely with the product/business owner and decide on the KPIs and metrics to help identify the performance of the product/feature in detail.
  3. Documentation:
    • Business Solution Design Document: Describe in detail how you are going to capture the data you need in your implementation.
    • Technical Solution Design Document: work with your implementation team (if you are lucky enough to have one)to clarify the above document in detail so that they can write the Technical Solution Document (e.g. detailed Data Layer requirements).
  4. Requirements Implementation: implement the final solution through a TMS (Tag Management Solution) and work with developers when necessary.
  5. Unit Testing: QA the implementation of the new features/products.
  6. User Acceptance Testing (UAT): submit your pass/fail findings back to the team for a final UAT and update the master dictionary with the new events, custom dimensions, evars, custom metrics, etc., if all is implemented as requested.
  7. Regression Testing and Release: do a full regression test and release the changes to production.
  8. Data Pull: pull the data you need for your analysis.
  9. Data Analysis: torture if needed.
  10. Storytelling: tell a story to the business based on your findings.

Meh! Not Sexy Enough

I probably wouldn't exaggerate if I told you that (3), (5), (6) and (7) above, i.e., the documentation and QA, are often overlooked and underappreciated because they are considered time consuming! I think they are at least as important, if not more, as the rest of the items above. Let me expand on this.

I completely agree that documentation is not the most entertaining part of what we do. You know what you want to track and you can probably save time (in the short run, anyway) if you have a quick chat or type up a quick email to your implementation team and/or your developers to get the stuff you want implemented so that you don't have to "waste" your time with documentation, but I highly encourage you to document everything you implement.

Assuming that there is a release every other week, there are around 26 releases a year (times the number of platforms you have). Without proper documentation, probably nobody would know/remember what's been implemented in what release. This is the obvious reason. The second one is more strategic.

The "Urgent" Requests

Tell me if you had a conversation like the one below with one of your product managers:

Product Manager (PM): You remember we implemented this new [amazing] product to increase exposure of the XYZ product category?

You: Yes

PM: Can you get me a report/do some analysis that tells me whether this product is actually performing well? We need to talk to the sales team so that they can start selling it

You: That can be done… but when do you need it by?

PM: End of day tomorrow

You: That is not possible… I am working on [insert a high priority project here]

PM: When can I get it?

You: We'll put you in a queue and see what can we can do...

PM: It's super important because [insert a name with a big title with large salary [aka a HIPPO]] asked for it

You: OK. As I said I will see what I can do about this but I cannot get it to you by the end of day tomorrow

PM: That's too late because the presentation is this Wed.

You: If it's super important, why is this a last min. request?

PM: [Provocative response]

You: [More provocative response]

... and boom there is unnecessary tension between you and your colleague! I am sure this happens to some of you all the time (it used to happen to us a lot but not as much anymore).

One solution (at least a partial one): Self-Serve and Automation!!! Let me type that out again: SELF-SERVE & AUTOMATION. Trust me, because I'm sure you don't want to worry about these type of requests anymore.

Documentation Is the Backbone of Analytics

At autoTRADER, we've been trying to solve this problem for a while.

  • How can we carve up more time to do strategic analyses that can move the business forward?
  • How can we modify our existing processes so that the individual business units could have access to their own data and do their jobs more efficiently and faster so that they can also move faster?

And now that we have been properly documenting every single tag on the site and trained pretty much everybody who needs digital analytics data for whatever they are doing, we have enabled product / agile teams to self-serve. It's a definite win-win both for us, the analytics folks, and the product teams.

Analyze Away

Do you want to know how your amazing product has been performing? Go to the 3rd tab on the live Master Dictionary spreadsheet (ours lives in Google Drive), see how the implementation was done for your product and pull whatever you need yourself. Do you want to slice and dice the data by date, hour of day, medium, platform? Go crazy, be my guest! Now I can concentrate on my more strategic project!

We've come a long way at At one point there was only one person (a.k.a., me) who used to do pretty much all of the stuff above (all but coding) but now we have a larger team. Everybody has their own speciality, allowing us to execute the fix I am proposing here. Yes, you need a team to be able to self-serve; if you are a one man show, let this be an incentive for you to make the business case to staff up (more on this for a future post).

The Analysis Framework

This is the framework we've designed at autoTRADER (kudos to my colleague Kevin, who was able to articulate what we wanted to achieve in this two by two framework).

Analysis Framework

I don't know about you, but we, the Analytics group, want to spend the most of our time on the right side of the framework that has an impact on the larger business opportunities and challenges. However, we used to spend most of the time on the left side of this matrix doing "micro-analyses", where the business impact is usually smaller. I am not condescending the micro analyses at all, but our group's mandate is to focus on the right side.

In order to change this imbalance, we've come up with processes and hired resources to carve up time for the team to focus on the more strategic stuff. We document everything that goes into the implementation and we created a measurement and data dictionary (we call it the Implementation Bible) so that the teams can do their own analyses without being dependant on us. We've also trained the internal teams on our Analytics platforms - how things are tagged, how teams can create and use their own reports through various tools like PowerBI, Supermetrics, Data Studio, GA Dashboards and Custom Reports - in order to create a win-win situation for everybody. They have access to what they want, we spend our time on what we need to spend more time on...

Slog Now to Become Cooler Later

Going back to the beginning of this article, the sexy part of the job is tied to the not-so-sexy part of the job and goes back to documentation. I am aware that documentation is time consuming but it's necessary to move your business forward. I highly recommend that you think about it if you are having issues like we used to!

You don't have time to document? Sure!


Online Behavior © 2012