Building Google Analytics Powered Widgets

There is a lot of useful and interesting data held in your Google Analytics account that could be used to drive content on your site and apps. For example, you might want to show your website visitors what are the most viewed products, or the most viewed articles, or the best performing authors, etc.

In this tutorial I provide a step-by-step guide showing how to create a Top Authors widget using the Google Analytics API, Google App Engine (Python) and Google Tag Manager. You can create a free Google App Engine account that should give you enough allowance to build and use your widget. You can see the end result of this tutorial right there on the right hand side of this site, see “Top Authors” widget.

There are 2 reasons we are using Google App Engine as a proxy instead of just calling the Google Analytics API directly:

Avoid exposing any sensitive information held in Google Analytics. Eg. Instead of sharing pageviews we will calculate and share a percentage of the maximum pageviews instead.
There is a limit to the number of API calls that can be made and with this method we only need to call the API once a day as we will cache the results. Therefore we don’t risk exceeding the API quota; also, as the data is cached, the results will return a lot faster.

The steps below will take you through all the way from creating your app engine project to adding the widget to your site using Google Tag Manager.

Create a New Google Cloud Project

Create Your Google App Engine App

Enable the Google Analytics API

Use Import.io To Scrape Extra Data

Create the Top Authors API

Serve the Widget using Google Tag Manager

1. Create a New Google Cloud Project

If you have not used Google cloud before sign up and create a new project at https://console.developers.google.com. For this tutorial you will be using the free version of App Engine and therefore you do not need to enable billing. Name the project and create a brand friendly project id as this will become your appspot domain, eg. yourbrandwidgets.appspot.com

2. Create Your Google App Engine App

Download the Google App Engine SDK for Python and create a folder on your computer called yourbrandwidgets.

In the folder create a file called app.yaml and add the code below. This is the configuration file and it is important that the application name matches the the project ID created in the first step.

application: onlinebehaviorwidgets version: 1 runtime: python27 api_version: 1 threadsafe: yes


handlers:

- url: .*

  script: main.app

libraries: - name: jinja2 version: "2.6" - name: markupsafe version: "0.15"

In the folder create a file called main.py and add the following code

from flask import Flask


app = Flask(__name__)

app.config['DEBUG'] = True
# Note: We don't need to call run() since our application is embedded within the App Engine WSGI application server.
@app.route('/')

def home():

    """Return a friendly HTTP greeting."""

    return 'Online Behavior Widgets'

@app.errorhandler(404) def page_not_found(e): """Return a custom 404 error.""" return 'Sorry, nothing at this URL.', 404

Create a file called appengine_config.py and add the following code.

"""'appengine_config' gets loaded when starting a new application instance.""" import sys import os.path


# add 'lib' subdirectory to 'sys.path', so our 'main' module can load third-party libraries.

sys.path.insert(0, os.path.join(os.path.dirname(__file__), 'lib'))

Create a folder called lib in the main folder.

Download the file called google-api-python-client-gae-.zip from this page.

Unzip the folder and add the 4 folders to the lib folder in your project.
Install the other required libs for Flask by creating a file called requirements.txt and add the following text.

# This requirements file lists all third-party dependencies for this project. # Run 'pip install -r requirements.txt -t lib/' to install these dependencies in 'lib/' subdirectory. # Note: The 'lib' directory is added to 'sys.path' by 'appengine_config.py'. Flask>=0.10

Run pip install -r requirements.txt -t lib/ in the terminal install these dependencies. You should now be ready to test locally. Using the Google App Engine Launcher add the application as described in this tutorial.

Then, select the app as shown in the screenshot below and click run; this will run locally and open a new tab in your current open browser.

If this works as expected you should be able to visit the site on your localhost at the port you set.

You are now ready to deploy this to the cloud. Click deploy and keep an eye on the logs to check that there are no errors.

if successful you can test the the app at yourbrandwidgets.appspot.com.

3. Enable the Google Analytics API

To use the Google Analytics API you will need to enable it for your project. Go to the API portal in the developer console under APIs & Auth and click on the Analytics API as shown in the screenshot below. Then, click on the Enable API button.

Get the App Engine service account email, which will look something like [email protected], under the Permissions tab following the steps shown in the screenshot below and add the email to your Google Analytics account with collaborate, read and analyze permission (learn more about User Permissions).

4. Use Import.io To Scrape Extra Data

One issue we had while creating the widget in the sidebar of this site was that the author images and links are not stored in Google Analytics. We therefore have 2 options to overcome this.

Option 1

If you are using Google Tag Manager, create a variable to capture the author image and author urls on each pageview as custom dimensions.

Option 2 (the option we will use in this tutorial)

We used import.io to scrape the authors page and turn it into an API that we can use in app engine.

In order to see how this works, go to https://import.io and copy and paste this URL into the box and press try it out. You should see the page scraped into a structured format that you can use by clicking on the Get API button, as shown below.

As you can see, the API has a record for each author in a neat JSON format including the 3 pieces of data we needed. The author’s name is under "value", the author’s page link is under "picture_link" and the author’s image is under "picture_image". That really is magic.

We can now create a function in our code that will call the import.io api, extract the 3 data points that we need, cache it for 24 hours, and the returns the result. We can test the result of this by creating an url for this. Update the main.py file with this code. You will notice we have now included some new modules at the top.

import json import pickle import httplib2


from google.appengine.api import memcache

from google.appengine.api import urlfetch

from apiclient.discovery import build

from oauth2client.appengine import OAuth2Decorator

from google.appengine.ext import webapp

from google.appengine.ext.webapp.util import run_wsgi_app

from oauth2client.appengine import AppAssertionCredentials

from flask import Flask

from flask import request

from flask import Response
app = Flask(__name__)

app.config['DEBUG'] = True
# Note: We don't need to call run() since our application is embedded within the App Engine WSGI application server.
@app.route('/')

def hello():

    """Return a friendly HTTP greeting."""

    return 'Hello World!'
@app.route('/importioauthors.json')

def importio():

    authors = importioOnlineBehaviorAuthors()
    return json.dumps(authors)
@app.errorhandler(404)

def page_not_found(e):

    """Return a custom 404 error."""

    return 'Sorry, nothing at this URL.', 404
def importioOnlineBehaviorAuthors():
    ob_authors_check = memcache.get('importioOnlineBehaviorsAuthors')

    if ob_authors_check:

        ob_authors_output = pickle.loads(memcache.get('importioOnlineBehaviorsAuthors'))

        ob_authors_output_method = 'memcache'

    else:

        importio_url = "https://api.import.io/store/data/6f4772f4-67ce-4f78-83f3-fa382e87c658/_query?input/webpage/url=http%3A%2F%2Fonline-behavior.com%2Fabout%2Fauthors&_user=ENTER-YOUR-USERID-HERE&_apikey=ENTER-YOUR-API-KEY-HERE"

        importio_url_result = urlfetch.fetch(importio_url)

        importio_result = json.loads(importio_url_result.content)

        importio_author_images = {}
        for row in importio_result['results']:

            name = row['value']

            importio_author_images[name] = {

                    'picture_image': row['picture_image'],

                    'picture_link': row['picture_link']

                    }
        ob_authors_output = importio_author_images
        memcache.set('importioOnlineBehaviorsAuthors', pickle.dumps(ob_authors_output), 86400)

return ob_authors_output

You can run this locally or deploy to live and then go to yourbrandwidgets.appspot.com/importioauthors.json to test this is working.

5. Create the Top Authors API

The code shown below will authenticate and call the Google Analytics API using the App Engine service account email we added earlier. As you will see in the API request below, we are getting Unique Pageviews for the top 20 authors from the past 30 days. The code then stitches the import.io data to the Google Analytics data so that we have the author images and links ready to be used.

The results are cached for 24 hours so that the API is only called once a day for all users and returns the data in the callback function name we define when calling the URL.
Add the following code to your main.py file above the line of code @app.errorhandler(404)

@app.route('/topauthors.jsonp') def topauthors(): # Get the callback function name from the URL callback = request.args.get("callback")


    # Check if the data is stored in the cache (it resets after 24 hours)

    output_check = memcache.get('gaApiTopAuthors')
    # If yes then used the cached data in the response

    if output_check:

      output = pickle.loads(memcache.get('gaApiTopAuthors'))
      # If no then request the Google Analytics API

    else:
      # Authenticate and connect to the Google Analytics service

      credentials = AppAssertionCredentials(

      scope='https://www.googleapis.com/auth/analytics.readonly')

      http = credentials.authorize(httplib2.Http(memcache))

      analytics = build("analytics", "v3", http=http)
      # Set the Google Analytics View ID

      view_id = '32509579'
      # Set the report options

      result = analytics.data().ga().get(

        ids='ga:' + view_id,

        start_date='30daysAgo',

        end_date='yesterday',

        dimensions='ga:contentGroup2',

        metrics='ga:uniquePageviews',

        sort='-ga:uniquePageviews',

        filters='ga:contentGroup2!~Online Behavior|admin|(not set)|Miklos Matyas',

        max_results='20'

        ).execute()
      # Get the authors extra data

      authors = importioOnlineBehaviorAuthors()
      # Loop through the results from Google Analytics API and push into output only the data we want to share publicly

      output = []

      max_unique_pageviews = float(result['rows'][0][1])
      for row in result['rows']:

        author = row[0]

        unique_pageviews = float(row[1])

        perc_of_max = str(int(100*(unique_pageviews/max_unique_pageviews)))
        # Only push the author if their image and link exist in the import.io API

        if (author in authors):

            output.append({

              "author":author,

              "perc":perc_of_max,

              "image":authors[author]['picture_image'],

              "link":authors[author]['picture_link']

              })
      # Save the output in cache for 24 hours (60 seconds * 60 minutes * 24 hours)

      memcache.set('widgetTopTenAuthors', pickle.dumps(output), 86400)
    # Create the response in the JSONP format

    jsonp_callback = callback+'('+json.dumps(output)+')'
    resp = Response(jsonp_callback, status=200, mimetype='application/json')

    resp.headers['Access-Control-Allow-Origin'] = '*'

# Return the response return resp

You will not be able to test this locally as it accesses the Google Analytics API so you will have to deploy to App Engine to see the output.

If it is all working as expecting you should see the result by directly accessing the URL in the browser. eg. http://yourbrandwidgets.appspot.com/topauthors.jsonp?callback=anyFunctionName

You can check for any errors in the developer console under Monitoring > Logs. Select App Engine and click the refresh icon on the right to see the latest logs for every time a URL is requested.

6. Serve the Widget using Google Tag Manager

Using the API we just created, which returns the top authors data, we can add a custom HTML tag to Google Tag Manager that will loop through the results and (using a bit of HTML and CSS) output the results in nice looking widgets, complete with bar charts based on the percentage of the maximum Pageviews we calculated server-side.

You will want to design the widget first, and a tip is to try and reuse as much of the current CSS styles for the website.

a) Add the widget code as a tag

Add the following code to Google Tag Manager as a Custom HTML tag.

b) Create a variable and trigger

In this example, we will be adding the new widget right above the Google+ widget so first we create a Custom JS variable that returns true if the div element holding the Google+ exists and false if it does not, as shown below.

c) Custom JS Variable – Google Plus Widget Exists

function(){ if ($( '#block-block-18' ).length > 0){ return true } else { return false } }

d) Preview and Publish the widget

Set the tag to trigger on all pages where the div we are appending the widget to exists, as shown below.

Save the tag and before you publish, go into preview mode to test that the tag is triggering as expected and the widget is appearing as you have designed. If you are happy it is all working you can publish the tag and launch the widget to all your users.

Your Turn To Create A Widget

This is just one simple example of what is possible and we would love to see what you create. How about sharing your top 10 products based on sales or top performing brands or categories. The possibilities are endless!