Building a custom Google Data Studio connector from A-Z. Part 1 — Basic setup.

In the previous article I described the idea of Google Data Studio connectors. Now I’d like to show you how to build one. I’ll be building a connector for showing stats about the tracks played in Spotify, so for example user will be able to see what was his most popular artist in a specified timeframe. In order to do this I’ll consume Spotify’s public API.

In this article I’ll describe the basic steps of creating the connector and show you how to setup your development environment.

If you’re not patient you can find the full code for this article at https://github.com/Bajena/spotify-gds-connector/tree/Tutorial1 ;). You can also test the connector at this link.

Let’s go!

Step 1 — Setup a Google Script

GDS connector is basically a Google Script that implements an interface provided by Google, so in order to work with Google Data Studio our connector will need to respond to following methods:
- getAuthType
- getConfig
- getSchema
- getData

In order to create a script go to https://script.google.com and create a new script.

Google Scripts dashboard

First thing after creating the script I recommend is to set correct access privileges. If you’d like the connector to work also for other people then you should select “Anyone with link can view”. Don’t worry, nobody will see your code unless you explicitly give them the link. This option will just allow Google Data Studio to properly access your script.

Select correct sharing settings

Step 2 — Setup environments (a.k.a. “Deployments”)

Second thing that took me some time to understand is how do I release new connector versions. Should I keep separate script files for production/dev environments? No, Google has invented a concept of “deployments” and “versions”.

You can think of “versions” as of labels code base state, similar to GIT’s tags. A version can be applied to a “deployment” which can be compared to a server on which you deploy your web apps.

By default there’s only one, special deployment called “Latest version” — it’s special, because you cannot apply any versions to it. The “Latest version” deployment will always run the exact code that you can see in Google’s editor, so basically it can be understood as your “dev” env.

We’d like to have another environment called “Production” that will not get broken every time we add a new character to our code. So let’s first go to File -> Manage versions and create a new version called “Initial version”.

Version management

Then go to Publish -> Deploy from manifest… , press “Create”, enter name as “Production” and select version 1 (the version we’ve just created). Now we’ll have the production environment locked on version 1 and if we ever want to update the production we just need to bump its version.

Creating new deployment

Step 3 — Setup local development

When I started working on the script I quickly started missing my favorite text editor’s (Sublime Text 3) features. Even though Google’s editor doesn’t work that bad and even has features like code completions it cannot replace the advanced editors/IDEs.

Of course one idea was to write the code in Sublime and then copy paste it to Google’s editor, however as soon as my code got bigger than 1 file it started getting pretty annoying, so I needed to find another way.

After a few minutes of googling I found a library called CLASP which is a CLI for Google Script operations. Even though it’s not very mature it’s good enough to use.
Here’s how to setup a project with CLASP:

• Add CLASP as a devDependency using yarn package manager. Runyarn add @google/clasp
• Create a src subfolder to keep the Google script files and CLASP config files (.clasp.json, .claspignore). The reason is that soon we’ll add tests folder for unit tests. Trust me ;)
• Run clasp login and login with your google credentials. You may also need to enable Google scripts API on this page: https://script.google.com/home/usersettings
• Run clasp clone <SCRIPT>.<SCRIPT_ID> can be found in the url of Google Script editor.
• Add .claspignorefile to src folder with following entries:

**/**
!*.js
!appsscript.json

• Now whenever you make a change run clasp push to update the remote code. You can also add a following script to package.json ‘s scripts : ”push”: “cd src;clasp push;cd -”

Step 4 — Prepare a connector with mock data

Ok, we already have our machine and script ready for the development, so we can now focus on connector’s code itself.

Manifest file

First let’s edit the manifest file (appsscript.json). This simple info is needed e.g. for displaying your connector in the Custom Connector Gallery.

{
  "dataStudio": {
    "name": "Spotify Connector",
    "company": "None",
    "logoUrl": "https://developer.spotify.com/assets/branding-guidelines/icon1@2x.png",
    "addonUrl": "https://github.com/Bajena/spotify-gds-connector",
    "supportUrl": "https://github.com/Bajena/spotify-gds-connector",
    "description": "This connector can be used to show stats about your last played tracks"
  }
}

getAuthType function

Let’s add the first required function to the Connectors code (typically Code.js file) — getAuthType. It informs Google Data Studio if additional authentication is needed in order to get the data. For now let’s return "NONE" here.

function getAuthType() {
  return {
    type: "NONE"
  };
}

You can find more info about this function here

getConfig function

Next step is defining a config — a list of inputs required from the user before adding the connector to the report. We won’t need a sophisticated config for the connector — only date range for fetching Spotify stats. Date range feature is built into the connectors by default, so our function will be as simple as:

function getConfig() {
  return {
    dateRangeRequired: true
  };
}

One tip from me here — if you need to add a text input field to the config and you’re trying to validate the text there’s no such way… The only thing you can do is raise an error and crash the connector completely in the runtime.

More info about this function here

getSchema function

In this function we’ll define the fields that users will be able to play around with. I’ll just paste the function here — it should be pretty self explanatory:

function getSchema() {
  return {
    schema: [
      {
        name: 'track_name',
        label: 'Track Name',
        dataType: 'STRING',
        semantics: {
          conceptType: 'DIMENSION'
        }
      },
      {
        name: 'artist',
        label: 'Artist',
        dataType: 'STRING',
        semantics: {
          conceptType: 'DIMENSION'
        }
      },
      {
        name: 'played_at_hour',
        label: 'Played at (date + hour)',
        dataType: 'STRING',
        semantics: {
          conceptType: 'DIMENSION',
          semanticGroup: 'DATETIME',
          semanticType: 'YEAR_MONTH_DAY_HOUR'
        }
      },
      {
        name: 'played_at_date',
        label: 'Played at (date)',
        dataType: 'STRING',
        semantics: {
          conceptType: 'DIMENSION',
          semanticGroup: 'DATETIME',
          semanticType: 'YEAR_MONTH_DAY'
        }
      },
      {
        name: 'plays',
        label: 'Plays',
        dataType: 'NUMBER',
        formula: 'COUNT(track_name)',
        semantics: {
          conceptType: 'METRIC',
          isReaggregatable: false
        }
      },
      {
        name: 'tracks_count',
        label: 'Played Tracks',
        dataType: 'NUMBER',
        formula: 'COUNT(track_name)',
        semantics: {
          conceptType: 'METRIC',
          isReaggregatable: false
        }
      },
      {
        name: 'popularity',
        label: 'Popularity',
        dataType: 'NUMBER',
        semantics: {
          conceptType: 'METRIC'
        }
      }
    ]
  };
}

If you need more info on what these keys mean check out Google’s docs

getData function

Now we came to the main part of each connector — getData function. It’s responsible for fetching data from third party APIs and forming it into the format accepted by Google Data Studio. For now we’ll just return mocked data. In the next part of this series we’ll start using the real Spotify API data. The following snippet is +/- the same code as in this tutorial:

function getData(request) {
  // Prepare the schema for the fields requested.
  var dataSchema = [];
  var fixedSchema = getSchema().schema;
  request.fields.forEach(function(field) {
    for (var i = 0; i < fixedSchema.length; i++) {
      if (fixedSchema[i].name == field.name) {
        dataSchema.push(fixedSchema[i]);
        break;
      }
    }
  });
// We'll query Spotify API here. For now let's just return two mocked records
  var mockedData = {
    items: [
      {
        track: {
          name: "Voice of the New Generation",
          popularity: 25,
          artists: [
            {
              name: "Santa Cruz"
            }
          ]
        },
        played_at: "2018-06-08T16:16:13.185Z"
      },
      {
        track: {
          name: "Shorty Wanna Be A Thug",
          popularity: 59,
          artists: [
            {
              name: "2Pac"
            }
          ]
        },
        played_at: "2018-06-08T14:11:02Z"
      }
    ]
  };
// Prepare the tabular data.
  var data = [];
  mockedData.items.forEach(function(play) {
    var values = [];
    var playTime = new Date(play.played_at);
    // Google expects YYMMDD format
    var playedAtDate = playTime.toISOString().slice(0, 10).replace(/-/g, "");
    // Provide values in the order defined by the schema.
    dataSchema.forEach(function(field) {
      switch (field.name) {
      case 'track_name':
        values.push(play.track.name);
        break;
      case 'artist':
        values.push(play.track.artists[0].name);
        break;
      case 'played_at_hour':
        values.push(
          playedAtDate +
          (playTime.getHours() < 10 ? '0' : '') + playTime.getHours()
        );
        break;
      case 'played_at_date':
        values.push(playedAtDate);
        break;
      case 'popularity':
        values.push(play.track.popularity);
        break;
      default:
        values.push('');
      }
    });
    data.push({
      values: values
    });
  });
return {
    schema: dataSchema,
    rows: data
  };
}

isAdminUser function

This one is not required but useful while developing the connector. If this function’s value is true then when connector crashes for some reason it’ll display the full error message to the user.

function isAdminUser() {
  return true;
}

Step 5 — Test the connector

Great we’ve built a super-basic connector! Now let’s see how it can be consumed. Go to Publish -> Deploy from manifest… and open the link to the Head deployment (you may first need to change it to to run in the context of correct user — check the GIF below).

Check out the connector in action:

Testing the connector

You can also check it out yourself here: https://datastudio.google.com/datasources/create?connectorId=AKfycbwSGHSMwF2IMeP97bzn9rO2IGaQo0d-qZD5YPNpW9ok_eMyjdQ-yL9eDorakhIABBavGg

Step 6 — debug your connector

Not always our code works great right away, especially when we’re learning new technologies, so sooner or later you’ll need a way to debug your connector code.

When your connector works on the first run

The easiest way is to sprinkle console.log("message", additionalObjectInfo) statements in your code. Later, after your script gets executed you can view the logs in Google script’s panel (Project details -> Stackdriver logs).

Also one important thing about debugging — you don’t need to create a new GDS report every time you refresh your script code. Most of the times it’ll be enough if you just press “Refresh Data” button in Google Data Studio.

Coming next…

It’s alive! In the next tutorial I’ll show you how to authenticate in Spotify’s API and query it in order to fetch the data. You’ll also learn how to cache API requests to make your connector faster.