Here’s an example of how APIs can be useful to journalists when they need to combine two sets of data.

I recently spoke to Lincoln investigative journalism student Sean McGrath who had obtained some information via FOI that he needed to combine with other data to answer a question (sorry to be so cryptic).

He had spent 3 days cleaning up the data and manually adding postcodes to it. This seemed a good example where using an API might cut down your work considerably, and so in this post I explain how you make a start on the same problem in less than an hour using Excel, Google Refine and the Google Maps API.

Step 1: Get the data in the right format to work with an API

APIs can do all sorts of things, but one of the things they do which is particularly useful for journalists is answer questions.

If we give the Google Maps API an address, for example, it will give us all sorts of information in return, such as latitude and longitude, postcode, and so on (I’ll explain how to do this later in the post). That’s what we’re going to use here – but we might use other APIs or datasets instead.

Sean’s spreadsheet had one column for school names and another for its town or city – but we needed those details to be together so we had a complete ‘address’. In order to do that we needed to open the spreadsheet in Excel and create a new column that combined the two.

I created the new column on the left (column A) and typed the following into cell A3:

=CONCATENATE(B3, ", ", I3)

This copies the value in cell B3, puts a comma and space after it (“, “), and then copies whatever is in I3. In other words, it combines the two cells to create a full address.

To copy the formula down the whole spreadsheet for all the other rows I used my favourite ever shortcut: hold down CTRL and clicked on the + in the bottom right corner of that cell.

Now it’s ready to use in Google Refine.

Step 2: Using Google Refine to ask Google Maps API a question

Now open the spreadsheet in Google Refine.

In Refine click on the arrow at the top of column A (the one you created using =CONCATENATE) and select Edit Column > Add Column by fetching URLs

A new window will appear with a code box. Type this:

"http://maps.googleapis.com/maps/api/geocode/json?sensor=false&address=" + escape(value, "url")

That basically creates a URL by adding the address in column A (‘value’) to the Google Maps API URL. The URL itself is basically the spreadsheet ‘asking’ the Google Maps API to give it all the information it has about the address – it also asks it to provide that information in a format called JSON (note ‘json’ in the URL)

You can see all this being done in Google Refine’s own video:

Give your new column a name, and click OK. You’ll see that the new column contains a raft of code – JSON – about each school. This contains all that geographical information – but we still need to extract an address from that.*

Step 3: Using Google Refine to extract the address from the Google Maps data

Create a further column based on the one you’ve just created by clicking the arrow at the top and selecting Edit Column > Add Column based on this column

We need to write some more code. This took a bit of trial and error but here’s what I ended up with:

value.parseJson().results[0].formatted_address

‘value’ is the value in the column we’re basing this new column on. parseJson looks through the JSON code. If you look in it you’ll see there’s a bit called ‘results’, and within that a bit called ‘formatted address’ which has what we need.

Now we have a new column with the full address – including postcode.

Step 4: Using Excel to split the address up again

We can now export this (Export is in the upper right corner of Google Refine) as a spreadsheet and open it up in Excel again.

To split that address column into its parts select it and then select Data > Text to columns to split that address into separate items, with postcodes in their own. (There are other ways you could do this, for example extracting the last 5 characters of each cell instead).

Alternatively, you could get the postcode from the JSON directly, with a different line of code at Step 3 (if you work this out let me know) – or you could extract the lat/long as detailed in the video and use the Postcodes API at http://www.uk-postcodes.com/api.php to get the postcode from that. As always there are various ways to crack the nut.

Note: I haven’t ‘learned’ JSON or GREL or any other language in this – just done a bit of searching (it took around an hour) to find the code that I needed and adapted it with educated guesswork.

*Problems 1 and 2: not all addresses return results from Google Maps because we haven’t given it enough detail. Also, there’s a 2500 limit on ‘free’ calls to their API – and we have 5000+ records, so almost 2000 are returned ‘LIMIT EXCEEDED’. A possible solution to the latter would be to split this into 2 spreadsheets and then merge the results later. A possible solution to the former may be to find – or create by scraping – another dataset that has more address information (for example this one).

FROM THE COMMENTS: Chip Oglesby suggests some other workarounds, including doing it all in Refine and using the Yahoo Maps API for half of the calls.

UPDATE: Tony Hirst follows on from this and finds other solutions to some of the problems outlined.

PHP Freelancer

If you have a spreadsheet containing geographical data such as postcodes you may want to know what constituency they are in, or convert them to local authority. That was a question that Bill Thompson asked on Twitter this week – and this is how I used Google Refine to do that: adding extra columns to a spreadsheet with geographic information.

You can watch a video tutorial of this here.

1. Find a website that gives information based on a postcode

First, I needed to find an API which would return a page of information on any postcode in JSON…

If that sounds like double-dutch, don’t worry, try this instead.

Translation: First, I needed either of these websites: http://www.uk-postcodes.com/ or http://mapit.mysociety.org/

Both of these will generate a page giving you details about any given postcode. The formatting of these pages is consistent, e.g.

• http://www.uk-postcodes.com/postcode/B422SU.json
• http://mapit.mysociety.org/postcode/B42%202SU

(The first removes the space between the two parts of the postcode, and adds .json; the second replaces the space with %20 – although I’m told by Matthew Somerville that it will work with spaces and postcodes without spaces)

This information will be important when we start to use Google Refine…

2. Create a new column that has text in the same format as the webpages you want to fetch

In Google Refine click on the arrow at the top of your postcode column and follow the instructions here to create a new column which has the same postcode information, but with no spaces. To replace the space with %20 instead you would replace the express with

value.split(" ").join("%20")

Let’s name this column ‘SpacesRemoved’ and click OK.

Now that we’ve got postcodes in the same format as the webpages above, we can start to fetch a bunch of code giving us extra information on those postcodes.

3. Write some code that goes to a webpage and fetches information about each postcode

In Google Refine click on the arrow at the top of your ‘SpacesRemoved’ column and create a new column by selecting ‘Edit column’ > ‘Add column by fetching URLs…’

You can read more about this functionality here.

This time you will type the expression:

"http://www.uk-postcodes.com/postcode/"+value+".json"

That basically creates a URL that inserts ‘value’ (the value in the previous column) where you want it.

Call this column ‘JSON for postcode’ and click OK.

Each cell will now be filled with the results of that webpage. This might take a while.

4. Write some code that pulls out a specific piece of information from that

In Google Refine click on the arrow at the top of your ‘SpacesRemoved’ column and create a new column by selecting ‘Edit column’ > ‘Add column based on this column…’

Write the following expression:

value.parseJson()["administrative"]["district"]["title"]

Look at the preview as you type this and you’ll see information become more specific as you add each term in square brackets.

Call this ‘Council’ and click OK.

This column will now be populated with the council names for each postcode. You can repeat this process for other information, adapting the expression for different pieces of information such as constituency, easting and northing, and so on.

5. Export as a standard spreadsheet

Click Export in the top right corner and save your spreadsheet in the format you prefer. You can then upload this to Google Docs and share it publicly.

Other possibilities

Although this post is about postcode data you can use the same principles to add information based on any data that you can find an API for. For example if you had a column of charities you could use the Open Charities API to pull further details (http://opencharities.org/info/about). For local authority data you could pull from the OpenlyLocal API (http://openlylocal.com/info/api).

If you know of other similarly useful APIs let me know.

PHP Freelancer

This post by Tony Hirst should be recommended reading for every journalist interested in the potential of computers for reporting.

Why? Because it shows you how you can use Google spreadsheets to interrogate data as if it was a database; and because it demonstrates the importance of news organisations releasing data to their users.

Put aside any intimidation you might feel at the mention of APIs and query languages. What it boils down to is this: you can alter the web address of a Google spreadsheet to filter the data and find the story.

Simple as that. 

Hirst uses the example of the spreadsheet of MPs expenses recently released by The Guardian (they’ve also published Lords expenses). By altering the URLs this is what he generates (I’m quoting his bullet points):

• the names of people who have claimed the maximum additional costs allowance (£23,083): fetch just columns B, C and I where the value in column I is 23083: select B,C,I where I=23083 (column I is the additional costs allowance column);
• How many people did claim the maximum additional costs allowance? Select the people who claimed the maximum amount (23083) and count them: select count(I) where I=23083
• So which people did not claim the maximum additional costs allowance? Display the people who did not claim total additional allowances of 23083: select B,C,I where I!=23083 (using <> for ‘not equals’ also works); NB here’s a more refined take on that query: select B,C,I where (I!=23083 and I>=0) order by I
• search for the name, party (column D) and constituency (column E) of people whose first name is Jane or is recorded as John (rather than “Mr John”, or “Rt Hon John”): select B,C,D,E where (C contains ‘Joan’ or C matches ‘John’)
• only show the people who have claimed less than £100,000 in total allowances : select * where F<100000
• what is the total amount of expenses claimed? Fetch the summed total of entries in column I (i.e. the total expenses claimed by everyone): select sum(I)
• So how many MPs are there? Count the number of rows in an arbitrary column: select count(I)
• Find the average amount claimed by the MPs: select sum(I)/count(I)
• Find out how much has been claimed by each party (column D): select D,sum(I) where I>=0 group by D (Setting I>0 just ensures there is something in the column)
• For each party, find out how much (on average) each party member claims:select%20D,sum(I)/count(I)%20where%20I%3E=0%20group%20by%20D”>select D,sum(I)/count(I) where I>=0 group by D

OK, you need to know the words to use (and if you have a link to an easy reference for these let me know*), but this is still a lot easier than using programming languages and databases.

As I say, this also illustrates the importance of publishing raw data so users can interrogate it in their own ways, which is precisely what The Guardian’s Data Store has been doing, meaning that people like Tony can create interfaces like this.

Wonderful.

*Tony has very generously created this page which helps you formulate your search – and generates the URL. If you were working on a different spreadsheet you could just replace the spreadsheet URL and change any column references accordingly.

UPDATE: Tony also has a version which allows you to pick from Guardian datasets.

PHP Freelancer