It’s been over 2 years since I stopped doing the ‘Something for the Weekend’ series. I thought I would revive it with a tutorial on They Work For You and Google Refine…

If you want to add political context to a spreadsheet – say you need to know what political parties a list of constituencies voted for, or the MPs for those constituencies – the They Work For You API can save you hours of fiddling – if you know how to use it.

An API is – for the purposes of journalists – a way of asking questions for reams of data. For example, you can use an API to ask “What constituency is each of these postcodes in?” or “When did these politicians enter office?” or even “Can you show me an image of these people?”

The They Work For You API will give answers to a range of UK political questions on subjects including Lords, MLAs (Members of the Legislative Assembly in Northern Ireland), MPs, MSPs (Members of the Scottish Parliament), select committees, debates, written answers, statements and constituencies.

When you combine that API with Google Refine you can fill a whole spreadsheet with additional political data, allowing you to answer questions you might otherwise not be able to.

I’ve written before on how to use Google Refine to pull data into a spreadsheet from the Google Maps API and the UK Postcodes API, but this post takes things a bit further because the They Work For You API requires something called a ‘key’. This is quite common with APIs so knowing how to use them is – well – key. If you need extra help, try those tutorials first.

The They Work For You API key

Unlike the previous APIs I’ve written about, the They Work For You API requires you to register for a ‘key’ to use it. If you don’t understand how this works the instructions on the TWFY website can be a little confusing. So here’s how it works:

The key is a password of sorts, used when you ask the API a question.

As your ‘question’ takes the form of a web address (URL) then that key needs to be included at a particular part of that URL.

You’ll see how that works when we get to asking the URL questions. But first, go to http://www.theyworkforyou.com/api/key to get a key.

Got it? OK, now copy it into a text document – or just keep this window open. You’ll need to paste it later.

Using the TWFY key

The API has a number of pre-set questions, called ‘functions’. These are listed in the right hand column, and include getMPs, getLord, getDebates and so on. If you click on any of these you will be given information on how they work, and you can also test the function with the ‘Explorer’.

To demonstrate how to use these functions, click on getConstituency.

If you use the ‘Explorer’ to test it (in this case with ‘Edinburgh South”) you will be shown a bunch of results at a URL like this:

http://www.theyworkforyou.com/api/docs/getConstituency?name=edinburgh+south&postcode=&output=js#output

Now you could manually use the Explorer to get information for each of the cells in a spreadsheet, but it’s much, much quicker to use the API to automate the process instead.

On that front the Explorer can be a little misleading. Because although it shows you the information you might get from the API, this is not the URL that you will need.

The URL you really need is shown above the results, and below the word ‘Output’ like so:

http://www.theyworkforyou.com/api/getConstituency?name=edinburgh+south&output=js

If you copy and paste that URL into your browser you will get the following warning:

{
error: “No API key provided. Please see http://www.theyworkforyou.com/api/key for more information.”
}

So now we need that key.

Using your key

Assuming you still have your API key copied somewhere, or still open in another window, you can find instructions on how to use it at http://www.theyworkforyou.com/api/

Here you are told to use the key as part of the following structure:

http://www.theyworkforyou.com/api/function?key=key&output=output&other_variables

The important bit is where it says key=key&

That is where you need to add your own key, so that that part of the URL looks something like

key=aTh0jklerJaHui7&

(where that random assortment of characters is your key, copied earlier, followed by the & sign)

Going back for a moment to the URL that wasn’t working without a key, we can see that it can be split into two parts:

http://www.theyworkforyou.com/api/getConstituency?

and

name=edinburgh+south&output=js

Adding in the key in the middle makes up a third part, like so:

http://www.theyworkforyou.com/api/getConstituency?

and

key=key&

and

name=edinburgh+south&output=js

So, you now need to edit the output URL to include your API key. It should then look something like this:

http://www.theyworkforyou.com/api/getConstituency?key=AHdajHUShajshaJ&name=edinburgh+south&output=js

UPDATE: Matthew Somerville points out that the key can be used anywhere after the ? so you can tag it on the end if that’s easier.

The URL broken down further

Just to clarify, these are the parts:

http://www.theyworkforyou.com/

(The website hosting the API)

api/

(The API)

getConstituency?

(The function – or question being asked)

key=AHdajHUShajshaJ

(Our API key – or password)

&name=edinburgh+south

(and the constituency name that we are asking the API for information on)

&output=js

(and the format we want the answer in – JSON, in this case)

You should now get a page of JSON code giving data for the question. If your browser doesn’t display it particularly well, try Chrome or Firefox.

Using with Google Refine to get a bunch of results

Great. But we could get one result by using the ‘Explorer’, so why did we need to do all that? Because we can now use Google Refine to automate the process of asking the same question hundreds of times.

To demonstrate this, here’s a spreadsheet with 4 constituencies. Open it, and select File > Download as… > CSV

Open Google Refine (download here) and create a new project with that spreadsheet. Create a new column from the one you have by clicking on the arrow at the top of the column and selecting Edit Column > Add Column by fetching URLs

In the window that appears adapt the following piece of Google Refine Expression Language (GREL) with your own API key (shown in bold):

(NOTE: Avoid copying and pasting as quotation marks may cause you problems. Instead try typing it in yourself – this also helps you remember things) This generates a URL in each cell based on the value of the original column: the start and end of the URL are in quotation marks; the value is inserted in the middle where it says +value+

Give the column a name and click OK. It will now run – this test example only has 4 rows so you can see the results quickly.

You’ll see that only one row has actually worked – Tatton. The others have failed. Why? Because they have more than one word.

Take another look at that URL that the API returned earlier with the test of Edinburgh South:

http://www.theyworkforyou.com/api/getConstituency?key=AHdajHUShajshaJ&name=edinburgh+south&output=js

When a constituency has two words the space between them is represented by a plus sign – so we need to format our data in the same way for it to work.

Formatting data for the API

You could use Find and Replace in Excel to replace all spaces in that column with a plus sign but you will still hit problems with unusual constituency names. But this is how to do it in Google Refine:

Click on the arrow at the top of the constituency column and selecting Edit Column > Add column based on this column…

In the window that appears type the following GREL:

value.split(” “).join(“+”)

To explain:

‘Value’ is the value in each cell.

‘.split(” “)’ splits each value where there is a space (” “).

‘.join(“+”) then joins the resulting items together, with a plus sign.

Give it a name and click OK. You’ll see a new column with plus signs replacing the spaces. [see comment from Matthew Somerville for explanation]

Create a new column from the one you have by clicking on the arrow at the top of the column and selecting Edit Column > Add Column by fetching URLs

In the window that appears adapt the following piece of Google Refine Expression Language (GREL) with your own API key (shown in bold):

“http://www.theyworkforyou.com/api/getConstituency?name=” + escape(value, “url”) + “&key=Gr7jUUlKdhB3fsihFnHzab&output=js”

The key part here is between the + signs. Whereas before we simply inserted the value of each cell, here we escape that value at the same time so that it will work in a URL.

This will change Edinburgh South to “edinburgh+south” but also Normanton, Pontefract and Castleford to ”Normanton%2C+Pontefract+and+Castleford” and any other unforeseen characters in similar ways.

Give this new column a name, click OK and watch your new column populate itself with the JSON from each URL.

Creating new columns from the JSON

Now we can populate new columns with data taken from that JSON as follows:

Click on the arrow at the top of the new JSON column and select Edit Column > Add column based on this column…

Type this GREL:

value.parseJson().bbc_constituency_id

(This looks in the JSON in each cell and pulls out the bit after bbc_constituency_id And click OK.

Repeat the process for further columns as follows:

value.parseJson().guardian_election_results

value.parseJson().pa_id

value.parseJson().guardian_id

Going further

That’s just a demonstration of how to use a small part of the They Work For You API – there are lots of other functions that you can use to get other information. Have a play with those.

Meanwhile, what about those IDs? Well, the Guardian ID will allow you to play with The Guardian’s API – which gives lots more information on each constituency. For an example see http://www.guardian.co.uk/politics/api/constituency/664/json

Based on that URL you can repeat the process above to grab more data.

Is this useful? Anything you can add? Or other data problems?

RSS girl by HeatherWeaver on Flickr

During some training in open data I was doing recently, I ended up explaining (it’s a long story) how to pull a feed from Delicious into a Google Docs spreadsheet. I promised I would put it down online, so: here it is.

In a Google Docs spreadsheet the formula =importfeed will pull information from an RSS feed and put it into that spreadsheet. Titles, links, datestamps and other parts of the feed will each be separated into their own columns.

When combined with Delicious, this can be a useful way to collect together pages that have been bookmarked by a group of people, or any other feed that you want to analyse.

Here’s how you do it:

1. Decide on your tag, network or user

The spreadsheet will pull data from an RSS feed. Delicious provides so many of these that you are spoilt for choice. Here are the main three:

A tag

Used by various people.

Advantages: quick startup – all you need to do is tell people the tag (make sure this is unique, such as ‘unguessable2012′).

Disadvantages: others can hijack the tag – although this can be cleaned from the resulting data.

A network

Consisting of the group of people who are bookmarking:

Advantages: group cannot be infiltrated.

Disadvantages: setup time – may need to create a new account to build the network around.

A user

Created for this purpose:

Advantages: if users are not confident in using Delicious, this can be a useful workaround.

Disadvantages: longer set up time – you’ll need to create a new account, and work out an easy way for it to automatically capture bookmarks from the group. One way is to pull an RSS feed of any mentions on Twitter and use Twitterfeed to auto-tweet them with a hashtag, and then Packrati.us to auto-bookmark all tweeted links (a similar process is detailed here).

The RSS feed for each will be found at the bottom of pages, and is consistently formatted like so:

Delicious.com/tag/unguessable2012

Delicious.com/network/unguessable2012

Delicious.com/unguessable2012

2. Create your spreadsheet

In Google Docs, create a new spreadsheet and in the first cell type the following formula:

=importfeed(“

…adding your RSS feed after the quotation mark, and then this at the end:

“)

So it looks something like this:

=importfeed(“http://feeds.delicious.com/v2/rss/tag/unguessable2012?count=15″)

Now press enter and after a moment the spreadsheet should populate with data from that feed.

You’ll note, however, that at most you will have only 15 rows of data here. That’s because the RSS feed you’ve copied includes that limitation.

If you look at the RSS feed you’ll see an easy clue on how to change this…

So, try editing it so that the count=15 part of that URL reads count=20 instead. You can put a higher number – but Google Docs will limit results to 20 at a time.

3. Collecting contributions

Technically, you’re now all set up. The bigger challenge is, of course, in getting people to contribute. It helps if they can see the results – so think about publishing your spreadsheet.

You’ll also need to make sure that you check it regularly and copy into a backup spreadsheet so you don’t miss results after that top 20.

But if you find it doesn’t work it may be worth thinking of other ways of doing this – for example, with a Google Form, or using =importfeed with the RSS feed for a search on results for a Twitter hashtag containing links (Twitter’s advanced search allows you to limit results accordingly – and all search results come with an RSS feed link like this one)

Of course there are far more powerful ways of doing this which are worth exploring once you’ve understood the basic possibilities.

Doing more with =importfeed

The =importfeed formula has some other elements that we haven’t used.

Another way to do this, for example, is to paste your RSS feed URL into cell A1 and type the following anywhere else:

=importfeed(A1, ”Items Title”, FALSE, 20)

This has 4 parts in the parentheses:

1. A1 – this points at the URL you just pasted in cell A1, and means that you only have to change what’s in A1 to change the feed being grabbed, rather than having to edit the formula itself
2. “Items Title” – this is the part of the feed that is being grabbed. If you look in the feed you will see a part that says <item> and within that, an element called <title> – that’s it. You could change this to “Items URL” to get the <URL> part of <title> instead, for example. Or you could just put “Items” and get all 5 parts of each item (title, author, URL, date created, and summary). You can also use “feed” to get information about the feed itself, or “feed URL” or “feed title” or “feed description” to get that single piece of information.
3. FALSE – this just says whether you want a header row or not. Setting to TRUE will add an extra row saying ‘Title’, for example.
4. 20 – the number of results you want.

You can see an example spreadsheet with 3 sheets demonstrating different uses of this formula here.

Anyway, I spent a fair chunk of yesterday trying to find a way of getting started with grappling with CSV data text files that are just a bit too big to comfortably manage in a text editor or simple spreadsheet (so files over 50,000 or so rows, up to low millions) and that should probably be dumped into a database if that option was available, but for whatever reason, isn’t… (Not feeling comfortable with setting up and populating a database is one example…But I doubt I’ll get round to blogging my SQLite 101 for a bit yet…)

Note that the following tools are Unix tools – so they work on Linux and on a Mac, but probably not on Windows unless you install a unix tools package (such as GnuWin – coreutils and sed, which look good for starters…). Another alternative would be to download the Data Journalism Developer Studio and run it either as a bootable CD/DVD, or as a virtual machine using something like VMWare or VirtualBox.

All the tools below are related to the basic mechanics of wrangling with text files, which include CSV (comma separated) and TSV (tab separated) files. Your average unix jockey will look at you with sympathetic eyes if you rave bout them, but for us mere mortals, they may make life easier for you than you ever thought possible…

[If you know of simple tricks in the style of what follows that I haven't included here, please feel free to add them in as a comment, and I'll maybe try to work then into a continual updating of this post...]

If you want to play along, why not check out this openurl data from EDINA (data sample; a more comprehensive set is also available if you’re feeling brave: monthly openurl data).

So let’s start at the beginning and imagine your faced with a large CSV file – 10MB, 50MB, 100MB, 200MB large – and when you try to open it in your text editor (the file’s too big for Google spreadsheets and maybe even for Google Fusion tables) the whole thing just grinds to a halt, if doesn’t actually fall over.

What to do?

To begin with, you may want to take a deep breath and find out just what sort of beast you have to contend with. You know the file size, but what else might you learn? (I’m assuming the file has a csv suffix, L2sample.csv say, so for starters we’re assuming it’s a text file…)

The wc (word count) command is a handy little tool that will give you a quick overview of how many rows there are in the file:

wc -l L2sample.csv

I get the response 101 L2sample.csv, so there are presumably 100 data rows and 1 header row.

We can learn a little more by taking the -l linecount switch off, and getting a report back on the number of words and characters in the file as well:

wc L2sample.csv

Another thing that you might consider doing is just having a look at the structure of the file, by sampling the first few rows of it and having a peek at them. The head command can help you here.

head L2sample.csv

By default, it returns the first 10 rows of the file. IF we want to change the number of rows displayed, we can use the -n switch:

head -n 4 L2sample.csv

As well as the head command, there is the tail command; this can be used to peek at the lines at the end of the file:

tail L2sample.csv
tail -n 15 L2sample.csv

When I look at the rows, I see they have the form:

logDate	logTime	encryptedUserIP	institutionResolverID	routerRedirectIdentifier ...
2011-04-04	00:00:03	kJJNjAytJ2eWV+pjbvbZTkJ19bk	715781	ukfed ...
2011-04-04	00:00:14	/DAGaS+tZQBzlje5FKsazNp2lhw	289516	wayf ...
2011-04-04	00:00:15	NJIy8xkJ6kHfW74zd8nU9HJ60Bc	569773	athens ...

So, not comma separated then; tab separated…;-)

If you were to upload a tab separated file to something like Google Fusion Tables, which I think currently only parses CSV text files for some reason, it will happily spend the time uploading the data – and then shove it into a single column.

I’m not sure if there are column splitting tools available in Fusion Tables – there weren’t last time I looked, though maybe we might expect a fuller range of import tools to appear at some point; many applications that accept text based data files allow you to specify the separator type, as for example in Google spreadsheets:

I’m personally living in hope that some sort of integration with the Google Refine data cleaning tool will appear one day…

If you want to take a sample of a large data file and put into another smaller file that you can play with or try things out with, the head (or tail) tool provides one way of doing that thanks to the magic of Unix redirection (which you might like to think of as a “pipe”, although that has a slightly different meaning in Unix land…). The words/jargon may sound confusing, and the syntax may look cryptic, but the effect is really powerful: take the output from a command and shove it into a file.

So, given a CSV file with a million rows, suppose we want to run a few tests in an application using a couple of hundred rows. This trick will help you generate the file containing the couple of hundred rows.

Here’s an example using L2sample.csv – we’ll create a file containing the first 20 rows, plus the header row:

head -n 21 L2sample.csv > subSample.csv

See the > sign? That says “take the output from the command on the left, and shove it into the file on the right”. (Note that if subSample.csv already exists, it will be overwritten, and you will lose the original.)

There’s probably a better way of doing this, but if you want to generate a CSV file (with headers) containing the last 10 rows, for example, of a file, you can use the cat command to join a file containing the headers with a file containing the last 10 rows:

head -n 1 L2sample.csv > headers.csv
tail -n 20 L2sample.csv > subSample.csv
cat headers.csv subSample.csv > subSampleWithHeaders.csv

(Note: don’t try to cat a file into itself, or Ouroboros may come calling…)

Another very powerful concept from the Unix command line is the notion of | (the pipe). This lets you take the output from one command and direct it to another command (rather than directing it into a file, as > does). So for example, if we want to extract rows 10 to 15 from a file, we can use head to grab the first 15 rows, then tail to grab the last 6 rows of those 15 rows (count them: 10, 11, 12, 13, 14, 15):

head -n 15 L2sample.csv | tail -n 6 > middleSample.csv

Try to read in as an English phrase (the | and > are punctuation): take the the first [head] 15 rows [-n 15] of the file L2sample.csv and use them as input [|] to the tail command; take the last [tail] 6 lines [-n 6] of the input data and save them [>] as the file middleSample.csv.

If we want to add in the headers, we can use the cat command:

cat headers.csv middleSample.csv > middleSampleWithHeaders.csv

We can use a pipe to join all sorts of commands. If our file only uses a single word for each column header, we can count the number of columns (single words) by grabbing the header row and sending it to wc, which will count the words for us:

head -n 1 L2sample.csv | wc

(Take the first row of L2sample.csv and count the lines/words/characters. If there is one word per column header, the word count gives us the column count…;-)

Sometimes we just want to split a big file into a set of smaller files. The split command is our frind here, and lets us split a file into smaller files containing up to a know number of rows/lines:

split -l 15 L2sample.csv subSamples

This will generate a series of files named subSamplesaa, subSamplesab, …, each containing 15 lines (except for the last one, which may contain less…).

Note that the first file will contain the header and 14 data rows, and the other files will contain 15 data rows but no column headings. To get round this, you might want to split on a file that doesn’t contain the header. (So maybe use wc -l to find the number of rows in the original file, create a header free version of the data by using tail on one less than the number of rows in the file, then split the header free version. You might then one to use cat to put the header back in to each of the smaller files…)

A couple of other Unix text processing tools let us use a CSV file as a crude database. The grep searches a file for a particular term or text pattern (known as a regular expression, which I’m not going to cover much in this post… suffice to note for now that you can do real text processing voodoo magic with regular expressions…;-)

So for example, in out test file, I can search for rows that contain the word mendeley

grep mendeley L2sample.csv

We can also redirect the output into a file:

grep EBSCO L2sample.csv > rowsContainingEBSCO.csv

If the text file contains columns that are separated by a unique delimiter (that is, some symbol that is only ever used to separate the columns), we can use the cut command to just pull out particular columns. The cut command assumes a tab delimiter (we can specify other delimiters explicitly if we need to), so we can use it on our testfile to pull out data from the third column in our test file:

cut -f 3 L2sample.csv

We can also pull out multiple columns and save them in a file:

cut -f 1,2,14,17 L2sample.csv > columnSample.csv

If you pull out just a single column, you can sort the entries to see what different entries are included in the column using the sort command:

cut -f 40 L2sample.csv | sort

(Take column 40 of the file L2sample.csv and sort the items.)

We can also take this sorted list and identify the unique entries using the uniq command; so here are the different entries in column 40 of our test file:

cut -f 40 L2sample.csv | sort | uniq

(Take column 40 of the file L2sample.csv, sort the items, and display the unique values.)

(The uniq command appears to make comparaisons between consecutive lines, hence the nee to sort first.)

The uniq command will also count the repeat occurrence of unique entries if we ask it nicely (-c):

cut -f 40 L2sample.csv | sort | uniq -c

(Take column 40 of the file L2sample.csv, sort the items, and display the unique values along with how many times they appear in the column as a whole.)

The final command I’m going to mention here is magic search and replace operator called sed. I’m aware that this post is already over long, so I’ll maybe return to this in a later post, aside from giving you a tease of scome scarey voodoo… how to convert a tab delimited file to a comma separated file. One recipe is given by Kevin Ashley as follows:

sed 's/"/\\\"/g; s/^/"/; s/$/"/; s/ctrl-V<TAB>/","/g;' origFile.tsv > newFile.csv

(See also this related question on #getTheData: Converting large-ish tab separated files to CSV.)

Note: if you have a small amount of text and need to wrangle it on some way, the Text Mechanic site might have what you need…

This lecture note on Unix Tools provides a really handy cribsheet of Unix command line text wrangling tools, though the syntax does appear to work for me using some of the commands as given their (the important thing is the idea of what’s possible…).

If you’re looking for regular expression helpers (I haven’t really mentioned these at all in this post, suffice to say they’re a mechanism for doing pattern based search and replace, and which in the right hands can look like real voodoo text processing magic!), check out txt2re and Regexpal (about regexpal).

TO DO: this is a biggie – the join command will join rows from two files with common elements in specified columns. I canlt get it working properly with my test files, so I’m not blogging it just yet, but here’s a starter for 10 if you want to try… Unix join examples

Why? Two reasons… Firstly, because it demonstrates a “have a go” attitude which you absolutely need to have if you’re going to appropriate technology and turn it to your own purposes. Secondly, because it maybe (or maybe not…) hints at a missed trick or two…

So what trick’s missing?

Here’s an example of the sort of thing you get back from the Google Geocoder:

{ “status”: “OK”, “results”: [ { "types": [ "postal_code" ], “formatted_address”: “Milton Keynes, Buckinghamshire MK7 6AA, UK”, “address_components”: [ { "long_name": "MK7 6AA", "short_name": "MK7 6AA", "types": [ "postal_code" ] }, { “long_name”: “Milton Keynes”, “short_name”: “Milton Keynes”, “types”: [ "locality", "political" ] }, { “long_name”: “Buckinghamshire”, “short_name”: “Buckinghamshire”, “types”: [ "administrative_area_level_2", "political" ] }, { “long_name”: “Milton Keynes”, “short_name”: “Milton Keynes”, “types”: [ "administrative_area_level_2", "political" ] }, { “long_name”: “United Kingdom”, “short_name”: “GB”, “types”: [ "country", "political" ] }, { “long_name”: “MK7″, “short_name”: “MK7″, “types”: [ "postal_code_prefix", "postal_code" ] } ], “geometry”: { “location”: { “lat”: 52.0249136, “lng”: -0.7097474 }, “location_type”: “APPROXIMATE”, “viewport”: { “southwest”: { “lat”: 52.0193722, “lng”: -0.7161451 }, “northeast”: { “lat”: 52.0300728, “lng”: -0.6977000 } }, “bounds”: { “southwest”: { “lat”: 52.0193722, “lng”: -0.7161451 }, “northeast”: { “lat”: 52.0300728, “lng”: -0.6977000 } } } } ] }

The data represents a Javascript object (JSON = JavaScript Object Notation) and as such has a standard form, a hierarchical form.

Here’s another way of writing the same object code, only this time laid out in a way that reveals the structure of the object:

{
  "status": "OK",
  "results": [ {
    "types": [ "postal_code" ],
    "formatted_address": "Milton Keynes, Buckinghamshire MK7 6AA, UK",
    "address_components": [ {
      "long_name": "MK7 6AA",
      "short_name": "MK7 6AA",
      "types": [ "postal_code" ]
    }, {
      "long_name": "Milton Keynes",
      "short_name": "Milton Keynes",
      "types": [ "locality", "political" ]
    }, {
      "long_name": "Buckinghamshire",
      "short_name": "Buckinghamshire",
      "types": [ "administrative_area_level_2", "political" ]
    }, {
      "long_name": "Milton Keynes",
      "short_name": "Milton Keynes",
      "types": [ "administrative_area_level_2", "political" ]
    }, {
      "long_name": "United Kingdom",
      "short_name": "GB",
      "types": [ "country", "political" ]
    }, {
      "long_name": "MK7",
      "short_name": "MK7",
      "types": [ "postal_code_prefix", "postal_code" ]
    } ],
    "geometry": {
      "location": {
        "lat": 52.0249136,
        "lng": -0.7097474
      },
      "location_type": "APPROXIMATE",
      "viewport": {
        "southwest": {
          "lat": 52.0193722,
          "lng": -0.7161451
        },
        "northeast": {
          "lat": 52.0300728,
          "lng": -0.6977000
        }
      },
      "bounds": {
        "southwest": {
          "lat": 52.0193722,
          "lng": -0.7161451
        },
        "northeast": {
          "lat": 52.0300728,
          "lng": -0.6977000
        }
      }
    }
  } ]
}

Making Sense of the Notation

At its simplest, the structure has the form: {“attribute”:”value”}

If we parse this object into the jsonObject, we can access the value of the attribute as jsonObject.attribute or jsonObject["attribute"]. The first style of notation is called a dot notation.

We can add more attribute:value pairs into the object by separating them with commas: a={“attr”:”val”,”attr2″:”val2″} and address them (that is, refer to them) uniquely: a.attr, for example, or a["attr2"].

Try it out for yourself… Copy and past the following into your browser address bar (where the URL goes) and hit return (i.e. “go to” that “location”):

javascript:a={"attr":"val","attr2":"val2"}; alert(a.attr);alert(a["attr2"])

(As an aside, what might you learn from this? Firstly, you can “run” javascript in the browser via the location bar. Secondly, the javascript command alert() pops up an alert box:-)

Note that the value of an attribute might be another object.

obj={ attrWithObjectValue: { “childObjAttr”:”foo” } }

Another thing we can see in the Google geocoder JSON code are square brackets. These define an array (one might also think of it as an ordered list). Items in the list are address numerically. So for example, given:

arr[ "item1", "item2", "item3" ]

we can locate “item1″ as arr[0] and “item3″ as arr[2]. (Note: the index count in the square brackets starts at 0.) Try it in the browser… (for example, javascript:list=["apples","bananas","pears"]; alert( list[1] );).

Arrays can contain objects too:

list=[ "item1", {"innerObjectAttr":"innerObjVal" } ]

Can you guess how to get to the innerObjVal? Try this in the browser location bar:

javascript: list=[ "item1", { "innerObjectAttr":"innerObjVal" } ]; alert( list[1].innerObjectAttr )

Making Life Easier

Hopefully, you’ll now have a sense that there’s structure in a JSON object, and that that (sic) structure is what we rely on if we want to cut down on the “trial an error” when parsing such things. To make life easier, we can also use “tree widgets” to display the hierarchical JSON object in a way that makes it far easier to see how to construct the dotted path that leads to the data value we want.

A tool I have appropriated for previewing JSON objects is Yahoo Pipes. Rather than necessarily using Pipes to build anything, I simply make use of it as a JSON viewer, loading JSON into the pipe from a URL via the Fetch Data block, and then previewing the result:

Another tool (and one I’ve just discovered) is an Air application called JSON-Pad. You can paste in JSON code, or pull it in from a URL, and then preview it again via a tree widget:

Clicking on one of the results in the tree widget provides a crib to the path…

Summary

Getting to grips with writing addresses into JSON objects helps if you have some idea of the structure of a JSON object. Tree viewers make the structure of an object explicit. By walking down the tree to the part of it you want, and “dotting” together* the nodes/attributes you select as you do so, you can quickly and easily construct the path you need.

* If the JSON attributes have spaces or non-alphanumeric characters in them, use the obj["attr"] notation rather than the dotted obj.attr notation…

PS Via my feeds today, though something I had bookmarked already, this Data Converter tool may be helpful in going the other way… (Disclaimer: I haven’t tried using it…)

If you know of any other related tools, please feel free to post a link to them in the comments:-)

I’ve blogged previously about what Yahoo! Pipes can do. The following describes some of the basic mashups you can create with Yahoo! Pipes – please add your own tips in the comments.

Signing up

First you’ll need to go to pipes.yahoo.com and register with the page. If you already have a Yahoo! or Flickr account you may be able to use that.

Aggregating feeds into one

1. Log on to Yahoo! Pipes and click on Create Pipe. You should be presented with a ‘graph’-style page.
2. On the left column are a number of buttons – called modules. These are arranged within different categories, the first category being Sources. In the Sources category should be a module called Fetch Feed. Click and drag this onto the graphed area.
3. You will need to copy the URL of the RSS feed you want to fetch and paste it into the Fetch Feed module input box.
4. To add extra feeds click on the plus (+) icon next to URL and further input boxes will appear. Paste the extra feeds into each new box.
5. Finally, you need to connect the Fetch Feed module to the Pipe Output. To do this click on the circle at the bottom of the Fetch Feed module and drag it to the circle at the top of Pipe Output. You should now see a pipe appear connecting the two.
6. Click on Pipe Output to see the results at the bottom of the screen.
7. That’s it. Click Save (top right), give the pipe a name, then click Run Pipe… at the top of the screen. Note: the results may be displayed as images – click List to see the text version.
8. Along the top of the results you will see various options. Click on the RSS symbol to get the option to have the output of this pipe as a standalone RSS feed. You can also click Get as a Badge to get some HTML to put on your blog and display results more attractively.

Note: with more than one feed Pipes will ‘cluster’ them together by feed rather than by date. To order the results by date, use the Sort module under the Operators category, connect it to Fetch Feed, and sort by item.pubDate in descending order.

If you want to aggregate feeds after filtering, etc. you can use the Union module under Operators category.

Filtering feeds

1. Follow steps 1-4 for Aggregating feeds, above.
2. On the left column are a number of buttons – called modules. These are arranged within different categories. Expand the Operators category. There should be a module called Filter. Click and drag this onto the graphed area.
3. You need to connect the Fetch Feed module to the Filter module. To do this click on the circle at the bottom of the Fetch Feed module and drag it to the circle at the top of Filter. You should now see a pipe appear connecting the two.
4. Using the settings in the Filter module you can choose to filter the aggregated feed by blocking items (posts) with certain words, or only allowing items with certain words to come through.
5. You will then need to choose from the drop-down menu which field (e.g. ‘title’ or ‘category’) you want to be the subject of the filter.
6. Finally, you need to connect the Filter module to the Pipe Output. To do this click on the circle at the bottom of the Filter module and drag it to the circle at the top of Pipe Output. You should now see a pipe appear connecting the two.
7. Follow steps 6-8 for Aggregating feeds, above to finish.

Note: you can use the Unique module instead to filter out multiple versions of the same post (e.g. when you’re using feeds from search results on different engines)

Translating feeds

The translation tool in Pipes (BabelFish) is pretty clumsy, and you couldn’t rely on it to produce a clear and understandable feed for readers – but it may be useful to highlight leads in other countries that you otherwise wouldn’t see, and which you can then follow up.

1. Follow steps 1-4 for Aggregating feeds, above.
2. On the left column are a number of buttons – called modules. These are arranged within different categories. Expand the Deprecated category. There should be a module called BabelFish. Click and drag this onto the graphed area.
3. You need to connect the Fetch Feed module to the BabelFish module. To do this click on the circle at the bottom of the Fetch Feed module and drag it to the circle at the top of BabelFish. You should now see a pipe appear connecting the two.
4. Using the settings in the BabelFish module you can choose to translate the feed from one language to another.
5. Finally, you need to connect the BabelFish module to the Pipe Output. To do this click on the circle at the bottom of the BabelFish module and drag it to the circle at the top of Pipe Output. You should now see a pipe appear connecting the two.
6. Follow steps 6-8 for Aggregating feeds, above to finish.

Tips and tools

• If the website you want to use doesn’t have an RSS feed you may be able to create one using a service like Page 2 RSS (http://page2rss.com/).
• Pipes has a specific module to help you pull images from Flickr (under Sources).
• The Feed Auto-Discovery module allows you to just input the URL of the website, not the RSS feed itself, and will automatically find feeds. This may pull in comment and other feeds, however.
• You can always search everyone’s pipes to find something that does what you want to do – and clone it to adapt it accordingly.
• Although Pipes allows you to create email alerts and widgets, Feedburner.com is even better at doing the same thing – and will tell you who has subscribed, etc.

If you liked this…

Webpages about Yahoo! Pipes I’ve bookmarked