It’s been over 2 years since I stopped doing the ‘Something for the Weekend’ series. I thought I would revive it with a tutorial on They Work For You and Google Refine…

If you want to add political context to a spreadsheet – say you need to know what political parties a list of constituencies voted for, or the MPs for those constituencies – the They Work For You API can save you hours of fiddling – if you know how to use it.

An API is – for the purposes of journalists – a way of asking questions for reams of data. For example, you can use an API to ask “What constituency is each of these postcodes in?” or “When did these politicians enter office?” or even “Can you show me an image of these people?”

The They Work For You API will give answers to a range of UK political questions on subjects including Lords, MLAs (Members of the Legislative Assembly in Northern Ireland), MPs, MSPs (Members of the Scottish Parliament), select committees, debates, written answers, statements and constituencies.

When you combine that API with Google Refine you can fill a whole spreadsheet with additional political data, allowing you to answer questions you might otherwise not be able to.

I’ve written before on how to use Google Refine to pull data into a spreadsheet from the Google Maps API and the UK Postcodes API, but this post takes things a bit further because the They Work For You API requires something called a ‘key’. This is quite common with APIs so knowing how to use them is – well – key. If you need extra help, try those tutorials first.

The They Work For You API key

Unlike the previous APIs I’ve written about, the They Work For You API requires you to register for a ‘key’ to use it. If you don’t understand how this works the instructions on the TWFY website can be a little confusing. So here’s how it works:

The key is a password of sorts, used when you ask the API a question.

As your ‘question’ takes the form of a web address (URL) then that key needs to be included at a particular part of that URL.

You’ll see how that works when we get to asking the URL questions. But first, go to http://www.theyworkforyou.com/api/key to get a key.

Got it? OK, now copy it into a text document – or just keep this window open. You’ll need to paste it later.

Using the TWFY key

The API has a number of pre-set questions, called ‘functions’. These are listed in the right hand column, and include getMPs, getLord, getDebates and so on. If you click on any of these you will be given information on how they work, and you can also test the function with the ‘Explorer’.

To demonstrate how to use these functions, click on getConstituency.

If you use the ‘Explorer’ to test it (in this case with ‘Edinburgh South”) you will be shown a bunch of results at a URL like this:

http://www.theyworkforyou.com/api/docs/getConstituency?name=edinburgh+south&postcode=&output=js#output

Now you could manually use the Explorer to get information for each of the cells in a spreadsheet, but it’s much, much quicker to use the API to automate the process instead.

On that front the Explorer can be a little misleading. Because although it shows you the information you might get from the API, this is not the URL that you will need.

The URL you really need is shown above the results, and below the word ‘Output’ like so:

http://www.theyworkforyou.com/api/getConstituency?name=edinburgh+south&output=js

If you copy and paste that URL into your browser you will get the following warning:

{
error: “No API key provided. Please see http://www.theyworkforyou.com/api/key for more information.”
}

So now we need that key.

Using your key

Assuming you still have your API key copied somewhere, or still open in another window, you can find instructions on how to use it at http://www.theyworkforyou.com/api/

Here you are told to use the key as part of the following structure:

http://www.theyworkforyou.com/api/function?key=key&output=output&other_variables

The important bit is where it says key=key&

That is where you need to add your own key, so that that part of the URL looks something like

key=aTh0jklerJaHui7&

(where that random assortment of characters is your key, copied earlier, followed by the & sign)

Going back for a moment to the URL that wasn’t working without a key, we can see that it can be split into two parts:

http://www.theyworkforyou.com/api/getConstituency?

and

name=edinburgh+south&output=js

Adding in the key in the middle makes up a third part, like so:

http://www.theyworkforyou.com/api/getConstituency?

and

key=key&

and

name=edinburgh+south&output=js

So, you now need to edit the output URL to include your API key. It should then look something like this:

http://www.theyworkforyou.com/api/getConstituency?key=AHdajHUShajshaJ&name=edinburgh+south&output=js

UPDATE: Matthew Somerville points out that the key can be used anywhere after the ? so you can tag it on the end if that’s easier.

The URL broken down further

Just to clarify, these are the parts:

http://www.theyworkforyou.com/

(The website hosting the API)

api/

(The API)

getConstituency?

(The function – or question being asked)

key=AHdajHUShajshaJ

(Our API key – or password)

&name=edinburgh+south

(and the constituency name that we are asking the API for information on)

&output=js

(and the format we want the answer in – JSON, in this case)

You should now get a page of JSON code giving data for the question. If your browser doesn’t display it particularly well, try Chrome or Firefox.

Using with Google Refine to get a bunch of results

Great. But we could get one result by using the ‘Explorer’, so why did we need to do all that? Because we can now use Google Refine to automate the process of asking the same question hundreds of times.

To demonstrate this, here’s a spreadsheet with 4 constituencies. Open it, and select File > Download as… > CSV

Open Google Refine (download here) and create a new project with that spreadsheet. Create a new column from the one you have by clicking on the arrow at the top of the column and selecting Edit Column > Add Column by fetching URLs

In the window that appears adapt the following piece of Google Refine Expression Language (GREL) with your own API key (shown in bold):

(NOTE: Avoid copying and pasting as quotation marks may cause you problems. Instead try typing it in yourself – this also helps you remember things) This generates a URL in each cell based on the value of the original column: the start and end of the URL are in quotation marks; the value is inserted in the middle where it says +value+

Give the column a name and click OK. It will now run – this test example only has 4 rows so you can see the results quickly.

You’ll see that only one row has actually worked – Tatton. The others have failed. Why? Because they have more than one word.

Take another look at that URL that the API returned earlier with the test of Edinburgh South:

http://www.theyworkforyou.com/api/getConstituency?key=AHdajHUShajshaJ&name=edinburgh+south&output=js

When a constituency has two words the space between them is represented by a plus sign – so we need to format our data in the same way for it to work.

Formatting data for the API

You could use Find and Replace in Excel to replace all spaces in that column with a plus sign but you will still hit problems with unusual constituency names. But this is how to do it in Google Refine:

Click on the arrow at the top of the constituency column and selecting Edit Column > Add column based on this column…

In the window that appears type the following GREL:

value.split(” “).join(“+”)

To explain:

‘Value’ is the value in each cell.

‘.split(” “)’ splits each value where there is a space (” “).

‘.join(“+”) then joins the resulting items together, with a plus sign.

Give it a name and click OK. You’ll see a new column with plus signs replacing the spaces. [see comment from Matthew Somerville for explanation]

Create a new column from the one you have by clicking on the arrow at the top of the column and selecting Edit Column > Add Column by fetching URLs

In the window that appears adapt the following piece of Google Refine Expression Language (GREL) with your own API key (shown in bold):

“http://www.theyworkforyou.com/api/getConstituency?name=” + escape(value, “url”) + “&key=Gr7jUUlKdhB3fsihFnHzab&output=js”

The key part here is between the + signs. Whereas before we simply inserted the value of each cell, here we escape that value at the same time so that it will work in a URL.

This will change Edinburgh South to “edinburgh+south” but also Normanton, Pontefract and Castleford to ”Normanton%2C+Pontefract+and+Castleford” and any other unforeseen characters in similar ways.

Give this new column a name, click OK and watch your new column populate itself with the JSON from each URL.

Creating new columns from the JSON

Now we can populate new columns with data taken from that JSON as follows:

Click on the arrow at the top of the new JSON column and select Edit Column > Add column based on this column…

Type this GREL:

value.parseJson().bbc_constituency_id

(This looks in the JSON in each cell and pulls out the bit after bbc_constituency_id And click OK.

Repeat the process for further columns as follows:

value.parseJson().guardian_election_results

value.parseJson().pa_id

value.parseJson().guardian_id

Going further

That’s just a demonstration of how to use a small part of the They Work For You API – there are lots of other functions that you can use to get other information. Have a play with those.

Meanwhile, what about those IDs? Well, the Guardian ID will allow you to play with The Guardian’s API – which gives lots more information on each constituency. For an example see http://www.guardian.co.uk/politics/api/constituency/664/json

Based on that URL you can repeat the process above to grab more data.

Is this useful? Anything you can add? Or other data problems?

Having worked for the BBC News Entertainment website for a decade, Darryl Chamberlain took voluntary redundancy and set up the widely successful 853 Blog. As part of the Hyperlocal Voices series he shares some of the secrets of his success.

1) Who where the people behind the blog, and what where their backgrounds?

853’s all mine. My background’s actually in showbiz news. I worked for the BBC News website’s entertainment desk for a decade in a variety of roles – mainly sub-editing and being the daily editor, but also reporting and feature writing.

I took voluntary redundancy and a career break in 2009 – standing in a council election in May 2010, and doing odd bits of freelance work. While standing in an election will probably leave me hopelessly biased in many eyes, it helped introduce me to local issues which simply weren’t being touched, and potential contacts of all political hues. After my glorious defeat, I realised I could do a bit more for my local area by sticking to what I was good at – finding things out and writing about them.

I have lived in the Greenwich area all my life, and it’s an under-reported patch, so here was my chance to do something about it. 853’s helped me keep my hand in the trade, too, which has been a nice spin-off.

More recently, I’ve set up a truly hyperlocal blog, the Charlton Champion , for the area where I live . I’m hoping to get more people involved in it, though, so it develops a different voice and its own distinctive identity. I’ve a few other people on board, but it’s very early days.

I’m also involved in a new project, The Scoop, about London news and politics.

2) When did you set up the blog and how did you go about it?

I’d blogged under a pseudonym on a couple of other sites for about five years – the usual “have a go at everyone/everything” stuff – before my impending redundancy convinced me I should try something under my own name.

I set 853 up in October 2008, using a basic WordPress template. Originally, it was going to be a showcase for my writing – I had all kinds of plans to go travelling. But the travel stuff only ended up being a small part of what the site became. Maybe I’ll pack my bags again one day and add a bit more travel.

3) What other blogs, bloggers or websites influenced you?

I’ve always thought a blog should tell you something you don’t know, instead of parroting the same old stuff. So I’ve always been in awe of Diamond Geezer , who’s looking at London’s lesser-known aspects for nearly nine years now.

Jason Cobb’s Onionbagblog was a huge influence – like me, he never set out to scrutinise his local council, but found himself doing it when nobody else was. I’m sure the leadership of Lambeth Council are breathing a sigh of relief now he’s chronicling life on the Essex coast instead.

Adam Bienkov has shown the benefits of persistence and building up good contacts in his chronicle of life at City Hall, while Brockley Central has become the model for just about anybody wanting to set up a hyperlocal blog.

My fellow Greenwich blogger The Greenwich Phantom has a distinctive take on local life which means we don’t tread on each other’s toes, Greenwich.co.uk has shown there is a demand for local news and information, while Transpontine is essential reading if you’re interested in south-east London’s music, culture and history. London SE1 is a fantastic news source which puts the big operators to shame, while Chislehurst News is a newcomer to the SE London scene which is well worth a look.

There’s a loose network of bloggers in south-east London and beyond which has been a great source of inspiration and support.

4) How did- and do you- see yourself in relation to a traditional news operation?

In south-east London, the hyperlocal blogs are partly filling a gap that’s come about because of market failure. The Greenwich area’s been largely abandoned by the big operators, leaving a couple of freesheets whose editorial is shared with neighbouring Lewisham.

The two boroughs are fairly similar socially but wildly different politically, despite both being Labour areas, and that’s where they hit problems. Combined, those freesheets are struggling to serve an area with the same population of Liverpool against a lack of interest from their proprietors – Tindle’s Mercury has great reporters but is horribly under-resourced and doesn’t even have a proper website, while Newsquest’s News Shopper is based far out in the suburbs and really doesn’t understand the area.

That said, I’d rather 853 complemented rather than competed with them – so when I deal with news I’m concentrating on council-related matters because that’s what’s getting neglected. But it still contains lots of opinion on other issues and anything else that takes my fancy.

5) What have been the key moments in the blog’s development editorially?

The turning point was going to a Greenwich Council meeting in July 2009 and watching a member of the public hectored by the mayor because he was having trouble asking a question about a housing development that affected him. It was horrifying to watch but here were no reporters there to see this – the entire meeting went unreported beyond my site.

Greenwich.co.uk’s Rob Powell asked me to cover a few meetings for him after this, and I’ve continued doing this on 853. A lot of the blog’s opinionated, but on council issues the facts usually speak for themselves.

More recently, revealing the closures of the Greenwich Foot Tunnel and the council pulling its funding from fireworks on Blackheath – claiming cuts-induced poverty despite blowing £30,000 on a mayor-making ceremony have been important moments for the blog.

Following the ongoing story of the cuts is going to become more important as time goes on – 853 was the first place to report on the initial swathe of Greenwich’s cuts and the Charlton Champion’s revealed the threat to a local petting zoo.

Covering the problems of the Southeastern train company whose press office refuses to deal with blogs – has been a boost for traffic; again, it’s an issue that’s often poorly covered elsewhere.

My background on the BBC News website’s served me well – I get frustrated if I’m not first to a story!

6) What sort of traffic do you get and how has that changed over time?

Traffic has doubled over the past year or so – it tends to go up in spurts with big stories.

There have been several events and reports worthy of note in the last week in the world of local blogs and websites .

Wired Magazine Intelligence Briefing

Recently I gave an interview about current directions in the world of local blogging, for one part of a Wired Intelligence Briefing, to @jamiedouglas.

The whole presentation identified “10 current trends in 20 minutes”, which characterise our environment:

1. We the people – isolation of the political class.
2. Abundance creates scarcity of attention.
3. Serendipity and shared experiences – where discovery happens.
4. Everybody is local – enrichment of neighbourhoods.
5. Collaborate and listen.
6. The media is unpoliceable.
7. Watch out, sport.
8. Social networks have a half-life.
9. Google’s Achilles’ heel.
10. An era of etiquette.

Talk About Local ’09

Also worth a look is a report in the Guardian about the Talk About Local ’09 Unconference, held on Saturday 3rd October in Stoke-on-Trent:

Almost 100 people left their bedrooms, home offices and local community halls for talkaboutlocal’s inaugural unconference this weekend. Some attendees in Stoke-on-Trent are professional journalists, starting out on their own against a backdrop of local and regional press lay-offs and closures, some have a political cause to fight while others quite simply want to give a voice to a community not well-served by a newspaper industry retracting and centralising.

Definitive numbers of these hyperlocal sites are hard to come by but the website www.nutshell.org.uk has already listed more than 50.

The event organiser, William Perrin, from TalkAboutLocal.org says: “People have always wanted to get involved to make things better and suddenly they can do it for themselves. The web 2.0 tools provide platforms that are incredibly easy to use, without any real cost.”

In the Guardian, the report author mentioned the practice of Stoke City Council to treat “Citizen Journalists” as being different from ‘real journalists’:

The town hall which PitsnPots set up to scrutinise has been less welcoming. Stoke Council’s head of PR and communications, Dan Barton, explained that bloggers would not be invited to briefings and are excluded from sitting at the press table in the council chamber.

“Opinion should be encouraged but we do draw a distinction between what is news otherwise we are in danger of de-valuing the role of journalists,” he said.

The National Association of Local Councils (NALC) is currently updating its guide to include the rise of social media networks but looks unlikely to change the definition of who gets treated as a journalist. A spokeswomen said: “We can say anecdotally that we would encourage councils to treat only accredited journalists as journalists. And treat citizen journalists as citizens. But that does not stop citizen journalists making enquiries in the normal way … And there is no reason why media releases cannot be available to everyone as they are public documents.”

Reflections

The comment from Dan Barton seems to imply that the content of newspapers is “news, not comment”, and that the content of blogs is “comment, not news”.

I’d suggest that this is a ludicrous position to take, bearing in mind the extent to which news and opinion are mixed in the local (and especially the national) media, and also the miraculous range of howlers and planted stories which appear regularly. To mention just two that I have covered in the the last 5 weeks, remember the non-fact-checked Mayor of Baltimore spoof, which appeared in half a dozen national media outlets, and the Alan Sugar on Terrorism Death List story, which was invented out of the air and swallowed by the Sun?

I’d assert the opposite position: that thoughtful and careful bloggers, whether our beat is “local”, “subject-based”, “professional”, “political”, “technical”, or anything else, have potential to increase significantly the range and quality of niche coverage, due our tighter focus and our freedom from a need to work tightly to deadlines and volume of output requirements.

There are two clusters of issues that we need to address:

1. We need to make sure that we do work to a level of quality which is equivalent to, or better than, our colleagues whom Mr Barton is willing to name as “journalists”.
2. We need to make sure that it is impossible for the councils we are scrutinising to refuse to accredit us for any reason.

I’ll cover these issues more in a future article, but I think it is time for local bloggers to go on the front-foot; the decline in local media has left the gate wide-open.

Two places to start are firstly to write for other outlets, such as your local newspaper and sites which are regarded as “media” not “blogs”, and to see if you can get a UK Press Card. There are a range of bodies which issue UK Press Cards, and some people may come within the rules who do not realise that they do so.

Matt Wardman is an online consultant, and edits the Wardman Wire group political blog and Nutshell directory of local blogs and websites.

The internet blows my mind. Ryan Carson opened my eyes to the power of it a few months ago. We can sit down and create a blog or web application and have it instantly accessible to the world. That’s unique, and it’s exciting.

We’re asking the BBC to join us in this creativity. Today, we’re launching BBC Free – it’s a campaign to convince the BBC to offer full article RSS feeds.

Currently, their feeds are just a single line or two and this hurts your RSS experience, and it also hinders creativity in online news. RSS feeds are machine readable and a ton of great startups base their news products off that content. By making the feed “full article”, we can be far more creative with how we improve your online news experience.

We’re not asking the BBC to create an amazing news API like The Guardian. The BBC doesn’t run adverts, any users of RSS will appreciate this change, and people who don’t use RSS won’t know anything has changed.

We’re imploring you, internets, to help us with our campaign. Full details are at our site http://bbcfree.net – the twitter hash tag is #bbcfree and you can follow the campaign at @bbcfree.

– Peter Clark, CEO of Broadersheet.

Retweet this.

As traditional media outlets close down, the relative importance of non-market players becomes more important.

Governments around the world were quick to see the opportunities for their news agencies. From Xinhua (China) to ITAR-TASS (Russia), from AFP (half of its budget comes from state subscriptions) to Voice of America, governments are trying to shape the world’s public opinion.

The coverage of Gaza by Al Jazeera is a case in point. They produced quality journalism no other outlet could dream of. Now, viewers should keep in mind that money for such newsgathering comes straight from the pocket of the Emir of Qatar. Believe me, I’m sure Al Jazeera’s journalists keep that in mind too.

To help you measure the amount of government-funded journalism, I built this little app, I smell a government rat in my news. Just type in any query and you’ll see the share of articles produced with state funds.

[iframe http://windowonthemedia.com/ismellarat/ismellaratSmall.php5 442 352]

Details about how it’s done and which sources are scanned are available at windowonthemedia.com.

by Nicolas Kayser-Bril

She was trying to make sure media (literally) used the “right” image of Barack Obama during the campaign. Jodi Williams was one of the many young brains behind Barack Obama’s media campaign.

Jodi Williams, who was part of Barack Obama's press team in the presidental campaign. (Photo: Bente Kalsnes)

I met her at the Digital News Affairs conference in Brussels to talk about the digital changes in campaigning and dealing with the media. She had no doubt that all the new digital tools made it easier for political candidates to communicate independently from mainstream media, on their own platforms.

- Absolutly. Even though mainstream media always will be important, the digital tools have given politicans fantastic opportunities to communicate directly with people. It will give a huge boom to any campaign. If Obama wants, he can address the nation every night as long as people are interested, says Williams.

One of the important differences is how easy and cost-efficient it has become to distribute campaign materials, unlike in the past, when candidates had spend fortunes on tv commercials, to use for example direct mail or hope the New York Times used a picture that made the him or her look pretty.The Obama campaign’s Flickr images from election night have already become legendary.

- When you see these images, you understand much more, you see that even a presidental candidate has to tie his shoe laces. That makes the candidate seem much more human, says Williams.

-In the past, candidates have been able to talk to people through the radio. Now, Obama was able to talk through images.

She is now working in the private sector for a company called Premier Digital Services, but before joining Obama’s team in 2007, she was also part of John Kerry and Howard Dean’s teams.

- I was present when Howard Dean did the famous “Dean Scream”. It wasn’t nearly as damaging in person as on tv. When I came back to my hotel room that night and saw this horrible soundbite on tv, I got surprised. There was so much energy in that room, so Dean’s outburst wasn’t that out of place. But the soundbite brought it out of proportions, says Williams.

She also mentions how the spread of George Allen’s 2006 racial slur on YouTube eventually took down his campaign. The digital tools are powerful, and can easily build up or break down a candidate.

As we’ve already read about, lots of bloggers followed the 2008 presidental campaign, and after Obama entered the White Office, a Huffington Post reporter made history when he became the first member of an Internet-based news organization to be called upon during an official White House press conference.

- Some bloggers, like Ben Smith from Politico, followed us almost full-time. And on all the stops we did, there were lots of bloggers who approached us. We had some safety concerns, because how can you decide who are credible or not? We only had one incident when we had to remove one blogger. But generally, they were asking really interesting questions, says Williams, and adds that a mainstream media reporter such as Brian Williams from NBC understood that he became less relevant in competition with all the bloggers.

During the DNA conference Williams got asked about mistakes done during the Obama campaign, which has been called “the perfect campaign” by many.

“The roll-out of the vice president announcement via SMS could have gone much smoother” is all she will say about that.

-Internet gave Barack Obama a fantastic opportunity to communicate directly to people.But the next five-ten years will be bumpy, and it will take time before we reach true participatory democracy, says Williams.

I’ve very quickly created a Yahoo! Pipes mashup for today’s council and London mayor elections in the UK. All it does at the moment is

• take the RSS feed for Tweetscan searches for ‘election’, ‘voted’, ‘voting’, ‘vote’, ‘Ken Livingstone‘ and ‘Boris Johnson‘,
• gets rid of duplicate results,
• and spits out a feed.
• UPDATE: Now it also takes feeds from Google News and Technorati searches for local election and the two london candidates
• It also filters out anything with ‘Zimbabwe’ in it, as reports on those elections were coming through.

I’d like to invite you to clone the mashup and make improvements. Or you can just suggest them here.

Some things I’d like to do are: add images; geo information and mapping; other feeds; filtering based on user input (e.g. location).

Meanwhile, here’s how the two mayoral candidates are faring on Twitter mentions according to a search on Twist: