Monthly Archives: March 2011

Getting full addresses for data from an FOI response (using APIs)


Here’s an example of how APIs can be useful to journalists when they need to combine two sets of data.

I recently spoke to Lincoln investigative journalism student Sean McGrath who had obtained some information via FOI that he needed to combine with other data to answer a question (sorry to be so cryptic).

He had spent 3 days cleaning up the data and manually adding postcodes to it. This seemed a good example where using an API might cut down your work considerably, and so in this post I explain how you make a start on the same problem in less than an hour using Excel, Google Refine and the Google Maps API.

Step 1: Get the data in the right format to work with an API

APIs can do all sorts of things, but one of the things they do which is particularly useful for journalists is answer questions. Continue reading

New York Times paywall: sense prevails over ideology (almost)

So, the plans for the New York Times paywall are out. I said when they were first mooted that they looked to be thinking along the right lines in allowing people to view content for free if they came via social media – but I feared that that innovation would be lost along the way.

It’s enormously encouraging to see that it hasn’t.

Why is it encouraging? For two main reasons: firstly, it recognises the importance of distribution in online publishing. If you erect an arbitrary paywall, many people will not bother to link to you because they don’t want to frustrate their friends. That not only hurts your social media traffic, it hurts your search engine ranking.

Variety magazine suffered from this so much recently, it seems, that they launched a blog outside of their paywall with an email begging other sites to link to it.

Secondly, it recognises that they need to balance quality with quantity. Online advertising has yet to settle into any sort of pattern, but metrics of engagement are rising in importance, and one of those metrics is how much traffic comes from recommendations, i.e. social media.

Another metric is, of course, how loyal a user is, how many articles they read, and how much you know about them. The subscription options will allow the NYT to gather that information too – without sacrificing the huge numbers that most advertisers will be looking for.

Curiously, the chairman of The New York Times Company is quoted as saying “A few years ago it was almost an article of faith that people would not pay for the content they accessed via the Web.”

But I don’t think they are paying just for the content. I think this system recognises that they are paying for convenience (you pay more to get the content across web, mobile and iPad than you do to get the same content on fewer platforms – and you could get all the content for free if you can bother to go through Bing), and reliability (not hitting a wall when you want to read the 21st article of the month).

In many ways it is no different to traditional subscriptions: it is the difference between paying for regular deliveries of the whole paper package, and picking up a newspaper that someone has left on the bus or the staff canteen, or borrowing one from a friend, for free.

In the past we accounted for those ‘freeloaders’ and ‘parasites’ – as we call them online – by adjusting our readership figures to reflect that every copy bought was read by 4 people. We didn’t lock down the newspapers and tell subscribers what they could do with them.

And so here we are, with the most mature, intelligent, and commercially sensible paywall model yet.

But we still have no idea if it will work…

UPDATE: Aside from the technical implementation I think Dave Winer has a point about the content proposition

What is investigative journalism (for)?

On Wednesday I attended a fascinating conference addressing the question of whether investigative journalism was “dead or alive”. As is now routine at these events the ‘Is ice cream strawberry?’ question reared its head as those assembled tried to establish just where they stood in this Brave New World – but this time it got me thinking…

It was a stunning line up of speakers – live, pre-recorded, and remote – and after being drawn into a Skype two-way with David Leigh by chair Kevin Marsh on whether “professional” journalists still had a role to play in all this, I started to literally sketch out – on paper – some of the key questions underlying investigative journalism’s own identity crisis.

Is investigative journalism defined by how its done, or what it does?

Investigative journalism was described in many ways throughout the afternoon: as “uncovering the hidden”; “expensive”; “difficult”; “requiring dedication”; “has impact”; “holding power to account”. These terms are important: I’ve blogged elsewhere about journalism’s professional ideology and how it compares to bloggers’, and investigative journalism has its own professional ideology within that. If we are going to ask “But is it investigative journalism?” then these will be particularly relevant.

For example, there was a focus on investigative journalism as process that particularly fascinated me: Donal Macintyre talked about the ‘undercover reporter’ as a “narrative device” to allow them to create a narrative around important but difficult-to-dramatise issues, rather than something inherent in investigative work itself. In other words, for his purposes the process of ‘going undercover’ had a storytelling function as much as – if not more than – an investigative one.

On the other hand, some members of the audience dismissed modern examples of investigative work because it did not fit into this mythology.

A comparison of the Wikileaks, MPs’ expenses and Watergate stories is useful to flesh this out: in looking at those three where is the cut-off point that makes this one ‘investigative’, and another not? More to the point, why do we care?

If Wikileaks hadn’t had a website, would it make those stories more ‘investigative’? Do the parts of Watergate based on public documents not count as ‘investigative’ because they were available to anyone with a library card?

It struck me that this idea of ‘uncovering the hidden’ was key – and not too dissimilar to the general journalistic idea of ‘reporting the new’.

Defining ‘hidden’

What is ‘new’? It can be what happened today – but it is not limited to that. It can also be what is happening tomorrow, or what happened 30 years ago. It can be something that someone has said about an ‘old story’ days later, or an emerging anger about something that was never seen as ‘newsworthy’ to begin with. The talent of the journalist is to be able to spot that ‘newness’, and communicate it.

So does journalism become investigative when that newness involves uncovering the hidden? And if so, what is ‘hidden’?

I would argue that it is anything that our audience couldn’t see before – it could be a victim’s story, a buried report, 250,000 cables accessible to 2.5 million people, or even information that is publicly available but has not been connected before.

Like the journalist’s eye for ‘the news’, ‘the hidden’ is subject to individual perceptions, and the talent of a particular journalist for finding something in it – or a way of seeing it – that is ‘newsworthy’.

I sketched out a thought experiment: what if all of the investigative journalist’s material was public: documents, sources (witnesses, experts, victims, actors in the story), and information?

The role of the investigative journalist would perhaps be as follows:

  • To make the ‘hidden’ (to their audience) ‘visible’;
  • To hold power to account;
  • To make connections;
  • To verify;
  • To test hypotheses.

This doesn’t sound very different to how we see their role now.

But in reality, all of the investigative journalist’s material will most likely not be online, so if we leave that thought experiment behind we can add other elements to acknowledge that, particularly in a digitised world:

  • Making the invisible visible (i.e. digitising offline material, from paper documents and witness accounts to the ‘invisible web‘)
  • Making the disconnected connected
  • Identifying gaps in information – and filling them

These are all in fact ‘making the hidden visible’ in another form. It is the final one which comes closest to the process-based model identified above. But does it matter whether they fill those gaps with material that is in the public domain or which only exists in a single witness’s diary?

(I may have missed elements here – if I have, please let me know)

Narrative and authority

The role of a journalist in creating a narrative came through strongly in the conference – and also comes through strongly here: hypotheses are about narratives; making connections is about making narratives.

The other role that comes through strongly is institutional: holding power to account involves (but does not require) being in a position of power to do so; verification involves (but does not require) the stamp of institutional ‘due process’.

My own experience with Help Me Investigate suggests that these two roles remain important bases for journalism as a profession: in crowdsourced journalism, ‘writing the story up’ did not particularly appeal to people (the story was in their minds already) – only journalists wanted to do that. And it took an established media outlet to get official reaction.

This is not to suggest that only journalists can “have impact” as was mentioned at the conference – there are plenty of examples of groundswells of opinion online instigating media coverage: Memogate is perhaps the best known example. But this does not mean we need journalists – it means that we need publishers and broadcasters. There is a difference.


Does deconstructing investigative journalism in the way outlined above make the craft any less special? I don’t believe so.

Does it make it less mysterious? Probably. But that’s no bad thing. I was heartened to hear the responses of two of the Coventry University journalism students in the room to a question from Kevin Marsh on how they saw investigative journalism: the first felt that institutional restrictions on time or money should not be an excuse for journalists failing to investigate important questions in their own time; the second felt that people no longer needed institutional validation to investigate something: they could publish on a blog and build an audience that way.

The mythology of our craft, however, has said that they have to get a job before doing investigative journalism. We have even – over the last 50 years – built an iconography to market it: the ‘undercover reporter’; ‘Deep Throat’. And in drawing a line between investigative journalism and journalism – and between journalism and everything else – we took a little bit of power away from our colleagues, and from our readers.

Giving some of that power back was one of the things that excited me most about Help Me Investigate – and research into its users suggests they have found it genuinely empowering. It’s not, of course, enough on its own: there remains a disconnect between citizens and journalists, and too often power is held to account for entertainment rather than the greater good.

Now, I’ve taught enough students to know the sort of initiative expressed by those two Coventry students is not shown by every aspiring journalist (which perhaps comes back to my differentiation between wanting to be a journalist and wanting to make journalism happen), but still: it demonstrated that they were not going to wait to be given the job title of ‘investigative journalist’ to get out there and do some investigating. That’s good: it also shows that they are doing so not for status but for the reasons for investigative journalism’s existence: to hold power to account, to make the hidden visible, and perhaps just for the pleasure of solving a problem and gaining knowledge.

If we can swallow our pride long enough to stop debating the membership requirements of who and what can be in ‘our club’ – whether that’s investigative journalism, watchdog journalism, or just ‘journalism’, we might just have time to help those students – and those who can’t afford to be students, or indeed journalists – do it better.

When is an online comment defamatory?

Rob Minto looks at two recent cases that leave the field of libel online as confusing as ever.

For several years, newspapers, bloggers and other online publishers have been waiting for a landmark case to clarify defamation online.

The unanswered questions have been along the lines of: who’s responsible – the author or publisher (or even ISP)? What jurisdiction will it fall in? What kind of audience is required (if at all?)

In the UK, in quick succession, there have been two cases which have, if anything, muddied the waters. Continue reading

Hyperlocal Voices: Darryl Chamberlain, 853 Blog

853 blog

Having worked for the BBC News Entertainment website for a decade, Darryl Chamberlain took voluntary redundancy and set up the widely successful 853 Blog. As part of the Hyperlocal Voices series he shares some of the secrets of his success.

1) Who where the people behind the blog, and what where their backgrounds?

853’s all mine. My background’s actually in showbiz news. I worked for the BBC News website’s entertainment desk for a decade in a variety of roles – mainly sub-editing and being the daily editor, but also reporting and feature writing.

I took voluntary redundancy and a career break in 2009 – standing in a council election in May 2010, and doing odd bits of freelance work. While standing in an election will probably leave me hopelessly biased in many eyes, it helped introduce me to local issues which simply weren’t being touched, and potential contacts of all political hues. After my glorious defeat, I realised I could do a bit more for my local area by sticking to what I was good at – finding things out and writing about them. Continue reading

Internet destroys Fred Goodwin’s super-injunction about alleged affair

I’ve written before about superinjunctions, the difficulties of bloggers learning about reporting restrictions (featured) and the problems the internet causes for super-injunctions.

This morning, however, has seen a deliberate attempt by some people to use the internet to reveal the alleged affair that the super-injunction about disgraced RBS boss Fred Goodwin supposedly covers. Continue reading

Hyperlocal Voices: Ian Wylie, Jesmond Local


Yessi Bello continues the Hyperlocal Voices series with an interview with JesmondLocal‘s Ian Wylie, who decided to dabble in local journalism after taking voluntary redundancy from a national newspaper. Still viewed as a “pro-bono”, ” good thing to do” Jesmond Local has now become an integral part of the Jesmond Community.

1)Who were the people behind the blog, and what where their backgrounds?

After 15 years working for The Guardian as a reporter, features writer and finally section editor, I took voluntary redundancy in 2009, and began thinking about what I would do with the next chapter of my career. I’d been involved mostly in national newspaper and magazine journalism, so local journalism was something I hadn’t dabbled in before.

The concept of “hyperlocal” fascinated me as an area for me to explore and an opportunity for me also to “give something back”. I discovered that Newcastle University lecturer David Baines had a research interest in the subject. We met to discuss and he suggested I offer some of his students the chance to launch a hyperlocal website, which we did almost exactly a year ago. Continue reading

A fantastic tool for presenting debates – come play

Debate tool Wrangl

Someone should ask Stef Lewandowski to look after children more often. Any children. Every time he does, he seems to invent something. His latest project – made while cradling his baby son – is Wrangl, a tool for “making sense of both sides of the argument”.

This is a problem that has been tackled before – most notably by Debategraph. But Wrangl – at least judging by the example on AV – is better: it looks beautiful, and works wonderfully.

That’s partly because it is aimed at for-and-against debates, rather than the broader issues which Debategraph focuses on.

With such a simple basis, it’s then possible to link each argument to a counter-argument – or add new arguments of your own.

The design of the site makes it easy to pick out each side’s arguments, and follow those through to counter arguments. These are often linked to evidence (for example, Channel 4’s Fact Check) or blog posts which expand on the argument in more detail.

It will be interesting to see how the site is used – and abused – and how it develops in response. At the moment it has a few design issues – such as the ‘Back’ button taking you out of the site, and responding to an argument involving a dropdown menu for arguments across the site – but those small tweaks aside this is a fantastic design solution to an ongoing journalism problem.

To test it out, I’ve started a Wrangl of my own – on NHS reforms. See if you can have a play.

*Full disclosure: Stef was a colleague with me on the crowdsourcing investigative journalism website Help Me Investigate, which he built.

Culture Clash: Journalism’s ideology vs blog culture

If you read the literature on journalism’s professional ideology – or just follow any argument about journalists-versus-the-rest-of-the-world – you’ll notice particular themes recurring.

Like any profession, journalism separates itself from other fields of work through articulating how it is different. Reading Mark Deuze’s book Media Work recently I was struck by how a similar, parallel, ideology is increasingly articulated by bloggers. And I wanted to sketch that out. Continue reading

5 predictions for journalism in 25 years

The following is cross-posted from XCity Magazine, the student magazine for City University, where I teach online journalism. They asked me to look ahead 25 years. I barely think you can look five years ahead at the moment, but I agreed anyway. This is, of course, not meant to be taken seriously…

If you’d asked someone in 1986 to predict what journalism would be like now, you would have ended up with Michael J Fox playing a techno-draped future hack. In a flying car. And lots of fluorescent pink.

We have a tendency to cast the future as an exaggerated present. We give too much power to technology, and not enough to people. Any prediction I can come up with for 25 years’ time will, of course, say more about 2011 than 2036.

But there’s nothing like a challenge… Continue reading