Category Archives: data journalism

When data goes bad

Bad data on sex trafficking: flow chart

Image by Lauren York on the Data Journalism Blog

Data is so central to the decision-making that shapes our countries, jobs and even personal lives that an increasing amount of data journalism involves scrutinising the problems with the very data itself. Here’s an illustrative list of when bad data becomes the story – and the lessons they can teach data journalists:

Deaths in police custody unrecorded

This investigation by the Bureau of Investigative Journalism demonstrates an important question to ask about data: who decides what gets recorded?

In this case, the BIJ identified “a number of cases not included in the official tally of 16 ‘restraint-related’ deaths in the decade to 2009 … Some cases were not included because the person has not been officially arrested or detained.” Continue reading

Get started in data scraping – and earn £75 for the pleasure

OpenlyLocal are trying to scrape planning application data from across the country. They want volunteers to help write the scrapers using Scraperwiki – and are paying £75 for each one.

This is a great opportunity for journalists or journalism students looking for an excuse to write their first scraper: there are 3 sample scrapers to help you find your feet, with many more likely to appear as they are written. Hopefully, some guidance will appear too (if not, I may try to write some myself).

Add your names in the comments on Andrew’s blog post, and happy scraping!

 

Comparing apples and oranges in data journalism: a case study

A must-read for any data journalist, aspiring or otherwise, is Simon Rogers’ post on The Guardian Datablog where he compares public and private sector pay.

This is a classic apples-and-oranges situation where politicians and government bodies are comparing two things that, really, are very different. Is a private school teacher really comparable to someone teaching in an unpopular school? What is the private sector equivalent of a director of public health or a social worker?

But if these issues are being discussed, journalists must try to shed some light, and Simon Rogers does a great job in unpicking the comparisons. From pay and hours worked, to qualifications and age (big differences in both), and gender and pay inequality (more women in the public sector, more lower- and higher-paid workers in the private sector), Rogers crunches all the numbers: Continue reading

La Nación: data journalism from Argentina

Guest post by Duarte Romero

Since the start of the year the Argentinian newspaper ‘La Nación’ has been publishing ‘Nación Data’, a blog dedicated to data visualization, interactive projects and especially, all the news related with data journalism.

During this time they have been posting interviews with experts from the community, reporting popular events such as NICAR and sharing the most innovative pieces made by other newspapers.

The multimedia development manager of ‘La Nación’, Momi Peralta, pointed out that their main goal so far is to release as much data as they can. Continue reading

The straw man of data journalism’s “scientific” claim

Guardian cover March 10 2012: Half UK's young black men out of work

Over the weekend Fleet Street Blues has had a bee in its bonnet about the “pretence” of data journalism and Saturday’s Guardian front page: “Half UK’s young black men out of work“.

This, says FSB, is a lie that demonstrates the “pretence” that “‘crunching the numbers’ is somehow an an abstract, scientific, mathematical task”. Continue reading

From CMS to DMS

There’s a persuasive argument being made by Francis Irving and Rufus Pollock in a joint blog post about the growth of data management systems – the ‘DMS’ to content management systems’ ‘CMS’:

“Just as then we wrote HTML in text files by hand and uploaded it by FTP, now we analyse data on our laptops using Excel, and share it with friends by emailing CSV files.

“But it reaches the point where using the filesystem and Outlook as your DMS stretches to breaking point. You’ll need a proper one.

“Nobody really knows what a proper one will look like yet. We’re all working on it.”

Their post lists what a DMS needs to do and the companies already trying to solve the ‘DMS problem’ from different directions: a list which includes Google Docs (“coming from the web spreadsheet direction”), the data social network BuzzData, visualisation tool Tableau, data marketplaces, operating systems, Scraperwiki, and PANDA (“making a DMS for newsrooms”)

It’s a well-drawn picture from an angle which I haven’t seen before. Certainly, a number of news organisations are trying to reduce the friction of producing content for different platforms by ‘atomising’ it in data-driven production processes (where a piece of content might be assembled and presented differently depending on the platform it is accessed through, for example), and their internal systems can probably be added to the list above.

What do you think? Is this a problem that’s being addressed in your own organisation?

FAQ: The stream as an interface; starting out in data journalism

Here are the latest answers to some questions – this time relating to these predictions for 2012:

Q: What are the advantages of “stream” as an interface for news website homepages?

The main advantages are that it’s very sticky – users tend to leave streams on in the same way that they leave 24 hour news channels on, or keep checking back to Facebook and Twitter (which have helped popularise the ‘stream’ interface).

If you compare that to the traditional story layout format, where users scan across the page but then leave the site if there’s nothing obviously of interest, you can see the difference.

I think there’s room for both, but if you want to know what’s new since the last time you looked, the stream works very well. And it’s not difficult to combine that with subject or region pages that show the most important news of that day, for example.

I think it can work for every kind of news: the stream says ‘Here’s what’s new’ across all topics; the ‘layout’ says ‘Here’s what we think is important’ – in other words, it performs a more traditional ‘snapshot’ function akin to the daily newspaper layout.

2) What are the skills a reporter should have in order to be a top-notch, first-rate data journalist?

The basic skills are the same as any journalist: a nose for a story, and the ability to communicate that clearly. In data journalism terms that means being able to interrogate data quickly and then focus on the most important facts within it.

That will most likely involve being able to use spreadsheet formulae to work out, for example, the proportion of time or money being spent on something, or to combine different datasets to gain new insights or overcome obstacles put in your way by those publishing the data.

You also need to be able to avoid mistakes by cleaning data, for example (often the same person or organisation will be named differently, for example), and by understanding the context of the data (for example, population size, or methodology used to gather it).

Finally, as I say, you need to be able to communicate the results clearly, which often means pulling back from the data and not trying to use it all in your telling of the story (just as you wouldn’t use every quote you got from a source) but keeping it simple.

Video: Heather Brooke’s tips on investigating, and using the FOI and Data Protection Acts

The following 3 videos first appeared on the Help Me Investigate blog, Help Me Investigate: Health and Help Me Investigate: Welfare. I thought I’d collect them together here too. As always, these are published under a Creative Commons licence, so you are welcome to re-use, edit and combine with other video, with attribution (and a link!).

First, Heather Brooke’s tips for starting to investigate public bodies:

Her advice on investigating health, welfare and crime:

And on using the Data Protection Act:

Moving away from ‘the story’: 5 roles of an online investigations team

In almost a decade of teaching online journalism I repeatedly come up against the same two problems:

  • people who are so wedded to the idea of the self-contained ‘story’ that they struggle to create journalism outside of that (e.g. the journalism of linking, liveblogging, updating, explaining, or saying what they don’t know);
  • and people stuck in the habit of churning out easy-win articles rather than investing a longer-term effort in something of depth.

Until now I’ve addressed these problems largely through teaching and individual feedback. But for the next 3 months I’ll be trying a new way of organising students that hopes to address those two problems. As always, I thought I’d share it here to see what you think.

Roles in a team: moving from churnalism to depth

Here’s what I’m trying (for context: this is on an undergraduate module at Birmingham City University):

Students are allocated one of 5 roles within a group, investigating a particular public interest question. They investigate that for 6 weeks, at which point they are rotated to a different role and a new investigation (I’m weighing up whether to have some sort of job interview at that point).

The group format allows – I hope – for something interesting to happen: students are not under pressure to deliver ‘stories’, but instead blog about their investigation, as explained below. They are still learning newsgathering techniques, and production techniques, but the team structure makes these explicitly different to those that they would learn elsewhere.

The hope is that it will be much more difficult for them to just transfer print-style stories online, or to reach for he-said/she-said sources to fill the space between ads. With only one story to focus on, students should be forced to engage more, to do deeper and deeper into an issue, and to be more creative in how they communicate what they find out.

(It’s interesting to note that at least one news organisation is attempting something similar with a restructuring late last year)

Only one member of the team is primarily concerned with the story, and that is the editor:

The Editor (ED)

It is the editor’s role to identify what exactly the story is that the team is pursuing, and plan how the resources of the team should be best employed in pursuing that. It will help if they form the story as a hypothesis to be tested by the team gathering evidence – following Mark Lee Hunter’s story based inquiry method (PDF).

Qualities needed and developed by the editor include:

  • A nose for a story
  • Project management skills
  • Newswriting – the ability to communicate a story effectively
This post on Poynter is a good introduction to the personal skills needed for the role.

The Community Manager (CM)

The community manager’s focus is on the communities affected by the story being pursued. They should be engaging regularly with those communities – contributing to forums, having conversations with members on Twitter; following updates on Facebook; attending real world events; commenting on blogs or photo/video sharing sites, and so on.

They are the two-way channel between that community and the news team: feeding leads from the community to the editor, and taking a lead from the editor in finding contacts from the community (experts, case studies, witnesses).

Qualities needed and developed by the community manager include:

  • Interpersonal skills – the ability to listen to and communicate with different people
  • A nose for a story
  • Contacts in the community
  • Social network research skills – the ability to find sources and communities online

6 steps to get started in community management can be found in this follow-up post.

The Data Journalist (DJ)

While the community manager is focused on people, the data journalist is focused on documentation: datasets, reports, documents, regulations, and anything that frames the story being pursued.

It is their role to find that documentation – and to make sense of it. This is a key role because stories often come from signs being ignored (data) or regulations being ignored (documents).

Qualities needed and developed by the data journalist include:

  • Research skills – advanced online search and use of libraries
  • Analysis skills – such as using spreadsheets
  • Ability to decipher jargon – often by accessing experts (the CM can help)

Here’s a step by step on how to get started as a data journalist.

The Multimedia Journalist (MMJ)

The multimedia journalist is focused on the sights, sounds and people that bring a story to life. In an investigation, these will typically be the ‘victims’ and the ‘targets’.

They will film interviews with case studies; organise podcasts where various parties play the story out; collect galleries of images to illustrate the reality behind the words.

They will work closely with the CM as their roles can overlap, especially when accessing sources. The difference is that the CM is concerned with a larger quantity of interactions and information; the MM is concerned with quality: much fewer interactions and richer detail.

Qualities needed and developed by the MMJ include:

  • Ability to find sources: experts, witnesses, case studies
  • Technical skills: composition; filming or recording; editing
  • Planning: pre-interviewing, research, booking kit 

The Curation Journalist (CJ)

(This was called Network Aggregator in an earlier version of this post) The CJ is the person who keeps the site ticking over while the rest of the team is working on the bigger story.

They publish regular links to related stories around the country. They are also the person who provides the wider context of that story: what else is happening in that field or around that issue; are similar issues arising in other places around the country. Typical content includes backgrounders, explainers, and updates from around the world.

This is the least demanding of the roles, so they should also be available to support other members of the team when required, following up minor leads on related stories. They should not be ‘just linking’, but getting original stories too, particularly by ‘joining the dots’ on information coming in.

Qualities needed and developed by the CJ include:

  • Information management – following as many feeds, newsletters and other relevant soures of information
  • Wide range of contacts – speaking to the usual suspects regularly to get a feel for the pulse of the issue/sector
  • Ability to turn around copy quickly

There’s a post on 7 ways to follow a field as a network aggregator (or any other journalist) on Help Me Investigate.

And here’s a post on ‘How to be a network journalist‘.

Examples of network aggregation in action:

  • Blogs like Created In Birmingham regularly round up the latest links to events and other reports in their field. See also The Guardian’s PDA Newsbucket.
  • John Grayson’s post on G4S uses a topical issue as the angle into a detailed backgrounder on the company with copious links to charity reports, politicians’ statements, articles in the media, research projects, and more.
  • This post by Diary of a Benefit Scrounger is the most creative and powerful example I’ve yet seen. It combines dozens of links to stories of treatment of benefit claimants and protestors, and to detail on various welfare schemes, to compile a first-person ‘story’.

Publish regular pieces that come together in a larger story

If this works, I’m hoping students will produce different types of content on their way to that ‘big story’, as follows:

  • Linkblogging – simple posts that link to related articles elsewhere with a key quote (rather than wasting resources rewriting them)
  • Profiles of key community members
  • Backgrounders and explainers on key issues
  • Interviews with experts, case studies and witnesses, published individually first, then edited together later
  • Aggregation and curation – pulling together a gallery of images, for example; or key tweets on an issue; or key facts on a particular area (who, what, where, when, how); or rounding up an event or discussion
  • Datablogging – finding and publishing key datasets and documents and translating them/pulling out key points for a wider audience.
  • The story so far – taking users on a journey of what facts have been discovered, and what remains to be done.

You can read more on the expectations of each role in this document. And there’s a diagram indicating how group members might interact below:

Investigations team flowchart

Investigations team flowchart

What will make the difference is how disciplined the editor is in ensuring that their team keeps moving towards the ultimate aim, and that they can combine the different parts into a significant whole.

UPDATE: A commenter has asked about the end result. Here’s how it’s explained to students:

“At an identified point, the Editor will need to organise his or her team to bring those ingredients into that bigger story – and it may be told in different ways, for example:

  • A longform text narrative with links to the source material and embedded multimedia
  • An edited multimedia package with links to source material in the accompanying description
  • A map made with Google Maps, Fusion Tables or another tool, where pins include images or video, and links to each story”

If you’ve any suggestions or experiences on how this might work better, I’d very much welcome them.