Data is so central to the decision-making that shapes our countries, jobs and even personal lives that an increasing amount of data journalism involves scrutinising the problems with the very data itself. Here’s an illustrative list of when bad data becomes the story – and the lessons they can teach data journalists:

Deaths in police custody unrecorded

This investigation by the Bureau of Investigative Journalism demonstrates an important question to ask about data: who decides what gets recorded?

In this case, the BIJ identified “a number of cases not included in the official tally of 16 ‘restraint-related’ deaths in the decade to 2009 … Some cases were not included because the person has not been officially arrested or detained.”

As they explain:

“It turns out the IPCC has a very tight definition of ‘in custody’ –  defined only as when someone has been formally arrested or detained under the mental health act. This does not include people who have died after being in contact with the police.

“There are in fact two lists. The one which includes the widely quoted list of sixteen deaths in custody only records the cases where the person has been arrested or detained under the mental health act. So, an individual who comes into contact with the police – is never arrested or detained – but nonetheless dies after being restrained, is not included in the figures.

“… But even using the IPCC’s tightly drawn definition, the Bureau has identified cases that are still missing.”

Cross-checking the official statistics against wider reports was key technique. As was using the Freedom of Information Act to request the details behind them and the details of those “ who died in circumstances where restraint was used but was not necessarily a direct cause of death”.

Cooking the books on drug-related murders

Cross-checking statistics against reports was also used in this investigation by Diego Valle-Jones into Mexican drug deaths:

“The Acteal massacre committed by paramilitary units with government backing against 45 Tzotzil Indians is missing from the vital statistics database. According to the INEGI there were only 2 deaths during December 1997 in the municipality of Chenalho, where the massacre occurred. What a silly way to avoid recording homicides! Now it is just a question of which data is less corrupt.”

Diego also used the Benford’s Law technique to identify potentially fraudulent data, which was also used to highlight relationships between dodgy company data and real world events such as the dotcom bubble and deregulation.

Poor records mean no checks

Detective Inspector Philip Shakesheff exposed a “gap between [local authority] records and police data”, reported The Sunday Times in a story headlined ‘Care home loses child 130 times‘:

“The true scale of the problem was revealed after a check of records on police computers. For every child officially recorded by local authorities as missing in 2010, another seven were unaccounted for without their absence being noted.”

Why is it important?

“The number who go missing is one of the indicators on which Ofsted judges how well children’s homes are performing and the homes have a legal duty to keep accurate records.

“However, there is evidence some homes are failing to do so. In one case, Ofsted gave a good report to a private children’s home in Worcestershire when police records showed 1,630 missing person reports in five years. Police stationed an officer at the home and pressed Ofsted to look closer. The home was downgraded to inadequate and it later closed.

“The risks of being missing from care are demonstrated by Zoe Thomsett, 17, who was Westminster council’s responsibility. It sent her to a care home in Herefordshire, where she went missing several times, the final time for three days. She had earlier been found at an address in Hereford, but because no record was kept, nobody checked the address. She died there of a drugs overdose.

“The troubled life of Dane Edgar, 14, ended with a drugs overdose at a friend’s house after he repeatedly went missing from a children’s home in Northumberland. Another 14-year-old, James Jordan, was killed when he absconded from care and was the passenger in a stolen car.”

Interests not registered

When there are no formal checks on declarations of interest, how can we rely on it? In Chile, the Ciudadano Inteligente Fundaciondecided to check the Chilean MPs’ register of assets and interests by building a database:

“No-one was analysing this data, so it was incomplete,” explained Felipe Heusser, executive president of the Fundacion. “We used technology to build a database, using a wide range of open data and mapped all the MPs’ interests. From that, we found that nearly 40% of MPs were not disclosing their assets fully.”

The organisation has now launched a database that “enables members of the public to find potential conflicts of interest by analysing the data disclosed through the members’ register of assets.”

Data laundering

Tony Hirst’s post about how dodgy data was “laundered” by Facebook in a consultants report is a good illustration of the need to ‘follow the data’.

We have some dodgy evidence, about which we’re biased, so we give it to an “independent” consultant who re-reports it, albeit with caveats, that we can then report, minus the caveats. Lovely, clean evidence. Our lobbyists can then go to a lazy policy researcher and take this scrubbed evidence, referencing it as finding in the Deloitte report, so that it can make its way into a policy briefing.”

“Things just don’t add up”

In the video below Ellen Miller of the Sunlight Foundation takes the US government to task over the inconsistencies in its transparency agenda, and the flawed data published on its USAspending.gov – so flawed that they launched the Clearspending website to automate and highlight the discrepancy between two sources of the same data:

Key budget decisions made on useless data

Sometimes data might appear to tell an astonishing story, but this turns out to be a mistake – and that mistake itself leads you to something much more newsworthy, as Channel 4′s FactCheck found when it started trying to find out if councils had been cutting spending on Sure Start children’s centres:

“That ought to be fairly straightforward, as all councils by law have to fill in something called a Section 251 workbook detailing how much they are spending on various services for young people.

“… Brent Council in north London appeared to have slashed its funding by nearly 90 per cent, something that seemed strange, as we hadn’t heard an outcry from local parents.

“The council swiftly admitted making an accounting error – to the tune of a staggering £6m.”

And they weren’t the only ones. In fact, the Department for Education  admitted the numbers were “not very accurate”:

“So to recap, these spending figures don’t actually reflect the real amount of money spent; figures from different councils are not comparable with each other; spending in one year can’t be compared usefully with other years; and the government doesn’t propose to audit the figures or correct them when they’re wrong.”

This was particularly important because the S251 form “is the document the government uses to reallocate funding from council-run schools to its flagship academies.”:

“The Local Government Association (LGA) says less than £250m should be swiped from council budgets and given to academies, while the government wants to cut more than £1bn, prompting accusations that it is overfunding its favoured schools to the detriment of thousands of other children.

“Many councils’ complaints, made plain in responses to an ongoing government consultation, hinge on DfE’s use of S251, a document it has variously described as “unaudited”, “flawed” and”not fit for purpose”.

No data is still a story

Sticking with education, the TES reports on the outcome of an FOI request on the experience of Ofsted inspectors:

“[Stephen] Ball submitted a Freedom of Information request, asking how many HMIs had experience of being a secondary head, and how many of those had led an outstanding school. The answer? Ofsted “does not hold the details”.

““Secondary heads and academy principals need to be reassured that their work is judged by people who understand its complexity,” Mr Ball said. “Training as a good head of department or a primary school leader on the framework is no longer adequate. Secondary heads don’t fear judgement, but they expect to be judged by people who have experience as well as a theoretical training. After all, a working knowledge of the highway code doesn’t qualify you to become a driving examiner.”

“… Sir Michael Wilshaw, Ofsted’s new chief inspector, has already argued publicly that raw data are a key factor in assessing a school’s performance. By not providing the facts to back up its boasts about the expertise of its inspectors, many heads will remain sceptical of the watchdog’s claims.”

Men aren’t as tall as they say they are

To round off, here’s a quirky piece of data journalism by dating site OkCupid, which looked at the height of its members and found an interesting pattern:

“The male heights on OkCupid very nearly follow the expected normal distribution—except the whole thing is shifted to the right of where it should be.

“Almost universally guys like to add a couple inches. You can also see a more subtle vanity at work: starting at roughly 5′ 8″, the top of the dotted curve tilts even further rightward. This means that guys as they get closer to six feet round up a bit more than usual, stretching for that coveted psychological benchmark.”

Do you know of any other examples of bad data forming the basis of a story? Please post a comment – I’m collecting examples.

UPDATE (April 20 2012): A useful addition from Simon Rogers: Named and shamed: the worst government annual reports explains why government department spending reports fail to support the Government’s claimed desire for an “army of armchair auditors”, with a list of the worst offenders at the end.

Also:

• This post on the lack of data on deaths from legal highs, by some of my students at City University.
• Sex trafficking: a story of data gone wrong, which is the source of the opening image for this post (by Lauren York, another student of mine)
• Chicago police crash reports are full of errors.

PHP Freelancer

Earlier this year I was asked to write a chapter for a book on the future of investigative journalism – ‘Investigative Journalism: Dead Or Alive?‘. I’m reproducing it here. The chapter was originally published on my Facebook page. An open event around the book’s launch, with a panel discussion, is being held at the Frontline Club next month.

We may finally be moving past the troubled youth of the internet as a medium for investigative journalism. For more than a decade observers looked at this ungainly form stumbling its way around journalism, and said: “It will never be able to do this properly.”

They had short memories, of course. Television was an equally awkward child: the first news broadcast was simply a radio bulletin on a black screen, and for decades print journalists sneered at the idea that this fleeting, image-obsessed medium could ever do justice to investigative journalism. But it did. And it did it superbly, finding a new way to engage people with the dry, with the political, and the complex.

Now the internet is growing up too, finding its feet with the likes of Clare Sambrook, Talking Points Memo, PolitiFact and VoiceOfSanDiego all winning awards, while journalists such as Paul Lewis (the death of Ian Tomlinson), Stephen Grey (extraordinary rendition) and James Ball (Wikileaks) explore new ways to dig up stories online that hold power to account. As these pioneers unearth, tell and distribute their stories in new ways we are beginning to discover just what shape investigative journalism might take in this new medium.

Funding investigative journalism

There is a now-familiar refrain that rumbles across the newsroom as regularly as a train: that online publishing cannot support what is needed for proper journalism – the journalism we have to call “investigative”. The argument is simple. Done the way it has been done for the past 50 years in newspapers and broadcasters, investigative journalism requires a reporter’s time – and, therefore, money. Online publishing – or at least, online advertising – does not currently offer a publisher the same margins that they enjoyed in the past.

But investigative journalism does not have to be pursued – or funded – in one particular way. The newsroom investigative journalist was an endangered species well before the internet arrived, while over the last decade NGOs and activist organisations have taken on an increasing role in funding investigations.

Indeed, the argument that the commercial pain of news organisations leads to cuts in investigative journalism is contradicted by research undertaken by Dutch-Flemish investigative journalism organisation VVOJ. They found that there was no relationship between the financial health of a news organisation and the amount of investigative journalism that was undertaken there.

It is notable that some of the biggest investigative stories in decades have come during one of the worst commercial periods for the newspaper industry: and while the MPs’ expenses and Wikileaks stories may not prove anything about the health of investigative journalism as a whole, they do serve as canonical examples of how it is changing. Because the web specifically – and digital technology more generally – offer new business models around investigative journalism. Primarily these come down to two features: a lowering of costs, and a broadening of revenue streams.

One of the costs of investigative journalism, for example, is that of organisation. As the internet makes it significantly easier to collaborate and communicate with others, the need for a formal news organisation is much reduced. The way that the Wikileaks revelations were managed both with that organisation and between publications in different countries is just one very visible example. My own project Help Me Investigate, meanwhile, proved that it was possible to conduct investigations (such as that into a £2.2m overspend on a council website) with the help of self-organising groups of individuals.

Another cost is time – and here, again, the internet offers efficiencies: a visit to the library is replaced with a visit to the library website, or a database. The FoI Act and related online services make it easier to obtain official documents. Social networks and forums make it easier to find leads, sources and experts.

This is not to argue that investigative journalism can be replaced by an entirely online process, merely to point out that previously time-consuming elements of the process have now been considerably accelerated.

The funding opportunities presented by the web are particularly interesting. Print and broadcast journalism relied on three streams of funding: advertising, for most; cover sales for some; and the licence fee.

Online, those organisational capabilities and reduced costs have opened up other streams: donation-funded investigations, for example, may not be new for charities and NGOs, but even those middlemen are now not always needed. The US website Spot.us, for instance, has successfully facilitated the sponsorship of numerous investigations by users. Other crowdfunding platforms offer the same possibilities to non-journalistic organisations. It is also difficult to pick apart how many subscribers to a platform such as Malaysiakini, for example, are paying for content, and how many to support a cause – its founder notes how subscriptions rise and fall in direct relation to negative actions by the government.

Meanwhile the funding of the Bureau of Investigative Journalism, ProPublica and the Huffington Post Investigations Fund (coming from sources other than traditional advertising or cover sales) suggest that we may be seeing a partial separation of the investigative and watchdog roles of the media from those of entertainment, information and current affairs which previously subsidised it. It is not yet clear, of course, how sustainable the individual examples are – but the broader trend towards a wider diversity of funding streams and business models remains.

Part 2, Investigative Journalism As A Genre, is now live here.

PHP Freelancer

[youtube:http://www.youtube.com/watch?v=1YeYn--rnIY]

Last week Channel 4 and the Bureau of Investigative Journalism “raised questions” over the election campaign expenses of Conservative MP Zac Goldsmith, specifically the practice of claiming partial expenses on the grounds that ‘not all material was used’.

The response from Goldsmith and the Conservative Party seemed to argue that this was standard practice. “The examples raised could be seen in the returns of other candidates.” (see video above)

So I decided to obtain the expenses receipts for two of the most closely-fought campaigns in Birmingham, and put them online, with the invitation for others to take a look to see if that is indeed true.

And now the receipts for the election campaigns of Gisela Stuart (Lab) and Deirdre Allen (Con) for Edgbaston can be found at EdgbastonElectionExpenses.posterous.com

Here’s the plan:

• Take a look at the expenses – they’re all individually tagged so you can click on the tags on the right hand side to just look at Labour receipts or leaflets.
• Post a comment to say you’ve looked, and to note anything of interest, or any detail that is missing (such as a company name or claim type).
• Join the investigation to discuss and follow up on the findings

But that’s only part of the story. I want to help repeat this in other cities and towns. And it’s quite simple: start your own investigation into your own local election candidates. by following the instructions here.

Remember, finding nothing is still a finding, as it challenges Goldsmith’s story.

PHP Freelancer

More questions from a student that I’m publishing as part of the FAQ section:

1. If News Corp starts charging for news stories, do you think readers would pay or they would just go to different newspapers?

Both, but mostly the latter. Previous experiments with paywalls saw audiences drop between 60 and 97%. And you also have to figure in that a paywall will likely make content invisible to search engines (either directly or indirectly, because no one will link to them which will drop their ranking). Search engines are responsible for a significant proportion of visits (even the Wall Street Journal receives a quarter of its traffic from Google). Still, some people will always pay – the question is: how many?

2. A newspaper website which introduces paid content is very likely to see a decline in number of visitors. How would this affect advertisers and the amount they agree to pay to that website/newspaper?

Advertisers will pay more per user, firstly. Both because they will know more about that user through registration details (and therefore advertising will be more targeted), and also because they know that that user has paid to see content, making them both more engaged and likely to be more affluent.

Of course, there will be fewer of those users, so the challenge is compensating for the loss of quantity through the increase in quality.

3. In your opinion, how could the concept of ‘charging for content’ affect the quality of journalism?

The interesting thing about the recent announcement by the editor of The Times is that he said they wouldn’t charge per article because that would influence their commitment to expensive journalism such as covering Sri Lanka.

An optimist would hope that charging for content would mean that a news organisation would focus more on unique journalism that doesn’t replicate what is available elsewhere for free. Sadly, I don’t think we’ll see that happen, at least in the near future.

It’s worth pointing out that many web operations churn out content because the advertising rates are so low they need to get as many views as possible.

On the flip side, if your paywall is preventing you from attracting enough readers to fund decent journalism, then you save the same problem.

More generally, putting up a paywall means that your journalism is seen – and criticised – by fewer people, which I would argue does present a quality issue. The future of journalism is collaborative, so if you’re putting up barriers you’re not enabling that opportunity to tap into the enormous knowledge in your former audience.

4. Do you think other newspaper publishers would follow News Corp and start charging for content or there would always be “free” places for news?

If News Corp makes it viable, then yes, others will surely follow. Until then I think almost all will sit back and see what happens with News Corp. But there will always be free places for news for a range of reasons: firstly, publicly funded organisations like the BBC and those with a social remit such as The Guardian; secondly, those funded by voluntary or foundation income such as The Bureau of Investigative Journalism and organisations like Amnesty; and finally, passionate citizens and those who simply like to chat.

5. Do you think that ‘charging for content’ is a vital business model which would last for long time?

I think it’s a business model that can work in some circumstances, if managed intelligently. The FT, for example, seems to be making it work, mainly because that content is financially valuable (I’d argue it’s information they’re charging for rather than content) but also because they’ve not cut it off entirely.

But broadly I think it’s the most difficult model because people never paid for ‘content’; they paid for a package and a service that included content. They bought a newspaper, not ‘the news’.

As for its longer term viability, as the means of production and distribution become more widely available, and advertisers themselves become content producers, it’s going to be increasingly difficult, and we’ll see increasing pressure on government to legislate to shore up publishers’ monopolies because of that, I fear.

PHP Freelancer

These days any journalist job ad is news, but this one is particularly worth blogging about. The recently formed Investigations Fund has in turn launched the Bureau of Investigative Journalism with a £2million grant from the David and Elaine Potter Foundation, and they’re looking for a Managing Editor.

Here’s the PDF of the job ad. The closing date is actually August 17 and not the 7th as stated in the ad. Although the job ad doesn’t particularly reflect it, the Director of the Centre for Investigative Journalism Gavin Macfadyen expresses a desire for the Bureau to experiment with new media:

“We will experiment with all the techniques available to us from ‘crowdfunding’ to ‘crowdsourcing’ and provide content across the media spectrum. But there is no substitute for first rate reporters being given time and resources to deliver great stories, which hold the powerful to account. The Bureau will offer investigative journalists both proper funding and the support of senior and experienced editors and researchers to carry out important investigations that are in the public interest.”

Here’s what they’re looking for:

• A noteworthy career as an investigative reporter in either print or electronic media; preferably in both newspaper and television.
• Experience as an editor or in leading teams of reporters and researchers, in commissioning the work of others.
• Experience in media law and a track record of defending publication.
• Dedication to the highest standards of truth in journalism and integrity.
• The imagination to be highly innovative; and to help the Board in building, and defending a high profile and potentially controversial new venture.

Expressions of interest with an up-to-date CV, can be emailed in confidence to Olga@crsearchandselection.com

PHP Freelancer