Earlier this year I was asked to write a chapter for a book on the future of investigative journalism – ‘Investigative Journalism: Dead Or Alive?‘. I’m reproducing it here. The chapter was originally published on my Facebook page. An open event around the book’s launch, with a panel discussion, is being held at the Frontline Club next month.

We may finally be moving past the troubled youth of the internet as a medium for investigative journalism. For more than a decade observers looked at this ungainly form stumbling its way around journalism, and said: “It will never be able to do this properly.”

They had short memories, of course. Television was an equally awkward child: the first news broadcast was simply a radio bulletin on a black screen, and for decades print journalists sneered at the idea that this fleeting, image-obsessed medium could ever do justice to investigative journalism. But it did. And it did it superbly, finding a new way to engage people with the dry, with the political, and the complex.

Now the internet is growing up too, finding its feet with the likes of Clare Sambrook, Talking Points Memo, PolitiFact and VoiceOfSanDiego all winning awards, while journalists such as Paul Lewis (the death of Ian Tomlinson), Stephen Grey (extraordinary rendition) and James Ball (Wikileaks) explore new ways to dig up stories online that hold power to account. As these pioneers unearth, tell and distribute their stories in new ways we are beginning to discover just what shape investigative journalism might take in this new medium.

Funding investigative journalism

There is a now-familiar refrain that rumbles across the newsroom as regularly as a train: that online publishing cannot support what is needed for proper journalism – the journalism we have to call “investigative”. The argument is simple. Done the way it has been done for the past 50 years in newspapers and broadcasters, investigative journalism requires a reporter’s time – and, therefore, money. Online publishing – or at least, online advertising – does not currently offer a publisher the same margins that they enjoyed in the past.

But investigative journalism does not have to be pursued – or funded – in one particular way. The newsroom investigative journalist was an endangered species well before the internet arrived, while over the last decade NGOs and activist organisations have taken on an increasing role in funding investigations.

Indeed, the argument that the commercial pain of news organisations leads to cuts in investigative journalism is contradicted by research undertaken by Dutch-Flemish investigative journalism organisation VVOJ. They found that there was no relationship between the financial health of a news organisation and the amount of investigative journalism that was undertaken there.

It is notable that some of the biggest investigative stories in decades have come during one of the worst commercial periods for the newspaper industry: and while the MPs’ expenses and Wikileaks stories may not prove anything about the health of investigative journalism as a whole, they do serve as canonical examples of how it is changing. Because the web specifically – and digital technology more generally – offer new business models around investigative journalism. Primarily these come down to two features: a lowering of costs, and a broadening of revenue streams.

One of the costs of investigative journalism, for example, is that of organisation. As the internet makes it significantly easier to collaborate and communicate with others, the need for a formal news organisation is much reduced. The way that the Wikileaks revelations were managed both with that organisation and between publications in different countries is just one very visible example. My own project Help Me Investigate, meanwhile, proved that it was possible to conduct investigations (such as that into a £2.2m overspend on a council website) with the help of self-organising groups of individuals.

Another cost is time – and here, again, the internet offers efficiencies: a visit to the library is replaced with a visit to the library website, or a database. The FoI Act and related online services make it easier to obtain official documents. Social networks and forums make it easier to find leads, sources and experts.

This is not to argue that investigative journalism can be replaced by an entirely online process, merely to point out that previously time-consuming elements of the process have now been considerably accelerated.

The funding opportunities presented by the web are particularly interesting. Print and broadcast journalism relied on three streams of funding: advertising, for most; cover sales for some; and the licence fee.

Online, those organisational capabilities and reduced costs have opened up other streams: donation-funded investigations, for example, may not be new for charities and NGOs, but even those middlemen are now not always needed. The US website Spot.us, for instance, has successfully facilitated the sponsorship of numerous investigations by users. Other crowdfunding platforms offer the same possibilities to non-journalistic organisations. It is also difficult to pick apart how many subscribers to a platform such as Malaysiakini, for example, are paying for content, and how many to support a cause – its founder notes how subscriptions rise and fall in direct relation to negative actions by the government.

Meanwhile the funding of the Bureau of Investigative Journalism, ProPublica and the Huffington Post Investigations Fund (coming from sources other than traditional advertising or cover sales) suggest that we may be seeing a partial separation of the investigative and watchdog roles of the media from those of entertainment, information and current affairs which previously subsidised it. It is not yet clear, of course, how sustainable the individual examples are – but the broader trend towards a wider diversity of funding streams and business models remains.

Part 2, Investigative Journalism As A Genre, is now live here.

Last week I published an inverted pyramid of data journalism which attempted to map processes from initial compilation of data through cleaning, contextualising, and combining that. The final stage – communication – needed a post of its own, so here it is.

UPDATE: Now in Spanish too.

Below is a diagram illustrating 6 different types of communication in data journalism. (I may have overlooked others, so please let me know if that’s the case.)

Modern data journalism has grown up alongside an enormous growth in visualisation, and this can sometimes lead us to overlook different ways of telling stories involving big numbers. The intention of the following is to act as a primer for ensuring all options are considered.

1. Visualisation

Visualisation is the quickest way to communicate the results of data journalism: free tools such as Google Docs allow it with a single click; more powerful tools like Many Eyes only require the user to paste their raw data and select from a range of visualisation options.

But ease does not equal effectiveness. The rise of chartjunk illustrates that visualisation is not immune to churnalism or spectacle without insight.

There is a rich history of print visualisation which remains relevant to the generation of online infographics: focusing on no more than 4 data points; avoiding 3D and ensuring the graphic is self-sufficient are just some.

Kaiser Fung’s trifecta is one useful reference point for ensuring a visualisation is effective, and this explanation of how a chart was transformed into something that could be used in a newspaper is also instructive (summarised by Kaiser Fung here).

In short: it’s not a simple process.

Visualisation has one major advantage which makes that effort worthwhile, however: it can make communication incredibly effective. And it can provide a method of distributing your content which cannot be matched by the other types of communication listed here.

But its major strength is also its main weakness: the instant nature of infographics also means that people often do not spend much time looking at it. It makes it very effective for distribution, but not for engagement, and so it is worth thinking strategically about 1) making sure the image contains a link back to its source; and 2) making sure that there is something more at the source when people arrive.

2. Narration

A traditional article can struggle to contain the sort of numbers that data journalism tends to turf up, but it still provides an accessible way for people to understand the story – if done well.

There are books providing useful guidance on how to write with numbers most clearly – and some guidance for web writing too (you should use numerals rather than words, as this helps people who are scanning the page).

As with visualisation, less is often more. But also, as in most narrative, you need to think about meaningfulness and your objectives in communicating these numbers.

Abstract amounts can be impressive, but meaningless and useless. What does it mean that £10m has been spent on something? Is that more or less than usual? More or less than something similar?

Try to bring down amounts to manageable quantities – the amount per person, or per day, for example.

Finally, use editing to focus in on the essentials: and make sure you link to the whole.

3. Social communication

Communication is a social act, and the success of infographics across social media is a testament to that. But it’s not just infographics that are social – data is too. The Guardian has demonstrated this particularly successfully with the cultivation of a healthy community around its Data Blog (which enjoys higher stickiness than the average Guardian article), and around its API.

Crowdsourcing initiatives aimed at gathering data can also provide a social dimension to the data. The Guardian are, again, pioneers here, with their MPs’ expenses project and Charles Arthur’s attempt to crowdsource predictions about the specifications of the iPad. But there are other examples, too – especially when it is difficult to obtain the data any other way.

The connectivity of the web presents new opportunities to present data journalism in a social way. ProPublica’s app that provides results based on your Facebook profile (schools attended; friends who have used the app) is one example of how data journalism can leverage social data, and, equally, how communicating the results of data journalism can be geared around social dynamics, using elements such as quizzes, sharing, competition, campaigning and collaboration. We are barely at the start of this aspect of online journalism.

4. Humanise

Broadcast news reports often use case studies to get around the problem of presenting numbers-based stories on television and radio. If waiting times have increased, speak to someone who had to wait a long time for an operation. In other words, humanise the numbers.

More recently the growth of computer-generated motion graphics has relaxed that pressure somewhat, as presenters can call on powerful animation to illustrate a story.

But once again, the point of making stories relevant to people comes through. As I wrote in One ambassador’s embarrassment is a tragedy, 15,000 civilian deaths is a statistic: when you move beyond scales we can deal with on a human level, you struggle to engage people in the issue you are covering – no matter how impressive the motion graphics (that post outlines some other considerations in humanising stories, such as ensuring that case studies are representative).

So after being buried in abstract data we need to remember that going out and recording an interview with a person whose life has been affected by that data can make a big difference to the power of our story.

5. Personalise

One of the biggest changes in journalism’s move online is that it opens up all sorts of possibilities around interactivity. When it comes to data journalism that means that the user can, potentially, control what information is presented to them based on various inputs.

There are some relatively well-established forms of this. For example, when a government presents its latest budget, news websites often invite the user to input their own details (for example, their earnings, or their family make up) to find out how the budget affects them. A recent variant of this are those interactives which invite the user to make their own decisions on how they might cut the deficit (the FT’s version took this further, adding in party strategies and policies).

Another common form is geographical personalisation: the user is invited to enter their postcode, zip code or other geographical information to find out how a particular issue is playing out in their home town.

A third is simply ‘your interests’, as demonstrated by Popvox’s approach to political engagement and the LA Times’ Newsmatch.

As more and more personal data is held by third party sites, the possibilities for personalisation expand. The ProPublica example given above, for example, demonstrates how Facebook profile information can be used to automatically personalise the experience of a story. And there are various apps that offer to present information based on location data provided via GPS.

This also indicates that there may be various ways in which personalisation and social strategies might be combined. Personalised stories can, in many ways, be used as an expression of our identity: this is where I live; this is how I am affected; this is what I’m interested in.

And when the COO of Facebook is predicting that all media will be personalised in 3-5 years, it’s clear that this is something the social networks are going to drive towards too.

6. Utilise

The most complex way of communicating the results of data journalism is to create some sort of tool based on the data. Calculators are popular choices, as are GPS-driven tools, but there is a lot of scope for more complex applications as more data becomes available both from the publisher and the user.

Again, there is overlap here with personalisation – but it is possible to provide utility without personalisation. And quite often, the complexity and consequent barrier to competitors presents commercial opportunities too.

At Reed Business Information, for example, their model is geared towards this sort of utility: attracting users at various points of the communication chain – online updates, printed magazines, mobile news – and steering them towards the point where they are closest to a purchasing decision. The idea is that the closer your information is to their action, the more valuable it is to the user.

Creating utility from data is currently relatively costly – but those costs are going down as a result of competition and standardisation. For example, as increasing numbers of news organisations adopt standard ways of storing story data (e.g. XML files), it is easier to create apps that pull data from datasets. Meanwhile, app creation becomes increasingly templated (in many ways you can see the process following a similar path to that of web design) and platform independent.

A medium up for grabs

What all of the above makes apparent – and I may have missed other methods of communicating data journalism (please let me know if you can think of any) – is that there are whole areas of online journalism that have yet to be properly explored, and certainly most have yet to establish clear conventions or ideas of best practice.

I’ve tried to scope out an overview of those conventions that are emerging, and the best practice that’s currently available, but it would be great if you could add more. What makes for good humanisation? Utility? What are great examples of personalisation or data journalism that involves a social dimension? Comments below please.

Meanwhile, here are both parts of the model shown together (click to magnify):

.

A few months ago I heard ProPublica’s Olga Pierce and Jeff Larson speak at the Digital Editors Network Data Meet, giving their advice on data journalism projects. I thought I might publish notes of five tips they had here for the record:

1. Three-quarters of the top 10 stories on the site were news apps

Online applications prove very popular with users – but they are more often a landing page for further exploration via stories.

2. When you publish your story, ask for data

Publication is not the end of the process. If you invite users to submit their own information, it can lead to follow-ups and useful contacts.

3. Have both quantitative and qualitative fields in your forms

In other words, ask for basic details such as location, age, etc. but also ask for ‘their story’ if they have one.

4. Aim for a maximum of 12 questions

That seems to be the limit that people will realistically respond to. Use radio buttons and dropdown menus to make it easier for people to complete. At the end, ask whether it is okay for the organisation to contact them to ensure you’re meeting data protection regulations.

5. Share data left over from your investigation

Just because you didn’t use it doesn’t mean someone else can’t find something interesting in it.

Here’s another collection of questions from a University of Montana student that I’m answering here as part of my FAQ section:

Q: What do you see is the future for investigative journalism? Do you still see it as having a home at newspapers?

I think the future of investigative journalism is already here – it’s just unevenly distributed,as William Gibson would say. Nonprofit organisations (such as Amnesty or Human Rights Watch) are an increasingly significant source of investigative journalism. Then there are the more general investigative journalism operations, funded by foundations and donations, such as ProPublica. Crowdfunding projects such as Spot.us are going to be increasingly important. And then there are crowdsourcing operations such as those done by Talking Points Memo and, of course, my own project Help Me Investigate.

Alongside that I expect a number of traditional publishers to make a decision whether to continue to process content to fill in the space between ads, or to rely on a networked approach for that sort of commodity content and invest in quality investigative journalism that makes them stand out (this appears to be the direction that the Huffington Post and Guardian are moving in).

Investigative journalism will very much still have a home in newspapers; how much of that originates in newspapers depends on how strong the journalistic culture is in the companies that fund them.

Q: Who are the main competition to newspaper investigative reporters now? What are your thoughts on groups such as ProPublica?

The accountants and shareholders are the main competition to investigative reporters! Investigative reporters themselves appears to be becoming less competitive and more collaborative – mainly because the subjects they are investigating are increasingly international in scope and require cross-border partnership (the Trafigura story is just one example).

Likewise, many are learning how to work with bloggers and other publishers to get to the bottom of the story, but I think this is the biggest journey that needs to be made: away from the idea of the perfect ‘exclusive’ and towards the idea of something that engages and involves readers as a force for positive change. Put another way, collaboration creates a market for your journalism, not competition.

As for ProPublica, I think it’s a great idea. I wouldn’t want it to be the only one, because there are weaknesses to a foundation-funded model just as there are to a commercially funded one.

Q: Do you think there is more pressure from editors with the advent of the 24-hour news culture? Are investigative journalists getting the time and resources necessary?

Yes there is more pressure on editors, not just from a 24-hour news culture but from reduced resources and commercial pressures. As a result, yes, investigative journalists are often under pressure too, but these are not new trends – investigative journalism has always been defined by its exceptional nature. If it wasn’t exceptional, we’d just call it ‘journalism’. In Phillip Knightley’s account of pursuing the Thalidomide story in the 1970s (one of the biggest stories of the past 50 years in the UK), he recounts how it took a few years before he was able to cover the story because he was busy with other stories. There wasn’t a queue of other investigative journalists ready to do that reporting instead.

So I don’t think we should pretend that investigative journalism wasn’t already suffering. And the economic effects of advertising migrating to the web are probably nowhere near as important as shareholder expectation of profit margins way in excess of anything outside publishing. Regional pubishing in the UK still makes a margin of around 11% compared to Tesco’s margins of 8%.

Meanwhile, the internet offers some very interesting opportunities to move away from commercial pressures; to establish independent editorial operations without the legacy costs of printing and distributing (often 60-90% of all costs); to organise investigations in a more efficient way; and to engage readers.

Karthika Muthukumaraswamy on how crowdsourcing experiments in journalism need to learn from their commercial counterparts – and how the end results could bring financial rewards for everyone.

The crowd has done a great deal for journalism: it has counted the number of SUVs on the streets of New York City, determined Bill Clinton’s financial impact on Hillary Clinton’s campaign, and offered valuable suggestions to transform an impoverished Ugandan village.

Ever since journalism jumped on the crowdsourcing bandwagon following innovative business models in T-shirt designing and problem solving, it has been baffled by the intensity of crowd response. Consequently, the media’s implementation of it has lacked the selection process that is essential to use crowdsourcing to its fullest potential.

There are only so many T-shirts that Threadless can make and sell; there are only so many solutions to Innocentive’s complex problems; and there are only so many photographs that iStockphoto consumers will purchase.

But when the News-Press in Southwest Florida turned to its citizens for help with investigating the rising costs of local public utilities, much of the voluminous response – amounting to 6,500 pieces of user-generated stories – was published in six weeks following the investigation.

The difference lies in the ultimate goal. A company that aims to create a product is merely looking for the best idea to create one, and one that is looking to solve a problem is looking for the best solution. Journalism, on the other hand, while seeking the best stories, is also hoping to mobilize the maximum number of civilians and fulfill the ideals of democracy.

Stimulating citizen participation is, and should, in fact be, an important goal of crowdsourced journalism.

However, when it comes at the price of quality, as any cursory glance at citizen journalism sites would reveal, it not only compromises the media’s role in society, but also belittles the effectiveness of civilian engagement.

Not surprisingly, there is now an increasing desire for more reliable information on the Web, as seen from the popularity of sites such as BigThink and Mahalo, which rely on expert and professional sources rather than random, large groups of people.

The aim of crowdsourcing is to effectively enhance the quality of journalism because of crowd contributions, not despite them. And that is why distilling the best ideas, and thereby their utilization, becomes important.

Selecting for the top contributions and contributors is not new to citizen journalism. Establishing a community of dynamic civilians that a news organization can tap into on a regular basis is an important objective for most crowdsourced journalism projects.

The citizen journalists who established their credibility through productive efforts in Off the Bus have been largely retained to help report on the parent news site, the Huffington Post.

The investigative journalism site, Propublica, hopes to build a similar community of citizen journalists through its recently announced pro-am project.

The News-Press’s Team Watchdog went one step further and implemented a rigorous screening process that involved resumes and interviews to select twenty citizen volunteers from the Fort Myers community.

While such organization is essential for the success of open-source projects, news entities should be careful so as not to replicate the top-down hierarchy that still prevails in conventional media. This could defy the tenets of decentralization and independence that are essential to James Surowiecki’s concept of crowd wisdom.

It also ends up reinforcing the digital, intellectual, and economic divide that crowdsourcing already perpetuates.

The pharmaceutical company, Innocentive, has used a less conventional approach to seek out experts. Its website posts open calls to solve complex chemical problems to its large global community. While many of the 160,000 registered members of Innocentive are from highly specialized fields with advanced degrees (over a third have doctorates), almost anyone can register and take a crack at a problem.

Little surprise, then, that the company has turned up some unlikely problem-solvers in the form of patent attorneys and college students. Hence, real-world degrees and professional experience may not be the defining parameters for expertise, a finding that is reinforced by research from Harvard University.

The open-source technical support site, Experts-exchange.com has the luxury of using a more democratic approach to source “experts” from the crowd. The best solutions to technical problems are voted on by users, and the higher a contributor’s rating, the higher his authority and credibility in the community.

While the idea of allowing communities to choose their own experts would be desirable to citizen journalism, this form of user rating does not appear to work in more subjective areas.

While quantifiable answers to technical support questions are easier to rate, crowd wisdom is less reliable in judging more creative fields such as art and journalism. We all know that sensationalism would sooner get a Digg story on the home page or make an Internet video go viral than high-quality journalism would.

Hence, it would probably be in the best interest of news organizations to make these determinations at the editorial level.

In addition to improving the quality of content, such a strategy would promote better submissions from users. Crowdsourcing ventures like iStockphoto and Innocentive have shown that providng rewards – in the form of fame or bounty – works. As Jeff Howe, who coined the very term that all the fuss is about, has learned, community standing and recognition might be the key motivators in crowdsourced operations.

If you are one among thousands of people and don’t get recognition for your particular effort, there is little motivation for you to come back and participate.

If the more deserving contributors are acknowledged, and given special access privileges (such as being able to post content without moderation, for instance), it would encourage them to contribute more, and urge other contributors to compete at a higher level.

The unique, creditable, and more attractive content that would result from such moderation will eventually lead to higher site traffic, increased number of unique visitors, and hence, more advertising revenue. This might legitimize charging for content, thus allowing greater profits for news organizations, and possibly payment of individual contributors.

With contributors specifically chosen for the merit of their submissions, news organizations could finally explore the possibility of compensating the crowd for the product it creates. The opportunity to make money has been shown to be the most popular reason to participate in crowdsourcing projects.

Three years ago, when crowdsourcing first made a splash in the world of business and journalism, its democratic, freewheeling ideal was intriguing in all its novelty. But now, critics – and contributors themselves – have begun to question the legitimacy of a concept that puts people to work for little or no monetary gain while holding complete ownership over the product. “Is crowdsourcing evil?” asks Howe in Wired this week, detailing a backlash that is brewing in the design community.

It may be argued that the weeding out of contributors goes against the grain of grassroots citizen journalism. However, it is important to remember that news organizations are also entities that offer a service to people, and it behooves them to perform this service well.

In the field of business and innovation, companies are implementing a division of labor – specialized tasks are sourced to “experts,” while more general assignments are sourced to crowds. It is tempting to speculate that such a practice would work well for journalism.

Seeking ideas for stories from general readers, as well as involving them in the debate and discussion would fulfill the core purposes of journalism. On the other hand, the knowledge and skills of more prolific contributors could be utilized for specialized reporting. This would ensure the dissemination of quality content while still utilizing crowd diversity.