In the final part of the research underpinning a new Help Me Investigate project I explore the qualities that successful crowdsourcing investigations shared. Previous parts are linked below:

• Part 1: Investigative journalism; conceptualising Help Me Investigate
• Part 2: Building the site
• Part 3: Reflections on the Proof of Concept phase
• Part 4: The London Weekly case study
• Part 5: What are the characteristics of a crowdsourced investigation?
• Part 6: What made the crowdsourcing successful?

Conclusions

Looking at the reasons that users of the site as a whole gave for not contributing to an investigation, the majority attributed this to ‘not having enough time’. Although at least one interviewee, in contrast, highlighted the simplicity and ease of contributing, it needs to be as easy and simple as possible for users to contribute (or appear to be) in order to lower the perception of effort and time needed.

Notably, the second biggest reason for not contributing was a ‘lack of personal connection with an investigation’, demonstrating the importance of the individual and social dimension of crowdsourcing. Likewise, a ‘personal interest in the issue’ was the single largest factor in someone contributing. A ‘Why should I contribute?’ feature on crowdsourcing projects may be worth considering.

Others mentioned the social dimension of crowdsourcing – the “sense of being involved in something together” – what Jenkins (2006, p244) would refer to as “consumption as a networked practice”, a motivation also identified by Yochai Benkler in his work on networks (2006). Looking at non-financial motivations behind people contributing their time to online projects, he refers to “socio-psychological reward”. He also identifies the importance of “hedonic personal gratification”. In other words, fun.

Although positive feedback formed part of the design of the site, no consideration was paid to negative feedback: users being made aware of when they were not succeeding. This element also appears to be absent from game mechanics in other crowdsourcing experiments such as The Guardian’s MPs’ expenses app.

While it is easy to talk about “Failure for free”, more could be done to identify and support failing investigations. A monthly update feature that would remind users of recent activity and – more importantly – the lack of activity might help here. The investigators in a group might be asked whether they wish to terminate the investigation in those cases, emphasising their responsibility for its progress and helping ‘clean up’ the investigations listed on the first page of the site.

However, there is also a danger in interfering too much in reducing failure. This is a natural instinct, and the establishment of a reasonable ‘success rate’ at the outset – based on the literature around crowdsourcing – helps to counter this. That was part of the design of Help Me Investigate: it was the 1-5% of questions that gained traction that would be the focus of the site. One analogy is a news conference where members throw out ideas – only a few are chosen for investment of time and energy, the rest ‘fail’.

It is the management of that tension between interfering to ensure everything succeeds (and so removing the incentive for users to be self-motivated) and not interfering at all (leaving users feeling unsupported and unmotivated) that is likely to be the key to a successful crowdsourcing project. More than a year into the project, this tension was still being negotiated.

In summing up the research into Help Me Investigate it is possible to identify five qualities which successful investigations shared: ‘Alpha users’ (highly active, who drove investigations forward); modularity (the ability to break down a large investigation into smaller discrete elements); public-ness (the ability for others to find out about an investigation); feedback (game mechanics and the pleasure of using the site); and diversity of users.

Relating these findings to other research into crowdsourcing more generally it is possible to make broader generalisations regarding how future projects might be best organised. Leadbeater (2008, p68), for example, identifies five key principles of successful collaborative projects, summed up as ‘Core’ (directly comparable to the need for alpha users identified in this research); ‘Contribute’ (large numbers, comparable to public-ness); ‘Connect’ (diversity); ‘Collaborate’ (self governance – relating indirectly to modularity); and ‘Create’ (creative pleasure – relating indirectly to feedback). Similar qualities are also identified by US investigative reporter and Knight fellow Wendy Norris in her experiments with crowdsourcing (Lavrusik, 2010).

The most notable connections here are the indirect ones. While the technology of Help Me Investigate allowed for modularity, for example, the community structure was rather flat. Leadbeater’s research (2008) and that of Lih (2009) into the development of Wikipedia and Tsui (2010, PDF) into Global Voices indicate that ‘modularity’ may be part of a wider need for ‘structure’. Conversely ‘feedback’ provides a specific, practical way for crowdsourcing projects to address users’ need for creative pleasure.

As Help Me Investigate reached its 18th month a number of changes were made to test these ideas: the code was released as open source, effectively crowdsourcing the technology itself, and a strategy was adopted to recruit niche community managers who could build expertise in particular fields, along with an advisory board that was similarly diverse. The Help Me Investigate design was replicated in a plugin which would allow anyone running a self-hosted WordPress blog to manage their own version of the site.

This separation of technology from community was a key learning outcome of the project. While the site had solved some of the technical challenges of crowdsourcing and identified the qualities of successful crowdsourced investigation, it was clear that the biggest challenge lay in connecting the increasingly networked communities that wanted to investigate public interest issues – and in a way that was both sustainable and scalable beyond the level of individual investigations.

References

1. Arthur, Charles. Forecasting is a notoriously imprecise science – ask any meteorologist, January 29 2010, The Guardian, http://www.guardian.co.uk/technology/2010/jan/29/apple-ipad-crowdsource accessed 14/3/2011
2. Beckett, Charlie (2008) SuperMedia, Oxford: Blackwell
3. Belam, Martin. Whatever Paul Waugh thinks, The Guardian’s MPs Expenses crowd-sourcing experiment was no “total failure”, Currybetdotnet, March 10 2010 http://www.currybet.net/cbet_blog/2010/03/whatever-paul-waugh-thinks-the.php accessed 14/3/2011
4. Belam, Martin. Abort? Retry? Fail? – Judging the success of the Guardian’s MP’s expenses app, Currybetdotnet, March 7 2011, http://www.currybet.net/cbet_blog/2011/03/guardian-mps-expenses-success.php accessed 14/3/2011
5. Belam, Martin. The Guardian’s Paul Lewis on crowd-sourcing investigative journalism with Twitter, Currybetdotnet, March 10 2011, http://www.currybet.net/cbet_blog/2011/03/paul-lewis-investigative-journalism-twitter.php accessed 14/3/2011
6. Benkler, Yochai (2006) The Wealth of Networks, New Haven: Yale University Press
7. Bonomolo, Alessandra. Repubblica.it’s experiment with “Investigative reporting on demand”, Online Journalism Blog, March 21 2011, http://onlinejournalismblog.com/2011/03/21/repubblica-its-experiment-with-investigative-reporting-on-demand/ accessed 23/3/2011
8. Bradshaw, Paul. Wiki Journalism: Are wikis the new blogs? Paper presented to The Future of Journalism conference, Cardiff University, September 2007, http://onlinejournalismblog.files.wordpress.com/2007/09/wiki_journalism.pdf
9. Bradshaw, Paul. The Guardian’s tool to crowdsource MPs’ expenses data: time to play, Online Journalism Blog, June 19 2009 http://onlinejournalismblog.com/2009/06/19/the-guardian-build-a-platform-to-crowdsource-mps-expenses-data/ accessed 14/3/2011
10. Brogan, C., & Smith, J. (2009). Trust Agents: Using the Web to Build Influence, Improve
11. Reputation, and Earn Trust (1 ed.), New Jersey: Wiley
12. Bruns, Axel (2005) Gatewatching, New York: Peter Lang
13. Bruns, Axel (2008) Blogs, Wikipedia, Second Life and Beyond, New York: Peter Lang
14. De Burgh, Hugo (2008) Investigative Journalism, London: Routledge
15. Dondlinger, Mary Jo. Educational Video Game Design: A Review of the Literature, Journal of Applied Educational Technology Volume 4, Number 1, Spring/Summer 2007, http://www.eduquery.com/jaet/JAET4-1_Dondlinger.pdf
16. Ellis, Justin. A perpetual motion machine for investigative reporting: CPI and PRI partner on state corruption project, Nieman Journalism Lab, March 8 2011 http://www.niemanlab.org/2011/03/a-perpetual-motion-machine-for-investigative-reporting-cpi-and-pri-partner-on-state-corruption-project/ accessed 21/3/2011
17. Graham, John. Feedback in Game Design, Wolfire Blog, April 21 2010 http://blog.wolfire.com/2010/04/Feedback-In-Game-Design accessed 14/3/2011
18. Grey, Stephen (2006) Ghost Plane, London: C Hurst & Co
19. Hickman, Jon. Help Me Investigate: the social practices of investigative journalism, Paper presented to the Media Production Analysis Working Group, IAMCR, Braga, 2010, http://theplan.co.uk/help-me-investigate-the-social-practices-of-i
20. Howe, Jeff. Gannett to Crowdsource News, Wired, November 3 2006, http://www.wired.com/software/webservices/news/2006/11/72067 accessed 14/3/2011
21. Jenkins, Henry (2006) Convergence Culture, New York: New York University Press
22. Lavrusik, Vadim. How Investigative Journalism Is Prospering in the Age of Social Media, Mashable, November 24 2010, http://mashable.com/2010/11/24/investigative-journalism-social-web/ accessed 14/3/2011
23. Leadbeater (2008) We-Think, London: Profile Books
24. Leigh, David. Help us solve the mystery of Blair’s money, The Guardian, December 1 2009, http://www.guardian.co.uk/politics/2009/dec/01/help-us-solve-blair-mystery accessed 14/3/2011
25. Lih, Andrew (2009) The Wikipedia Revolution, London: Aurum Press
26. Marshall, Sarah. Snow map developer creates ‘Cutsmap’ for Channel 4′s budget coverage, Journalism.co.uk, 22 March 2011, http://www.journalism.co.uk/news/snow-map-developer-creates-cutsmap-for-channel-4-s-budget-coverage/s2/a543335/ accessed 22/3/2011
27. Morozov, Evgeny (2011) The Net Delusion, London: Allen Lane
28. Nielsen, Jakob. Participation Inequality: Encouraging More Users to Contribute, Jakob Nielsen’s Alertbox, October 9, 2006, http://www.useit.com/alertbox/participation_inequality.html accessed 14/3/2011
29. Paterson and Domingo (2008) Making Online News: The Ethnography of New Media Production, New York: Peter Lang
30. Porter, Joshua (2008) Designing for the Social Web, Berkeley: New Riders
31. Raymond, Eric S. (1999) The Cathedral and the Bazaar, New York: O’Reilly
32. Scotney, Tom. Help Me Investigate: How working collaboratively can benefit journalists, Journalism.co.uk, August 14 2009, http://www.journalism.co.uk/news-features/help-me-investigate-how-working-collaboratively-can-benefit-journalists/s5/a535469/ accessed 21/3/2011
33. Shirky, Clay (2008) Here Comes Everybody, London: Allen Lane
34. Snyder, Chris. Spot.Us Launches Crowd-Funded Journalism Project, Wired, November 10, 2008, http://www.wired.com/epicenter/2008/11/spotus-launches/ accessed 21/3/2011
35. Surowiecki, James (2005) The Wisdom of Crowds, London: Abacus
36. Tapscott, Don & Williams, Anthony (2006) Wikinomics, London: Atlantic Books
37. Tsui, Lokman. A Journalism of Hospitality, unpublished thesis, Presented to the Faculties of the University of Pennsylvania, 2010 http://dl.dropbox.com/u/22048/Tsui-Dissertation-Deposit-Final.pdf accessed 14/3/2011
38. Weinberger, David (2002) Small Pieces, Loosely Joined, New York: Basic Books

In the penultimate part of the serialisation of research underpinning a new Help Me Investigate project I explore the qualities that successful crowdsourcing investigations shared. Previous parts are linked below:

• Part 1: Investigative journalism; conceptualising Help Me Investigate
• Part 2: Building the site
• Part 3: Reflections on the Proof of Concept phase
• Part 4: The London Weekly case study
• Part 5: What are the characteristics of a crowdsourced investigation?

What made the crowdsourcing successful?

Clearly, a distinction should be made between what made the investigation successful as a series of outcomes, and what made crowdsourcing successful as a method for investigative reporting. This section concerns itself with the latter.

What made the community gather, and continue to return? One hypothesis was that the nature of the investigation provided a natural cue to interested parties – The London Weekly was published on Fridays and Saturdays and there was a build up of expectation to see if a new issue would indeed appear.

The data, however, did not support this hypothesis. There was indeed a rhythm but it did not correlate to the date of publication. Wednesdays were the most popular day for people contributing to the investigation.

Upon further investigation a possible explanation was found: one of the investigation’s ‘alpha’ contributors – James Ball – had set himself a task to blog about the investigation every week. His blog posts appeared on a Wednesday.

That this turned out to be a significant factor in driving activity suggests one important lesson: talking publicly and regularly about the investigation’s progress is key to its activity and success.

This data was backed up from the interviews. One respondent mentioned the “weekly cue” explicitly. And Jon Hickman’s research also identified that investigation activity related to “events and interventions. Leadership, especially by staffers, and tasking appeared to be the main drivers of activity within the investigation.” (2010, p10)

He breaks down activity on the site into three ‘acts’, although their relationship to the success of the investigation is not explored further:

• ‘Brainstorm’ (an initial flurry of activity, much of which is focused on scoping the investigation and recruiting)
• ‘Consolidation’ (activity is driven by new information)
• ‘Long tail’ (intermittent caretaker activity, such as supportive comments or occasional updates)

Networked utility

Hickman describes the site as a “centralised sub-network that suits a specific activity” (2010, p12). Importantly, this sub-network forms part of a larger ‘network of networks’ which involves spaces such as users’ blogs, Twitter, Facebook, email and other platforms and channels.

“And yet Help Me Investigate still provided a useful space for them to work within; investigators and staffers feel that the website facilitates investigation in a way that their other social media tools could not:

““It adds the structure and the knowledge base; the challenges, integration with ‘what do they know’ ability to pose questions allows groups to structure an investigation logically and facilitates collaboration.” (Interview with investigator)” (Hickman, 2010, p12)

In the London Weekly investigation the site also helped keep track of a number of discussions taking place around the web. Having been born from a discussion on Twitter, further conversations on Twitter resulted in further people signing up, along with comments threads and other online discussion. This fit the way the site was designed culturally – to be part of a network rather than asking people to do everything on-site.

The presence of ‘alpha’ users like James and Judith was crucial in driving activity on the site – a pattern observed in other successful investigations. They picked up the threads contributed by others and not only wove them together into a coherent narrative that allowed others to enter more easily, but also set the new challenges that provided ways for people to contribute. The fact that they brought with them a strong social network presence is probably also a factor – but one that needs further research.

The site had been designed to emphasise the role of the user in driving investigations. The agenda is not owned by a central publisher, but by the person posing the question – and therefore the responsibility is theirs as well. This cultural hurdle – towards acknowledging personal power and responsibility – may be the biggest one that the site has to address, and the offer of “failure for free” (Shirky, 2008), allowing users to learn what works and what doesn’t, may support that.

The fact that crowdsourcing worked well for the investigation is worth noting, as it could be broken down into separate parts and paths – most of which could be completed online: “Where does this claim come from?” “Can you find out about this person?” “What can you discover about this company?”. One person, for example, used Google Streetview to establish that the registered address of the company was a postbox. Other investigations that are less easily broken down may be less suitable for crowdsourcing – or require more effort to ensure success.

Momentum and direction

A regular supply of updates provided the investigation with momentum. The accumulation of discoveries provided valuable feedback to users, who then returned for more. In his book on Wikipedia, Andrew Lih (2009 p82) notes a similar pattern – ‘stigmergy’ – that is observed in the natural world: “The situation in which the product of previous work, rather than direct communication [induces and directs] additional labour”. An investigation without these ‘small pieces, loosely joined’ (Weinberger, 2002) might not suit crowdsourcing so well.

Hickman’s interviews with participants in the Birmingham council website investigation found a feeling of the investigation being communally owned and led:

“Certain members were good at driving the investigation forward, helping decide on what to do next, but it did not feel like anyone was in charge as such.”

“I’d say HMI had pivital role in keeping us together and focused but it felt owned by everyone.” (Hickman 2010, p10)

One problem, however, was that the number of diverging paths led to a range of potential avenues of enquiry. In the end, although the core questions were answered (was the publication a hoax and what were the bases for their claims) the investigation raised many more questions. These remained largely unanswered once the majority of users felt that their questions had been answered. As in a traditional investigation, there came a point at which those involved had to make a judgement whether they wished to invest any more time in it.

Finally, the investigation benefited from a diverse group of contributors who contributed specialist knowledge or access. Some physically visited stations where the newspaper was claiming distribution to see how many copies were being handed out. Others used advanced search techniques to track down details on the people involved and the claims being made, or to make contact with people who had had previous experiences with those behind the newspaper. The visibility of the investigation online also led to more than one ‘whistleblower’ approach providing inside information, which was not published on the site but resulted in new challenges being set.

The final part of this series will outline some conclusions to be taken from the project, and where it plans to go next.

Continuing the serialisation of the research underpinning a new Help Me Investigate project, in this fourth part I describe how one particular investigation took shape. Previous parts are linked below:

• Part 1: Investigative journalism; conceptualising Help Me Investigate
• Part 2: Building the site
• Part 3: Reflections on the Proof of Concept phase

Case study: the London Weekly investigation

In early 2010 Andy Brightwell and I conducted some research into one particular successful investigation on the site. The objective was to identify what had made the investigation successful – and how (or if) those conditions might be replicated for other investigations both on the site and elsewhere online.

The investigation chosen for the case study was ‘What do you know about The London Weekly?’ – an investigation into a free newspaper that was, the owners claimed (part of the investigation was to establish if the claim was a hoax), about to launch in London.

The people behind The London Weekly had made a number of claims about planned circulation, staffing and investment which went unchallenged in specialist media. Journalists Martin Stabe, James Ball and Judith Townend, however, wanted to dig deeper. So, after an exchange on Twitter, Judith logged onto Help Me Investigate and started an investigation.

A month later members of the investigation (most of whom were non-journalists) had unearthed a wealth of detail about the people behind The London Weekly and the facts behind their claims. Some of the information was reported in MediaWeek and The Guardian podcast Media Talk; some formed the basis for posts on James Ball’s blog, Journalism.co.uk and the Online Journalism Blog. Some has, for legal reasons, remained unpublished.

Methodology

Andrew Brightwell conducted a number of semi-structured interviews with contributors to the investigation. The sample was randomly selected but representative of the mix of contributors, who were categorised as either ‘alpha’ contributors (over 6 contributions), ‘active’ (2-6 contributions) and ‘lurkers’ (whose only contribution was to join the investigation). These interviews formed the qualitative basis for the research.

Complementing this data was quantitative information about users of the site as a whole. This was taken from two user surveys – one conducted when the site was three months’ old and another at 12 months – and analysis of analytics taken from the investigation (such as numbers and types of actions, frequency, etc.)

In the next part I explore some of the characteristics of a crowdsourced investigation and how these relate to the wider literature around crowdsourcing in general.

Continuing the serialisation of the research underpinning a new Help Me Investigate project, in this third part I describe how the focus of the site was shaped by the interests of its users and staff, and how site functionality was changed to react to user needs. I also identify some areas where the site could have been further developed and improved. (Part 1 is available here; Part 2 is here)

Reflections on the proof of concept phase

By the end of the 12 week proof of concept phase the site had also completed a number of investigations that were not ‘headline-makers’ but fulfilled the objective of informing users: in particular ‘Why is a new bus company allowed on an existing route with same number, but higher prices?’; ‘What is the tracking process for petitions handed in to Birmingham City Council?’ and ‘The DVLA and misrepresented number plates’

The site had also unearthed some promising information that could provide the basis for more stories, such as Birmingham City Council receiving over £160,000 in payments for vehicle removals; and ‘Which councils in the UK (that use Civil Enforcement) make the most from parking tickets?’ (as a byproduct, this also unearthed how well different councils responded to Freedom of Information requests#)

A number of news organisations expressed an interest in working with the site, but practical contributions to the site took place largely at an individual rather than organisational level. Journalist Tom Scotney, who was involved in one of the investigations, commented: “Get it right and you’re becoming part of an investigative team that’s bigger, more diverse and more skilled than any newsroom could ever be” (Scotney, 2009, n.p.) – but it was becoming clear that most journalists were not culturally prepared – or had the time – to engage with the site unless there was a story ‘ready made’ for them to use. Once there were stories to be had, however, they contributed a valuable role in writing those stories up, obtaining official reactions, and spreading visibility.

After 12 weeks the site had around 275 users (whose backgrounds ranged from journalism and web development to locally active citizens) and 71 investigations, exceeding project targets. It is difficult to measure ‘success’ or ‘failure’ but at least eight investigations had resulted in coherent stories, representing a success rate of at least 11%: the target figure before launch had been 1-5%. That figure rose to around 21% if other promising investigations were included, and the sample included recently initiated investigations which were yet to get off the ground.

‘Success’ was an interesting metric which deserves further elaboration. In his reflection on The Guardian’s crowdsourcing experiment, for example, developer Martin Belam (2011a, n.p.) noted a tendency to evaluate success “not purely editorially, but with a technology mindset in terms of the ‘100% – Achievement unlocked!’ games mechanic.”. In other words, success might be measured in terms of degrees of ‘completion’ rather than results.

In contrast, the newspaper’s journalist Paul Lewis saw success in terms of something other than pure percentages: getting 27,000 people to look at expense claims was, he felt, a successful outcome, regardless of the percentage of claims that those represented. And BBC Special Reports Editor Bella Hurrell – who oversaw a similar but less ambitious crowdsourcing project on the same subject on the broadcaster’s website, felt that they had also succeeded in genuine ‘public service journalism’ in the process (personal interview).

A third measure of success is noted by Belam – that of implementation and iteration (being able to improve the service based on how it is used):

“It demonstrated that as a team our tech guys could, in the space of around a week, get an application deployed into the cloud but appear integrated into our site, using a technology stack that was not our regular infrastructure.

“Secondly, it showed that as a business we could bring people together from editorial, design, technology and QA to deliver a rapid turnaround project in a multi-disciplinary way, based on a topical news story.

“And thirdly, we learned from and improved upon it.“ (Belam, 2010, n.p.)

A percentage ‘success’ rate of Help Me Investigate, then, represents a similar, ‘game-oriented’ perspective on the site, and it is important to draw on other frameworks to measure its success.

For example, it was clear that the site did very well in producing raw material for ‘journalism’, but it was less successful in generating more general civic information such as how to find out who owned a piece of land. Returning to the ideas of Actor-Network Theory outlined above, the behaviour of two principal actors – and one investigation – had a particular influence on this, and how the site more generally developed over time. Site user Neil Houston was an early adopter of the site and one of its heaviest contributors. His interest in interrogating data helped shape the path of many of the site’s most active investigations, which in turn set the editorial ‘tone’ of the site. This attracted users with similar interests to Neil, but may have discouraged others who did not – further research would be needed to establish this.

Likewise, while Birmingham City Council staff contributed to the site in its earliest days, when the council became the subject of an investigation staff’s involvement was actively discouraged (personal interview with contributor). This left the site short of particular expertise in answering civic questions.

At least one user commented that the site was very ‘FOI [Freedom Of Information request]-heavy’ and risked excluding users interested in different types of investigations, or who saw Freedom of Information requests as too difficult for them. This could be traced directly to the appointment of Heather Brooke as the site’s support journalist. Heather is a leading Freedom of Information activist and user of FOI requests: this was an enormous strength in supporting relevant investigations but it should also be recognised how that served to set the editorial tone of the site.

This narrowing of tone was addressed by bringing in a second support journalist with a consumer background: Colin Meek. There was also a strategic shift in community management which involved actively involving users with other investigations. As more users came onto the site these broadened into consumer, property and legal areas.

However, a further ‘actor’ then came into play: the legal and insurance systems. Due to the end of proof of concept funding and the associated legal insurance the team had to close investigations unrelated to the public sector as they left the site most vulnerable legally.

A final example of Actor-Network Theory in action was a difference between the intentions of the site designers and its users. The founders wanted Help Me Investigate to be a place for consensus, not discussion, but it was quickly apparent users did not want to have to go elsewhere to have their discussions. Users needed to – and did – have conversations around the updates that they posted.

The initial challenge-and-result model (breaking investigations down into challenges with entry fields for the subsequent results, which were required to include a link to the source of their information) was therefore changed very early on to challenge-and-update: people could now update without a link, simply to make a point about a previous result, or to explain their efforts in failing to obtain a result.

One of the challenges least likely to be accepted by users was to ‘Write the story up’. It seemed that those who knew the investigation had no need to write it up: the story existed in their heads. Instead it was either site staff or professional journalists who would normally write up the results. Similarly, when an investigation was complete, it required site staff to update the investigation description to include a link to any write-up. There was no evidence of a desire from users to ‘be a journalist’. Indeed, the overriding objective appeared rather to ‘be a citizen’.

In contrast, a challenge to write ‘the story so far’ seemed more appealing in investigations that had gathered data but no resolution as yet. The site founders underestimated the need for narrative in designing a site that allowed users to join investigations while they were in progress.

As was to be expected with a ‘proof of concept’ site (one testing whether an idea could work), there were a number of areas of frustration in the limitations of the site – and identification of areas of opportunity. When looking to crowdfund small amounts for an investigation, for example, there were no third party tools available that would allow this without going through a nonprofit organisation. And when an investigation involved a large crowdsourcing operation the connection to activity conducted on other platforms needed to be stronger so users could more easily see what needed doing (e.g. a live feed of changes to a Google spreadsheet, or documents bookmarked using Delicious).

Finally investigations often evolved into new questions but had to stay with an old title or risk losing the team and resources that had been built up. The option to ‘export’ an investigation team and resources into a fresh question/investigation was one possible future solution.

‘Failure for free’ was part of the design of the site in order to allow investigations to succeed on the efforts of its members rather than as a result of any top-down editorial agenda – although naturally journalist users would concentrate their efforts on the most newsworthy investigations. In practice it was hard to ‘let failure happen’, especially when almost all investigations had some public interest value.

Although the failure itself was not an issue (and indeed the failure rate lower than expected), a ‘safety net’ was needed that would more proactively suggest ways investigators could make their investigation a success, including features such as investigation ‘mentors’ who could pass on their experience; ‘expiry dates’ on challenges with reminders; improved ability to find other investigators with relevant skills or experience; a ‘sandbox’ investigation for new users to find their feet; and developing a metric to identify successful and failing investigations.

Communication was central to successful investigations and two areas required more attention: staff time in pursuing communication with users; and technical infrastructure to automate and facilitate communication (such as alerts to new updates or the ability to mail all investigation members)

The much-feared legal issues threatened by the site did not particularly materialise. Out of over 70 investigations in the first 12 weeks, only four needed rephrasing to avoid being potentially libellous. Two involved minor tweaks; the other two were more significant, partly because of a related need for clarity in the question.

Individual updates within investigations, which were post-moderated, presented even less of a legal problem. Only two updates were referred for legal advice, and only one of those rephrased. One was flagged and removed because it was ‘flamey’ and did not contribute to the investigation.

There was a lack of involvement by users across investigations. Users tended to stick to their own investigation and the idea of ‘helping another so they help you’ did not take root. Further research is needed to see if there was a power law distribution at work here – often seen on the internet – of a few people being involved in lots of investigations, most being involved in one, and a steep upward curve between.

In the next part, published tomorrow, I look at one particular investigation in an attempt to identify the qualities that made it successful. 

If you want to get involved in the latest Help Me Investigate project, get in touch on paul@helpmeinvestigate.com

A research paper I’ve contributed to, with Jean-Christophe Pascal and Neil Thurman, on a regional publisher’s experiment with hyperlocal publishing, has now been published on City University’s website. You can download the full PDF from here.

Hold The Front Page (which is part-owned by Northcliffe, the subject of the research), reported on the research here, which includes a response from Northcliffe.

Earlier this year I and Andy Brightwell conducted some research into one of the successful investigations on my crowdsourcing platform Help Me Investigate. I wanted to know what had made the investigation successful – and how (or if) we might replicate those conditions for other investigations.

I presented the findings (presentation embedded above) at the Journalism’s Next Top Model conference in June. This post sums up those findings.

The investigation in question was ‘What do you know about The London Weekly?‘ – an investigation into a free newspaper that was (they claimed – part of the investigation was to establish if this was a hoax) about to launch in London.

The people behind the paper had made a number of claims about planned circulation, staffing and investment that most of the media reported uncritically. Martin Stabe, James Ball and Judith Townend, however, wanted to dig deeper. So, after an exchange on Twitter, Judith logged onto Help Me Investigate and started an investigation.

A month later members of the investigation had unearthed a wealth of detail about the people behind The London Weekly and the facts behind their claims. Some of the information was reported in MediaWeek and The Media Guardian podcast Media Talk; some formed the basis for posts on James Ball’s blog, Journalism.co.uk and the Online Journalism Blog. Some has, for legal reasons, remained unpublished.

A note on methodology

Andrew conducted a number of semi-structured interviews with contributors to the investigation. The sample was randomly selected but representative of the mix of contributors, who were categorised as either ‘alpha’ contributors (over 6 contributions), ‘active’ (2-6 contributions) and ‘lurkers’ (whose only contribution was to join the investigation). These interviews formed the qualitative basis for the research.

Complementing this data was quantitative information about users of the site as a whole. This was taken from two user surveys – one when the site was 3 months’ old and another at 12 months – and analysis of analytics taken from the investigation (such as numbers and types of actions, frequency, etc.)

What are the characteristics of a crowdsourced investigation?

One of the first things I wanted to analyse was whether the investigation data matched up to patterns observed elsewhere in crowdsourcing and online activity. An analysis of the number of actions by each user, for example, showed a clear ‘power law’ distribution, where a minority of users accounted for the majority of activity.

This power law, however, did not translate into a breakdown approaching the 90-9-1 ‘law of participation inequality‘ observed by Jakob Nielsen. Instead, the balance between those who made a couple of contributions (normally the 9% of the 90-9-1 split) and those who made none (the 90%) was roughly equal. This may have been because the design of the site meant it was not possible to ‘lurk’ without being a member of the site already, or being invited and signing up.

Adding in data on those looking at the investigation page who were not members may have shed further light on this.

What made the crowdsourcing successful?

Clearly, it is worth making a distinction between what made the investigation successful as a series of outcomes, and what made crowdsourcing successful as a method.

What made the community gather, and continue to return? One hypothesis was that the nature of the investigation provided a natural cue to interested parties – The London Weekly was published on Fridays and Saturdays and there was a build up of expectation to see if a new issue would indeed appear.

I was curious to see if the investigation had any ‘rhythm’. Would there be peaks of interest correlating to the expected publication?

The data threw up something else entirely. There was indeed a rhythm but it was Wednesdays that were the most popular day for people contributing to the investigation.

Why? Well, it turned out that one of the investigation’s ‘alpha’ contributors – James Ball – set himself a task to blog about the investigation every week. His blog posts appeared on a Wednesday.

That this turned out to be a significant factor in driving activity tells us one important lesson: talking publicly and regularly about the investigation’s progress is key.

This data was backed up from the interviews. One respondent mentioned the “weekly cue” explicitly.

More broadly, it seems that the site helped keep track of a number of discussions taking place around the web. Having been born from a discussion on Twitter, further conversations on Twitter resulted in further people signing up, along with comments threads and other online discussion. This fit the way the site was designed culturally – to be part of a network rather than asking people to do everything on-site.

But the planned technical connectivity of the site with the rest of the web (being able to pull related tweets or bookmarks, for example) had been dropped during development as we focused on core functionality. This was not a bad thing, I should emphasise, as it prevented us becoming distracted with ‘bells and whistles’ and allowed us to iterate in reaction to user activity rather than our own assumptions of what users would want. This research shows that user activity and informs future development accordingly.

The presence of ‘alpha’ users like James and Judith was crucial in driving activity on the site – a pattern observed in other successful investigations. They picked up the threads contributed by others and not only wove them together into a coherent narrative that allowed others to enter more easily, but also set the new challenges that provided ways for people to contribute. The fact that they brought with them a strong social network presence is probably also a factor – but one that needs further research.

The site has always been designed to emphasise the role of the user in driving investigations. The agenda is not owned by a central publisher, but by the person posing the question – and therefore the responsibility is theirs as well. In this sense it draws on Jenkins’ argument that “Consumers will be more powerful within convergence culture – but only if they recognise and use that power.” This cultural hurdle may be the biggest one that the site has to address.

Indeed, the site is also designed to offer “Failure for free”, allowing users to learn what works and what doesn’t, and begin to take on that responsibility where required.

The investigation also suited crowdsourcing well, as it could be broken down into separate parts and paths – most of which could be completed online: “Where does this claim come from?” “Can you find out about this person?” “What can you discover about this company?”. One person, for example, used Google Streetview to establish that the registered address of the company was a postbox.

Other investigations that are less easily broken down may be less suitable for crowdsourcing – or require more effort to ensure success.

A regular supply of updates provided the investigation with momentum. The accumulation of discoveries provided valuable feedback to users, who then returned for more. In his book on Wikipedia, Andrew Lih (2009 p82) notes a similar pattern – ‘stigmergy‘ – that is observed in the natural world: “The situation in which the product of previous work, rather than direct communication [induces and directs] additional labour”. An investigation without these ‘small pieces, loosely joined’ might not suit crowdsourcing so well.

One problem, however, was that those paths led to a range of potential avenues of enquiry. In the end, although the core questions were answered (was the publication a hoax and what were the bases for their claims) the investigation raised many more questions.

These remained largely unanswered once the majority of users felt that their questions had been answered. Like any investigation, there came a point at which those involved had to make a judgement whether they wished to invest any more time in it.

Finally, the investigation benefited from a diverse group of contributors who contributed specialist knowledge or access. Some physically visited stations where the newspaper was claiming distribution to see how many copies were being handed out. Others used advanced search techniques to track down details on the people involved and the claims being made, or to make contact with people who had had previous experiences with those behind the newspaper.

The visibility of the investigation online led to more than one ‘whistleblower’ approach providing inside information.

What can be done to make it better?

Looking at the reasons that users of the site as a whole gave for not contributing to an investigation, the majority attributed this to ‘not having enough time’. Although at least one interviewee, in contrast, highlighted the simplicity and ease of contributing, it needs to be as easy and simple as possible for users to contribute in order to lower the perception of effort and time needed.

Notably, the second biggest reason for not contributing was a ‘lack of personal connection with an investigation’, demonstrating the importance of the individual and social dimension of crowdsourcing. Likewise, a ‘personal interest in the issue’ was the single largest factor in someone contributing. A ‘Why should I contribute?’ feature on each investigation may be worth considering.

Others mentioned the social dimension of crowdsourcing – the “sense of being involved in something together” – what Jenkins (2006) would refer to as “consumption as a networked practice”.

This motivation is also identified by Yochai Benkler in his work on networks. Looking at non-financial reasons why people contribute their time to online projects, he refers to “socio-psychological reward”. He also identifies the importance of “hedonic personal gratification”. In other words, fun. (Interestingly, these match two of the three traditional reasons for consuming news: because it is socially valuable, and because it is entertaining. The third – because it is financially valuable – neatly matches the third reason for working).

While it is easy to talk about “Failure for free”, more could be done to identify and support failing investigations. We are currently developing a monthly update feature that would remind users of recent activity and – more importantly – the lack of activity. The investigators in a group might be asked whether they wish to terminate the investigation in those cases, emphasising their role in its progress and helping ‘clean up’ the investigations listed on the first page of the site.

That said, there is also a danger is interfering too much in reducing failure. This is a natural instinct, and I have to continually remind myself that I started the project with an expectation of 95-99% of investigations ‘failing’ through a lack of motivation on the part of the instigator. That was part of the design. It was the 1-5% of questions that gained traction that would be the focus of the site (this is how Meetup works, for example – most groups ‘fail’ but there is no way to predict which ones. As it happens, the ‘success’ rate of investigations has been much higher than expected). One analogy is a news conference where members throw out ideas – only a few are chosen for investment of time and energy, the rest ‘fail’.

In the end, it is the management of that tension between interfering to ensure everything succeeds – and so removing the incentive for users to be self-motivated – and not interfering at all – leaving users feeling unsupported and unmotivated – that is likely to be the key to a successful crowdsourcing project. More than a year into the project, this is still a skill that I am learning.

Global news provider Small World News Service and online research company OnePoll are looking to undertake a large study which will research how the public access and use news online.

Great post by Claire Wardle and Matthew Eltringham on some research they conducted into how social network users use news. Here are the highlights. Firstly, news as a social object:

“They all saw comment and discussion as a key component of enjoying news on Facebook. They shared and posted stories they were interested in, sure, but also so they could make a point or start a conversation. But the vast majority really only wanted to have that conversation within their own group of friends, partly because that was where they felt comfortable.”

And secondly, it’s all about the niche:

“They were ‘only interested in the news they were interested in’ – not what they thought they ought to be interested in, or what news organisations thought they should be interested in. Would they join a general news group that provided a wide diet of content? Unlikely. Would they join a specialised or thematic group offering education or entertainment or business news? Quite possibly. Would they join a programme page – like BBC Breakfast? Maybe.”

Finally, an argument against Facebook Pages:

“When we showed the participants BBC Facebook pages, they saw them as a discrete space on Facebook which they would have to choose to go and visit (seemingly confusing them with Facebook groups) and they all said ‘why would I do that when I could just go to the BBC website’.”

There’s more, including findings on how much users trust news on Facebook, at the post.

This post is cross-published from my new journalism/new media-blog. Previous posts in this series:

• Part 1: The revolution that never happened
• Part 2: The assets

In the third part of this series I will take a closer look at the research on hypertext in online journalism and to what degree this asset of new technology has been and is utilized in online journalism. The general assumption of researchers interested in hypertextual online journalism is that if hypertext is used innovatively it would provide a range of advantages over print journalism:

• No limitations of space
• The possibility to offer a variety of perspectives
• No finite deadline
• Direct access to sources
• Personalized paths of news perception and reading
• Contextualization of breaking news
• Simultaneous targeting of different groups of readers – those only interest in the headlines and those interested in the deeper layers of information and sources

This list is generated from several sources, out of which the most important are:

• Peter Dahlgren (1996) “Media logic in cyberspace: Repositioning journalism and its publics”, Javnost/The Public 3, no. 3, pp. 59-72 (pdf)
• Mark Deuze (1999) “Journalism and the Web: An Analysis of Skills and Standards in an Online Environment”, International Communication Gazette 61, no. 5, pp. 373-390.
• Martin Engebretsen (2000) “Hypernews and coherence”, Journal of Digital Information 1, no. 7
• Nicholas W. Jankowski and Martine van Selm (2000) “Traditional news media online: an examination of added values”, Communications 25, no. 1, pp. 85-102
• Kevin Kawamoto (ed.) (2003) Digital journalism: emerging media and the changing horizons of journalism. Lanham, Md.: Rowman & Littlefield

Content analysis studies

Empirical research on the presence and relevance of hypertext in online journalism tends to rely on the methodology of quantitative content analysis to statistically count the amount of links present in online news sites. The findings are generally (but with many variations) categorized according to the three different types of hypertext identified by Chris Shipley and Matt Fish in their 1996 book “How the World Wide Web Works” ; Target links (links within documents), relative links (links to other pages within a site), and external links (links from one site to another site).

Most of the content analysis studies of hyperlinks in online journalism are snapshots of a situation at specific moments in time. A few of them are larger, cross-country studies, like Kenny et al (2000), who investigated 100 online newspapers (62 from the United States and 38 from “other countries”) at the end of the millennium and found that 33 percent of them offered links within news stories (target links) and only 52 percent of them offered some kinds of hyperlinks.

Jankowski and van Selm (2000) investigated 13 online news sites in the United States, The Netherlands and Canada and found similar results. A few years later, van der Wurff and Lauf (eds) (2005) presented studies of 72 European online newspapers and found that hyperlinks was the least developed “internet feature” (page 37). In their research on the front-pages of 26 leading online newspapers in 17 countries worldwide in 2003, Dimitrova and Neznanski (2006) found that use of hyperlinks had become “an established feature of online news”, but that the majority of the links was relative links (within-site links, mostly to archived material). Only eight percent of the online newspapers provided external links “despite the theoretically limitless possibilities for external linking”.

Compared to these studies, Quandt (2008) found in an extensive study of 10 online news sites in the United States, France, Germany, United Kingdom and Russia that hyperlinks were used to a somewhat greater extent: 73 percent of the 1600 full-text articles he analyzed had relative links, 14.3 percent had target links and 24.7 percent had external links.

Other, more nation-specific studies conclude that hyperlinks/hypertext is not utilized to its potential in online journalism, especially concerning the use of target links and external links (In Scandinavia: Engebretsen 2006; In Slovenia: Oblak 2005; In Ireland: O’Sullivan 2005; In Flandern: Paulussen 2004; In the United States: Pitts 2003; In Spain: Salaverria 2005).

A common explanation in these studies for the perceived lack of hypertext in the online news sites investigated is that a majority of the stories published online is shovel ware (stories that are originally published in print). Only a few studies offer more theoretically informed explanations of the findings, and even fewer offer a longitudinal approach.

One study that does both is Tremayne’s analysis of front-pages of ten online newspapers in the United States over a period of six years (1999-2004). He found that the amount of external links decreased during these years, while relative links increased. He explained this by network theory:

“[a]s each organization builds up its own archive of Web content, this material is being favored over content that is off-site. This is just one example of preferred attachment, which is the driving principle of network theory” (page 60).

Preferred attachment may be the result of a protectionist strategy aiming at keeping readers on-site, even though it is not portrayed as such in network theory. Such a strategy conflicts with the utilization of hypertext technology.

Surveys

While content analysis has been the preferred method to investigate hyperlinks/hypertext in online journalism, other methods have also been utilized. In their2002  report Online News Media and Their Audience (which is not to be found online) Quinn and Trench presents a survey of 138 “media professionals” engaged in online news production in Denmark, France, Ireland and United Kingdom. The respondents agreed that providing hyperlinks could make stories more valuable to the readers. However, they were skeptical as to whether the readers “should be left to make their own judgment about the relevance of links, rather than […] having the news services provide guidance to users” (page  35).

O’Sullivan (2005) interviewed Irish online journalists and found that few of them found hyperlinks to be an important feature of online journalism. On the contrary, they expressed concerns as to whether (external) hyperlinks would lead readers away from their site. In his 2009 PhD thesis The Online News Factory: A Multi-Lens Investigation of the Strategy, Structure, and Process of Online News Production at CNN and NRK (not available online) Krumsvik found that hypertext was to a little extent utilized – external links were “ignored” (page  145).

User studies

In an experimental study of how readers in the United States evaluate in-text (target) links in news stories Eveland et al. (2004) found that only the experienced web users found such hypertext structured news stories valuable. For in-experienced users, the hypertext structure was a disadvantage. Sundar (2009) found similar result in his experimental study. However, users seem to be satisfied with relative links. According to a survey amongst readers of Flemish online newspapers, the utilization of links to archived material (relative links) is regarded as an important reason to read online newspapers.

Based on these studies, it seems that relative hyperlinks, i.e. hyperlinks to other stories within the online news site, is the most common form of hypertext structure found in online journalism, while target links (links within stories) and external links are used to a lesser degree. A protectionist attitude might prevent utilization of external links; while utilization of target links may be obstructed by a high degree of shovel ware material and uncertainty as to whether users actually benefit from such links.

In the next post I’ll take a closer look at what the research on interactivity in online journalism might tell us. Until then – please feel free to comment any thoughts/disagreements.