The following post was originally published by Gary Herman on the NUJ New Media blog. It’s reproduced here with permission.

Here at Newmedia Towers we are being swamped by events which at long last are demonstrating that the internet is really rather relevant to the whole debate about media ethics and privacy. So this is by way of a short and somewhat belated survey of the news tsunami – Google, Leveson, Twitter, ACTA, the EU and more.

When Camilla Wright, founder of celebrity gossip site Popbitch (which some years ago broke the news of Victoria Beckham’s pregnancy possibly before she even knew about it), testified before Leveson last week (26 January 2012) [Guardian liveblog; Wright's official written statement (PDF)] the world found out (if it could be bothered) how Popbitch is used by newspaper hacks to plant stories so that they can then be said to have appeared on the internet. Anyone remember the Drudge report, over a decade ago?

Wright, of course, made a somewhat lame excuse that Popbitch is a counterweight to gossip magazines which are full of stories placed by the PR industry.

But most interesting is the fact that Wright claimed that Popbitch is self-regulated and that it works.

Leveson pronounced that he is not sure there is ‘so much of a difference’ between what Popbitch does and what newspapers do – which is somehow off the point. Popbitch – like other websites – has a global reach by definition and Wright told the Inquiry that Popbitch tries to comply with local laws wherever it was available – claims also made more publicly by Google and Yahoo! when they have in the past given in to Chinese pressure to release data that actually or potentially incriminated users and, more recently, by Twitter when it announced its intention to regulate tweets on a country-by-country basis.

Trivia – like the stuff Popbitch trades – aside, the problem is real. A global medium will cross many jurisdictions and be accessible within many different cultures. What one country welcomes, another may ban. And who should judge the merits of each?

Confusing the internet with its applications

The Arab Spring showed us that social media – like mobile phones, CB radios, fly-posted silkscreen prints, cheap offset litho leaflets and political ballads before them – have the power to mobilise and focus dissent. Twitter’s announcement should have been expected – after all, tweeting was never intended to be part of the revolutionaries’ tool-kit.

There are already alternatives to Twitter – Vibe, Futubra, Plurk, Easy Chirp and Blackberry Messenger, of course – and the technology itself will not be restrained by the need to expand into new markets. People confuse the internet with its applications – a mistake often made by those authorities who seek to impose a duty to police content on those who convey it.

Missing the point again, Leveson asked whether it would be useful to have an external ombudsman to advise Popbitch on stories and observed that a common set of standards across newspapers and websites might also help.

While not dismissing the idea, Wright made the point that the internet made it easy for publications to bypass UK regulators.

This takes us right into the territory of Google, Facebook and the various attempts by US and international authorities to introduce regulation and impose duties on websites themselves to police them.

ACTA, SOPA and PIPA

The latest example is the Anti-Counterfeit Trade Agreement (ACTA) – a shadowy international treaty which, according to Google’s legal director, Daphne Keller, speaking over a year ago, has ‘metastasized’ from a proposal on border security and counterfeit goods to an international legal framework covering copyright and the internet.

According to a draft of ACTA, released for public scrutiny after pressure from the European Union, internet providers who disable access to pirated material and adopt a policy to counter unauthorized ‘transmission of materials protected by copyright’ will be protected against legal action.

Fair use rights would not be guaranteed under the terms of the agreement.

Many civil liberty groups have protested the process by which ACTA has been drafted as anti-democratic and ACTA’s provisions as draconian.

Google’s Keller described ACTA as looking ‘a lot like cultural imperialism’.

Google later became active in the successful fight against the US Stop Online Piracy Act (SOPA) and the related Protect Intellectual Proerty Act (PIPA), which contained similar provisions to ACTA.

Google has been remarkably quite on the Megaupload case, however. This saw the US take extraterritorial action against a Hong Kong-based company operating a number of websites accused of copyright infringement.

The arrest of all Megaupload’s executives and the closure of its sites may have the effect of erasing perfectly legitimate and legal data held on the company’s servers – something which would on the face of it be an infringement of the rights of Megaupload users who own the data.

Privacy

Meanwhile, Google – in its growing battle with Facebook – has announced its intention to introduce a single privacy regime for 60 or so of its websites and services which will allow the company to aggregate all the data on individual users the better to serve ads.

Facebook already does something similar, although the scope of its services is much, much narrower than Google’s.

Privacy is at the heart of the current action against Google by Max Mosley, who wants the company to take down all links to external websites from its search results if those sites cover the events at the heart of his successful libel suit against News International.

Mosley is suing Google in the UK, France and Germany, and Daphne Keller popped up at the Leveson Inquiry, together with David-John Collins, head of corporate communications and public affairs for Google UK, to answer questions about the company’s policies on regulation and privacy.

Once again, the argument regarding different jurisdictions and the difficulty of implementing a global policy was raised by Keller and Collins.

Asked about an on-the-record comment by former Google chief executive, Eric Schmidt, that ‘only miscreants worry about net privacy’, Collins responded that the comment was not representative of Google’s policy on privacy, which it takes ‘extremely seriously’.

There is, of course, an interesting disjuncture between Google’s theoretical view of privacy and its treatment of its users. When it comes to examples like Max Mosley, Google pointed out – quite properly – that it can’t police the internet, that it does operate across jurisdictions and that it does ensure that there are comprehensive if somewhat esoteric mechanisms for removing private data and links from the Google listings and caches.

Yet it argues that, if individuals choose to use Google, whatever data they volunteer to the company is fair game for Google – even where that data involves third persons who may not have assented to their details being known or when, as happened during the process of building Google’s StreetView application, the company collected private data from domestic wi-fi routers without the consent or knowledge of the householders.

Keller and Collins brought their double-act to the UK parliament a few days later when they appeared before the joint committee on privacy and injunctions, chaired by John Whittingdale MP.

When asked why Google did not simply ‘find and destroy’ all instances of the images and video that Max Mosley objected to, they repeated their common mantras – Google is not the internet, and neither can nor should control the websites its search results list.

Accused by committee member Lord MacWhinney of ‘ducking and diving’ and of former culture minister, Ben Bradshaw of being ‘totally unconvincing’, Keller noted that Google could in theory police the sites it indexed, but that ‘doing so is a bad idea’.

No apparatus disinterested and qualified enough

That seems indisputable – regulating the internet should not be the job of providers like Google, Facebook or Twitter. On the contrary, the providers are the ones to be regulated, and this should be the job of legislatures equipped (unlike the Whittingdale committee) with the appropriate level of understanding and coordinated at a global level.

The internet requires global oversight – but we have no apparatus that is disinterested and qualified enough to do the job.

A new front has been opened in this battle by the latest draft rules on data protection issued by Viviane Reding’s Justice Directorate at the European Commission on 25 January.

Reding is no friend of Google or the big social networks and is keen to draw them into a framework of legislation that will – should the rules pass into national legislation – be coordinated at EU level.

Reding’s big ideas include a ‘right to be forgotten’ which will apply to online data only and an extension of the scope of personal data to cover a user’s IP address. Confidentiality should be built-in to online systems according to the new rules – an idea called ‘privacy by design’.

These ideas are already drawing flak from corporates like Google who point out that the ‘right to be forgotten’ is something that the company already upholds as far as the data it holds is concerned.

Reding’s draft rules includes an obligation by so-called ‘data controllers’ such as Google to notify third parties when someone wishes their data to be removed, so that links and copies can also be removed.

Not surprisingly, Google objects to this requirement which, if not exactly a demand to police the internet, is at least a demand to ‘help the police with their enquiries’.

The problem will not go away: how do you make sure that a global medium protects privacy, removes defamation and respects copyright while preserving its potential to empower the oppressed and support freedom of speech everywhere?

Answers on a postcard, please.

In this guest post, Damian Radcliffe highlights some recent developments in the intersection between hyper-local SoLoMo (social, location, mobile). His more detailed slides looking at 20 developments across the sector during the last two months of 2011 are cross-posted at the bottom of this article.

Facebook’s recent purchase of location-based service Gowalla (Slide 19 below,) suggests that the social network still thinks there is a future for this type of “check in” service. Touted as “the next big thing” ever since Foursquare launched at SXSW in 2009, to date Location Based Services (LBS) haven’t quite lived up to the hype.

Certainly there’s plenty of data to suggest that the public don’t quite share the enthusiasm of many Silicon Valley investors. Yet.

Part of their challenge is that not only is awareness of services relatively low – just 30% of respondents in a survey of 37,000 people by Forrester (Slide 27) – but their benefits are also not necessarily clearly understood.

In 2011, a study by youth marketing agency Dubit found about half of UK teenagers are not aware of location-based social networking services such as Foursquare and Facebook Places, with 58% of those who had heard of them saying they “do not see the point” of sharing geographic information.

Safety concerns may not be the primary concern of Dubit’s respondents, but as the “Please Rob Me” website says: “….on one end we’re leaving lights on when we’re going on a holiday, and on the other we’re telling everybody on the internet we’re not home… The danger is publicly telling people where you are. This is because it leaves one place you’re definitely not… home.”

Reinforcing this concern are several stories from both the UK and the US of insurers refusing to pay out after a domestic burglary, where victims have announced via social networks that they were away on holiday – or having a beer downtown.

For LBS to go truly mass market – and Forrester (see Slide 27) found that only 5% of mobile users were monthly LBS users – smartphone growth will be a key part of the puzzle. Recent Ofcom data reported that:

• Ownership nearly doubled in the UK between February 2010 and August 2011 (from 24% to 46%).
• 46% of UK internet users also used their phones to go online in October 2011.

For now at least, most of our location based activity would seem to be based on previous online behaviours. So, search continues to dominate.

Google in a recent blog post described local search ads as “so hot right now” (Slide 22, Sept-Oct 2011 update). The search giant launched hyper-local search ads a year ago, along with a “News Near You” feature in May 2011. (See: April-May 2011 update, Slide 27.)

Meanwhile, BIA/Kelsey forecast that local search advertising revenues in the US will increase from $5.1 billion in 2010 to $8.2 billion in 2015. Their figures suggest by 2015, 30% of search will be local.

The other notable growth area, location based mobile advertising, also offers a different slant on the typical “check in” service which Gowalla et al tend to specialise in. Borrell forerecasts this space will increase 66% in the US during 2012 (Slide 22).

The most high profile example of this service in the UK is O2 More, which triggers advertising or deals when a user passes through certain locations – offering a clear financial incentive for sharing your location.

Perhaps this – along with tailored news and information manifest in services such as News Near You, Postcode Gazette and India’s Taazza – is the way forward.

Jiepang, China’s leading Location-Based Social Mobile App, offered a recent example of how to do this. Late last year they partnered with Starbucks, offering users a virtual Starbucks badge if they “checked-in” at a Starbucks store in the Shanghai, Jiangsu and Zhejiang provinces. When the number of badges issued hit 20,000, all badge holders got a free festive upgrade to a larger cup size. When coupled with the ease of NFC technology deployed to allow users to “check in” then it’s easy to understand the consumer benefit of such a service.

Mine’s a venti gingerbread latte. No cream. Xièxiè.

The Scotsman has a newish data blog, set up (I’m rather proud to say) by one of my former PA/Telegraph trainees: Jennifer O’Mahony. This is particularly important as so much data covered in the ‘national’ press tends to be English-only due to devolution.

The Department of Education, for example, only publishes English education data. If you want Scottish education data you need to go to the Scottish Government website or Education Scotland. Ofsted inspects schools in England; for Scottish schools reports you need to visit HM Inspectorate of Education. (Meanwhile, the National Statistics site, publishes data from England, Scotland, Wales and Northern Ireland).

So if there’s any Scottish data – or that of Wales or Northern Ireland – that you want me to help with, let me or Jennifer know. By way of illustrating the process, here’s a post over on Help Me Investigate: Education on how I helped Jennifer collect data on free school meals in Scotland.

A treemap in Liverpool

On the same note of non-national data journalism, here’s a particularly nice bit of data visualisation at the Liverpool Post. It’s not often you see treemaps on a local newspaper website – this one was designed by Ilan Sheady based on data gathered by City Editor David Bartlett after a day’s data journalism training.

One of the easiest ways to get someone started on data visualisation is to introduce them to word clouds (it also demonstrates neatly how not all data is numerical).

Using tools like Wordle and Tagxedo, you can paste in a major speech and see it visualised within a minute or so.

But is a word cloud the best way of visualising speeches? The New York Times appear to think otherwise. Their visualisation (above) comparing President Obama’s State of the Union address and speeches by Republican presidential candidates chooses to use something far less fashionable: the bar chart.

Why did they choose a bar chart? The key is the purpose of the chart: comparison. If your objective is to capture the spirit of a speech, or its key themes, then a word cloud can still work well, if you clean the data (see this interactive example that appeared on the New York Times in 2009).

But if you want to compare it to speeches of others – and particularly if you want to compare on specific issues such as employment or tax – then bar charts are a better choice. Compare, for example, ReadWriteWeb’s comparison of inaugural speeches, and how effective that is compared to the bar charts.

In short, don’t always reach for the obvious chart type – and be clear what you’re trying to communicate.

UPDATE: More criticism of word clouds by New York Times software architect here (via Harriet Bailey)

Obama inaugural speech word cloud by ReadWriteWeb

via Flowing Data

In this guest post, Damian Radcliffe highlights some topline developments in the hyper-local space during 2011. He also asks for your suggestions of great hyper-local content from 2011. His more detailed slides looking at the previous year are cross-posted at the bottom of this article.

2011 was a busy year across the hyper-local sphere, with a flurry of activity online as well as more traditional platforms such as TV, Radio and newspapers.

The Government’s plans for Local TV have been considerably developed, following the Shott Review just over a year ago. We now have a clearer indication of the areas which will be first on the list for these new services and how Ofcom might award these licences. What we don’t know is who will apply for these licences, or what their business models will be. But, this should become clear in the second half of the year.

Whilst the Leveson Inquiry hasn’t directly been looking at local media, it has been a part of the debate. Claire Enders outlined some of the challenges facing the regional and local press in a presentation showing declining revenue, jobs and advertising over the past five years. Her research suggests that the impact of “the move to digital” has been greater at a local level than at the nationals.

Across the board, funding remains a challenge for many. But new models are emerging, with Daily Deals starting to form part of the revenue mix alongside money from foundations and franchising.

And on the content front, we saw Jeremy Hunt cite a number of hyper-local examples at the Oxford Media Convention, as well as record coverage for regional press and many hyper-local outlets as a result of the summer riots.

I’ve included more on all of these stories in my personal retrospective for the past year.

One area where I’d really welcome feedback is examples of hyper-local content you produced – or read – in 2011. I’m conscious that a lot of great material may not necessarily reach a wider audience, so do post your suggestions below and hopefully we can begin to redress that.

In this guest post Ofcom’s Damian Radcliffe cross-publishes his latest presentation on developments in hyperlocal publishing for September-October, and highlights how partnerships are increasingly important for hyper-local, regional and national media in terms of “making it pay”.

When producing my latest bi-monthly update on hyper-local media, I was struck by the fact that media sales partnerships suddenly seem to be all the rage.

In a challenging economic climate, a number of media providers – both big and small – have recently come together to announce initiatives aimed at maximising economies of scale and potentially reducing overheads.

At a hyperlocal level, the launch on 1^st November of the Chicago Independent Advertising Network (CIAN), saw 15 Chicago community news sites coming together to offer a single point of contact for advertisers. These sites “collectively serve more than 1 million page views each month.”

This initiative follows in the footsteps of other small scale advertising alliances including the Seattle Indie Ad Network and Boston Blogs.

These moves – bringing together a range of small scale location based websites – can help address concerns that hyper-local sites are not big enough (on their own) to unlock funding from large advertisers.

CIAN also aims to address a further hyper-local concern: that of sales skills. Rather than having a hyperlocal practitioner add media sales to an ever expanding list of duties, funding from the Chicago Community Trust and the Knight Community Information Challenge allows for a full-time salesperson.

Big Media is also getting in on this act.

In early November Microsoft, Yahoo! and AOL agreed to sell each other’s unsold display ads. The move is a response to Google and Facebook’s increasing clout in this space.

Reuters reported that both Facebook and Google are expected to increase their share of online display advertising in the United States in 2011 by 9.3% and 16.3%.

In contrast, AOL, Microsoft and Yahoo are forecast to lose share, with Facebook expected to surpass Yahoo for the first time.

Similarly in the UK, DMGT’s Northcliffe Media, home to 113 regional newspapers, recently announced it was forging a joint partnership with Trinity Mirror’s regional sales house, AMRA.

This will create a commercial proposition encompassing over 260 titles, including nine of the UK’s 10 biggest regional paid-for titles. Like The Microsoft, Yahoo! and AOL arrangement, this new partnership comes into effect in 2012.

These examples all offer opportunities for economies of scale for media outlets and potentially larger potential reach and impact for advertisers.  Given these benefits, I wouldn’t be surprised if we didn’t see more of these types of partnership in the coming months and years.

Damian Radcliffe is writing in a personal capacity.

Other topics in his current hyperlocal slides  include Sky’s local pilot in NE England and research into the links between tablet useand local news consumption. As ever, feedback and suggestions for future editions are welcome.

Since I wrote about the need for more data journalism around sentencing in August, the Manchester Evening News have been beavering away keeping track of riot sentencing data on their own patch with stories on the first 60 looters to be sentenced and the role of poverty. Last week the newspaper finally made a splash on the figures.

The collected data led to this front page story: Looters jailed straight after Manchester riots given terms 30 per cent longer than those punished later.

While another article builds up a detailed profile of the rioters with plenty of visualisation, and links to the raw data.

The MEN’s Paul Gallagher had previously told me in an email correspondence that they were expecting at least 250-300 cases to be going through the courts in total, making “enough to make a very interesting and useful dataset but not so many as to make it too big a job.

“This spreadsheet is being completed using information provided by our journalists in court. The MEN is committed to staffing every court hearing so we should be able to fill this over time. This is a trial project limited only to the riots, and I don’t know if we will do anything with other court data in future.”

At the time Paul was trying to set up a system that would see court reporters add information when they covered a case, a system that could be used to publish court data in future.

“One of the biggest problems I have found is that we can produce graphics quite easily for online using Google Fusion Tables and other tools but it is difficult to turn these into graphics that will work in print without getting a graphic designer to recreate the image.”

A couple months on Paul remarks that the project has required significant editorial resources:

“Around ten MEN journalists have either sat in court to take down details of one or more riot cases in the last three months, or have been involved in the data analysis.”

He also says the exercise has raised some questions about the use, and sharing, of court data.

“Although the names and home addresses of adult defendants are published in court reports in the media, it does not seem appropriate to include them in shared spreadsheets, or to plot them on street level maps.

“For that reason, I decided to remove the names and personal details when we plotted home addresses of defendants on a map of Greater Manchester to visualise the correlation between rioters and high levels of poverty and deprivation.

“At the moment there is no other way of getting this information than to have a person sat in court at every hearing, jotting down the details in their notebook and then copying them into a spreadsheet.”

The data and visualisation was also used in last night’s Panorama: Inside The Riots. Disappointingly, the Panorama website and solitary blog post include no links to the MEN coverage or data, and the official Twitter account not only failed to link – it has failed to tweet at all in almost two weeks.

This is the final part of a guest post by Paul Lewis that originally appeared in the book Investigative Journalism: Dead or Alive? You can read the introduction here and the second part – on the investigation of Ian Tomlinson’s death – here.

Mubenga’s death had been similarly “public”, occurring on a British Airways commercial flight to Angola surrounded by passengers. As with Tomlinson, there was a misleading account of the death put out by the authorities, which we felt passengers may wish to contest. Within days, open journalism established that Mubenga had been handcuffed and heavily restrained by guards from the private security firm G4S. He had been complaining of breathing prior to his collapse. After the investigation was published, three G4S guards were arrested and, at the time of writing, remained on bail and under investigation by the Met’s homicide unit.

Our strategy for finding out more about Mubenga’s death centred on two approaches, both aided by Twitter. The BA flight, which had been due to depart on 12 October, was postponed for 24 hours, and by the time we began investigating the following day the passengers had left Heathrow and were on route to Angola’s capital, Luanda. Raising our interest in the story via Twitter, we asked for help in locating someone who could visit the airport to interview disembarking passengers.

A freelance did just that, and managed to speak to one who said he had seen three security guards forcibly restrain Mubenga in his seat. We instantly shared that breakthrough, in the hope that it would encourage more passengers to come forward. At the same time we were publishing what we knew about the case, while being candidly open about what we did not know.

Hence the very article, published before any passengers had been tracked down, stated: “There was no reliable information about what led to the man’s death of how he became unwell.” It added, perhaps controversially: “In the past, the Home Office’s deportation policy has proved highly controversial.”

The tone was necessarily speculative, and designed to encourage witnesses to come forward. So too were the tweets. “Man dies on Angolan flight as UK tries to deport him. This story could be v big,” said one.

This articles and tweets, contained relevant search-able terms – such as the flight number – so that they could serve as online magnets, easily discoverable for any passengers with important information and access to the internet. Another tweet said: “Please contact me if you were on BA flight 77 to Angola – or know the man in this story.”

One reply came from Twitter user @mlgerstmann, a passenger on the flight who felt inappropriate force was used against Mubenga. He had come across the tweet – and then read the article – after basic Google searches. “I was also there on BA77 and the man was begging for help and I now feel so guilty I did nothing,” he tweeted.

Within hours, his shocking account of Mubenga’s death was published alongside several other passengers who had found us via the internet. An interactive graphic of the seating arrangements on the aircraft was created, enabling users to listen to audio clips of the passengers give personal accounts of what they had seen.

How verification was crucial

As with the Tomlinson investigation, verification, something paid journalists do better than their volunteer counterparts, was crucial. The fact the passengers had disseminated to remote parts of Africa – @mlgerstmann was on an oil-rig – explains why the only way to contact them was through an open, Twitter-driven investigation.

But this methodology also poses problems for authenticating the validity of sources. Journalists are increasingly finding that a danger inherent in opening up the reporting process is that they become more susceptible to attempts to mislead or hoax. This is particularly the case with live-blogs which need regular updates, require authors to make split-second decisions about the reliability of information and take care to caveat material when there are questions.

For journalists with more time, it is incumbent, therefore, to apply an equal if not more rigorous standard of proof when investigating in the open. In the Tomlinson case, when sources were encountered through the internet it was mostly possible to arrange meetings in person. That was not possible when investigating Mubenga, where there was an attempt by a bogus passenger to supply us false information.

In lieu of face to face meetings, we were able to use other means, such as asking prospective sources to send us copies of their airline tickets, to verify their accounts. What the investigations into the deaths of both Tomlinson and Mubenga show is that journalists don’t always need to investigate into the dark. Through sharing what they do know, they are most likely to discover what they don’t.

This is the second of a three-part guest post by Paul Lewis that originally appeared in the book Investigative Journalism: Dead or Alive? You can read the first part here.

The investigation into Tomlinson’s death began in the hours after his death on 1 April 2009, and culminated, six days later, in the release of video footage showing how he had been struck with a baton and pushed to the ground by a Metropolitan police officer, Simon Harwood. The footage, shot by an American businessman, was accompanied by around twenty detailed witness accounts and photographs of the newspaper seller’s last moments alive and successfully disproved the police’s explanation of the death.

The result was a criminal investigation, a national review of policing, multiple parliamentary inquiries and, by May 2011, an inquest at which a jury concluded Tomlinson had been “unlawfully killed”. At the time of writing, Harwood, who was on the Met’s elite Territorial Support Group, was awaiting trial for manslaughter.

In media studies, the case was viewed as a landmark moment for so-called “citizen journalism”. Sociologists Greer and Laughlin argue the Tomlinson story revealed a changing narrative, in which the powerful – in this case, the police – lost their status of “primary definers” of a controversial event.

Significantly, it was the citizen journalist and news media perspective, rather than the police perspective, that was assimilated into and validated by the official investigations and reports. Ultimately, it was this perspective that determined “what the story was”, structured the reporting of “what had happened and why” and drove further journalistic investigation and criticism of the Metropolitan Police Services.

The initial account of Tomlinson’s death put out by police was that he died of a heart attack while walking home from work in the vicinity of the protests, and that protesters were partly to blame for impeding medics from delivering life-saving treatment. Neither of these claims were true, but they fed into coverage that was favourable to police.

A public relations drive by the Met and City of London police was bolstered by “off the record” briefings to reporters that suggested – also wrongly – that Tomlinson’s family were not surprised by his death and upset by internet speculation it could be suspicious. These briefings contributed to a broader media narrative that endorsed police and criticised protesters.

How the police account left so many questions unanswered

The morning after father of nine died, the newspaper he had been selling outside Monument tube station, the Evening Standard, carried the headline: “Police pelted with bricks as they help dying man.” But it was plain to us, even at an early stage, that there could be more to the story. The overlydefensive police public relations campaign gave the impression there was something to hide. Embedded in the small-print of press releases, there were clues – such as the Independent Police Complaints Commission’s notification of the death – that left unanswered questions.

Most obviously, anyone who had ventured near to the protests near the Bank of England on the evening Tomlinson died would have known he collapsed in the midst of violent clashes with police. It seemed implausible, even unlikely, that the death of a bystander would not have been connected in some way to the violence. But pursuing this hunch was not easy, given the paucity of reliable information being released by police, who at times actively discouraged us from investigating the case.

All that was known about Tomlinson in the 48 hours after his death was that he had been wearing a Millwall football t-shirt. That, though, was enough to begin pursuing two separate lines of inquiry. One involved old school “shoe leather”; trawling through notepads to identify anyone who may have been in the area, or know someone who was, who could identify Tomlinson from press photographs of him lying unconscious on the ground.

That yielded one useful eye-witness, with photographic evidence of Tomlinson alive, with images of him walking in apparent distress, and lying at the feet of riot police 100 yards from where he would eventually collapse. Why was Tomlinson on the ground twice, in the space of just a few minutes? And if those photographs of the father of nine stumbling near police officers, moments before his death, were put online, would anyone make the connection?

Becoming part of a virtual G20 crowd

The answer was yes, as a direct result of the second line of inquiry: by open sharing information online, both through internet stories and Twitter, we became part of a virtual G20 crowd that had coalesced online to question the circumstances of his death. In this environment, valuable contributions to the debate, which were more sceptical in tone than those adopted by other media organisations, worked like online magnets for those who doubted the official version of events. Twitter proved crucial to sharing information with the network of individuals who had begun investigating the death of their own accord.

I had signed-up to the social media website two days before the protest, and became fascinated with the pattern of movement of “newsworthy” tweets. For example, a YouTube video uploaded by two protesters who did not see the assault on Tomlinson, but did witness his collapse minutes later and strongly disputed police claims that officers treating him were attacked with bottles, was recommended to me within seconds of being uploaded. Minutes later, Twitter investigators had identified the protesters in the film and, shortly after that, found their contact details.

Similarly, those concerned to document Tomlinson’s last moments alive, including associates of the anarchist police-monitoring group Fitwatch, were using the internet to organise.

Through Twitter I discovered there were Flickr albums with hundreds of photographs of the vicinity of this death, and dissemination of blog-posts that speculated on how he may have died. None of these images of course could be taken at face value, but they often contained clues, and where necessary the crowd helped locate, and contact, the photographer.

Journalists often mistakenly assume they can harness the wisdom of an online crowd by commanding its direction of travel. On the contrary, in digital journalism, memes (namely, concepts that spread via the internet) take their own shape organically, and often react with hostility to anyone who overtly seeks to control their direction. This is particularly the case with the protest community, which often mistrusts the so-called mainstream media. Hence it was incumbent on me, the journalist, to join the wider crowd on an equal playing-field, and share as much information as I was using as the investigation progressed.

Establishing authenticity and context

There were times, of course, when we had to hold back important material; we resisted publishing images of Tomlinson at the feet of riot police for four days, in order to establish properly their authenticity and context.

Internet contact usually does not suffice for verification, and so I regularly met with sources. I asked the most important witnesses to meet me at the scene of Tomlinson’s death, near the Bank of England, to walk and talk me through what they had seen. We only published images and video that we had retrieved directly from the source and later verified.

A different standard applies to sharing images already released on Twitter, where journalists such as National Public Radio’s Andy Carvin in the US have proven the benefits from sharing information already in the public domain to establish its significance and provenance. The break, though, as with most scoops, was partly the result of good luck, but not unrelated to the fact that our journalism had acquired credibility in the online crowd.

Chris La Jaunie, an investment fund manager, who had recorded the crucial footage of Harwood pushing Tomlinson on a digital camera, had become part of that crowd too, having spent days monitoring coverage on the internet from his office in New York. He knew the footage he had was potentially explosive. The options available to Mr La Jaunie were limited. Fearing a police cover-up, he did not trust handing over the footage. An alternative would have been to release the video onto YouTube, where would it lack context, might go unnoticed for days and even then could not have been reliably verified.

He said he chose to contact me after coming to the conclusion that ours was the news organisation which had most effectively interrogated the police version of events. It was more than a year later that my colleague Matthew Taylor and I began inquiring into the death of Mubenga. By then we had recognised the potential reach of Twitter for investigative journalism and our decision to openly investigate the death of the Angolan failed asylum seeker was a deliberate one.

Not all investigations are suited to transparent digging, and, indeed, many stories still demand top secrecy. This has been true for the three outstanding UK investigations of our times: the Telegraph’s MPs’ expenses scandal and, at the Guardian, the investigations into files obtained by WikiLeaks and phone-hacking by the News of the World. However, Tomlinson had shown that open investigations can succeed, and there were parallels with the death of Mubenga.

In the third and final part, published tomorrow, Lewis explains how he used Twitter to pursue that investigation into the death of Jimmy Mubenga, and the crucial role of verification.

In a guest post for the Online Journalism Blog, Paul Lewis shows how Twitter helped the Guardian in its investigations into the deaths of news vendor Ian Tomlinson at the London G20 protests and Jimmy Mubenga, the Angolan detainee, while he was being deported from Heathrow.

This originally appeared in the book Investigative Journalism: Dead or Alive?, which also includes another chapter previously published on the blog: Has investigative journalism found its feet online?.

Investigative journalists traditionally work in the shadows, quietly squirrelling away information until they have gathered enough to stand-up their story. That silence reassures sources, guarantees targets do not discover they are being scrutinised and, perhaps most importantly, prevents competitors from pinching the scoop.

But an alternative modus operandi is insurgent. It is counter-intuitive to traditionalist mind-set, but far more consistent with the prevailing way readers are beginning to engage with news.

Investigating in the open means telling the people what you are looking for and asking them to help search. It means telling them what you have found, too, as you find it. It works because the ease with which information can be shared via the internet, where social-media is enabling collaborative enterprise between paid journalists and citizens who are experts in their realm.

Journalism has historically been about the hunt for sources, but this open method reverses that process, creating exchanges of information through which sources can seek out journalists. There are drawbacks, of course. This approach can mean forfeiting the short-term scoop. At times, the journalist must lose control of what is being investigated, how and by whom, and watch from a distance as others make advances on their story.

They have to drop the fallacy that their job title bestows upon them a superior insight to others. But all these are all worthwhile sacrifices in the context of what can be gained.

This is illustrated by Guardian investigations into the deaths of Ian Tomlinson, the newspaper seller who died at the London G20 protests in 2009, and Jimmy Mubenga, the Angolan detainee who died while being deported from Heathrow on 12 October 2010. In both cases, eliciting cooperation through the internet – particularly Twitter – allowed us to successfully challenge the official accounts of the deaths.

In the second part Lewis explains how he used Twitter and Flickr to pursue his investigation into the death of Ian Tomlinson.

UPDATE: The stories described in these posts can also be seen in this video of Paul speaking at the TEDx conference in Thessaloniki: