The following 3 videos first appeared on the Help Me Investigate blog, Help Me Investigate: Health and Help Me Investigate: Welfare. I thought I’d collect them together here too. As always, these are published under a Creative Commons licence, so you are welcome to re-use, edit and combine with other video, with attribution (and a link!).

First, Heather Brooke’s tips for starting to investigate public bodies:

Her advice on investigating health, welfare and crime:

And on using the Data Protection Act:

I’ve written a couple of guest posts for Nieman Journalism Lab and the tech news site Memeburn. The Nieman post is part of a series looking forward to 2012. I’m never a fan of futurology so I’ve cheated a little and talked about developments already in progress: new interface conventions in news websites; the rise of collaboration; and the skilling up of journalists in data.

Memeburn asked me a few months ago to write about social media’s impact on journalism’s role as the Fourth Estate, and it took me until this month to find the time to do so. Here’s the salient passage:

“But the power of the former audience is a power that needs to be held to account too, and the rise of liveblogging is teaching reporters how to do that: reacting not just to events on the ground, but the reporting of those events by the people taking part: demonstrators and police, parents and politicians all publishing their own version of events — leaving journalists to go beyond documenting what is happening, and instead confirming or debunking the rumours surrounding that.

“So the role of journalist is moving away from that of gatekeeper and — as Axel Bruns argues — towards that of gatewatcher: amplifying the voices that need to be heard, factchecking the MPs whose blogs are 70% fiction or the Facebook users scaremongering about paedophiles.

“But while we are still adapting to this power shift, we should also recognise that that power is still being fiercely fought-over. Old laws are being used in new ways; new laws are being proposed to reaffirm previous relationships. Some of these may benefit journalists — but ultimately not journalism, nor its fourth estate role. The journalists most keenly aware of this — Heather Brooke in her pursuit of freedom of information; Charles Arthur in his campaign to ‘Free Our Data’ — recognise that journalists’ biggest role as part of the fourth estate may well be to ensure that everyone has access to information that is of public interest, that we are free to discuss it and what it means, and that — in the words of Eric S. Raymond — “Given enough eyeballs, all bugs are shallow“.”

Comments, as always, very welcome.

Continuing the serialisation of the research underpinning a new Help Me Investigate project, in this third part I describe how the focus of the site was shaped by the interests of its users and staff, and how site functionality was changed to react to user needs. I also identify some areas where the site could have been further developed and improved. (Part 1 is available here; Part 2 is here)

Reflections on the proof of concept phase

By the end of the 12 week proof of concept phase the site had also completed a number of investigations that were not ‘headline-makers’ but fulfilled the objective of informing users: in particular ‘Why is a new bus company allowed on an existing route with same number, but higher prices?’; ‘What is the tracking process for petitions handed in to Birmingham City Council?’ and ‘The DVLA and misrepresented number plates’

The site had also unearthed some promising information that could provide the basis for more stories, such as Birmingham City Council receiving over £160,000 in payments for vehicle removals; and ‘Which councils in the UK (that use Civil Enforcement) make the most from parking tickets?’ (as a byproduct, this also unearthed how well different councils responded to Freedom of Information requests#)

A number of news organisations expressed an interest in working with the site, but practical contributions to the site took place largely at an individual rather than organisational level. Journalist Tom Scotney, who was involved in one of the investigations, commented: “Get it right and you’re becoming part of an investigative team that’s bigger, more diverse and more skilled than any newsroom could ever be” (Scotney, 2009, n.p.) – but it was becoming clear that most journalists were not culturally prepared – or had the time – to engage with the site unless there was a story ‘ready made’ for them to use. Once there were stories to be had, however, they contributed a valuable role in writing those stories up, obtaining official reactions, and spreading visibility.

After 12 weeks the site had around 275 users (whose backgrounds ranged from journalism and web development to locally active citizens) and 71 investigations, exceeding project targets. It is difficult to measure ‘success’ or ‘failure’ but at least eight investigations had resulted in coherent stories, representing a success rate of at least 11%: the target figure before launch had been 1-5%. That figure rose to around 21% if other promising investigations were included, and the sample included recently initiated investigations which were yet to get off the ground.

‘Success’ was an interesting metric which deserves further elaboration. In his reflection on The Guardian’s crowdsourcing experiment, for example, developer Martin Belam (2011a, n.p.) noted a tendency to evaluate success “not purely editorially, but with a technology mindset in terms of the ‘100% – Achievement unlocked!’ games mechanic.”. In other words, success might be measured in terms of degrees of ‘completion’ rather than results.

In contrast, the newspaper’s journalist Paul Lewis saw success in terms of something other than pure percentages: getting 27,000 people to look at expense claims was, he felt, a successful outcome, regardless of the percentage of claims that those represented. And BBC Special Reports Editor Bella Hurrell – who oversaw a similar but less ambitious crowdsourcing project on the same subject on the broadcaster’s website, felt that they had also succeeded in genuine ‘public service journalism’ in the process (personal interview).

A third measure of success is noted by Belam – that of implementation and iteration (being able to improve the service based on how it is used):

“It demonstrated that as a team our tech guys could, in the space of around a week, get an application deployed into the cloud but appear integrated into our site, using a technology stack that was not our regular infrastructure.

“Secondly, it showed that as a business we could bring people together from editorial, design, technology and QA to deliver a rapid turnaround project in a multi-disciplinary way, based on a topical news story.

“And thirdly, we learned from and improved upon it.“ (Belam, 2010, n.p.)

A percentage ‘success’ rate of Help Me Investigate, then, represents a similar, ‘game-oriented’ perspective on the site, and it is important to draw on other frameworks to measure its success.

For example, it was clear that the site did very well in producing raw material for ‘journalism’, but it was less successful in generating more general civic information such as how to find out who owned a piece of land. Returning to the ideas of Actor-Network Theory outlined above, the behaviour of two principal actors – and one investigation – had a particular influence on this, and how the site more generally developed over time. Site user Neil Houston was an early adopter of the site and one of its heaviest contributors. His interest in interrogating data helped shape the path of many of the site’s most active investigations, which in turn set the editorial ‘tone’ of the site. This attracted users with similar interests to Neil, but may have discouraged others who did not – further research would be needed to establish this.

Likewise, while Birmingham City Council staff contributed to the site in its earliest days, when the council became the subject of an investigation staff’s involvement was actively discouraged (personal interview with contributor). This left the site short of particular expertise in answering civic questions.

At least one user commented that the site was very ‘FOI [Freedom Of Information request]-heavy’ and risked excluding users interested in different types of investigations, or who saw Freedom of Information requests as too difficult for them. This could be traced directly to the appointment of Heather Brooke as the site’s support journalist. Heather is a leading Freedom of Information activist and user of FOI requests: this was an enormous strength in supporting relevant investigations but it should also be recognised how that served to set the editorial tone of the site.

This narrowing of tone was addressed by bringing in a second support journalist with a consumer background: Colin Meek. There was also a strategic shift in community management which involved actively involving users with other investigations. As more users came onto the site these broadened into consumer, property and legal areas.

However, a further ‘actor’ then came into play: the legal and insurance systems. Due to the end of proof of concept funding and the associated legal insurance the team had to close investigations unrelated to the public sector as they left the site most vulnerable legally.

A final example of Actor-Network Theory in action was a difference between the intentions of the site designers and its users. The founders wanted Help Me Investigate to be a place for consensus, not discussion, but it was quickly apparent users did not want to have to go elsewhere to have their discussions. Users needed to – and did – have conversations around the updates that they posted.

The initial challenge-and-result model (breaking investigations down into challenges with entry fields for the subsequent results, which were required to include a link to the source of their information) was therefore changed very early on to challenge-and-update: people could now update without a link, simply to make a point about a previous result, or to explain their efforts in failing to obtain a result.

One of the challenges least likely to be accepted by users was to ‘Write the story up’. It seemed that those who knew the investigation had no need to write it up: the story existed in their heads. Instead it was either site staff or professional journalists who would normally write up the results. Similarly, when an investigation was complete, it required site staff to update the investigation description to include a link to any write-up. There was no evidence of a desire from users to ‘be a journalist’. Indeed, the overriding objective appeared rather to ‘be a citizen’.

In contrast, a challenge to write ‘the story so far’ seemed more appealing in investigations that had gathered data but no resolution as yet. The site founders underestimated the need for narrative in designing a site that allowed users to join investigations while they were in progress.

As was to be expected with a ‘proof of concept’ site (one testing whether an idea could work), there were a number of areas of frustration in the limitations of the site – and identification of areas of opportunity. When looking to crowdfund small amounts for an investigation, for example, there were no third party tools available that would allow this without going through a nonprofit organisation. And when an investigation involved a large crowdsourcing operation the connection to activity conducted on other platforms needed to be stronger so users could more easily see what needed doing (e.g. a live feed of changes to a Google spreadsheet, or documents bookmarked using Delicious).

Finally investigations often evolved into new questions but had to stay with an old title or risk losing the team and resources that had been built up. The option to ‘export’ an investigation team and resources into a fresh question/investigation was one possible future solution.

‘Failure for free’ was part of the design of the site in order to allow investigations to succeed on the efforts of its members rather than as a result of any top-down editorial agenda – although naturally journalist users would concentrate their efforts on the most newsworthy investigations. In practice it was hard to ‘let failure happen’, especially when almost all investigations had some public interest value.

Although the failure itself was not an issue (and indeed the failure rate lower than expected), a ‘safety net’ was needed that would more proactively suggest ways investigators could make their investigation a success, including features such as investigation ‘mentors’ who could pass on their experience; ‘expiry dates’ on challenges with reminders; improved ability to find other investigators with relevant skills or experience; a ‘sandbox’ investigation for new users to find their feet; and developing a metric to identify successful and failing investigations.

Communication was central to successful investigations and two areas required more attention: staff time in pursuing communication with users; and technical infrastructure to automate and facilitate communication (such as alerts to new updates or the ability to mail all investigation members)

The much-feared legal issues threatened by the site did not particularly materialise. Out of over 70 investigations in the first 12 weeks, only four needed rephrasing to avoid being potentially libellous. Two involved minor tweaks; the other two were more significant, partly because of a related need for clarity in the question.

Individual updates within investigations, which were post-moderated, presented even less of a legal problem. Only two updates were referred for legal advice, and only one of those rephrased. One was flagged and removed because it was ‘flamey’ and did not contribute to the investigation.

There was a lack of involvement by users across investigations. Users tended to stick to their own investigation and the idea of ‘helping another so they help you’ did not take root. Further research is needed to see if there was a power law distribution at work here – often seen on the internet – of a few people being involved in lots of investigations, most being involved in one, and a steep upward curve between.

In the next part, published tomorrow, I look at one particular investigation in an attempt to identify the qualities that made it successful. 

If you want to get involved in the latest Help Me Investigate project, get in touch on paul@helpmeinvestigate.com

Justice photo by mira66

Few things sum up the division of the UK around the riots like the sentencing of those involved. Some think courts are too lenient, while others gape at six month sentences for people who stole a bottle of water.

These judgments are often made on the basis of a single case, rather than any overall view. And you might think, in such a situation, that a journalist’s role would be to find out just how harsh or lenient sentencing has been – not just across the 1,600 or more people who have been arrested during the riots, but also in comparison to previous civil disturbances – or indeed, to similar crimes outside of a riot situation.

As Martin Belam argues:

“Really good data journalism will help us untangle the truth from those prejudiced assumptions. But this is data journalism that needs to stay the course, and seems like an ideal opportunity to do “long-form data journalism”. How long will these looters serve? What is the ethnic make-up and age range of those convicted? How many other criminals will get an early release because our jails are newly full of looters? How many people convicted this week will go on to re-offend?”

And yet, amazingly, we cannot reliably answer these questions – because it is still not possible to get raw data on sentencing in UK courts, not even through FOI.

In the absence of such data, The Guardian are attempting to collate data on convictions at magistrates courts (the Crown Courts have yet to get going), with over 200 cases in their spreadsheet.

It’s a laudable endeavour – but the very fact that they have to undertake it should be an embarrassment to the English and Welsh justice system.

Justice is blind – but so are we

The aphorism – from a case in the 1920s – that justice must not only be done, but “must also be seen to be done” is still not carried through into practical acts. A year ago Heather Brooke wrote about her own experiences along these lines, and the court’s desire to maintain “control” over recordings of court proceedings:

“Control” is exactly what a court shouldn’t be exerting. Once it is decided that it is open, there should be no restriction on how that open hearing is processed. She went on to say that she’d allow me to record now but I’d have to wait for a future ruling before I could “use” the recording. The next day in court the Judge announced she’d made her ruling. “Please turn your tape recorder off,” she said, looking sternly at me over her glasses. I did so.

8 months before that Will Perrin wrote about the problems of hyperlocal bloggers wishing to report on their courts.

Little has changed.

In frustration at this, a number of coders around the country have been attempting to find ways to record what happens in court. Examples include Cause List (cases being heard in the courts of England and Wales right now), InCourts Daily (simply names and times of cases being heard – it has dozens of Twitter accounts for particular regions) and the long-running BAILII, which publishes partial information on case law but this year hit funding problems (another useful resource is The Law Pages).

There is generic sentencing data on Data.gov.uk – but not at a level of detail or timeliness that would allow you to answer basic questions about ‘justice being done’ (in contrast, sentencing data for Scottish courts at the Judiciary of Scotland site, includes RSS feeds for sentencing statements and summaries of opinion, among others).

Newspapers: publish in structured formats

Meanwhile, there’s been a steady decline in court reporting across the regional press and national news agencies which has made the situation worse.

What remains – those dry court reports that fill the space between adverts in most local newspapers – could in theory be of use to those who want to ask bigger questions than “Which one of my neighbours has been in trouble?”

But it is being published online in a format which makes it far from easy for anyone to interrogate – including newspapers’ own journalists.

Take, for example, the Telegraph & Argus’s court pages: an RSS feed which says nothing more than “The following have been dealt with by Bradford magistrates” and dozens of judgments condensed into four paragraphs (there is some structure which you can extract into a table – but it could be easier).

Even a specialist service like Court News UK only appears to deal in stories, while, interestingly, the Wigan World website (which has partnered with the local paper) is a rarity in presenting court sentences in a basic table with an accompanying search facility. (If you know of any good examples of online court data, please let me know).

There’s something to be learned here: if newspapers published comprehensive sentencing data in tables, it’s more likely that users will help find the stories in them.

Meanwhile, sentencing data is increasingly (and arbitrarily) being published on Twitter by police authorities – starting with the West Midlands and Manchester – and journalists need to be able both to deal with that information, and check it.

If journalists are to do more than provide a platform for a blame game, we need to put pressure on the courts to publish as much sentencing data as possible, in as open a format as possible, and as close to real time as possible. Justice must be seen to be done.

Various commentators over the past year have made the observation that “Data is the new oil“. If that’s the case, journalists should be following the money. But they’re not.

Instead it’s falling to the likes of Tony Hirst (an Open University academic), Dan Herbert (an Oxford Brookes academic) and Chris Taggart (a developer who used to be a magazine publisher) to fill the scrutiny gap. Recently all three have shone a light into the move towards transparency and open data which anyone with an interest in information would be advised to read.

Hirst wrote a particularly detailed post breaking down the results of a consultation about higher education data.

Herbert wrote about the publication of the first Whole of Government Accounts for the UK.

And Taggart made one of the best presentations I’ve seen on the relationship between information and democracy.

What all three highlight is how control of information still represents the exercise of power, and how shifts in that control as a result of the transparency/open data/linked data agenda are open to abuse, gaming, or spin.

Control, Cost, Confusion

Hirst, for example, identifies the potential for data about higher education to be monopolised by one organisation – UCAS, or HEFCE – at extra cost to universities, resulting in less detailed information for students and parents.

His translation of the outcomes of a HEFCE consultation brings to mind the situation that existed for years around Ordnance Survey data: taxpayers were paying for the information up to 8 times over, and the prohibitive cost of accessing that data ended up inspiring the Free Our Data campaign. As Hirst writes:

“The data burden is on the universities?! But the aggregation – where the value is locked up – is under the control of the centre? … So how much do we think the third party software vendors are going to claim for to make the changes to their systems? And hands up who thinks that those changes will also be antagonistic to developers who might be minded to open up the data via APIs. After all, if you can get data out of your commercially licensed enterprise software via a programmable API, there’s less requirement to stump up the cash to pay for maintenance and the implementation of “additional” features…”

Meanwhile Dan Herbert analyses another approach to data publication: the arrival of commercial-style accounting reports for the public sector. On the surface this all sounds transparent, but it may be just the opposite:

“There is absolutely no empiric evidence that shows that anyone actually uses the accounts produced by public bodies to make any decision. There is no group of principals analogous to investors. There are many lists of potential users of the accounts. The Treasury, CIPFA (the UK public sector accounting body) and others have said that users might include the public, taxpayers, regulators and oversight bodies. I would be prepared to put up a reward for anyone who could prove to me that any of these people have ever made a decision based on the financial reports of a public body. If there are no users of the information then there is no point in making the reports better. If there are no users more technically correct reports do nothing to improve the understanding of public finances. In effect all that better reports do is legitimise the role of professional accountants in the accountability process.

Like Hirst, he argues that the raw data – and the ability to interrogate that – should instead be made available because (quoting Anthony Hopwood): “Those with the power to determine what enters into organisational accounts have the means to articulate and diffuse their values and concerns, and subsequently to monitor, observe and regulate the actions of those that are now accounted for.”

This is a characteristic of the transparency initiative that we need to be sharper around as journalists. The Manchester Evening News discovered this when they wanted to look at spending cuts. What they found was a dataset that had been ‘spun’ to make it harder to see the story hidden within, and to answer their question they first had to unspin it – or, in data journalism parlance, clean it. Likewise, having granular data – ideally from more than one source – allows us to better judge the quality of the information itself.

Chris Taggart meanwhile looks at the big picture: friction, he says, underpins society as we know it. Businesses such as real estate are based on it; privacy exists because of it; and democracies depend on it. As friction is removed through access to information, we get problems such as “jurisdiction failure” (corporate lawyers having “hacked” local laws to international advantage), but also issues around the democratic accountability of ad hoc communities and how we deal with different conceptions of privacy across borders.

Questions to ask of ‘transparency’

The point isn’t about the answers to the questions that Taggart, Herbert and Hirst raise – it’s the questions themselves, and the fact that journalists are, too often, not asking them when we are presented with yet another ‘transparency initiative‘.

If data is the new oil those three posts and a presentation provide a useful introduction to following the money.

(By the way, for a great example of a journalist asking all the right questions of one such initiative, however, see The Telegraph’s Conrad Quilty-Harper on the launch of Police.uk)

Data is not just some opaque term; something for geeks: it’s information: the raw material we deal in as journalists. Knowledge. Power. The site of a struggle for control. And considering it’s a site that journalists have always fought over, it’s surprisingly placid as we enter one of the most important ages in the history of information control.

As Heather Brooke writes today of the hacking scandal:

“Journalism in Britain is a patronage system – just like politics. It is rare to get good, timely information through merit (eg by trawling through public records); instead it’s about knowing the right people, exchanging favours. In America reporters are not allowed to accept any hospitality. In Britain, taking people out to lunch is de rigueur. It’s where information is traded. But in this setting, information comes at a price.

“This is why there is collusion between the elites of the police, politicians and the press. It is a cartel of information. The press only get information by playing the game. There is a reason none of the main political reporters investigated MPs’ expenses – because to do so would have meant falling out with those who control access to important civic information. The press – like the public – have little statutory right to information with no strings attached. Inside parliament the lobby system is an exercise in client journalism that serves primarily the interests of the powerful. Freedom of information laws bust open the cartel.”

But laws come with loopholes and exemptions, red tape and ignorance. And they need to be fought over.

One bill to extend the FOI law to “remove provisions permitting Ministers to overrule decisions of the Information Commissioner and Information Tribunal; to limit the time allowed for public authorities to respond to requests involving consideration of the public interest; to amend the definition of public authorities” and more, for example, was recently put on indefinite hold. How many publishers and journalists are lobbying to un-pause this?

So let’s simplify things. And in doing so, there’s no better place to start than David Eaves’ 3 laws of government data.

This is summed up as the need to be able to “Find, Play and Share” information. For the purposes of journalism, however, I’ll rephrase them as 3 questions to ask of any transparency initiative:

1. If information is to be published in a database behind a form, then it’s hidden in plain sight. It cannot be easily found by a journalist, and only simple questions will be answered.
2. If information is to be published in PDFs or JPEGs, or some format that you need proprietary software to see, then it cannot be easily be questioned by a journalist
3. If you will have to pass a test to use the information, then obstacles will be placed between the journalist and that information

The next time an organisation claims that they are opening up their information, tick those questions off. (If you want more, see Gurstein’s list of 7 elements that are needed to make effective use of open data).

At the moment, the history of information is being written without journalists.

This is the fourth part of my inaugural lecture at City University London, ‘Is Ice Cream Strawberry?’. You can find part one here, part two here, and part three here.

Human capital

So here’s person number 4: Gary Becker, a Nobel prize-winning economist.

Fifty years ago he used the phrase ‘human capital’ to refer to the economic value that companies should ascribe to their employees.

These days, of course, it is common sense to invest time in recruiting, training and retaining good employees. But at the time employees were seen as a cost.

We need a similar change in the way we see our readers – not as a cost on our time but as a valuable part of our operations that we should invest in recruiting, developing and retaining.

Any online operation that does not incorporate its users in production is not only democratically deficient, it is commercially inefficient.

Of course some are inclined to see user generated content as a mass of ignorance, abuse and waffle. Those people should ask how much work has been put into attracting good contributors? Into developing a healthy commenting culture? And how much has been invested into giving the good users a reason to keep coming back?

Journalism’s conflicted future

I have spoken about journalism’s ego problem. This is worsened by the fact that journalism is going through an identity crisis, which will become increasingly problematic as it tries to reinvent itself for an uncertain future.

And as always, this is nothing new. In the 1920s journalism faced a similar conflict: between the journalism of information and the journalism of stories. Should we, as journalists, perform a role of providing citizens with the information that they need to make informed decisions? Or are we just in the business of great stories?

The source of that conflict then was the rise of the scientific method, as I explained at the start of this lecture. The source of today’s conflict could be traced to institutional change in news organisations becoming part of larger entertainment empires – and the melting pot of online publication.

Where you stand on the role of journalists will likely depend on whether you think you’re in the business of building cars, constructing roads or organising picnics, and what role you think journalism should perform in a democracy.

Is journalism part of a deliberative democracy, in which the media provides a public forum for debate and consensus?

Is journalism’s role is to provide citizens with information – as part of a liberal democracy?

Or should the media encourage participation and engagement as part of a participatory democracy?

The institutional history of journalism kept those views somewhat separated – as Lokman Tsui explores in his ethnography of Global Voices.

But as these cultures of journalism clash in the online space it is more important than ever that we reflect on our own views of where we stand.

And as educators we should be teaching our students to be aware of their positions and how that affects what they report on, how they report it, and who gets a voice in its coverage.

If objectivity is to remain a journalistic value, then it should be modern objectivity, not this 19th century construct that passes for objectivity in most newswriting: the setting up of an arbitrary fence, and the selection of a source from each side of it as an indication of ‘balance’.

Culture shift

In many ways culture is the way that people and institutions communicate with each other. And just as institutional culture shapes the journalism that we create, for the last couple of decades there has been a growing movement outside of news institutions that sees democracy as both participatory and information driven.

The campaign for Freedom of Information, the work of MySociety in opening up voting records and debate transcripts so the public could see what their representatives were doing and saying in their name. The Free Our Data campaign – which sought to give the public the right to use information that was paid for with public money. And the Linked Data and Open Data movements which have campaigned to make public bodies publish data in a form that makes it easier to interrogate.

What these people – and I want to name some of them here:

• Tom Steinberg and Tom Loosemore
• Heather Brooke
• Charles Arthur
• Tim Berners Lee and
• Chris Taggart

What these people have done – and are still doing – is making power accountable, promoting a cultural expectation that we should have access to information about how our money is spent, and that most publicly funded information should be available to the people who paid for it.

This of course is one of the first steps to ‘holding power to account’, the great argument that publishers make for their existence. That is what the Telegraph did with the MPs’ expenses; what The Guardian have done with Wikileaks data.

But too much of this groundwork is lying ignored and unsupported by the mainstream press.

When Walsall Council released their spending data last year the webpage received more visits than the rest of the council website. They received several enquiries from people like Chris Taggart asking for information about why particular items had been redacted – but they received only one enquiry from the local newspaper.

And that was to ask: ‘Why have you released the data early?’

For comparison I want to show you a video of Ellen Miller of the Sunlight Foundation in the US talking about their government’s transparency initiative.

Where is the news organisation in the UK that is lobbying like this?

A similar cultural shift is happening around public meetings and hearings, with hyperlocal blogs who want to make processes of law and democracy transparent.

Simon Perry of the Ventnor Blog was ejected from a coroner’s court last year on the grounds that he was neither a member of the press nor a member of the public. Richard Taylor was investigated by Cambridge City Council for recording public meetings – he was not told what the grounds of the investigation were. In Brighton a councillor was disciplined for posting clips of council meetings to YouTube. And Heather Brooke was told that she could not make an audio recording of a hearing because the tribunal could not “maintain the necessary degree of control over the transcript.” When Brooke asked for a copy of the ruling she was told that there was to be no written record of it.

This is one area where journalists and news organisations can be investing in their users. It should not just be bloggers pushing for these changes.

Part five can be found here.

UPDATE: You can vote to repeal the ban on recording court proceedings here (Thanks to Alistair Kelman in the comments)

Heather Brooke is calling for a campaign to allow recording in UK courts. I agree. In the comments below, let’s talk strategy.

Meanwhile, here’s some of the background from Brooke’s related blog post:

How:

“The simple answer is to allow tape recorders for all: no party is disadvantaged and an ‘official’ recording is there for checking. This is how it works in other countries. But this is to ignore the root objection of the courts: that they are losing control of how court proceedings are presented to the public.”

Why:

“You might like to know whether the builder you’re going to give your keys to has any convictions for theft or if the company you’re about to do business with has a report for fraud. Tough. This information is not a click of a button away. Instead you’ll have to know the details of the case before you can call up any records – even though it’s the existence of cases that you’re trying to find in the first place. It’s Catch-22. If you do know the details of the case you’re then forced to undergo a tortuous and tedious process which involves battling a raft of petty officials across a number of court offices all for the simple purpose of accessing information that is supposedly public.”

And what:

“There are three main things that would make the courts useful to the general public:

1. knowing by name who is using them (the court list);
2. why (the particulars of claim);
3. the result (the verdict, sentence or settlement).

“Yet trying to get any, let alone all, of these is fraught with difficulty.”

So: strategy. To kick things off, I’ll give you 3 starters:

• the much-mocked Your Freedom website inviting suggestions for laws to get rid of (how seriously is this being taken in government?)
• the Number 10 petitions site (likewise)
• Contact your MP.

Come up with some better ideas than that, and we’re somewhere.

Meanwhile, to spread awareness of this, why not tweet about this with the hashtag #opencourts (UPDATE: Also #courtrecord thanks to @JackofKent)

In the week that a general election is called, Heather Brooke’s latest book couldn’t have been better timed. The Silent State is a staggeringly ambitious piece of work that pierces through the fog of the UK’s bureaucracies of power to show how they work, what is being hidden, and the inconsistencies underlying the way public money is spent.

Like her previous book, Your Right To Know, Brooke structures the book into chapters looking at different parts of the power system in the UK – making it a particularly usable reference work when you want to get your head around a particular aspect of our political systems.

Chapter by chapter

Chapter 1 lists the various databases that have been created to maintain information on citizens - paying particular focus to the little-publicised rack of databases holding subjective data on children. The story of how an old unpopular policy was rebranded to ride into existence on the back of the Victoria Climbie bandwagon is particularly illustrative of government’s hunger for data for data’s sake.

Picking up that thread further, Chapter 2 explores how much public money is spent on PR and how public servants are increasingly prevented from speaking directly to the media. It’s this trend which made The Times’ outing of police blogger Nightjack particularly loathsome and why we need to ensure we fight hard to protect those who provide an insight into their work on the ground.

Chapter 3 looks at how the misuse of statistics led to the independence of the head of the Office of National Statistics – but not the staff that he manages – and how the statistics given to the media can differ quite significantly to those provided when requested by a Select Committee (the lesson being that these can be useful sources to check). It’s a key chapter for anyone interested in the future of public data and data journalism.

Bureaucracy itself is the subject of the fourth chapter. Most of this is a plea for good bureaucracy and the end of unnamed sources, but there is still space for illustrative and useful anecdotes about acquiring information from the Ministry of Defence.

And in Chapter 5 we get a potted history of MySociety’s struggle to make politicians accountable for their votes, and an overview of how data gathered with public money – from The Royal Mail’s postcodes to Ordnance Survey – is sold back to the public at a monopolistic premium.

The justice system and the police are scrutinised in the 6th and 7th chapters – from the twisted logic that decreed audio recordings are more unreliable than written records to the criminalisation of complaint.

Then finally we end with a personal story in Chapter 8: a reflection on the MPs’ expenses saga that Brooke is best known for. You can understand the publishers – and indeed, many readers – wanting to read the story first-hand, but it’s also the least informative of all the chapters for journalists (which is a credit to all that Brooke has achieved on that front in wider society).

With a final ‘manifesto’ section Brooke summarises the main demands running across the book and leaves you ready to storm every institution in this country demanding change. It’s an experience reminiscent of finishing Franz Kafka’s The Trial – we have just been taken on a tour through the faceless, logic-deprived halls of power. And it’s a disconcerting, disorientating feeling.

Journalism 2.0

But this is not fiction. It is great journalism. And the victims caught in expensive paper trails and logical dead ends are real people.

Because although the book is designed to be dipped in as a reference work, it is also written as an eminently readable page-turner – indeed, the page-turning gets faster as the reader gets angrier. Throughout, Brooke illustrates her findings with anecdotes that not only put a human face on the victims of bureaucracy, but also pass on the valuable experience of those who have managed to get results.

For that reason, the book is not a pessimistic or sensationalist piece of writing. There is hope – and the likes of Brooke, and MySociety, and others in this book are testament to the fact that this can be changed.

The Silent State is journalism 2.0 at its best – not just exposing injustice and waste, but providing a platform for others to hold power to account. It’s not content for content’s sake, but a tool. I strongly recommend not just buying it – but using it. Because there’s some serious work to be done.

There have been rumblings for a while about the establishment of a UK investigation foundation, and now it’s here. They’re not accepting cash at the moment, just pledges of support and help. So go help them.

Here’s their open letter:

Sir,

A group of journalists – among them some of Britain’s most experienced investigative reporters – has been meeting to consider what should be done to address the deep and sustained crisis in our craft.

Even before the onset of the recession, thousands of media jobs had been lost across Britain. The internet, digital television, falling advertising revenues and the commercial pressures of the 24/7 news cycle have all had an impact. While there continues to be great examples of courageous journalism, a growing number of news outlets are increasingly putting emphasis on entertainment, on rapidly-delivered and recycled news rather than the investigation and discovery of what the public wants and needs to know.

And yet the need for information that can hold public institutions to account is as great as ever. Scandals such as MPs’ expenses, the intelligence failings in the run-up to war in Iraq or the behaviour of bankers which led to crisis in the world economy show how many aspects of public life remain critically under-examined.

We have decided to announce the formation of a Foundation for Investigative Reporting to look at what practical steps can be taken, both to experiment with new means of funding essential investigations and to inspire a new generation of reporters. The Foundation will act as an incubator for new ways of conducting journalism and for new ideas of how to finance this kind of reporting.

None of us profess to know the answers, but we are convinced it is time for some bold experiments. In particular, we would like to persuade all those who believe in the value of serious reporting in the public interest to join an open debate about journalism’s future – and to support this work financially.

As of today, we are creating a new rolling fund that will aim to help provide the initial cash required for the kind of risky, challenging reporting and film-making for which there is a crying demand, but few sponsors. This not-for-profit venture will not compete directly with established media, but will instead provide the seeds from which the big stories can grow.

We ask anyone interested in joining the debate to pledge their support or partnership, as well as to offer their views about what should be brought to public light – and isn’t. Further information can be obtained from our website: www.investigationsfund.org.

Yours sincerely,

Antony Barnett, Martin Bright, Heather Brooke, Peter Barron, Nick Davies, Nick Fielding, Misha Glenny, Stephen Grey (editor), Mark Hollingsworth, Andrew Jennings, Philip Knightley, Paul Lashmar, David Leigh, Jason Lewis

Following the tone set so succinctly by Glyn Moody, I thought I would add my own thoughts on what Sir Tim should say to the government when he bends their ear on transparency.

Firstly, I would second everything that Glyn says.

But I’m going to be cynical and strategic, and urge Sir Tim to emphasise the importance of open data on a couple of areas that are close to the government’s hearts.

1. Stimulating growth in the economy.

You could compare a genuinely significant release of public data to an economic stimulus.

Like cutting VAT, only cheaper.

At minimal cost you could have a new raw material that startups and established media organisations alike could create new value out of. Some of those would create commercial implications far exceeding any revenue generated within government (as research recently suggested in relation to the comparably valuable Ordnance Survey data).

Repeat after me: jobs and money, jobs and money.

2. Efficiencies and passing on costs in the public sector

Samuel Butler’s Erewhon puts it particularly well:

You will sooner gain your end by “appealing to men’s pockets, in which they have generally something of their own, than to their heads, which contain for the most part little but borrowed or stolen property”

Public sector spending is going to drop whichever party is in power. Let’s play to that.

By opening up public data the government will effectively be able to pass on some development costs to willing volunteers who mash up the data in their own ways. The difference is that people will do this to their own agendas and for their own benefit.

But more importantly, the results of this experimentation – if supported and encouraged – should produce work that makes it more efficient to interact with public data and therefore public bodies. If I can use a slider to find out which schools are within 3 miles, that saves 20 minutes of someone answering a phonecall in the local education department. If I can have a Facebook app which tells other users how much money alcohol abuse is costing my local hospital, it might save the NHS a bob or two. You get the picture. 

Oh yes, and it’s important for democracy, civic engagement and digital literacy

The limited data that’s available in the UK is an embarrassment. Imagine what MySociety could do with what’s available in the US.

Likewise, for all the talk of transparency, the recent announcement that Cabinet Papers and information relating to the Royal Family would be exempt from the Freedom of Information act is a backward step. Heather Brooke’s concerns proved right.

The cynic in me sees the appointment of Berners-Lee as an action intended to generate the illusion of movement – “We’re working on it”. But the Freedom of Information act is possibly the most positive contribution the Labour government has made to this country’s political health since it came to power, and not to follow through on promises made would be an enormous political mistake.

So I will add one request to my advice above: I would stress that any discussion of transparency acknowledges the importance of requiring any organisation using public funds to make their data public too. So much public work is outsourced to the private sector that it is particularly difficult to see whether public money is spent responsibly.

More at Podnosh, BBC, Emma Mulqueeny, Simon Dickson and Amused Cynicism.