Journalist SA Mathieson has used open data in Britain to put together an impressive new ebook. In a guest post for OJB he looks at the country’s strengths when it comes to open data — and the problems still facing journalists who want to see how the public’s money is spent.

It is tough for a British journalist to admit that their government does something well, but here goes: when it comes to openly releasing data, Great Britain (in other words England, Scotland and Wales) is second only to Taiwan according to the Global Open Data Index.

Westminster gets maximum marks for releasing data on the government’s budget, national statistics, administrative boundaries, national maps, air quality and company registers.

Northern Ireland, assessed separately, is a respectable 10th out of the 94 countries assessed.

Some specific recent improvements have made it easier and cheaper to do good journalism with public data: in June 2015, for example, Companies House, which covers England and Wales, dropped charges for online access to documents including companies’ annual filings. I was able to use that access to track how little tax companies including Facebook, Google and Apple pay in Britain.

More generally, the Office for National Statistics releases a wealth of data in machine-readable formats.

Reusable by law, not just design

The UK government also makes it straightforward for people to reuse this data through the Open Government Licence, which is broadly similar to the Creative Commons Attribution licence with a few exceptions including images and personal data. [UPDATE: thanks to Owen Bosvara in the comments for pointing out the OGL doesn’t exclude images (other than narrow exclusions for government logos etc.)]

The fact that commercial reuse is clearly allowed is helpful for journalists trying to find new uses for their research. I have taken advantage of this in my new e-book Britdata — as well as providing a guide to data available on Britain it also includes mini-profiles of all the UK’s 206 top-tier council areas with topline numbers for population, health and economic output.

The same open data has been used in the Journalists’ Local Authority Directory, an information and contacts service already available to members of the Chartered Institute of Journalists and the Society of Authors, and in the near future the National Union of Journalists.

The inflatable toy hammer

What lets Britain down is how it deals with the data it does not want to release. The grudging approach taken is clear in the infamous words of former prime minister Tony Blair, whose government brought in the UK’s Freedom of Information (FoI) Act.

“The truth is that the FoI Act isn’t used, for the most part, by ‘the people’,” he wrote in A Journey.

“It’s used by journalists. For political leaders, it’s like saying to someone who is hitting you over the head with a stick, ‘Hey, try this instead’, and handing them a mallet. It’s used as a weapon.”

As someone who uses this weapon a lot, it often feels more like an inflatable toy hammer with half the air let out.

An organisation that doesn’t want to answer a request can usually find a way to do so, with security a favourite excuse.

There is a process for challenging refusals, but this can drag on for several years — one of mine took three years to resolve, others have taken twice that — and at its later stages it is best undertaken with the backing of a large news organisation’s legal department.

Sometimes the existence of open data is used as an excuse to refuse FoIs, even if answering the request through open data will require a lot of number crunching.

Lateness is common

More prosaically, it is common for public sector organisations to answer requests late (after 20 working days in normal circumstances) or not at all. Any journalist using FoI in Britain should expect to do a lot of administration (a process I cover in the book).

When I reminded one large council recently about a late response, it told me that even though the web form through which it accepts requests had generated a reference number, the query had not been submitted.

To be fair, it then rushed through its response in just a week… saying it didn’t hold that data.

It’s not all bad. It is easy to ask for more information than an organisation is willing to provide within the law’s time budget, not least because it is very hard for an outsider to know how much work it will take to retrieve it. So I always offer to discuss the scope of requests that fall foul of this and resubmit a cut-down version, which quite often succeeds.

And some FoI officers go beyond the call of duty — such as the officer from the Northern Ireland Executive who told me within 90 minutes of making the request that it would not apply to his department, and I should try this one instead.

The second department provided a full response a fortnight later.

On this evidence, on FoI perhaps Northern Ireland deserves a rather higher rating than the rest of the UK.

Britdata is available at SAMathieson.com for £9.99 or Leanpub for US$11.99 plus VAT.