Tag Archives: foi

Managing a mass FOI project? Here’s an AI-assisted methodology for that

Sending FOIs to multiple bodies across the country to get the big picture on an issue sounds like a great idea — until the responses start to trickle in. Differences between responses often make mass FOI projects extremely time-consuming as you try to get everything into a format that allows you to ask journalistic questions and compare different authorities. Can AI help?

On one recent project I decided to put together a methodology that made the process less stressful, faster and more accurate. Here’s how it works.

Data structure

Extract & reshape

Check & verify

Combine

Audit & prioritise

Audit responses to identify the level of detail in each response and identify edge cases. Include a caveats column.
Augment manual audit with NotebookLM audit.
Identify a priority order for data, e.g. totals by outcome, hospital, category or year where these are provided separately


Design a data structure that can accommodate all responses
Structure should follow ‘tidy’ data principles, i.e. one row per combination of features (force, category, hospital, outcome, year)
Structure should include source details, e.g. filename, sheet name, name of person entering data


PDFs: use Tabula or 
vibe coding (design a prompt template to generate code to attempt to extract data). Multi-sheet XLS files: use Open Refine to import and combine sheets
Design a prompt template for generating code to reshape CSV responses


Manual checks (e.g. compare entries, check page-ending rows)
Analysis-based checks (e.g pivots, totals)
AI-based checks using a prompt template (e.g. compare files)


Use OpenRefine or: Design a prompt template for generating code to combine the resulting CSV files.
Continue reading

How to use FOI to develop good journalism habits

Freedom of Information (FOI) requests are not only one of the best ways to get original and exclusive stories that set your reporting apart — they’re also a good way to develop core journalism habits like curiosity, scepticism, and creativity. Here are some tips on how to get started with FOI while developing those qualities.

Being curious: how often is this happening? How much has it increased?

Curiosity is the first quality I identified in my series on the 7 habits of successful journalists — and FOI is a great way to hone that.

One good way to get started with FOI is to identify an event or problem that you’ve read about, and get curious about it: how many times is that event happening? How much is that problem costing? These are perfect questions for FOI.

Continue reading

9 takeaways from the Data Journalism UK conference

Attendees in a lecture theatre with 'data and investigative journalism conference 2025 BBC Shared Data Unit' on the screen.

Last month the BBC’s Shared Data Unit held its annual Data and Investigative Journalism UK conference at the home of my MA in Data Journalism, Birmingham City University. Here are some of the highlights…

Continue reading

How do I get data if my country doesn’t publish any?

Spotlight photo by Paul Green on Unsplash

In many countries public data is limited, and access to data is either restricted, or information provided by the authorities is not credible. So how do you obtain data for a story? Here are some techniques used by reporters around the world.

Continue reading

I’ve updated the Inverted Pyramid of Data Journalism — and brought together resources for every stage

Inverted pyramid of data journalism: conceive, compile, clean, context, combine (with 'question' throughout). Communicate: vis, narrate, humanise, personalise, audiolise/materialise, utilise

It’s over a decade since I published the Inverted Pyramid of Data Journalism. The model has been translated into multiple languages, taught all over the world, and included in a number of books and research papers. But in that time the model has also developed and changed through discussion and teaching, so here’s a round up of everything I’ve written or recommended on the different stages — along with a revised model in English (shown above; versions have been published in German, Spanish, Finnish, Russian and Ukrainian).

The most basic change to the Inverted Pyramid of Data Journalism is the recognition of a stage that precedes all others — idea generation — labelled ‘Conceive’ in the diagram above.

This is often a major stumbling block to people starting out with data journalism, and I’ve written a lot about it in recent years (see below for a full list).

The second major change is to make questioning more explicit as a process that (should) take place through all stages — not just in data analysis but in the way we question our sources, our ideas, and the reliability of the data itself.

A third change is to remove the ‘socialise‘ option from the communication pyramid: in conversation with Alexandra Stark I realised that this is covered sufficiently by the ‘utilise’ stage (i.e. making something useful socially).

Replacing that is a new communication option — in fact, two: audiolise and physicalise. This recognises the emergence of sonification as a method of communicating data, and physical methods of representing data from crochet to art installations.

Alongside the updated pyramid I’ve been using for the past few years I also wanted to round up links to a number of resources that relate to each stage. Here they are…

Continue reading

FOI, diversity, and imposter syndrome — an interview with Jenna Corderoy

Lyra McKee Memorial Lecture - Tuesday March 28 5.30-7pm at Birmingham City University Curzon Building. Jenna Corderoy on holding power to account and getting into journalism.

On Tuesday I will be hosting the award-winning investigative journalist and FOI campaigner Jenna Corderoy at the Lyra McKee Memorial Lecture. Ahead of the event, I asked Jenna about her tips on investigations, FOI, confidence, and the challenges facing the industry.

What’s the story you have learned the most from?

The story that I learned the most from was definitely our Clearing House investigation. Back in November 2020, we revealed the existence of a unit within the heart of government, which screened Freedom of Information (FOI) requests and instructed government departments on how to respond to requests. The unit circulated the names of requesters across Whitehall, notably the names of journalists. 

Continue reading

Investigating the World Cup: tips on making FOIA requests to create a data-driven news story

Image by Ambernectar 13

Beatriz Farrugia used Brazil’s freedom of information laws to investigate the country’s hosting of the World Cup. In a special guest post for OJB, the Brazilian journalist and former MA Data Journalism student passes on some of her tips for using FOIA.

I am from Brazil, a country well-known for football and FIFA World Cup titles — and the host of the World Cup in 2014. Being a sceptical journalist, in 2019 I tried to discover the real impacts of that 2014 World Cup on the 213 million residents of Brazil: tracking the 121 infrastructure projects that the Brazilian government carried out for the competition and which were considered the “major social legacy” of the tournament.  

In 2018 the Brazilian government had taken the website and official database on the 2014 FIFA World Cup infrastructure projects offline — so I had to make Freedom of Information (FOIA) requests to get data.

The investigation took 3 months and more than 230 FOIA requests to 33 different public bodies in Brazil. On August 23, my story was published.

Here is everything that I have learned from making those hundreds of FOIA requests:

Continue reading

Meet the man who fought a dozen FOI battles to prove that data doesn’t cause crime

Empty Shops

Empty Shops image by Dan Thompson

For the last three years Gavin Chait has been fighting — and winning — multiple Freedom of Information cases to unlock data on vacant properties. In a special guest post ahead of his latest hearing, he explains how he uses a range of evidence to fight a widely misused exemption.

I don’t know how to break this to you, but you’re probably a terrorist.

According to Richard Woolford, Strategic Director of Security and Counter Terrorism with the City of London Corporation, unoccupied properties — and especially unoccupied commercial properties — are attractive for those intent on committing terrorism.

Any knowledge about vacant properties is so dangerous, he believes, that no information about them should be placed in the public domain.

Estate agents against terrorism

This will come as tough news to terrorism enablers, especially real-estate agents, property developers, banks, insurers, and Google Street View.

If you were hoping to find somewhere new to live, or somewhere to open your dream business, then – for the safety of everyone – you’ll need to stay put. Continue reading

Emma Youle: “Local newspapers are one of the best places to do in-depth investigations because they are very well connected to the community”

emma youle

Emma Youle speaking at the Data Journalism UK conference in 2017 – photo by Wan Ulfa Nur Zuhra

As Archant’s award-winning Emma Youle announces she is to leave local newspapers to join Huffington Post UK as a special correspondent. Victoria Oliveres spoke to the investigative journalist about setting up local investigations, using data, and campaigning.

If you’ve looked at any UK journalism awards ceremony in the last few years, chances are you will have seen Emma Youle’s name: winner of the Private Eye Paul Foot Award in 2017, and the Weekly Reporter of the Year at Regional Press Awards 2016, she has also been shortlisted in many others, largely for her approach to showing the impact of national decisions at local level.

This success has come after a career of over a decade in journalism, including the last three years as part of Archant‘s investigations unit, where she uncovered in-depth stories from London boroughs.

Setting up local investigations

The unit was set up in 2015, which Youle considers to be quite pioneering at the time.

“I think local newspapers are one of the best places to do in depth investigations because they are very well connected to the community,” Emma says. Continue reading