Category Archives: data journalism

Managing a mass FOI project? Here’s an AI-assisted methodology for that

Leave a reply

Sending FOIs to multiple bodies across the country to get the big picture on an issue sounds like a great idea — until the responses start to trickle in. Differences between responses often make mass FOI projects extremely time-consuming as you try to get everything into a format that allows you to ask journalistic questions and compare different authorities. Can AI help?

On one recent project I decided to put together a methodology that made the process less stressful, faster and more accurate. Here’s how it works.

Data structure Extract & reshape Check & verify Combine Audit & prioritise Audit responses to identify the level of detail in each response and identify edge cases. Include a caveats column. Augment manual audit with NotebookLM audit. Identify a priority order for data, e.g. totals by outcome, hospital, category or year where these are provided separately Design a data structure that can accommodate all responses Structure should follow ‘tidy’ data principles, i.e. one row per combination of features (force, category, hospital, outcome, year) Structure should include source details, e.g. filename, sheet name, name of person entering data PDFs: use Tabula or vibe coding (design a prompt template to generate code to attempt to extract data). Multi-sheet XLS files: use Open Refine to import and combine sheets Design a prompt template for generating code to reshape CSV responses Manual checks (e.g. compare entries, check page-ending rows) Analysis-based checks (e.g pivots, totals) AI-based checks using a prompt template (e.g. compare files) Use OpenRefine or: Design a prompt template for generating code to combine the resulting CSV files.
Continue reading →

Showing charts on video? Here are two essential techniques to make them effective

Leave a reply

Using visualisation on TV and video is very different to using charts and maps online. In video, the audience has very little time to absorb the information contained in the chart — so you need to get them to that information as quickly as possible.

Every bad example of charts in videos forgets this. And every good example uses two essential techniques: keeping things simple, and adding motion.

Continue reading →

Caught in a trap: what journalists can learn from systems thinking

Leave a reply

One of the most powerful ways to generate original journalism is to look at the systems behind stories — particularly the points where those systems fail.

For investigative work, those points are central. Surface-level scandals often stem from deeper systemic problems. So what tools do we have for recognising those patterns?

Donella Meadows’s classic book Thinking in Systems offers one: “system traps” — patterns that explain how systems get stuck, break down, or behave in ways nobody intends. They are “traps” because attempts to escape them often backfire.

System trap Journalism examples Policy resistance The war on drugs; reforms that fail; missed targets Overuse leading to shortages; climate change impacts; AI Tragedy of the commons Drift to low performance Normalisation of poor performance or low productivity Escalation Arms races; races to the bottom Success to the successful Increasing concentration of wealth or resources Shifting the burden to the intervenor Subsidies, price fixes and delaying the impact/cost of a policy Rule beating Tax avoidance, loopholes Seeking the wrong goal Schools focusing on targets over pupil welfare;

In this post I’ll explain each trap, what it looks like in the wild, and how to use it as a lens for story ideas.

Continue reading →

How to: generate hundreds of maps by combining QGIS with Python (code included!)

Leave a reply

At this year’s Dataharvest I delivered a workshop on using Python in QGIS to automate the process of exporting maps for multiple locations. Here’s how to do it (you can find a GitHub repository with materials and links here).

Making a map for a story is cool — but what if you could make a map for every reader? Or if you’re working on a project involving teams in different regions or countries, what if you could give each one of those teams a map centred on their own patch?

Normally you would have to manually move the map to centre it on a key city, and then export an image. Then do it again and again and again for every area.

Luckily, QGIS has the ability to run code. And this is a great excuse to start using it.

By organising the layers on the left you can put shapes such as flood defences over a base OpenStreetMap layer. You can also change the scale in the box underneath the map
Continue reading →

Words as data: how data journalists tell stories about documents and text

Leave a reply

Documents and other collections of text can be goldmines for data journalism — if you know how to approach them as data. Here are some techniques and inspiration for your next data project.

From stories about political speech and song lyrics, to street names and social media chatter, data journalists now have a wide range of examples of text-as-data to draw inspiration and guidance from, while tools such as Pinpoint and NotebookLM are making text analysis easier than ever.

I compiled a list of over 200 pieces of data journalism where text or documents were used as sources. Quantification techniques ranged from counting the frequency of a single word and using Google’s ngram viewer, to machine learning and topic modelling.

Looking at those articles it’s clear that, once quantified, journalists tell the same stories about text as any other piece of data: using the seven most common angles.

But how those angles are used — and how often — is where it gets interesting…

7 common angles for data stories: text and documents Scale: how often words/phrases are used Change: how language has changed Ranking: the most/least common words/phrases Variation: e.g. in relation to gender, ethnicity, ideology etc. Exploration: journeys through multiple angles; interactives Relationships: correlations, similarities and connections Meta: ‘how we quantified text’ Leads: clusters, patterns or themes for further digging
Continue reading →

How to use FOI to develop good journalism habits

Leave a reply

Freedom of Information (FOI) requests are not only one of the best ways to get original and exclusive stories that set your reporting apart — they’re also a good way to develop core journalism habits like curiosity, scepticism, and creativity. Here are some tips on how to get started with FOI while developing those qualities.

Being curious: how often is this happening? How much has it increased?

Headlines: Rising numbers of hospital patients so fed up they discharge themselves Figures reveal how many lives firefighters have saved Welsh parents owe thousands in school dinner debts
Headlines: How the cost of paying up is sending bailiffs' diaries wild Council use of bailiffs to chase debts jumps 16% in two years Acid attack hospital admissions have almost doubled Student Loans Company overcharges 78,000 graduates Schools converting to academies cost councils £30m
All these stories involve asking the question “how much” or “how many” about an issue or event

Curiosity is the first quality I identified in my series on the 7 habits of successful journalists — and FOI is a great way to hone that.

One good way to get started with FOI is to identify an event or problem that you’ve read about, and get curious about it: how many times is that event happening? How much is that problem costing? These are perfect questions for FOI.

Continue reading →

6 formas de comunicar jornalismo de dados (a Pirâmide Invertida do Jornalismo de Dados – parte 2)

Leave a reply
etapas de comunicação (visualizar, narrar, humanizar, personalizar, sonorizar/materializar, utilizar).

A pirâmide invertida do jornalismo de dados mapeia o processo de utilização de dados na reportagem, desde a geração de ideias, passando pela limpeza, contextualização e combinação, até à comunicação. A fase final — a comunicação — apresenta uma série de opções: desde a visualização e sonificação até à personalização e ferramentas. Mas quais são as melhores práticas para cada uma?

(Também disponível em inglês, alemão e espanhol, russo e ucraniano).

1. Visualização

A visualização é normalmente a forma mais rápida de comunicar os resultados do jornalismo de dados: ferramentas gratuitas como Datawrapper e Flourish muitas vezes exigem apenas que você carregue os seus dados e escolha entre várias opções de visualização.

Continue reading →

A pirâmide invertida do jornalismo de dados: Do conjunto de dados à história

Leave a reply
Diagrama mostrando a pirâmide invertida do jornalismo de dados com duas pirâmides conectadas: uma preta com as etapas de produção (conceber, compilar, limpar, contextualizar, combinar) ligada por "questionar" a uma verde com as etapas de comunicação (visualizar, narrar, humanizar, personalizar, sonorizar/materializar, utilizar).

Os projetos de jornalismo de dados envolvem várias etapas, cada uma apresentando seus próprios desafios. Para ajudar a compreendê-las, criei o que chamei de ‘Pirâmide Invertida do Jornalismo de Dados’. Ela delineia as etapas que precisam ser consideradas à medida que a matéria avança desde a conceção inicial até a comunicação dos resultados, e como elas se relacionam entre si. Abaixo, explico cada etapa, identifico questões a considerar conforme o projeto avança e ofereço conselhos e dicas sobre como enfrentá-las.

(Também disponível em Inglês, Alemão, Espanhol, Finlandês, Russo e Ucraniano.)

Etapa 1: Conceber

O primeiro desafio que um jornalista enfrenta é conceber uma ideia viável para uma matéria baseada em dados.

Continue reading →

FAQ: On data journalism and open data

Leave a reply

In the second part of this FAQ (first part here), I respond to more answers to questions from a Turkish PR company (published on LinkedIn here)…

Q: What skills do you think a journalist must absolutely have when working with data?

There are three core skills I always begin with: sorting, filtering, and calculating percentages (proportion and change). You can do most data journalism stories with those alone.

Alongside those basic technical skills it’s important to have the basic editorial skills of checking a source against other sources (following up your data by getting quotes or interviews), and being able to communicate what you’ve found clearly for a particular audience.

Continue reading →

FAQ: How has journalism been transformed?

1 Reply

In the latest FAQ, I’m publishing here answers to some questions from a Turkish PR company (published on LinkedIn here)…

Q: In your view, what has been the most significant transformation in digital journalism in recent years? 

There have been so many major transformations in the last 15 years. Mobile phones in particular have radically transformed both production and consumption — but having been through all those changes, AI feels like a biggest transformation than all the changes that we’ve already been through. 

It’s not just playing a role in transforming the way we produce stories, it’s also involved in major changes around what happens with those stories in terms of how they are distributed, consumed, and even how they are perceived: the rise of AI slop and AI-facilitated misinformation is going to radically accelerate the lack of trust in information (not just the media specifically). I’m being careful to say ‘playing a role’ because of course the technology itself doesn’t do anything: it’s how that technology is designed by people and used by people. 

Continue reading →