In the second of a series of posts from a workshop at the Centre for Investigative Journalism Summer School (read the first part on idea generation here), I look at using generative AI tools such as ChatGPT and Google Gemini to improve sourcing and story research.

Research is arguably the second-highest risk area (after content generation) for using generative AI within journalism. The most obvious reason for this is AI’s ability to make things up (“hallucinate“) — but there are other reasons too.

One is that tools like ChatGPT are only trained on data up to a particular date — what is known as their ‘knowledge cutoff’ (this can be partly addressed through Retrieval Augmented Generation). 

Another is that there is little quality control on training data to ensure that algorithms understand the difference between facts, speculation and satire (custom GPTs like Factiverse attempt to tackle this).

Training data is also biased in a range of ways, from gender imbalance in authorship to the under-representation of numerous other groups, nationalities and languages. If you want a balanced or critical perspective to be part of your background research, you should assume you won’t get it unless you ask for it.

As a result, any use of genAI in the research phase of an investigative project needs to consider these risks and address them.

Suggesting (diverse) sources for stories

One low-risk use case is to ask generative AI to suggest sources. Like idea generation, this has the advantage of focusing on stimulating ideas rather than needing to be ‘true’ — and the journalist will quickly discover if a source has been hallucinated.

Here’s an example prompt — one that not only addresses the risk of bias, but actively uses genAI to tackle the same bias that exists in journalists’ own inherently limited backgrounds and experiences. As in the previous post, you can replace the parts in square brackets with your own particular scenario:

You are a specialist [health journalist] with years of experience working in the sector. You are planning an investigative feature on the [influence of the private sector on NHS dentistry] and are especially keen to make sure that the story draws on a more diverse range of sources than stories normally do. As one editor puts it, diversity can look like and mean a lot of different things based on the story: race and ethnicity, gender, age, geographic location, perspective, sexual orientation or economic background.
Make a list of sources that you might approach to get quotes and background for the story.

Once you’ve already established the context and rules, responses to follow-up prompts should factor in your initial preferences. For example, if your next prompt is Suggest academic experts you should get a much more diverse range of suggestions than if your first prompt about sources hasn’t established the ground rules.

Human sources aren’t the only type of source you can ask about. Here’s that prompt adapted to ask for data sources (again, if you’ve already asked the above question in the same conversation, you don’t need to repeat information from that):

You are a specialist [health journalist] with years of experience working in the sector. You are planning an investigative feature on the [influence of the private sector on NHS dentistry]. You are looking for data on this issue and are conscious that many data sources lack information on diversity. As one editor puts it, diversity can look like and mean a lot of different things based on the story: race and ethnicity, gender, age, geographic location, perspective, sexual orientation or economic background.
Make a list of data sources that you might use. Make sure sources are included which have data on under-represented groups.

Suggest document-based sources for the feature. Make sure sources are included which relate to under-represented groups.

Remember that these are just starting points — both for your research and for your interaction with generative AI: be prepared to iterate by providing further context to the AI chatbot, including feedback on responses, examples of what you are looking for, and specific information you want it to draw from (such as documents).

Assisting with Freedom of Information requests

Many investigations employ freedom of information (FOI) laws to obtain data and documents that will be vital to the story. Drafting FOI requests is an area where generative AI can help enormously, because it is a task which is all about predicting language.

Here’s a template prompt which can generate useful responses:

You are a [UK journalist] with extensive experience of writing FOI requests. You are working on an investigation into [schools' contracts with suppliers of school uniform and sports kits]. Write an FOI request that can be sent to [each school] asking for details of the contract - anticipating potential exemptions that they may try to use to refuse the request

The template demonstrates an important principle when using genAI: the more knowledge and experience you can put into the prompt, the better the result. Or put another way: genAI cannot be a replacement for your own learning.

So, knowing (through your own learning) that a key problem with FOI requests is refusals due to exemptions, you can add that vital dimension to your prompt. If you don’t know enough about an area to feel confident prompting, do a little reading to brush up on it).

This all feeds into the CAREful process (Context, Ask, Rules, Examples) outlined in the previous post.

Helping with searching (‘Google dorking’)

There’s a world of difference between how most people use a search engine and how to use one as a journalist. Advanced search skills are something that genAI can provide advice for — and there are many advanced search tools powered by AI.

One of those is DorkGPT: when given a description of the type of document you are looking for (every webpage is a document), it will return a ‘Google dork‘ (advanced search query).

For example, if you typed the description “PDFs on the Metropolitan police website mentioning strip search between October 1 2023 and December 31 2023” into DorkGPT, it will return this “dork”, with a link to run that search on Google:

site:met.police.uk filetype:pdf "strip search" after:2023-10-01 before:2023-12-31

The query uses a number of advanced search operators:

• site: narrows the search to particular domains.
• filetype: narrows a search to specific types of documents (in this case, PDFs);
• The quotation marks around "strip search" limit results to that exact phrase (rather than the two words separately); and
• after: and before: narrow the search to results that were noticed by Google at particular points in time (this doesn’t necessarily mean they were published then)

These suggestions can help you develop your own knowledge of search operators — and tools like Henk Van Ess’s Google Word Sniper and AI Search Whisperer can help further: the first uses the operator AROUND() to conduct searches where you want to look for words near each other (e.g. within the same headline, sentence or paragraph) while the second helps refine and suggest queries by combining operators with other information such as specialist domains.

Other useful custom GPTs in this field include Dork Genius and, for satellite imagery specifically, EarthGPT. Search chatgpt.com/gpts for ‘dork’ for others as they are developed.

Exploring documents using AI

One of the most common use cases for genAI is summarising documents. In Finding Newsworthy Documents using Generative AI, for example, Nick Diakopoulos outlines a methodology for using ChatGPT to surface and rank potential leads in research papers.

Summarisation should be treated with caution, however: uploading a document does not prevent hallucinations, and it’s important to take steps to reduce the likelihood of hallucinations — and make it easier for you to check the response (which is essential). For example, you should always ask it to specify which pages you can find particular information on.

Techniques used to reduce hallucinations include:

1. Limiting possible outcomes in your prompt (specify that you want a yes/no answer, or one of a list of possible responses)
2. Including relevant and/or unique data and sources
3. Creating a data template for the model to follow
4. Giving the AI a specific role—and telling it not to lie (you can also explicitly instruct the LLM to stick to the provided material)
5. Telling it what you want—and what you don’t want
6. Experimenting with the temperature in ChatGPT Playground (a lower temperature makes the responses more predictable and less ‘creative’)

Further techniques identified by BBC R&D‘s Henry Cooke include breaking long summaries down and summarising parts iteratively, and prompting the LLM to check its own work.

A template prompt incorporating some of those techniques, then, might look like this:

You are a prize-winning professional [crime] reporter. You have years of experience as a reporter and editor, extensive editorial knowledge, and excellent judgment for what makes a compelling news story. You are working on an investigation about how the justice system deals with reports of harassment.

Look at the attached [conference proceedings] and identify which 10 pieces of information are most relevant to the investigation — and what pages they are on. Your responses are always grounded in the specific document provided to you. Do not lie.

Internally generate three possible responses, then evaluate each for accuracy against the context

AI can be particularly useful in digging into company accounts. Remember that it will need a version of the accounts that is readable, so you will need to find the original PDF on the company website rather than scanned versions that might have been submitted to company registries like Companies House.

Note: it is quite common for responses to use inappropriate numbers from the accounts (i.e. numbers that are in the accounts but have been misunderstood) so do not rely on any interpretation of the numbers.

Here is a template prompt for exploring accounts:

Here is the latest annual report for a company. Use the document to understand the structure of parent companies and subsidiaries that this company sits within (include the numbers of pages where the information can be found). Generate code in Mermaid that describes that structure (remove any parentheses as these will cause problems in Mermaid)

The resulting code can be pasted into the Mermaid editor to generate a flow chart. As always, this will need checking against the sections in the accounts identified, as well as other sources. Further research into parent and subsidiary companies will be needed to expand the diagram into more levels of detail.

You may be tempted to use genAI to understand the numbers in a set of company accounts, but I’d advise caution on this: tools like ChatGPT have a tendency to misinterpret numbers, even if the numbers themselves are correct (for example failing to factor in other figures on the same page).

In general a good principle to follow is to use genAI to help direct your attention, but not to do the work of interpretation or communication based on what it’s directing you to.

Google’s AI tool NotebookLM is especially good on this front. It not only allows you to upload multiple documents and then query across those documents, but also includes footnotes in its responses which allow you to see the part of the document that each piece of information is drawn from.

A little thread on NotebookLM and journalism uses. First and most obviously, if you’re a journalist working a specific beat and often referring to reports, research or inquiry evidence, NotebookLM is just an excellent way of keeping those reports in one place, and sharing them with collaborators /1

— Chris Moran (@chrismoranuk.bsky.social) 2024-11-22T10:22:00.992Z

There is a further way that genAI tools can be used as part of sourcing, which is helping with code for scraping and cleaning data — I will cover this in a separate post specifically focusing on coding.

UPDATES

Henk Van Ess provides some excellent tips on summarising in his post Do Not “Summarise This” including the following table of types of summaries you might prompt for. “I love to use “Analyze this text semantically”” all the time,” he writes, adding that you should specify the length of the summary you want, its purpose and audience, and which parts of the text you are most interested in (e.g. data or conclusions rather than background). Finally: “Make sure to include any key actions or next steps.”