What are the risks of information you put into an AI chat becoming public? Evaluating those risks is slightly different to other information security challenges because of the way large language models work, so here’s a guide to assessing and managing information security with AI.

Understanding risk with AI is about understanding probabilities

One of the most common misunderstandings about large language models (LLMs) is the idea that if you upload a document such as an interview transcript, confidential leak or sensitive dataset to an AI tool like ChatGPT, it might reveal that document to another user.

There are no publicly documented cases of this happening.

The reason for this is the same reason why AI shouldn’t be treated as a search engine: large language models like GPT-5 don’t work by retrieving documents for users — they work by separating documents into parts of language (‘tokens’ in the jargon), adding those to a vast ocean of other tokens, and training on a database of relationships between those.

When you ask a question like “Who is the CEO of Apple?” a genAI tool doesn’t retrieve information from a document. Instead it breaks down your question into tokens like “CEO” and “Apple” and “Who”, calculates the most statistically likely meaning of those things (e.g. is that apple probably a fruit or a company), and then the most statistically likely sequence of words which might form a meaningful answer to that question.

(In this example, “Tim” and “Cook” are likely to be more strongly associated with “CEO” and “Apple” in its training data than other tokens, and so they will be formed into a response: “Tim Cook”.)

If information you provide in a prompt is used in an LLM’s training (see below), then, it will be a) broken down into tokens and b) those tokens’ relationships with other tokens will be competing with lots of other token relationships to be the ‘most likely’ answer to any question.

So when do those tokens and relationships become a risk?

Probabilities are shaped by training

There are two main steps in a large language model’s training: the most important is pretraining: this is when the model is trained using an enormous collection of language. If your information ends up being included in that collection, it is a drop in the ocean of those trillions of words.

There will also be a delay before it is included in responses, because each model is trained a number of months before it is release — this is called the knowledge cutoff.

ChatGPT’s latest model, GPT-5.6, has a knowledge cutoff of five months ago, which means it doesn’t contain any information after mid-February. If Apple’s CEO has changed since then, that won’t be reflected in its core training data. You can see each model’s cutoff dates here.

The other key step in a model’s training is called reinforcement learning from human feedback (RLHF).

Whenever ChatGPT presents you with two possible responses and asks to choose which one you prefer, that’s you helping to train the model by reinforcing a preference.

AI companies pay humans to train the model too (there are human rights and exploitation concerns around this). Those people train the model by looking at a small sample of conversations and selecting the better responses.

Your information might be selected for this sample (out of millions of conversations every day). If that happens it will be incorporated into a model’s training with a shorter time lag, and weighted more heavily. For example, even where the majority of pretraining data says the Apple CEO is Tim Cook, if a human has ranked a different, more up-to-date, response as more or less accurate, then it can outweigh that other training.

The chances of your information either being incorporated into pretraining data, or selected for reinforcement training, will depend to some extent on policy.

Check the policies

Free consumer accounts with ChatGPT, Gemini, Copilot and Claude opt users in by default to agreeing to allow their conversations to be used for training. The chances of a conversation being used is still tiny, but you can reduce it further by opting out (you might also consider opting out of AI training on platforms like Adobe and AWS where you might use sensitive data).

Enterprise and business accounts and APIs tend to opt users out by default.

Data retention is another factor to consider. Some developers retain chats indefinitely, and even ‘incognito’ or ‘temporary’ chats are likely to be retained for at least 30 days. Some accounts will have a zero data retention policy — PC Tech Magazine has a breakdown of different providers’ policies and processes for this.

Your chats are more likely to be selected for reinforcement training — and retained for longer — if they are ‘flagged’ for training for some reason. The most obvious reason that a chat might be flagged is for safety reasons, or because you have given feedback on a response in some way.

Chats can be flagged for training even if you have opted out of training on a free account, as a recent update to Anthropic’s privacy policy revealed:

“Conversations that Anthropic’s systems flag for safety review can still be used to train its models, regardless of a user’s stated preference. Anthropic’s policy does not define what triggers a safety flag, nor does it commit to notifying users when one occurs.”

The less information about an entity, the higher the risk

If your information was used to train a large language model — either through pretraining or reinforcement training — there is still no guarantee that it will find its way into a response to a user’s prompt.

What will make this more likely, however, is if your information relates to an entity or a question that the LLM ‘knows’ less about (i.e. has less training data).

Here are two scenarios to illustrate:

In Scenario A, a journalist uploads a sensitive document about Microsoft’s hiring policies to Gemini and it ends up being part of Gemini’s training data. All the ‘tokens’ (words and parts of words) from that document are already in its training, and this document changes the strengths of their relationships slightly so some have a stronger relationship with tokens like ‘Microsoft’ or concepts like ‘bias’.

At various points Gemini users will ask a question about Microsoft’s hiring policies. Because the sensitive document is one of many documents about Microsoft and hiring that Gemini has been trained on, its information is very unlikely to make a meaningful difference to most responses.

In Scenario B, on the other hand, a journalist uploads a list of the names of victims of a known predator. In this case, some of the names are quite unique and Gemini has either very little or no training data related to those. As a result the information in the document (the relationships between words) will be much more heavily weighted. Instead of being a drop in this particular ocean, the document constitutes a much larger part of it.

Now when users ask a question about one of those people, Gemini has much less training to draw on — and is therefore more likely to replicate the particular patterns of words (the ‘fact pattern’) in the sensitive document than other patterns it might predict.

Remember, however: it is still not retrieving a sentence from a document — it is only predicting one based on mathematical relationships.

Notably, if they asked about the known predator in general the chances of those names being generated by the AI tool is lower, because there are more, and stronger, connections to other facts, which might be more heavily weighted (depending on the question).

The complicating factor, of course, is that the user has no way of knowing whether the ‘fact’ in either scenario is true. When asked questions where very little information exists, AI tools are much more likely to hallucinate (hence court cases where AI tools have defamed individuals). And because the sensitive material is not online in either scenario, the chatbot cannot point to it as a source for the user to check with. The user has the suggestion of a fact, but not the proof.

When the risk calculation is different: sharing chats, internal access, hacks and law

Outside of user interactions with the AI tool, there are a number of other risks to consider.

The first is a simple one: if you choose to share the chat at any point this creates a risk that it becomes public.

Last year an investigation by Henk Van Ess‘s Digital Digging newsletter uncovered over 100,000 ChatGPT conversations which were findable with advanced search techniques. Last month he found another 90,000 shared conversations from multiple chatbots. The problem isn’t the AI tool — it’s the user sharing their chat. As Henk writes: “It looks like people thought “share” means send. But it means publish.”

It’s also worth mentioning that some engineering, support, and safety/abuse teams at AI companies may have authorisation to access AI conversations. As with reinforcement training, this is more likely if content is flagged for specific reasons, to provide account support, or for legal reasons, and in the context of millions of prompts each day. As with any online service, there is also the potential for an employee to misuse their access.

The same security risks that apply to online behaviour in general apply here, too: if you are using unsecured or public wifi to access an AI tool, it could be intercepted. The provider’s servers could be hacked or breached, as could your account with the provider if, for example, your password is weak. Your employer’s network and your own computer may also be hacked or accessed.

A more specific security risk is extraction attacks: these are specific attempts to extract training data from large language models (documented in literature listed here).

Finally, there are legal orders, which may be more important to consider than technical risks. The most famous of these is the court ruling that forced OpenAI to give millions of user chats to news organisations in a copyright case, but chats are increasingly showing up as evidence in criminal and civil cases, with one legal website advising “to prepare for a steady increase of government and litigation requests for Gen AI data”. In the US, court rulings have indicated that chats with AI tools “were not protected by attorney-client privilege or the work product doctrine” (see Munir v Secretary of State for the Home Department for a similar point in the UK).

Some of these requests will be specific to a particular user, but a warrant in October 2025 revealed that they can also relate to a particular prompt: what is known as “reverse” search warrants.

What can you do about it? Anonymisation and switching to local models

One way to reduce the risk involved in uploading sensitive information to an AI chatbot is to anonymise it. This involves either removing information that identifies individuals, or replacing it with unique IDs that you can use to re-identify them outside of the chat.

It’s not just names that might identify individuals: any collection of details might be connected with an individual (jigsaw identification) — remember that it’s these relationships that large language models are designed to calculate. Some technical solutions and libraries already exist for this.

It’s also worth considering whether uploading the information is necessary at all: just providing a description of the document or data may be enough to get a useful response, especially if you are focusing on a process (journey prompting) rather than an end result.

If the risk of using an AI service is too high, another option is to use a large language model locally — i.e. on your own computer rather than a server belonging to OpenAI, Google or another provider.

A number of tools make it relatively easy to run AI models locally, including LM Studio, AnythingLLM and Ollama. These allow you to choose from a range of models, download them to your computer, (make sure you use the local models as some also offer ‘cloud’ models), and then run prompts against them.

You will need a computer with enough processing power and RAM (CanIRun.ai provides a breakdown of which models you may be able to run based on detecting your computer’s specs), and the larger the model, the more power you will need. Prompts will typically run more slowly than on mainstream AI chatbots, too.

Make a threat model

A useful process to map all this out is to create a threat model that asks four simple questions:

1. What information do you not want other people to know? (This can be anything from passwords to contacts’ details, data and documents)
2. Why might someone want that information? Who?
3. What can they do to get it?
4. What might happen if they do?

Using the information in this post to map out realistic threats will make any subsequent steps much more manageable — and less anxiety-provoking.

This is a work in progress. Contributions, suggestions and updates are welcome.

Thanks to Laura Isotalo who raised a question around security and suggested writing about it after my talk on FOI management with AI at Dataharvest this year.