AI privacy risks put users and small firms on notice

UK regulators and privacy specialists are urging individuals and small businesses to rethink how they use generative artificial intelligence tools, warning that routine prompts can expose personal data to retention, reuse and overseas transfer in ways that may breach data protection law.

The Information Commissioner’s Office, the UK’s data protection watchdog, has said that text, images and files entered into generative AI systems may constitute personal data and are therefore subject to the UK General Data Protection Regulation, even when users believe the information is trivial or anonymised.

ICO guidance

“Personal data is any information relating to an identified or identifiable individual,” the ICO said in recent guidance. “This can include names and contact details, but also online identifiers, free text responses, images, audio and opinions where a person can be identified directly or indirectly.”

The warning comes as generative AI tools are rapidly adopted by home users, sole traders and small companies for tasks such as drafting emails, producing marketing copy, summarising documents and responding to customer queries. Many users remain unclear about how their inputs are processed or whether they are stored or used to train future models.

Unlike traditional software, generative AI tools often rely on cloud-based systems that process data outside the user’s device. In many cases, providers reserve the right to retain prompts and outputs for quality assurance, abuse monitoring or model improvement, unless users take explicit steps to opt out or subscribe to enterprise services.

Data controllers

The ICO has stressed that where personal data is involved, responsibility does not sit solely with the AI provider. Users who determine why and how data is processed may themselves be acting as data controllers under UK law.

“Organisations using generative AI must understand their role and responsibilities,” the regulator said. “This includes identifying a lawful basis for processing personal data, ensuring data minimisation, setting appropriate retention periods and addressing international data transfer risks.”

For individuals using AI tools at home, the implications may appear remote. However, privacy experts say even casual use can involve personal data. Asking a chatbot to rewrite a complaint email that includes a full name, address and account number, or uploading a family photograph to generate a stylised image, may expose personal or biometric data to third-party systems.

Privacy applies to everyone

“People often think privacy law only applies to companies,” said Madeleine Stone, a UK-based data protection consultant. “But when you input someone else’s personal data into an AI system, you are still disclosing it. If that data is retained or reused, you have lost control over it.”

The risks become more acute for small businesses, which often lack dedicated legal or information security teams but increasingly rely on AI tools to save time and cut costs. Customer service responses, sales emails and internal reports are frequently drafted with the assistance of chatbots, sometimes using real customer data.

In one common scenario, a small retailer might paste a customer complaint into an AI tool and ask for a polite response. That complaint may include the customer’s name, email address, order history and details of a dispute. Unless the data has been removed or anonymised, the entire prompt may be stored by the provider.

Specific purposes

Under UK data protection law, personal data must be processed lawfully, fairly and transparently. Organisations must have a lawful basis for processing, such as consent, performance of a contract or legitimate interests. They must also ensure that only the minimum necessary data is used for a specific purpose.

The ICO has repeatedly emphasised the principle of data minimisation. “You should only process the personal data you need,” it said. “If you can achieve your purpose without personal data, you should do so.”

Retention is another area of concern. Some AI providers state that user inputs may be stored for a defined period, while others reserve the right to retain data indefinitely unless users opt out. From a UK compliance perspective, organisations must not keep personal data longer than necessary for the purpose for which it was collected.

International data transfers present a further complication. Many generative AI services are developed and operated by companies based outside the UK, with data processed or stored in multiple jurisdictions. UK law requires that personal data transferred abroad is protected by appropriate safeguards, such as an adequacy decision or standard contractual clauses.

Where does the data go?

“Users should be asking where their data goes,” the ICO said. “If personal data is transferred outside the UK, you must ensure that appropriate protections are in place.”

Large technology companies have responded to regulatory pressure by offering enterprise-grade AI services that limit or exclude the use of customer data for training purposes. These services often include contractual commitments on data handling, retention and security, as well as options for data residency.

However, such protections are typically not available to users of free or consumer-grade tools. Privacy notices for these services often permit the use of prompts and outputs to improve the underlying models.

In a public statement, OpenAI has said: “We may use content such as prompts and responses to improve our services. Enterprise customers are excluded from this by default.”

Similar distinction

Microsoft and Google have made similar distinctions between consumer and business offerings in their AI products, according to their published terms.

For small firms, the choice of tool can therefore have legal consequences. “If you are using a free AI service for business purposes, you need to read the terms very carefully,” said James Barrett, a solicitor specialising in technology law. “You may be disclosing customer data to a third party in a way that your privacy notice does not cover.”

Transparency is another requirement under UK law. Organisations must inform individuals about how their personal data is used, including whether it is shared with third parties or processed using automated tools. Failing to disclose the use of generative AI in handling customer data could expose businesses to complaints or enforcement action.

The ICO has not announced any fines specifically related to generative AI prompts, but it has signalled that misuse of personal data in AI systems falls within its existing enforcement powers.

Governance and training

For teams within larger organisations, the focus has shifted towards governance and staff training. Many companies now restrict the use of public AI tools or require employees to use approved platforms that meet internal security standards.

“Most data breaches involving AI are not technical,” said Stone. “They are behavioural. Someone copies and pastes something they should not.”

The ICO has encouraged organisations to provide clear guidance to staff on what can and cannot be shared with AI systems. This includes prohibiting the entry of customer lists, login credentials, financial details or health information into external tools.

Even where data appears anonymised, risks remain. Combinations of details can re-identify individuals, particularly in small datasets or niche contexts. A description of a role, location and incident may be enough to identify a person internally or externally.

The regulator has also warned against assuming that AI systems automatically anonymise inputs. “You should not rely on the tool to remove personal data,” it said. “Responsibility rests with the user.”

Coordinating guidance

The ICO is continuing to develop its broader approach to artificial intelligence. It is working with other UK regulators through the Digital Regulation Cooperation Forum to align guidance across privacy, competition and online safety. Further consultation on AI and data protection is expected next year.

In the meantime, the message to users is one of caution rather than prohibition. The regulator has said it supports innovation and recognises the productivity benefits of AI, but only where data protection principles are respected.

“Generative AI can be used responsibly,” the ICO said. “But organisations and individuals must understand how personal data is processed and take steps to protect people’s rights.”

Update policies

For home users, that may mean keeping prompts generic and adding personal details manually after the fact. For small businesses, it may involve updating internal policies, reviewing supplier terms and ensuring that staff understand the risks. For larger teams, it increasingly requires structured controls over how AI tools are deployed.

As generative AI becomes embedded in everyday work and personal life, privacy specialists say awareness will be critical. “AI feels conversational and informal,” said Barrett. “But legally, it is still data processing. Treat it with the same care you would any other system.”

The ICO’s advice is blunt. If you would not post the information on a public website, do not paste it into an AI tool unless you are certain how it will be used.