Confidential data

AI tools have also leaked confidential data, either directly or because employees have uploaded confidential documents to an AI tool.

Then there is bias. The latest AI algorithms, especially in LLMs, are highly complex. This makes it difficult to understand exactly how an AI system has come to its conclusions. For an enterprise, this in turn makes it hard to explain or even justify what an AI tool, such as a chatbot, has done.

This creates a range of risks, especially for businesses in regulated industries and the public sector. Regulators rapidly update existing compliance frameworks to cover AI risks, on top of legislation such as the European Union’s (EU’s) AI Act.

Research by industry analyst Forrester identifies more than 20 new threats resulting from GenAI, some of which relate to security. These include a failure to use secure code to build AI systems, or malicious actors that tamper with AI models. Others, such as data leakage, data tampering and a lack of data integrity, risk causing regulatory failures even when a model is secure.

The situation is made worse by the growth of “shadow AI”, where employees use AI tools unofficially. “The most common deployments are likely to be those that enterprises aren’t even aware of,” warns James Bore, a consultant who works in security and compliance.

“This ranges from shadow IT in departments, to individuals feeding corporate data to AI to simplify their roles. Most companies haven’t fully considered compliance around AI, and even those who have, have limited controls to prevent misuse.”

This requires chief information officers (CIOs) and data officers to look at all the ways AI might be used across the business and put control measures in place.

AI’s source data issue

The first area for enterprises to control is how they use data with AI. This applies to model training, and to the inference, or production, phase of AI.

Enterprises should check they have the rights to use data for AI purposes. This includes copyright, especially for third-party data. Personal identifiable information used for AI is covered by the General Data Protection Regulation (GDPR) and industry regulations. Organisations should not assume existing data processing consent covers AI applications.

Then there’s the question of data quality. If an organisation uses poor-quality data to train a model, the results will be inaccurate or misleading.

This, in turn, creates compliance risk – and these risks might not be removed, even if an organisation uses anonymised data.

“Source data remains one of the most overlooked risk areas in enterprise AI, warns Ralf Lindenlaub, chief solutions officer at Sify Technologies, an IT and cloud services provider. “These practices fall short under UK GDPR and EU privacy laws,” he says. “There is also a false sense of security in anonymisation. Much of that data can be re-identified or carry systemic bias.

“Public data used in large language models from global tech providers frequently fails to meet European privacy standards. For AI to be truly reliable, organisations must carefully curate and control the datasets they use, especially when models may influence decisions that affect individuals or regulated outcomes.”

A further level of complexity comes with where AI models operate. Although interest in on-premise AI is growing, the most common LLMs are cloud-based. Firms need to check they have permission to move data to where their cloud suppliers store it.

AI outputs and compliance

A further set of compliance and regulatory issues applies to the outputs of AI models.

The most obvious risk is that confidential results from AI are leaked or stolen. And, as firms link their AI systems to internal documents or data sources, that risk increases.

There have been cases where AI users have exposed confidential information either maliciously or inadvertently through their prompts. One cause is using confidential data to train models, without proper safeguards.

Then there’s the risk the AI model’s output is simply wrong.

“AI outputs can appear confident but be entirely false, biased, or even privacy-violating,” warns Sify’s Lindenlaub. “Enterprises often underestimate how damaging a flawed result can be, from discriminatory hiring to incorrect legal or financial advice. Without rigorous validation and human oversight, these risks become operational liabilities.”

And the risk is greater still with “agentic” AI systems, where a number of models work together to run a business process. If the output from one model is wrong, or biased, that error will be compounded as it moves from agent to agent.

Regulatory consequences could be severe, as one erroneous output might result in numerous customers being refused credit or denied a job interview.

“The most obvious problem with outputs from AI is that they generate language, not information,” says James Bore. “Despite the way they’re presented, LLMs do not analyse, they do not have any understanding, or even weightings for fact versus fiction, except those built into them as they are trained.

“They hallucinate wildly, and worse, they do so in very convincing ways, since they are good at language,” he adds. “They can never be trusted without thorough fact-checking – and not by another LLM.”

Enterprises can, and do, use AI in a compliant way, but CIOs and chief digital officers need to give careful consideration to compliance risks in training, inference and how they use AI’s results.