NCSC warns of confusion over true nature of AI prompt injection
Malicious prompt injections to manipulate GenAI large language models are being wrongly compared to classical SQL injection attacks. In reality, prompt injection may be a far worse problem, says the UK’s NCSC.
The UK’s National Cyber Security Centre (NCSC) has highlighted a potentially dangerous misunderstanding surrounding emergent prompt injection attacks against generative artificial intelligence (AI) applications, warning that many users are comparing them to more classical structured query language (SQL) injection attacks, and in doing so, putting their IT systems at risk of compromise.
While they share similar terminology, prompt injection attacks are categorically not the same as SQL injection attacks, said the NCSC in an advisory blog published on 8 December. Indeed, said the GCHQ-backed agency, prompt injection attacks may be much worse, and harder to counteract.
“Contrary to first impressions, prompt injection attacks against generative artificial intelligence applications may never be totally mitigated in the way SQL injection attacks can be,” wrote the NCSC’s research team.
In their most basic form, prompt injection attacks are cyber attacks against large language models (LLMs) in which threat actors take advantage of ability such models to respond to natural language queries and manipulate them into producing undesirable outcomes – for examply, leaking confidential data, creating disinformation, or potentially guiding on the creation of malicious phishing emails or malware.
SQL injection attacks, on the other hand, are a class of vulnerability that enable threat actors to mess with an application’s database queries by inserting their own SQL code into an entry field, giving them the ability to execute malicious commands to, for example, steal or destroy data, conduct denial of service (DoS) attacks, and in some cases even to enable arbitrary code execution.
SQL injection attacks have been around a long time and are very well understood. They are also relatively simple to address, with most mitigations enforcing a separation between instructions and sensitive data; the use of parameterised queries in SQL, for example, means that whatever the input may be, the database engine cannot interpret it as an instruction.
While prompt injection is conceptually similar, the NCSC believes defenders may be at risk of slipping up because LLMs are not able to distinguish between what is an instruction and what is data.
“When you provide an LLM prompt, it doesn’t understand the text it in the way a person does. It is simply predicting the most likely next token from the text so far,” explained the NCSC team.
“As there is no inherent distinction between ‘data’ and ‘instruction’, it’s very possible that prompt injection attacks may never be totally mitigated in the way that SQL injection attacks can be.”
The agency is warning that unless this spreading misconception is addressed in short order, organisations risk becoming data breach victims at a scale unseen since SQL injection attacks were widespread 10 to 15 years ago, and probably exceeding that.
It further warned that many attempts to mitigate prompt injection – although well-intentioned – in reality do little more than try to overlay the concepts of instructions and data on a technology that can’t tell them apart.
Should we stop using LLMs?
Most objective authorities on the subject concur that the only way to avoid prompt injection attacks is to stop using LLMs altogether, but since this is now no longer really possible, the NCSC is now calling for efforts to turn to reducing the risk and impact of prompt injection within the AI supply chain.
It called for AI system designers, builders and operators to acknowledge that LLM systems are “inherently confusable” and account for manageable variables during the design and build process.
It laid out four steps that taken together, may help alleviate some of the risks associated with prompt injection attacks.
- First, and most fundamentally, developers building LLMs need to be aware of prompt injection as an attack vector, as it is not yet well-understood. Awareness also needs to be spread across organisations adopting or working with LLMs, while security pros and risk owners need to incorporate prompt injection attacks into their risk management strategies.
- It goes without saying that LLMs should be secure by design, but particular attention should be paid to hammering home the fact LLMs are inherently confusable, especially if systems are calling tools or using APIs based on their output. A securely-designed LLM should focus on deterministic safeguards to constrain an LLM’s actions rather than just trying to stop malicious content from reaching it. The NCSC also highlighted the need to apply principles of least privilege to LLMs – they cannot have any more privileges than the party/ies interacting with them does.
- It is possible to make it somewhat harder for LLMs to act on instructions that may be included within data fed to them – researchers at Microsoft, for example, found that using different techniques to mark data as separate to instructions can make prompt injection harder. However, at the same time it is important to be wary of approaches such as deny-listing or blocking phrases such as ‘ignoring previous instructions, do Y’, which are completely ineffective because there are so many possible ways for a human to rephrase that prompt, and to be extremely sceptical of any technology supplier that claims it can stop prompt injection outright.
- Finally, as part of the design process, organisations should understand both how their LLMs might be corrupted and the goals an attacker might try to achieve, and what normal operations look like. This means organisations should be logging plenty of data – up to and even including saving the full input and output of the LLM – and any tool use or API calls. Live monitoring to respond to failed tool or API calls is essential, as detecting these could, said the NCSC, be a sign a threat actor is honing their cyber attack.
