AI-augmented cyber operations

Nation-state actors have been increasingly integrating GenAI into cyber attacks to enhance efficiency, automation, and precision. More than 57 advanced persistent threat (APT) groups linked to nation states have been observed using AI in cyber operations. AI can automate research, translate content, assist with coding, and develop malware to advance cyber operations.

One of the most concerning challenges is the use of AI in crafting highly convincing phishing messages, increasing both the pace and scale of cyber-attacks. Large language models (LLMs) can generate highly plausible messages, targeted to individuals and organisations. Criminals are deploying believable, personalised AI-generated deepfake videos, audio, and images to enhance social engineering campaigns. The case of Arup, the design and engineering firm, which lost $25 million as a result of a deepfake ‘CFO’, shows how convincing AI-enabled operations can gain meaningful access to companies.

Supply chain vulnerabilities

Beyond direct cyberattacks, threat actors are also targeting AI supply chains from hardware to software. The infamous SolarWinds Sunburst attack demonstrated how sophisticated nation state actors can infiltrate enterprise networks by targeting supply chains. The risk extends to AI software as well. By embedding vulnerabilities at the manufacturing or development stage, adversaries can target a broad range of adversaries, profiting from economies of scale.

Supply chain vulnerabilities are a key trend dominating cyber security. The Bureau of Industry and Security’s recent prohibition on the import and sale of hardware or software for connected vehicles from certain nations highlights the US’s growing concern. Malicious actors have targeted Python packages for LLMs like ChatGPT and Claude to deliver malware that can harvest browser data, screenshots and session tokens. Those procuring AI systems and their components need to consider both where the AI has come from and how users will interact with it.

AI governance and security frameworks

To defend against AI-augmented nation-state threats, security leaders must adopt a range of strategies including AI governance frameworks, targeted training, robust data protection measures, third-party risk management processes, and proactive threat intelligence.

AI frameworks aligning with best practice for governance – such as NIST AI RMF, ISO 42001 and MITRE, OWASP and NCSC for security – provide the basis to a structured defence. By establishing clear roles and accountabilities for AI, policies defining acceptable and unacceptable use, and robust approaches to monitoring and auditing, the framework can implement defences against exposing sensitive information.

The role of people and culture needs to change in response to AI risks. Training, starting with AI literacy to cover foundational AI awareness and its impact on security, can empower staff to spot, challenge, and mitigate AI cyber threats. An inventory of AI systems is a foundational part of AI governance. CISOs need to know where and how AI is being used across the enterprise, and technology companies need to know what and where their critical assets are.

Data protection measures

Data access controls can limit adversaries’ ability to exfiltrate proprietary secrets. Data segmentation to restrict AI models from processing sensitive data, privacy-enhancing technologies like encryption, and monitoring systems for unauthorised loss of corporate data make it harder for nation-states to extract valuable intelligence. Applying data protection principles like minimisation, purpose limitation, and storage limitation can further both security and responsible AI objectives.

Securing AI supply chains

Meanwhile, supply chain risk management prevents infiltration of compromised AI tools. Important steps include conducting security assessments for third-party AI vendors, ensuring that AI models do not rely on foreign-hosted APIs that could introduce vulnerabilities, and documenting software bills of materials (SBOMs) to track dependencies and detect risks.

AI-driven threat detection and response

Finally, AI itself can be a tool to defend against AI-powered threats. AI-driven anomaly detection can identify suspicious behaviour or data loss patterns, deploying adversarial AI to test enterprise AI systems for vulnerabilities, increase monitoring for AI-generated phishing, and assess the effectiveness of controls. As AI-enabled cyber attacks accelerate beyond human response capabilities, automated monitoring and defensive systems are necessary to prevent exploitation of vulnerabilities at machine speed.

Clearly, the rise of AI-powered nation state threats demands a proactive and strategic response from security leaders. By adopting AI governance frameworks, enforcing strict data governance, securing supply chains, and leveraging AI-driven threat detection, enterprises can strengthen their defences against industrial espionage.

Elisabeth Mackay is a cyber security expert at PA Consulting