Challenges in data and AI deployment

Most AI projects in government today layer a large language model (LLM) on top of existing legacy datasets. This approach can deliver some useful outcomes. For example, an AI model can sift through hours of Hansard transcripts — the official record of parliamentary debates — and instantly summarise discussions, or scan government policy documents to pinpoint where a particular issue has been raised. These examples demonstrate AI’s potential to cut through information overload and support faster decision-making.

However, these use cases are limited and struggle when applied to large, disparate, and unstructured datasets spread across multiple departments. The underlying challenge remains – fragmented, incomplete, and siloed data.

Without consistent data foundations, these proofs of concept cannot be scaled or integrated into critical public services. The infrastructure required to process and analyse this data – from cloud platforms and high-performance computing systems to modern networks and storage – must also evolve with growing demand and complexity.

The UK is also facing a skills gap, with a shortage of public sector employees trained to manage, interpret, and apply AI responsibly. Valuable AI talent is leaving the UK for roles overseas, drawn by more advanced facilities and higher salaries.

A refreshed National Data Strategy

A refreshed National Data Strategy can offer a roadmap for overcoming these challenges and turning AI pilots into scalable, productive outcomes.

It should look to establish clear standards for data quality, interoperability, and accessibility across government. This requires not only cleansing and standardising legacy datasets but also creating governance models that incentivise collaboration rather than siloed pilots.

Crucially, this must be supported by a structured approach — from auditing what data exists, to consolidating and de-duplicating it, to migrating it onto modern platforms where it can be securely managed and accessed. Furthermore, governance frameworks should enable safe data sharing between departments, allowing citizens to be treated as whole individuals rather than fragmented records scattered across multiple systems.

The government can further accelerate progress by investing in skills and culture, equipping staff with AI capabilities and fostering a mindset that embraces digital transformation. Cross-department collaboration should be encouraged, with frameworks and incentives to share data and undertake joint initiatives, reducing silos and maximising the value of AI across services.

Public sector data often involves sensitive personal or national information, making robust security guardrails non-negotiable. A refreshed strategy should address not only how data is stored and shared, but also how it is protected against increasingly complex threats. By embedding strong data sovereignty and security principles into every stage of AI deployment, the government can give public sector organisations the confidence to embrace AI while managing risk effectively.

Embedding AI into everyday public services

AI has the potential to transform public services, improving efficiency, decision-making, and citizen outcomes across healthcare, infrastructure, and everyday government operations. Realising this potential will require more than pilots; it will take a coordinated effort to embed AI into the fabric of public sector work.

By investing in workforce skills, modernised infrastructure, cross-department collaboration, and strong governance, the UK can move from isolated experiments to AI solutions that scale, deliver tangible benefits, and maintain public trust.

Matt Harris is managing director for UK, Ireland, Middle East and Africa at HPE.