Federal Leaders Confront the Next Wave of AI Security Risks

The rapid rise of artificial intelligence across government is reshaping how agencies think about security and governance, according to a recent panel discussion at Zscaler’s Public Sector Summit.

Some of the key risks organizations face around AI adoption include areas like data poisoning, data exfiltration and prompt injection. One statistic from Zscaler’s ThreatLabz report noted that 70% or more of AI-generated code goes unchecked.

The threat is only increasing as AI adoption grows.

The same report noted a recent red-teaming exercise that found 90% of AI systems were compromised in under 90 minutes. Further, Zscaler’s cloud has blocked more ransomware attempts in the last year than in any year at 10.8 million — a 146% year over year increase.

Growing AI User Base

Panelists called out developers as one of the largest users of AI tools, yet there are still gaps between those users and governance policies. For many organizations, this means AI use cases are flying under the radar as shadow AI.

“Developers who are under a lot of pressure to deliver software and applications want to be productive, and what better way to be productive and to use AI in creative ways,” Zscaler Public Sector CTO Chad Tetreault told GovCIO Media & Research. “That’s one of the exciting things that will come out of this year is where you can prioritize, where to make investments based on pulling these things out of the dark, and really shining some light on what your workforce is doing with these tools.”

The experts suggested giving developers a seat at the table in the governance development process, “so that developer experience becomes part of the governance discussion,” said Robert Brown, senior vice president at Alpha Omega and former CTO at U.S. Citizenship and Immigration Services.

“There’s still a lot of teeth that need to be put into governance,” he added.

Brown called attention to some of the burgeoning solutions for organizations to explore like threat modeling, writing governance into procurement and also “MBOM,” or model bills of materials.

“We’ve all heard of ‘SBOM.’ I think we really need to adopt a hard and fast MBOM,” said Brown. “These are some of the areas that are still burgeoning and still need help.”

Rise of Red Teaming

Few organizations have systematic, continuous AI red‑teaming and model‑evaluation frameworks baked into their lifecycles. Red-teaming exercises continuously test models for exploitable weaknesses before an attacker can find them.

Panelists urged organizations to bake red-teaming into their policy compliance processes to keep up with the evolution of AI and machine-learning models.

“These models are evolving very quickly. … You have to change quickly, and you have to be able to reevaluate. So setting up that testing rubric now is what I’m instructing our team, showcasing for our account team, how to have those conversations with the customers and the partners earlier, rather than later,” said Kevin Meredith, federal AI technology strategist at AWS. “When you’re developing your AI product, you need to be able to understand which model best suits this use case.”

What Does the Future Look Like?

Looking ahead to 2027, panelists noted optimistic predictions about AI’s impact on government. They foresee AI quietly reshaping familiar processes like license renewals and passport applications.

“We’re not even [going to] notice … all these things that it took a lot of analysts weeks to do, [are] going to be done in hours,” said Meredith.

Perhaps the most immediate shift will be in education and literacy across the workforce.

“That education of understanding what AI is, it’s critical, and I think that will be probably one of the bigger measurable outputs from a year from today,” said Brown.