Federal Experts Urge Rigorous Testing Before Deploying AI Tools

AI-powered tools in government and commercial enterprises enable federal agencies to work more efficiently and be more secure, but they also open new vectors for threats.

Federal agencies are navigating this changing environment by setting firm guardrails and governance to ensure AI performs as intended, government and industry experts said at the Carahsoft DevSecOps Conference in Reston, Virginia Tuesday.

AI presents several challenges from a governance and control standpoint, said Anil Chaudhry, senior advisor of AI at the Transportation Department. While external AI-based threats are important, managing enterprise AI systems for security and governance present their own challenges.

These new tools require CIOs to change their approach to practices such as DevSecOps, Chaudhry said.

“DevSecOps is important — it’s foundational. It needs to happen. It’s a proven model. But what you need to realize is, in the age of AI, you have to think about the way stuff gets developed. It’s no longer deterministic models. It’s no longer static script-based orchestration infrastructure as code with third-party library calls. It’s no longer APIs. It’s just no longer linear scans,” he said.

Focusing development on large language models and vibe coding is probabilistic, dynamic, goal-oriented decision-making, meaning that applications agencies deploy “are not a set of instructions anymore,” Chaudhry said. Instead, they are outcomes developers want to happen. However, how an agency’s mix of AI agents and programming orchestration work together may not be readily apparent until something goes wrong, he said.

“What you need to start thinking about is, what do your goals look like? Are your goals and the instructions that you have set out for your application clear enough?” Chaudhry said.

USDA’s Approach to AI and Zero Trust

The Agriculture Department contains a wide variety of data supporting farmers, ranchers, producers, financial and international users. This also includes confidential information such as agricultural forecasts that can affect how commodities such as grain are sold and purchased. This presents a tempting target for hackers, said Rudolf Rojas, USDA’s IT program manager and security lead.

To counter these threats, especially as the agency moves much of its data, applications and software to the cloud, USDA is turning to AI cybersecurity tools and employing zero-trust principles. Rojas added that zero trust is important for network security, allowing for continuous monitoring and continuous integration to ensure data safety.

Because AI tools are becoming more powerful and capable, it is important to set and maintain guardrails. An important part of this is keeping humans in the decision loop.

“That’s where zero trust comes in to make sure that we’ve vetted the proper uses of those tools and the right people are getting into it, and then using tools to continually scan and monitor applications and tools and platforms. So keeping the human in the loop is going to be key,” Rojas said.

Evaluating AI Tools

There are AI security tools available on the market with impressive benchmarks, but agencies need to accurately evaluate them to ensure they meet their needs, noted Nihal Singh, a software engineer with Moody’s Corporation.

Agencies should query vendors on how and what they test for and if the statistical outcomes of those tests vary under more accurate and rigorous benchmarks.

“My advice to agencies would be to ask the vendors the correct questions. Did they do the temporal drift testing, did they evaluate the statistical significance, and what split did they use for train and test of those models because that is more important? Once those questions are correctly answered by vendors, then I feel like we would be more clearly confident in deploying those tools or the stack and the production,” Singh said.

Security as Narrative

AI tools becoming more pervasive in agencies makes communication between departments critical, said Matthew Graviss, public sector CTO at Altassian. He noted that one thing many organizations lack is communicating the “why” around security to all employees.

Graviss was previously at the State Department where his group was responsible for initiating and conducting impact assessments for every cyber incident. He noted that the department’s CIO was good at driving communication between developers and end users about any cyber instances they encountered and how that affected their missions.

This approach meant that security remained entirely the purview of the developer community or as a box to check or gate to pass. Another facet is storytelling about why security matters to help align goals across an organization. This helps to focus priorities and strategy.

“Making sure that your strategy and the metrics associated with that are tied to the actual work that’s getting done, whether it’s the operations or the modernization or whatnot, I think that’s really important,” Graviss said.