AI Adoption Pushes Agencies to Rethink Infrastructure, Costs

AI is moving beyond chatbots into enterprise workflows and autonomous systems, forcing government agencies to rethink how they build infrastructure, manage costs and maintain accountability.  

As AI tools take on more complex tasks, technology leaders said last week at GovCIO Media & Research’s Federal Cloud & Data Forum that organizations must prioritize flexible architectures, stronger safeguardsand meaningful human oversight. 

Joe Croghan, associate director of informatics in the Office of Cyber Infrastructure and Computational Biology at NIAID, said his agency is focused on creating a shared software platform that enables developers to integrate AI into enterprise systems, databases and workflows. Rather than competing with commercial providers on chatbot capabilities, the agency is concentrating on a flexible platform can help agencies keep pace with rapidly changing technology.  

“One of the challenges we have is that things are changing so fast. This platform environment helps us pivot for that. If we let people build applications from the ground up, usually by the time they’ve built the application, it’s obsolete,” he said. “By doing this … we can build applications quickly and we can access enterprise resources, and as we’re moving into agentic solutions we can start to tie some of these solutions together.” 

Meanwhile, growing AI usage is driving financial pressure by way of computing demand, said Jeff Malec, principal technologist at HP. Rising consumption could change how cloud and software providers charge customers, making it harder for agencies to predict expenses. 

To manage those costs, organizations are considering moving some workloads away from cloud-based services and onto local devices and infrastructure. 

“We’re starting to see how do we architect that or shift that, especially some of the light work stuff, onto the device using the [neural processing unit] in the EliteBook or the laptop. How do I build my own local models that run on local servers or local workstations so I’m not spinning that meter constantly? And that’s the concern. That’s sort of the new question around AI is, ‘How do I architect for the future so I’m not breaking the bank?’” Malec said.  

But controlling costs and building flexible infrastructure are only part of the challenge. As agencies give AI greater autonomy, they must also ensure the technology operates securely, meets existing compliance requirements and remains accountable when decisions go wrong. Kevin Walsh, director of IT and cybersecurity at GAO, said agencies should not treat AI as an entirely separate compliance category. 

“You still have to ensure your users are not putting personally identifiable information on the internet. They should be appropriate stewards. They need to do all their cybersecurity training and make sure that cyber is checked. They need to do their same job regardless of what tool they are using,” Walsh said. “Don’t build a separate silo, but you have to think about the challenges that AI is introducing that may be different.” 

He warned, however, that autonomous systems create additional challenges when they begin making consequential decisions rather than simply generating responses. 

Croghan summarized the concern as a question of whether AI is doing the right task, not just performing it correctly. As organizations delegate more decisions to AI, he said, they must ensure the systems’ broader objectives reflect the outcomes they actually want.