DOW CDAO: Agencies Enter ‘Top of the Third Inning’ on AI’s Value

National security agencies may still be in their salad days when it comes to capitalizing on artificial intelligence, but growing operational demands could soon make the technology essential.

As a result, Intelligence Community agencies are weighing not only where AI can deliver the most value, but also how to scale those applications and balance human-in-the-loop explainability with the speed automation provides.

War Department Chief Digital & Artificial Intelligence Officer Cameron Stanley, during an Aug. 29 panel at the AFCEA/INSA Intelligence and National Security Summit, said agencies are currently in the “bottom of the second, top of the third inning” when it comes capitalizing on AI’s value, but real-world gains are already at play.

“By doctrine, for example, the Fires Centers of Excellence in Fort Sill still teaches that a corps-level fires support coordination cell can execute about 13-14 targets a day. In Operation Epic Fury, we were executing more than 1,000 a day,” said Stanley about AI-assisted operations in Iran. “That wasn’t done by following doctrine, that was done be completely reimagining the approach using digital tools and then optimizing the decision-making process to incorporate those tools in a very dynamic way.”

But while AI can accelerate and optimize processes, onboarding capabilities and designing the processes in which they will be used can be nuanced and vary by mission, with several considerations to weigh. As a result, agencies have to be deliberate in their approach.

Where’s the current value?

AI’s potential is not yet fully realized in the War Department, but officials are seeing early gains in areas such as data synthesis and generative capabilities, while more advanced agentic AI solutions are still being developed.

“At least in our experience, with intelligence, it’s the aggregation of AI,” said Maj. Gen. Robert Kinney, military and mobilization assistant to the director at the Defense Intelligence Agency. “We’re in the very early stages I think, not just the agency, but I think the federal government as it relates to truly mature capabilities and what is in the realm of the possible.”

Kinney said there is currently significant interest in the potential of agentic AI systems that can act autonomously, but laying the groundwork for them will require DIA to continue building the technology ecosystem needed to support functions such as testing, evaluation, verification and continuous monitoring of AI agents.

The agency is currently working through a 90-day sprint to stand up an enterprise AI platform it announced this month. The platform will include deliverables such as a retrieval-augmented generation (RAG) capability to help users source internal information, an internal gateway to deploy and manage AI tools dubbed ChatDIA Gateway, an open-source Model Context Protocol (MCP) connection to some of its legacy datasets and a collection management capability.

Kinney said the platform will allow DIA to query large language models, RAG capabilities and legacy datasets and aggregate the results to better support users and analysts. It will also provide an ecosystem for onboarding new capabilities.

The Black Box Problem

Alongside AI’s speed and potential gains come inevitable questions about where humans and AI meet in decision-making — the so-called human-in-the-loop.

Those concerns tend to center on the potential tradeoff between the speed and scale AI systems provide and the need for users to understand how models reached their conclusions.

“The issue you are getting at is what we call the black box problem with neural nets and the like,” Kinney said. “That’s something we look very seriously at DIA, and it’s held us back, if you will, and given us pause with certain capabilities that we’ve seen that have come in front of us.”

He said Office of the Director of National Intelligence directives ICD-203 and ICD-206 around analytic standards and sourcing requirements make clear that AI models should be able to explain how they reached their outcomes.

“In my mind, it’s absolutely unimpeachable that we have to be able explain where the information came from, how the model derived to the answers that it got,” Kinney said, adding that he didn’t envision a time when AI replaced human analysts, but rather assisted them.

Stanley, however, said some degree of automation will be unavoidable as the scale of operations grows beyond human capacity.

“You heard me bring up the statistic 13,000 targets in 38 days. Is intelligence ready to keep up with 1,000 targets having to be updated every single day using human-centered manual approaches? The answer is no, it’s not ready to do that. So when you are thinking about what intelligence is going to be asked to do operationally, automation is the only way forward,” he said.