Defense Officials Highlight ‘Problem-First’ Approach to AI Development

Defense officials are sharpening data practices as artificial intelligence gains traction at the War Department, aiming to make systems more reliable and mission-ready for real-world scenarios, leaders said Thursday at GovCIO Media & Research’s Defense IT Summit in Arlington, Virginia.

Problems, Not Technology, First

The Marine Corps is avoiding a technology-first approach for developing and acquiring AI systems, explained Maj. Christopher Clark, AI lead for the Deputy Commandant for Information at the Marine Corps Service Data Office.

Clark described a technology-first mindset as “trying to pigeonhole everything into an AI hole that may not fit,” often creating unnecessary complexity and risk.

Instead, the service is taking a problem-first approach: clearly defining the operational issue, then working step by step through potential solutions — from simple, non-AI methods to machine learning and deep learning techniques. That progression helps avoid many of the pitfalls that derail AI initiatives, Clark said.

Another challenge is using data “that may not represent the real world.” Clark noted that synthetic data used in laboratory environments used to train prototype AI models is very clean, structured and organized. However, when a system is deployed, they often do not function well because real data is often messy, varied and constantly changing. He explained that the core of the issue is that many systems are not trained for real-world scenarios.

Building systems with the actual data they will use is challenging, “but doing that is an effective approach for making sure that the system you push forward is actually going to be the system you want to employ,” Clark said.

Making Training Data More Reliable

Many core AI technologies — including entity resolution, natural language processing and computer vision — have existed for decades, said Srini Iyer, senior vice president and chief technology officer for health and services at Leidos. What has changed is users’ ability to interact directly with systems to build agents and applications.

In recent years, Iyer said, AI pilots have consistently faced three major challenges: data, security and validation.

“Data is critical to how we go about doing this,” Iyer said. He noted that pilot projects often work in a sandbox environment limiting the kinds of data than can be accessed. This is often synthetic data “that doesn’t truly represent the real world.”

Security is equally critical. Without securing data in real-world operational environments, AI capabilities cannot scale effectively, Iyer said.

Validation presents an additional layer of complexity. Because many AI models are non-deterministic, their outputs can drift over time as data changes. That raises a fundamental question: “How do you validate models as you go through that?” Iyer said.

Iyer added that the War Department and researchers are still refining best practices for validating and operationalizing AI systems at scale.

Iteration and Speed

Synthetic data is an important step in training prototype AI systems, said Guy Kiyokawa, vice president and account manager for Leidos’ War Department Personnel Readiness and Military Health System. But as prototypes move into pilot phases, ingesting real operational data becomes critical to validating performance and mission utility.

Speed is equally important.

“We’ve got all these new acquisition tools that are coming out, and it’s pushing speed. So how do we do that quickly?” said Kiyokawa.

That requires building synthetic environments that closely mirror operational conditions and engaging end users early and often.

“When you start off with a problem set, you need validation of the problem set,” said Kiyokawa.

That validation extends to workflows, proofs of concept and mission needs. He noted that the DOW is more open to early engagement with industry partners, creating opportunities to refine problem sets and operational workflows before full-scale deployment.

Moving AI From the Lab to Warfighters

The Defense Health Agency’s Appraise-HRI app is one success story of an AI-based system moving from the lab to operational use.

It is the first software as a medical device to be cleared with the Food and Drug Administration, said Jaques Reifman a senior research scientist with the DHA’s Medical Research and Development Command. Appraise-HRI is an AI-enabled decision support system designed to help medics triage combat casualties for hemorrhage risk.

Appraise-HRI is a major development because hemorrhage is the leading cause of preventable battlefield fatalities, accounting for more than 90% of personnel who die before reaching a medical treatment facility.

The software resides on a medic’s smartphone where it wirelessly communicates with a vital signs monitor collecting heart rate and blood pressure data. The system looks for patterns in vital sign data and within a minute can determine if there is a low or high risk of hemorrhage. This gives medics the opportunity to prioritize care and evacuation procedures for that individual.

FDA approval also means the software can be used by device manufacturers. This allows vendors to integrate Appraise-HRI into devices such as vital sign monitors without having to demonstrate that the device is both safe and effective. They only have to demonstrate that they integrated the software correctly.

“By clearing Appraise with the FDA, we have de-risked the technology,” Reifman said.