NIST Launches ARIA to Redefine How AI Is Evaluated

The National Institute of Standards and Technology (NIST) is launching its Assessing Risks and Impacts of AI (ARIA) 0.1, a new evaluation environment designed to assess how AI systems behave when used by people in real-world settings.

Unlike previous NIST evaluations that focused on accuracy, bias or discrete technical capabilities, ARIA measures system behavior across real contexts and use cases, agency officials said.

“AI systems are extremely complex,” NIST’s Information Access Division Chief Mark Przybocki told GovCIO Media & Research in an interview. “And a single number to characterize the level of performance is often insufficient.”

Instead, ARIA uses a three‑tier structure — model testing, red‑teaming and field testing — to generate a multidimensional view of system behavior.

A Layered Approach

NIST publicly released its first ARIA evaluation plan last month, reflecting what Przybocki described as growing recognition that traditional benchmarks fail to capture the full range of AI risks.

According to the plan, each tier examines different aspects of robustness and risk. Model testing assesses whether a system performs as advertised. Red-teaming probes adversarial or malicious use. Field testing evaluates how systems behave when deployed in realistic scenarios with human users.

Przybocki said the layered approach is critical to identifying impacts developers may not anticipate.

“By incorporating human testers into AI evaluation,” he said, “Red‑teaming and field testing can help to reveal positive and negative impacts that are not known ahead of time or that relate to AI use in its operating environment.”

Measuring ‘Technical and Contextual Robustness’

While existing evaluation frameworks often emphasize benchmark scores or accuracy, ARIA is designed to assess how AI systems perform across varied real-world conditions, according to the evaluation plan. In the pilot, NIST is focusing on large language models (LLMs).

“NIST focused on LLMs due to the immediate need to understand the wide variety of contexts in which they may be used across private industry and the public sector,” Przybocki said.

He added ARIA’s metrics differ from traditional evaluations in several ways. First, they combine expert annotation with human tester feedback. Second, they aggregate results into a multi‑scale tree structure that allows evaluators to zoom in on specific behaviors or zoom out for a high‑level score.

A Sector‑Agnostic Framework with Sector‑Specific Flexibility

Although ARIA 0.1 focuses on LLMs, the program is designed to be sector‑agnostic, Przybocki told GovCIO Media & Research. NIST officials say the goal is to build a framework across government services, critical infrastructure and commercial applications. Przybocki said the key is balancing generalizability with customization.

“NIST recognizes the importance of both generalizable and context‑specific evaluation approaches,” he said.

In the pilot, the team selected impacts that could materialize across multiple sectors, such as misinformation, harmful content or flawed reasoning. Future iterations will allow agencies or industries to tailor robustness scores to their own risk tolerances.

The evaluation plan emphasizes that ARIA is not a certification program. Instead, it will help organizations understand how AI systems behave under varied conditions and to inform future standards and best practices.

Red‑Teaming as a Tool for Uncovering the Unknown

Red-teaming is a central component of ARIA’s design. While model testing verifies stated capabilities and field testing examines everyday use, red-teaming is intended to uncover unanticipated behaviors.

“Such findings can inform future evaluations which focus on specific previously unanticipated impacts,” he said.

Why Now

The ARIA program arrives at a moment when federal agencies, private companies, and international bodies are all grappling with how to evaluate AI systems that increasingly shape public life. Traditional benchmarks — often built around static datasets — struggle to capture the dynamic, context‑dependent risks of generative AI. Przybocki said that ARIA is meant to fill that gap.

“ARIA seeks to improve AI evaluation by accounting for these varied contexts via realistic scenarios and multiple testers,” he said.