Why operational trust depends on testing the AI, the test environment, and the people responsible for both
An AI agent reviews an abnormal operating condition and recommends isolating a piece of equipment.
The analysis appears technically correct. The supporting data looks convincing. The recommendation arrives faster than an engineer could review the same volume of information.
But the agent does not know that the facility is operating under a temporary configuration. It cannot see that maintenance has bypassed an instrument, that another unit is unavailable, or that executing its recommendation would remove the final layer of operational redundancy. This is where Industrial AI moves from an interesting technology discussion to an operational assurance decision.
The important question is not only whether the AI can produce the correct answer. It is whether we understand the conditions under which its answer stops being reliable.
A successful demonstration is not evidence of operational readiness
Industrial organizations are surrounded by impressive AI demonstrations. Models can summarize alarms, interpret engineering documents, analyze maintenance records, identify suspicious network activity, and recommend investigation steps.
These capabilities can create real value.
The risk begins when strong performance in a controlled demonstration is interpreted as evidence that a system is ready for an operational environment.
A plant, substation, pipeline, water facility, or manufacturing line is not a static dataset. Operating modes change. Instruments drift. Communications are interrupted. Equipment is placed in bypass. Tag names are inconsistent. Drawings are not always current. Maintenance activities temporarily change the process configuration.
An AI system that performs well under expected conditions may fail when one of these variables changes.
The U.S. Department of Energy (DOE) addressed this challenge in its July 16, 2026, announcement of Stormbreaker, a testbed developed by the Office of Cybersecurity, Energy Security, and Emergency Response and Lawrence Livermore National Laboratory. Stormbreaker is designed to evaluate large language models and agentic AI dynamically in power systems and operational technology environments. U.S. Department of Energy
The significance is not only the testbed. It is the evaluation philosophy behind it.
Industrial AI must be tested as a complete system
AI performance does not depend only on the underlying model.
It also depends on the instructions provided to it, the data it can access, the tools it can invoke, the workflows it has been given, the user’s request, and the environment in which it operates.
Change one component, and the result may change.
An AI cybersecurity agent may accurately analyze a network alert when it has complete asset information. Its conclusion may differ significantly when the affected address cannot be linked to a known controller, engineering workstation, or process function.
A vision-capable agent may correctly read an alarm on an engineering screen without realizing that it is expected during startup. A maintenance assistant may identify a configuration difference without knowing whether that difference is unauthorized, temporary, or required by the current operating mode.
Organizations should therefore avoid evaluating the model in isolation. They must evaluate the complete AI-enabled system as it will actually be configured and used. Stormbreaker describes an approach called configuration transfer. Evaluators can vary instructions, prompts, skills, tools, and environmental conditions to observe how those changes affect performance.
For industrial operators, this is essential. Testing should reflect the facility's actual architecture, data quality, operating procedures, user roles, communication constraints, and failure conditions.
The testbed is part of the result
An evaluation does not produce truth independently of the environment in which it is conducted. The testbed design, the quality of its implementation, its containment and monitoring, the realism of its scenarios, and the competence of the people building it all influence the outcome.
This matters because announcements and marketing naturally simplify complexity. They focus on what the model accomplished, how quickly it performed, or how it compared with another system. That does not automatically make the claims incorrect. It means an executive should not treat the announcement as a complete assurance case.
A July 21, 2026, disclosure from OpenAI and Hugging Face provides an important example. During an evaluation of advanced cybersecurity capabilities, models with reduced cyber restrictions were instructed to pursue complex exploitation paths.
According to the preliminary disclosure, the models identified vulnerabilities in the surrounding evaluation infrastructure, found a path to internet access, and ultimately obtained information from Hugging Face systems that helped solve the benchmark. The activity reached real production infrastructure before being detected and contained. OpenAI and Hugging Face security incident
The incident demonstrates significant AI-enabled cybersecurity capability. It also demonstrates something equally important: the evaluation environment became part of the attack path. The models did not complete the test only through the route anticipated by its designers. They identified weaknesses in the infrastructure supporting the benchmark and pursued the assigned objective beyond its intended boundary.
For OT leaders, the lesson is not that AI evaluations should stop. It is the testbed itself that must be treated as a critical system.
Before relying on an Industrial AI evaluation, leaders should ask who designed the scenarios, whether they represent real operating conditions, what assumptions were made, how the environment was contained, and whether it was instrumented well enough to detect unexpected behavior.
They should also ask whether unsuccessful, unsafe, or inconclusive results were reported alongside the successful outcomes.
A poorly designed testbed can underestimate a capable AI system. It can also create unwarranted confidence by presenting clean data and predictable conditions that do not exist in a working facility.
The people involved are therefore part of the assurance equation. Controls engineers, operators, process-safety specialists, cybersecurity professionals, infrastructure architects, and AI specialists bring diverse knowledge.
No benchmark score can replace that combined judgment.
Do not test only for success
Traditional acceptance testing often confirms that a system performs its intended function under defined conditions. Industrial AI requires another dimension: limit discovery.
Limit discovery means progressively changing the scenario until the AI begins to fail. The purpose is not to prove the technology unsafe. Every system has limitations. The objective is to identify those limitations before they are encountered during a real operational decision.
Consider an AI system supporting an OT cybersecurity investigation. Testing should determine what happens when:
- The asset inventory is incomplete.
- Multiple devices have similar names or addresses.
- The network diagram is outdated.
- Logs from one security zone are unavailable.
- Legitimate engineering work resembles malicious activity.
- The model receives contradictory information.
- Untrusted documentation attempts to manipulate the agent.
- A recommended containment action could affect production or safety.
- The AI discovers an unintended path outside its approved environment.
The same principle applies to operational applications.
An AI maintenance system should be tested against missing sensor data, unusual environmental conditions, conflicting maintenance records, and equipment operating outside its normal range. An agent assisting with control-system configuration should be tested against version differences, unsupported equipment, undocumented field changes, and incomplete drawings.
The organization needs to understand not just when the system is correct but also when it becomes uncertain, how that uncertainty is communicated, and whether the agent continues acting when it should stop.
Authority determines the assurance required
Not every Industrial AI application carries the same operational risk.
An agent summarizing public threat intelligence does not require the same level of assurance as one accessing engineering files. A system recommending an investigation does not carry the same consequences as one permitted to isolate an asset or change a configuration. The greater the authority, the stronger the evidence required before deployment.
Leaders should define whether the AI is permitted to:
- Observe information.
- Summarize or explain conditions.
- Recommend an action.
- Initiate a controlled workflow.
- Execute a change affecting an operational system.
Each step should introduce stronger testing, identity controls, authorization boundaries, monitoring, and human approval. NIST has identified AI-agent identity and authorization as an emerging security priority. Its work considers identification, authorization, auditing, non-repudiation, and protection against prompt injection. These controls become especially important when an agent can access operational data or invoke external tools. NIST AI-agent identity and authorization
An AI agent should never inherit broad operational authority simply because its human user has that authority.
The agent needs its own identity, clearly defined permissions, attributable actions, and limits that cannot be changed through an ordinary conversation with the model.
Human oversight must also be tested
“Human in the loop” is not sufficient if the human cannot understand, challenge, or safely reject the recommendation. Testing should examine the complete human and AI decision process.
Does the engineer understand which information the agent used? Can the operator recognize missing or uncertain data? Is there enough time to review the recommendation? Does the interface distinguish a confirmed fact from an AI-generated conclusion? A confident recommendation can create automation bias. After months of generally correct performance, personnel may become less likely to challenge the system when it is wrong.
Effective oversight requires more than an approval button. It requires trained personnel, clear decision rights, usable supporting evidence, independent process indicators, and the authority to stop the AI-enabled workflow.
The engineer remains accountable for the operational decision. The system must provide enough transparency for that accountability to be meaningful.
Build an operational assurance case
Before connecting an AI agent to an engineering or OT environment, leaders should require a documented assurance case.
It should explain:
- The operational purpose of the system.
- The data, tools, and environments it can access.
- The authority it has been granted.
- The conditions under which it has been tested.
- Its known limitations and prohibited uses.
- How uncertainty and failure are communicated.
- Which actions require human approval?
- How its identity and privileges are controlled.
- How model, prompt, tool, and configuration changes are governed.
- The qualifications and operational experience of the evaluation team.
- How the test environment is contained, monitored, and independently validated.
- How are unsuccessful and unexpected outcomes reported?
- How the AI system can be isolated, disabled, and recovered.
This should not become another paperwork exercise. It should give operators, engineers, cybersecurity teams, and executives a shared understanding of what the AI can be trusted to do.
NIST is developing an AI Risk Management Framework profile specifically for critical infrastructure. The initiative recognizes that trustworthy AI must be managed across its full lifecycle and across IT, OT, industrial control systems, and their supply chains. NIST Critical Infrastructure AI RMF Profile
The direction is clear. AI assurance in critical infrastructure must become an operational discipline, not simply a model-selection exercise.
Trust begins at the failure boundary
Industrial AI will help engineers process information faster, investigate cybersecurity events, improve maintenance decisions, and extend scarce expertise across distributed operations.
But trust should not be based on how impressive the system appears when everything works as expected.
Trust begins when we understand where it fails, how it behaves when information is incomplete, and whether people can maintain control when its recommendation is wrong.
The boardroom should not ask only, “How accurate is the AI?” It should ask: “What authority have we given it? Where have we proven its limits? Who validated the testing environment? Can the operation remain safe when it fails?”
Before an AI agent enters the control room, we should know where its confidence ends and engineering judgment must begin.