the OTCISO Newsletter
ENISA published its Handbook for Cyber Stress Tests in May 2025.
I read it the way I read every new regulatory artifact arriving in the OT space, looking for the gap between the paper and the plant.
The handbook is intelligent. It is timely. It is, in places, exactly what NIS2 needs the EU regulatory toolkit to have. And the framing resilience focus, scenario-based, systemic risk view is the right framing for a sector that has spent the last decade conflating compliance evidence with operational capability.
But the methodology ENISA has defined, if transposed unchanged into OT environments, will create the same audit-theater problem that the industry has spent the last five years trying to escape.
This is not a rejection of stress testing. Stress testing is overdue, and what ENISA has built is a meaningful step forward from broad compliance audits.
This is a warning about what happens next.
What the Handbook Gets Right
Before I argue with it, let me credit what is genuinely valuable in the document.
The shift from compliance focus to resilience focus is real. The recognition that resilience is a multi-stage property, preparedness, response, and recovery, is correct. The acknowledgment that gaps revealed by stress tests can be discussed in collaborative settings before stricter supervision is a mature regulatory posture. The 5-step methodology is sound at a high level. The case studies, the ECB's 2024 cyber resilience test of 109 banks and Belgium's 2024 hybrid threat test on energy infrastructure, show what good practice looks like at scale.
These are real contributions. They deserve serious engagement.
But the handbook also defines, with admirable transparency, exactly where the methodology stops short.
The Structural Problem
On page 7, the handbook tells you what a cyber stress test is and is not:
"In practice, stress tests are mostly 'desktop-based', relying on a technical questionnaire, centered around one or more risk scenarios, which is filled out independently by the entities/organizations that are being tested."
And then:
"A cyber stress test is not a penetration test, conducted in real time, or a cyber exercise."
Read those two paragraphs together. The handbook says: " This is a paper exercise, completed individually by the entity, scored against scenarios that exist only as text.
In an IT environment, that is a defensible methodology. Many of the variables in IT incident response are documentable. Recovery time on a domain controller can be estimated. Backup verification can be paper-evidenced. Patch coverage can be sampled and reported. The questionnaire approximates reality reasonably well.
In an OT environment, the questionnaire approximates something entirely different.
A questionnaire is what an organization with no real capability fills out competently when the program manager is good at writing. A questionnaire is what auditors have been collecting for fifteen years under ISO 27001, SOC 2, IEC 62443 self-assessments, and NIS2 transposition exercises. A questionnaire is exactly the artifact I named in The Compliance Trap as the difference between audit-ready and incident-ready.
Renaming the questionnaire a "stress test" does not change what it measures.
What OT Stress Testing Actually Has to Verify
In an OT environment, the difference between resilient and non-resilient cannot be captured on paper. Here is what real OT stress testing has to verify and what a desktop questionnaire structurally cannot.
Mean time to safe operational state, not mean time to restore. The handbook's example resilience metrics include MTTD and MTTR, Mean Time to Detect and Mean Time to Respond. Both are IT vocabulary. In OT, the relevant clock is Mean Time to Safe State — the time required to bring the physical process into a verified, stable, non-hazardous condition. Recovering a server while a chemical process drifts toward an unsafe envelope is not recovery. It is an ongoing incident. No questionnaire I have seen, including ENISA's example, captures this distinction.
Operator-in-the-loop decision chains. The first hour of an OT incident is almost always managed by operators who must make containment decisions before cybersecurity has fully characterized the event. A questionnaire asking "Do you have an incident response plan?" cannot verify whether the operator on shift has read it, practiced it, has the authority to act on it, or has a working escalation contact at 3 a.m. on a Sunday. The plan is on paper. The operator is in the control room. Only one of them is the actual response capability.
Vendor escalation under real conditions. A contract can state a response target and still leave operational questions unanswered. A tabletop or controlled call-tree exercise can reveal overloaded support queues, unfamiliar responders and parallel approval chains that delay access. A questionnaire cannot test that path; an exercise can.
Process safety integrity under a cyber event. OT environments are protected by layered safety systems — SIS layers, LOPA discipline, and hard-coded safety interlocks. These layers were designed under the assumption that failures are random and that operators are human. Agentic systems and modern cyberattacks break both assumptions. A questionnaire asking whether safety controls exist cannot test whether those controls remain effective during a coordinated cyber event that specifically targets the assumptions on which the safety architecture depends.
Drift between documentation and running configuration. A recurring finding in OT reviews is a network architecture diagram that no longer matches the running configuration. Emergency change windows, vendor modifications, undocumented patches, and field workarounds accumulate over months and years. A questionnaire collects the documented architecture and treats it as truth. An exercise discovers the drift. There is no middle option.
Cross-discipline coordination. A live OT incident requires operations, process engineering, safety, IT, cybersecurity, and external vendors to coordinate within minutes. A questionnaire completed by the cybersecurity team alone — or even by a steering committee in a conference room cannot verify that those disciplines actually communicate when the alarm cascade is in its second hour.
Insecure-by-design protocols and legacy reality. Modbus, DNP3, certain implementations of IEC 61850, BACnet, and much of the building automation stack — these are not vulnerable because of implementation mistakes. They are vulnerable by design, built for a network era when physical access was the boundary. A questionnaire asking whether segmentation exists cannot tell you whether the segmentation actually holds against a lateral movement attempt across an insecure-by-design protocol. Only a live test can.
The Case for Live Testing — Already Made in the Handbook Itself
The handbook tells the reader what good practice looks like, even if it does not center that practice in the methodology.
The ECB's 2024 cyber resilience test was applied to 109 banks. Twenty-eight of those banks were required to perform an actual IT recovery test, provide evidence that the recovery was successful, and undergo on-site audits by supervisors. That is stress testing. The other 81 banks completed a questionnaire. The handbook itself, in Annex B, shows the gradient between paper and reality, and the more rigorous tier was reserved for institutions where the consequences of surface were highest.
Belgium's 2024 hybrid threat test on critical energy infrastructure went further. It combined cyber, physical, and hybrid scenarios at three escalation levels. It included new threats — drones, social unrest, supply chain shortages that no purely cyber questionnaire would surface. The findings called out cross-border dependencies, supply chain risks, and the need for closer public-private cooperation in extreme scenarios. Closer to operational reality. Still primarily questionnaire-based, but informed by stakeholders who understood that paper alone would not capture what was being tested.
The pattern in both cases is the same. The closer the stress test gets to actual operational verification, the more useful it becomes and the more expensive, complex, and politically uncomfortable. The handbook acknowledges this implicitly. OT operators and national authorities should make it explicit.
The Trap to Avoid
I have written before about what happens when programs optimize for the artifact instead of the outcome.
Programs reorganize around the question auditors actually ask: Can you prove you have the control? That is a fair question. It is also a different question from the one that gets asked during an incident: Will the control hold when the plant is on fire?
The cyber stress test, as currently framed in the ENISA handbook, risks becoming the newest version of the artifact problem. National authorities will use it. Entities will complete it. Consultants will help. Templates will circulate. Programs will optimize for the questionnaire. The questionnaire will be returned. The authority will analyze the responses. Recommendations will be issued.
The next questionnaire will look better than the last one. And nothing structural will change. The audit will be clean. The incident, when it comes, will reveal what the questionnaire could not.
This is the compliance trap, wearing the vocabulary of resilience.
What To Do
For OT operators: treat the ENISA questionnaire as the warm-up, not the test. Add a live exercise tier operator-led, safety-validated, vendor-included, executed under realistic time pressure. Test the recovery clock against mean time to safe state, not mean time to restore. Verify drift between documentation and running configuration before the regulator does it for you. The questionnaire is a useful gap analysis. It is not a resilience proof.
For national authorities under NIS2, follow the ECB model. Require actual recovery testing for a subset of entities, the highest-consequence operators in each critical sector, with independent verification of recovery claims and on-site review. Treat the broader questionnaire results as a maturity signal, not a resilience signal. A clean stress test response from an entity that has never executed a live recovery is not evidence of resilience. It is evidence of a competent program manager.
For ENISA: publish an OT-specific addendum. The current health sector example acknowledges OT assets connected medical devices, IoT, and building infrastructure — but the questionnaire it offers is structurally IT-centric. Stress level 3 mentions OT in a single bullet about lateral movement. That is not enough. OT stress testing requires its own methodology: safety integration, operator-in-the-loop validation, vendor-escalation testing under real conditions, mean time to safe state as the primary metric, and live drift verification between documentation and configuration. The handbook has the structure to host that addendum. It just has not been written yet.
Final Thought
A questionnaire measures what an organization can describe. A stress test measures what it can do.
Most OT programs in 2026 can describe more than they can do, and the gap between the two is exactly where the next public OT incident will land.
The ENISA handbook is a useful tool. It is the floor of a real stress testing regime, not the ceiling.
The first real incident in a hospital, a substation, a water utility, or a refinery is what a stress test actually is. And there is no rewriting the report after the fact.
Subscribe today to the OTCISO Newsletter