<< Return to Page One
I feel their pain... but do they feel ours?
I do feel the pain of security researchers who feel that they have been ignored. We see vendors and utilities living with known vulnerabilities rather than fix them. My team and I have been frustrated too. But these revelations can impact something that is rarely discussed though vitally important -- trust.
I am not talking about the geek definition, such as a trust relationship in a Windows 2003 multi-domain server environment. Rather I mean the trust that must exist between people. The trust that comes into play when you need quick advice with confidence the information you disclose will not be shared.
Without trust, we will never be able to stay ahead of our adversaries. We will never be able to limit the risk to our assets. The recent vulnerability and exploit disclosure may have gotten “atta-boys” from the cyber researcher community. It did, after all, force the issue of persistent, unaddressed vulnerabilities into the open. Yet I am concerned it may reduce the ability of responsible security engineering and consulting firms to establish and maintain the trust needed with their utility and vendor clients.
Let me give an example. I recently lead a webinar with the EnerNex cyber security team for the Utility Telecom Council (UTC). About 60 utilities participated. The topic was the latest changes in the NERC CIP world. The most remarkable thing: the time allocated to live Q&A was deadly silent. Anonymous questions were submitted in advance but no one wanted to say anything. Finally, a couple of brave souls did ask one or two “safe” follow-up questions that would not reveal internal information.
We've been seeing this dangerous symptom for a while. No one wants to imply lack of knowledge. No one wants to imply they might have a critical infrastructure vulnerability. We know several security engineers and practitioners who have been forbidden from speaking to anyone – even their peers at other utilities – in any public venue. This implies no more presentations on real-world problems and their solutions.
Our company has had the fortune of developing a solid reputation for discretion, confidentiality and integrity. It can be difficult, yet it is doable with solid trust relationships. However, events like the recent revelations put strains on relationships by making the parties wonder – will they disclose information that could impact me?
I very much respect the researchers who made the revelation even though I disagree with it. As a committed optimist I am hopeful that this “experiment” will have positive results and that the impact on trust between stakeholders will be minimal and short lived. The vulnerabilities and exploits released were real and actionable, not academic. Even so, they are all manageable with disciplined risk management. I recommend the "systems of systems" engineering approach. Working together, we can uncover vulnerabilities, develop solutions, educate key stakeholders, and implement solutions and workarounds.
Erich W. Gunther is Chairman and CTO of EnerNex, an electric power research, engineering and consulting firm. He is also the interim director for EnerNex Smart Grid Labs. EnerNex and SGL are active in the development of standards and best practices for utility cyber security as well as the practical application of cyber security disciplines for their vendor and utility clients. Erich’s cyber security team performs physical and cyber security evaluations and penetration tests on new and legacy utility infrastructure equipment and systems in the field and at their lab in Knoxville, Tennessee.
<< Return to Page One
| Trust indeed |
| Great article, Erich! Very true indeed. Trust is what makes the whole thing work. It allows the vendors and utilities to work together, it allows utilities to work with security researchers, and allows the researchers to remain quiet when appropriate and for a certain time. I will be forwarding this to a number of individuals because your points are well articulated and valid. It is this kind of trust which InGuardians has always striven for, sometimes alongside Enernex as partners. I would like to indicate another recent issue which raises the issue of trust, as a contrast to the circumstances you rightfully call out. Don Weber's talk was recent pulled from a security "hackers" conference, by an as yet unnamed AMI vendor, even though extreme discretion was not warranted. Although the topic is uncomfortable for some, Weber's talk was about significant and novel use of what has thus far been considered "like a physical attack." Having given the power industry early access to his materials for review, Don's preso was to point out how C12.18 security is being abused in deployment, indicating real-world impact that is not being widely considered, providing guidance on mitigations that utilities can and should be putting in place, and providing a tool that allows asset owners to discover and validate the weaknesses. Don's tool, humorously titled "SMACK" (Don has a strange preoccupation with the Hulk) does *not* provide any weaponized exploit for power meters, but rather allows for validating whether a hacker has stumbled upon the C12.18 password, after significant hardware-hacking. Many utilities - although fewer and fewer, based in part on the work of InGuardians - still consider a single network-wide C12.18 password acceptable and beneficial. This presentation and tool does not represent clear and present danger to our power grid, the value of motivating those who *can* make changes is great, and many of the changes/mitigations can be rolled out starting tomorrow... if the presentation was not muzzled. The trust needs to flow both ways. Applying pressure for InGuardians to pull Don's talk for little reason does not bolster that trust. Hackers will hack. Just because the good guys shut up, doesn't mean the problem goes away. Please encourage your readership to be cautious about not damaging the trust going the other way. I'm sure they will pay attention to what you have to say. Best Regards, Matt Carpenter |
| Matthew Carpenter - 02/06/2012 - 18:27 |
| Nomalization of Deviance in the Power Industry |
| Congratulations! You give an example of the Normalization of Deviant. Next see a segment of an interview I found on Internet and info about a blog post. Deviance, normalization of deviance. What was exactly that normalization of deviance in the case of NASA? Diane Vaughan: Social normalization of deviance means that people within the organization become so much accustomed to a deviant behaviour that they don’t consider it as deviant, despite the fact that they far exceed their own rules for the elementary safety. But it is a complex process with some kind of organizational acceptance. The people outside see the situation as deviant whereas the people inside get accustomed to it and do not. The more they do it, the more they get accustomed. For instance in the Challenger case there were design flaws in the famous « O’rings », although they considered that by design the O-rings would not be damaged. In fact it happened that they suffered some recurrent damage. The first time the O-rings were damaged the engineers found a solution and decided the space transportation system to be flying with « acceptable risk ». The second time damage occurred, they thought the trouble came from something else. Because in their mind they believed they fixed the newest trouble, they again defined it as an acceptable risk and just kept monitoring the problem. And as they recurrently observed the problem with no consequence they got to the point that flying with the flaw was normal and acceptable. Of course, after the accident, they were shocked and horrified as they saw what they had done. Smart Grid: SoS "… interacting in unpredictable ways that regulators and investors cannot comprehend, far less control.” ( bit.ly/GMH049 ) Summary: Similar to financial markets, system crashes are expected in smart grid, because they have been though to be just complex technological systems, when they are in fact ultra large scale socio-technical systems. The difference between the two kinds of systems is told in “… the story of the London Millennium Bridge, which opened in June 2000 and two days later was closed for two years to remedy destabilizing swaying motions induced when groups of people walked over it.” As industry restructuring was flawed, legislators, regulators, and investors have a change to minimize the damage in the making on the power industry, by learning about their responsibility of the now known error of the Normalization of Deviance before it is too late. |
| José Antonio Vanderhorst-Silverio, PhD - 02/07/2012 - 19:43 |
| Normalization of Deviance Leading to Systemi Risk |
| The following is the text of a tweet in response to a comment by Ben Rolfe on the IEEE Smart Grid Group on Linkedin. http://lnkd.in/njKHYW Thank you Ben. The problem is not just risk, but systemic risk (small changes that produce big destruction)... |
| José Antonio Vanderhorst-Silverio, PhD - 02/09/2012 - 05:18 |
Talk Back
