For utilities, banks, and other critical infrastructure providers, Y2K arrives April 8 2014
By Stuart McCafferty and Andy Bochman
This is definitely a different article and subject area than my regular stories on microgrids, renewable energy, and the federal government’s energy-related activities. However, after reading it, many will walk away thinking microgrid solutions are more viable than ever... And, people that know me know that I'm not a "chicken little” and I regularly complain about some very notable cyber security mouthpieces that identify and teach where and how to exploit vulnerabilities, but offer little or nothing in the way of solutions. So, I enlisted my friend and fellow zoomie graduate, Andy Bochman from Bochman Advisors, to help with content, fact-checking, and editing support.
Here’s some background on why we felt compelled to do some research, make some phone calls, and write this story.
One of my co-workers, Kevin Brown, works for a BUNCH of utility companies doing penetration testing on a variety of devices. Sometime last year, Kevin and I talked about Microsoft’s decision to end support for Windows XP on April 8, 2014. We talked about potential impact to banks and utilities - and the effect it might have on U.S. and world economies if those systems were compromised. This long-telegraphed move by Microsoft (first announced in 2007) has profound implications for multiple critical infrastructure industries upon which entire economies are based. Here in the U.S., many electric utility field devices use Windows XP embedded to provide processing, communication, and intelligence to transmission and distribution networks. These devices have served very useful lives, providing rugged, upgradeable, patchable environments that can be kept current and which help manage electricity service to hundreds of millions of consumers.
It makes financial sense for Microsoft...
For Microsoft, it makes financial sense to stop supporting an operating system that is well over a decade old - released in 2001. For utility and banking systems, it is a shot across the bow, a visceral reminder of their dependency on 3rd party technologies, and an event that has profound financial implications for their organizations. For utilities that normally have field equipment life expectancies of 20+ years, it reinforces the new reality of operating in a "smart grid” where shorter technology lifecycles defy the "business as usual” paradigm utilities and regulators previously operated under. Their urgent task now: develop a strategy and budget CAPEX resources to address the end-of-life cyber security risks exposed by this decision.
For embedded systems deployed in ATMs and substations over large geographic areas, this presents a real predicament and risk to operational capabilities and at least two of our must-have critical infrastructures - banking and electricity delivery. With no patch capabilities and concerns over similar future end-of-life situations, utility companies and vendors find themselves in a conundrum.
They HAVE to do something. But what? Microsoft does offer paid migration support (kudos) and a migration path to Windows 7 or 8. And, the banks and utility customers are listening. Unfortunately, there are so many of these systems and devices that organizations trying to identify, locate, and replace them all are quickly overwhelmed.
This is not to say that XP systems and devices will suddenly seize up when support is terminated. Rather, over time, new vulnerabilities will be discovered, shared amongst various attacker communities, attack modules will be created, and these vulnerabilities will remain forever unpatched. XP systems will be sitting ducks, punching bags, target rich environments. Prior to April 8, as security issues were identified, Microsoft moved quickly to reverse engineer and provide solutions to its XP customer base. On April 9 and beyond, Microsoft disappears from the equation, and every customer is on their own to remedy these problems. For a utility, bank, or any other critical infrastructure of any size, the operative word for this challenge is: insurmountable.
So, we have a Y2K-like situation here with a hard date of April 8, 2014. Like Y2K, we’ve seen it coming from a long way off but somehow haven’t mustered the will to take action in time. The silver lining and lessons learned from Y2K is that at midnight, 2000, there were no missiles launched, the stock market systems did not crash, and prophetic End of Days scenarios never happened. The risk was managed, but there were very clear technologies like 32 bit operating systems that resolved the two-digit date issue very cleanly and we didn’t have the volume and frequency of cyber security threats we face today. This Y2K is quite different in scale and type.
So, what’s the answer? I believe it is fourfold:
1. Microsoft has provided a "custom support” option for retired XP that, according to a Computer World article, costs around $200 per PC with a $500k cap. For large vendors, this cost may make financial sense and provide an extended transition window to allow for unrushed upgrades after XP’s retirement deadline. Perhaps Microsoft is willing to consider lower fees or smaller caps for vendors that cannot afford the costs of Microsoft’s security hotpatches. There is a new visionary CEO in town with Satya Nadella and a man in Bill Gates that has dedicated his life to philanthropy and good will. The opportunity is there and vulnerable critical infrastructure is as much a threat to Microsoft’s business as anyone.
2. Utility companies should develop a corporate cyber security strategy and perform physical audits of their equipment and systems (similar to those done in the late 90’s) and designate CAPEX budgets to address the costs of upgrade and replacement. CAPEX and operational planning for 2015 is underway now with most large utilities. This can’t happen fast enough to meet this looming deadline, but coupled with a Microsoft XP extended warranty, it provides additional time for a structured approach to upgrade and replacement. It will also put them in a much stronger position next time. And since optimal technology solutions are a moving target, there will always be a next time.
3. Vendors need to recognize the looming threat and upgrade or develop new solutions to replace end-of-life XP systems. They should immediately stop selling systems based on Windows XP., and they should team with security solutions providers who can provide their client base with mitigation strategies.
4. Although some may argue that regulators should have rung the alarm bell on this years ago, they now need to appreciate the threat of patchless cyber security vulnerabilities and support utilities when they come with rate cases addressing the upgrade requirements that have now officially arrived.
The good news is that this is not new news to all of the above entities. Most utilities are quite aware of this threat and have already begun addressing it. Vendors are also introducing new products into the marketplace that are based on updated interoperability standards and operating systems. We have the capability of mitigating this threat, but it is simply going to take more time than remains and there will be a period of uncomfortably long exposure. Like the Y2K challenge, there is more at stake here than profits. This issue needs to be at the top of every vendor, regulator, and utility company risk management and budgeting plans. Y14 has arrived.
Stuart McCafferty is the President of GridIntellect, a certified Veteran-Owned Small Business. Recently awarded the Project Management Institute’s (PMI) 2013 Distinguished Project Award, Stuart is a U.S. military veteran, business leader executive, program manager, and system engineering professional with nearly 30 years of experience. He has specialized expertise in microgrids, smart grid technologies, managing complex technical projects, system architecture design, software development and implementation, real-time data acquisition, and managing Program Management Offices (PMOs).
Andy Bochman is the founder of Bochman Advisors LLC where the focus is on helping utility orgs narrow culture and communications gaps between senior business leaders and their security teams. A contributor to industry and national security working groups, Andy was recently Energy Security Lead for IBM, where he worked closely with the company's global energy sector clients.