The Coming Smart Grid Data Surge

Smart Grid diagram courtesy of U.S. Department of Energy

As has often been written, the advancements of the Smart Grid are founded in information. Data is used to inform consumption, to make rates more dynamic, and to enable the next-generation power prosumer. In reading a recent piece on potentially mandated Smart Metering in the UK, the Telegraph raises the issue of data handling relative to today's data management. In short strokes, 44 million homes were typically measured twice a year, making for 88 million entries for data. In the new system, every home is measured twice a day, meaning that those 88 million entries have now become over 32 billion. This sounds like a lot, so let's quickly look at the new challenges that arise for organizations seeing this kind of increase:

• Data Center Expansion. The types and volume of data associated with Smart Grid use will mean a new need to bring Internet-style data centers into the complex mesh of utility control systems.
• Data Organization and Retention. With Time of Use pricing and user charge recovery for power generated, a sizable subset of this data will no longer be simply transient and used in the aggregate. Individual elements will need to be captured and tagged for later retrieval over whatever period is chosen by regulators as appropriate for looking back.
• Data Privacy. While there may be dubious benefit to stealing the private data from individual citizen's smart meters, it is naive to think that privacy concerns will not find their way into regulation. That means data will have to be partitioned when needed longer term, destroyed when transient, and never left in an unknown state.

I led with the UK piece, because it does a relatively nonthreatening analysis of data gathering trends from a smarter grid.

The U.S. Smart Grid, however, has a series of challenges that expand on this by many times. Back in May, Beth Pariseau did a piece on Smart Grid storage for SearchStorageChannel.com where she interviewed a variety of players, including Austin Energy’s CIO, Andres Carvallo.  The data usage trends described are nothing short of mind-boggling.

In the Austin Energy data, for phase one of the roll-out which included 500,000 meters, the increase in yearly data storage went from 20 TB to 200 TB, with disaster recovery redundancy. This is for 15 minute sampling, and first stage (home-oriented) integration. Ignoring smaller sampling frequencies (resulting in much higher data storage) necessary for some Smart Grid functionality, this presents a model of about 400 MB per meter per year ( 200,000,000,000,000/500,000 ).

While this sounds mind-numbing, there is substantiation (and a reasonably close ratio) in the same piece, this from Pacific Gas and Electric, who added 1.2 PB of memory (and growing) to support 700,000 meters, or over 170 MB per meter per year. (And this was sampling only twice per day).

Finally, in their August 2009 report on “Assessment of Demand Response and Advanced Metering,” FERC presented a partial scenario (80 million meters) and a full deployment scenario (140 million meters) by 2019. Assuming that we feel comfortable in the midrange of the data descriptions used above, this would imply the need for the creation of infrastructures necessary to organize and manage roughly 100 PB of information within the next ten years. (Good luck to us all!)

What conclusions can we draw from all of this?

Massive Data is about to swamp existing infrastructure, requiring some hard thinking about how to architect, secure, segment, and deploy the data centers that will accommodate it.

• There is striking variability in the amount of data organizations are expecting, seeing, and preparing for. Work is needed on what information should be gathered, what needs to be stored long-term, what needs to be tagged with user information, and what needs to be treated as private.
• This is a new area for providers. The storage, record keeping, and maintenance of all of this data, particularly that which needs to be held for longer regulated periods, is unlikely to be a current function of the provider budget and functional organization. The steps to rationalize this area financially are critically important. Any plan to advance smart metering should include these costs in justification or grant request.
• Every new idea must detail the additional burden on providers, particularly those in the soft grid investment space, from a data acquisition and data management perspective.

Like so much of our economy, these advancements are changing the grid from a power economy to a data and power economy. To survive and thrive these new requirements must be considered. In the medium and long term, those organizations that consider, and then capitalize on, all of this data will find themselves in a much better position to add services, ensure satisfaction levels, and find new ways to make the Smart Grid even smarter.

Jack Danahy and Andy Bochman are authors of the Smart Grid Security Blog.

   Email Jack Danahy

   Telegraph article on data handling

   Beth Pariseau article on data storage

   Interview with Andres Carvallo

   FERC report on DR and advanced metering (PDF)