Why and how to convert Big Data into little data (aka actionable data)
Big Data has been a utility preoccupation for the past two years. Guess what â€“ we've been thinking about it wrong. Sure, when you consider the vast amounts of data flowing in from smart meters and sensors, the input is indeed Big. But we should be thinking about the output, argues Steve Ehrlich of Space-Time Insight in this guest editorial.
It was just a short while ago that we presented a webinar called â€œForget about Big Data! Start with small data, then work your way upâ€ which featured experts from Duke, Centerpoint and eMeter. (Click the link to review the slides and view replays.)
I'm seeing a theme here. I hope you are noticing it too. â€“ Jesse Berst
By Steve Ehrlich
With the massive volume of smart grid data threatening to overwhelm many utilities, executive are looking for answers. Not just answers as to how to cope with the volume, but answers as to how their investment in the smart grid is going to deliver tangible benefits.
Of course, to get answers you have to ask questions. As a visual analytics vendor, we are often on the receiving end of questions like this: â€œWe have 5 million meters. How is your software going to deal with displaying all of them on the screen?â€
Although it is possible to put 5 million dots on a screen (and one could imagine scenarios where displaying all of those meters at once is useful) one has to question whether that is, indeed, the right question. After all, how much data can a person reasonably consume at once?
Converting big to little
To really understand and get value from Big Data, you have to convert it to little data. Take those 5 million meters for instance. It would probably be more interesting to understand which ones havenâ€™t responded in the last hour (or day or week). Now you might be looking at tens of thousands of meters. Smaller but still too big.