Data Management Q&A Series: Relying on Giant Data Warehouses

Why isn’t a giant data warehouse a good option for managing meter data?

.

Just putting data in a large DW is not the answer. One has to approach it top-down with an understanding of the uses and applications of this data. This is what will allow data warehouse designers to develop data aggregates and summaries, populate multi-dimensional fact tables and create appropriate DW schemas  (star/snowflake/cube) based on the needs of that specific application.

.

Some applications of the data may require near real-time correlation with other data or information acquired to determine informational context and notify a specific set of end-users or systems. For instance, consider a hypothetical Smart Grid deployment wherein the smart meters are capable of sending notification on every power interruption at the meter. Depending on other information from the enterprise as well as information collected  along with the interruption or on subsequent restoration within a correlated timeframe, this could be interpreted as a:

·         Sustained interruption at the customer (restoration follows 5 minutes following the outage and /or customer trouble ticket)

·         Momentary interruption /power reliability issue (repeated interruptions in a defined timeframe)

·         Revenue assurance issue (momentary interruption followed by tilt/tamper status or alert from meter, or energy backflow into grid reported for meter  and customer  record in CIS does not indicate authorized distributed generation on premise)

Typical data warehouse architectures have latencies wherein historical data is summarized and used for analysis and presentation to end users. Although real-time data warehouse architectures can be developed to support such analysis and alert users, the architectural complexity is not insignificant. Applications such as these may be better implemented using complex event processing (CEP) architectures which can support the same kinds of analysis (correlation of seemingly disparate events within a timeframe).

.