Next page >> Faster, better, cheaper results

Editor's note: JD Hammerly formerly ran North America for Alstom T&D (since purchased and renamed Areva Grid). He's one of that rare breed with a deep understanding of the technical side AND the financial aspects AND the business issues AND the sales side. This broad knowledge base makes him adept at spotting trends and their implications. Over lunch recently he told me his firm was getting amazing results from a transformational approach to coding aimed at data analytics. Since readers tell me analytics is simultaneously their biggest opportunity and their biggest headache, I asked him to share his approach with Smart Grid News. - Jesse Berst

.

By JD Hammerly

The electrical industry currently uses a myriad of applications for data analysis. These applications take in data, apply algorithms, and present results. But they are built "one-off" with a heavy emphasis on coding. Typically, 75% of the total code is dedicated to "infrastructure" – data preparation, application flow, storage of results, etc. Thorough testing of the algorithms must await the completion of that extensive infrastructure code.  

 

Historically, this approach was effective. But the smart grid has meant an enormous increase in the available data, which resides in a “system of record” archive scaling to trillions of records representing tens of petabytes. This data tsunami puts the traditional approach in jeopardy. 

This new reality requires a new approach to analytics. Competitiveness hinges not just on quality, but also on cost and time to market. This balancing act creates an inherent conflict and requires fundamental change.

 

Why heavy coding is obsolete

Next-generation applications can no longer afford to rely on heavily code-based

solutions. In their place, we need to substitute an approach that makes use of commercial “solvers.”  Solvers are general purpose mathematical engines into which you plug specific problems. Solvers take away some of the need for infrastructure coding. To use them effectively, developers merely need to know the basics of data, how to present the problem to the solver, and which of several different mathematical approaches to use (or, at least, how to test different approaches to see which one gives the best results).

By taking advantage of these powerful, commercially available engines, we can divide the development process into:

1.     Problem formulation

2.     Integrating the data

3.     Selecting the solver  and algorithm approach

4.     Presenting the results

 

The importance of problem formulation

Commercial solvers are driven by mathematical and statistical constructs but they have no knowledge of the application’s intent. Thus, they require a domain expert who can formulate the problem in a way the solver can "understand." Further, the domain expert often needs to select from several algorithms. Commercial solvers typically include a selection of predefined mathematical techniques. You can switch from one mathematical approach to another without investing in any code. Better yet, you can test several alternatives and then choose a winner.

Next page: Faster, better, cheaper results

Talk Back to the Author

Current Comments (10)

Leave a Comment

---

Data Analytics

Finally, something truly fresh on Data Analytics! Thanks, JD - great article!!

M.A. Marullo - 04/06/2012 - 09:07

Data Analytics

Congratulations to Mr. Hammerly; he has his finger squarely on one of the most current operational issues in the energy domain. We have found in working with our clients that an abstracted method for problem formulation driven by a generic computational engine empowers the client to focus on real and current business intelligence needs. This approach is responsive to the rapidly changing demands of the enterprise and leverages the core organizational competencies while minimizing the "pick and shovel" work that has typically impeded such efforts in the past. However, we have found that it is crucial to devote adequate time toward careful evaluation and analysis of the data structures and the dimensions along which analysis occurs, so that efficient data structures can be implemented to ensure that data access and transformation do not become choke points in the process. This is particularly true when performing comparative evaluations of multiple approaches for problem solving. Our experience indicates that while low-level coding may be receding in importance, it is crucial to engage competent and domain-knowledgable data architects and designers at the outset to ensure that advanced BI analytics are not crippled by long response times, particularly when analyzing over long time horizons.

Jeff W Chapman - 04/06/2012 - 11:01

Solver Software

Can you mention some examples of such solver software, or a source to look for examples? A google for Big Data Solver only turned up one example that seemed relevant to this article.

Bill Bruns - 04/06/2012 - 12:25

Follow up - Solver software

AIMMS is an example. http://www.aimms.com/aimms/ It was created by Paragon Decision Technology, a Dutch company and has a US operation that is mostly focused on the financial community. Once a formulation exists you can experiment with different engines to evaluate which provides better answers, faster (e.g.mix-integer programming, dynamic programming) by simply changing a parameter or two.

JD Hammerly - 04/06/2012 - 13:18

The distinction is not quite so clear cut

This really isn't anything new, as these systems have been around for years. There is always a trade-off between having a domain expert working interactively with an analytical system or the same domain expert working interactively with a programmer. People who don't work with these systems (but read the marketing literature) tend to underestimate the skill level required to use them. Don't kid yourself, they require programming skill, just a different kind than rwriting procedural code in a high-level language. Both options have their uses, and some analytical systems allow "compilation" of the resulting models so they can be called from other software applications. That "hybrid" approach can give you the best of both worlds, the rapid prototyping ability inherent in an interactive tool combined with the full automation and rapid results of a fully automated custom solution.

Greg Tinfow - 04/06/2012 - 16:44

Dark matter

The dark matter is coding the interfaces and maintaining the system of records. Solvers like Mathematica, Matlab, Aiims, CPlex, etc. have been available for decades, nothing new there. But you will never be able to get rid of the coders, because 80% of each project is bringing the data in and getting it out for interpretation, then maintaining the process day to day. Hire the coders back, please. Cheers,

Valentine - 04/06/2012 - 17:41

Current solver software

What are some problems with the current solvers in the application of smart grid big data analysis? Is it necessary to expect a new solver software specifically tailored for this application? Thank you very much for your insightful article.

M. Jin - 04/07/2012 - 07:34

Example for Solver Softare

Interesting. An example will help, both for planning and real time optimizations.Thanks.

Shailesh Jain - 04/07/2012 - 10:50

Actionable Analytics

Thanks JD for writing an insightful post. It is good to see that others share our position. Powerrunner is one such energy-focused software company executing the sentiments in this article. For several years, the Forrunner Load forecasting application has used a “data solver” at its core for non-linear optimization for prediction of short-term and long-term load. This has allowed our industry specific skills to focus on the business processes and functionality specific to energy companies in dealing with analyzing advanced meter data and performing actionable analytics to truly approach forecasting from a ground-up direction.

Dan Garvey - 04/09/2012 - 11:18

Collection & Delivery of the raw material

Well done JD. Thanks. I just want to throw another dimension to the analytics discussion, one often overlooked. Can solvers disseminate chunks of their smarts to the edge? I fully agree with the 'solver' approach, remembering though that analytics can only deal with the raw material it has. Collecting, transporting and storing are all critical. Recent modelling research prooves that an ounce of intelligence at the edge of the grid, at the IED, can save a pound (often a tonne) at the server. Can solvers disseminate chunks of their smarts to the edge?

Yochai Glick - 04/09/2012 - 17:18

Leave a Comment

New to this Blog? Need some help?

Email: [?] We will send you an email with an appproval link you must click for your comment to appear.
Full Real Name:	[?] Real comments need a real name.
Subject:	[?] Please provide a short subject for your comment.
Comment:[?] Do not post commercial/sales messages here. They will be deleted. Try: pr@smartgridnews.com PLEASE NOTE:  All HTML codes will be removed from your comment.
CAPTCHA Security Code The numbers (0) and (1) are not used. [?] Please type the characters you see into the box above.
Remember my name? (Uncheck to forget.)
You can not edit or delete your comment once it is posted. Please check for spelling now. Your IP Address is: 107.22.156.205