1. Predicting Electricity Distribution Feeder Failures using Machine Learning Marta Arias 1, Hila Becker 1,2 1 Center for Computational Learning Systems 2 Computer Science Columbia University LEARNING ‘06

2. Overview of the Talk Introduction to the Electricity Distribution Network of New York City  What are we doing and why? Early solution using MartiRank, a boosting- like algorithm for ranking Current solution using Online learning Related projects

3. Overview of the Talk Introduction to the Electricity Distribution Network of New York City  What are we doing and why? Early solution using MartiRank, a boosting- like algorithm for ranking Current solution using Online learning Related projects

4. The Electrical System 1. Generation2. Transmission 3. Primary Distribution 4. Secondary Distribution

5. Electricity Distribution: Feeders

6. Problem Distribution feeder failures result in automatic feeder shutdown  called “Open Autos” or O/As O/As stress networks, control centers, and field crews O/As are expensive ($ millions annually) Proactive replacement is much cheaper and safer than reactive repair

7. Our Solution: Machine Learning Leverage Con Edison’s domain knowledge and resources Learn to rank feeders based on susceptibility to failure How?  Assemble data  Train model based on past data  Re-rank frequently using model on current data

8. New York City

9. Some facts about feeders and failures About 950 feeders:  568 in Manhattan  164 in Brooklyn  115 in Queens  94 in the Bronx

10. Some facts about feeders and failures About 60% of feeders failed at least once On average, feeders failed 4.4 times (between June 2005 and August 2006)

11. Some facts about feeders and failures mostly 0-5 failures per day more in the summer strong seasonality effects

12. Feeder data Static data  Compositional/structural  Electrical Dynamic data  Outage history (updated daily)  Load measurements (updated every 5 minutes) Roughly 200 attributes for each feeder  New ones are still being added.

13. Feeder Ranking Application Goal: rank feeders according to likelihood to failure (if high risk place near the top) Application needs to integrate all types of data Application needs to react and adapt to incoming dynamic data  Hence, update feeder ranking every 15 min.

14. Application Structure Static data SQL Server DB ML Engine ML Models Rankings Decision Support GUI Action Driver Action Tracker Decision Support App Outage data Xfmr Stress data Feeder Load data

15. Goal: rank feeders according to likelihood to failure

16. Overview of the Talk Introduction to the Electricity Distribution Network of New York City  What are we doing and why? Early solution using MartiRank, a boosting-like algorithm for ranking  Pseudo ROC and pseudo AUC  MartiRank  Performance metric  Early results Current solution using Online learning Related projects

17. (pseudo) ROC sorted by score 0 0 0 1 2 1 3 outagesfeeders

18. (pseudo) ROC Number of feeders Number of outages 941 210

19. Fraction of outages (pseudo) ROC 1 1 Area under the ROC curve Fraction of feeders

20. Some observations about the (p)ROC Adapted to positive labels (not just 0/1) Best pAUC is not always 1 (actually it almost never is..) E.g.: pAUC = 11/15 = 0.73  “Best” pAUC with this data is 14/15 = 0.93 corresponding to ranking 21000 11 20 32 40 50 ranking outages 12345 3 2 1

21. MartiRank Boosting-like algorithm by [Long & Servedio, 2005] Greedy, maximizes pAUC at each round Adapted to ranking Weak learners are sorting rules  Each attribute is a sorting rule  Attributes are numerical only If categorical, then convert to indicator vector of 0/1

22. MartiRank feeder list begins in random order sort list by “best” variable divide list in two: split outages evenly divide list in three: split outages evenly choose separate “best” variables for each part, sort choose separate “best” variables for each part, sort continue…

23. MartiRank Advantages:  Fast, easy to implement  Interpretable  Only 1 tuning parameter “nr of rounds” Disadvantages:  1 tuning parameter “nr of rounds” Was set to 4 manually..

24. Using MartiRank for real-time ranking of feeders MartiRank is a “batch” algorithm, hence must deal with changing system by:  Continually generate new datasets with latest data Use data within a window, aggregate dynamic data within that period in various ways (quantiles, counts, sums, averages, etc.)  Re-train new model, throw out old model Seasonality effects not taken into account  Use newest model to generate ranking Must implement “training strategies”  Re-train daily, or weekly, or every 2 weeks, or monthly, or…

25. Performance Metric Normalized average rank of failed feeders Closely related to (pseudo) Area-Under-ROC-Curve when labels are 0/1:  avgRank = pAUC + 1 / #examples Essentially, difference comes from 0-based pAUC to 1-based ranks

26. Performance Metric Example 10 21 31 40 51 60 70 80 rankingoutages 3 2 1 12345678 pAUC=17/24=0.7

27. How to measure performance over time Every ~15 minutes, generate new ranking based on current model and latest data Whenever there is a failure, look up its rank in the latest ranking before the failure After a whole day, compute normalized average rank

28. MartiRank Comparison: training every 2 weeks

29. Using MartiRank for real-time ranking of feeders MartiRank seems to work well, but..  User decides when to re-train  User decides how much data to use for re-training  …. and other things like setting parameters, selecting algorithms, etc. Want to make system 100% automatic!  Idea: Still use MartiRank since it works well with this data, but keep/re-use all models

30. Overview of the Talk Introduction to the Electricity Distribution Network of New York City  What are we doing and why? Early solution using MartiRank, a boosting-like algorithm for ranking Current solution using Online learning  Overview of learning from expert advice and the Weighted Majority Algorithm  New challenges in our setting and our solution  Results Related projects

31. Learning from expert advice Consider each model as an expert Each expert has associated weight (or score)  Reward/penalize experts with good/bad predictions  Weight is a measure of confidence in expert’s prediction Predict using weighted average of top- scoring experts

32. Learning from expert advice Advantages  Fully automatic No human intervention needed  Adaptive Changes in system are learned as it runs  Can use many types of underlying learning algorithms  Good performance guarantees from learning theory: performance never too far off from best expert in hindsight Disadvantages  Computational cost: need to track many models “in parallel”  Models are harder to interpret

33. Weighted Majority Algorithm [Littlestone & Warmuth ‘88] Introduced for binary classification  Experts make predictions in [0,1]  Obtain losses in [0,1] Pseudocode:  Learning rate as main parameter, ß in (0,1]  There are N “experts”, initially weight is 1 for all  For t=1,2,3, … Predict using weighted average of each experts’ prediction Obtain “true” label; each expert incurs loss l i Update experts’ weights using w i,t+1 = w i,t pow(ß,l i )

34. In our case, can’t use WM directly Use ranking as opposed to binary classification More importantly, do not have a fixed set of experts

35. Dealing with ranking vs. binary classification Ranking loss as normalized average rank of failures as seen before, loss in [0,1] To combine rankings, use a weighted average of feeders’ ranks

36. Dealing with a moving set of experts Introduce new parameters  B: “budget” (max number of models) set to 100  p: new models weight percentile in [0,100]   : age penalty in (0,1] When training new models, add to set of models with weight corresponding to p th percentile (among current weights) If too many models (more than B), drop models with poor q-score, where  q i = w i pow( , age i )  I.e.,  is rate of exponential decay

37. Other parameters How often do we train and add new models?  Hand-tuned over the course of the summer Every 7 days  Seems to achieve balance of generating new models to adapt to changing conditions without overflowing system  Alternatively, one could train when observed performance drops.. not used yet How much data do we use to train models?  Based on observed performance and early experiments 1 week worth of data, and 2 weeks worth of data

38. Performance

39. Failures’ rank distribution

40. Daily average rank of failures

41. Other things that I have not talked about but took a significant amount of time DATA  Data is spread over many repositories. Difficult to identify useful data Difficult to arrange access to data  Volume of data. Gigabytes of data accumulated on a daily basis. Required optimized database layout and the addition of a preprocessing stage  Had to gain understanding of data semantics Software Engineering (this is a deployed application)

42. Current Status Summer 2006: System has has been debugged, fine-tuned, tested and deployed Now fully operational Ready to be used next summer (in test mode) After this summer, we’re going to do systematic studies of  Parameter sensitivity  Comparisons to other approaches

43. Related work-in-progress Online learning:  Fancier weight updates with better guaranteed performance in “changing environments”  Explore “direct” online ranking strategies (e.g. the ranking perceptron) Datamining project:  Aims to exploit seasonality  Learn “mapping” from environmental conditions to good performing experts’ characteristics When same conditions arise in the future, increase weights of experts that have those characteristics Hope to learn it as system runs, continually updating mappings MartiRank:  In presence of repeated/missing values, sorting is non-deterministic and pAUC takes different values depending on permutation of data  Use statistics of the pAUC to improve basic learning algorithm Instead of input nr of rounds, stop when AUC increase is not significant Use better estimators of pAUC that are not sensitive to permutations of the data

44. Other related projects within collaboration with Con Edison Finer-grained component analysis  Ranking of transformers  Ranking of cable sections  Ranking of cable joints  Merging of all systems into one Mixing ML and Survival Analysis

45. Acknowledgments Con Edison:  Matthew Koenig  Mark Mastrocinque  William Fairechio  John A. Johnson  Serena Lee  Charles Lawson  Frank Doherty  Arthur Kressner  Matt Sniffen  Elie Chebli  George Murray  Bill McGarrigle  Van Nest team Columbia:  CCLS: Wei Chu Martin Jansche Ansaf Salleb Albert Boulanger David Waltz Philip M. Long (now at Google) Roger Anderson  Computer Science: Philip Gross Rocco Servedio Gail Kaiser Samit Jain John Ioannidis Sergey Sigelman Luis Alonso Joey Fortuna Chris Murphy  Stats: Samantha Cook