1. http://www.hume.ictas.vt.edu Smart Grid: Where Computation, Communication and Power Systems Meet Sandeep K. Shukla shukla@vt.edu with Hua Lin, Yi Deng, James Thorp, Lamine Mili This work was partially supported by NSF grant EFRI-0835879 & an NSF IUCRC - S2ERC Project

2. About ACM ACM, the Association for Computing Machinery is the world’s largest educational and scientific computing society, uniting educators, researchers and professionals to inspire dialogue, share resources and address the field’s challenges. ACM strengthens the computing profession’s collective voice through strong leadership, promotion of the highest standards, and recognition of technical excellence. ACM supports the professional growth of its members by providing opportunities for life-long learning, career development, and professional networking. With over 100,000 members from over 100 countries, ACM works to advance computing as a science and a profession. www.acm.orgwww.acm.org

3. The Distinguished Speakers Program is made possible by For additional information, please visit http://dsp.acm.org/http://dsp.acm.org/

4. Outline Motivation – Need for Infrastructure Interdependence Study – Power System & Computing/communication – Smart Grid Need for Co-Simulation GECO – Our Co-simulator Designing New Relaying Scheme with GECO All PMU-State Estimator with GECO – Experimental Framework – Experimental Results and Interpretations Conclusions

5. Infrastructure Interdependencies “Our nation’s infrastructures have become increasingly interconnected and interdependent … this creates an increased possibility that a rather minor and routine disturbance can cascade into a regional outage … it also creates new assurance challenges that can only be met by a partnership between owners and operators and government at all levels.” President’s Commission on Critical Infrastructure Protection 1997

6. Examples of Critical Infrastructures Energy (electric power, oil, natural gas) Telecommunications Transportation Water systems Banking and finance Emergency services Government services Agriculture Others

7. * CMU SEI Study

8. What is “Power System” 8

9. Generation 9 renewable coal natural gasnuclear

10. Transmission 10 substation power tower power pole

11. Distribution 11 residential industrial

12. What is “Smart Grid” 12 http://www.elp.com/index/display/article-display/0045209435/articles/utility-products/volume-7/issue-7/product-focus/test-__measurement/measurement-tools.html

13. Smart Grid Vision Generation: – Micro-grid – Renewable energy – Gas turbines Transmission: – Wide area monitoring – Wide area protection and control – Real-time state estimation Distribution Level: – Smart metering – Demand response – Self-healing distribution network 13

14. Communication Infrastructure 14

15. Communication Techniques Communication Link – Telephone – Microwave – Co-axial – Fiber – Power line communication Communication Network – LAN – WAN – MAN – WLAN 15

16. A Wide Area Measurement Scenario 16 Control Center

17. Motivation Smarter Grid entails more Cyber components Wide area measurement and Control Communication Infrastructure New Cyber Security Vulnerabilities Smart Grid is a Extremely Large Scale Cyber Physical System ELCPS Physical Dynamics controlled by Cyber Networked Control Attack on the networked control can lead to disastrous Physical Dynamics Need to Study ELCPS Too large for Analytical Study Scalable but Accurate Co-Simulation is needed Need for co-simulation tools Leveraging Existing Scalable Tools Study Wide Area Control issues but Security is Extremely Important to Study

18. Co-Simulation for CPS 18 Power System Simulation Cyber & Network Simulation To design a CPS system, engineers need tools to explore possible architectures, protocols, and configurations. Smart Grid engineers should be able to precisely model the power system and the communication network together so that the system behaviors can be suitably predicted. Synchronization

19. Other Power System/Cyber Co-Simulators EPOCHS: PSLF + NS2 [Cornell] DEVS method: adevs + NS2 [ORNL] PowerWorld + RINSE [UIUC] PowerWorld + OPNET [UIUC] PowerWorld + NS3 [Ga Tech] OPNET extension [Jia Tong] 19 [1] K. Hopkinson, X. Wang, R. Giovanini, J. Thorp, K. Birman, and D. Coury. Epochs: a platform for agent-based electric power and communication simulation built from commercial off-the-shelf components. [2] J. Nutaro, P. T. Kuruganti, L. Miller, S. Mullen, and M. Shankar. Integrated hybridsimulation of electric power and communications systems. In Proc. IEEE Power Engineering Society General Meeting, pages 1–8, 2007. [3] C. M. Davis, J. E. Tate, H. Okhravi, C. Grier, T. J. Overbye, and D. Nicol. Scada cybersecurity testbed development. In Proc. 38th North American Power Symp. NAPS 2006, pages 483–488, 2006. [4] D. C. T. C. Malaz Mallouhi, Youssif Al-Nashif and S. Hariri. A testbed for analyzing security of scada control systems (tasscs). In Second IEEE PES Innovative Smart Grid Technologies Conference, 2011. [5] X. Tong. The co-simulation extending for wide-area communication networks in power system. In Proc. Asia-Pacific Power and Energy Engineering Conf. (APPEEC), pages 1–4, 2010.

20. Continuous Time System Simulation Discretize differential equations and time 20

21. Power System Dynamic Simulation 21

22. Discrete Event System Simulation Occurrence of events are not uniform Event-Driven – Scheduler – Event Queue – Event Processing 22

23. Communication Network Simulation 23 1 2 4 3

24. Synchronization with errors in EPOCHS 24 Power Communication

25. Global Event-Driven Synchronization 25 Power Communication

26. Implementation of the Co-simulation Framework GECO PSLF – Power system – Written in Java – Script: EPCL 26 NS2 –Communication network –Written in C++ –Script: OTcl

27. Co-Simulation Platform Structure 27

28. GECO To Study All PMU linear state estimator Global Event-driven Co-simulation

29. Power System Protection Relays protect power systems when faults happen – Over current – Over voltage – Directional – Distance (Impedance) – Differential – Pilot 29

30. Distance Relay Protection Zones Primary: Zone 1 Backup: Zone 2, Zone 3 Time-delayed manner for backups: Zone 2(300ms), Zone 3(1s) 30

31. Problems with Backup Relays Drawbacks – Long waiting time – Over sensitivity – Hidden failures However, zone 3 is still needed 31 [1] S. Protection and C. T. Force. Rationale for the use of local and remote (zone 3) protective relaying backup systems. Technical report, North American Electric Reliability Council, 2005.

32. Network-based Backup Relay Protection Backup distance relays proactively communicate with other relays to obtain wider system visibility and make global protection decision – Software agents take control – Supervisory (master - slave) – Ad-hoc (peer - peer) 32

33. Supervisory Protection Scheme 33

34. Supervisory Scheme Operation (Slave) 34

35. Supervisory Scheme Operation (Master) 35

36. Ad-Hoc Protection Scheme 36

37. Ad-Hoc Scheme Operation (Peer) 37

38. Relay Searching Find the responsible relay group 38

39. Searching Algorithm 39

40. Decision Making Decision is made by “OR” manner voting Upper and lower time threshold 40

41. Co-Simulation Settings New England 39-bus system Communication network share same topology with power system 100Mbps bandwidth and 3ms latency for each communication link Without background traffic 41

42. Supervisory Protection on 39-bus System (Case 1) 42

43. Robustness against primary failure 43 AMS

44. Supervisory Protection on 39-bus System (Case 2) 44

45. Robustness against hidden failure 45 AMS

46. Supervisory Protection Communication Delay 46 Relay Agent ID

47. Supervisory Protection Communication Delay Analysis 47

48. Ad-hoc Protection Communication Delay 48 Relay Agent ID

49. Supervisory Protection with Link Failure 49 Relay Agent ID

50. Supervisory Protection Delay with Link Failure 50

51. Ad-hoc Protection with Link Failure 51 Relay Agent ID

52. Comparison Real system implementation – Supervisory: extra master agent needed – Ad-hoc: peer relays store system information locally – Hybrid mode Reaction time – Supervisory: long, uneven – Ad-hoc: short, even Robustness to network failures – Supervisory: increase by 20%-100% – Ad-hoc: increase by multiple times 52

53. Outline Motivation Need for Co-Simulation GECO – Our Co-simulator Relay Case Study All PMU-State Estimator – Experimental Framework – Experimental Results and Interpretations Conclusions

54. Power System State Estimation Conventional – Slow scanning rate – Power injection, power flow, voltage magnitude – Non-linear, iterative solution All-PMU – 30 times/sec – Complex voltage and current – Linear, non-iterative solution

55. Cyber Security Considerations All-PMU state estimation is superior than conventional ones. But it can still be vulnerable to cyber attacks or network failures. – Intranet not completely safe – Many conceivable threat models

56. WAMS Infrastructure 56 Timerto catch up measurement rate

57. Outline Motivation Need for Co-Simulation GECO – Our Co-simulator Relay Case Study All PMU-State Estimator – Experimental Framework – Experimental Results and Interpretations Conclusions

58. New England 39-bus System 58 Area 1 Area 2 Area 3 Area 4

59. Co-Simulation Settings 59 [1] Kun Zhu, M. Chenine, and L. Nordstrom. ICT architecture impact on wide area monitoring and control systems’ reliability. 26(4):2801–2808, 2011. [1]

60. Outline Motivation Need for Co-Simulation GECO – Our Co-simulator Relay Case Study All PMU-State Estimator – Experimental Framework – Experimental Results and Interpretations Conclusions

61. Co-Simulation Results Use estimated voltage at Bus 3 to represent if the estimation is done successfully Attacks at critical locations to show typical vulnerability

62. Network Link Failure at Bus16-Bus17 (Tp=50ms) 62

63. Network Link Failure at Bus16-Bus17 (Tp=60ms) 63

64. Network Link Congestion at Bus16-Bus17 64

65. Router Congestion: Bus16 65

66. Data Spoofing: Bus 3 66

67. Data Spoofing : Bus 3 (with a Real Fault) 67

68. Outline Motivation Need for Co-Simulation GECO – Our Co-simulator Relay Case Study All PMU-State Estimator – Experimental Framework – Experimental Results and Interpretations Conclusions

69. Smart Grid is an ELCPS Cyber Security Vulnerability for WAMS applications must be studied in depth Co-Simulation is a good way to study Smart Grid applications GECO is built for such studies These case studies enhanced our confidence in GECO as a tool to study new smart grid protocols and cyber security impacts on smart grid Can we draw any general conclusions? – Possibly not without stretching our imagination – Need for identifying critical bottle neck links and nodes and safe guarding them – Further studies needed to develop More threat models Defense mechanisms against threat models