1. NERC Lessons Learned Summary March 2015

2. NERC lessons learned published in March 2015 Two NERC lessons learned (LL) were published in March2015 LL20150301 Importance of Backup Energy Management System Failover Testing after Network Device Reconfiguration LL20150302 Importance of State Estimator save Cases and Troubleshooting Guide

3. Importance of Backup Energy Management System Failover Testing after Network Device Reconfiguration Performing maintenance on the primary control center’s (PCC) uninterruptable power supply (UPS), all functionality of the EMS system, including the communication circuits, were successfully transferred from the PCC to the alternate control center (ACC) After maintenance was complete EMS analysts attempted to bring system functionality back to the PCC but the attempt was unsuccessful Full system restart was then performed to establish system functionality at the PCC

4. Importance of Backup Energy Management System Failover Testing after Network Device Reconfiguration PCC came on-line, but the ACC failed, and EMS analysts were unable to restore ACC functionality EMS functionality was lost due to communication circuits still connected to the failed ACC It should be noted that an EMS loss-of-functionality event will occur every time a full system restart is performed, and the duration of a typical event is five to eight minutes. Loss of EMS/SCADA functionality for 49 minutes during a scheduled transfer of the EMS from the alternate control center (ACC) to the primary control center (PCC)

5. Importance of Backup Energy Management System Failover Testing after Network Device Reconfiguration Upon subsequent investigation, it was discovered that due to a recent and extensive network device reconfiguration, one of the parameters was in error, and this resulted in the inability to restore ACC functionality via a full system restart Device configuration modifications were then performed, and the ACC was successfully restarted with functionality restored

6. Importance of Backup Energy Management System Failover Testing after Network Device Reconfiguration EMS maintenance and reconfiguration operations should be closely coordinated with vendors that are needed to support the changes to ensure that there are no overlaps in planned maintenance schedules Succinct and accurate communication between registered entities and vendors is essential to ensure that both parties fully understand their roles and obligations during planned maintenance operations Procedures for EMS system restart operations should be rigorously documented to ensure that the EMS can be restarted in the most rapid and secure manner

7. Importance of Backup Energy Management System Failover Testing after Network Device Reconfiguration Step-by-step checklist for the procedure is desirable to ensure that no steps are overlooked Entities should periodically review EMS redundancy to ensure ongoing independence between sites, including full functional failover testing. Entities should review and identify the extent of testing to be performed following significant EMS infrastructure reconfiguration.

8. Importance of State Estimator save Cases and Troubleshooting Guide State estimator (SE) failed to solve for 37 minutes, resulting in real- time contingency analysis (CA)also being unavailable During this event, operators had system visibility via SCADA and could still take control actions, including the ability to shed load Inter-Control Center Communications Protocol (ICCP) continued to function, providing real-time data to the RC and the other local entities in the RC’s footprint Entity confirmed with its RC that the RC’s SE and provide real-time CA

9. Importance of State Estimator save Cases and Troubleshooting Guide A SE should be able to automatically and frequently save cases to assist in post-event analysis. It should also automatically save non convergent cases Operators and support staff should have clear guidance and training on troubleshooting state estimator failures. An online state estimator guide for systems operators should be available to ensure consistent troubleshooting. Periodic refresher training should also be employed, including reviews of recent aborted cases. A joint review of state estimator issues with other entities should be periodically conducted to ensure applicable common solutions are implemented.

10. Link to Lessons Learned Directions to Lessons Learned: Go to www.NERC.com > “Program Areas & Departments” tab > “Reliability Risk Management” (left side menu) > “Event Analysis” (left side menu) > “Lessons Learned” (left side menu) NERC’s goal with publishing lessons learned is to provide industry with technical and understandable information that assists them with maintaining the reliability of the bulk power system. NERC requests that industry provide input on lessons learned by taking the short survey. The survey link is provided on each Lesson Learned.

11. Questions?