NERC Published Lessons Learned Summary November 2016

NERC Lessons Learned - November 2016 Three NERC Lessons Learned (LL) were published in the November 2016 LL20161101 - “Redundant Systems may not Cold-Start unless fully intact to prevent Dual Primary Operation” LL20161102 - “Failover Configuration Leads to Loss of EMS” LL20161103 - “Loss of ICCP due to Database Sizing Issue”

Redundant Systems may not Cold-Start – Details The entity deployed a patch on several noncritical but similar hardware installations throughout the organization in addition to the identical QA/test EMS When the patch was executed on the production EMS communications halted The failure was attributed to a corrupt switch configuration (part of a redundant pair) The second switch’s configuration was verified intact, and the upgrade process had not yet executed on the second switch

Redundant Systems may not Cold-Start – Details The freshly upgraded switch was powered down and traffic was expected to resume to normal – Did not The switches were power cycled in various orders in an attempt to restore service – Still not working Once the corruption/missing configuration was identified, it was restored from backup and system functionality resumed immediately

Redundant Systems may not Cold-Start – Details The network switches will not forward traffic when the two units do not have matching configuration A single switch would not start without its mate in a cold- start situation This is to prevent dual primary operation (split-brain scenario), where two isolated switches each think that they are the only operating switch

Redundant Systems may not Cold-Start – Corrective Actions Vendor modified their recommended configuration baseline to include the ability to cold-start a single switch after a waiting period This balances dual primary protection (split-brain scenario) with the operational need to start a system using a single network switch Engineers now have processes to quickly compare configurations with known good baselines during maintenance operations Standard commissioning and testing procedures include cold-starting redundant systems when the mate is not present

Redundant Systems may not Cold-Start – Corrective Actions These concepts redundant system must be understood at all levels of technicians, operators, and engineers of redundant systems See lesson learned document for details In addition to common testing, redundant systems should be tested with partial outages Ensure backups and disaster recovery procedures are readily available before performing maintenance

Failover Configuration Leads to Loss of EMS– Details A failover was initiated from EMS-C server to EMS-D server, the AGC application aborted twice within a minute during EMS-D’s initialization/startup This caused an automatic failover to the next backup EMS server in line (EMS-A) The same condition was experienced by EMS-A, which initiated another automatic failover to the next backup EMS server in line (EMS-B) When the system reached the final available server (EMS-B), all systems were in a DISABLED state

Failover Configuration Leads to Loss of EMS– Corrective Actions The entity removed the scheme that initiated an automatic failover after two consecutive AGC failures (within a minute) from the EMS process manager model The entity also reviewed all other schemes to ensure that the triggering of an automatic failover is properly defined

Failover Configuration Leads to Loss of EMS– Lessons Learned Review all failover configuration settings in the EMS that could initiate an automatic failover of the EMS to determine the value of the scheme Remove schemes that are not necessary or could lead to a cascading failover scenario Evaluate whether to allow these applications to fail in lieu of automatic failovers

Loss of ICCP due to Database Sizing Issue – Details An entity was updating and expanding it’s state estimator (STE) network model This STE update required an additional 13,000 points from the ICCP These had already been added to the ICCP database and to the development and staging STE servers When the production STE server was updated, it began requesting the 13,000 additional points again and the database table was increased by 26,000 points (13,000 for each of the two production STE servers)

Loss of ICCP due to Database Sizing Issue – Details This exceeded the maximum allowed size of the database table and caused the ICCP processes to abort Investigation revealed that the database table was 97% full prior to the expansion, and the extra points caused it to exceed its maximum size A failback to the previous ICCP database was attempted, but this did not successfully resolve the issue

Loss of ICCP due to Database Sizing Issue –Corrective Action The database table was temporarily resized to accommodate the appropriate number of entries More extensive research was done to determine the total size that will be needed through the end of the network model expansion project A report was created to compare the current size of database tables to their maximum limit This is now reviewed at each ICCP database change Two ICCP support staff completed a vendor class on ICCP support and maintenance

Loss of ICCP due to Database Sizing Issue – Lessons learned Database sizes need to be carefully monitored as a system is expanded Sizes should be large enough to accommodate all data being requested, not just what is currently being transferred Primary databases, as well as peripherally associated databases, need to be evaluated for size constraints The vendor may need to be contacted to verify that database sizes can be increased without causing problems or to provide a more comprehensive validation routine Alternative ICCP configurations need to be evaluated to determine if there is a more efficient means to feed data into the staging and development systems

Loss of ICCP due to Database Sizing Issue – Lessons Learned ICCP databases should be set up so that external companies cannot inadvertently request data that does not originate in the host utility Backup support staff should be fully trained so that discovery of problems does not rest completely with primary support personnel Support staff should meet regularly to discuss questions, discoveries, and findings

Lessons Learned Survey Link NERC’s goal with publishing lessons learned is to provide industry with technical and understandable information that assists them with maintaining the reliability of the bulk power system NERC requests that industry provide input on lessons learned by taking the short survey. A link is provided in the PDF version of each Lesson Learned