1. 1 Lessons Learned 2011 Cold Snap Event System Operator Conference September 11-13 and September 25-27, 2012 Memphis, TN Steve Corbin Reliability Specialist SERC Reliability Corp.

2. 2012 System Operator Conference 2 At the conclusion of this training session, you should be able to: Give examples of preparations needed for impending cold weather at a generating plant Identify issues an operator must know about manual load shedding Discuss the importance of Lessons Learned Recommendations relative to institutionalizing and training on such recommendations. Learning Objectives

3. 2012 System Operator Conference 3 Outline of Topics Highlights of previous winter events in ERCOT Highlights of the cold-weather events of February 1, 2, 2011 in the Southwest Review the 4 of the 8 NERC published Lessons Learned from the 2011 cold snap event

4. 2012 System Operator Conference 4 Cold Front The arctic cold front that descended on the Southwest during the first week of February 2011 was indisputably severe. Many cities in Texas and New Mexico experienced a 50 degree drop in temperature over an eighteen-hour period. Temperatures dropped to the low teens in Texas and below zero in New Mexico. Much of north Texas experienced record setting sleet and snow, totaling up to seven inches. Exacerbating the effects of the cold temperatures were accompanying sustained winds of 30-40 mph, with gusts as high as 51 mph.

5. 2012 System Operator Conference 5 This had happened before, no sweat 1983, 1989, 2003, 2006, 2008, and 2010

6. 2012 System Operator Conference 6 December 21-23, 1989February 1-2, 2011 Min. Temps & Wind Chills in Dallas Area Temperature: -1 degrees F Wind Chill: -12 degrees F Temperature: 13 degrees F Wind Chill: -6 degrees F Peak System Load38,300 MW56,334 MW Net Generating Capacity Reduction 11,809 MW 31% of peak load 14,702 MW 26% of peak load Gross Generating Capacity Reduction 56+ units 16,805 MW 193 units 29,729 MW Firm Load Shed1710 MW 4.5% of peak load 4900 MW 8.7% of peak load Overall Duration of Firm Load Shedding 5 hours, 47 minutes7 hours, 24 minutes

7. 2012 System Operator Conference 7 PUCT Recommendations in 1989 All utilities should ensure that they incorporate the lessons learned during December of 1989 into the design of new facilities in order to ensure their reliability in extreme weather conditions. All utilities should implement procedures requiring a timely annual (each Fall) review of unit equipment and procedures to ensure readiness for cold weather operations.

8. 2012 System Operator Conference 8 PUCT Recommendations in 1989 All utilities should ensure that procedures are implemented to correct defective freeze protection equipment prior to the onset of cold weather. All utilities should maintain insulation integrity and heat tracing systems in proper working order. Generating unit control systems and equipment essential to cold weather operations should be included in a correctly managed preventive maintenance program. Additional training programs for plant personnel on the emergency cold weather procedures, including periodic drills, should be implemented by each responsible utility.

9. 2012 System Operator Conference 9 PUCT Recommendations in 1989 PUC Engineering Staff should modify procedures for power plant CCN [Certificates of Convenience and Necessity] reviews to include a specific review for plant reliability under adverse weather conditions. Of special interest would be the selection of proper design temperature ranges for the power plant site.

10. 2012 System Operator Conference 10 PUCT Recommendations of 2003 The Market Oversight/Reliability Division of the PUCT investigated the 2003 cold weather event and issued a number of recommendations. Notable among these are the following: –Stricter enforcement of Resource Plan accuracy. –Improved weather and electric demand forecasting. –Consider providing financial incentives for fuel oil inventories to be maintained for use by dual fueled units. –Curtailment prioritization – development of a joint curtailment methodology for natural gas and electricity production. –ERCOT should communicate with both QSEs and Transmission / Distribution Service Providers in the future when the power system is under stress.

11. 2012 System Operator Conference 11 2011 Lessons Learned 4 LLs Reviewed of 8 Currently Posted by NERC Rotational load shed Transmission facilities and winter weather operations Plant fuel switching and cold weather Generating unit temperature design parameters and extreme weather conditions

12. 2012 System Operator Conference 12 Rotational Load Shed Primary Interest Groups (PIGs): TOP, LSE, DP, TO Problem Statement Registered entities implemented rolling blackouts in response to directives issued by the reliability coordinator (RC) during an Energy Emergency Alert (EEA) event. During the rolling blackouts, operators experienced difficulties that may have been avoided with the use of improved operator tools and processes.

13. 2012 System Operator Conference 13 Rotational Load Shed Details The following challenges and issues were identified: SCADA alarms were not distinguishable between non-rolling blackout circuits and rolling blackout circuits. The number of circuits available for rolling blackouts was limited by exempt customers, critical loads, and under-frequency or under-voltage load-shedding circuits. Several circuits were not correctly identified as critical load, including hospitals and jails Issues were identified with the long duration of the rolling blackouts for some individual circuits, i.e. there were problems with re-energizing circuits due to high cold-load pickup inrush currents

14. 2012 System Operator Conference 14 Rotational Load Shed - Details Real-time communications were primarily by phone, making it difficult to keep circuit status documented. Entities used manual spreadsheets to track and update rolling blackout circuit status and had trouble keeping the spreadsheets updated. Maps of area outages were not immediately available to meet the needs of management, stakeholders, and media. Communications processes were not fully defined or fully implemented with other local government agencies, local law enforcement, local emergency management teams, and other critical facilities.

15. 2012 System Operator Conference 15 Rotational Load Shed Lesson Learned Periodic system operator training proved to be invaluable for this event. No issues were identified due to lack of personnel training. Load-Serving Entities (LSE) should ensure they have identified enough load circuits that can be used in manual load sheds to meet the needs of the area load shed, to meet the regions criteria for under-frequency relaying requirements and to protect critical loads. These lists should be reviewed and update periodically.

16. 2012 System Operator Conference 16 Rotational Load Shed Lessons Learned, continued Operational tools and EMS SCADA displays which show and track load shed outages and outage durations must be available to system operators to help manage load shed events. Registered entities must have procedures in place to periodically review and update circuit data for load shed events, to keep critical customer lists updated and to ensure under-frequency or under-voltage load-shed circuit lists are updated.

17. 2012 System Operator Conference 17 Rotational Load Shed Lessons Learned, continued Registered entities should have fully defined communication processes with other local government agencies, local law enforcement, and local emergency management teams. Periodic exercises should be in-place to test these communication processes. Maps showing outage areas should be readily available for management, stakeholders and media. It is critical in such an event that these entities are kept updated. This responsibility should be performed by support personnel so as not to distract operators from operations.

18. 2012 System Operator Conference 18 Transmission Facilities and Winter Weather Operations PIGs: GO, GOP, TO, TOP Problem Statement Unit 1 at a 1700 Mw plant was forced off line during a severe weather event due to the failure of the disconnect "B" phase switch.

19. 2012 System Operator Conference 19 Transmission Facilities and Winter Weather Operations Details After a winter storm at a generating plant, the "B" phase disconnect switch connection started to arc in the plant switchyard. The Control Room was notified immediately and operations manually tripped the turbine and took the generator off line. It was found that the arc originated toward the pivot point of the switch just inside the contact region. The disconnect switch did not arc in the contact area of the switch. Once electricians removed the disconnect switch arm it was found to be full of ice.

20. 2012 System Operator Conference 20 Transmission Facilities and Winter Weather Operations Details, continued Under normal circumstances water should not be inside the disconnect switch arm. Water, which had ingress in the disconnect arm, froze during the cold weather, causing the weld on the end of the arm to crack and reduced the current carrying capabilities of the arm. This added heat to the area, which progressed to an arc.

21. 2012 System Operator Conference 21 Transmission Facilities and Winter Weather Operations Lessons Learned All plant and transmission electrical disconnect equipment should be inspected for areas where water could collect and freeze – especially in geographic areas not accustomed to freezing temperatures. Any areas found with the potential to retain moisture should be modified as prescribed by the equipment manufacturer or by an adequate engineer design. With regards to transmission and substation equipment, other lessons which the GO, GOP, TO and TOP should consider in their winter preparation plans are:

22. 2012 System Operator Conference 22 Transmission Facilities and Winter Weather Operations Lessons Learned, continued Power transformers should have their equipment checked to ensure: –Control cabinet heaters are working correctly –Oil is at its appropriate level for the oil temperature. –Bushing oil levels are appropriate –Nitrogen levels are correct –SF6 gas ‐ insulated devices should be checked to ensure: SF6 gas in breakers, current transformers and other electrical equipment is at the correct pressure and temperature to operate safely during extreme cold

23. 2012 System Operator Conference 23 Transmission Facilities and Winter Weather Operations –Heaters (when so ‐ equipped) and associated support circuits are functioning correctly The minimum operating ambient temperature should be known on all transmission facilities. This temperature should also take into account fire protection systems. Lessons Learned, continued

24. 2012 System Operator Conference 24 Plant Fuel Switching and Cold Weather PIGs: RC, BA, GO, GOP Problem Statement During a cold weather event, both equipment freezing and infrequent use of alternative fuels caused a plant’s generation to become unreliable, contributing to the BA and RC having to shed load.

25. 2012 System Operator Conference 25 Plant Fuel Switching and Cold Weather Details A two unit 900 MW natural gas powered plant developed significant fuel switching issues and unit reliability problems during a severe winter weather event. The plant experienced temperatures low as 18 degrees (F), wind chill as low as 1 degree (F). For three days, the daytime temperatures stayed below freezing. During the winter weather event, the plant’s gas supply was interrupted.

26. 2012 System Operator Conference 26 Plant Fuel Switching and Cold Weather Operation on fuel oil was infrequent: –fuel oil burning equipment remained dormant much of the time. –As a result, equipment required for burning fuel oil had high failure rates, in part due to sub ‐ freezing temperatures, but also due to infrequent use.

27. 2012 System Operator Conference 27 Plant Fuel Switching and Cold Weather The plant experienced: –Instrumentation sensor lines freezing –Condensation freezing in the air inlet needle valve for a Gas Turbine (GT) –Reduced unit output due to gas transmission curtailments –Failed oil cooler fan due to freezing issues –Unit tripping multiple times when attempting to fire on fuel oil due to equipment malfunctioning and operator error

28. 2012 System Operator Conference 28 Plant Fuel Switching and Cold Weather Lessons Learned GOs and GOPs should review their winter maintenance plans and ensure: –critical sensing and operating equipment are protected and can operate at the plant’s designed minimum operating ambient temperature for all fuels the plant is capable of burning.

29. 2012 System Operator Conference 29 Plant Fuel Switching and Cold Weather Lessons Learned, continued GO/GOPs should consider the following if they have dual fuel operation capability: –Reliable unit fuel oil operation requires dormant equipment to be tested and maintained. Entities should evaluate their testing programs to insure all systems can operate reliably at all times and provide training to appropriate personnel on the maintaining and operation of the equipment.

30. 2012 System Operator Conference 30 Plant Fuel Switching and Cold Weather Lessons Learned, continued GO/GOPs should consider the following if they have dual fuel operation capability: –Perform review and update of procedures periodically for unit fuel oil operation and maintenance based upon expected operation capabilities to ensure reliable operation. Training plans: –consider incorporation of processes to institutionalize the knowledge of procedures for fuel switching and operation for the non primary fuel mode of operation.

31. 2012 System Operator Conference 31 Plant Fuel Switching and Cold Weather Alternate fuel supplies should be tested for contamination or degradation BAs and RCs should consider the following if they have plants which have dual fuel capability: –Verification of the output of each fuel capability provides more accurate unit commitment planning. –Verification might include the following: Document time to switch Document unit capacity and maximum runtimes while on alternate fuel

32. 2012 System Operator Conference 32 Generating Unit Temperature Design Parameters and Extreme Winter Conditions PIGs: GO, GOP, RC, BA Problem Statement Extreme low winter temperatures forced a plant offline which had been providing over 1000 Mw. The plant was not designed to operate under such conditions. The Balancing Authority (BA) and Reliability Coordinator (RC) were not informed that the plant was not designed to operate during the low temperature conditions.

33. 2012 System Operator Conference 33 Generating Unit Temperature Design Parameters and Extreme Winter Conditions Details During a period of extreme cold, a large power plant experienced temperatures below freezing for as much as 100 hours. –Sustained winds of over 20 mph with gusts much higher created wind chills in the single digits. These weather conditions had been predicted a week earlier by weather services in long range forecasts. The plant had taken some precautions before the cold period to protect equipment. Precaution steps included applying covering at various places, using heating equipment and adding items such as checks of equipment for freezing to operator “rounds” checklists, etc.

34. 2012 System Operator Conference 34 Generating Unit Temperature Design Parameters and Extreme Winter Conditions Yet the plant was not designed to handle these temperatures and had numerous freezing issues. Critical pieces of equipment froze. For example: –the plant had instrumentation freeze on its Zero Liquid Discharge Facility (ZLDF) which stopped it from making demineralized water. –a freezing regulator on one unit caused oil to leak out of the seal on the generator. Oil from the seal got on the field brushes which started arcing. Units, some of which had just started to come online, tripped offline.

35. 2012 System Operator Conference 35 Generating Unit Temperature Design Parameters and Extreme Winter Conditions –The plant implemented further freeze protection such as: operating heaters for wind-sensitive transmitters continuously to stop further freezing adding more tarps as windbreaks for the ZLDF tower. obtaining demineralization trailers from various sources in other states, bringing them in to replace their ZLDF.

36. 2012 System Operator Conference 36 Generating Unit Temperature Design Parameters and Extreme Winter Conditions –The BA and RC did not know this plant had these low temperature design limitations and were counting on the plant for its generation to help meet the cold weather load peaks. –As a result of the loss of this generation and other generation, the BA and RC were forced to implement rolling load sheds in order to reliability manage the load.

37. 2012 System Operator Conference 37 Generating Unit Temperature Design Parameters and Extreme Winter Conditions Lesson Learned Generator Owner/Operator should: Know the ambient temperature limitations of their plant If they don’t know……. –they should perform a detailed engineering design analysis of their plants to determine their ability to operate during extreme cold conditions. –A low ambient temperature limit according to the design of the plant should be established.

38. 2012 System Operator Conference 38 Generating Unit Temperature Design Parameters and Extreme Winter Conditions Lesson Learned continued Generator Owner/Operator should: Fix issues which do not allow the plant to operate at extreme low ambient temperature: –Identify freezing places and ensure insulation, heat tracing, lagging and wind breaks are sufficient to maintain water temperature at 40 degrees or above. –This practice should also be performed to protect the plant if it is offline.

39. 2012 System Operator Conference 39 Generating Unit Temperature Design Parameters and Extreme Winter Conditions Establish ongoing maintenance practices which stress identification of possible freeze areas after maintenance or design changes Ensure those areas can meet the requirements to continue to operate during low ambient temperature periods. If ambient air temperatures exist which could cause a unit to runback its output, these temperature limitations should be documented and procedures and training modified accordingly.

40. 2012 System Operator Conference 40 Generating Unit Temperature Design Parameters and Extreme Winter Conditions Work with the BA and RC to establish communications practices by which the GO and GOP informs the BA and RC of the plant design low ambient temperature limitation and when the plant cannot meet the design specification. These communications should include informing the BA and RC if weather conditions dictate that the plant must be kept online to be able to operate the next day. Include such notification in winter operation drills which involve the plants and other entities.

41. 2012 System Operator Conference 41 Generating Unit Temperature Design Parameters and Extreme Winter Conditions AND FINALLY…….. Document and institutionalize knowledge and experiences from previous severe winter weather events, and apply this learning to the winterization procedures of plant equipment. Severe Weather Procedures

42. 2012 System Operator Conference 42 Summary Some examples of recommended preparation for impending cold weather at a generating plant: Perform analyses to determine low ambient temperature limit Identify freezing places and ensure sufficient insulation & other fixes Identify sufficient manual load shed circuits Test alternate fuel supplies Conduct cold weather drills for operators Work with the BA and RC to establish communications practices regarding temperature limitations, and when the plant cannot meet design specs. It is important to learn from past events – document & institutionalize your severe weather procedures based on lessons learned!

43. 2012 System Operator Conference 43 Questions?