1. August 14, 2003 Blackout Summary Based on Interim Report of the
United States – Canada Power Outage Task Force
November 19, 2003

2. U.S.-Canada Interim Report
Released November 19, 2003
Result of an exhaustive bi-national investigation
Working groups on electric system, nuclear plant performance and security
Hundreds of professionals on investigation teams performed extensive analysis
Interim report produced by the teams and accepted by the bi-national Task Force

3. Overview
Overview of power system and reliability
Pre-outage conditions on August 14
Trigger events and start of cascade
Wide area cascade
Root causes
Next steps

4. Power System Overview

5. Reliably operate the system you have!
Reliability Overview
Balance generation and demand
Balance reactive power supply and demand
Monitor flows and observe thermal limits
Observe power and voltage stability limits
Operate for unplanned contingencies
Plan, design and maintain a reliable system
Prepare for emergencies
Reliably operate the system you have!

6. 3 Interconnections / 10 NERC Regions

7. NERC Control Areas

8. NERC Reliability Coordinators

9. Footprints of Reliability Coordinators in Midwest

10. NERC Immediate Response to Blackout
First hours
Worked closely with NERC Reliability Coordinators
Identified what had tripped and extent of outage
Assessed restoration efforts
Maintained open line with DOE/FERC
Communicated with DHS, White House, and NRC
First days
Assigned project manager
Established Steering Group with industry executive experts
Began organizing investigation teams
90+ volunteers + entire NERC staff

11. Investigation Organization Overview
Steering Group
U.S – Canada
Task Force
Investigation Team Lead
Project Planning and Support
Root Cause Analysis
Cooper Systems
Investigation Process Review
Vegetation/ROW Management
NERC & Regional Standards/Procedures & Compliance
Sequence of Events
Transmission System Performance, Protection, Control
Maintenance & Damage
MAAC/ECAR/NPCC
Coordinating Group
MAAC
Restoration
Operations - Tools, SCADA/EMS Communications Op Planning
Generator Performance, Protection, Controls
Maintenance & Damage
ECAR
Data Requests and Management
Frequency/ACE
NPCC
System Modeling and Simulation Analysis
System Planning, Design, & Studies
MEN Study Group

12. Data Gathering and Analysis
Three fact-finding meetings
August 22
September 8-9
October 1-3
Onsite interviews and inspections
Secure database of outage information
Extensive corroboration of data to determine facts
Analysis by teams of technical experts

13. Root Cause Analysis
Logical structure for investigating complex problems
Identifies changes, conditions, actions, or inactions at each causal step
Starts with final event and drills back through each branch of causal tree.
Asks “why?” at each step.
Accurate, reliable, defensible understanding of the root causes.
Successfully used to investigate root causes of PJM voltage stability condition
in July and established history in nuclear and defense industries.

14. Root Cause Analysis Phases
16:15
BLACKOUT
16:06
Sammis – Star
Star – South Canton
Hanna – Juniper
Chamberlin - Harding
Initial Focus
15:05
Pre-Existing Conditions
E.g. voltages, wide- area transfers,
line and generator outages, etc.

15. August 14 Conditions Prior to Blackout
Planned outages
Cook 2, Davis Besse nuclear plants
East Lake 4, and Monroe 1
Transfers high to northeast U.S. + Ontario
Not unusually so and not above transfer limits
Critical voltage day
Voltages within limits
Operators taking action to boost voltages
Frequency
Typical for a summer day
System was within limits prior to 15:05, on both actual and contingency basis

16. Warm But Not Unusual for August

17. August 14 Imports to Northeast-Central Compared to 6/1 to 8/13/2003

18. Voltages Prior to 15:05 EDT August 14

19. Frequency Typical for Summer Day

20. Blackout was NOT Caused by
Heavy wide-area transfers
Low voltages, voltage collapse
Lack of IPP voltage/reactive support
Frequency anomalies
Cinergy outages starting at 12:08
East Lake 5 trip at 13:31
Contributing factor to later events, but not by itself causal to the blackout
DPL Stuart-Atlanta trip at 14:02
Contributing factor to loss of MISO real-time monitoring, but not electrically significant

21. Outage Sequence of Events Transmission Map Key

22. East Lake 5 Trip: 1:31:34 PM
ONTARIO
ONTARIO 2 1
There were several incidents occurring earlier in the day on August 14. However, the investigation has not found any of them to be electrically related to the blackout. The first significant event setting the stage for the cascade is at 1:31:34 when the East Lake 5 generating unit trips while the unit operator is attempting to increase reactive output. The East Lake 5 forced outage causes increased flows into northern Ohio over transmission paths into the area.

23. East Lake 5 Exciter Failure Causes Trip

24. Stuart Atlanta Trip: 2:02 PM
In the Dayton Power & Light system about 2:02 pm, the Stuart-Atlanta 345 kV line trips and locks out due to a ground fault. The loss of Stuart-Atlanta is not electrically connected to the start of the cascade, but it is noteworthy later from an operational perspective.
At 2:27:16, in a precursor of subsequent trips, the Star-South Canton 345 kV line trips on a ground fault and recloses.

25. MISO State Estimator and Reliability Analysis
MISO state estimator and contingency analysis ineffective from 12:37 to 16:04
State estimator not solving due to missing information on lines out in Cinergy then DPL
Human error in not resetting SE automatic trigger
Using Flowgate Monitoring tool to monitor conditions on previously identified critical flowgates

26. FirstEnergy Computer Failures
14:14 Alarm logger fails and operators are not aware
No further alarms to FE operators
14:20 Several remote consoles fail
14:41 EMS server hosting alarm processor and other functions fails to backup
14:54 Backup server fails
EMS continues to function but with very degraded performance (59 second refresh)
FE system data passed normally to others: MISO and AEP
AGC function degraded and strip charts flat-lined
15:08 IT warm reboot of EMS appears to work but alarm process not tested and still in failed condition
No contingency analysis of events during the day including loss of East Lake 5 and subsequent line trips

27. Phone Calls to FirstEnergy
FE received calls from MISO, AEP, and PJM indicating problems on the FE system but did not recognize evolving emergency
14:32 AEP calls regarding trip and reclose of Star-S. Canton
15:19 AEP calls again confirming Star-S. Canton trip and reclose
15:35 Calls received about “spikes” seen on system
15:36 MISO calls FE regarding contingency overload on Star-Juniper for loss of Hanna-Juniper
15:45 FE tree trimming crew calls in regarding Hanna-Juniper flashover to a tree
PJM called MISO at 15:48 and FE at 15:56 regarding overloads on FE system

28. Chamberlin-Harding (3:05:41)
At 3:05 a sequence of three key events begins and sets the trigger for the cascade.
First, at 3:05:41 the Chamberlin-Harding 345 kV line in FirstEnergy trips and locks out. The trip was caused by a high impedance ground fault.

29. Chamberlin-Harding Indication of Ground Fault Due to Tree Contact as Measured by DFR at Juniper

30. Hanna-Juniper (3:05:41) (3:32:03)
At 3:32:03 the Hanna-Juniper 345 kV line in FirstEnergy trips and locks out. The Hanna-Juniper line trips due to contact with a tree, visually confirmed by a tree-trimming crew working a few spans away who saw the flash.

31. Hanna Juniper Confirmed as Tree Contact at Less than Emergency Ratings of Line

32. Effects of Ambient Conditions on Ratings

33. (3:05:41) (3:32:03) Star- S. Canton (3:41:35)
At 3:38:48 the Star-South Canton 345 kV trips and recloses again.
At 3:41:33-35 the Star-South Canton 345 kV line trips and recloses; then trips a final time and locks out. Each trip of the Star-South Canton line is caused by a short circuit to ground.

34. Situation after Initial Trips 3:05:41 – 3:41:35
ONTARIO
With the loss of the Chamberlin-Harding, Hanna-Juniper, and Star-South Canton 345 kV lines, three critical paths delivering heavy imports into northern Ohio are lost, aggravated by the prior loss of the East Lake 5 unit. The power that had been flowing over these 345 kV lines is now flowing over the underlying 138 kV lines in this area, and these lines quickly become overloaded. At 3:42, 36 minutes after the trip of Chamberlin-Harding, the cascade is electrically set to happen with no further triggers required, although intervening actions still appeared possible at this time.

35. Canton Central – Tidd (3:45:41)
At 3:45:41 the Canton Central – Tidd 345 kV line trips and recloses. Several operations of a transformer circuit breaker at Canton Central cause low air pressure on the breaker. As a result, the 345/138 kV transformer is disconnected and remains open at Canton Central.
At 3:39 a “local cascade” of 138 kV lines begins in the Akron area, resulting from line overloads. The lines are rapidly overheating and sagging into obstructions, with most tripping on ground faults.
Following the trip of one 138 kV line at 3:59, a circuit breaker failure on the West Akron Transformer #1 causes a 138 kV bus trip and the five remaining 138 kV lines connected to West Akron are tripped.
In a 30 minute period from 3:39 to 4:09, a total of sixteen 138 kV lines are lost in the Akron area. In this period, voltages are severely degraded in the area and about 600 MW of load is lost in Akron and areas to the south and west of Akron.
Canton Central – Tidd
(3:45:41)

36. 138 kV Lines Overload and Cascade Near Akron

37. 138 kV Cascade Contributes Further to Overload of Sammis-Star
15:51:41 EDT
15:05:41 EDT
15:32:03 EDT
16:05:55 EDT
15:41:35 EDT

38. Sammis-Star
(4:05:57.5)
At 4:05:57, having become incrementally overloaded with the loss of underlying 138 kV lines over the prior 30 minutes, the Sammis-Star 345 kV line trips and locks out. The line trips in a steady state overload with the operation of a Zone 3 impedance relay. The Sammis-Star trip is a significant turning point for two reasons. First, this is the first 345 kV trip not caused by a ground fault. The line tripped due to an “apparent overload” – the impedance relay seeing a combination of increased current and sustained low voltage. Secondly, the trip of Sammis-Star is a turning point because it truly is the beginning point of a rapid, uncontrolled wide-area cascade. After Sammis-Star, there is likely nothing that could have been practically done to prevent the wide-area cascade with the electrical system and capabilities that were in place on that day .

39. Sammis-Star Zone 3 Relay Operates on Steady State Overload

40. Actual Loading on Critical Lines

41. Actual Voltages Leading to Sammis-Star

42. Major Path to Cleveland Blocked after Loss of Sammis-Star 4:05:57.5 PM
Remaining
Paths
With the 345 kV and 138 kV paths from eastern Ohio blocked, the still significant demand for imports in the northern Ohio area are met by lines along the southern shore of Lake Erie from Pennsylvania in the east and from Michigan and northwest Ohio in the west.

43. 345 kV Lines Trip Across Ohio to West
ONTARIO
Over the next minute-plus, three more 345 kV lines in Ohio trip:
4:08:59 – Galion-Ohio Central-Muskingum 345 kV line trips
4:09:06 – East Lima-Fostoria Central 345 kV line trips
4:09:07 – Harding-Fox 345 kV #2 line trips
When the Galion-Ohio Central-Muskingum and East Lima-Fostoria Central transmission lines disconnect, the transmission paths from southern and western Ohio into northern Ohio and eastern Michigan are disconnected. Thus the combined northern Ohio and eastern Michigan load centers are now connected only by the transmission lines from: 1) northwestern Pennsylvania along the southern shore of Lake Erie; 2) lines west and northwest of Detroit, Michigan; and 3) Ontario. Northern Ohio is connected to eastern Michigan by only three 345 kV transmission lines near the southwestern bend of Lake Erie.
The reduced transmission capacity serving the northern Ohio load center results in the transmission voltage becoming severely depressed in that area as load exceeds the rapidly degrading power delivery capability.
We begin to see signs of the grid ripping along natural seams where the system is less tightly connected.

44. Generation Trips 4:09:08 – 4:10:27 PM
ONTARIO
From 4:09:08 - 4:10: MW of generation is lost:
MCV Plant drops 300MW from 1263 MW to 963 MW,
Avon Lake #7 unit trips (82 MW),
Two Kinder Morgan units trip (200 MW total),
Berger #3, 4, & 5 Units trip (355MW total)
Power flows into Michigan from Indiana become heavy in order to serve loads in eastern Michigan and northern Ohio. This reduces transmission capacity serving northern Ohio load centers resulting in depressed transmission voltage as load exceeds the rapidly declining power delivery capability over the remaining transmission lines.

45. 345 kV Transmission Cascade Moves North into Michigan 4:10:36 – 4:10:37 PM
From 4:10:36 to 4:10:37 three lines trip in rapid succession, continuing the ripping of the system through central Michigan.
Argenta – Battlecreek 345kV line trips
Battlecreek – Oneida 345kV line trips
Argenta – Tompkins 345kV line trips
These line outages interrupt the transmission paths into the Detroit area from south-central Michigan. Michigan lines northwest of Detroit then begin to trip, as shown on the next slide.

46. Northern Ohio and Eastern Michigan Served Only from Ontario after 4:10:37.5 – 4:10:38.6 PM
From 4:10:37.5 – 4:10:38.4 a peninsula consisting of northern Ohio and eastern Michigan is formed as:
the Hampton – Pontiac 345kV line trips;
the MCV Plant is reduced in output by an additional 835MW;
and the Thetford – Jewell 345kV line trips.

47. Power Transfers Shift at 4:10:38.6 PM
At this point, the load centers in eastern Michigan and northern Ohio have little generation left available to them and the voltage is declining. The major connection between those load centers and the rest of the Eastern Interconnection is at the interface between the Michigan and Ontario systems.
Line flows in Pennsylvania, New York and Ontario have now reversed direction and are flowing northeastward through Pennsylvania and into New York and Ontario before reaching Michigan by the remaining transmission path from Ontario. Power flows to serve eastern Michigan and northern Ohio loads, directly connected to Ontario, now all appear on the Pennsylvania – New York interface. This sudden large change in power flows drastically impacts the voltage and current levels on the transmission lines along the Pennsylvania – New York transmission interface.

48. Ontario – Michigan Interface Flow and Voltages Beginning 16:10:38
Eastern Eastern Michigan (Detroit) Unstable Voltage and Frequency Collapse and Pole Slipping
Ontario – Michigan Interface Flow and Voltages Beginning 16:10:38

49. Generator Trips to 16:10:38

50. Generator Trips – Next 7 Seconds

51. Overloads on PJM – NY Ties 4:10:39 PM
Responding to the surge of power flowing north out of Pennsylvania through New York and Ontario into Michigan, two 345 kV lines between Pennsylvania and New York disconnect within seconds of each other:
4:10:39.5 – Homer City-Watercure Road 345 kV
4:10:39.8 – Homer City-Stolle Road 345 kV

52. PJM – NY Separating 4:10:44 PM
With the two 345 kV lines gone, several 230 kV lines quickly become overloaded and trip at 4:10:44:
South Ripley - Erie East 230kV
South Ripley - Dunkirk 230 kV
East Towanda-Hillside 230 kV
At this point, the northern part of the Eastern Interconnection (which still includes eastern Michigan and northern Ohio) remains connected to the rest of the Interconnection at only two locations:
1) in the east through the 500 kV and 230 kV ties between New York and New Jersey, and
2) in the west through the long and weak 230 kV transmission path in Ontario connecting it to Manitoba and Minnesota.
Heavy power flows are now moving northward over the New Jersey to New York tie lines into what is now the great peninsula of northern Ohio, eastern Michigan, Ontario, New York and New England.

53. Cleveland – Toledo Island 4:10:39 - 4:10:46 PM Cleveland Blacks Out
From 4:10:39 to 4:10:46 Toledo and Cleveland split off into a separate island. Frequency in this island collapses quickly. When the Beaver-Davis Besse 345 kV line trips, Cleveland and Toledo separate from each other. A race begins in Cleveland between under-frequency load shedding and generator trips on under-frequency. Load shedding of 1300 MVA is unable to arrest the frequency collapse and the Cleveland island blacks out. The Toledo island also experiences load shedding as portions of the Toledo area black out, but some loads are restored by automatic reclosures.

54. Northeast Completes Separation from Eastern Interconnection 4:10:43 – 4:10:45 PM
North of Lake
Superior
At 4:10:43, eastern Michigan is still connected to Ontario, but the Keith-Waterman 230 kV line that forms part of that interface then disconnects.
At 4:10:45, the northwest Ontario system separates from the remainder of Ontario when the Wawa-Marathon 230 kV lines disconnect along the northern shore of Lake Superior. The portion of Ontario to the west of Wawa remains connected to the Eastern Interconnection through Manitoba and Minnesota.
At this time, the Branchburg-Ramapo 500 kV line and underlying 230 and 138 kV lines are the last major transmission paths remaining between the Eastern Interconnection and the northeast. Branchburg-Ramapo trips at 4:10:45 taking the underlying 230 and 138 kV lines along with it. This leaves the northeastern part of New Jersey connected to New York. Pennsylvania and the remainder of New Jersey remain connected to the Eastern Interconnection.
New York City, northern New Jersey, New York state, New England and the Canadian Maritime provinces, eastern Michigan, and the majority of Ontario are in a large island separated from the Eastern Interconnection. The areas outside this island were not significantly affected by the blackout.

55. Conditions at Niagara Indicate Progressively Worsening Stability Conditions with Prior Events

56. Island Breaks Up: 4:10:46 – 4:13 PM
The voltage falls to 50 kV on the 230 kV system feeding eastern Michigan. Large units in Detroit appear to pull out of synch and slip two poles before tripping off. The remaining units in Detroit area are no match for the remaining loads and the frequency collapses at about 2 Hz/sec. High speed power swing oscillations occur on the lines feeding Detroit from Ontario. Two more system-to-system pole slips are observed in Detroit before the Keith line trips by zone 1 relay and eastern Michigan is blacked out.
At 4:10:50, New York and Ontario separates an initial time, leaving New York and Ontario hydro generation at Niagara and St. Lawrence, along with some thermal generators and a tie to Quebec, connected to the western New York system, supporting the demand in upstate New York. A large portion of demand in Ontario is automatically disconnected by under-frequency load shedding. At 4:11:10, the three lines between New York and Ontario, which had reclosed after about 8 seconds, trip a second time west of Niagara, separating New York and Ontario a final time. Following this separation, the frequency in Ontario declines to 56 Hz by 4:11:57, resulting in an almost total blackout of most of Ontario, leaving 22,500 MW of demand lost out of a total demand of about 24,000 MW. The eastern New York island also blacked out with only scattered small pockets of service remaining. The western New York island continued to serve about 50% of its demand.
Moments later, the southwest portion of Connecticut, which had been connected to the eastern New York island, separates and blacks out.
As a result of the severe frequency and voltage changes, many large units in New York and Ontario have tripped. The eastern island of New York including the heavily populated areas of southeastern New York, New York City, and Long Island experiences severe frequency and voltage decline. At 4:11:29, the eastern island of New York splits into northern and southern sections. The small remaining load in the Albany area remains energized by local generation.

57. Frequency in Ontario and New York during Breakup Niagara Generation Stays with Western NY
There were two very sharp frequency excursions to 63 Hz on the Upper State New York area and the IMO Beck and Saunders hydro-electric plants after this. This plot shows the frequency peaks and the separating moments as lines into Ontario tripped, successfully reclosed (accidentally), tripped again, and then separated for good. In between the two frequency peaks, the Ontario system and Upper State New York almost held it together, however, two 500 MW Nanticoke units tripped by negative sequence protective relays in rapid succession at this time and the resulting additional surge of generation into the deficient interior region of Ontario was too much for the weak three remaining ties at Beck. They tripped again by zone 2 distance relays. Definite-time reclose attemps after that were unsuccessful as the frequencies were widely separated.
After this, the interior Ontario system frequency decayed, shed what remaining under frequency load shed locations at 58.8 Hz, and then fell, and fell some more until the total collapse of frequency (and simultaneously voltage collapse) at 16:11:56.
The wild ride of frequency on the New York side shook several unit (especially the nuclear units) into tripping off with in-plant control limit violations.

58. Generator Trips – After 16:10:44

59. Areas Affected by the Blackout Some Local Load Interrupted
End of the Cascade
Areas Affected by the Blackout
Service maintained
in some area
Some Local Load Interrupted
Starting at 4:10, and over the next 40 seconds, three more 345 kV transmission lines trip in Michigan, Ohio and Pennsylvania, as do twenty generators along Lake Erie, which had been producing 2,174 MW.
The loss of this generation increased the power flows into the northern Ohio and eastern Michigan load centers on the remaining paths, which included the west-east transmission lines that cross Michigan.
When the west-east Michigan 345 kV paths disconnected, it left eastern Michigan connected by only a circuitous path around northern Michigan that disconnected one second later, and the connections to Ontario and northern Ohio. Investigators are still studying the power flows that resulted.

60. Blackout Root Cause Group 1 FE Situational Awareness
FE did not ensure a reliable system after contingencies occurred because it did not have an effective contingency analysis capability
FE did not have effective procedures to ensure operators were aware of the status of critical monitoring tools
FE did not have effective procedures to test monitoring tools after repairs
FE did not have additional high level monitoring tools after alarm system failed

61. Blackout Out Root Cause Group 2 Vegetation Management
FE did not adequately manage tree growth in its transmission rights of way

62. Blackout Cause Group 3 Reliability Coordinator Diagnostics
MISO’s state estimator failed due to a data error.
MISO’s flowgate monitoring tool didn’t have real-time line information to detect growing overloads
MISO operators couldn’t easily link breaker status to line status to understand changing conditions.
PJM and MISO ineffective procedures and wide grid visibility to coordinate problems affecting their common boundaries

63. Near-Term Industry Actions
Responses from Control Areas and Reliability Coordinators Due December 15
Voltage support/reactive supply
Reliability communications
Computer failure response & notifications
Emergency action plans and capabilities
Operator training for emergencies
Vegetation management

64. Next Steps U.S./Canada Power Outage TF hearings NERC next steps
Public hearings to allow comment on report and input on recommendations
December 4
December 5
December 8 – Toronto
Industry technical conference
December 10 – Philadelphia
NERC next steps
NERC executive committees December 11
NERC committees meet January 13-14
Continue investigation
Near term analysis and recommendations in support of U.S. Canada Task Force
Long term analysis and recommendations for NERC