1. August 14, 2003 Blackout Final Report
U.S.-Canada Power System Outage Task Force
Final Report Summary

2. Overview The blackout and the investigation
Findings – how the blackout happened
Why the blackout happened
Report recommendations

3. The blackout August 14, 2003, starting at 16:05:57 EDT
Over 50 million people out of power in northeast US and Ontario, Canada
61,800 NW of load lost
Millions of work hours and billions of $ in economic costs

4. The investigation
Directed by President Bush and Prime Minister Chrétien on 8/15/03
Task Force headed by US Secretary of Energy and Canadian Minister of Natural Resources, with US and Canadian officials
Three work groups – Electric System Outage, Nuclear and Security
Investigators from both nations, public and private sector
Electric system investigation team composed of NERC, industry experts across US and Canada, US and Canada federal staff

5. Computer problems
At 12:15 EDT, MISO began having problems with its state estimator; it didn’t return to full functionality until 16:04 EDT.
Sometime after 14:14 EDT, FirstEnergy began losing its energy management system (EMS) alarms but didn’t know it.
At 14:20 EDT, parts of FE’s EMS began to fail – first remote sites, then core servers – but FE system operators didn’t know and FE IT support didn’t tell them.
Without a functioning EMS, FE operators didn’t know their system was losing lines and voltage until perhaps 15:45 EDT.

6. Reactive power problems
At 13:31 EDT, FirstEnergy lost the Eastlake 5 plant, a critical source of real and reactive power for the Cleveland-Akron area.
FE did not perform contingency analysis after this trip.
FE did not have sufficient understanding of the Cleveland-Akron area to know their system was seriously deficient in reactive power for voltage support.

7. Cleveland area voltage began falling

8. Line outages
Between 15:05 and 15:41 EDT, FE lost kV lines in the Cleveland-Akron area under normal loading due to contact with too-tall trees – but didn’t know it without EMS.
Line loading shifted and reactive power demands increased with each line loss.
Between 15:39 and 16:08 EDT, FE lost kV lines in the Cleveland-Akron area due to overloads and ground faults.

9. The Ohio tipping point
At 15:57:05 EDT, FE lost the Sammis-Star 345 kV line due to overload.
This shut down a major path for power imports into the Cleveland-Akron area and was the starting point for the full cascade.

10. The cascade
A cascade is a dynamic phenomenon in an electric system that can’t be stopped by human intervention once it starts
Power swings, voltage fluctuations and frequency fluctuations cause sequential trips of numerous transmission lines and generators, and automatic load-shedding in a widening geographic area.
System oscillations grow so large the system can’t rebalance and stabilize.

12. 265 power plants lost

13. Why did Ohio start happen?
The Ohio phase began because:
1. FirstEnergy and ECAR failed to study and understand the inadequacies of the FE system and FE didn’t operate the system with appropriate voltage criteria.
2. FE had inadequate situational awareness and didn’t recognize its system deteriorating.

14. Why did Ohio start happen?
3. FE failed to trim trees in its rights of way, so each early 345 kV line faulted on a tree that was too tall.
4. MISO and neighboring PJM didn’t provide effective real-time diagnostic support to FE.
5. FE didn’t act to restore its system to a secure condition (due to the other causes).

15. Why the cascade happened
Sammis-Star and the Cleveland-Akron line trips shifted load burden onto limited paths
After Sammis-Star, Zone 3 (and 2) relays in Ohio and Michigan caused 13 line trips between 15:57:05 and 16:10: EDT that would not have happened so quickly without those relays.

16. The cascade (2)
3. Relay settings on lines, generators and load-shedding across the northeast were not coordinated and integrated to reduce the likelihood of a cascade, so the grid’s elements and regions couldn’t rebalance.
4. Physics – once the frequency, voltage and power swings started, the grid couldn’t recover

17. Report recommendations for better reliability
Institutional issues (14 recommendations), including:
Make reliability standards mandatory and enforceable
Develop an independent funding mechanism for NERC
Strengthen reliability institutions, including NERC and regional councils, and define minimum requirements and cleaner footprints for control areas and reliability authorities
Reliability investments should be recoverable in transmission rates

18. Reliability recommendations
Strengthen and support NERC’s actions of February 14, 2004 (17), including:
Correct the direct causes of the blackout
Strengthen NERC compliance program
Support NERC’s Reliability Readiness Audits
Improve operator training and certification
Use better system protection measures
Use better real-time tools for grid monitoring and operation
Accelerate and improve reliability standards

19. Reliability recommendations
Physical and cyber-security (13):
Implement NERC IT standards
Develop IT management procedures
Improve IT forensic and diagnostic capabilities
Establish authority for physical and cyber security
Control access to critical equipment
Canadian nuclear power (2)

20. Will More Blackouts Occur?
System is man-made, and subject to mechanical failure and human error.
NERC’s readiness audits are a key preventive measure.
The U.S – Canada Task Force has been extended for a year to provide oversight for implementation of recommendations.