11/08/2005 Ken Donohoo Report on Lessons Learned – Hurricane Rita November 10, 2005 ROS Meeting Austin Control Center (MET Center)

Slides:



Advertisements
Similar presentations
Florida Public Service Commission 2007 Hurricane Preparedness Briefing May 23, 2007 Kissimmee Utility Authority Ken Davis Vice President Engineering &
Advertisements

Lee County Electric Cooperative 2006 Hurricane Preparedness Plan.
Hospital Emergency Management
FPL Hurricane Assessment Operating Committee January 25, 2005.
Westar Energy Emergency Preparedness and Restoration TFR Meeting June 26, 2012 Jim Tyler Director, Distribution Services.
Planning for the Future Disaster Recovery Plan / Business Continuity Plan Jim Zukowski, Ed.D. Texas State Board of Dental Examiners 2006 Annual ConferenceAlexandria,
ORGANIZATION. 2 Problem scenario  Develop an organizational chart for your laboratory showing lines of authority from the head of the organization to.
TRANSPORTATION & EMERGENCY PREPAREDNESS CHECKLIST National Resource Center for Human Service Transportation Coordination (NRC)
Comprehensive State Energy Plan – Public Meeting #3 Joplin, Missouri October 14, 2014 Ray Wilson.
1 JULY 2010 PRELIMINARY TECHNICAL CONFERENCE REPORT – WORK IN PROGRESS Session 4 Discussion of Alternatives and Implementation Activities for Real- Time.
Our Technology Comes with People Disaster Recovery Planning Glenn Lytle, Vice President Sales, Lumos Networks July 28,
Spring 2008 Campus Emergency Management Program Overview
ERCOT’S COMPLIANCE ASSESSMENT PROGRAM Mark Henry Compliance Manager March 23, 2005.
Services Tailored Around You® Business Contingency Planning Overview July 2013.
Continuity of Operations Planning COOP Overview for Leadership (Date)
Organizational Tabletop Exercise 1 Hurricane Scenario (Community-Based Organizations) Date | Location.
ERCOT 2013 Winter Storm Drill (WSD) Summary Brian Barcalow Training Specialist, Sr.
Community Preparedness & Disaster Planning. Why Disasters occur ?
Kenneth (Kenny) Mercado Division Vice President of Distribution Electric Operations Turning on the Lights.
Module 3 Develop the Plan Planning for Emergencies – For Small Business –
Hurricane Rita. 2 ERCOT Preparation ERCOT started alerts to QSEs and TOs on 9/21 Did a site failover of EMMS system Austin Taylor Procured satellite voice.
PAR CONFERENCE Homeland Defense A Provider’s Perspective Lessons from TMI Dennis Felty November 15, 2001.
A Major Business Disruption A Strategy for Minimising the Downtime Anthony Hegarty Mitigating Risks.
1 Practical Tips to Emergency Preparedness John Dahlin CHT, CWS-1 Texas ESRD Emergency Coalition.
Unit 8:COOP Plan and Procedures  Explain purpose of a COOP plan  Propose an outline for a COOP plan  Identify procedures that can effectively support.
Power Outages and Emergency Communications November 13, 2012.
ISA 562 Internet Security Theory & Practice
Insurance Institute for Business & Home Safety Even if the worst happens, be prepared to stay.
HURRICANES WHAT A DIFFERENCE A tropical storm becomes a hurricane when its winds reach what speed? 45 mph 64 mph 74 mph 80 mph 45 mph 64 mph 74 mph 80.
ADM 677 Crisis Management in Educational Settings Karen McCuiston Kentucky Center For School Safety.
1 Crisis Management / Emergency Management Overview.
Continuity of Operations Planning (COOP) for Community Colleges Welcome Introductions Administration Agenda Classroom Decorum Participant-Instructor.
ERCOT MARKET EDUCATION
ISO Outlook Summer 2005 and Beyond Senate Energy, Utilities and Communications Committee February 22, 2005 Jim Detmers Vice President of Grid Operations.
National Communications System Briefing Template Hurricane Isabel: Government-Industry Partnership in Planning, Response and recovery CAPT J. Katharine.
Disaster Recovery and Business Continuity Planning.
ERCOT 2014 Hurricane Drill (HD) Summary Brian Barcalow Training Specialist, Sr.
Utility Engineers, PC.  Generation  Transmission  Distribution.
Lessons Learned from Block Load Transfers following Hurricane Rita Reliability and Operations Subcommittee Meeting January 13, 2006.
1 Verizon Florida, LLC Hurricane Season Preparation PSC Workshop – May 6, 2009.
Operating Topics Part 2 Mike Oatts May 9, 2008 Transmission Customer Forum Destin, Florida Part 2 Mike Oatts May 9, 2008 Transmission Customer Forum Destin,
2015 HURRICANE SEASON PREPARATIONS March 18, 2015.
Erman Taşkın. Information security aspects of business continuity management Objective: To counteract interruptions to business activities and to protect.
2006 Hurricane Preparedness June 5, 2006 Sumter Electric Cooperative, Inc R. Ben Brickhouse: Director of Engineering & IT John J. LaSelva: Director of.
Hurricane Season Preparation Briefing Florida Public Service Commission May 1, 2008 Kevin Wailes General Manager – Electric Utility.
2008 Hurricane Season Preparedness Briefing Gulf Power Company Presented by Andy McQuagge Power Delivery Services Manager May 1, 2008.
Role of Account Management at ERCOT 2006 TAC Subcommittee Review ERCOT Board February 21, 2006.
2015 Hurricane Season Preparedness Briefing Jorge Puentes Engineering Manager – Northeast and Northwest Divisions March 18, 2015.
Verizon Florida, LLC Hurricane Season Preparation PSC Workshop – May 1, 2008.
Oncor Transmission Service Provider Kenneth A. Donohoo Director – System Planning, Distribution and Transmission Oncor Electric Delivery Co LLC
PJM©2014www.pjm.com A System Operator’s Resilience Wish List Tom Bowe Executive Director Reliability and Compliance PJM Interconnection
2014 Hurricane Season Preparedness Briefing Jorge Puentes Technical Projects Manager – Northeast and Northwest Divisions March 26, 2014.
Informal Meeting with FPSC Staff 2015 Hurricane Season Preparation March 18, 2015.
CBIZ RISK & ADVISORY SERVICES BUSINESS CONTINUITY PLANNING Developing a Readiness Strategy that Mitigates Risk and is Actionable and Easy to Implement.
©2003 PJM 1 Presentation to: Maryland Public Service Commission May 16, 2003.
Impact of Hurricane Ike on CenterPoint Energy’s Transmission System Paul Rocha Director, Transmission Planning Presented at October 2, 2008 ERCOT TAC Meeting.
Emergency Preparedness. Proposed Emergency Preparedness Rules NFR/LMC §19.326(a) deleted and moved to § for Emergency Preparedness Rules Places.
11 S I A M E M B E R C O M P A N I E S Satellite Industry Response to 2005 Hurricanes Satellites = Redundancy, Ubiquity, Interoperability.
Utility Needs and Smart Grid for Major Storm Events
Impact of Hurricane Ike on
MANAGEMENT of INFORMATION SECURITY, Fifth Edition
Florida Public Service Commission’s 2007 Storm Preparedness Workshop
Hurricane Rita.
Satellites = Redundancy, Ubiquity, Interoperability
Pacific Power Seismic Preparedness Update
Business Contingency Planning
Public Safety Power Shutoff (PSPS)
For Stake Emergency Communication Specialists
Presentation transcript:

11/08/2005 Ken Donohoo Report on Lessons Learned – Hurricane Rita November 10, 2005 ROS Meeting Austin Control Center (MET Center)

11/08/2005 Ken Donohoo It is better to be missed than hit… ERCOT was not significantly impacted by hurricane RITA. ENTERGY was significantly impacted; and ERCOT is attempting to learn from their experience. ERCOT does not yet have a response.

11/08/2005 Ken Donohoo ERCOT is able to setup and execute Block Load Transfer rapidly ERCOT was able to quickly implement block load transfers within the market structure –Registration – Financial qualification was done quickly through picking an existing representative in ERCOT. We were able to push through the database in 1 day what normally takes 3-4 days. Registered delivery point as a new NOIE LSE. The LSE pointed to an existing QSE. Set EZID as a wholesale delivery point. Nothing went to MOS. –We are required by protocol to have metering at BLT’s– Centerpoint and TXU got meters installed and registered EZIDs at ERCOT very quickly.

11/08/2005 Ken Donohoo Backup Center activation Several QSE’s/TDSP’s activated their emergency backup centers/plans. –Some QSE’s have backup centers –Some delegate another service provider to provide services In general, this was successful. –One center did not successfully communicate during the storm period. Storm impact was minimal

11/08/2005 Ken Donohoo Loss of Communications ERCOT implemented a plan for loss of communications ERCOT procured satellite telephones for the event. It was not needed during this event

11/08/2005 Ken Donohoo ERCOT Can expect timely government action if needed Actions: –Governor issued order suspending eminent domain requirements for temporary facilities –On 9/27 PUCT issued order suspending CCN requirements and allowing ERCOT TDSPs to extend service in area –DOE agreed to issue order maintaining ERCOT’s FERC exemption for these emergency connections

11/08/2005 Ken Donohoo ERCOT does NOT have an effective method of forecasting load in either Commercial or Operational systems We should expect high UFE amounts on and around projected hurricanes landfall We should expect high errors in the ERCOT operational load forecast.

11/08/2005 Ken Donohoo ERCOT has sent a list of questions to Entergy… What did you learn from your experiences with hurricanes this summer? What would you do differently when preparing for hurricanes in the future? Do you have an existing emergency plan that deals with hurricanes? Can you share it? Did you shut down plants in preparation for the storm? Should you have? What advice can you give us on how to prepare? Should we have a special emergency plan for hurricane preparation? Do you set distribution relays to one-shot near the coast? Can you advise us on how to plan for fuel curtailments/issues? Do you believe additional units should be gotten on line, or should we reduce the number of units online so we can get under the load? What did you do with staff? Did you put people in critical locations?

11/08/2005 Ken Donohoo Lessons Learned from Block Load Transfers following Hurricane Rita Reliability and Operations Subcommittee Meeting January 13, 2006

11/08/2005 Ken Donohoo Lesson’s Learned Purpose: Brief evaluation of Block Load Transfers (BLTs) in the aftermath of Hurricane Rita to pick up load in Entergy Gulf States Service territory Involved Parties: TXU Energy Delivery – Deep East Texas Cooperative BLT (≈10 MWs) Centerpoint – Crosby Station BLT (≈80 MWs) Entergy Gulf States

11/08/2005 Ken Donohoo Lesson’s Learned Physical Issues: Very little ERCOT can do. Work needed to block load over and all closely associated activities are between TDSPs. One-line drawings delivered ASAP are very important for a timely transfer (acceptable format) Contractual issues between the TDSP’s (how to compensate TDSPs for work) can slow up transfer

11/08/2005 Ken Donohoo Lesson’s Learned Commercial Issues: Need to create a LSE in ERCOT’s system was a hindrance Contractual Issues between the TDSPs should be considered in advance of the need to use an emergency BLT Settlement quality metering during a BLT for emergency restoration (.2% vs.6% CT accuracy) Frequency for providing metering data as well as method for providing data

11/08/2005 Ken Donohoo Lesson’s Learned Recommendations: ERCOT will investigate the creation entities and their relationships ahead of time (LSE for BLT into ERCOT; RE for BLT into other control area). Metering CT accuracy (.6%) will be allowed for emergency load restoration. ERCOT recommends actual meter data be provided before initial settlement run. Use of default profiles would introduce significant error if used. ERCOT will be flexible as to the format for data submittal (Loadstar, Excel, etc.). Investigate clearing Interstate Commerce issues in advance for BLTs that could block loads across state lines. Develop checklist for entities to use as a guide to prepare for an emergency.

11/08/2005 Ken Donohoo SMEPA – Lessons Learned During Storm 1. Alarms/Events (Avg 25k/day) – During Katrina 6.5K every two minutes. 1. Set your alarm priorities. 2. Make sure your archive size is sufficient. 3. Keep a unit spinning off-line. 4. Put the Control Center on emergency power before the storm. 5. You set up in the Control Center for real-time database and display modifications and additions. 6. Single transmission one-line with NSP 7. Limit access to the main control room

11/08/2005 Ken Donohoo SMEPA – Lessons Learned Day After 1. Approximately 1500 miles of destroyed/down high voltage transmission 2. Almost all distribution, distribution/transmission down/destroyed 3. No ties or communication with other utilities (Isolated) 4. One unit spinning, supplying it’s own load 5. All communications to plants - Dead 6. All roads, state highways, interstate highways – Completely Blocked 7. All SCADA communications – Dead 8. All voice, cell and internet communications – Dead 9. Main control center running on backup generator 10.All stores, gas stations, supply houses – Closed (extended time) 11.Employee travel impossible to home or to work 12.Employees at work had no clue about their families or homes the main control room

11/08/2005 Ken Donohoo SMEPA –First Needs 1. Long term fuel for Control Center emergency generator 2. Fuel for SMEPA and contractor vehicles 3. Food and water for local, plant and field personnel. 4. No motels – Housing for contract and SMEPA personnel 5. No running water – facilities for both men and women 6. Purchasing 7. Who is in charge (Emergency organization list) 1. Overall coordination – from Control Center 2. Transmission, Generation/Fuels, Facility Needs, SCADA, Communications (Phone, Radio, Internet, )

11/08/2005 Ken Donohoo SMEPA – Additional Needs 1. Satellite phones 2. Additional consoles (Switch, ES), Power Application Studies 3. PI Playback 4. Have a management command center outside the control center 5. Critical Loads, Hospitals, SMEPA facility, VP Call 6. Fly transmission lines/substations – visually (open/closed) 7. Transmission restoration based on 1. Critical load list 2. Available generation / Can you sustain voltage 3. Load needed to stabilize generation 4. Line/Substation condition

11/08/2005 Ken Donohoo SMEPA – Additional Needs 1. Transmission 1. Assemble crews (Road passage, ROW passage, Pole/Wire) 2. Coordinate work to be done 3. Coordinate transmission and distribution to same area 2. Generation 1. Fuel supply 2. Damage - cooling towers, etc. 3. CIS network including internet, (no outside network for over a week) 4. Phone communications – satellite (some phone communications out for two to three months) – SCADA & Land Lines 5. Communications – Radio, Microwave

11/08/2005 Ken Donohoo SMEPA – Lessons Radio Needs Satellite – Phone, SCADA, Internet Mapboard – NSP, Lines of Resolution Personnel Location by crew/team/individual (GPS) Procedures and Emergency Organization Chart Reclosers - Off Test black-start plan, units-Isoch, Frequency source, 60HZ Written Event Log – Lines put in service, Transformers heated Ties closed, … The guiding principal: 1. Safety 2. Reliability 3. Economics After restoration 1. Fix patched lines 2. Complete right of way clearing

11/08/2005 Ken Donohoo FPL Preparedness plan Excerpts from the Geisha Testimony – FPL response to FLA PUC

11/08/2005 Ken Donohoo Plan Objective –To safely restore the greatest number of customers in the least amount of time Meeting this objective is the most prudent response after a major storm. –Planning –Training –Adherence to established storm processes –Execution

11/08/2005 Ken Donohoo Key Components Disaster response policies and procedures Adjustable internal organizational structures based on the required response Timeline of activities to assure rapid notification and response; Mutual assistance agreements and vendor contracts and commitments Plans for movement of resources, personnel, materials, and equipment to areas requiring service restoration Communication and notification plans for employees, customers, community leaders, emergency operating centers, and regulators

11/08/2005 Ken Donohoo Key Components An established centralized command center with an organization for command and control of emergency response forces Checklists and conference call agendas to organize, plan, and report situational status Damage assessment modeling and reporting procedures; Field and aerial patrols to assess damage Comprehensive circuit patrols to gather vital information needed to identify the resources required for effective restoration Systems necessary to support outage management procedures and customer communications.

11/08/2005 Ken Donohoo Staffing All business units in the company identify personnel for staffing the emergency response organization. In many cases, employees assume roles different than their regular responsibilities. Training is conducted for many storm personnel each year regardless of whether they are in a new role or a role in which they have served many times. This includes training on processes that range from analytical and clerical to reinforcing restoration processes for managers and directors.

11/08/2005 Ken Donohoo Logistics Increasing material inventory Establishing staging site plans Expanding and verifying lodging arrangements Securing agreements and contracts for –Catering –Busing –Office trailers All of these agreements and activities provide the foundation to begin any restoration effort.

11/08/2005 Ken Donohoo Logistics - Functions Acquisition, preparation and coordination of: –Staging sites, environmental, salvage, lodging, laundry, buses, caterers, ice and water, office trailers, light towers, generators, port-o-lets, security guards, communications, and fuel delivery.

11/08/2005 Ken Donohoo Testing the plan Annual, full scale drill –Active participation from employees represented from every business unit in the company. –Begin 72 hours from the mock hurricane’s forecasted time and date of impact. –Simulate Damage assessments Communications Acquiring resources Updating storm systems

11/08/2005 Ken Donohoo Response to a Storm Threat – 72 Hours out Staff meteorologists are monitoring conditions Initiate preliminary preparations for addressing internal and external resource requirements, logistics needs, and system operation conditions. General Office Command Center (GOCC) is activated Storm personnel are alerted Resource requirements are forecasted Initial restoration plans are developed Contingency resources are activated Available resources from mutual assistance utilities are identified

11/08/2005 Ken Donohoo Response to a Storm Threat – 48 Hours out Computer models are run based on the projected intensity and path of the storm to forecast –Damage –Restoration workload –Potential customer outages Commitments are confirmed for restoration personnel, materials, and logistics support Identify staging site locations based on expected path of storm

11/08/2005 Ken Donohoo Response to a Storm Threat – 24 Hours out Positioning personnel and supplies to begin restoration as soon as it is safe to do so Continuously re-run Damage models Community leaders and County Emergency Operations Centers are contacted to share FPL’s restoration plans Verify critical infrastructure facilities Provide information to the news media, customers and community leaders regarding storm preparation, what to do in the event of an outage, as well as public safety messages.

11/08/2005 Ken Donohoo 5/30 DHS Report Florida utility prepares for major hurricanes. ‘Earlier this month the Florida Public Service Commission concluded that Florida Power & Light had not done enough to limit hurricane damage to the power grid following Hurricane Wilma in Up to 6.5 million people were left without lights, air conditioning, or hot water. FPL has instituted a five-point plan dubbed “Storm Secure”, to make major steps to strengthen its transmission and distribution system and make it better able to withstand hurricane-force winds, said Geisha Williams, FPL’s vice president of distribution. She said Storm Secure, launched on January 30, was only just getting underway, however. And while FPL has improved its reliability by about 50% since 1997, Williams acknowledged it still has a long way to go. “It won’t be overnight; it’s going to take a long time,” said Williams, referring to a key goal of “hardening” the electric grid to withstand wind gusts up to 150 mph.’