Lessons Learned Response and Recovery in a Disaster Robert DiLossi Director, Crisis Management

2 Copyright  2006 SunGard Availability Services LP, all rights reserved. Top 8 lessons from DR Scenarios Key players not available or missing Communication issues Lack of transportation/lodging/logistics No plan for extended recoveries Documentation/Scripts out dated (i.e. DDA, Technical Scripts, Employee Contact Lists, etc.) Difficulty accessing tapes/data/documentation Incomplete contracts-any vendor (i.e. Media Storage, Insurance, Fuel, etc.) Lack of content “Business Continuity Plans”

3 Copyright  2006 SunGard Availability Services LP, all rights reserved. Lessons Learned – Comparison Grid

4 Copyright  2006 SunGard Availability Services LP, all rights reserved. People - Employee Issues Encountered  Key players missing, safety  Not focused on recoveries  Insufficient funds  Who’s in charge? / No communication  Lack of personal necessities  Transportation unavailable  No lodging / logistics  Lack of corporate presence at recovery site (i.e. HR, Corporate Communications, etc.) Include employees / people with plans and tests Address employees families in plan Engage HR, Corporate, and Local Government Use alternate staff / alternate locations Establish Employee safe and travel kits Create cash account with debit cards Travel, lodging agreements near recovery locations

5 Copyright  2006 SunGard Availability Services LP, all rights reserved. Incident and Crisis Communication Plans  Internal documentation not current  Lack of involvement during planning / testing  Business unit requirements not accurate  Corporate participation minimal  Vendor contacts out dated  External agencies not engaged (i.e. Local or Federal Governments, Red Cross, EMS, etc.) Enforce change control / mandatory Develop / test your incident and crisis communication plans Engage all business units earlier / ownership of plan Rally Corporate involvement / include while testing Validate vendors contacts quarterly Engage local agencies (i.e. Fire, Police, Red Cross, EMS, Local Government, etc)

6 Copyright  2006 SunGard Availability Services LP, all rights reserved. Business Continuity / Disaster Recovery Preparedness Outdated equipment configurations with vendor contracts Crisis leader or communication not clearly defined No plans for extended recoveries Documentation / technical scripts out of date No vendor agreements in place Single point of failures Develop and ‘TEST’ your plans Change control/management Designate leads, single contact number and distribute Develop robust plans for longer stays Update scripts as they change Establish vendor agreements Establish backups for key technical personnel

7 Copyright  2006 SunGard Availability Services LP, all rights reserved. Data Availability Destroyed Tape, DASD and Paper Documents Tapes lost during shipment No access to Tapes, DASD or Paper Documents Travel Restrictions (curfews, government occupancy) Recovery scripts outdated / unknown Investigate the need for Electronic media Utilize disk to disk mirroring Use alternate locations for off-site storage vendor Store recovery scripts in electronic format off-site Test scripts and add to your BCP plan Rotate technical staff while testing

8 Copyright  2006 SunGard Availability Services LP, all rights reserved. Travel and Communication Challenges Air travel unavailable Highways closed No rentals available Vehicles broken down Mass evacuations enforced earlier Long delays Limited fuel available Disabled vehicles Deploy key team members and families earlier Consider alternate routes Utilize alternate travel means Understand and follow evacuation routes-safety Extra gas on hand or delivery Enter into agreements with fuel providers Utilize local government agencies for support

9 Copyright  2006 SunGard Availability Services LP, all rights reserved. Travel and Communication Challenges Cont…. Cell towers and land lines down Press/media statements for public sectors Consider alternate communication tools Extra cell phones and batteries Activate stand-by satellite phones Wireless cards for laptops Establish VPN, text messaging, direct connect Corporate address Augment staff from alternate business locations Establish external communication plan Update local radio stations with reports Establish company wide Crisis Bridge for updates

10 Copyright  2006 SunGard Availability Services LP, all rights reserved. After the Disaster… Insurance coverage - unknown Inventory not current Relocation Understand your policy and risks Test different disaster scenarios Keep inventory current Utilize independent review of insurance Record all disaster activities Use external security agency to monitor

11 Copyright  2006 SunGard Availability Services LP, all rights reserved. Long term recoveries… Recovery & Business Continuity plan while at the hotsite New equipment and circuit procurement  Cold-site and New Facility Location for a new facility Relocating your employees and their families New contact lists for e mployees Damage assessment report/length of stay No “return home” content Test scenarios for all levels of recovery Third party to search for new location Utilize vendor for facility plan HR and corporate involvement for employees New facility evacuation and fire routes Longer term housing Possible school registration Update communication plans

12 Copyright  2006 SunGard Availability Services LP, all rights reserved. Develop and test plan regularly Implement change management Maintain recovery scripts and procedures Assign and rotate backup roles Execute your plan as documented Familiarize yourself with offsite storage vendor Utilize professional organizations to augment staff Engage corporate and external agencies during testing Test the way you recover, recover the way you test! Key Points to a successful recovery

13 Copyright  2006 SunGard Availability Services LP, all rights reserved. Philadelphia Crisis Management Center

14 Copyright  2006 SunGard Availability Services LP, all rights reserved. California Fires Mt. St. Helens Eruption Hurricanes Proactive Calling Map N. CA. Earthquake 2003 Northeast Power Outage

15 Copyright  2006 SunGard Availability Services LP, all rights reserved. SunGard’s Pro-Active Calling Communication tools and protocols SunGard Contracts and DDA Damage assessment teams Discuss recovery plans and last test Logistics / Travel / Evacuations Roles and Responsibilities Tape and Vendor Availability Timeline Corporate and External Agency Activation

16 Copyright  2006 SunGard Availability Services LP, all rights reserved. Authorization Password (if applicable) or Disaster Declaration Authorization List Identify Declaration Needs/Alternatives Site Preparation Appropriate Staff Notified - Call Back Verification How to Declare a Disaster

17 Copyright  2006 SunGard Availability Services LP, all rights reserved. Alert Process What are the advantages? No cost to You, our Customer Heightened awareness Review last test and DDA on file Logistics Review contract information Document configurations needed Follow up on every alert until issue is resolved

18 Copyright  2006 SunGard Availability Services LP, all rights reserved. DDA Online - The Disaster Declaration Authorization (DDA) Form is a list of individuals who are authorized to declare a disaster for their company. Directly logon to Access and Update DDA 24/7, by person and by schedule Delegate DDA Information access by company, person or schedule Data is update within the hour or final changes Alerts Online - SunGard Customers may open an Alert online Logon to Access and submit online Alerts 24/7 Customer will receive call back on all alerts Still no costs to Customer Hotline calls are now handled by the IOC in Denver. What’s new?

Questions? Lessons Learned Response and Recovery in a Disaster.