1. Presented by Steve Davis, Principal, DavisLogic & All Hands Consulting*
07/16/96
Business Continuity Planning For Research and Development Organizations
Presented by Steve Davis, Principal, DavisLogic & All Hands Consulting
Introduction. Mention working with TIGR. R&D not a typical client.
Terrorist Attacks caused $500 M data and systems damage. $15.8 B Cost of restoring All IT and Communications
$8.1B long term cost to enterprises.
FEMA: WTC disaster: NY economy $60B
Insurance payouts: $25B, which will top Hurricane Andrew 1992 $19B
Biggest loss was in paper.
BCP is disaster preparedness. Might be a legal or regulatory requirement. Might be your insurance company pressuring you.
Research animal facilities must comply with the Animal Welfare Act, ILAR guidelines to have a disaster preparedness plan, Association for the Accreditation and Assessment of Laboratory Animal Care International (AAALAC) certification standards, institutional environmental health and safety or Occupational Safety and Health Administration (OSHA) standards, and other institutional documents, such as mission statements to perform research, teaching and service activities, and empowerment of institutional security and fire departments. *

2. *
07/16/96
“Stuff” Happens
How should you help your company maintain "business continuity" in the wake of disaster?
Stuff happens all the time. While R&D organizations face many unique risk management challenges. Like any operation, R&D firms have both property and intellectual capital at risk. While the recent terrorist acts have heightened disaster preparedness concerns, disastrous calamities can happen anywhere and any time. For instance, this past June, medical researchers in Houston suffered a devastating blow when flooding from Tropical Storm Allison destroyed thousands of records and research animals. Research data was lost due to a lack of proper disaster planning. Both paper and electronic data were lost due to a lack of basic back-up precaution. One-of-a-kind property was lost when floodwaters burst through a wall and destroyed centuries old art collections, equipment, and music scores. *

3. Are You Ready For Anything?*
07/16/96
Are You Ready For Anything?
Eighty-one per cent of CEOs say that their company's plans were inadequate to handle the myriad of issues arising from the World Trade Center tragedy
Recent events have made it clear that you need to be ready for anything.
Business that are not prepared typically fail after suffering a disaster. *

4. Disaster Causes & Effects Common Causes*
07/16/96
Disaster Causes & Effects Common Causes
Natural Hazards
Ice Storm
Earthquake
Wind
Flood
Lightning
Snow
Frost
Man-made Hazards (Deliberate)
Theft
Violence
Fraud
Arson
Malicious Damage
Strike *

5. Disaster Causes & Effects Common Causes*
07/16/96
Disaster Causes & Effects Common Causes
Man-made Hazards (Deliberate)
Riot
Bomb Damage
Bomb Hoax
Terrorists
Hacking
Man-made Hazards (Accidental)
Operator Error
Explosion
Fire
Water Leaks
Fire Extinguisher Discharge
Until 3 weeks ago the probability of a terrorist attack was considered low. When planning for contingencies, two factors are taken into consideration: probability and amount of damage. *

6. Disaster Causes & Effects Common Effects*
07/16/96
Disaster Causes & Effects Common Effects
Man-made Hazards (Indirect)
Power Failure
Telecommunications Failure
Smoke Damage
Fire Suppression Agents
Hardware/Software failure
These are common indirect effects of hazards. Power failure is the most common. *

7. Disaster Causes & Effects Common Effects*
07/16/96
Disaster Causes & Effects Common Effects
Denial of Service
Data Loss
Loss of Personnel
Loss of System Function
Lack of Information
Denial of Access
Compromised or Corrupted Data
Damaged Environment
Productivity Loss
Picture is a recent case where a water main broke outside of the offices, and the washout broke the sewer main, which forced lots of water and sand back through the toilets and sink drains into the basement of the building where the servers were on the floor with the UPS. Keep important equipment out of harms way.
Other types of flooding can cause DOS – Flood of transactions Can’t get to your systems
Data Loss – stolen or destroyed
Personnel – temporary or permanent – most devastating
System Function – Have data, but no software or workstations
Lack of Information – employees don’t know where to go or what to do
Poor or no access to building, computers –
Compromised data – if hacker attack or most common: disgruntled employee
Damaged environment: equipment, furniture, carpet
Here are some plans
1. Setting up a backup server in an offsite location using a point to
point DSL link.
2. Put the servers up one shelf, less likely to be drowned, but not on
the top so they are a prime target for smoke.
3. Vendors are going to be identified that actually use a clock to time
overnight, not a calendar.
4. We are looking at sensors to notify us early in an event (smoke,
heat, water, etc. augmenting the security system).
5. They are now moving to a paperless system, hence reducing the files. *

8. Disaster Causes & Effects Common Effects*
07/16/96
Disaster Causes & Effects Common Effects
Loss of Control
Loss of Communication
Interrupted Cash Flow
Loss of Image
Loss of Market Share
Costs of Repair
Cost of Recovery
Lower Morale
Loss of Profits
Loss of Control
Interrupted cash flow can’t get invoices out, collections interrupted, sales interrupted
Loss of Image – if something you caused or through negligence
Loss of Market
Cost of Repair, Recovery, Morale
Profitability
Not intended to scare or depress. Empower you. *

9. Special Considerations*
07/16/96
Special Considerations
Animals
Evacuation - where
Ongoing care and feeding
Bites/Scratches
Hazardous Materials
Bio Hazards
Radiation
Chemicals
Alternate Space
Wet Labs
Power Needs
Containment
Animal holding facilities have special considerations. A decision will need to be made on evacuation – evac authority and procedures must be clearly established. Where are you going to move them to? If done in a rush it exposes staff to risk from bites and scratches. Over 30,000 animals drowned as a result of severe flooding of basement laboratories in several research institutions in the Houston, Texas area last June.
Since 1996, the Institute of Laboratory Animal Resources (ILAR) Guide for Care and Use of Laboratory Animals recommends that research and laboratory animal facilities have a disaster preparedness plan. This is a prudent recommendation, because over US$10 billion a year are spent at nearly 2,000 facilities on biological research involving animals in the United States.
Hazardous materials Bodily fluids, blood, infectious waste, or other potentially infectious material pose a special Biohazard risk. Any body fluid may contain micro organisms capable of causing disease. Appropriate protective attire must be worn.
Special procedure if radiological materials involved.
Alternate space can have special requirement based on use. *

10. Terminology Business Continuity Planning*
07/16/96
Terminology
Business Continuity Planning
More than IT: people, premises, legal contracts, vital records, market knowledge.
Work with senior management and other outside professionals such as corporate attorneys, succession planners, insurance companies.
Discipline of thinking of contingencies that can happen to a business and developing a Plan to prevent, recover, repair and continue the business with the least disruption and cost. *

11. What is Business Continuity Planning?*
07/16/96
What is Business Continuity Planning?
Planning to ensure the continuation of operations in the event of a catastrophic event.
Business continuity planning includes the actions to be taken, resources required, and procedures to be followed to ensure the continued availability of essential services, programs, and operations in the event of unexpected interruptions. *

12. Business Continuity Planning
Contingency Planning
Disaster Recovery
Security
Business Recovery
Crisis Management

13. BC Plan Components BCP Disaster Recovery Business Recovery
Business Resumption
Contingency Planning
Objective
Critical Computer Apps
Critical Business Processes
Process Restoration
Process Workaround
Focus
Data Recovery
Process Recovery
Return to Normal
Make Do
Example
Event
Mainframe or server failure
Laboratory Flood
Building Fire
Loss of Application
Solution
Hot Site Recovery
Dry Out & Restart
New Equip. New Bldg.
Use Manual Process

14. Create a Business Continuity Management Team
Lead by Top Management
Project BoD Monitors
Regular Status Reporting to Management
Broad-based
Awareness for Everyone
Key Players
Senior Officials
Facilities/Safety
Risk Management
Legal
Finance/Budget
Procurement

15. Business Continuity Process
Assess - identify and triage all threats (BIA)
Evaluate - assess likelihood and impact of each threat
Mitigate - identify actions that may eliminate risks in advance
Prepare – plan for contingent operations
Respond – take actions necessary to minimize the impact of risks that materialize
Recover – return to normal as soon as possible

16. Building a BCP Plan

17. Business Impact Assessment*
07/16/96
Business Impact Assessment
The purpose of the BIA is to:
Identify critical systems, processes and functions;
Establish an estimate of the maximum tolerable downtime (MTD) for each business process
Assess the impact of incidents that result in a denial of access to systems, services or processes; and,
Determine the priorities and processes for recovery of critical business processes. *

18. BIA Review Factors All Hazards Analysis Likelihood of Occurrence
Impact of Outage on Operations
System Interdependence
Revenue Risk
Personnel and Liability Risks

19. Risk Analysis Matrix Area of Major Concern High Medium Low Low Medium
Probability of Likelihood
Medium
Area of
Major
Concern
Low
Low
Medium
High
Severity of Consequence

20. Developing Business Continuity Strategies
Understand alternatives and their advantages, disadvantages, and cost ranges, including mitigation and mutual aid as recovery strategies.
Identify viable recovery strategies with business functional areas.
Consolidate strategies.
Identify off-site storage requirements and alternative facilities.
Develop business unit consensus.
Present strategies to management to obtain commitment.

21. Contingency Planning Process Phases
Assessment - organizing the team, defining the scope, prioritizing the risks, developing failure scenarios
Planning - building contingency plans, identifying trigger events, testing plans, and training staff on the plan
Plan Execution - based on a trigger event, implementing the plan (either preemptively or reactively)
Recovery - disengaging from contingent operations mode and restarting primary processes of normal operations by moving from contingency operations to a permanent solution as soon as possible.

22. Evaluating Alternatives
Functionality - provides an acceptable level of service
Practicality - is reasonable in terms of the time and resources needed to acquire, test, and implement the plan
Cost Benefit - cost is justified by the benefit to be derived from the plan

23. Emergency Management Planning
Work with local and regional disaster agencies and business associations
Assess special problems with disasters
Loss of lifelines
Emergency response
Review and revise existing disaster plans
Look for new areas for disaster plans
Include Disaster Recovery Planning

24. *
07/16/96
Elements of a Good Plan
Prevention, Response, Recovery, Remediation, Restoration
Top Priorities addressed first
Keep things from happening if you can
If they do happen, act quickly and properly
Salvage what you can
Fix what caused the problem
Get back to business as usual
Challenges: establishing High priorities. What is a high priority to the CFO is not what is important to the Director of Sales, is not what is important to the COO.
CFO: financial records
Sales: Customer data
COO: Intellectual Capital *

25. Elements of a Good Plan Action Plan responsibilities clearly defined*
07/16/96
Elements of a Good Plan
Action Plan responsibilities clearly defined
Communication alternatives are considered
Redundancies are in place
Action Plans spell out who is to do what, when and how. Third parties need to know. Vendor who will supply. For spare parts, facilities. First in command
Communication alternatives considered. Internet walkie talkies
Redundancies: communication lines/carriers, redundant drives, servers, redundant data: tape drives, data vaulting. Hot sites.. Question to ask: How much does it cost us for every hour our systems are inoperable? *

26. Elements of a Good Plan Product sources are identified*
07/16/96
Elements of a Good Plan
Product sources are identified
Personnel sources are identified
Where are you going to get PC’s? Spare parts
Personnel sources: temp agencies, cross training and knowledge bases.
Gartner Group: New approaches since 9/11: increased use of telecommuting, moving out of the city into cheaper space and split technology and staff into multiple locations. People trained in multiple jobs, so if you have loss of lives, that knowledge base survives.
Collaboration and knowledge bases software will increase. *

27. Keys to Success Vulnerabilities Clearly Identified*
07/16/96
Keys to Success
Vulnerabilities Clearly Identified
Comprehensive Plan in Place
Plan Understood, Communicated and Updated
Tested quarterly
Adequately funded
Specific Vulnerabilities – changes from company to company, industry to industry – R&D clearly has some unique issue but many are the same.
Comprehensive – covering all types of disasters: deliberate, unintentional, all aspects of operations: people, premises, hardware, software, communications, data
Understood, communicated and updated: Plan is no good if no one knows where it is. Must evolve as businesses evolve.
Tested quarterly: Kemper Insurance: all 225 employees survived. Full disaster recovery drill June including PC’s. All data was backed up to Chicago. Didn’t plan on air space to be closed. Fully operational by Friday the 14th.
Oppenheimer Funds had 598 staff, all survived. Plan in Place since 1993, tested every six months. Hot site in New Jersey. Had to rebuild an NT domain so workers could access over a VPN. Forgot about how employees would access the data.
Adequately funded: When times are tight, it’s one of the first things to go. *

28. Disaster Alert If you have advanced warning:
People come first. Provide assistance. Note special needs.
Move or secure vital records/high priority items if it can be done safely.
Screw plywood over windows or use tape to reduce shattering.
Verify master switch shut-off (water, gas, electricity) by trained staff.
Secure outdoor objects.

29. Disaster Alert If you have advanced warning:
Move items away from windows and below-ground storage into water-resistant areas.
Wrap shelves and storage units in heavy plastic sealed with waterproof tape.
Take Emergency Contact Lists, insurance and financial data, inventory, emergency plan and supplies with you.
Give instructions on returning to work.

30. Safety First! Remain calm. Alert staff to potential hazards.
Look for loose or downed power lines. Avoid area and report problems to local utility.
Look for electrical damage: sparks, broken/frayed wires, burning smell. Turn off electricity at main switch if you can without risk.
Shut off water.
If you smell gas, open a window and immediately leave the building. Turn off gas if trained to do so. Call gas company at once.
Do not reenter the building until declared safe by security or emergency management officials.

31. Getting Started Off-Site
Gather staff off-site to assign tasks and review priorities.
Establish a Command Center.
Create a secure salvage area with necessary materials.
Notify officials of the extent of damage.
Establish alternative work sites.
Appoint a PIO to report conditions to public and employees.
Verify amount and terms of insurance, government assistance, potential funding.
Contact service providers for disaster recovery equipment and services.
Arrange for repairs as needed.

32. Stabilize the Building and Environment
Do not enter without proper personal protective equipment.
Identify structural hazards. Brace shelves. Remove debris.
Stabilize vital equipment or experiments.
Reduce temperature and humidity at once to prevent mold. Use air conditioning; or commercial dehumidification.
In cool, low-humidity weather open windows, use circulating fans. If mold is already present, do not circulate air.
Do not turn on heat unless required.
Remove standing water and empty items containing water; remove wet carpets and furnishings.

33. Documentation
Once it is safe to enter the building, make a preliminary tour of all affected areas.
Do not move objects without documenting their condition.
Use a camera to record the condition of property. Make sure images clearly record damage. Make notes and voice recordings to accompany photographs.
Keep written records of contacts with insurance agents and other investigators, and decisions on retrieval and salvage.
Make visual, written and voice records for each step of salvage procedures.

34. Retrieval And Protection
Leave undamaged items in place if the environment is stable and area secure. If not, move them to a secure, environmentally controlled area.
If no part of the building is dry, protect all objects with loose plastic sheeting.
Separate undamaged from damaged items.
Until salvage begins, maintain each group in the same condition you found it; i.e., keep wet items wet, dry items dry, and damp items damp.
Retrieve all pieces of broken objects and label them.
Check items daily for mold. If mold is found, handle objects with extreme care and isolate them.

35. Damage Assessment
Notify insurance representative - You may need an on-site evaluation before taking action.
Make a rough estimate of the area affected and the extent and nature of damage. A detailed evaluation can slow recovery now.
Look for threats to worker safety or collections. Determine status of security systems.
Look for evidence of mold. Note how long the materials have been wet and the current inside temperature and relative humidity.
Documenting the damage is essential for insurance and will help you with recovery.

36. Salvage Priorities Irreplaceable items and related documentation.
Vital information; employee and accounting records, succession lists, inventories, and data.
Other items that directly support your mission.
Items that are unique, most used, most vital for research, most representative of subject areas, least replaceable or most valuable.
Items most prone to continued damage.
Materials most likely to be successfully salvaged.

37. Indoor Air Quality
Failure to remove contaminated materials and to reduce moisture and humidity can present serious long-term health risks.
Standing water and wet materials are a breeding ground for microorganisms, such as viruses, bacteria, and mold.
They can cause disease, trigger allergic reactions, and continue to damage materials long after the flood.
Source: EPA

38. Some DR Questions
Do you have an alternate person for every key function?
Do the Fire and Police departments have pre-plans including key contact information?
Are your key technology rooms protected from "falling" water?
Do each of your locations have emergency cabinets, first-aid kits, and disaster supplies?
Do you have off-site storage of critical documents like contact information and forms?

39. *
07/16/96
Emergency Response Action Steps The first 48 hours can make the difference.
Safety First!
Getting Started Off-Site
Stabilize the Building & Environment
Documentation
Retrieval & Protection
Damage Assessment
Salvage Priorities
Adapted from FEMA – handout contains details.
The handout covers these in detail.Being prepared can make all the difference. *

40. For More Information Contact: Steve Davis, Principal
DavisLogic & All Hands
DavisLogic.com
AllHandsConsulting.com