Presentation on theme: "Presented by Steve Davis, Principal, DavisLogic & All Hands Consulting"— Presentation transcript:
1 Presented by Steve Davis, Principal, DavisLogic & All Hands Consulting *07/16/96Business Continuity Planning For Research and Development OrganizationsPresented by Steve Davis, Principal, DavisLogic & All Hands ConsultingIntroduction. Mention working with TIGR. R&D not a typical client.Terrorist Attacks caused $500 M data and systems damage. $15.8 B Cost of restoring All IT and Communications$8.1B long term cost to enterprises.FEMA: WTC disaster: NY economy $60BInsurance payouts: $25B, which will top Hurricane Andrew 1992 $19BBiggest loss was in paper.BCP is disaster preparedness. Might be a legal or regulatory requirement. Might be your insurance company pressuring you.Research animal facilities must comply with the Animal Welfare Act, ILAR guidelines to have a disaster preparedness plan, Association for the Accreditation and Assessment of Laboratory Animal Care International (AAALAC) certification standards, institutional environmental health and safety or Occupational Safety and Health Administration (OSHA) standards, and other institutional documents, such as mission statements to perform research, teaching and service activities, and empowerment of institutional security and fire departments.*
2 *07/16/96“Stuff” HappensHow should you help your company maintain "business continuity" in the wake of disaster?Stuff happens all the time. While R&D organizations face many unique risk management challenges. Like any operation, R&D firms have both property and intellectual capital at risk. While the recent terrorist acts have heightened disaster preparedness concerns, disastrous calamities can happen anywhere and any time. For instance, this past June, medical researchers in Houston suffered a devastating blow when flooding from Tropical Storm Allison destroyed thousands of records and research animals. Research data was lost due to a lack of proper disaster planning. Both paper and electronic data were lost due to a lack of basic back-up precaution. One-of-a-kind property was lost when floodwaters burst through a wall and destroyed centuries old art collections, equipment, and music scores.*
3 Are You Ready For Anything? *07/16/96Are You Ready For Anything?Eighty-one per cent of CEOs say that their company's plans were inadequate to handle the myriad of issues arising from the World Trade Center tragedyRecent events have made it clear that you need to be ready for anything.Business that are not prepared typically fail after suffering a disaster.*
4 Disaster Causes & Effects Common Causes *07/16/96Disaster Causes & Effects Common CausesNatural HazardsIce StormEarthquakeWindFloodLightningSnowFrostMan-made Hazards (Deliberate)TheftViolenceFraudArsonMalicious DamageStrike*
5 Disaster Causes & Effects Common Causes *07/16/96Disaster Causes & Effects Common CausesMan-made Hazards (Deliberate)RiotBomb DamageBomb HoaxTerroristsHackingMan-made Hazards (Accidental)Operator ErrorExplosionFireWater LeaksFire Extinguisher DischargeUntil 3 weeks ago the probability of a terrorist attack was considered low. When planning for contingencies, two factors are taken into consideration: probability and amount of damage.*
6 Disaster Causes & Effects Common Effects *07/16/96Disaster Causes & Effects Common EffectsMan-made Hazards (Indirect)Power FailureTelecommunications FailureSmoke DamageFire Suppression AgentsHardware/Software failureThese are common indirect effects of hazards. Power failure is the most common.*
7 Disaster Causes & Effects Common Effects *07/16/96Disaster Causes & Effects Common EffectsDenial of ServiceData LossLoss of PersonnelLoss of System FunctionLack of InformationDenial of AccessCompromised or Corrupted DataDamaged EnvironmentProductivity LossPicture is a recent case where a water main broke outside of the offices, and the washout broke the sewer main, which forced lots of water and sand back through the toilets and sink drains into the basement of the building where the servers were on the floor with the UPS. Keep important equipment out of harms way.Other types of flooding can cause DOS – Flood of transactions Can’t get to your systemsData Loss – stolen or destroyedPersonnel – temporary or permanent – most devastatingSystem Function – Have data, but no software or workstationsLack of Information – employees don’t know where to go or what to doPoor or no access to building, computers –Compromised data – if hacker attack or most common: disgruntled employeeDamaged environment: equipment, furniture, carpetHere are some plans1. Setting up a backup server in an offsite location using a point topoint DSL link.2. Put the servers up one shelf, less likely to be drowned, but not onthe top so they are a prime target for smoke.3. Vendors are going to be identified that actually use a clock to timeovernight, not a calendar.4. We are looking at sensors to notify us early in an event (smoke,heat, water, etc. augmenting the security system).5. They are now moving to a paperless system, hence reducing the files.*
8 Disaster Causes & Effects Common Effects *07/16/96Disaster Causes & Effects Common EffectsLoss of ControlLoss of CommunicationInterrupted Cash FlowLoss of ImageLoss of Market ShareCosts of RepairCost of RecoveryLower MoraleLoss of ProfitsLoss of ControlInterrupted cash flow can’t get invoices out, collections interrupted, sales interruptedLoss of Image – if something you caused or through negligenceLoss of MarketCost of Repair, Recovery, MoraleProfitabilityNot intended to scare or depress. Empower you.*
9 Special Considerations *07/16/96Special ConsiderationsAnimalsEvacuation - whereOngoing care and feedingBites/ScratchesHazardous MaterialsBio HazardsRadiationChemicalsAlternate SpaceWet LabsPower NeedsContainmentAnimal holding facilities have special considerations. A decision will need to be made on evacuation – evac authority and procedures must be clearly established. Where are you going to move them to? If done in a rush it exposes staff to risk from bites and scratches. Over 30,000 animals drowned as a result of severe flooding of basement laboratories in several research institutions in the Houston, Texas area last June.Since 1996, the Institute of Laboratory Animal Resources (ILAR) Guide for Care and Use of Laboratory Animals recommends that research and laboratory animal facilities have a disaster preparedness plan. This is a prudent recommendation, because over US$10 billion a year are spent at nearly 2,000 facilities on biological research involving animals in the United States.Hazardous materials Bodily fluids, blood, infectious waste, or other potentially infectious material pose a special Biohazard risk. Any body fluid may contain micro organisms capable of causing disease. Appropriate protective attire must be worn.Special procedure if radiological materials involved.Alternate space can have special requirement based on use.*
10 Terminology Business Continuity Planning *07/16/96TerminologyBusiness Continuity PlanningMore than IT: people, premises, legal contracts, vital records, market knowledge.Work with senior management and other outside professionals such as corporate attorneys, succession planners, insurance companies.Discipline of thinking of contingencies that can happen to a business and developing a Plan to prevent, recover, repair and continue the business with the least disruption and cost.*
11 What is Business Continuity Planning? *07/16/96What is Business Continuity Planning?Planning to ensure the continuation of operations in the event of a catastrophic event.Business continuity planning includes the actions to be taken, resources required, and procedures to be followed to ensure the continued availability of essential services, programs, and operations in the event of unexpected interruptions.*
12 Business Continuity Planning Contingency PlanningDisaster RecoverySecurityBusiness RecoveryCrisis Management
13 BC Plan Components BCP Disaster Recovery Business Recovery Business ResumptionContingency PlanningObjectiveCritical Computer AppsCritical Business ProcessesProcess RestorationProcess WorkaroundFocusData RecoveryProcess RecoveryReturn to NormalMake DoExampleEventMainframe or server failureLaboratory FloodBuilding FireLoss of ApplicationSolutionHot Site RecoveryDry Out & RestartNew Equip. New Bldg.Use Manual Process
14 Create a Business Continuity Management Team Lead by Top ManagementProject BoD MonitorsRegular Status Reporting to ManagementBroad-basedAwareness for EveryoneKey PlayersSenior OfficialsFacilities/SafetyRisk ManagementLegalFinance/BudgetProcurement
15 Business Continuity Process Assess - identify and triage all threats (BIA)Evaluate - assess likelihood and impact of each threatMitigate - identify actions that may eliminate risks in advancePrepare – plan for contingent operationsRespond – take actions necessary to minimize the impact of risks that materializeRecover – return to normal as soon as possible
17 Business Impact Assessment *07/16/96Business Impact AssessmentThe purpose of the BIA is to:Identify critical systems, processes and functions;Establish an estimate of the maximum tolerable downtime (MTD) for each business processAssess the impact of incidents that result in a denial of access to systems, services or processes; and,Determine the priorities and processes for recovery of critical business processes.*
18 BIA Review Factors All Hazards Analysis Likelihood of Occurrence Impact of Outage on OperationsSystem InterdependenceRevenue RiskPersonnel and Liability Risks
19 Risk Analysis Matrix Area of Major Concern High Medium Low Low Medium Probability of LikelihoodMediumArea ofMajorConcernLowLowMediumHighSeverity of Consequence
20 Developing Business Continuity Strategies Understand alternatives and their advantages, disadvantages, and cost ranges, including mitigation and mutual aid as recovery strategies.Identify viable recovery strategies with business functional areas.Consolidate strategies.Identify off-site storage requirements and alternative facilities.Develop business unit consensus.Present strategies to management to obtain commitment.
21 Contingency Planning Process Phases Assessment - organizing the team, defining the scope, prioritizing the risks, developing failure scenariosPlanning - building contingency plans, identifying trigger events, testing plans, and training staff on the planPlan Execution - based on a trigger event, implementing the plan (either preemptively or reactively)Recovery - disengaging from contingent operations mode and restarting primary processes of normal operations by moving from contingency operations to a permanent solution as soon as possible.
22 Evaluating Alternatives Functionality - provides an acceptable level of servicePracticality - is reasonable in terms of the time and resources needed to acquire, test, and implement the planCost Benefit - cost is justified by the benefit to be derived from the plan
23 Emergency Management Planning Work with local and regional disaster agencies and business associationsAssess special problems with disastersLoss of lifelinesEmergency responseReview and revise existing disaster plansLook for new areas for disaster plansInclude Disaster Recovery Planning
24 *07/16/96Elements of a Good PlanPrevention, Response, Recovery, Remediation, RestorationTop Priorities addressed firstKeep things from happening if you canIf they do happen, act quickly and properlySalvage what you canFix what caused the problemGet back to business as usualChallenges: establishing High priorities. What is a high priority to the CFO is not what is important to the Director of Sales, is not what is important to the COO.CFO: financial recordsSales: Customer dataCOO: Intellectual Capital*
25 Elements of a Good Plan Action Plan responsibilities clearly defined *07/16/96Elements of a Good PlanAction Plan responsibilities clearly definedCommunication alternatives are consideredRedundancies are in placeAction Plans spell out who is to do what, when and how. Third parties need to know. Vendor who will supply. For spare parts, facilities. First in commandCommunication alternatives considered. Internet walkie talkiesRedundancies: communication lines/carriers, redundant drives, servers, redundant data: tape drives, data vaulting. Hot sites.. Question to ask: How much does it cost us for every hour our systems are inoperable?*
26 Elements of a Good Plan Product sources are identified *07/16/96Elements of a Good PlanProduct sources are identifiedPersonnel sources are identifiedWhere are you going to get PC’s? Spare partsPersonnel sources: temp agencies, cross training and knowledge bases.Gartner Group: New approaches since 9/11: increased use of telecommuting, moving out of the city into cheaper space and split technology and staff into multiple locations. People trained in multiple jobs, so if you have loss of lives, that knowledge base survives.Collaboration and knowledge bases software will increase.*
27 Keys to Success Vulnerabilities Clearly Identified *07/16/96Keys to SuccessVulnerabilities Clearly IdentifiedComprehensive Plan in PlacePlan Understood, Communicated and UpdatedTested quarterlyAdequately fundedSpecific Vulnerabilities – changes from company to company, industry to industry – R&D clearly has some unique issue but many are the same.Comprehensive – covering all types of disasters: deliberate, unintentional, all aspects of operations: people, premises, hardware, software, communications, dataUnderstood, communicated and updated: Plan is no good if no one knows where it is. Must evolve as businesses evolve.Tested quarterly: Kemper Insurance: all 225 employees survived. Full disaster recovery drill June including PC’s. All data was backed up to Chicago. Didn’t plan on air space to be closed. Fully operational by Friday the 14th.Oppenheimer Funds had 598 staff, all survived. Plan in Place since 1993, tested every six months. Hot site in New Jersey. Had to rebuild an NT domain so workers could access over a VPN. Forgot about how employees would access the data.Adequately funded: When times are tight, it’s one of the first things to go.*
28 Disaster Alert If you have advanced warning: People come first. Provide assistance. Note special needs.Move or secure vital records/high priority items if it can be done safely.Screw plywood over windows or use tape to reduce shattering.Verify master switch shut-off (water, gas, electricity) by trained staff.Secure outdoor objects.
29 Disaster Alert If you have advanced warning: Move items away from windows and below-ground storage into water-resistant areas.Wrap shelves and storage units in heavy plastic sealed with waterproof tape.Take Emergency Contact Lists, insurance and financial data, inventory, emergency plan and supplies with you.Give instructions on returning to work.
30 Safety First! Remain calm. Alert staff to potential hazards. Look for loose or downed power lines. Avoid area and report problems to local utility.Look for electrical damage: sparks, broken/frayed wires, burning smell. Turn off electricity at main switch if you can without risk.Shut off water.If you smell gas, open a window and immediately leave the building. Turn off gas if trained to do so. Call gas company at once.Do not reenter the building until declared safe by security or emergency management officials.
31 Getting Started Off-Site Gather staff off-site to assign tasks and review priorities.Establish a Command Center.Create a secure salvage area with necessary materials.Notify officials of the extent of damage.Establish alternative work sites.Appoint a PIO to report conditions to public and employees.Verify amount and terms of insurance, government assistance, potential funding.Contact service providers for disaster recovery equipment and services.Arrange for repairs as needed.
32 Stabilize the Building and Environment Do not enter without proper personal protective equipment.Identify structural hazards. Brace shelves. Remove debris.Stabilize vital equipment or experiments.Reduce temperature and humidity at once to prevent mold. Use air conditioning; or commercial dehumidification.In cool, low-humidity weather open windows, use circulating fans. If mold is already present, do not circulate air.Do not turn on heat unless required.Remove standing water and empty items containing water; remove wet carpets and furnishings.
33 DocumentationOnce it is safe to enter the building, make a preliminary tour of all affected areas.Do not move objects without documenting their condition.Use a camera to record the condition of property. Make sure images clearly record damage. Make notes and voice recordings to accompany photographs.Keep written records of contacts with insurance agents and other investigators, and decisions on retrieval and salvage.Make visual, written and voice records for each step of salvage procedures.
34 Retrieval And Protection Leave undamaged items in place if the environment is stable and area secure. If not, move them to a secure, environmentally controlled area.If no part of the building is dry, protect all objects with loose plastic sheeting.Separate undamaged from damaged items.Until salvage begins, maintain each group in the same condition you found it; i.e., keep wet items wet, dry items dry, and damp items damp.Retrieve all pieces of broken objects and label them.Check items daily for mold. If mold is found, handle objects with extreme care and isolate them.
35 Damage AssessmentNotify insurance representative - You may need an on-site evaluation before taking action.Make a rough estimate of the area affected and the extent and nature of damage. A detailed evaluation can slow recovery now.Look for threats to worker safety or collections. Determine status of security systems.Look for evidence of mold. Note how long the materials have been wet and the current inside temperature and relative humidity.Documenting the damage is essential for insurance and will help you with recovery.
36 Salvage Priorities Irreplaceable items and related documentation. Vital information; employee and accounting records, succession lists, inventories, and data.Other items that directly support your mission.Items that are unique, most used, most vital for research, most representative of subject areas, least replaceable or most valuable.Items most prone to continued damage.Materials most likely to be successfully salvaged.
37 Indoor Air QualityFailure to remove contaminated materials and to reduce moisture and humidity can present serious long-term health risks.Standing water and wet materials are a breeding ground for microorganisms, such as viruses, bacteria, and mold.They can cause disease, trigger allergic reactions, and continue to damage materials long after the flood.Source: EPA
38 Some DR QuestionsDo you have an alternate person for every key function?Do the Fire and Police departments have pre-plans including key contact information?Are your key technology rooms protected from "falling" water?Do each of your locations have emergency cabinets, first-aid kits, and disaster supplies?Do you have off-site storage of critical documents like contact information and forms?
39 *07/16/96Emergency Response Action Steps The first 48 hours can make the difference.Safety First!Getting Started Off-SiteStabilize the Building & EnvironmentDocumentationRetrieval & ProtectionDamage AssessmentSalvage PrioritiesAdapted from FEMA – handout contains details.The handout covers these in detail.Being prepared can make all the difference.*
40 For More Information Contact: Steve Davis, Principal DavisLogic & All HandsDavisLogic.comAllHandsConsulting.com