1. Business Continuity Planning In a Post-Katrina Environment May 2006 Brian D. Voss Chief Information Officer Enterprise 2006—The EDUCAUSE Enterprise Technology Conference Copyright Brian D. Voss, 2006. This work is the intellectual property of the author. Permission is granted for this material to be shared for non-commercial, educational purposes, provided that this copyright statement appears on the reproduced materials and notice is given that the copying is by permission of the author. To disseminate otherwise or to republish requires written permission from the author.

2. A few caveats I was not at the center of Katrina –John Lawson, my colleague from Tulane; Jim Burgard, my colleague from University of New Orleans; Dave Troendle, my colleague from LSU Health Science Center NO were … and they have stories to tell –Our IT colleagues at Xavier, Loyola, Dillard, Southern (NO), and Southeastern, in the Louisiana Technical and Community Colleges, and those in the states of Mississippi, Alabama, and Texas impacted by Katrina and Rita. –Clifford Woodruff at Lamar University, who rode out Rita, has some excellent insights to share; ask him I was on this disaster’s edge; they were at its center. My campus – LSU A&M in Baton Rouge was not subject to any devastation or flooding. We were, however, impacted greatly by the aftermath of the storm

3. Don’t call me Doctor … I am not an expert on disaster recovery –But I am learning more and faster than I could have imagined I don’t have all the answers –But I sure have thought about a lot of questions and some possible answers LSU does not have a ‘model’ BCP to share –But we’re working on one with urgency

4. A Little History from 2005: Katrina’s Impacts on Louisiana Higher Ed Basically, those in Katrina’s path were put temporarily out of business UNO/LSUHSC in New Orleans –portions of campus under water –classes resumed Oct-05 at Jefferson Facility –Resumed operations in NO Spring Semester 06 Tulane, Xavier, Dillard & Loyola –Campuses all impacted to one degree or another –Established temporary admin HQs elsewhere –Resumed operations in NO Spring Semester 06

5. Katrina’s Impacts on Louisiana Higher Ed Louisiana Technical Colleges & Community Colleges –Lost several NO-based campuses; serving students at other campuses that were less (or not at all) impacted Louisiana State University –Became perhaps the most critical facility in support of disaster relief/response in the State of Louisiana –Despite this, continued to function as the State’s Flagship Research University; added over 3,000 temporary/displaced students; serve over 9,000 students from areas impacted by Katrina & Rita

6. Katrina Impact Continues: Smaller Classes in New Orleans 3,000 3,000 4,000 4,000 Loyola U. New Orleans*:Loyola U. New Orleans*: 1,507 1,507 4,205 4,205 Xavier University of Louisiana:Xavier University of Louisiana: 3,400 3,400 4,244 4,244 University of New Orleans:University of New Orleans: 420 420 1,050 1,050 Our Lady of Holy Cross College:Our Lady of Holy Cross College: 20,857 20,857 18,666 18,666 Tulane University:Tulane University: ApplicationsApplications Applications:Applications: Fall 2005 Fall 2005 Fall 2006 Fall 2006 3,000 3,000 4,000 4,000 Loyola U. New Orleans*:Loyola U. New Orleans*: 1,507 1,507 4,205 4,205 Xavier University of Louisiana:Xavier University of Louisiana: 3,400 3,400 4,244 4,244 University of New Orleans:University of New Orleans: 420 420 1,050 1,050 Our Lady of Holy Cross College:Our Lady of Holy Cross College: 20,857 20,857 18,666 18,666 Tulane University:Tulane University: ApplicationsApplications Applications:Applications: Fall 2005 Fall 2005 Fall 2006 Fall 2006 The Chronicle of Higher Education May 26 issue Story by Jeffrey Selingo

7. Katrina Impact Continues: Smaller Classes in New Orleans The Chronicle of Higher Education May 26 issue Story by Jeffrey Selingo

8. A word about New Orleans Don’t believe all of what you see in the media – good or bad; you need to see it to understand The city is about 40% its pre-Katrina population Whole areas – many square miles – are not inhabited and little or no recovery has occurred Blue tarps, FEMA trailers, and abandoned vehicles abound; storm damage is still evident, even downtown The French Quarter is pretty much the same as it was and tourism has resumed Optimism and hope fight a daily battle with despair It will take a decade or more for the city to become whatever it will become – but it will never be like it was before (which may have a hopeful aspect)

9. Post-Katrina Disaster Recovery Thinking Traditional Disaster Recovery –What if my data center is lost Broader Disaster Recovery –What if my campus is lost –What if the city where my campus is located is lost Survivor Disaster Recovery –What if I’m fine … but everyone around me is not

10. Traditional Disaster Recovery You’re down, everything else is fine Do you have a workable DR plan? Do you know where on campus you’ll go? Did you take necessary back-ups and do you have them ready to re-produce production files? What vendors will you need to tap – and for what? How will you quickly re-establish network connectivity? Phone service? Web presence? E- mail? Mission critical information systems?

11. Traditional Disaster Recovery You’re down, everything else is fine Lessons: –It’s the data, stupid. Hardware can be replaced; data can not be –If you lose just your data center, the need for rapid response will be acute and immediate; your institution can not operate without IT and it will notice it (maybe for the first time) –The CIO will be definitely on the hot seat; job at risk

12. Your Views/Experiences? Do you have a “traditional” DR Plan? –Are you building one? Have you had to use it? How’d that go? Do you ‘drill’ on this plan? What has been your experience with your administration’s responsiveness to DR/BCP? –Has that changed since 9-11, or Katrina?

13. Broader Disaster Recovery You and everyone around you is down Are your off-sites conveniently (and perhaps tragically) close? Do you have arrangements to get key services restored at a distance –Web, E-mail, Financial/HR, Student Information, CMS Hot-sites may be too much $$$$ – but can you find suitable raised floor/HVAC/power to ‘re-build’ Can you support your administration “in exile?” –Internet access, computers, cell phones, e-mail, IM Is your ‘life-boat’ plan portable over larger distances? Can you grab your key people? Can you care for them?

14. Broader Disaster Recovery You and everyone around you is down Lessons: –People are your most key resource – but expect them to be burdened with other priorities –Knowing what you’ll need to do and having it organized is more important than knowing all about ‘how’ you’ll do it when you get there (wherever ‘there’ might be) –Once again, the CIO will be on the hot seat as the institution realizes just how dependent it is upon IT

15. One Possible Tool In The Arsenal: Data Center Lifeboat Situation: What if we had very short notice (4-8 hours) notice of the need to abandon our data center/campus and set-up elsewhere (>50miles away) Goal #1: Re-establish some critical subset of services Goal #2: Support the re-establishment of some subset of university administration

16. Lifeboat Key things to recover: –Payroll/Financial Data –Web presence Splash/priority information screens As much content as possible –E-mail service for faculty/staff/students –Portal interface –Student Information Systems –HR, Procurement Systems –CMS –What else? Budgets ($25K, $50K, $100K) Key things to address –Off-site storage of critical back-ups –Ability to ‘grab and go’ key data and hardware –List of key hardware needed later from vendors –Disaster Supplies Crate What would we put into an 8x12 truck for rapid evac? –Equipment for a mobile or relocated university command post Laptops, radios, phones, etc. –Identify Key IT personnel Who does what w/back-up “Scoop ‘em up” –Where might we go?

17. Your Views/Experiences? Does your IT DR plan included a ‘broader impact’ consideration? Or Does your Institution have such a plan contingency? –Has that changed since 9-11 or Katrina? Have you had to use it? How’d that go? How do you think your Administration would respond to this type of broader crisis?

18. Survivor Disaster Recovery You’re the last ones standing Dealing with unimaginable demands –Start imagining it Do you have a stock of equipment to set up a large support operation in short-order? –Networking gear, computers, cables, supplies, telephone service Value of a flexible and capable staff –They’ll see things no one should have to see Consider how you’ll do all this on top of your normal jobs, as campus life resumes and student enrollment increases How ready is your campus administration to take on the role of disaster response center? –Facilities, public safety/police, communications, academic affairs –Is the CEO (Chancellor, or President) prepared?

19. Lessons Learned at LSU Have a good stock of networking equipment, and mobile and desktop computing in the storeroom –Plan to raid campus labs & empty desks if need be Have strong relationships with key vendors Great to have terrific, dedicated, service-oriented people Architectures count – how divisible are the components? What’s removable as a component, and what’s too-tightly integrated? Be prepared to be flexible; adapt, improvise, overcome –Don’t be thin-skinned

20. Lessons Learned continued Good to “sit at the big table” –But know when to speak, and when not to – if you’ve watched ‘Survivor’ you know what I mean Keep your friends close – and don’t have enemies Everything we’ve been saying about the strategic value of IT is valid; IT enables everything in the 21 st Century –But even now – does HE administration ‘get it?’ Disaster Recovery and Business Continuity Planning is not a luxury

21. Your Views/Experiences? Have you ever been on the edge of someone else’s disaster? Does your DR/BCP plan include this aspect? Does your institution have such a plan component? –Has that changed since 9-11 or Katrina? How do you think your Administration would respond to this type of broader crisis?

22. Hurricane Season 2006 Starts … next week! We’re working on a basic, nimble IT DR/BCP document – will be done by 5/31 –Addressing all three forms of disaster –Requesting funds ~$500K –Primed the pump with $100K from year-end cash Campus is constructing a ‘permanent’ EOC –Rather than relying upon hastily assembled one –Spending ~$150K to do it Chancellor put out a call for DR/BCP plans across key campus units –Met with EOC commander to review plans and coordinate preparation

23. IT DR/BCP at LSU – Post-Katrina Before 29-August 2005 Information Technology Services (ITS) has always made backups of data on its servers - Mainframe (daily, stored off-site in Port Allen) - Lots of non-mainframe servers (housed in BR) Fairly strong, redundant network topology Computer Center has a UPS, generator, chillers Stand-by or parallel servers in separate building on campus

24. IT DR/BCP at LSU – Post-Katrina After 15-September 2005 Organizational focus/emphasis on DR/BCP - Creates IT DR/BCP Officer; within IT Policy & Security Office Started to update disaster recovery plan –The ‘old’ formal one was circa 1984 (useless) Double-checked Payroll contingency - Able to meet payroll deadlines without mainframe

25. IT DR/BCP at LSU – Post-Katrina More work Long distance offsite storage --- Houston, Birmingham, Nashville, Dallas? –Security of tapes with Breach Notification Law Establish LSU Rapid Recovery Site - 100 + miles, not south - Relocate servers - Increase stand-by presence Formal hot-site contract for mainframe E-mail emergency service contract

26. Some Key Items to Address Conduct a risk evaluation and business impact analysis. –Define and prioritize your mission-critical systems. –What must be recovered immediately (within 24 hours)? What can wait (and how long)? Identify your backup/recovery site. –Vendors provide offsite storage of mission-critical backup tapes, remote data centers, and temporary office locations. –Consider co-sourcing or reciprocal agreements with other regional higher education institutions or for facility and equipment use. –Develop a plan with your key hardware vendors to rapidly replace any damaged hardware/communication systems.

27. Some Key Items to Address continued Develop and document a communications and contact plan. –The Internet can be a crucial external communications tool. –Designate and equip a central command and communication center. Who will be primary spokesperson to respond to questions, as well as how information will be disseminated? Be wary of relying on wireless. –Cellular circuits can quickly become overloaded and unavailable during a regional or national incident. –At your centralized command and communication center, use a variety of communication links (Web, cellular, fax, landline, radio, and sticky-note bulletin boards).

28. Key Items to Address in a DRP continued Don’t forget about the people side of your institution. –Do you know where your staff, faculty, and students are? Do you have a tracking or check-in system? –Who is on your IT Emergency Response Team? How will you communicate with them? –Do you need temporary offices, temporary classrooms, or temporary housing? Finally, document and distribute your plan (including hard copies). –Test it, evaluate it, fix it, and retest it. –Do this at least annually, as well as after major system or infrastructure upgrades.

29. Ozymandias I met a traveler from an antique land Who said: Two vast and trunkless legs of stone Stand in the desert. Near them, on the sand, Half sunk, a shattered visage lies, whose frown, And wrinkled lip, and sneer of cold command, Tell that its sculptor well those passions read, Which yet survive, stamped on these lifeless things, The hand that mocked them, and the heart that fed, And on the pedestal these words appear: "My name is Ozymandias, King of Kings: Look upon my works, ye Mighty, and despair!“ Nothing beside remains. Round the decay Of that colossal wreck, boundless and bare The lone and level sands stretch far away. -Percy Bysshe Shelley

30. Thoughts 8 months after In a disaster, do the rules – and the plan – go out the window? Isn’t the very nature of disaster its unpredictability? Can we ever plan for every possible event and circumstance? During the crisis will we have time to refer to a detailed disaster plan document? Not a chance! No Yes

31. What then should we do? Focus on the process of planning and not on the plan itself (so sayeth Capt. Joe Castillo USCG) Examine how we will position ourselves and our assets to be flexible in responding to a disaster Focus on knowing what will need to be done in the first stages, what we’ll need to do those things, and who will do them Plan to be flexible. Plan to improvise, adapt, and overcome Drill on these things

32. Your Views? Do you think having a DR/BCP matters, or is ad-hoc the best way to handle things? How much is too much to spend on DR/BCP –From your perspective –From the viewpoint of your Administration

33. Again, What’s Important? Hardware and facilities can be replaced in the periods following a disaster Data is the primary focus of what you need to be prepared to restore and the basis of continuity People are your most key asset

34. IT Personnel = First Responders Everyone on campus uses and relies upon IT today (whether they fully realize it or not) If Anyone is on campus, they’ll need IT (and thus need IT support) Ergo, if someone is on campus during a disaster, they’ll need IT and IT personnel on campus also.

35. Your Views? Do you think this is true – that IT is now a ‘first responder?’ Is your organization (i.e., staff) ready to accept and fulfill this role? Are you ready for them to do this – are you (as a CIO or IT leader) prepared to deal with this aspect of our role?

36. “We had a failure of imagination.” Lessons from NASA; Apollo 1 and Columbia Shuttle disasters We need to imagine the questions first so that we can find the answers We need to – as a community – seek answers together –How can we leverage national cyberinfrastructure? –Individual arrangements versus broader approaches How seriously do CIOs, take the strategic nature of IT? How about your administration? –IT truly is an enabler of everything we do now –Are our people ready to be First Responders?

37. Those who cannot learn from history are doomed to repeat it. -George Santayana CIOs can no longer say they can’t imagine what could happen – because it just did. –Or an earthquake, or a tsunami, or a terrorist attack, or an accident, or a pandemic Next time, you may not be watching it on CNN – you may be living it Now is the time to think, plan, and take action – later it will be too late

38. Brian D. Voss Chief Information Officer Business Continuity Planning In a Post-Katrina Environment May 2006