1. “…most universities are doing at least the minimum necessary to protect computer data, although officials tend to think more along the lines of server crashes than widespread disasters. And while many institutions have plans to protect their data, few have well-thought-out plans for recovery once it happens.” Chronicle. February 28, 2003 A Tumultuous Year in the Life of a Data Center Presenter:Dwight Fischer, CIO Plymouth State University University System of New Hampshire dcfischer@plymouth.edu

2. Agenda Overview of UNH Enterprise Computing Review the numerous events—planned and unplanned—of the year Discuss implications of Disaster Plan Share lessons learned Discussion

3. University System of NH University of New Hampshire, Durham & Manchester Plymouth State University, Plymouth Keene State College, Keene College of Lifelong Learning

4. Enterprise Computing Group Serving UNH and U. System of NH Enterprise Computing Group Data CenterSysAdminDBAMISMgt. Rptg. Supporting Banner Student Banner Finance Banner HR (July 2003) Hosting and support for all UNH information systems

5. Impact of 9/11 Wake up call for many organizations Disaster recovery became paramount concern USNH was already in the process of rewriting the disaster plan Striving toward new policy and practice to educate the campuses and allocate appropriate resources to mitigate risk Business continuity suddenly the buzz

6. Chronicle of Events in 2002 Went live on Banner Finance Jan. 1 Completed construction a new data center Began planning of the data center move Rewrote disaster recovery plan Experienced disaster with major power outage Recovery and lessons learned Two staged move of data center

7. USNH Data Center Evolved in Computer Science and Engineering building, centrally located on the UNH campus Decision made in 2001 to move data center and combined IT support to new location on outskirts of campus Wide variety of VMS, UNIX, Windows and related systems An array of new hardware for ERP implementation CHALLENGE: Constant change for an operation that strives for control and steadiness

8. Disaster Recovery Plan Dusted off 10-year DR plan Updated the plan to incorporate new hardware, vendors and key campus personnel Developed alert, assessment, triage and communication plans among key technical and business stakeholders Incorporated various components of network and telecom systems Rolled out to stakeholders, posted in key areas for IT staff review and training

9. Planning the Data Center Move Reviewed business cycles Looked for timing with LEAST impact Developed consensus with key business partners on timing Divided move into two stages –VMS based Student Systems –UNIX based FIS, HR, development and others Engaged vendor partners to conduct moves

10. Overseeing Construction Design-build process Infrastructure Power and network HVAC Fire Suppression Alarms Vigilance over process was critical to ensure no corners cut

11. July 5, 2002… Major Power Outage Prolonged spell of heat and humidity AC units started to overheat Facilities put water sprinklers under them for cooling Water leaked into basement where transformer located Sump pump failed Bang!...transformer blew. Quick, the disaster plan!

12. Restoration of Systems Fuses had to be ordered Power generator brought in, insufficient New generator rented the next day (Saturday) Payroll needed to be run on legacy, fed from new Banner Systems powered up, down, up, down System restoration priorities changed over five days Planning the down time with users Bang.2 Final restoration

13. Debrief Facilities and computing staff –What happened? –What went well? –What could we have done differently? –Changes to DR plan Report and recommendations to executives –Alternative power –Alternative network Funding for new generators secured

14. Gaps No alternative data center site No redundant hardware beyond what we have in the main data center If data center was compromised, USNH would be out of operation for weeks, possibly months Continuing to work with business units to clarify limits to disaster recovery and identify plans for business continuity

15. Move in October All VMS/VAX systems hosting SIS Prepared campuses for being off-line up to a week Targeted systems to be back online in 2-3 days Successful, no problems

16. Move in December 75% of equipment and infrastructure moved Planned move over holiday break when all campuses quiet Aggressive work schedule by vendors and staff to be done in two days Accomplished

17. Subsequent Goals Internal audit & risk assessment Mitigation strategies based on cost and time to recovery Engage business users in genuine business continuity planning Engage engineer firm to identify redundant network and viable alternative data center site

18. Presentation of Risk COST Time to Recovery Greater investment in redundant infrastructure and hardware, quicker recovery Less investment in redundant infrastructure and hardware, greater time to recovery

19. Discussion Dwight Fischer, CIO Plymouth State University University System of New Hampshire dcfischer@plymouth.edu

20. Copyright Dwight Fischer 2003. This work is the intellectual property of Dwight Fischer. Permission is granted for this material to be shared for non- commercial, educational purposes, provided that this copyright statement appears on the reproduced materials and notice is given that the copying is by permission of the author. To disseminate otherwise or to republish requires written permission from the author.