VERMONT STATE COLLEGES WHO WE ARE

VERMONT STATE COLLEGES SHARED SERVICES From the data center: (located in a state-owned building) Wide-area network & single Internet connection ERP Identity & Access Management Portal IT Help Desk Data warehouse Conference & events application Enterprise IT Staff: 6 programmers; 3 system admins; 1 network; 1 security & data center

VERMONT STATE COLLEGES DISASTER RECOVERY First plan in 2007 Approved by Council of Presidents Reviewed twice a year - updated February, 2011 two weeks Key feature: assumed local and business process planning “to accommodate the absence of critical systems for a period of up to two weeks” In-scope and out-of-scope clearly articulated

VERMONT STATE COLLEGES BUSINESS CONTINUITY Plans for admissions, accounts payable, accounts receivable, finance, financial aid, payroll and registrars processes Reviewed and updated annually Some business areas could go for several weeks without extraordinary measures Some could only function for few hours or days

IRENE

VERMONT STATE COLLEGES WHAT HAPPENED On Aug 28, 2011, at 8:13 PM, "Pollak, Dianne M" > Just got a text from michael... First floor of the hospital is under water... They are evacuating randall st. We are on generator. > > Dianne Pollak > Director of Administrative Information Systems Vermont State Colleges

VERMONT STATE COLLEGES WHAT HAPPENED

VERMONT STATE COLLEGES-WHAT HAPPENED By 6 am on 8/29 – water had filled basement and up to 5 feet of the first floor Generator under water = power gone Entire state office complex flooded Roads to town closed

VERMONT STATE COLLEGES BUSINESS IMPACT Two colleges had started classes Two colleges were about to start classes on Tuesday One college was still in “marketing mode”

VERMONT STATE COLLEGES COMMUNICATIONS

VERMONT STATE COLLEGES SHORT-TERM SOLUTIONS Monday - Emergency web site up by Monday Monday - Alternate addresses collected Tuesday - Wide-area network and Internet restored Monday - Colleges using internal Monday-Thursday – Some paper paychecks handwritten Tuesday-Friday – Air-conditioning and server issues Friday morning – ERP and other critical systems restored

VERMONT STATE COLLEGES EARLY LESSONS Even if you can’t afford what you really need, have a plan! Business continuity plans had some issues Never underestimate the power of denial Good relationships sustain you; bad ones are killer Stay on the lookout for good ideas and be flexible It’s true: never waste a good disaster Keep track of who’s doing what Communications are critical & difficult

Data center

DATA CENTER GENERATOR... High Water Mark

VERMONT STATE COLLEGES EARLY LESSONS Take notes everywhere, in every conversation Have a conference call system set up Texting is critical (and set up those contacts!) Have an alternate address for all critical contacts – IT and priority business areas Have a plan for alternate web site hosting Staff check-in: food and sleep Staff in different places on the resilience scale As everyone gets tired, mistakes will be made Danger in false sense of stability/reliability

UP NEXT: SNOWTOBER From: The EDUCAUSE CIO Constituent Group Listserv On Behalf Of Theresa Rowe Sent: Thursday, November 03, :44 AM To: Subject: Re: [CIO] Snowtober Every situation has been different, so you really need thinking, problem-solving people on hand, rather than a scripted action. Thanks so much for taking the time to write out the details of your experiences. We all learn from these posts. Our experiences with electrical power failures and generators have been very difficult to manage. Every situation has been different, so you really need thinking, problem-solving people on hand, rather than a scripted action. You've given us more to think about. Theresa

“Thinking, problem – solving people”

VERMONT STATE COLLEGES OPPORTUNITIES Reinforce the importance of both disaster recovery and business continuity planning and underscore the differences Make the case for disaster recovery resources-less difficult in the reality of business interruption Change things that were “unchangeable” Focus on what’s important: people and mission

VERMONT STATE COLLEGES TODAY Access to building restricted to 2-3 staff Off generator; data center is only room with power Legislature to determine future use of the complex IT Staff distributed across the state Building two data centers – looking at co-location Dealing with displaced employees Re-projecting the annual project list Getting back to “normal” - IT reporting cycle