1. Making the best of a bad situation — Dealing with the aftermath of a major server crash Patty Brown & Roger Gant Instructional Design & Development Elon University Copyright Patty Brown & Roger Gant, 2004. This work is the intellectual property of the authors. Permission is granted for this material to be shared for non-commercial, educational purposes, provided that this copyright statement appears on the reproduced materials and notice is given that the copying is by permission of the authors. To disseminate otherwise or to republish requires written permission from the authors.

2. Alternate title: The crash of ’03 — How we survived without being lynched

3. Overview Focus is not technical Background Chronology of crash-related events Lessons learned Continuing challenges Suggestions, resources

4. Background Elon University Private university in NC Piedmont Focus on undergraduate education & liberal arts 252 full-time & 72 part-time faculty 4,584 students in fall 2003

5. Background Technology @ Elon Academic buildings & res. halls wired #43 on Intel’s "Most Unwired College Campuses" list Student access to 620 workstations in 19 computer labs 98% of students bring their own computers

6. Background Course Mgmt System @ Elon Bb campus since CourseInfo in 1998-99 Our role –End-user support & training –Sys admin privileges for managing daily course & account operations Sys Admin unit –Handles hw/sw installation & configuration –Reports to different manager

7. Instructional Des. Ins. Support. Liaison Assoc. Director Instructional Des. Multimedia Dev. Liaison for Sci/Math n = 53

8. Background Course Mgmt System – Spring 2003 Running Bb 5.5 ‘Level I’ –Basic, non-enterprise version –1 server –Heavily used, widely accepted Plan to upgrade from Bb 5.5 to Bb 6.0 enterprise level over summer for use in fall

9. Chronology of crash The best laid plans of management & system administrators often go awry June 2003, shipping mix up & subsequent negotiations cause delay Revised plan: –Postpone rollout until spring 2004 –Test small # of courses winter term

10. Chronology of crash “Uh oh…” In October, log files cause storage issues Resourceful fix enables business as usual Routine safeguards (backups) normal to all appearances Preparations continue for rollout of Bb 6 Server crashes – hard – just before Thanksgiving Sys admin eventually restores from system backup and log files

11. Chronology of crash From “Uh oh” to “Oh, no!” Server restored BUT… All transactions after ‘fix’ are lost — appx. 4 weeks of data –Midterm grades and more No better outcome despite all attempts to recover data Some content files accessible from server Alternate solution found for original storage problem – so 5.5 OK through Dec.

12. Artner’s Law of Intermediate Crisis Management: “The only small problem is one that happens to the other guy.”

13. Response and recovery Now what? Meetings & communications –No search for guilty, no blame for innocent Lesson: Working relationship is all important Decide to move up Bb 6 rollout to winter We’re still in this together –System Admin readies new servers –IDD deals with Fallout from crash Creating & implementing expedited rollout

14. Response and recovery Fallout from crash Communicating frequently to faculty –Lesson: An e-mail in time saves 9 Responding to calls, e-mails Long hours in December Managers don flak jackets Faculty response: Sympathy! –Lesson: Good service and relationship built over time pays off

15. Response and recovery Other saving graces Official grades not maintained in Bb Most faculty had source material & grades outside of Bb Made salvaged Bb files available by request Timing; semester ends soon

16. Response and recovery Getting to Bb 6 sooner, not later Archiving Bb 5.5 sites, restoring in 6 Coordinating creation of accounts, courses –Batch enrollment a plus Training, user support needs for Bb 6 –Workshops –New front end for new log-in page

18. Response and recovery Getting to Bb 6 sooner, not later Archiving Bb 5.5 sites, restoring in 6 Coordinating creation of accounts, courses Training, user support needs for Bb 6 –Workshops –New front end for new log-in page Shepherding the switch

19. Crisis management challenges Physical & organizational structure –Reassess? Increased collaboration, joint decision-making Calls for informal relationship-building

20. Ongoing challenges An ounce of prevention… Differing priorities for System Admin & IDD –Means we have to forecast & articulate needs Shared responsibility –Requires proactive measures Planning for upgrades Communication

21. Are you ready for a small catastrophe?

22. Suggestions Identify areas of potential concern Build relationships in advance Make plans but stay flexible Be prepared –To communicate, communicate –To modify plans Don’t sweat the small stuff –To negotiate solutions –To accept less than 100% satisfaction Books, articles, consultants Make your own action plan & test

23. Keep working at it…