1. Reinventing Blackboard Backups California State Polytechnic University, Pomona

2. Introduction Reinventing Blackboard Backups Presenters: –David C. Lyon Bb Systems Administrator/Analyst –Pauline Salim Learning Management Systems Specialist

3. About Cal Poly Pomona Polytechnic – Integrating Technology with a Liberal Arts education Learn-by-doing philosophy One of 23 Universities in the California State University System 17000 FTE Land Grant from W.K. Kellogg

6. OVERVIEW LMS Organization and History Blackboard Deployment WebCT and Blackboard Backups Course Archives and Backup Issues Backup Solution and Issues Restore Procedure Future Plans Backup Monitoring Policy Development Challenges

7. LMS Management Instructional and Information Technology Division Learning – LMS Direction, Course design/management, Web documentation, User support, Restore requests Helpdesk & Studio Six - General help requests –Studio Six - high-end digital video and web development computing lab (course design help) Systems: UNIX - Systems administration Systems: application – Installation/mgmt, backups, Web CGI Applications - Database Management Computer Ops - Tape library management

8. Organization Instructional and Information Technology Learning Studio Six Systems LMS Applications UNIX Admin Applications DBA Support Computer OpsHelpdesk

9. Virtual Learning History Course Web Pages – late 90s WebCT – 2000 Blackboard Pilot – 2003 Blackboard Production - 2004 Content System Pilot – 2004 Bb 6.2.23/Solaris 2.8 currently Webct 4.1/Solaris 2.8 currently

10. WebCT Current Usage

11. Blackboard Current Usage

12. WebCT Backup Procedure All courses were backed up (about 800) daily Course archive tool created zip archives Zip archives copied to tape daily Recent zip archives kept on-line Restore requests made through Learning (informally)

13. Blackboard Pilot WebCT users were accustomed to individual course archives and on- demand restores – no course recycle policy Migrating them to Bb would be difficult if the level of support was not on par with WebCT

14. Reliable Backups Don’t assume what you back up can be restored Closely monitor backup process Tape quality and storage are key Plan for when a backup fails – what action will you take? Strongly consider disaster recovery

15. Developing Bb Backups Bb Support recommended Bb archive tool but did not have a best practice Wrapper script was developed Script was scheduled to run daily All Bb courses were archived to zip daily As with WebCT, all zip archives were copied to tape daily

16. Bb Production Backups The beta course archive script was ported Complete oracle exports and cold backups were done daily (batch_ImportExport.sh - java) Oracle and course zip archives were copied to tape

17. Why Course Archive Tool? Identify course issues – use restore tool to repair a course Restoring from DB restore complex Report course issues (next slide) Quick restores without DBA involvement – short staffed Restoring multiple dates would require more work of DBA and take longer Less reliance on DBAs Empower LMS specialists to restore Partial restores

18. Course Backup Error

19. Bb Backup Problems Downtime of 2 hours/night advertised The number of courses grew The downtime grew to over four hours Users were unhappy! WebCT did not have downtime for zip archives (not database driven) WebCT problems were related to tape overruns

20. Backup Solution LMS Director suggested disk-to-disk (back up against the image copy) Feasibility was studied and meetings were held There were differing opinions about the proposed solution and necessity of individual course archives Course archive tool was thought to be inadequate - unsubstantiated In the end, the solution had to be mandated by management

21. Bb Architecture - Before Front End BB6 DB and File Bb Shared Hot Backup Backup - half

22. Bb Architecture - After Front End BB6 BBBACKUP Backup Front End DB and File Bb Shared Bb Shared Copy Hot Backup Backup all

23. Solution: Technical Details Bb shuts down at 2:00 AM Oracle cold backups Oracle export Solaris snapshot of shared device (content) Bb starts up at 3:45 AM Backup database created Backup image of content (from snapshot) created Course archive runs on backup server - Map shared drive – connect to backup DB Disk copies are written to tape

24. Solution: Scripts

25. Problems with Solution Lack of backup policy Lack of course controls (quotas, etc) Weekly backup running 24 hours Handful of very large courses Snapshot mechanism issues –Archive tool needs write access to shared device

26. Modifications Snapshot used to create disk image copy – rather than a live mounted snapshot Policy development in progress – talking to power users (encourage streaming server for video/music) Course sizes are monitored Back up all organizations and only current term courses for weekly backup Backup is more stable, finishes completely by afternoon

27. Course restore procedure –Faculty fill out course restore form –Form submitted to LMS Specialist (I&IT Learning) –Restore options Restore it from our archive tape –Up to 4 business days Restore it from faculty’s personal backup file –Up to 2 business days –Systems restores the course YEARTERM_COLLEGE_COURSECODE. SECTION.DATE_BACKUPDATE (05Su_CBA_CIS101.01_20060215) Restoring Courses – How?

28. Restore Courses

29. Ticketing system is utilized – Learning assigns ticket to Systems Systems requests tape from operations (if backup no longer on-line) Restore of zip archive done manually (Solaris ufsrestore) Script to complete un-zip to a course Ticket is returned to Learning Specialist – with all history records

30. Script Sample All activity is logged so that we can go back and identify when a course was restored – see what errors might have occurred during the restore.

31. WebCT Improvements Currently active courses archived (WebCT experienced some growth in course size) Backup and restore procedures very similar to Bb Retention policy similar

32. Backing Up and Archiving Courses Why? Is it Useful? Clean up procedures –Courses over two quarters old are archived and removed –Keep the courses that faculty wish to keep How is course back up and archive useful to faculty? –Faculty can always request to restore the deleted course –Need to find materials from deleted course

33. Backing Up and Archiving Courses Why? Is it Useful? Resolving course issues –Data or files deleted from Bb accidentally –Data corruption Example: Valentine’s day incident Some data corruption in discussion table resulted from a hardware reset. Faculty could view a previous restored version – showing all discussion messages.

34. Backing Up and Archiving Courses Why? Is it Useful? –Grade dispute – student inadvertently removed from course Faculty could view several previous versions of the course Faculty was able to reveal the work the student had completed – examining previous courses Could have queried database for specific changes

35. Backing Up and Archiving Courses Why? Is it Useful? –Black Monday – enrollment feed removed students Organizations were untouched Archives enabled restore of courses only

36. The Future Bb Building Block – restore course request with list selection (My backups) Plan for double number of courses – add backup front end or add a thread Implement course quotas Implement content system – courses will be smaller in size Distributed architecture (backup DB server) Shorten down time for Oracle backup (hot backups)

37. Backup Monitoring All LMS backups monitored as a group Completion success monitored Completion time is monitored Other measurements Analyst automatically paged, operators emailed (success/failure) System health monitored (http, login) Periodic testing of restores and starting of backup Bb application

38. Backup Monitoring

40. Tape Rotation Daily – one week Weekly – one month Monthly – permanent Quarterly – permanent (all courses accessed since last quarterly) Tapes are kept offsite - daily

41. Lessons Learned Develop good policy before implementing LMS (Course size limits, courses that are backed up, retention policy) Listen more to faculty and students and understand their expectations Tape rotation should be strong consideration Do not leave decision to Computer Ops Make spare copy on DVD at quarter break Do not allow reuse of WebCT courses – implement naming convention Plan for growth in advance

42. Policy Development Course Restore There are two ways to restore old courses that have been deleted during quarterly maintenance. LMS administrators can restore the course archive from our system that is kept on a backup tape or from faculty’s personal backup or archive file. System Course Restore. Faculty can request a restore through a form that can be found at http://www.csupomona.edu/~iit/learning/blackboard/vle_restore.shtml for both Blackboard and WebCT. The form should indicate “System” for archive type. The process will take up to FOUR business days. http://www.csupomona.edu/~iit/learning/blackboard/vle_restore.shtml Personal backup or archive file restore. Faculty can request through a course restore form that can be found at http://www.csupomona.edu/~iit/learning/blackboard/vle_restore.shtml for both Blackboard and WebCT. The form should indicate “Personal” for archive type. Once the form submission has been submitted, the faculty must bring the zip archive file to LMS Support immediately. The LMS support will open the archive zip file through an admin access. It will take up to two business days after the zip archive file has been received. http://www.csupomona.edu/~iit/learning/blackboard/vle_restore.shtml

43. Challenges Potential of 100 new courses (from WebCT) Bb course growth (double/triple) Train faculty to mange courses – result in less restores Pandemic – all courses have Bb shell

44. Closing Email contact information dclyon@csupomona.edu psalim@csupomona.edu Scripts available upon request Questions?