1. www.ts.vcu.edu How We Maintain a Healthy Bb Environment Sheila Chandler Director, Learning Systems Virginia Commonwealth University

2. Presentation Overview Background –Who am I? –Virginia Commonwealth University (VCU) –VCU Blackboard Environment ____________________________________ How We Maintain a Healthy Bb Environment Sharing Lessons Learned

3. Who am I? High School – Computer programming (IBM) University of Louisville –Sociology Degree –First programming job (PDP 11/45 - BASIC) Community Hospital of Central California (MUMPS, BASIC) Virginia Commonwealth University (VCU) -Accounting Degree Auditor of Public Accounts – CPA, CISA VCU –Programmed on Web Course in a Box (1998 to 2000) –Blackboard purchased WCB (2000)

4. About VCU Richmond, VA Students: 32,000+ Faculty: 1,927 Adjunct: 1,161 Staff: 2,586 Monroe Park, MCV, Qatar, NOVA

5. Bb App Environment Bb Application (2000 to present) –Bb Classic Enterprise (7.3 -> v9 – May 2010) –Bb Content Management System Building Blocks –Bb Faculty Tools (VCU developed) –CPS Connection (Clickers) –Echo360 Content Creator –Health Check (modified) –Help Desk Tools (VCU developed) –Wimba –SafeAssign –SoftChalk

7. Bb User Environment Fall 2008 3319 Courses 2031 Instructors 29688 Students (85%) 723 Organizations Spring 2009 3099 Courses 1983 Instructors 28236 Students (87%) 804 Organizations

8. Lesson Learned #1 Designate Bb Coordinator and Support Team VCU Support Team (Tribe) –Technology Services Learning Systems Help Desk University Computer Center Networking –Center for Teaching Excellence –Support staff in Schools (School of Nursing, Business, etc) –Vendor (Blackboard)

9. Lesson Learned # 2 Recognize Mission Critical Path to Mission Critical –Student demand (85%+) –Faculty use (email, gradebook, document storage) –Administration/Staff found uses –Keep statistics – show growth

10. Lesson Learned # 3 - Scale Out

11. Lesson Learned #4 Request System Audit

12. Lesson Learned # 5 Monitor and Take Action Hobbit (Internal) Pingdom (External) Custom Scripts –Check db connections –Search logs for Oracle errors –Scan Snapshot files (header & footer) –Monitor SAN space (set to expand at 90%)

13. #5 Monitor and Take Action System cleanup –Rotate logs –Run Bb Cleanbb once a week –Restart app servers twice a week Track usage and react to trends Ask Users –My VCU Portal Polls –Annual Tech Fair suggestion box –Talk to Students on Campus –blackboard@vcu.edu

14. #5 Monitor and Take Action External Listserv –ASU list (BBADMIN-L@lists.asu.edu ) –BIE list (BIEACAMDEMIC@lists.blackboard.com ) –Developers (BB-OPENSRC@lists.vcu.edu ) Internal VCU Listserv –Bb-core –Bb-mgmt –Bb faculty list

15. Lesson Learned #6 Document and Communicate Technical –Bb Diary –Internal wiki – notes –BbCMS – Documentation –Emergency Call list End-User –Help Desk - blackboard@vcu.edu –Blogs (Tech Services, Bb system-wide, Bb faculty) –Learning Systems Newsletter –Technology Services Knowledge Base (http://www.ts.vcu.edu/kb/blackboard.html)http://www.ts.vcu.edu/kb/blackboard.html

16. #6 Document/Communicate Regular Meetings –Weekly LS meeting –Monthly Bb Tech meeting (agenda & minutes) –Monthly Center for Teaching Excellence meeting –Periodic meetings with IT Directors of Schools

17. #6 Document/Communicate Special Meetings –Special Task Force (iptables fix) Discovered that net.ipv4.netfilter.ip_conntrack_tcp_be_liberal = 1 doesn't survive an iptables restart –Bb TSM bi-weekly meetings Periodically request to review Bb tickets

18. Lesson Learned #7 “No such thing as a SMALL change” ALL Changes follow Written Procedures –Development Environment (maybe multiple times) –Test Environment (always multiple times) –Production

19. #7 “No such thing as a SMALL change” System Maintenance – Regularly Scheduled –Second and Fourth Friday of the month –Perform Downtime Checklist One week before downtime During downtime After downtime –Group IM (Instant Messenger) –Follow emergency communication plan (if needed)

20. Lessons Learned Summary 1.Designate Bb Coordinator and Support Team 2.Recognize Mission Critical 3.Scale out 4.Request System Audit 5.Monitor and Take Action 6.Document and Communicate 7.“No Such Thing as a SMALL Change”

21. Additional Information –Sheila Chandler (sdchandl@vcu.edu)sdchandl@vcu.edu –Bb Connections site –http://healthybb.wetpaint.com Thank you to… –Learning Systems Staff –University Computer Center Staff –Networking Services –Center for Teaching Excellence