1. From Entrepreneurial to Enterprise IT Grows Up Nate Baxley – ATLAS nbaxley@illinois.edu Rami Dass – ATLAS rpd@illinois.edu

2. Learn@Illinois A partnership between Colleges of LAS and Education Learning Management System (LMS) based on Moodle – Open Source – Released in 2002 – Currently at version 2.7 – Over 80,000 installations worldwide – http://moodle.org

3. Learn@Illinois A brief history Moodle came to LAS and Educaiton in 2003 – http://courses.las.illinois.edu – http://learn.education.illinois.edu Implemented on single machines 2010 – Sharing a single developer 2012 – Service was combined – http://learn.illinois.edu Statistics (Fall 2014) – 500 courses – 26,000 enrollments 20,000 daily logins 23,000 unique users

4. Early Days Separate Installations A single web server in tandem with a separate database server Every year courses were copied manually along with a new database Rosters were managed with daily file uploads or registration keys sent out via email Code was pushed as needed with minimal testing Changes were often implemented as soon as they were ready

5. Early Days Separate Installations (cont) Support was by email directly to the developers or service managers Monitoring was done through home grown script Logging was decentralized, each host kept logs locally We lacked scalability and flexibility

6. Growing Up The Partnership Evolves Opportunity to design infrastructure for scalability and stability Code release procedures were tightened Support team was expanded Partnership became stronger Moodle 2.0 release coincided with partnership – Major code changes – Revisit old plugins

7. Growing Up Evolution of Infrastructure Scalable, flexible, and redundant Multiple web front ends were deployed behind a load balancer Offloaded scheduled task from the web front ends to a dedicated server Moved from a single individual to a team to better manage the rapid growth Transparent modifications to the systems with minimal interruptions to the users

8. Growing Up Dev & Release Process Improved reliability Thorough documentation of changes Reduced the frequency of releases Increased usage made ad hoc fixes less desirable Established a stricter dev/test/prod environment

9. Growing Up Dev & Release Process (cont) Changes tracked in Redmine and bundled into releases Development team expanded to support other LMS CVS and GIT used for code versioning Code releases through direct file transfers

10. Today Infrastructure The database runs on a MySQL cluster with nodes spread across 3 data centers for high availability. We have several web front ends spanned across multiple data centers The file server replicates to a “hot spare” server in another data center that can be manually switched to.

11. Today Infrastructure (cont) The framework is duplicated to varying degrees for the dev, test, and shadow systems Use Zabbix for monitoring, and get a lot of information on the health of the systems keeping us ahead of the curve Logging is centralized using Graylog to review the logs and event notifications

12. Today Application/Service Code releases are done through GIT 5 issue severity levels ranging from “emergency fix needed” to “annual major upgrades” Issues tracked from requirements gathering through testing and resolution Coordination of support team between partners Instructional designers at LAS, Education, and CITL help faculty design and build their courses

13. Future Growth Automatic failover for file server Implement configuration management to automate, standardize, and document infrastructure changes. Future performance increases to handle growing enrollments. Dedicated "admin" web front end, tweaked to handle power user requests Moving beyond building maintenance tools to building teaching tools

14. Questions Nate Baxley – College of LAS - ATLAS – Client Relations Manager – nbaxley@illinois.edu Rami Dass – College of LAS – ATLAS – Lead IT Infrastructure Engineer – rpd@illinois.edu