From Entrepreneurial to Enterprise

IT Grows Up

Nate Baxley – ATLAS nbaxley@illinois.eduRami Dass – ATLAS rpd@illinois.edu

Learn@Illinois

• A partnership between Colleges of LAS and Education

• Learning Management System (LMS) based on Moodle– Open Source– Released in 2002– Currently at version 2.7– Over 80,000 installations worldwide– http://moodle.org

Learn@IllinoisA brief history

• Moodle came to LAS and Educaiton in 2003– http://courses.las.illinois.edu– http://learn.education.illinois.edu

• Implemented on single machines• 2010 – Sharing a single developer• 2012 – Service was combined

– http://learn.illinois.edu

• Statistics (Fall 2014)– 500 courses– 26,000 enrollments

20,000 daily logins23,000 unique users

Early DaysSeparate Installations

• A single web server in tandem with a separate database server

• Every year courses were copied manually along with a new database

• Rosters were managed with daily file uploads or registration keys sent out via email

• Code was pushed as needed with minimal testing• Changes were often implemented as soon as

they were ready

Early DaysSeparate Installations (cont)

• Support was by email directly to the developers or service managers

• Monitoring was done through home grown script

• Logging was decentralized, each host kept logs locally

• We lacked scalability and flexibility

Growing UpThe Partnership Evolves

• Opportunity to design infrastructure for scalability and stability

• Code release procedures were tightened• Support team was expanded• Partnership became stronger• Moodle 2.0 release coincided with partnership

– Major code changes– Revisit old plugins

Growing UpEvolution of Infrastructure

• Scalable, flexible, and redundant • Multiple web front ends were deployed behind

a load balancer• Offloaded scheduled task from the web front

ends to a dedicated server• Moved from a single individual to a team to

better manage the rapid growth• Transparent modifications to the systems with

minimal interruptions to the users

Growing UpDev & Release Process

• Improved reliability• Thorough documentation of changes• Reduced the frequency of releases• Increased usage made ad hoc fixes less

desirable• Established a stricter dev/test/prod

environment

Growing UpDev & Release Process (cont)

• Changes tracked in Redmine and bundled into releases

• Development team expanded to support other LMS

• CVS and GIT used for code versioning • Code releases through direct file transfers

TodayInfrastructure

• The database runs on a MySQL cluster with nodes spread across 3 data centers for high availability.

• We have several web front ends spanned across multiple data centers

• The file server replicates to a “hot spare” server in another data center that can be manually switched to.

TodayInfrastructure (cont)

• The framework is duplicated to varying degrees for the dev, test, and shadow systems

• Use Zabbix for monitoring, and get a lot of information on the health of the systems keeping us ahead of the curve

• Logging is centralized using Graylog to review the logs and event notifications

TodayApplication/Service

• Code releases are done through GIT • 5 issue severity levels ranging from “emergency

fix needed” to “annual major upgrades”• Issues tracked from requirements gathering

through testing and resolution• Coordination of support team between partners• Instructional designers at LAS, Education, and

CITL help faculty design and build their courses

Future Growth

• Automatic failover for file server• Implement configuration management to automate,

standardize, and document infrastructure changes.• Future performance increases to handle growing

enrollments. • Dedicated "admin" web front end, tweaked to

handle power user requests• Moving beyond building maintenance tools to

building teaching tools

Questions

• Nate Baxley – College of LAS - ATLAS– Client Relations Manager– nbaxley@illinois.edu

• Rami Dass – College of LAS – ATLAS– Lead IT Infrastructure Engineer– rpd@illinois.edu