1. Using CMS Data as a Force for Good? Applying Academic Analytics to Teaching and Learning Leah P. Macfadyen Science Centre for Learning and Teaching, UBC, Canada Shane Dawson Queensland University of Technology/University of Wollongong, Australia

2. Project Foundations 1.Emergence of “Academic Analytics” 2.Increased use of ICTs in teaching and learning 3.Increasing availability and detail of Course Management System data 4.Increasinging interest in socio-constructivist learning theory (and its implications)

3. Teaching and Learning for Engagement Socio-constructivist theories of learning Importance of engagement and learning communities Increasing use of ICTs Questions i.Which web-based tools and activities can promote student engagement and community online? ii.How do engagement and sense of community correlate with student achievement?

4. CMS data CMS usage is now prevalent (US data 2006: 93% student adoption in average of 2.5 courses; UBC data: >25,000 student users of Bb Vista) CMS data is immediate (can be mined at any time) CMS data is non-intrusive (does not require faculty intervention) (Bart Collins, Purdue University, 2006)

5. Bb Vista Tracking Data

6. Data points available to instructors Date of first access# Chat sessions Date of last access# Assessments begun Total # sessions# Assessments finished Total time onlineTime on Assessments # Mail messages read# Assignments read # Mail messages sent# Assignments submitted # Artifacts createdTime on Assignments # Artifacts saved# Goals viewed # Discussion messages read# Weblinks viewed # Discussion messages posted# Content folders viewed # Viewed Calendar entries# Files viewed # Added Calendar entries# Media library entries viewed # Media library collections viewed

7. Project goals Develop a data interpretation and visualization tool to: aid faculty and students in the interpretation of the vast array of data currently captured by Bb Vista permit ongoing formative evaluation of student engagement in learning activities and allow early identification of at risk students provide administrators and institutions with benchmarks of activity, usage trends, disciplinary differences

8. Case Study: BIOL200

9. Descriptive Data BIOL 200 online, 2008 BIOL200, 2007 web-supported N (completers)1191112 Average final grade60%65% Average online sessions/ student 15377 Average hours online/stud ent 10241 Average discussion messages read/student 4589643 Average messages posted722 Average # files viewed826423

16. Summary of correlations (bivariate) Variable BIOL200 online BIOL200 web-supported RR2R2 RR2R2 # Disc. messages posted.522.272.206.042 # Files viewed.332.110.244.060 # Online sessions.402.162.231.053 Total time online.338.114.226.051 # Assignments submitted.256.066n/a # web links viewed.251.063ns # Mail messages sent.282.080ns # Disc. messages read.290.084.183.033 # Assessments finished.310.096.290.084

17. Bivariate Correlations Categories of variables: 1.Measures of effort time online, number of sessions online, time on specific activities 2.Engagement and community activities discussion forums, chat 3.Administrative activities mail, calendar, announcements, tracking, grades 4.Content-related activities files, folders, media 5.Assessment activities assignments, assessments

18. Predictive modelling BIOL200 online multiple regression (with variables for tools used) BIOL200 web-supported multiple regression (with variables for tools used): (Compare to: Morris, Finnegan & Wu (2005): R 2 =.310 for online courses) RR2R2 F changeSig. F change.660.4354.503.000 RR2R2 F changeSig. F change.316.10010.119.000

19. Discussion participation….

20. Visualizing student engagement Instructor http://www.randomsyntax.com/blackboard-forum-social-network-analysis/

21. Instructor Disconnected students

22. Institutional tool use 27 Aug 200706 Jan 2008 Percentage of total interactions

23. Lessons learned so far… Some (but not all) CMS data variables are useful predictors of eventual student achievement Several seem to support theoretical propositions regarding the importance of community in learning Correlation ≠ causality… Significance of variables depends on course design