2. Why Predictive Modeling? Predict future events using data already available. E-Learning schools collect lots of behavioral data useful in predictive modeling. Detailed tracking of student activities. Logins Submissions Class discussions etc.

3. Data Sources Enrollment histories and demographics Online student activity Advisement records Assignment & test scores

4. Goals Construct classification models to identify online students at-risk of not successfully completing their course. Course Success = Student completes course w/ ‘C’ or better. Estimate the probability that a student will be unsuccessful. Separate High, Medium, and Low risk students. Perform predictions immediately after the 1st week of class.

5. Variables Enrollment History New student? Has previously been successful in any course? Has previously been unsuccessful in any course? Is student re-taking this course? Has taken a Developmental level course? etc.

6. Variables (Cont’d) Current Enrollment Taking other classes as of start date? Taking more than 6 other credits as of start date? etc. Online Activity Logged in before start date? Logged in on 1 st day? Logged in on 2 nd day? Logged in during 1 st week? Has opened an assessment during 1 st week? etc.

7. Variables (Cont’d) Financials On financial aid? Advising Had previously scheduled an advising appointment? Demographics Age Gender

8. Naïve Bayesian Model Our models are constructed using the Naïve Bayesian technique. Chosen for its accuracy and robustness. Every variable always “has its say” on the prediction. Unlike other popular methods (i.e. decision trees). Presents fair interpretation of student’s likelihood to succeed.

9. Naïve Bayesian Model (Cont’d) How does it work? Model takes input from each variable independently. Each variable has unique influence on result. Weight of influence based on how often that variable has been associated with previous cases of success. Some variables more predictive – have more influence.

10. Naïve Bayesian Model (Cont’d) Most significant variables Final success probability derived from combined input of all variables. Increases Probability of Success? Variable Description Yes Logged in to course homepage during first week of class. Yes Logged in to course homepage on first day of class. Yes Logged in to course homepage prior to start date. No Student taking other classes at Rio Salado College simultaneously. Yes Student has been successful in a previous course at Rio Salado College. No Student has been unsuccessful in a previous course at Rio Salado College. No Student is re-taking the course.

11. Naïve Bayesian Model (Cont’d) Predicted outcome Risk level Risk LevelProbability of Success GreenGreater than or equal to 70% YellowBetween 30% and 70% RedLess than or equal to 30% Predicted Outcome Probability of Success SuccessGreater than 50% Non-SuccessLess than or equal to 50%

12. Demonstration Spring 2009 Students Taken from Humanities course - Start date January 26th StudentFinal GradeProbability of Success Student 1?? Student 2?? Student 3?? Student 4?? Student 5?? Student 6?? Student 7?? Student 8?? Student 9?? Student 10?? Student 11?? Student 12?? Student 13?? Student 14?? Student 15?? Student 16?? Student 17?? Student 18??

13. Demonstration Spring 2009 Students Taken from Humanities course - Start date January 26th StudentFinal GradeProbability of Success Student 1? 100% Student 2? 94% Student 3? 88% Student 4? 82% Student 5? 76% Student 6? 70% Student 7? 64% Student 8? 64% Student 9? 58% Student 10? 52% Student 11? 46% Student 12? 40% Student 13? 34% Student 14? 28% Student 15? 22% Student 16? 22% Student 17? 22% Student 18? 16%

14. Demonstration Spring 2009 Students Taken from Humanities course - Start date January 26th StudentFinal GradeProbability of Success Student 1? 100% Student 2? 94% Student 3? 88% Student 4? 82% Student 5? 76% Student 6? 70% Student 7? 64% Student 8? 64% Student 9? 58% Student 10? 52% Student 11? 46% Student 12? 40% Student 13? 34% Student 14? 28% Student 15? 22% Student 16? 22% Student 17? 22% Student 18? 16%

15. Demonstration Spring 2009 Students Taken from Humanities course - Start date January 26th StudentFinal GradeProbability of Success Student 1 100% Successful Student 2 94% Student 3 88% Student 4 82% Student 5 76% Student 6 70% Student 7 64% 43% Successful Student 8 64% Student 9 58% Student 10 52% Student 11 46% Student 12 40% Student 13 34% Student 14 28% 20% Successful Student 15 22% Student 16 22% Student 17 22% Student 18 16% Success rates calculated after final grades recorded

16. Model Validation Model applied to Fall 08 and Spring 09 enrollments from select disciplines 1. Predictions compared to outcomes already recorded in student information system. How accurately does model predict correct outcome? Predicted Outcome SuccessfulUnsuccessful 59%41%Successful Actual Outcome 30%70%Unsuccessful 1: Includes select courses from Science, Biology, English, Math, Languages, Communication, Social Sciences, Humanities/History, and Reading. Students in special programs, such as dual enrollment, military, and incarcerated re-entry were not included. Run Aug 09.

17. Model Validation (Cont’d) How well does model assign students to risk levels?

18. “Yellow” Students Green students succeed most often, Red students succeed least often, and Yellow students fall somewhere in the middle. Yellow students do not show a strong tendency towards either outcome – could go either way. Moral obligation to help Yellow students succeed. More on Red students a bit later…

19. Retaining the online student “Adult learning theories are built on the premise that teachers will assist their students to become self- directed and independent” (Muirhead and Min, 2001, p. 1). How does this best work online?

20. Retaining the online student Research suggests that students are unaware of what strategies are needed to be self-monitoring online learners (Muirhead and Min, 2001; Ormrod, 2004; Youngblood, 2001). This is the challenge and area faculty need to focus on to increase student retention and success.

21. Share your thoughts… Take a moment, what strategies can you incorporate online to help your students become self-monitoring learners? How can we help our students become effective online learners?

22. Focusing interventions Course design and interactions with instructors have been identified as key areas for online student success (Hillsheim, 1998; Paloff & Pratt, 2003; Swan, 2001) The Rio Model is one course many instructors, so the majority of our instructors do not control the course design.

23. What we do before the interventions… To enable students to be self-monitoring, our courses are designed to engage students in self-check activities Students are taught to monitor their progress by completing pre and post tests Students are engaged in summarizing lessons Best practices in andragogy are embedded in our online courses… We try to go beyond this with additional Faculty Interventions…

24. Department Interventions Communication Department: Students receive a phone call from the instructor during 2nd week, plus follow-up phone call 1 week prior to midpoint. Humanities/History Department: Phone call from instructional helpdesk. Social Sciences Department: Students receive a phone call from instructor

25. Phone Call to Student “I am calling because you are currently in your 2 nd week of your course. I want to make sure you have a successful experience. The first thing you can do to ensure this occurs is to be sure to communicate your content questions to me. If you do not understand something you have read or are attempting to answer, be sure to ask. If you are still struggling to grasp the content, you may want to speak with a tutor. We offer free tutoring at Rio Salado college, both in-person and online. Do you currently have any questions that I can assist you with?”

26. Student feedback… Student appreciate the time and effort put forth by the instructor! “Eight weeks ago I came into this course very unsure if I could do this extremely intense course, but I have made it, with your help and guidance I have learned so much in such a short period of time. I now find myself looking at the world a lot differently.” –Rio Student

27. Student feedback “Now that the course is over and grades are posted I just wanted to thank you for your help with this class. I have to tell you that this class was one of the classes I was dreading most on my prereq list, because I remember being so lost with it in high school. Your quick feedback (and patience) with my numerous questions was a lifesaver”. “It is really hard to get to know your instructors in an online class since you don't actually meet, but I wanted to let you know you are by far the friendliest, most helpful, upbeat, and prompt professor I have this semester, and I am taking 18 credits, so thats saying something”.

28. Student Success Interventions 3 disciplines conducted initial intervention trials in Summer I 2009 using the models described previously. Focus on Yellow students. Random 50% receive intervention, other 50% placed in control group for future comparison. 1 Intervention strategies varied by discipline. Designed by faculty chair. 1: Control group students still had access to all Rio Salado College services and were still exposed to the traditional forms of success and retention outreach efforts that all students receive.

29. Preliminary Results Category Successful Enrollments N N% Green16586.8%190 Yellow (Intervention Group)8675.4%114 Yellow (Control Group)7169.6%102 Red4554.9%82 Overall36775.2%488 *Preliminary results only include Summer I students graded out as of 9/23/09.

30. Preliminary Results (Cont’d) Intervention success rate 8% higher than control. Not statistically significant at the 0.05 or 0.10 levels. Results are preliminary – sample size currently too small to make strategic decisions. More on that a bit later…

31. Preliminary Results (Cont’d) Online activity is a significant factor in course success. One of the potential metrics of course engagement. How did online activity in intervention group compare to control group? Category Average Log-In Total Average Weekly Log-In RateN Green81.87.8190 Yellow (Intervention Group)78.76.8114 Yellow (Control Group)67.46.1102 Red53.84.582 Overall73.36.7488

32. Preliminary Results (Cont’d) Average log-in total for intervention group 17% higher than control. Difference is statistically significant at the 0.05 level. 1 Average weekly log-in rate also higher, but only at the less significant 0.10 level. 2 Interventions applied to Yellow students may have influenced their online behavior. 1: One-tailed t-test; p-value = 0.047. 2: One-tailed t-test; p-value = 0.085.

33. Future Work Ongoing intervention trials in Summer II and Fall semesters. Continue improving interventions based on results of controlled trials. Collect sample size large enough to disaggregate results and determine which strategies worked best.

34. Future Work (Cont’d) College also researching new models capable of generating prediction at time of registration. Front-line staff can intervene with Yellow and Red students as early as possible. Will create two pronged approach when combined with faculty-designed interventions. Phase II – FY10-11 Phase I – FY09-10

35. Tell us where you are… Has anyone used predictive modeling? Plans to include it? What have your results been?

36. Questions? Please feel free to contact us: Shannon Corona, Physical Science Faculty Chair Shannon.corona@riosalado.edu or 480-517-8285 Shannon.corona@riosalado.edu Adam Lange, Program Analyst Adam.lange@riosalado.edu or 480-517-8401 Adam.lange@riosalado.edu