Early Alert System (EAS) in Online Education: Student Demographic Profile Helen Zaikina-Montgomery, Ph.D. Scott Burrus, Ph.D. Meryl Epstein, Ed.D. Elizabeth Young, Ed.D. Roger Gung, Ph.D.

Overview and Purpose of Study Examine the demographic characteristics of students who receive Early Alerts (EAs) in courses Compare demographics of students who receive EAs to students who do not receive EAs Existing research on course intervention strategies, such as EA identifies a need to better understand demographics of students who struggle in course work Build statistical models to evaluate the impact of demographic characteristics to receive Early Alerts Further evaluate the impact of Early Alerts on students’ probability of passing course [Ortagus, 2017]

EAS in the Present Study Context Implemented in 2007 with the goal of increasing course completion by alerting the student’s Academic Counselor to contact the student & discuss concerns. Updated January 2016 to automatically alert the student’s Academic Counselor (AC) if the student did not submit an assignment or participate in the online discussions in the course. Current EAS Process Faculty files EA through classroom or EA issued by LMS Academic Counselor is notified of issue Academic Counselor reaches out to student

Supporting Research Overview Student retention and graduation rates are a topic of institutional concern and academic examination Due to a difference in modality of delivery, online courses are structured differently than traditional on-ground or blended courses Online courses require students to possess more intrinsic motivation and higher levels of organizational and self-management skills Through a mutually collaborative ecosystem, [Allen, Seaman, Poulin, & Taylor, 2016; Braxton, 2002; Eaton, 2011; McElroy & Lubich, 2013]

Supporting Research Overview National Postsecondary Student Aid Study (NPSAS) shows that being married, being a parent, and a full-time employee were positively associated with online course enrollment. NPSAS data showed that minority students were less likely to engage in online education than their non-minority peers Effective interventions for students in online courses need to be well-matched to the online learning environment and to the demographic characteristics of those who are most likely to struggle in their course work [Allen, et al., 2016; Donnelly, 2010; McElroy & Lubich, 2013; NPSAS, 2012; Ortagus, 2017]

Questions guiding this Research What is the demographic profile of students who received an Early Alert, and how do they differ demographically from those who did not receive EA? What is the demographic profile of students who pass the courses, and can we measure the impact of EA on passing course?

Data Source Collected from the university Office of Business Analytics and Operations Research. Included records from students who were issues an early alert either by the course instructor or the learning management system (LMS) sometime during their course Included courses with start dates between November 24, 2015 and January 26, 2016. A total of 26,573 student records were accessed and used in the study of which 2.4% (n = 640) were students who received an Early Alert

Early alert Profiling results

What is the demographic profile of students who received an Early Alert? Demographic Characteristic EAS Students (N = 640) Non-EAS Students (N = 25,933) Age in years M (SD) 33.9 (8.59) 35.1 (9.15) Number of people in family M (SD) 2.69 (1.41) 2.83 (1.50) GPA M (SD) 2.57 (0.59) 3.10 (0.62) Total transfer credits into the university M (SD) 14.17 (19.38) 15.8 (20.44) Gender N (%)   Male 173 (27.4) 8,045 (31.3) Female 458 (72.6) 17,677 (68.7) Marital status N (%) Single 379 (63.5) 12,935 (54.7) Married 145 (24.3) 7,686 (32.5) Separated 36 (6.0) 1,193 (5.0) Divorced 37 (6.2) 1,819 (7.7)

Demographic Characteristic EAS Students (N = 640) Non-EAS Students (N = 25,933) Have dependents N (%)   Yes 86 (14.5) 3,475 (14.8) No 507 (85.5) 19,944 (85.2) Military status N (%) Military (past or current) 140 (21.9) 5,446 (21.0) Not Military 500 (78.1) 20,487 (79.0) Passed course N (%) 268 (41.9) 21,351 (82.3) 372 (58.1) 4,582 (17.7) Course grade N (%) A 10 (1.6) 5,890 (22.7) A- 4,150 (16.0) B+ 14 (2.2) 2,052 (7.9) B 18 (2.8) 2,011 (7.8) B- 28 (4.4) 2,196 (8.5) C+ 26 (4.1) 1,297 (5.0) C 31 (4.8) 1,331 (5.1) C- 67 (10.5) 1,243 (4.8) D+ 22 (3.4) 614 (2.4) D 42 (6.6) 554 (2.1) D- 30 (4.7) 546 (2.1) F 108 (16.9) 1,240 (4.8) W 234 (36.6) 2,800 (10.8)

Demographic Characteristic EAS Students (N = 640) Non-EAS Students (N = 25,933) Form of payment N (%)   Student Direct Pay 27 (11.2) 2,313 (8.9) Financial Aid 558 (87.2) 22,638 (87.3) Military 10 (1.6) 797 (3.1) Scholarship (non-EAS only) 0 (0.0) 185 (< 1.0) Program level N (%) Associate 172 (27.1) 6,546 (25.5) Undergraduate 383 (60.4) 16,262 (63.3) Graduate 79 (12.5) 2,875 (11.2)

Female students more likely to receive an EA than male students How students who received an Early Alert differ demographically from students who did not receive an Early Alert? Female students more likely to receive an EA than male students Younger students and students who were single were more likely to have an early alert Students who received an EA were less likely to pass the course in which early alert was filed, three times more likely to withdraw from the course, and four times more likely to fail the course

Cramer’s V ɸc (p-value) How students who received an Early Alert differ demographically from students who did not receive an Early Alert? Chi square tests applied to EAS status related to categorical variables Variable Chi-square df p-value Phi ɸ (p-value) Cramer’s V ɸc (p-value) Gender 4.276* 1 .039 -.031 (.039)   Marital status 23.047** 3 .000 .031 (.000) Dependents .052 .820 -.001 (.820) Military status .288 .592 .003 (.592) Passed course 673.994** .159 (.000) Course grade 930.036** 13 .187 (.000) Form of payment 4.362 2 .076 .088 (.076) Program level 2.365 .307 .009 (.307) * = Values significant at the .05 level ** = Value significant at the .0001 level

Early alert Modeling

Modeling Approach    

Early Alert Variables Exploration

Early Alert Variables Exploration

Early Alert Variables Coefficient Factors Estimate Std Error P-Value Intercept -1.521 0.250 0.000 GPA: 2 < · <= 2.67 -0.406 0.121 0.001 GPA: 2.67 < · <= 3.14 -0.950 0.135 GPA: > 3.14 -2.147 0.149 AGI: <= 10647 0.254 0.200 0.204 AGI: 10647 < · <= 40205 0.104 0.192 0.587 AGI: > 40205 -0.061 0.205 0.767 Gender: M -0.119 0.100 0.232 Gender: O 0.623 0.353 0.078 Payment Option: DEF -1.814 0.719 0.012 Payment Option: MIL -0.771 0.374 0.039 Payment Option: OTPO -0.501 0.134 Program Level: Graduate 1.166 0.153 Program Level: Undergraduate 0.044 0.096 0.648 AC Tenure: 5 -0.048 0.279 0.864 AC Tenure: 5 < · <= 10 -0.320 0.101 0.002 AC Tenure: > 10 -0.031 0.128 0.809 CCC Flag: Non-CCC -0.650

Odds Ratio Note: Odds Ratio is the odds of the positive outcome at one level of Xi relative to the odds of the positive outcome at reference level of Xi

Early Alert Model Performance The early alert model (7 variables) has a C statistic of 0.736, which can be considered a good model performance

COURSE-PASS Modeling

Course-Pass Variables Exploration

Course-Pass Variables Exploration

Course-Pass Variables Coefficient Factors Estimate Std Error P-Value Intercept -3.782 0.184 0.000 EAS Flag: 0 1.485 0.094 Age: 33 < · <= 48.4 0.086 0.042 Age: > 48.4 0.133 0.074 Age: Missing 0.414 0.393 0.292 GPA: 2 < · <= 2.67 1.010 0.066 GPA: 2.67 < · <= 3.14 1.688 0.070 GPA: > 3.14 2.643 0.071 AGI <= 10647 0.006 0.120 0.958 AGI: 10647 < · <= 40205 0.135 0.123 0.272 AGI: > 40205 0.285 0.128 0.026 Payment Option: DEF 0.206 0.188 0.274 Payment Option: MIL 0.499 0.161 0.002 Payment Option: OTPO -0.139 0.068 AC Tenure: 5 0.096 0.141 0.497 AC Tenure: 5 < · <= 10 -0.083 0.049 AC Tenure: > 10 -0.140 0.063 0.027 CCC Flag: Non-CCC 2.115 0.081 Num Family: 3 -0.050 0.051 0.332 Num Family: 3 < · <= 6 -0.007 0.050 0.885 Num Family: 6 -0.057 0.166 0.731 Num Family: Missing 0.339 0.079 Course Level: 2 -0.117 0.017 Course Level: 3 0.453 Course Level: 4 0.592 Course Level: 5 0.306 0.001

Odds Ratio Note: Odds Ratio is the odds of the positive outcome at one level of Xi relative to the odds of the positive outcome at reference level of Xi

Course-Pass Model Performance The early alert model (9 variables) has a C statistic of 0.798, which can be considered a good model performance

Conclusions & Future research Students who receive EA tend to be more academically “at risk” than non-EA students, and have 41% lower passing rate. Some of the expected factors (those that add more responsibility, such as being married and having dependents) were not associated with EA status Additional data and further analysis, including hierarchical linear modeling to account for potential nested effects should be conducted Universities can continue to develop interventions for “at risk” students keeping in mind factors from this study Students’ GPA, collection status, form of payment, income have strong impact on students’ receiving EA For a non-EA student, the odds of passing a course is 4.4 times as the odds of an EA student, due to the inherent risk

Questions & Discussion

Campus Collection Center (CCC) Flag Variables List Variable Attribute Description Early Alert Categorical 0: No Early Alert 1: Has Early Alert Passed Course Passing Grade is C- for graduate students, D- for bachelor and associate students GPA <=2; 2-2.67; 2.67-3.14; >=3.14 Campus Collection Center (CCC) Flag If the student’s risk code is CCC, then “CCC”; Otherwise “Non-CCC” Age <=33; 33-48; >=48 AGI Annual Gross Income <= $10,647; $10,647 - $40,205; Missing (Missing may indicate cash students) Program Level Associate, Undergraduate, Graduate Form of Payment Payment Option DBEB: Direct Bill or Employee Bill; MIL: Military Payment; DEF: Deferred or Delayed payment OTPO: Other Payment Options

Variables List (Cont’d) Attribute Description Total Transfer Credit Categorical <=20.67 credits; 20.67 - 55.95 credits; >=55 credits Military Flag Y, N Gender F, M, O High School Diploma Y,N Academic Counselor Tenure Academic Counselor’s tenure with the company. <=4, 4-5, 5-10, >=10 Course Level 1, 2, 3, 4, 5 Number of People in the Family <=2, 3, 3-6, >6 Marital Status Divorced; Married; Separated; Single

Variable Rank Order Based on Information Value (IV)     Information Value Predictive Power  >0.5 Suspicious or too good to be true 0.3 to 0.5 Strong predictor 0.1 to 0.3 Medium predictor 0.02 to 0.1 Weak predictor < 0.02 useless for prediction