Office of Institutional Research Antelope Valley College Probability Based Advising for Basic Skills Courses By Ted Younglove and Aaron Voelcker Office of Institutional Research and Planning Antelope Valley College

Office of Institutional Research Antelope Valley College Note From Ted: If you would like to try the ‘cheat sheet’ part of this project I can send you an example file for the data, and the data can be run for an hourly fee ($105/hr) by: Scott M Lesch, Ph.D. Principal Consulting Statistician C&C / Statistical Consulting Collaboratory University of California Riverside

Office of Institutional Research Antelope Valley College Modeling Prediction Intervention Evaluation

Office of Institutional Research Antelope Valley College Modeling and Prediction Update Predictive models for success and persistence developed in 2006 using SAS Stepwise Discriminant Analysis 11 parameter models and 4 parameter models Persistence defined as continued enrollment from Fall to Spring Success defined as Success in all courses taken (A,B,C,P,CR)

Office of Institutional Research Antelope Valley College Modeling and Prediction Update 4 parameter model selected for ease of use Success Age at start of term Ethnicity Black (Yes/No) Enrolled in at least one basic skills course (Yes/No) Units completed beyond 30

Office of Institutional Research Antelope Valley College Modeling and Prediction Update Validated in Fall 2007 on independent data

Office of Institutional Research Antelope Valley College Modeling and Prediction Update Validated in 2008 on independent data.

Office of Institutional Research Antelope Valley College Prediction Works! So what do we do about it? More of the same? New efforts?

Office of Institutional Research Antelope Valley College Intervention Many (90% Fall 2009) students enter AVC with reading, writing and/or math skills that are below college level, Students are placed into courses by their performance on an entrance exam, 3 levels of English (ENGL 095, 097, and 099) 2 levels of Reading (READ 097 and 099) 3 levels of Math (MATH 050, 060, 070) Students may be placed into one or more Basic Skills courses.

Office of Institutional Research Antelope Valley College Intervention Lack of success in Basic Skills courses is a significant impediment to persistence and success at AVC, Given that a new student tests into a specific Basic Skills course and can successfully pass this course, what other college level courses can this student concurrently enroll in and pass? Can we help these students by providing guidance to counselors and students based on past students success (or lack of success)?

Office of Institutional Research Antelope Valley College Intervention Our solution to the problem posed previously is a Logistic Regression model (specifically a Logistic ANOCOVA), Logistic regressions are used to predict probabilities of occurrence of binary variables, Frequently used in medical research and marketing, Predicting the effect smoking has on the probability of a heart attack, Predicting the probability a customer will purchase a product.

Office of Institutional Research Antelope Valley College Intervention One possible alternative solution: simply calculate the percent success in concurrent classes for students in the different basic skills courses, Logistic regression chosen to provide a statistical framework.

Office of Institutional Research Antelope Valley College Intervention In our case, we are interested in the covariates, the courses taken with the Basic Skills course. For ENGL095, ENGL097, ENGL099 the model would be: Represents the global mean Represents the specific non ENGL course effect Represents the adjustment effects of placement into ENGL095 and ENGL097 Represents the effect of passing the ENGL course

Office of Institutional Research Antelope Valley College Intervention Past data was used to estimate the parameters of the logistic equation,

Office of Institutional Research Antelope Valley College Intervention The model was estimated using SAS proc logistic, Minimum sample size for a course to be included was 30, Two important assumptions:  Individual student effect assumed to be random and negligible,  Individual instructor effect assumed to be random and negligible, Because of the large number of students and instructors these effects can not be easily estimated.

Office of Institutional Research Antelope Valley College Intervention Once the model has been estimated, the parameter estimates are then be used to calculate the specific probabilities for passing each analyzed secondary course for each Basic Skills English level, All probabilities are estimated for the case where the student has passed the ENGL course.

Office of Institutional Research Antelope Valley College Intervention Counseling ‘Cheat Sheets’ ‘Cheat Sheets’ were produced to help counselors and students in selecting courses to improve success in the other courses, It is hoped that by improving selection of concurrent courses success will improve in ENGL as well, The project has been implemented in the Intersession 2009 and Spring 2009 registration period.

Office of Institutional Research Antelope Valley College Intervention Counseling ‘Cheat Sheets’ 122 concurrent courses had sufficient data for estimates for ENGL 095, 097, and concurrent courses had sufficient data for estimates for MATH 050, 060, and concurrent courses had sufficient data for estimates for READ 097, and 099

Office of Institutional Research Antelope Valley College Intervention Example: ENGL095

Office of Institutional Research Antelope Valley College Intervention Discussion: What guidance do you give to counselors?

Office of Institutional Research Antelope Valley College Modeling Prediction Intervention Evaluation

Office of Institutional Research Antelope Valley College Evaluation Intersession/Spring 2009 Plan Test for changes in registration pattern, Test for increase in percent success Overall Basic Skills Test for differential effect on students predicted not likely to succeed.

Office of Institutional Research Antelope Valley College Evaluation Intersession/Spring Complications New variable created for tracking which students were advised using new method was not used consistently, Consistency in identification of students provided counseling lacking, Counseling = 1, student received counseling during this term, probably advised using ‘cheat sheets’ Counseling = 0, student probably did not receive counseling during this term, probably not advised using ‘cheat sheet’.

Office of Institutional Research Antelope Valley College Evaluation (Registration Behavior) Spring ENGL

Office of Institutional Research Antelope Valley College Evaluation (Registration Behavior) Spring MATH

Office of Institutional Research Antelope Valley College Evaluation (Registration Behavior) Spring 2009 – ENGL 099

Office of Institutional Research Antelope Valley College Evaluation Intersession/Spring Complications Additional evaluation suggestions? Two focus areas: Change in registration behavior, Change in Success percentage.

Office of Institutional Research Antelope Valley College Evaluation Spring 2009 After the end of the term: Are persistence rates higher in the students who were advised using the ‘cheat sheets’? Are success rates higher in the students who were advised using the ‘cheat sheets’?

Office of Institutional Research Antelope Valley College Conclusions Spring 2009 LANOCOVA analysis provides a workable way to estimate pass probabilities for concurrent courses, Adoption by counselors is under way and leading to changes in registration behavior, Effects on success may be difficult to estimate on Spring data.

Office of Institutional Research Antelope Valley College Discussion Spring 2009 Suggestions on improving use of the ‘cheat sheets’?, Suggestions on analysis of success? Other courses?