Presentation on theme: "Evaluating Student Success: The Impact of a First-year Experience Program When Students Self-select David Fairris Tarek Azzam This evaluation was funded."— Presentation transcript:
Evaluating Student Success: The Impact of a First-year Experience Program When Students Self-select David Fairris Tarek Azzam This evaluation was funded by the United States Department of Education, Fund for the Improvement of Post Secondary Education (FIPSE) Program. The contents of this presentation were developed under a grant from the U.S. Department of Education. However, those contents do not necessarily represent the policy of the U.S. Department of Education, and you should not assume endorsement by the Federal Government.
Introduction A national dialogue has begun around the issue of improving college-going and college-completion rates. First year persistence is a key indicator of college success, leading to graduation. Many colleges and universities have turned to “first year experience” programs – such as first year seminars and learning communities – to enhance first-year retention and thus graduation. How well are these programs working?
The Prevalence of First Year Experience Programs First year experience programs are prevalent across much of higher education. – The National Resource Center for the First-Year Experience and Students in Transition found that 90% of reporting institutions possessed at least one freshman seminar on their campus (2012). – The John N. Gardner Institute for Excellence in Undergraduate Education found that 96% of four-year institutions possessed freshman seminars of one form or another (2010). – This same survey found that 91% of reporting institutions possessed a learning community of some form at their institution. – These are extraordinarily high participation numbers.
Student Participation and Self-Selection Survey data do not offer precise measures of the extent to which freshman seminars or learning communities are populated by students who self-select into these programs. According to the National Resource Center survey of 2012, 43% of institutions that possess freshman seminars require them of all incoming freshmen. The Gardner Institute survey from 2009 yields an average student participation rate for seminars, among institutions that possess them, of 80%. – However, this masks great variation among institutions; for example, larger institutions (over 20,000 students) reported that less than half of their incoming freshman class participated in seminars. The average student participation rate for learning communities in the Gardner Institute survey is much lower, at 33%.
Student Participation and Self-Selection There is a high prevalence of first year experience programs across the higher education landscape The numbers on participation rates suggest that the percentage of students enrolled in these programs through a voluntary process of self-selection is not insignificant. Why does this matter for an evaluation of the effectiveness of these programs, and perhaps more generally for any academic or academic support program into which students self select?
The Freshman Seminar/Learning Community Structure Under Study One Year-Long Theme Fall Breadth Requirement 75 students Winter Breadth Requirement 75 students Spring Breadth Requirement 75 students Discussion Section One Year-Long TA for 3 sections of 25 students Student Success Seminar Two Year-Long Peer Educators for 3 sections of 25 students
Seminal Analysis Using a Quasi- Experimental Research Design (Fairris, Castro, & Son, 2010) Enrollment is voluntary and students self select into the program The analysis compares retention rates for treated self-selected students with non-self-selected students Comparison of mean first-year retention rates Comparison of conditional means using multiple regression analysis with controls for a host of student characteristics Data were from the 2002 and 2003 freshman cohorts Student characteristics such as high-school GPA, SAT scores, race, ethnicity, gender, and first-generation status were used as controls Mean and conditional mean retention rates were statistically significantly higher for treated self-selected students
A Randomized Control Trial Research Design A randomized control trial design was chosen as a follow-up exercise because: – It dealt with some of the selection bias concerns of previous evaluations. – Using a lottery system to assign students to the program was a logistical possibility. – It could be viewed as a more equitable way to distribute limited resources. Previously it was first-come, first-served.
Comparative Research Designs Self Selected (signed up for lottery) Treatment Students Control Students Not Self Selected (did not sign up for lottery) Other Students Randomized Control Trial Design Quasi-Experimental Design
Background Characteristics High School GPA SAT VerbalSAT MathSAT Writing On Campus Status Low Income Status First Generation Status Gender Treatment (N=425) Mean3.45485.06497.34490.380.770.640.670.69 Std. Deviation0.46122.93126.83121.310.420.480.470.46 Control (N=388) Mean3.47497.6506.93504.120.760.620.630.67 Std. Deviation0.35113.81120.03111.960.430.490.480.47 Not Self Selected (N=2854) Mean3.55*507.82*552.06*513.65*0.72*0.55*0.57*0.51* Std. Deviation0.46120.29132.74121.270.450.500.490.50 #statically significant difference between control and treatment * statistically significant difference between treatment and other. p<.05
Background Characteristics African AmericanHispanicAsianCaucasian Treatment 10.2%49.6%27.3%12.8% Control 10.7%47.9%29.2%11.7% Not Self Selected 5.3%37.4%43.9%13.1%
Comparing Retention Treatment vs. Control (ANOVA) Retention Year 1 Retention (%) Treatment (N=425) Mean92 Std. Deviation27 Control (N=388) Mean91 Std. Deviation29
Comparing Retention Treatment vs. Not Self Selected (ANCOVA) Retention Year 1 Retention (%) Treatment (N=425) Mean92** Std. Deviation27 Not Self Selected (N=2854) Mean87** Std. Deviation33 **p<.01
Comparing Retention Control vs. Not Self Selected (ANCOVA) Retention Year 1 Retention (%) Control (N=388) Mean91** Std. Deviation29 Not Self Selected (N=2854) Mean87** Std. Deviation33 **p<.01
Program Impact Decomposition Equation Treatment Students Not Self Selected Students 92% - 87% = 5% Conventional Quasi- Experimental Result Control Students Treatment Students = 92% - 91% = 1% True Program Impact Not Self Selected Students + Control Students 91% - 87% = 4% Selection Bias
Conclusions These results offer a “cautionary tale” regarding program evaluation in environments of student self selection. Our results suggest that students who self-select into first year experience programs are a non-random draw from the larger freshman student population. They appear to be more motivated and committed to academic success. When students self select, virtually all of the estimated program impact from a typical quasi-experimental, multiple regression research design is biased due to the positive selection of more motivated students.