Presentation on theme: "David Fairris Tarek Azzam"— Presentation transcript:
1David Fairris Tarek Azzam Evaluating Student Success: The Impact of a First-year Experience Program When Students Self-selectDavid FairrisTarek AzzamThis evaluation was funded by the United States Department of Education, Fund for the Improvement of Post Secondary Education (FIPSE) Program. The contents of this presentation were developed under a grant from the U.S. Department of Education. However, those contents do not necessarily represent the policy of the U.S. Department of Education, and you should not assume endorsement by the Federal Government.
2IntroductionA national dialogue has begun around the issue of improving college-going and college-completion rates.First year persistence is a key indicator of college success, leading to graduation.Many colleges and universities have turned to “first year experience” programs – such as first year seminars and learning communities – to enhance first-year retention and thus graduation.How well are these programs working?
3The Prevalence of First Year Experience Programs First year experience programs are prevalent across much of higher education.The National Resource Center for the First-Year Experience and Students in Transition found that 90% of reporting institutions possessed at least one freshman seminar on their campus (2012).The John N. Gardner Institute for Excellence in Undergraduate Education found that 96% of four-year institutions possessed freshman seminars of one form or another (2010).This same survey found that 91% of reporting institutions possessed a learning community of some form at their institution.These are extraordinarily high participation numbers.
4Student Participation and Self-Selection Survey data do not offer precise measures of the extent to which freshman seminars or learning communities are populated by students who self-select into these programs.According to the National Resource Center survey of 2012, 43% of institutions that possess freshman seminars require them of all incoming freshmen.The Gardner Institute survey from 2009 yields an average student participation rate for seminars, among institutions that possess them, of 80%.However, this masks great variation among institutions; for example, larger institutions (over 20,000 students) reported that less than half of their incoming freshman class participated in seminars.The average student participation rate for learning communities in the Gardner Institute survey is much lower, at 33%.
5Student Participation and Self-Selection There is a high prevalence of first year experience programs across the higher education landscapeThe numbers on participation rates suggest that the percentage of students enrolled in these programs through a voluntary process of self-selection is not insignificant.Why does this matter for an evaluation of the effectiveness of these programs, and perhaps more generally for any academic or academic support program into which students self select?
6The Freshman Seminar/Learning Community Structure Under Study One Year-Long ThemeFallBreadth Requirement75 studentsWinterSpringDiscussion SectionOne Year-Long TA for 3 sections of 25 studentsStudent Success SeminarTwo Year-Long Peer Educators for 3 sections of 25 students
7Seminal Analysis Using a Quasi-Experimental Research Design (Fairris, Castro, & Son, 2010) Enrollment is voluntary and students self select into the programThe analysis compares retention rates for treated self-selected students with non-self-selected studentsComparison of mean first-year retention ratesComparison of conditional means using multiple regression analysis with controls for a host of student characteristicsData were from the 2002 and 2003 freshman cohortsStudent characteristics such as high-school GPA, SAT scores, race, ethnicity, gender, and first-generation status were used as controlsMean and conditional mean retention rates were statistically significantly higher for treated self-selected students
8A Randomized Control Trial Research Design A randomized control trial design was chosen as a follow-up exercise because:It dealt with some of the selection bias concerns of previous evaluations.Using a lottery system to assign students to the program was a logistical possibility.It could be viewed as a more equitable way to distribute limited resources.Previously it was first-come, first-served.
10Comparative Research Designs Self Selected(signed up for lottery)Not Self Selected(did not sign up for lottery)Treatment StudentsControl StudentsOther StudentsRandomized ControlTrial DesignQuasi-ExperimentalDesign
11Background Characteristics High School GPASAT VerbalSAT MathSAT WritingOn Campus StatusLow Income StatusFirst Generation StatusGenderTreatment (N=425)Mean3.45485.06497.34490.380.770.640.670.69Std. Deviation0.46122.93126.83121.310.420.480.47Control(N=388)3.47497.6506.93504.120.760.620.630.35113.81120.03111.960.430.49Not Self Selected(N=2854)3.55*507.82*552.06*513.65*0.72*0.55*0.57*0.51*120.29132.74121.270.450.50#statically significant difference between control and treatment* statistically significant difference between treatment and other. p<.05
17ConclusionsThese results offer a “cautionary tale” regarding program evaluation in environments of student self selection.Our results suggest that students who self-select into first year experience programs are a non-random draw from the larger freshman student population. They appear to be more motivated and committed to academic success.When students self select, virtually all of the estimated program impact from a typical quasi-experimental, multiple regression research design is biased due to the positive selection of more motivated students.