1. David Fairris Tarek Azzam
Evaluating Student Success: The Impact of a First-year Experience Program When Students Self-select
David Fairris
Tarek Azzam
This evaluation was funded by the United States Department of Education, Fund for the Improvement of Post Secondary Education (FIPSE) Program. The contents of this presentation were developed under a grant from the U.S. Department of Education. However, those contents do not necessarily represent the policy of the U.S. Department of Education, and you should not assume endorsement by the Federal Government.

2. Introduction
A national dialogue has begun around the issue of improving college-going and college-completion rates.
First year persistence is a key indicator of college success, leading to graduation.
Many colleges and universities have turned to “first year experience” programs – such as first year seminars and learning communities – to enhance first-year retention and thus graduation.
How well are these programs working?

3. The Prevalence of First Year Experience Programs
First year experience programs are prevalent across much of higher education.
The National Resource Center for the First-Year Experience and Students in Transition found that 90% of reporting institutions possessed at least one freshman seminar on their campus (2012).
The John N. Gardner Institute for Excellence in Undergraduate Education found that 96% of four-year institutions possessed freshman seminars of one form or another (2010).
This same survey found that 91% of reporting institutions possessed a learning community of some form at their institution.
These are extraordinarily high participation numbers.

4. Student Participation and Self-Selection
Survey data do not offer precise measures of the extent to which freshman seminars or learning communities are populated by students who self-select into these programs.
According to the National Resource Center survey of 2012, 43% of institutions that possess freshman seminars require them of all incoming freshmen.
The Gardner Institute survey from 2009 yields an average student participation rate for seminars, among institutions that possess them, of 80%.
However, this masks great variation among institutions; for example, larger institutions (over 20,000 students) reported that less than half of their incoming freshman class participated in seminars.
The average student participation rate for learning communities in the Gardner Institute survey is much lower, at 33%.

5. Student Participation and Self-Selection
There is a high prevalence of first year experience programs across the higher education landscape
The numbers on participation rates suggest that the percentage of students enrolled in these programs through a voluntary process of self-selection is not insignificant.
Why does this matter for an evaluation of the effectiveness of these programs, and perhaps more generally for any academic or academic support program into which students self select?

6. The Freshman Seminar/Learning Community Structure Under Study
One Year-Long Theme
Fall
Breadth Requirement
75 students
Winter
Spring
Discussion Section
One Year-Long TA for 3 sections of 25 students
Student Success Seminar
Two Year-Long Peer Educators for 3 sections of 25 students

7. Seminal Analysis Using a Quasi-Experimental Research Design (Fairris, Castro, & Son, 2010)
Enrollment is voluntary and students self select into the program
The analysis compares retention rates for treated self-selected students with non-self-selected students
Comparison of mean first-year retention rates
Comparison of conditional means using multiple regression analysis with controls for a host of student characteristics
Data were from the 2002 and 2003 freshman cohorts
Student characteristics such as high-school GPA, SAT scores, race, ethnicity, gender, and first-generation status were used as controls
Mean and conditional mean retention rates were statistically significantly higher for treated self-selected students

8. A Randomized Control Trial Research Design
A randomized control trial design was chosen as a follow-up exercise because:
It dealt with some of the selection bias concerns of previous evaluations.
Using a lottery system to assign students to the program was a logistical possibility.
It could be viewed as a more equitable way to distribute limited resources.
Previously it was first-come, first-served.

9. Cohort Design

10. Comparative Research Designs
Self Selected
(signed up for lottery)
Not Self Selected
(did not sign up for lottery)
Treatment Students
Control Students
Other Students
Randomized Control
Trial Design
Quasi-Experimental
Design

11. Background Characteristics
High School GPA
SAT Verbal
SAT Math
SAT Writing
On Campus Status
Low Income Status
First Generation Status
Gender
Treatment  
(N=425)
Mean
3.45
485.06
497.34
490.38
0.77
0.64
0.67
0.69
Std. Deviation
0.46
122.93
126.83
121.31
0.42
0.48
0.47
Control
(N=388)
3.47
497.6
506.93
504.12
0.76
0.62
0.63
0.35
113.81
120.03
111.96
0.43
0.49
Not Self Selected
(N=2854)
3.55*
507.82*
552.06*
513.65*
0.72*
0.55*
0.57*
0.51*
120.29
132.74
121.27
0.45
0.50
#statically significant difference between control and treatment
* statistically significant difference between treatment and other. p<.05

12. Background Characteristics
African American
Hispanic
Asian
Caucasian
Treatment
10.2%
49.6%
27.3%
12.8%
Control
10.7%
47.9%
29.2%
11.7%
Not Self Selected
5.3%
37.4%
43.9%
13.1%

13. Comparing Retention Treatment vs. Control (ANOVA)
Retention
Year 1 Retention
(%)
Treatment
(N=425)
Mean 92
Std. Deviation 27
Control
(N=388) 91 29

14. Comparing Retention Treatment vs. Not Self Selected (ANCOVA)
Retention
Year 1 Retention
(%)
Treatment
(N=425)
Mean
92**
Std. Deviation 27
Not Self Selected (N=2854)
87** 33
**p<.01

15. Comparing Retention Control vs. Not Self Selected (ANCOVA)
Retention
Year 1 Retention
(%)
Control
(N=388)
Mean
91**
Std. Deviation 29
Not Self Selected
(N=2854)
87** 33
**p<.01

16. Program Impact Decomposition Equation
Treatment Students
Not Self Selected Students
92% % = 5%
Conventional Quasi-Experimental Result
Control Students
Treatment Students =
92% % = 1%
True Program Impact
Not Self Selected Students +
Control Students
91% % = 4%
Selection Bias

17. Conclusions
These results offer a “cautionary tale” regarding program evaluation in environments of student self selection.
Our results suggest that students who self-select into first year experience programs are a non-random draw from the larger freshman student population. They appear to be more motivated and committed to academic success.
When students self select, virtually all of the estimated program impact from a typical quasi-experimental, multiple regression research design is biased due to the positive selection of more motivated students.