1. Making a Difference in Science Education for Underrepresented Students: The Impact of Undergraduate Research Programs Kevin Eagan Gina Garcia Felisha Herrera Juan Garibay Sylvia Hurtado, Principal Investigator Mitchell Chang, Principal Investigator Higher Education Research Institute, UCLA 2010 AIR Annual Forum Chicago, Illinois

2. Introduction Graduate enrollment in science and engineering has been increasing However, URM enrollment continues to lag behind – Proportion of URMs in graduate programs during 2006- 2007 academic year American Indian 0.4% Black 4.9% Latina/o 3.6% STEM completion rates remain low (esp. for URMs) – Huang, Taddese, & Walter (2000) – Higher Education Research Institute (2010)

3. Purpose To examine the effects of undergraduate research programs on students’ intentions to enroll in graduate school through the use of advance statistical techniques on multi- institutional data. – Propensity score matching

4. Background Graduate School Enrollment – Prior academic achievement – Race/socioeconomic status – Parent education – Institutional selectivity – Level of involvement – Student faculty interaction

5. Background Benefits of undergraduate research programs Retrospective AnalysesSingle Time PointLongitudinal Hurtado, Cabrera, Lin, Arellano, & Espinosa (2009) Barlow & Villarejo (2004) Bauer & Bennett (2003) Hathaway, Nagda, & Gregerman (2002) MacLachlan (2006) Lopatto (2004) Maton & Hrabowski (2004) Russell, Hancock, & McCullough (2007) Hunter, Laursen, & Seymour (2006) Seymour, Hunter, Laursen, & Deantoni (2004)

6. Conceptual Framework Social and Cultural Capital Capital inherited through social position and family background Social capital acquired in college complements the capital that students bring with them Science Identity Fostering knowledge growth Opportunities to display scientific knowledge & practices Acknowledgement of being a science person

7. Research Questions 1.What pre-college experiences and characteristics of entering college students predict their likelihood of participating in a structured undergraduate research program during college? 2.After accounting for students’ chances of participating in an undergraduate research program, what effect does participation in such a program have on students’ intention to enroll in graduate/professional school, particularly in a STEM field?

8. Methods: Sample CIRP Longitudinal Sample (n=4,212) – 2004 Freshman Survey (TFS) – 2008 College Senior Survey (CSS) Targeted institutions: – Strong reputations in STEM graduation rates – Undergraduate research programs funded by NSF and NIH – Minority-serving institutions

9. Methods: Variables DV: 3-part variable representing post-college intentions: – Enroll in graduate/professional STEM program – Enroll in graduate/professional non-STEM program – No intentions to pursue graduate/professional degree IVs – Undergraduate research participation – Science identity – Career focus in 2008 – College GPA – College experiences – Pre-college preparation – Demographics – Institutional characteristics

10. Methods: Analyses Missing data Propensity score matching – Discussion of the counterfactual – Estimation of the propensity score related to participation in an undergraduate research program – Multinomial hierarchical generalized linear modeling

11. Methods: Analyses Issues of selection bias/endogeneity Counterfactual framework – “a potential outcome, or the state of affairs that would have happened in the absence of the cause” (Guo & Fraser, 2010, p. 24) – Comparing a “treated” individual with a “non- treated” individual Propensity score estimation

12. Methods: Analyses Reweighting of the data with derivations of the propensity score – Average treatment effect – Average treatment of the untreated (ATU) effect – Average treatment of the treated (ATT) effect Multinomial hierarchical generalized linear modeling

13. Limitations Secondary data analysis Limited DV: intentions and combination of graduate and professional school Unobservable variables affecting undergraduate research participation Weighting adjustment using propensity score rather than matching by propensity score

14. Findings: Predictors of Participating in Undergraduate Research Programs Major: Physical sciences (10.77%), Life sciences (7.34%), Health sciences (4.90%) Race: Black (5.71%) Participated in pre-college research program (4.03%) Degree aspiration in 2004: Ph.D. (3.54%) Composite SAT score (100-point change): 2.27%

15. Findings: Effects of Undergraduate Research Program Participation on Graduate/Professional School Enrollment Intentions Intend to Enroll in a STEM Graduate/Professional Program Intend to Enroll in a non-STEM Graduate/Professional Program Delta-P Log oddsS.E.Sig.Delta-P Log OddsS.E.Sig. Average treatment effect (ATE)7.84%0.390.16*4.96%0.230.17 Average treatment for the untreated (ATU)7.95%0.400.16*5.98%0.280.18 Average treatment for the treated (ATT)6.91%0.340.15*-0.45%-0.020.15 Unadjusted multinomial HGLM8.38%0.420.14**1.77%0.080.15 Simple mean comparison13.50%***1.80%

16. Discussion Confirmation of results from prior studies – “Effect” is much more modest than prior studies might suggest UG research programs attract students who already identify as scientists Average treatment of the untreated (ATU) effect – Expand the reach of these programs – Ensure programs not only harvest talent but develop it, too

17. Conclusion and Directions for Future Research Follow these students into graduate school and examine matriculation patterns Investigate via qualitative methods the quality of students’ research experiences UG research programs as wise investments

18. Contact Information Acknowledgments: This study was made possible by the support of the National Institute of General Medical Sciences, NIH Grant Numbers 1 R01 GMO71968-01 and R01 GMO71968-05 as well as the National Science Foundation, NSF Grant Number 0757076. This independent research and the views expressed here do not indicate endorsement by the sponsors. Papers and reports are available for download from project website: http://heri.ucla.edu/nih Project e-mail: herinih@ucla.edu Faculty and Co-PIs: Sylvia Hurtado Mitchell Chang Monica Lin Gina Garcia Felisha Herrera Postdoctoral Scholars: Kevin Eagan Josephine Gasiewski Administrative Staff: Aaron Pearl Graduate Research Assistants: Christopher Newman Minh Tran Jessica Sharkness Cindy Mosqueda Juan Garibay