Presentation is loading. Please wait.

Presentation is loading. Please wait.

Evaluating the Efficacy of the Research Initiative for Scientific Enhancement (RISE) by using Propensity Scores to Identify a Matched Comparison Group.

Similar presentations


Presentation on theme: "Evaluating the Efficacy of the Research Initiative for Scientific Enhancement (RISE) by using Propensity Scores to Identify a Matched Comparison Group."— Presentation transcript:

1 Evaluating the Efficacy of the Research Initiative for Scientific Enhancement (RISE) by using Propensity Scores to Identify a Matched Comparison Group Mica Estrada-Hollenbeck 1 Anna Woodcock 2 David Morella 3 Wesley Schultz 1 California State University, San Marcos 2 Purdue University 3 Kent State University Presented at the November 14, 2009 AEA Conference, Orlando, Florida.

2 Research Questions 1.Does participating in the RISE program increase the likelihood that a minority student will pursue a career in the biomedical sciences? 2.Are there some types of students who benefit more from the RISE program than others? 3.Are there elements of the RISE program that are linked with the success of the students? 4.Are the underlying assumptions regarding the efficacy of the elements of the RISE program valid?

3 The Challenge POPULATION TREATMENT P P P P P P P P P P P P PP P P T T T TP PP

4 Potential Statistical Solutions When there is no randomized control group Blocked design Analysis of covariance Propensity scores

5  Corrects for selection bias when randomization is not possible Propensity Scores Purpose Population Treatment

6 Propensity Score Definition Probability that a subject would receive treatment given a set of observed variables. People with similar propensity scores are similarly likely to receive treatment and can be compared to estimate the effect of treatment. Rosenbaum and Rubin (1983:420) suggest: If treatment assignment is strongly ignorable given [covariates used to estimate propensity scores], then the difference between treatment and control means at each value of a [propensity] score is an unbiased estimate of the treatment effect…”

7 > Propensity score matching assumes that all variables related to both outcomes and treatment assignment are included in the vector of observed variables (Rosenbaum & Rubin, 1983) > The size of the population/sample from which the propensity scores are derived is important - Minimally, there need to be people in the population who are similar to the treatment group. Propensity Score Caveats

8 Previous Uses Used primarily with studies retrospectively Education (Lee and Staff 2007; Wu et al. 2007) Economics (Dehejia and Wahba 2002, Benjamin 2003, Michalopoulos et al. 2004) Medical literature (Rubin 1999, Weitzen et al. 2004, Erosheva et al. 2007)

9 Overview: The Science Study  Longitudinal study of minority science students  From 45 campuses nationwide, 25 of these have RISE programs  1,380 participants  Data collected twice yearly from students  Propensity score matched control design  Completing fourth year (8 waves of data)

10 Longitudinal Panel (at recruitment)

11

12 Longitudinal Panel  72% Female Ethnicity/Race:  49% African American  39% Hispanic/Latino(a)  1% Native American Major:  63% Biological Sciences  21% Natural Sciences  12% Behavioral & Social Sciences  4% Mathematics & Engineering

13 Survey Data Collection Data collected through secure web interface www.TheScienceStudy.com

14 Propensity Score Generation 1. Identify what variables are predictive of treatment group -- Must be measurable items from both population and treatment sample -- In a prospective longitudinal study, we use the first data collected to identify the groups. 2. Using variables as predictors, a propensity score is generated for each person in the sample (logistic regression, SPSS). 3. Recruited match pool using faculty referrals: - RISE funded: N=750 possible, recruited 402 (added new students in W2 and W4) - Match pool: N=2166, wanted 402 (plus overmatch)

15 Propensity Score Generation Additionally, age squared and interactions of Gender and GPA, Gender and Transfer Student Status, and Gender by educational status were added. 1. GPA6. Parent Education 2. Educational Status (LD, UD, Grad) 7. Academic major 3. Intention to pursue a scientific career 8. First Generation College Student 4. Ethnicity9. Gender 5. Transfers student status 10. Age

16  Everyone received a score Propensity Scores Population Treatment =.75 =.85 =.93 =.97 =.93 =.94 =.67 =.87

17 Propensity Scores Matching Process RISEMATCH Subject 1=.86Subject 10=.66 Subject 2=.77Subject 11=.72 Subject 3=.81Subject 12=.80 Subject 4=.75Subject 13=.43 Subject 5=.68Subject 14=.79 Subject 6=.45Subject 15=.44 Subject 7=.57Subject 16=.73 Subject 8=.71Subject 17=.59 Subject 9=.82Subject 18=.63 Subject 19=.51 Subject 20=.85

18 Note: Change over time analyses conducted as a hierarchical linear model, with both linear and quadratic terms. Analyses are based on students who were undergraduates (jr. or sr.) at W0. Propensity score (W0) used as time invariant covariate. RISE = students continuously funded (N=101), and MATCH = students never funded (N=200) by any program and enrolled on a RISE campus. Dropped = students who were at one time enrolled in RISE but did not complete it. Intention to pursue career as biomedical scientist.

19 Baccalaureate Graduation (Fall, 08) No difference between Dropped & Match, χ 2 (1)=0.58, p=.81 Significant difference between RISE and Combined Dropped/Match, χ 2 (1)=10.37, p=.001

20 Graduate School: Applications & Offers Applications: Science-related Programs Offers: Science-related Programs

21 Post Baccalaureate Outcomes (Fall, 08)  21 panel members with a Ph.D. (W6, 12/08 )  1 panel member MD  2010/2011 first large wave of Ph.D. eligible panel members Enrollment Attainment

22 Thank You

23 Bibliography 1.Benjamin, Daniel. 2003. “Does 401(k) Eligibility Increase Savings? Evidence from Propensity Score Classification.” Journal of Public Economics 87: 1259-1290. 2.Dehejia, Rajeev and Sadek Wahba. 2002. “Propensity Score Matching Methods for Non- Experimental Causal Studies.” Review of Economics and Statistics 84 (1): 151-161. 3.Lee, Jennifer C. and Jeremy Staff. 2007. “When Work Matters: The Varying Impact of Work Intensity on High School Dropout.” Sociology of Education 80 (2): 158-178. 4.Michalopoulos, Charles, Howard S. Bloom ­ Carolyn J. Hill. 2004. “Can Propensity-Score Methods Match the Findings from a Random Assignment Evaluation of Mandatory Welfare- to-Work Programs?” Review of Economics and Statistics 86 (1): 156-179. 5.Rosenbaum, Paul and Donald Rubin. 1983. “The Central Role of the Propensity Score in Observation Studies for Causal Effects.” Biometrika 70 (1): 41-56. 6.Rubin, Donald. 2001. “Using Propensity Scores to Help Design Observational Studies: Application to the Tobacco Litigation.” Health Services and Outcomes Research Methodology 2: 169-188. 7.Weitzen, Sherry, Kate Lapane, Alicia Toledano, Anne Hume and Vincent Mor. 2004. “Principles for Modeling Propensity Scores in Medical Research: A Systemic Literature Review.” Pharmacoepidemiolgy and Drug Safety 13: 841-853. 8.Wu, Wei, Stephen West, and Jan Hughes. 2007. “Short-Term Effects of Grade Retention of the Growth Rate of Woodcock Johnson III Broad Math and Reading Scores.” Journal of School Psychology.

24


Download ppt "Evaluating the Efficacy of the Research Initiative for Scientific Enhancement (RISE) by using Propensity Scores to Identify a Matched Comparison Group."

Similar presentations


Ads by Google