1. Findings from a National Longitudinal Study of the RISE Program P. Wesley Schultz California State University, San Marcos Presented at the 2009 NIGMS Council Meeting, January 23, 2009, Washington DC.

2. Findings from a National Longitudinal Study of the RISE Program Research Team: Mica Estrada-HollenbeckRichard Serpe Anna WoodcockDavid Morolla Randie ChancePaul Hernandez Maria AguilarPerla Rivas Victor RochaBrian McDonald

3. Minority Training Programs NIH values diversity Underrepresentation is an important issue ◦STEM and biomedical sciences ◦Women in science ◦Racial and ethnic representation (Black, Native American, Hispanic/Latino) Many NIH-funded programs promote research careers among members of underrepresented groups ◦Primarily through NIGMS-MORE

4. Do These Programs Work? 1. National-level Data ◦80% increase in minority PhDs since 1977  62% increase in minority population during same period 2. Site-specific Data ◦Each NIH program submits yearly progress reports, and longer-term evaluation results ◦Typically track their student success, but what about a control group?

5. Do These Programs Work? 3. Large-Scale Evaluation Assessment of NIH Minority Research and Training Programs (Phase III). Science, 2006. Surveyed students from 47 programs (1970 - 99) Substantial limitations in finding former students 5,371 identified participants, of which data was obtained from 739 (only 83 trainees funded as undergraduates--19 U*STAR, 31 Bridges, 33 COR)

6. Problems with Existing Data 1. Programs can “cherry pick” students who are likely to succeed - These students are likely to succeed without the program (growing new talent versus harvesting abilities) 2. No control group 3. Long-term evaluations outside of funding scope 4. No way to examine the “mechanisms” of success 5. Retrospective accounts can be biased

7. Overview: The Science Study  Prospective, propensity matched control  Longitudinal study of R.I.S.E. and MARC students  Participants from 25 programs nationwide  Twice yearly surveys from students  Matched control group ◦For each R.I.S.E. or MARC student, we find a similar student who does not go through the program ◦Matching variables: ethnicity, gender, major, GPA, intention to become a scientist, enrollment level (LD, UD, Grad) ◦Secondary matching: age, parental education, community college transfer, English as first language

8. Overview: The Science Study

9. Longitudinal Panel  Entering Panel N=1,228 (Fall 2005) ◦457 R.I.S.E./MARC ◦457 Matched Control ◦157 Other funded (and 157 matched)

10. Longitudinal Panel

11. Survey Data Collection Data collected through web interface www.TheScienceStudy.com

12.  Psychological constructs self-efficacy, scientific identity, task-goal orientation, value orientation, ethnic identity, integrative identity, satisfaction, self- esteem, etc.  Accomplishments/milestones publications, presentations, research participation, conference attendance, degree completion, etc.  Perceptions of the academic environment mentorship, belonging, financial support and MTP experience W0 Fall ‘05 100% W1 Spring ‘06 84.7% W3 Spring ’07 78.0% W2 Fall ‘06 78.9% W4 Fall ‘07 75.4% W5 Spring ’08 76.0% Survey coverage: 93.1% Survey Measures Response rates

13. Questions we can (ultimately) answer 1. Does participating in the RISE/MARC program cause an increase in the likelihood that a minority student will pursue a career in the biomedical sciences? (summative) 2. What are the types of activities that students involved in RISE/MARC programs are exposed to? 3. Are there some types of students who benefit more from the RISE/MARC program than others? 4. Are there elements of the RISE/MARC program that are linked with student success? (Research experience, faculty mentoring, financial support, motivated peers, resume of accomplishments)

14. Preliminary Findings 1.Basic effects (summative outcome) Intention to become a scientist Graduation rates Applications and acceptance to doctoral programs 2. Program elements (mediators) Financial support, research experience, faculty mentorship 3. Reasons for success (process) Self-efficacy, identity, values

15. Q: To what extent do you intend to pursue a career as a biomedical scientist? (0 – 10 ) Note: Juniors and Seniors at screening (W0). Change over time analyses conducted as a hierarchical linear model, with both linear and quadratic terms. RISE = students continuously funded, and MATCH = students never funded by any program. Dropped = students who were at one time enrolled in RISE but did not complete it (N=55).

16. Q: To what extent do you intend to pursue a career as a biomedical scientist? (0 – 10 ) C=1.58

17. Note: Juniors and Seniors at screening (W0). Change over time analyses conducted as a hierarchical linear model, with both linear and quadratic terms. RISE = students continuously funded, and MATCH = students never funded by any program. Dropped = students who were at one time enrolled in RISE but did not complete it (N=55).

18. Baccalaureate Graduation W0 Juniors and Seniors. MTP = Minority Training Program

19. Graduate School: Applications * W0 Juniors and Seniors who have graduated with B.A./B.S. MTP = Minority Training Program.

20. Graduate & Medical School: Enrollment * W0 Juniors and Seniors who have graduated with B.A./B.S. MTP = Minority Training Program. 10% 13% 1%

21. 2. Program Elements (Mediators) Support to attend professional conferences Faculty Mentorship Research Experience Paid Tuition & Stipends Tutoring Invited Speakers & Workshops

22. 2. Program Elements (Mediators) Note: Table reported at the 2007 NIH progress report meeting. Results based on survey responses from 25 RISE directors, and 457 RISE students.

23. 2. Program Elements (Mediators) Mediation analyses of programs Focus on three program elements Research Experience Faculty Mentorship Financial Support (hours spent in employment off campus)

24. Mediation: Research Experience c=1.58** c’=.68* a=.36 ** b=2.38 ** Research Experience RISE Status Intention to Pursue a Career in the Biomedical Sciences Note: a, b and c path’s are unstandardized coefficients. * p <.05, ** p <.001 Bootstrapped Indirect Effect: Mean = 0.68, CI 99% = 1.31 to 0.45 Sobel: Z = 4.67, p <.001

25. Note: Change over time analyses conducted as a hierarchical linear model, with both linear and quadratic terms. Analyses are based on students who were undergraduates (jr. or sr.) at W0. Propensity score (W0) used as time invariant covariate. RISE = students continuously funded, and MATCH = students never funded by any program and enrolled on a RISE campus. Research is any research experience ever during undergraduate education. Intention to pursue career as biomedical scientist.

26. Mediation: Faculty Mentor c=1.58** c’= 1.33** a=.32** b=.78* Faculty Mentor RISE Status Intention to Pursue a Career in the Biomedical Sciences Note: a, b and c path’s are unstandardized coefficients. * p <.05, ** p <.001 Sobel: Z = 2.08, p <.05

27. Mediation: Financial Support c=1.60** c’= 1.49** a=-.26** b=-.42 ns Financial Support RISE Status Intention to Pursue a Career in the Biomedical Sciences Note: a, b and c path’s are unstandardized coefficients. * p <.05, ** p <.01 Sobel: Z =.85, ns Financial support is hours spent working off campus

28. PROCESS MODEL FOR RISE OUTCOMES ProgramConstructsOutcomes Faculty Mentorship (Student-Faculty Contact) Conference (attending) Cohort (peer support) Research Experience Financial Support Motivation to Become a Scientist -Learning motivation - Performance motivation (approach) -Performance motivation (avoid) Identity as a Scientist -Salience -Commitment -Integrative (with ethnicity) Efficacy to Achieve -Scientific Self Efficacy -Self Esteem Intention to pursue graduate education in the biomedical sciences Behavior -GPA - GRE/Other -Schools applied to/accepted -Degree(s) attained -Professional career choice Accomplishments -Presentations -Publications Value of Scientific Community Objectives and Lifestyle

29. 3. Process of Influence

30. 3. Reasons for Success (process) c=2.55 ** c’=2.28** a=.28* b=.98 ** Scientific Self-Efficacy (rule) Research Experience Intention to Pursue a Career in the Biomedical Sciences Note: a, b and c path’s are unstandardized coefficients. * p <.05, ** p <.01 Bootstrapped indirect effect: mean =.27, CI 99%.06 to.56 Sobel: Z = 2.57, p<.01

31. 3. Reasons for Success (process) c=2.55 ** c’=1.80** a=.56** b=1.34 ** Scientific Identity (role) Research Experience Intention to Pursue a Career in the Biomedical Sciences Note: a, b and c path’s are unstandardized coefficients. * p <.05, ** p <.01 Bootstrapped indirect effect: mean =.75, CI 99%.33 to 1.23 Sobel: Z = 4.37, p<.001

32. 3. Reasons for Success (process) c=2.51** c’=2.24** a=.24** b=1.08 ** Scientific Values (value) Research Experience Intention to Pursue a Career in the Biomedical Sciences Note: a, b and c path’s are unstandardized coefficients. * p <.05, ** p <.01 Bootstrapped indirect effect: mean =.27, CI 99%.01 to.60 Sobel: Z = 2.43, p<.01

33. Key Points 1. NIH programs for undergraduates (RISE and MARC) have a strong effect on student intentions to pursue a career in the sciences. 2. RISE and MARC undergraduates have higher graduation rates, applications to doctoral programs, and enrollment in graduate programs (preliminary). 3. RISE works primarily because it provides students with research experience. 4. Research Experience Scientific Identity Persistence