1. Assessing the Impact of Undergraduate Research and Mentoring (URM) on Student Learning in Biological Sciences
Howard R. Mzumara, Ph.D.
Director of Testing Center
Indiana University – Purdue University Indianapolis (IUPUI)
Testing Center
Phone: (317)
Web site:

2. Presentation Outline
Brief Overview of the URM Program in Biological Sciences
Basic Information about the URM Program at IUPUI
IUPUI URM Project Objectives and Focus of the Evaluation Study
Key Evaluation Questions
Methodology for the Evaluation Study
Mixed-Method Evaluation Approach
Qualitative & Quantitative Data Sources
Direct & Indirect Measures
Issues & Framework for Data Analysis
Highlights of Preliminary Study Findings …
Evaluation 2011 Conference

3. Synopsis of URM Program
Goal of the URM Program:
Increase the number and diversity of individuals pursuing graduate studies in all areas of biological research supported by the National Science Foundation (NSF).
Academic Institutions receive support to establish innovative programs to engage undergraduates in year-round research and mentoring activities.
URM projects must have strong research and mentoring activities…
Particular emphasis is placed on broadening participation of groups historically underrepresented in science and engineering.
Evaluation 2011 Conference

4. IUPUI URM Program Key Strategy: Target pool: Goal:
Engage student participants in a multi-year immersion in interdisciplinary research around the theme of biological signaling
Target pool:
Underrepresented minorities in Biological Sciences (initially recruit 8 URM Fellows per grant year)
Goal:
Increase the number of underrepresented minorities achieving graduate degrees in Biological Sciences
Evaluation 2011 Conference

5. IUPUI URM Program (continued)
Key Program Elements include:
Recruitment of URM Fellows & Mentors
Faculty Orientation
Summer Camp & 9-Week Summer Research Program that include:
Multi-tiered Mentoring Sessions & Brown-bag lunches
URM-focused Research Sessions / Workshop Series
Career Development Sessions
Graduate School Boot Camp
Poster Session
Academic year activities that key into science:
URM Fellows spend an average of 10 hrs/wk in labs (to earn at least one research credit per semester); and
Attend bi-weekly research cohort meetings
Evaluation 2011 Conference

6. IUPUI URM Project Objectives and Focus of Evaluation Study
Key Objective: Assess the effectiveness and impact of the IUPUI URM Program
Formative Evaluation
Improve design and delivery of the interventions
Track effectiveness of project development and implementation
Summative Evaluation
Assess impact on learners (URM Fellows)
Measure knowledge gained by participating in an intensive summer camp/research program
Assure project objectives are achieved
Evaluation 2011 Conference

7. Key Evaluation Questions
Assessing Effectiveness of the IUPUI URM Program:
To what extent did the URM Summer Camp/Research Program enhance student learning or career development?
To what extent did the URM Summer Camp/Research Program yield substantive changes in knowledge, attitudes, skills or performance of participants in Biological Sciences?
Did the URM Fellows develop a working knowledge of Biological Signaling to the extent that they could design and conduct appropriate research projects?
Assessing Impact on internal & external stakeholders:
To what degree did the IUPUI URM Program achieve its objectives?
In what ways did the URM Summer Camp/Research Program produce benefits among the study participants?
Evaluation 2011 Conference

8. (Mixed-Method Evaluation Approach) Types of Evaluation Measures
Direct Measures:
Assessments of specific PULs
Lab or Course-embedded assessments
Assessment of Learning Gains
Scores on standardized tests
Student products
(research papers, Portfolio artifacts)
Performance Assessment (Poster/Research Presentations
Project documents (e.g., RFP document, progress reports)
Indirect Measures:
Pre/Post Knowledge or Attitude Surveys (Questionnaires)
Research Self-Efficacy Survey
Interviews
Focus Groups with URM Fellows
Participant Evaluations of Workshops
Mentor Survey
Course/Institutional Data (e.g., enrollment, participation, completion/retention, course grades, and other related data)

9. Evaluation Instruments/Data Sources
Pre/Post Program Surveys
Student Attitude/Confidence Scales
URM Summer Camp/Research Program Evaluation Surveys
Research Self-Efficacy Scale
Workshop Evaluations
Interviews / Focus Groups with URM Fellows
Mentor Evaluation
Event Tracking (collect enrollment, participation, and completion data)
Extant Data (course grades, GPAs, %DFW, retention, persistence, & graduation rates) extracted from institutional databases
Evaluation 2011 Conference

10. Issues to Address in Data Analysis
Small sample size (n) for treatment group
Self-selection or research participant bias
Lack of an appropriate comparison group
Equivalence of Treatment vs. Comparison Groups
Appropriateness of theoretical framework
Statistical assumptions for using parametric or non- parametric statistical methods to assess project impact or effectiveness of interventions
Inherent limitations of self-reported measures
Evaluation 2011 Conference

11. Project Impact Analysis
Defining clear objectives for URM Summer Program is essential
Assessing clearly articulated outcomes is useful
Confidence-building among URM Fellows
Skill development, including communication, critical thinking, quantitative reasoning, etc.
Career choices affected (e.g., interest in preparing for graduate study or internships in STEM fields)
Evaluation 2011 Conference

12. Project Impact Analysis (continued)
Investigate performance and retention of URM Fellows (vs. an appropriate comparison group)
Assess impact on student retention and graduation
Increase in % of students completing degrees in STEM fields
(Compare URM Fellows with an appropriate comparison group that did not participate in summer research activities)
Track student self-reported attitudes
Where possible, examine the effect on student retention and performance of the project overall and for each of the project strategies
Evaluation 2011 Conference

13. Participant Demographic Summary for Initial Cohort (N=4)
Gender
Females: 50%
Males: 50%
Age
18-23 years: 75%
24-29 years: 25%
Race/Ethnicity
Black/African American: 50%
Other Hispanic or Latino: 25%
Multiracial: 25%
Educational Classification:
Junior: 75%
Senior: 25%
STEM Field Enrolled:
Biology: 50%
Chemistry: 25%
Physics: 25%
URM Fellows (Summer 2011)
URM Fellows (Summer 2011)
Evaluation 2011 Conference

14. Preliminary Findings …
Key Benefits of URM Program (most enjoyable aspects as reported by URM Fellows at IUPUI):
One-on-one interaction with different scientists (“mentorship”)
Interdisciplinary nature of the URM Program enhanced the understanding of multiple perspectives of Biological Signaling as a research area
Sessions on how to prepare for graduate school
Scholarships for URM Fellows
Evaluation 2011 Conference

15. Preliminary Findings … (continued)
Obstacles encountered/Least enjoyable aspects:
Time spent in weekly lab sessions is not helpful
Communication of program requirements (e.g., expectations regarding the amount of student work hours per week) is not articulated clearly for URM Fellows, Faculty or Mentors
Lack of a glossary of common terms (for use by URM Faculty, Mentors, and Fellows)
Evaluation 2011 Conference

16. Greatest Gains (reported elsewhere)*
Readiness for more demanding research
Learning a topic in depth
Opportunities for presentations
Laboratory techniques
Enhancement of academic credentials
Developing a relationship with a mentor
Skill development
Learning to persevere
Tolerance for obstacles
Understanding current research
*Source: Adopted from a presentation by Brakke & Crowe (retrieved 11/2/2011 from: )
Evaluation 2011 Conference