1. Victor M. H. Borden Associate Vice President, University Planning, Institutional Research & Accountability (IU) Professor of Psychology (IUPUI) Using Evaluation Research to Meet Expectations for Educating Students from Disadvantaged Backgrounds Scott Evenbeck Dean, University College Professor of Psychology Indiana University Purdue University Indianapolis (IUPUI)

3. Overview ► Introduction and Context ► The Nina Mason Pulliam Legacy Scholars Program ♦ What is it and what does it entail? ► The evaluation model ♦ Combining qualitative and quantitative methods to optimize usefulness ► The impact of evaluation on program and student success

4. Introduction and Context ► Four institutions invited by the Trust to write proposals to support the high risk students as scholars ♦ AZ: Arizona State University and Maricopa County Community College District ♦ IN: Indiana University Purdue University Indianapolis and Ivy Tech Community College- Central Indiana ► Separate call for evaluation program ♦ Qualitative Experts(Peg Bortner at ASU; Ken Duckworth at IUPUI) ♦ Quantitative Expert (Yours Truly)

5. The Scholarship Program ► Providing educational opportunities to individuals who would not normally receive traditional academic scholarships ♦ Adults, age 25 + with dependents in family unit ♦ College-age youths and adults with physical disabilities ♦ Young adults, 18-25 yrs, raised in the child welfare (foster care) system, responsible for own financial support ► Support provided ♦ Money for tuition and fees, books and supplies ♦ Annual living allowance (orig $2500, now $2750) ♦ Coordinated services and support in a “cohort” model

6. The Evaluation Model ► Three components ♦ One-on-one interviews with scholars, staff, advisors ♦ Annual Scholar survey: ratings of 11 service components ♦ Tracking of persistence and academic progress of Scholars against matched comparison group

7. One-on-one interviews ► Initially interviewed every Scholar in first two years ► Moved on to Program Managers, Advisory Councils and some other involved constituents ► Intensive qualitative analysis (NUDIST/NVivo) ► Phased down as program matured and Scholar survey implemented

8. Scholar Survey ► Started in fourth year ► Ratings regarding importance to Scholar success ♦ 8 Nina Program-specific components: Program director; Relationship with other scholars; Scholar orientation; Academic advisement; Nina Scholar luncheon; Tutoring; Scholar meetings; Mentoring ♦ 3 Campus-wide services Services for reentry students; Counseling services; Services for students with physical disabilities ► Open-ended comments for each item and in general

9. Tracking System ► Scholars tracked along with a “Matched” comparison group ♦ Comparison group derived from scholarship eligible students who were not offered the scholarship but who subsequently enrolled ► IRB-approved contracts signed with each institution to share student records data annually ♦ Fall request for enrollment information for new scholars ♦ End of year request for academic performance information, including credits attempted, completed, and grades

10. Tracking System (cont) ► Stabilized over first three years with lessons learned ♦ E.g., you can not create academic year by combining fall and spring

11. Tracking Results

13. Tracking Results – Multivariate

14. Intentional Program Development ► Are we on the right track? ♦ Quantitative data as a dashboard ♦ Qualitative feedback and narratives as a map ► Validation of holistic and intrusive approaches ♦ Are we right? ♦ Where are the boundaries ► Emphasis on cohesiveness, mutual support, building community ♦ Is the director a guru? ► Educational goals revised (two-year to four- year and four-year to grad school) and Development of passport program

15. Evaluation as the “Third Leg” ► Emerging literature on best practices ♦ “High Impact” programs ► Doing things right is as important as doing the right thing ► Evaluation drives organizational learning for doing things right ♦ Including doing evaluation right