1. Pittsburgh Promise Research & Evaluation
Dr. Jennifer Iriti & Dr. William Bickel
Evaluation for Learning Group
Prepared for Lumina Foundation Convening October 10-11, 2013
Studying since inception in Pittsburgh (2008 was first cohort)
Foundation-funded studies
Promise-funded studies
Sketches studies over 5 years
In order to fit in to the 5-10 minute time frame, just sharing at the conceptual level. Specific cohorts and technical information about analyses not included.
Our work on the Promise has been oriented toward providing data for program designers, implementers and stakeholders rather than for drawing conclusions about scholarship impact/potential overall. Recently, work is more summative focused.

2. EFL work on Pittsburgh Promise
Since years, variety of studies
Most work geared toward understanding implementation and providing formative feedback
Data stores and access improving
Prepared for Lumina Foundation Convening October 9-10, 2013
Early work on PK-12; later work on post-secondary
Also work outside these domains
Some funded by Heinz Endowments others by Pgh Promise directly

3. Research Questions: PK-12
What strategies are Pittsburgh high schools using to prepare students for post-secondary education?
In what ways, if at all, is the presence of the Promise influencing educators’, parents’, and students’ expectations and behaviors?
What practices are used by high schools that are most effective at getting students eligible for the Promise?
Why do HS graduates who are eligible for and have been awarded the Promise fail to use the scholarship in the subsequent semester?
To what extent do targeted programs influence participants’ “Promise-readiness?”
Prepared for Lumina Foundation Convening October 9-10, 2013
How are Pittsburgh high schools preparing students for post-secondary education?
In what ways, if at all, is the presence of the Promise influencing educators’, parents’, and students’ expectations and behaviors?
What practices are used by high schools that are most effective at getting students ready to use the Promise?
Why do HS graduates who are eligible for and have been awarded The Promise fail to use the scholarship in the subsequent semester?
To what extent do targeted programs influence participants’ “Promise-readiness?”
5. BA6GM for example. Also others possible.

4. PK-12 Methodologies
Case studies of 4 high schools guided by framework for college-career readiness
Pittsburgh Promise eligibility/award data paired with HS graduate data to determine high performing school in terms of proportion of graduates eligible for and using the Promise
Interviews with HS graduates who were awarded the Promise, had attained acceptance to a post-secondary institution but did not enroll in the semester after graduation
Prepared for Lumina Foundation Convening October 9-10, 2013
We used David Conley’s framework and added Kenneth Gray’s career maturity?
Case studies included:
Parent, teacher, student surveys
Interviews with administrators and other key school personnel
Focus groups with teachers, support personnel, and students
Review of school materials (Strategic Plan, Graduation requirements, etc.)
High school graduates data (demographics, GPA, attendance, advanced course taking, SAT/ACT participation and scores)

5. PK-12 Key Findings
Change in ASPIRATION does not equal change in BEHAVIOR
Schools need research-based college and career readiness framework
Use of coherent set of college and career ready practices linked with greater proportions of graduates eligible for the Promise
Graduates who could have but didn’t reported no strong internal drive to attend post-secondary in the summer after graduation but universally regretted this 9 months later
Prepared for Lumina Foundation Convening October 9-10, 2013
These findings influenced practices of schools, district, and Promise policies. District adopted college and career ready framework and began infusing it into language and practices.
Promise focused more on outreach to those who did not initially use the Promise.
Early changes in aspirations and intentions had to be intentionally guided to changed behaviors.
Most schools were not using a comprehensive research-based framework for thinking about college and career readiness.
Schools with greater proportions of graduates eligible for the Promise were employing a range of practices well supported by literature and aligned with college readiness framework.
In schools with fewer graduates eligible, college and career readiness practices were not explicit, coordinated, or systematically and uniformly available to students.
Graduates who could have but did not use the Promise were students who did not have strong internal drive to attend post-secondary in the summer after graduation but universally regretted this 9 months later. All wanted to go to post-secondary.

6. Research Questions: Post-Secondary
At the outset of their post-secondary studies, how prepared were Promise Scholars to meet the demands of college?
What are the Promise Scholar rates of retention in post-secondary? How do these compare to national rates?
Do particular high schools or post-secondary institutions have higher retention rates when student characteristics are controlled?
What are the most significant challenges to student post-secondary retention?
What strategies do post-secondary institutions use to support student success in higher education?
Prepared for Lumina Foundation Convening October 9-10, 2013

7. Post-Secondary Methodologies
Online survey of Promise Scholars in 1st year of post-secondary
Analysis of Promise Scholar post-secondary enrollment, retention, and persistence
Comparison to ACT national rates
Institution Promise Scholar retention rates compared to Institution cohort retention rates
Logistic regressions to identify predictors of Scholar retention
Case studies of 5 higher education institutions (high, moderate, and low performers)
Prepared for Lumina Foundation Convening October 9-10, 2013

8. Post-Secondary Key Findings
Scholars felt less well prepared for time management, study skills, and research skills
Scholar retention rates are higher than for ACT national sample for all institution types
For 8 of the 12 higher education institutions Scholars retained at higher rates than their peers at that institution
Family income level, higher high school GPA, and being female were predictive of retention
Once student variables were controlled there was almost no statistically significant differences among high schools
Statistically significant differences were found for individual higher education institutions
Higher education personnel identify challenges to student success ranging from academic preparedness, to “college knowledge,” to financial demands, and personal characteristics
Higher performing post-secondary institutions typically had elaborate programming in place to support Scholars
Prepared for Lumina Foundation Convening October 9-10, 2013
Scholars felt less well prepared for post-secondary on dimensions such as time management, study skills, and research skills.
Promise Scholar retention rates are higher than for ACT national sample for all institution types.
For 8 of the 12 higher education institutions serving the most Promise Scholars, the Scholars retained at higher rates than their peers at that institution who started the same year.
Family income level, higher high school GPA, and being female were predictive of retention.
Once student variables were controlled there was almost no statistically significant differences among high schools.
Statistically significant differences were found for individual higher education institutions. These findings were used to guide the selection of institutions for qualitative case studies.
Higher education personnel identify challenges to student success ranging from academic preparedness, to “college knowledge,” to financial demands, and personal characteristics.
The case study of higher performing higher education institutions typically had elaborate programming in place to support Promise Scholars.

9. Prepared for Lumina Foundation Convening October 9-10, 2013
SO YOU WILL ADD QUALIFIER RE ACT DATA?

10. Other Data/Evaluation Work
Articulating theory of change to support research design and programming development
Identifying indicators in the PK-12, higher education, and community that are either pre-conditions or targeted outcomes to serve as “dashboard”
Prepared for Lumina Foundation Convening October 9-10, 2013
Dashboard:
Advanced math/science course-taking
AP participation and performance
SAT/ACT participation and performance
Guidance counselor to student ratio
Teacher mobility rates by school
Teacher and student absence rates by school

11. Promising Opportunities in Pittsburgh
Linking data systems across sectors to allow more sophisticated and quasi-experimental designs.
Understanding the Promise “ecosystem” ways it changed as a result of the scholarship, and whether these changes resulted in any observed PK-12 and post-secondary attainment improvements.
Prepared for Lumina Foundation Convening October 9-10, 2013
EXPLAIN ECO SYSTEM??
Cautions: Developmentally appropriate timing of inquiries
Linking data systems across sectors to allow individual student PK-12 data, post-secondary data (enrollment, retention, and attainment), DHS data and scholarship use data for more sophisticated and quasi-experimental designs.
Linked data:
. Compare rates of post-secondary preparedness, enrollment, retention, and attainment pre- and post-Promise for various subgroups
Exploiting the Promise GPA threshold of 2.5 to understand whether and in what ways presence of the Promise influences college enrollment, retention, and attainment.
Examine how various support programs, school experiences, and DHS services relate to Promise influence

12. EFL Research/Evaluation Related to the Pittsburgh Promise
Iriti, J.E., Bickel, W.E., Meredith, J., & Walker, M. (2009). Looking inward to keep The Promise: What do Pittsburgh charter high schools do to prepare students for post-secondary education? Pittsburgh, PA: Learning Research and Development Center, University of Pittsburgh.
Iriti, J.E., & Bickel, W. E. (2010). Looking inward to keep The Promise: How Pittsburgh high schools prepare students for post-secondary education- Year 2. Pittsburgh, PA: Learning Research and Development Center, University of Pittsburgh.
Iriti, J. E., Bickel, W. E., & Nelson, C. A. (2010). Keeping “The Promise:” Practices for building post-secondary readiness and Promise scholarship eligibility in high schools. Paper presented at the Annual Meeting of the American Educational Research Association in Denver, CO.
Iriti, J. E., & Bickel, W. E. (November 11, 2011). “Promises” not yet realized: Those who could but don’t use Promise scholarship funds. Pittsburgh, PA: Heinz Endowments.
Iriti, J. E., & Bickel, W. E., with C. Long (May 31, 2012). Mentoring to The Promise: Evaluation of the Be a 6th Grade Mentor Program. Pittsburgh, PA: United Way of Pittsburgh.
 Iriti, J.E., Bickel, W. E., & Kaufman, J. (August 2012). Realizing “The Promise:” Scholar persistence and retention in post-secondary education. Pittsburgh, PA: Learning Research and Development Center, University of Pittsburgh.
Iriti, J., E., Bickel, W. E., Kaufman, J., & Wang, E. (2013). Pittsburgh Promise potential annual report indicators. Pittsburgh, PA: Learning Research and Development Center, University of Pittsburgh.
Prepared for Lumina Foundation Convening October 9-10, 2013