The Intersection of Performance Measurement and Program Evaluation: Searching for the Counterfactual The Intersection of Performance Measurement and Program Evaluation: Searching for the Counterfactual Moscow 2011 Douglas J. Besharov School of Public Policy University of Maryland

Performance Management  Efficiency studies (“outputs”)  How much does the program cost?  Monetary, nonmonetary, and opportunity costs  Could it be delivered more efficiently?  Effectiveness studies (“outcomes” and “impacts”)  Does the program achieve its goals?  Could it be more effective?  Both require a comparison, or a “counterfactual” 8 Douglas J. Besharov, June 2011

2

It Matters How Children Are Raised Douglas J. Besharov, June

Ineffective Early Childhood Education Programs IHDP Low-birth weight, pre- term infants and their parents Home visits, parenting education, and early childhood education $20,400 per child per year No significant impacts, initial IQ gains fade CCDP Poor children under age 1 and their parents Case management, parenting education, early childhood education, and referrals to community- based services $19,000 per family per year ($60 million annually) No significant impacts Early Head Start Poor children ages 0-2 and their parents Child development, parenting education, child care, and family support services $18,500 per child per year ($700 million annually) No significant impacts Douglas J. Besharov, June

Douglas J. Besharov (October 21, 2008) “The closest thing to immortality on this Earth is a federal government program.” – Ronald Reagan Program Improvement, not Program Dismantling

Performance Management  Leadership,  Management, and  Measurement 6 Douglas J. Besharov, June 2011

Performance Management  Leadership,  Management, and  Measurement 7 Douglas J. Besharov, June 2011

Point #1 Counterfactuals are needed for accurate performance measurement. 9 Douglas J. Besharov, June 2011

 Head Start Impact Study (2010): 10 years and running  Moving to Opportunity Study (1994): 17 years and running  Employment Retention and Advancement evaluation (1998): 13 years and running  Building Strong Families Project (2002): 9 years and running  National Job Corps Study (1993): 15 years to complete Impact Evaluations Take Too Long to Manage Performance Impact Evaluations Take Too Long to Manage Performance Douglas J. Besharov, June

Logic Model for Job Training Programs Inputs Training facilities Staff Funding Client characteris tics Inputs Training facilities Staff Funding Client characteris tics Activities Job search/job readiness training Classroom instruction Job skills training Activities Job search/job readiness training Classroom instruction Job skills training Outputs Hours of training instruction Hours of practice Staff admin Skill certificates Outputs Hours of training instruction Hours of practice Staff admin Skill certificates Outcomes Job search skills Technical job skills Interpersonal skills Outcomes Job search skills Technical job skills Interpersonal skills Proximal Impacts Earnings Employment UI/Welfare Receipt Crime Proximal Impacts Earnings Employment UI/Welfare Receipt Crime Distal Impacts Higher lifetime earnings/ employment Lower poverty Stronger economy Distal Impacts Higher lifetime earnings/ employment Lower poverty Stronger economy Theory : If government provides job training to the unemployed, than the unemployed will receive job skills necessary for good jobs, increased earnings, and a stronger economy Design : (1) Job search/job readiness training, (2) skills training, (3) in a classroom. External Community and Societal Context Problem : Some unemployed do not have the necessary skills to obtain and keep well-paying employment, leading to lower income, greater use of government benefits, and a weaker economy.

Point #2 Carefully applied, a measured outcome coupled with a logic model’s theory of change —often buttressed by other evidence— can serve as a more timely and more useful performance measure than a formal evaluation of long-term impacts. 12 Douglas J. Besharov, June 2011

 There is no output, so no positive outcome can be reasonably predicted.  The output itself is sufficiently suggestive of a likely outcome.  The output is produced at such a prohibitively high cost, that, regardless of its likely outcome, it does not meet cost-effectiveness or cost-benefit tests. When “Outputs” Imply “Outcomes” Douglas J. Besharov, June

 Evaluations of on-going programs  Rolling randomized experiments  Pre-post studies (with embedded counterfactual)  Regression-discontinuity designs  Evaluations of specific program “improvements”  Randomized experiments  Pipeline studies (or rolling implementation)  Interrupted time series studies Feasible “Outcome” Evaluations Douglas J. Besharov, June

A Clear Interrupted Time Series Douglas J. Besharov, June

Circling the Wagons Douglas J. Besharov, June

 Top-down administrative and funding incentives -- together with --  Bottom-up voucher-like programs Accountability Systems Douglas J. Besharov, June

2

When “Outcomes” Imply “Impacts”  When the desired impact is reasonably predicted to follow from the measured outcome Douglas J. Besharov, June

Ineffective Job Training Programs Job Corps Low-income youth JTPA Low-income adults, dislocated workers, and out- of school youth Classroom training, on-the- job training, job search assistance, adult basic education, and other services $2,400 per participant for 3-4 months ($60 million annually) Women: Small initial gains in earnings, employment, and GED receipt fade by 5 yrs Men: Small initial gains in earnings fade by 5 yrs, no other impacts Youth: On significant impacts WIA (dislocated) cts Douglas J. Besharov, June