Evaluating the Options Analyst’s job is to: gather the best evidence possible in the time allowed to compare the potential impacts of policies

Time Outcomes Activities, Policies, and Programs Before During After What should we do?:  Policy Analysis  Benefit-cost analysis  Cost-Effectiveness  Needsassessment What was the impact?:  Outcome evaluation  Performance Measurement What are we doing? How does it work?:  Process Evaluation

WSIPP: “Return on Investment: Evidence-Based Options to Improve Statewide Outcomes” What information do we get from WSIPP study? How did they create it? What principles can we take away for our predictions?

Benefits: To whom? For what period?

Costs: What’s included? How can they be positive?

Summary stats: What are they? How are they different?

Net present value (WSIPP 2011, Tech appdx p.6)) Q is how much of outcome you get with program P the value of the outcome C is cost of program Dis is discount rate Proage is age of participant when program starts

Benefit-Cost Ratio (WSIPP 2012, tech appdx p.6) Internal Rate of Return: discount rate at which NPV is zero =0

WSIPP study elementsWSIPP study elements: Meta-analysis: averages results across multiple studies to get program impacts Q (meta example)(meta example) Estimates monetary benefits: –private gains to participants, –public value of avoiding outcomes like abuse and crime, and –private value of not being victim, P Puts these together to get impact on outcomes over life with discount for later benefits and costs (dis) Benefit Cost analysis adds up benefits and costs

WSIPP adjusts estimates for quality of the evaluation evidence it collects: (tech appdx p.17) Determine empirically if have enough studies (i.e., which types of estimates are largest).

From WSIPP case: What outcomes will you account for? Impacts on who? –(state or local budget, participants, by-standers) What time frame will you use? –(discounting and NPV) How will you weight multiple sources of evidence given its qualities? How will you add it all up? Will you demonstrate Sensitivity analysis of results? What other criteria “count” other than monetized?

General starting points for predictions: Need detailed descriptions of the alternatives (but not TOO detailed) Focus on key impacts and most important costs Common metrics (dollars, DALYS, etc.) are useful if they capture key outcomesDALYS, May need to adjust estimates for your scale or context Get the best possible evidence—won’t get perfect information Need to understand strengths and weaknesses of your evidence and communicate it

Where to get evidence on costs and impacts (from Hatry): Previous experience with similar changes Pilot study in your organization Information from other organizations that implemented similar policies (program evaluations) Academic or think tank studies (academic journal search and web search) Modeled or “engineered” estimates Theories and logical inference about causal connections (tragically leads to “high,” “medium,” or “low”!) WEAKEST!

Does the evidence from elsewhere apply to your organization (external validity)? Is the policy or political context different in important ways? Are the economic conditions different? Is the target of the policy (e.g., client population or location) different in critical ways? Would the policy or program be implemented in the same way? To the same scale?  You must assess the severity of the differences and predict their impacts on your outcomes.

Sources of uncertainty in estimates: Validity of comparison and study methodology (see WSIPP report) Statistical uncertainty (randomness) Uncertainty in how would be implemented in new context Possible changes in other policies or conditions (e.g., economic or social)

What do you do with uncertainty? Give explicit range estimates for costs or impacts Perform sensitivity analysis and discuss effects on trade-offs (e.g., Monte Carlo) Use worst case/best case scenarios Give best guess estimates with caveats Build resilience into your policy options But…. Clients like certainty There is limited time/space to explain details Need to make decisions in face of uncertainty

How do you add up? Most likely you don’t!--Any adding up scheme inherently weights the criteria Can use cost/benefit analysis (monetize) Use go/no go (minimum threshold) for each criteria and pick policy that meets all, then maximize one Don’t have to recommend one policy but MUST point out KEY trade-offs across policy options

Your mandate: Find at least one quantitative outcome criterion that you can find evidence to estimate. You must provide cost estimates of your options Use at least one academic or think tank study as evidence for at least one outcome (and preferably more) I challenge you to find the most informative quantitative and qualitative evidence from the broadest sources.