1. Educational Webinar: Evaluation tied to Return on Investment Wednesday, January 21, 2009 National Network of Public Health Institutes Fostering Emerging Institutes Program Call in Number: (800) 504-8071 Code: 3019823

2. Please mute your line by pressing *6 You can un-mute your line by pressing *7 Do not put your phone on hold.

3. Agenda Welcome, Today’s Webinar, Who’s on the CallErin Marziale IntroductionKaren Minyard Overview of Organizational Evaluation Dr. Julia Heany Dr. Clare Tanner Michigan Public Health Institute Using Return on Investment Dr. Bill Custer, Health Economist Georgia State University Examples from Public Health InstitutesDina Baker Public Health Management Corporation Elaine Beane Michigan Public Health Institute Discussion and QuestionsSylvia James Webinar EvaluationErin Marziale Wrap up and Adjourn Erin Marziale

4. FEIP Focus on Sustainability Emotional Intelligence & Collaborative Leadership October 2008 Evaluations tied to ROI January 2009 Communication with Multiple Stakeholders April 2009 New Orleans Conference – Sustainability: Focus on Special Topics of Interest May 2009

5. Evaluation Clare Tanner, Ph.D. Julia Heany, Ph.D. Michigan Public Health Institute

6. Why evaluate?  To ensure a program is implemented as designed  To test the assumptions that drive a program’s design  To test the results of changes in a program  To determine whether a program is operating efficiently  To assess the degree to which the value of achieving a program’s goals exceeded the cost of producing them  To identify whether a program achieves its intended objectives  To determine whether progress toward a program’s goals can be related to the program

7. AEA’s Guiding Principles  Systematic Inquiry Evaluators conduct systematic, data-based inquiries  Competence Evaluators provide competent performance to stakeholders  Integrity/Honesty Evaluators ensure the honesty and integrity of the evaluation process from conceptualization to dissemination and use of results

8. AEA’s Guiding Principles  Respect for People Evaluators respect the security, dignity, and self-worth of evaluation participants, program participants, clients, stakeholders, and the broader community  Responsibilities for General and Public Welfare Evaluators articulate and take into account the diversity of general and public interests and values that may be related to the evaluation

9. CDC’s Framework for Program Evaluation in Public Health 1 1 MMWR 1999; 48(No. RR011)

10. Types of Evaluation  Evaluability Assessment Is the program ready to be evaluated?  Process Evaluation Was the program implemented as designed? Did the program reach the intended audience?  Formative Evaluation How can the program be improved over the course of implementation?  Outcome Evaluation What happened as a result of the program?  Summative Evaluation Did the program work?

11. Logic Models: A Key Evaluation Tool

12. Evaluation Stakeholders  Stakeholders in an evaluation may include: Program staff & clients/consumers Partner agencies Community leadership Funders  Why engage stakeholders in the evaluation process? Reduce anxiety Buy-in to the process and the outcomes Access to multiple, varied perspectives Ability to communicate results in a way that speaks to your audience  Participatory Evaluation

13. Organizational Evaluation  Examines the relationship between organizational outcomes and organizational goals, plans, and projects  Can focus on an organization’s administration, structure, strategic plan, staff structure and competencies, services, projects, facilities, or finances.  General evaluation principles apply here as well: Know your stakeholders and your audience, and build internal and external support for evaluation Plan, plan, and plan some more Ensure the necessary resources, including technical expertise, are in place

14. Resources on Evaluation  CDC Framework for Program Evaluation in Public Health (online): http://www.cdc.gov/eval/framework.htm http://www.cdc.gov/eval/framework.htm  RWJF Guide to Evaluation Primers: http://www.rwjf.org/files/publications/RWJF_Research Primer_0804.pdf http://www.rwjf.org/files/publications/RWJF_Research Primer_0804.pdf  WF Kellogg Foundation: Evaluation Handbook: http://www.nationalserviceresources.org/node/14473 http://www.nationalserviceresources.org/node/14473

15. Questions? Dr. Julia Heany 517-324-7349 jheany@mphi.org Dr. Clare Tanner 517-324-7381 ctanner@mphi.org

16. Estimating Return on Investment in Public Health Initiatives William Custer, Ph.D Georgia State University Wcuster@GSU.edu

17. Return on Investment Return on Community Investment: Cost-Benefit Analysis Looks at total Social Benefit Return on Investment Comes from the private, for-profit sector Examines returns to specific ‘investors’

18. Cost-Benefit Analysis Grew out of Public Good framework: How to compare competing projects that provide public goods Provide return on investment estimates for public goods Net Social Benefit= Total Social Benefit-Total Social Cost NSB= TSB-TSC The major issue is capturing all relevant costs and benefits.

19. Cost-Benefit Analysis: Measurement Issues Opportunity Costs For example: Benefits health intervention can include: Value of medical care avoided Value of increased production when death is avoided Value of increased production due to good health Value of increased utility due to increased life expectancy or better health (or both)

20. Cost-Benefit Analysis: Measuring Value of Life  Human Capital: income measures value  Life time Earnings  Do not capture non-market values  Compensating differentials: How much are people willing to pay to avoid death (or injury)  Market price of safety features  Risk differentials in wages reflect value of life

21. Cost-Effectivness Analysis Grew out of private return on investment literature: Compares competing projects that attempt to achieve same goal Avoids attempting to put monetary value on life. Does not evaluate net benefits of the goal CEA= (C 1 -C 2 )/(E 1 -E 2 )

22. Cost-Effectivness Analysis CEA= (C 1 -C 2 )/(E 1 -E 2 ) E is a measure of health outcome: usually incremental expected life-years E =  F i Where F is the probability the individual at current age is alive at time i Measurement of Costs similar to CBA Measurement of E critical

23. Cost Effectiveness Analysis Example

24. CBA vs. CEA vs. ROI CBA evaluates social welfare CEA looks at specific outcomes but does not look across outcomes ROI identifies specific individuals and groups who benefit and estimates the return to them.

25. Estimating Costs & Benefits Identify outcomes relevant to project and audience Describe outcomes in dollar terms Include all benefits Incidence of benefits (who reaps the benefits) Identify costs of achieving goals Include all costs Incidence of costs (who bears costs)

26. Estimating Costs & Benefits Identify sources of information Primary data  Internal data Secondary Data Existing research literature

27. Measuring Benefits and Costs Structure–Process–Outcomes Structure measures Capacity-infrastructure-people Process measures Methods--utilization Outcome Measures Results

28. Measuring Benefits and Costs Structure–Process–Outcomes Structure measures Descriptive Process measures Flows Outcome Measures Requires control groups

29. ROI Calculation Logic Model identifies stakeholders within Structure Process Outcome Value (monetarize) Outcome Willingness to pay Costs Research on Values Relate measureable characteristics to outcomes Structure => Outcome Process => Outcome

30. Data Sources and Resources Cost of Illness Handbook: http://www.epa.gov/oppt/coi/http://www.epa.gov/oppt/coi/ Also: http://www.rti.org/page.cfm?objectid=CA1E1F48-8B6C- 4F07-849D6A4C12CBF3C3 Medical Expenditure Panel Data http://www.ahrq.gov/data/mepsix.htm Hospital Discharge Data National data: HCUP http://www.ahrq.gov/data/hcup/http://www.ahrq.gov/data/hcup/ That site has individual state contacts listed Center for Studying Health System Change http://www.hschange.com/ Behavioral Risk Factor Surveillance System CDC survey administered by many states http://www.cdc.gov/brfss/sitemap.htm

31. Linking data with story-telling What works? What could work better?

32. Example 1 While a recession affects the contents of our wallets, it can also influence our health. Through Public Health Management Corporation’s Community Health Data Base (CHDB), those in the Philadelphia region have the opportunity to track the link between the economy and chronic disease. “Considering that our region’s residents consume less than the recommended minimum of five servings of produce each day, Philadelphians are more likely to suffer from chronic disease,” explains CHDB program director Francine Axler. “One reason area residents are not maintaining a healthy diet is cost and a recession can certainly add to the problem.”

33. Example 2 In 2004, D’Juan Diggs moved to Philadelphia with his girlfriend and three children. More than anything, Diggs wanted to create a stable environment for his children. Diggs recalls, “We were living in a small apartment trying to save for something bigger.” Unfortunately for Diggs and his family, time and money soon ran out. Their family had to be placed into the shelter system through the Department of Human Services. While in temporary housing with his family, Diggs was approached by Focus on Fathers parent educator, Robert McIntyre. Although wary at first, Diggs decided to sign up for the program – he didn’t have much to lose. “In my own personal thinking, this was the last thing that I needed considering my housing situation.” But the stress of being the primary caretaker of his family was taking a toll on Diggs. “The program offered me a way to voice some issues about being a better father and caretaker. It also helped me deal with the shelter environment.” As he continued in the program, Diggs began to appreciate its benefits. “I realized the distinctiveness of the program - something just for men - a forum to talk about serious issues pertaining to our own individual families.” Research from the National Conference of State Legislatures (2000) has found that fathers are often “unsure of what is expected of them as men, partners and fathers" and assume their families are “better off without them.” RCH recognized this problem.

34. Example 3 Seventeen-year-old Nicholas Shanks is your average, teenage boy. He spends his time playing video games with friends, watching cartoons on Saturday mornings and listening to music on his computer. But when this soft-spoken teen took the stage on June 17, 2008 to deliver his speech as valedictorian of Martin Luther King High School to 287 members of his graduating class–more than one person in the audience had tears in their eyes. When he was in 9th grade, Nicholas and his mother, Sherri Newton, became homeless. “I lost my job,” recalls Newton, “and unemployment wasn’t paying for my rent.” When his father was laid off, according to Nicholas “things started to fall apart.” McMillan and Ligons-Ham estimate that almost 10,000 children in the School District of Philadelphia are homeless–living in emergency shelters, transitional housing, or ‘doubled-up’ (living with other families).

35. Return on Investment for the Michigan Nursing Corps Elaine Beane, Ph.D. Michigan Public Health Institute

36. Discussion and Questions

37. Thank You! Join us for the next FEIP educational webinar: Communication with Multiple Stakeholders April 14 th, 2009 (3:00- 4:30 eastern)