1. Program Performance: Using Management Tools and Evaluation Data Assessing Performance to Drive Change Jean L. Lloyd US Administration on Aging 4 th State Units on Aging Nutritionists/Administrators Conference August, 2006

2. Overview Panel –Jean Lloyd –Bob Hornyak –Nancy Wellman –Shannon Dukes

3. Objectives Describe context for AoA assessment of program performance Describe AoA program performance components Demonstrate how to use program performance data Identify resources available from AoA

4. AoA Mission “ Help elderly individuals maintain their independence and dignity in their homes and communities through integrated, high-quality, cost-effective systems of home and community-based long-term care.”

5. Context Modernizing the Older Americans Act –Empowering consumers –Providing greater choice for high-risk individuals –Encouraging healthy lifestyles –Enhancing the long term care connection within the aging network Medicare Modernization Act Government Performance and Results Act (GPRA) Office of Management and Budget (OMB) Program Assessment Rating Tool (PART)

6. AoA Responsibility Assist the network in developing tools that will allow them to efficiently and effectively assess and evaluate program performance to ensure that the impact of our programs aligns with our mission and enhances program performance.

7. Why Do We Measure Performance? To serve older adults more effectively & efficiently To comply with GPRA & PART To meet the OAA requirements To demonstrate that OAA programs work To justify budgets To satisfy key stakeholders To plan, manage and administer programs To identify training and technical assistance needs To identify participant, program, service trends and needs To develop programs to better meet needs To identify best practices to share and replicate To inform research

8. AoA Practices Integrated Program Evaluation AoA Integrated Program Evaluation Impact Process Outcome Cost-Benefit For true transparency, all four types of evaluations are important to inform the aging network, the Executive Branch, citizens, Congress, advocacy groups, the private sector and other stakeholders.

9. GPRA Passed by Congress, 1993 Requires federal agencies and programs to document results Requires federal agencies and programs to improve performance based on results Strategic Plan http://www.aoa.gov/about/strategic/strategic.asp Annual Performance Plans that accompany the budget justification http:// www.aoa.gov/about/legbudg/performance/legbudg _performance.asp

10. Core Outcome Measures Improve program efficiency (NAPIS/SPR,) Improve client outcomes and quality (National Surveys, Performance Outcome Measurement Project, Program Evaluation) Improve targeting (NAPIS/SPR, National Surveys, Performance Outcome Measurement Project, Program Evaluations)

11. PART Required by the OMB, within the Executive Office of the President Plays a major role in the budget process Establishes a systematic, consistent process for rating program performance in order to make budget decisions Emphasizes assessment criteria for program performance & management The measure of compassion is more than good intentions, it is good results. Sympathy is not enough. President George W. Bush, April, 2002

12. PART http://www.whitehouse.gov/omb/part/ http://www.whitehouse.gov/omb/expectmore/index.html http://www.whitehouse.gov/omb/part/ Section Scores CY 2003: –1 – 100% - Program purpose –2 – 75% - Program performance measures –3 – 100% - Program accountability –4 – 67% - Long-term performance Overall Score Changes –CY 2002 – 49 –CY 2003 – 81 –Score change due to new performance data, National Survey data Overall Rating: Moderately Effective, second highest rating

13. OMB Program Assessment of the OAA Program

14. Performance Outcomes Measures Project (POMP) Developing and refining performance indicators for the past 7 years Developed SUA/AAA capacity in performance measurement Collaborative federal/state/local effort Grantees are SUAs who are required to work with at least one AAAs; single PSA states must involve providers

15. Performance Outcomes Measures Project (POMP) Developing core performance measurement tools POMP 8 grantees will collaboratively be: –Validating tools that have been developed –Pilot testing statewide performance measurement methodology –Developing performance measurement toolkits for the aging network Survey tools include: 1.Consumer assessment of service quality for core services 2.Consumer satisfaction 3.Consumer reported outcomes 4.Consumer demographic characteristics

16. POMP OAA Title III Service Domains Caregivers Case Management Congregate Nutrition Program Information & Assistance Assessment Senior Centers Homemaker Service Home Delivered Nutrition Program Transportation Service

17. Instruments Available on GPRA.Net

18. Sample POMP Successes New York – AAAs used POMP Survey Results to support their case: –Wyoming County used POMP nutrition survey results to prove the importance of meals, with increased funds, did not have a waiting list. –Tompkins County used POMP nutrition survey results to show that clients could remain home longer with HDMs; Additional funding obtained which resulted in the expansion of the HDM routes.

19. Advanced POMP Study the impact of OAA Services on costs, including cost avoidance Looking at nursing home diversion and other models, including a frailty index 10 Advanced POMP projects

20. Sample Advanced POMP Success Preliminary SC – Used their results to increase state funding –Showed that participation in congregate and home delivered meals reduced emergency room visits and inpatient hospital admissions for older adults enrolled in Medicare and Medicaid –Received $2.9 M supplemental appropriation for HCBS, first new state funds in 10 years

21. Work In Progress POMP, Advanced POMP National Surveys AAA Survey Revised SPR Implementation & State Reporting Tool Program Information Management Study Demographic Data Program Evaluations Data Integration Project

22. Title IIIC and Title VI Evaluation Applies AoA evaluation standards Will develop a logic model for nutrition services as well as Title VI services Will Follow CDC Program Evaluation Model Two Phases –Design phase –Implementation phase

23. Goals for Title IIIC & Title VI Evaluation Program Monitoring - document overall program results Identify opportunities for program improvement Aid program planning process Show programs’ contributions to older adults’ independence Assess best practices including those programs demonstrating the most effective cost-benefit outcomes and impacts

24. Status of Title IIIC and VI Evaluation 2006 Extensive statement of work compiled Proposals received, solicitation closed Expect an award in the next month Final evaluation due within 2 years Will be the first complete evaluation of Title VI Will build on results of the Mathematica July 1996 Title IIIC & Title VI Nutrition Evaluation

26. National SPR Data Tables Congregate and Home Delivered Participants Meals Expenditures Program income Meal costs

27. Proportion of 60+ Served in OAA Nutrition Programs (S&MW&E) FYUSState A State B State C 20045.5%3.7%8.4%10.3% 20035.7%3.9%7.6%10.5% 20026.1%4.0%7.9%11.3%

28. Proportion of Congregate and Home Delivered Participants Served (%) In OAA Nutrition Programs [E&MW(similar proportion of older adults)] FYUS Cong State A State B US HD State A State B 20046552.570.23547.529.8 20036653.971.53446.128.5 20026652.767.63447.332.4

29. Proportion of Congregate and Home Delivered Meals Served (%) In OAA Nutrition Programs [E&MW(similar proportion of older adults)] FYUS Cong State A State B US HD State A State B 200442.535.658.857.564.441.2 20034335.660.65764.439.4 20024337.859.55762.240.5

30. Proportion of Congregate and Home Delivered Total Expenditures from OAA Funds (%) [W&W (geographic area)] FYUS Cong State A State B US HD State A State B 2004411756301854 2003451758321851 2002441755311852

31. Proportion of Congregate and Home Delivered Total Expenditures from Program Income (%) [W,E&S (large)] FYUS Cong State A State B State C US HD State A State B State C 200420.517.08.515.017.115.311.85.5 200319.017.710.015.217.015.711.86.4 200219.418.49.115.217.316.412.17.5

32. Comparison of Gross Cost of Meals [E,E (adjacent states) &S] FYUS Cong State A State B State C US HD State A State B State C 2004$5.81$9.59$9.17$3.76$4.68$8.05$5.32$4.06 2003$5.87$9.12$9.22$3.91$4.63$7.77$5.07$4.01 2002$5.46$8.83$8.04$4.82$4.41$6.68$4.73$.4.38

33. So, what does your state look like? How does your state compare with others? Do you measure outcomes in your state? How can you use these or other tools to help inform decisions? How does this affect program administration/implementation? Does your nutrition program need to change?