1. Update on Performance Measures Pilot and Development of the Cancer Plan Index Presented by Deborah Porterfield, MD, MPH RTI International Presented at CCC Program Directors Meeting Atlanta, GA, May 13, 2008 RTI International is a trade name of Research Triangle Institute 3040 Cornwallis Road ■ P.O. Box 12194 ■ Research Triangle Park, NC 27709 Phone 919-541-1293 e-mail dporterfield@rti.orgFax 919-541-6683

2. Outline Performance measures pilot Background and purpose Methods Examples of preliminary results and issues raised in the pilot Next steps Cancer Plan Index Background and goals Methods and where we are Next steps

3. Background: Performance Measures Since 1998, the Division of Cancer Prevention and Control (DCPC) has provided funding to states, tribal organizations, and jurisdictions to conduct comprehensive cancer control (CCC) in their areas. Evaluation was part of the Building Blocks and has been stressed by CDC from the beginning In 2003 the DCPC initiated a process of developing an evaluation framework and performance measures for all CCC programs Now CDC is asking programs to submit information on performance measures

4. Developed by CDC to assess recipient’s progress toward reaching required Program Activities One assessment tool for the entire US, including Tribes and Territories, and USAPI Jurisdictions Performance is judged by CDC through various methods including a Performance Measurement Assessment tool “Measures of Effectiveness” are to be listed on the work plan and can be whatever programs decide. CDC does not “judge” them in any way.

5. Purpose of Performance Measurement To establish accountability for public investments in cancer prevention and control; To document the outcomes and accomplishments of funded programs and partners; and, To facilitate a quality improvement process at the program level.

6. Further Purposes of Performance Measures for CDC General information on CCC nationally Ex: The number of sectors, organizations and individuals meeting around the CCC table Ex: Funds and in-kind other than from CDC General knowledge of the program to help in technical assistance Ex: Sharing of satisfaction surveys Ex: Proportion of interventions that are truly “evidence-based”

7. Further Purposes of Performance Measures for CDC Monitoring of the most important activities Ex: Reviewing data at specified intervals Ex: Coalition representing “all” of your people Documenting outcomes that we can be used to show that CCC “works” At this point, results from the Performance Measurement worksheet are not being used for funding decisions

8. Methods Advisory workgroup Within CCC, both Program Evaluation and Scientific Support teams represented Framework based on CCC logic model Draft performance measures Extensive review Development of a worksheet Pilot of worksheet in 2008

9. Preliminary Results and Issues Raised Year 01 Performance Measures

10. Response Rate 48 of 51 worksheets for states and D.C. 6 of 6 worksheets for tribes 8 of 10 worksheets for USAPI jurisdictions Total of 62 of 67 worksheets were received

11. Update on Analysis Focused review of USAPI jurisdictions and tribes already complete Data entry underway for states Numbers presented today represent 45 of the states, tribal organizations and USAPI jurisdictions

12. Response Burden Median of 10 hours to complete the worksheet (range of 3–100 hours)

13. Some Potential Performance Measures Appeared to Go Smoothly 1.3Assessment of members’ satisfaction with aspects of the partnership was conducted 62% of programs YES 2.1Assessment of cancer burden using specified data sources and appropriate periodicity was conducted 96% of programs YES 6.1The program has a written evaluation plan64% of programs YES

14. And Some Additional “Informational” Measures Appeared to Go Smoothly 3.1Amount of non-CDC fundsMean$2.3 million Median$158 k 3.2Amount of in-kind contributionsMean$642 k Median$31 k

15. And Some Questions Need Work 5.1Proportion of partner organizations implementing interventions directly related to the plan How to know? What counts? 5.2Proportion of plan objectives being addressed How to know? 5.3Proportion of interventions that are evidence based Definition of evidence based needs more discussion 7.1Number and list of policy changesNeed criteria for what “counts” — perhaps also in-process activities?

16. General and Measure-specific Revisions Refinements in instructions Calculation of proportions Even if the performance measures is a proportion, the worksheet will not ask it as such Suggestions for additional sectors in 1.1

17. Next Steps Analysis of all worksheets Engage CCC Branch to review the data and suggest revisions to the worksheet Continued involvement of programs in refinement of measures and worksheets for Year 02 Ongoing work of the CDC/RTI workgroup to refine the worksheet Determination of benchmarks or standards Inclusion of revisions in next funding announcement in fall/winter 2009

18. Conclusions Performance measurement is an important tool for public health agencies to establish accountability and a culture of quality improvement. With further refinement of the measures and evolution of a system of routine measurement, this activity has the potential to stimulate improvements in CCC at the program level and to document nationwide progress in CCC efforts

19. Cancer Plan Index

20. Background A State Plan Index was developed by the Obesity program at CDC to review existing plans and provide guidance for new plan development 2 articles published in Preventing Chronic Disease DCPC interested to adapt this methodology for cancer plans

21. Goals of the Cancer Plan Index Activity Describe the current status of plans Inform guidance from the DCPC to programs on CCC plans Provide information to programs who are currently writing or revising their plans Provide information for one of the performance measures in the current funding announcement

22. Methods for the Cancer Plan Index Advisory workgroup Program volunteers’ extensive review Workgroup reworked components and added items (currently) Review by larger group of program volunteers

23. Next Steps Pilot in summer 2008 Review of all existing cancer plans Outputs: Report with descriptive information about current plans Revised instrument that will be available to programs  Guidance from CCC on plan content Future performance measure for CCC programs

24. Questions?