Creating National Performance Indicators that are Relevant to Stakeholders: Participatory Methods for Indicator Development, Selection, and Refinement Demia L. Sundra, M.P.H., Margaret Gwaltney, M.B.A., Lynda A. Anderson, Ph.D., Ross Brownson, Ph.D., Jennifer Scherer, Ph.D. Evaluation 2004 November 3, 2004

Contributors to Project DEFINE [Developing an Evaluation Framework: Insuring National Excellence] Evaluation Advisory Group Ross Brownson, Ph.D. (co-chair) Alan Cross, M.D. Robert Goodman, Ph.D., M.P.H. Richard Mack, Ph.D. Randy Schwartz, M.S.P.H. Tom Sims, M.A. Avra Warsofsky Carol White, M.P.H. CDC Lynda A. Anderson, Ph.D., Robert Hancock, Demia Sundra, M.P.H. COSMOS Corporation Jennifer Scherer, Ph.D., Margaret Gwaltney, M.B.A. (Currently with Abt Associates), Thérèse van Houten, D.S.W., Cynthia Carter Concept Systems, Inc. Dan McLinden, Ed.D., Mary Kane

Goals of Presentation Describe the context in which the performance indicators for the CDCs Prevention Research Centers (PRCs) program were developed Review the participatory methodology utilized to develop and select national indicators Discuss the benefits, challenges, and lessons learned from developing performance indicators in an established, diverse, national program

Background and Context

Features of CDCs Prevention Research Centers Program 33 academic-based extramural research centers across United States Academic centers partner with community and state organizations to develop, conduct, and disseminate prevention research through participatory methods Diversity across centers: When founded (newly funded to 18 yrs) Community setting and partners Focus of research

Context for Developing a National Evaluation National evaluation planning project initiated in response to: Institute of Medicine (IOM) report on the PRC program Support for evaluation at CDC Growth of program and related need for accountability Project DEFINE Goals (Planning Phase) Engage stakeholders, develop a logic model and performance indicators, and draft an evaluation plan Maintain a participatory and utilization-focused approach throughout

Intended Purposes of Performance Indicators Individual data on each PRC Evaluation Monitoring Technical assistance needs Cross-center summary data Accountability Program improvement Information sharing and communications with internal and external stakeholders

Anticipated Challenges in Developing PRC Indicators Centers strive to achieve diverse health outcomes Program had few previous cross-center requirements Centers are at various stages of growth and maturity Interests of diverse stakeholders had to be considered Concern existed about how performance indicators would be used Indicators had to be meaningful and impose minimal burden on PRCs in terms of time and cost

Methodology

Basis of Project DEFINE Concept Mapping Gained national and community perspectives on PRC program through 2-tiered approach Engaged diverse stakeholders in brainstorming statements describing PRC program Statements analyzed to create visual maps of concepts Concepts used to build draft logic models Community and national logic models combined Engage the Community Diversity & Sensitivity Relationships & Recognition Active Dissemination Technical Assistance Training Research Methods Research Agenda Core Expertise & Resources

Development of Draft Indicators More than 70 indicators first drafted Mapped to all components of program logic model Some indicators dropped, others refined based on input received at regional meetings and PRC contextual visits 52 candidate indicators remained on the list

Stakeholder Recommendations from Regional Meetings Select a limited number of indicators focused on features common across Centers Collect data on some components of logic model in other ways as part of the national evaluation, rather than through indicators Develop indicators through an iterative process, with multiple opportunities for input Link the performance indicators to the PRC Information System

Stakeholder Selection of the National Performance Indicators 52 indicators listed in structured feedback tool (workbook) All stakeholder groups provided feedback and comments PRCs, Community, State, and CDC Planned on having core and optional indicators

Results of Performance Indicator Feedback 100% response rate received on workbooks Comments from workbook summarized within each stakeholder group 3 of the 4 stakeholder groups recommended 8 indicators 2 of the 4 stakeholder groups recommended an additional 11 indicators

Resulting National Performance Indicators Evaluation advisory group selected and refined 13 indicators based on Results and feedback of workbook Map of indicators across the logic model Cross-walk of recommended indicators with IOM report recommendations Indicators correspond to various logic model components, e.g. Community input on selecting health priorities (input) Existence of explicit research agenda (activity) Evidence of peer-reviewed publications (output)

Collecting the Information Performance indicators were integrated into the information system that was in development Conceptualized from the beginning Reinforced through stakeholder feedback Fields were created in the information system for each performance indicator Information system was developed and reviewed by: Evaluation Contractors Centers and partners (usability and pilot tests) CDC staff Evaluation Advisory Group PRC Steering Committee

Core and Optional Indicators Only core indicators developed through Project DEFINE. Consensus allowed us to: Focus on 13 indicators Not use resources for optional indicator development and integration into information system 11 out of 28 PRCs developed center-specific indicators on their own Topics areas such as community satisfaction with partnership; funding generated; web site hits; infrastructure measures; research methods appropriate for minority population

PRC Performance Indicators: Summary Specific component requirements across all grantees Indicators reflect both process and outcome measures, with focus on process Initial requirements as part of new funding cycle Prospective evaluation Assess general information across PRCs rather than specific health topics Defining common outcomes, e.g. community capacity Indicators will be refined during evaluation implementation

Challenges, Benefits, and Lessons Learned

Current Challenges with Performance Indicators Requests for more guidance on how to further define and collect data Development of summary reports and provision of feedback to all stakeholders Need to increase specificity of indicators over time Balance between participatory processes and program requirements

Buy-in, support, and ownership of indicators Evaluation advisory group was critical for trust and support from larger stakeholder groups Community voice is reflected in indicators Perspective of the PRCs staff and partners reflected in utility and feasibility issues surrounding indicators and information system Benefits of Participatory Approach for Performance Indicator Development

Lessons Learned and Recommendations Utilize participatory methods for selecting and refining indicators to increase stakeholder support Build sufficient time into schedule to allow multiple opportunities for stakeholder input Acknowledge inherent challenge in developing indicators for an established program Include community input in indicator development to increase accountability to partners

For more information on the Prevention Research Centers Program Click on about the program to view the Conceptual Framework (logic model) and narrative Contact information: Demia Sundra:

PRC IS: General Information Page

PRC IS: Community Committees

PRC IS: Health Priorities