MAPP Process & Outcome Evaluation

Good Evaluation… Is not an afterthought or something that is done only if there is extra money. Measures progress and identifies reasons for celebration. Informs continuous improvements. Doesn’t have to be difficult. Doesn’t happen only during the action cycle—evaluation should be considered at every phase of the process.

Phase 1 Process Evaluation Did we fulfill phase 1 objectives? Ask the community, did we… Ensure that the appropriate participants were included? Educate participants on the activities that will be undertaken, their responsibilities, how long it will take, and the results that are expected? If the community answers no, ask… Who else needs to be included? What about the process remains unclear? These questions should be asked and addressed throughout the process.

Phase 1 Process Evaluation Ask the community, did we… Structure a planning process that engages participants as active partners? Structure a planning process that uses participants' time well? Structure a planning process that builds commitment? Set a tone of openness and sustained commitment among participants? Structure a planning process that results in a realistic plan? Continuously assess what can be improved as the community moves forward in the process.

Preparing for Outcome Evaluation in Phase 1 If we implement MAPP successfully, what will the process look like? What do we expect to achieve after one iteration of the MAPP process? Increased community engagement Stronger partnerships Better system efficiency More system resources Less system redundancies Return on investment Improved health How will we know we’ve successfully met our expectations?

Phase 2 Process Evaluation Did we fulfill phase 2 objectives? Ask the community, did we… Guide the community through a collaborative process resulting in a shared vision and values? If the community answers no, Why wasn’t the process collaborative? Why isn’t the vision and values shared? What could have been improved? What will be done in subsequent phases to ensure better collaboration?

Phase 2 Process Evaluation Ask the community, did we… Create values that are fundamental principles and beliefs that will guide the community through the process? Create a vision that provides focus, purpose, and direction to the MAPP process? These questions can also be answered during the action cycle and can inform improvements in subsequent iterations.

Phase 3 Process Evaluation Ask the community, did we… Include indicators that are important to our community? Effectively include the community in the design, implementation, and analysis of assessment data? Use the most effective data collection methods? Ensure results from one assessment did not overshadow another assessment?

Phase 3 Process Evaluation Did we create an effective data management system for future data collection efforts? If the community doesn’t respond favorably to these questions, identify what could have been improved in terms of… Community engagement Division of labor Methodology

Preparing for Outcome Evaluation in Phase 3 Reflecting on our vision, did we collect data that will inform whether we moved toward our vision? Reflecting on what we want to achieve through the process, will our data help confirm success? Increased community engagement Stronger partnerships Better system efficiency More system resources Less system redundancies Return on investment Improved health

Phase 4 Process Evaluation Did we fulfill phase 4 objectives? Did we… Effectively analyze and synthesize the data from all four MAPP assessments? Use an effective process to identify the fundamental policy choices or critical challenges that must be addressed in order for our community to achieve its vision? Ensure partners collectively identified and felt ownership of strategic issues? Ensure strategic issues were not biased by any one agency’s agenda? Ensure our strategic issues resonate with the community?

Phase 5 Process Evaluation Did we fulfill phase 5 objectives? Did we… Use an effective process for formulating our goals and strategies? Ensure our goals and strategies reflect what the community collectively wants to achieve? Effectively formulate goals and strategies such that we can develop practical work plans?

Preparing for Outcome Evaluation in Phase 5 Phase 5 provides a critical connection between your community’s strategic plan and how public health improvements will be made. It is important to explicitly detail how strategies will help reach goals and how goals address strategic issues.

Phase 6 Process Evaluation Did we fulfill phase 6 objectives? Did we… Effectively use assessment data to inform action cycle planning? Identify SMART objectives related to each strategy and goal? Use an effective division of labor and organizational structure? Ensure system partners have ownership of action cycle activities? Ensure energy and progress was sustained throughout the action cycle? Effectively use evaluation results to improve action cycle activities?

MAPP Outcome Evaluation Did we implement action plans that moved us closer to our goals and that addressed our strategic issues? Did our iteration of the MAPP process meet our initial expectations identified during phase 1?

MAPP Outcome Evaluation: Example 1 Continued (a) Did we implement action plans that moved us closer to our goals and that addressed our strategic issues? Objective: To decrease the cost of travel via public transportation to health centers in the county by 50% by June 1, 2010 Activity: To negotiate reduced fares with the public transit authority Hypothesis: If the community was successful in decreasing the cost of travel via public transportation, then reduced fares would improve access to care for residents with limited transportation—one strategy formulated in phase 5. If one action cycle activity was to negotiate reduced fares with the public transit authority, the effectiveness of that activity would be evaluated against the measurable objective to decrease the cost of travel via public transportation to health centers in the county by 50% by June 1, 2010. **Evaluation could involve having health center visitors answer a few questions during registration about how they arrived at the health center and whether they used reduced fare public transportation. If the community was successful in decreasing the cost of travel via public transportation, they would then have to determine if those reduced fares improved access to care for residents with limited transportation—one strategy formulated in phase 5. **It would be important to determine whether residents knew about and took advantaged of the reduced fares. **It would also be important to determine if reducing the cost of public transportation was enough to improve access. Perhaps getting to the bus station is a more prohibited than cost.

MAPP Outcome Evaluation: Example 1 (b) Did we implement action plans that moved us closer to our goals and that addressed our strategic issues? Measure to what extent the strategy contributed to the goal that all persons living in our community will have access to affordable quality health care. Continue existing and/or add strategies/activities to ultimately address the strategic issue: How can the public health community ensure access to population-based and personal health care services? If that strategy was successful, the community would have to measure to what extent that strategy contributed to their goal that all persons living in our community will have access to affordable quality health care. **Remember there are numerous strategies that could be associated with one goal. Some might address only a segment of the population while other strategies might address only one facet of a larger challenge. The community would then consider continuing and or supplementing the program with additional strategies and/or activities to ultimately address the strategic issue: How can the public health community ensure access to population-based and personal health care services? **Remember, strategic issues shouldn’t be questions that are easily answered. However, it is important to continuously assess how well activities are addressing the larger, fundamental cross-cutting strategic issue.

MAPP Outcome Evaluation: Example 2 (a) Did our iteration of the MAPP process meet our initial expectations identified during phase 1? Expectation: MAPP will improve collaboration among local public health system and community partners. Baseline Measures: At the beginning of the process, partners: Rated system collaboration according to levels of trust, interaction, and sharing of resources. Conducted a network analysis using the PARTNER tool to determine how different system partners are connected. Next slide… In phase 1, your Steering Committee identified the following expectation: MAPP will improve collaboration among local public health system and community partners. At the beginning of the process, partners rated system collaboration according to levels of trust, interaction, and sharing of resources. At the beginning of the process, your community also conducted a network analysis using the PARTNER tool to determine how different system partners are connected. Perceptions of collaboration and network analysis are conducted again during phase 6 and compared to data collected at the beginning of process.

MAPP Outcome Evaluation: Example 2 Continued (b) Did our iteration of the MAPP process meet our initial expectations identified during phase 1? Comparison: Perceptions of collaboration and network analysis are conducted again during phase 6 and compared to data collected at the beginning of process. The accumulation of process evaluation data compared with the network analysis and perceptions of improved partnerships provide a measure of how well the MAPP process improved collaboration during one iteration of the process. Perceptions of collaboration and network analysis are conducted again during phase 6 and compared to data collected at the beginning of process. In addition, as a part of your process evaluation efforts, you asked participants to evaluate how well each phase facilitated consensus, improved relationships among partners, aligned partner missions, and worked toward a shared vision. The accumulation of process evaluation data compared with the network analysis and perceptions of improved partnerships would then provide a measure of how well the MAPP process improved collaboration during one iteration of the process.

MAPP Outcome Evaluation Things to remember: Evaluation planning starts at the beginning of the MAPP process. Logic models help articulate why a MAPP process should result in improvement and provide a framework for process and outcome evaluations.