1. Managing for Change: The Art of Results Based Management Module 3 - Risk and Performance Measurement Mosaic Workshop - Ottawa July 14th - 18th, 2008

2. Enablers and Constraints… Are forces affecting any part of your project both positively and negatively

3. Why Identify Assumptions? Programs/projects are implemented in a dynamic environment Factors external to the project are often the cause of failure Make explicit the conditions external to the program/project upon which its internal logic is dependent

4. Test Your Design Logic OUTCOME Participating citizens increasingly challenge intolerance, racism and hatred in their communities. OUTPUT Individuals and groups gain awareness of racism and discrimination. Internal Logic What conditions are you assuming will exist in the operation’s environment? External Logic

5. Risk Risk is the uncertainty that surrounds project results. It is the likelihood and impact of an event with the potential to influence the achievement of results.

6. Risk Analysis Matrix Effects Significant (3) Moderate (2) Minor (1) Low (1)Medium (2)High (3) Likelihood Monitor Risks Acceptable Risks Monitor Risks Risk Mitigation Required Acceptable Risks Project Example… Internal Staff Training on Public Consultation Practice Outcome: Local Government Authority staff consistently apply public consultation guidelines in the implementation of their unit responsibilities; they do so with a solid grasp of the subject matter and with a command of the leading techniques of engagement 1. What could hamper progress? 2. What is the likelihood of a problem? 3. What effect would the problem have on the Outcome?

7. Risk mitigation strategies should reduce the likelihood of an undesirable event, or minimize the effect the event could have on the project. Risk areas that remain above the level of acceptable risk should be monitored. Risk Mitigation & Monitoring

8. Workshop Exercise - Risk Management 1.Develop your matrix 2.Pick one output and one outcome. 3.Define your assumptions. 4.Define your risks and and conduct risk analysis 5.Consider mitigation and monitoring strategies

9. Assumptions, Risk Identification and Mitigation Strategy Template Expected Result (as per LFA) AssumptionsRisksRisk Ranking/Rating Likelihood/Effect (L/M/H) Mitigation Strategy Output: Outcomes: Impact:

10. Risk Identification and Mitigation Strategy Template Expected Result (as per LFA) AssumptionsRisksRisk Ranking/Rating Likelihood/Effect (L/M/H) Mitigation Strategy Output: Outcomes: Impact:

11. Risk Identification and Mitigation Strategy Template Expected Result (as per LFA) AssumptionsRisk Ranking/Rating Likelihood/Effect (L/M/H) Mitigation Strategy Output: Outcomes: Impact:

12. Keeping Track of Results Shifting from the design and planning side of RBM to information gathering

13. Monitoring and Evaluation - The Distinction Monitoring A continuous self-assessment of progress -collecting data and comparing current performance with planned activities and results. Evaluation Independent (often external), periodic, strategically focused assessment of a project’s continuing relevance, management, governance, results, coverage, external relationships, sustainability, etc.

14. Keeping Track of Results… Means paying attention to … –Indicators –Sources of Information –Methods and Frequency –Person(s) to do the Work –Who needs to know

15. Indicators… Point to information you need to determine if the project is creating the results you want. They can be both quantitative and qualitative. They should help the project make wise management decisions. They should be cost effective to use.

16. Quantitative Indicators Expressed as a percentage, ratio, proportion, comparison, or number that can be analyzed statistically Examples? –Number of households represented on Village Development Committees –Ratio of men to women in senior management roles in the partner organization –Proportion of households using servicable latrines –Number of community organizations committing resources to address hate crimes in ‘x’ community

17. Quantitative Indicators - Your Turn Climate Change Conference – Stakeholder Dialogue on Provincial Action Plan Outcome - Public discourse on the Province’s Climate Change Action Plan Update peaks through community organized events ?

18. Qualitative Indicators Expressed as a change or comparison between two states of understanding or experience that can be analyzed for patterns Relies on people’s judgment or perception Examples: –Villager (men and women) stories of how their lives have changed following completion literacy training –Change in the level of rapport between local government and higher levels of government. –Change in community perceptions of the role/efficacy of Village Development Committees –Before-after comparison in the coverage of race relations issues in the local media

19. Qualitative Indicators - Your Turn Climate Change Conference – Stakeholder Dialogue on Provincial Action Plan Outcome - Public discourse on the Province’s Climate Change Action Plan Update peaks through community organized events ?

20. Indicators, Baseline, and Target Indicator Baseline Target Meaning The information to be collected What the indicator would tell you at the beginning What you want the indicator to tell you at a specific point in the future Example Proportion of school children routinely washing their hands after using the latrine/washroom 1 in 5 4 in 5

21. Qualitative Indicators - Setting out Markers of Progress Sometimes you cannot meaningfully track progress by counting “units” at the beginning (baseline), middle and end (e.g. ratio of local government units/service areas that have institutionalized public involvement as a way of working) You may need to set out… –Stages of completion, or –Markers of progress

22. Qualitative Indicators - Setting out Markers of Progress Consider the policy formulation… –Indicator is ‘ Comparison of actual to planned progress in updating ‘x’ policy’ You could say… –Baseline - ‘0’ updated policy –Target - ‘1’ updated policy But how useful is that from a management point of view?

23. Qualitative Indicators - Setting out Markers of Progress Would be more useful to identify progress markers. For example: a)Established need for updated policy b)Policy research underway c)Policy options formulated d)Discussion and approvals underway e)Updated policy approved f)Updated policy enacted Baseline might be ‘a’ Yr 1 target might be ‘c’ Yr 3 target might ‘e’

24. Validity - Does it measure the result?Yes No Is it cost-effective to collect the information?Yes No Does it help understand how the program or project is affecting men and women, and/or specific sub-groups of people differently? Yes No Does it provide useful information for management decisions? Yes No Does it communicate well to stakeholders (public, policy makers, donors)?Yes No Indicator Selection Checklist

25. Test Case #1 New skills through training - Output Participant farmers understand post harvest handling procedures and are willing to try these out on their own after the training. Indicator: a)Change in the average amount of production lost between field and market among participating farmers b)Proportion of participant farmers who can describe how they will replicate handling procedures following the training Yes No Does it communicate well to stakeholders (public, policy makers, donors)? Yes NoDoes it provide useful information for management decisions? Yes No Does it help understand how the program or project is affecting men and women, and/or specific sub-groups of people differently? Yes NoIs it cost-effective to collect the information? Yes NoValidity - Does it measure the result?

26. Test Case # 2 Climate Change Conference – Stakeholder Dialogue on Provincial Action Plan Outcome - Public discourse on the Province’s Climate Change Action Plan Update peaks through community organized events Indicator: Pick your favourite from your group’s list of quantitative and qualitative indicators. Test it… Yes No Does it communicate well to stakeholders (public, policy makers, donors)? Yes NoDoes it provide useful information for management decisions? Yes No Does it help understand how the program or project is affecting men and women, and/or specific sub-groups of people differently? Yes NoIs it cost-effective to collect the information? Yes NoValidity - Does it measure the result?

27. Test Case #3 Household Participation in Community Decision- making - Outcome Households participate more fully in community decision- making and improvement initiatives; they help each other out when in trouble Indicator: Reported change in the way households respond to common hardships (e.g. abusive relationships, financial shortfalls, childcare, infirmity) Yes No Does it communicate well to stakeholders (public, policy makers, donors)? Yes NoDoes it provide useful information for management decisions? Yes No Does it help understand how the program or project is affecting men and women, and/or specific sub-groups of people differently? Yes NoIs it cost-effective to collect the information? Yes NoValidity - Does it measure the result?

28. Performance Measurement Plan Results Levels IndicatorsBase -Line TargetData Source/ Method Report Frequency & Resp. Impact Outcome Output

29. Exercise Prepare a Matrix of the Performance Measurement Framework Select at least one output and one outcome. Brainstorm a total of 3 indicators per result: 1 quantitative and 1 qualitative indicator and 1 other indicator. Develop your sources of information, methods, responsibility and frequency.