1. Monitoring & Evaluation Presentation for Technical Assistance Unit, National Treasury 19 August 2004 Fia van Rensburg

2. Outline What is monitoring What is monitoring …and evaluation …and evaluation Monitoring issues Monitoring issues Evaluation issues Evaluation issues Resources Resources

3. Monitoring and Evaluation Monitoring: a systematic management activity that involves the analysis of efficiency and effectiveness (actual vs. planned) deliverables Monitoring: a systematic management activity that involves the analysis of efficiency and effectiveness (actual vs. planned) deliverables Evaluation: analysis of efficiency, effectiveness, impact relevance and sustainability Evaluation: analysis of efficiency, effectiveness, impact relevance and sustainability

4. M&E Focus and Context Determined by… Specific project Specific project M&E policy M&E policy Contributes to… Annual reports Annual reports Performance management Performance management General good governance General good governance Future planning Future planning Influenced by … Line functional units Line functional units Other projects Other projects Organisational strategy, culture and objectives Organisational strategy, culture and objectives Government objectives Government objectives External factors External factors

5. Different levels and perspectives What is important for… The public or beneficiaries The public or beneficiaries Implementing partners Implementing partners Project managers Project managers Line managers Line managers Top management Top management Political leadership Political leadership

6. Action Hierarchy (Bennet, 1979) Inputs: Resources expended, time, money human resources Inputs: Resources expended, time, money human resources Activities: Implementation data, what programme offers or does Activities: Implementation data, what programme offers or does Participation: Characteristics of participants and clients, nature of involvement, background Participation: Characteristics of participants and clients, nature of involvement, background Reactions: Client satisfaction, interest in project, perceptions of strengths and weaknesses Reactions: Client satisfaction, interest in project, perceptions of strengths and weaknesses Knowledge, attitude and skills changes : measures on individual and group levels Knowledge, attitude and skills changes : measures on individual and group levels Practice and behaviour: measured over time Practice and behaviour: measured over time End results: impact on overall problem, ultimate goals, side-effects, social and economical consequences. End results: impact on overall problem, ultimate goals, side-effects, social and economical consequences.

7. Monitoring approach Part of planning or an afterthought? Part of planning or an afterthought? Does it cost money or save money? Does it cost money or save money? Special responsibility or integrated in business? Special responsibility or integrated in business? Specific moments or throughout project? Specific moments or throughout project? Different approaches at different project stages? Different approaches at different project stages? Top down or bottom up? Top down or bottom up? Qualitative or quantitative? Qualitative or quantitative? Usefulness of information? Usefulness of information?

8. Monitoring issues Indicators - output, outcome, impact Indicators - output, outcome, impact Situation before and after intervention - what are we measuring Situation before and after intervention - what are we measuring Variables, control groups, longitudinal studies Variables, control groups, longitudinal studies Integration of technical and financial implementation, monitoring and reporting Integration of technical and financial implementation, monitoring and reporting Watch those risks and assumptions! Watch those risks and assumptions! Do you monitor your monitoring and evaluate your evaluations? Do you monitor your monitoring and evaluate your evaluations?

9. Monitoring Issues Annual Operational Plans Annual Operational Plans Cross-cutting issues - gender Cross-cutting issues - gender Integration of social accountability - financial, human and environmental Integration of social accountability - financial, human and environmental Stats SA Stats SA Lessons learnt from 10 year review Lessons learnt from 10 year review

10. Views on Evaluation Always required Always required The project is not “proud” yet The project is not “proud” yet Conclusive change – quality and degree Conclusive change – quality and degree Reasonable impact Reasonable impact Too many contributing variables Too many contributing variables Criteria for project selection Criteria for project selection Formative evaluation Formative evaluation Baseline data available Baseline data available Evidence of improvement and learning Evidence of improvement and learning Reverse logic Reverse logic

11. Evaluation Issues Can government evaluate itself? Can government evaluate itself? Role of Public Service Commission? Role of Public Service Commission? Is empowerment evaluation relevant in SA government context? Is empowerment evaluation relevant in SA government context? Are external evaluators objective? Are external evaluators objective?

12. Information systems and management Does the pieces of the puzzle match and do we have a complete picture? Does the pieces of the puzzle match and do we have a complete picture? Integrated databases? Integrated databases? What information is required by decision makers, project managers, other stakeholders? What information is required by decision makers, project managers, other stakeholders? Integrity of data? Integrity of data?

13. Standards Utility – practical information needs of primary intended users Utility – practical information needs of primary intended users Feasibility – realistic, prudent, diplomatic and frugal Feasibility – realistic, prudent, diplomatic and frugal Propriety – conducted legally, ethically, with due regard for those involved in evaluation and affected by results Propriety – conducted legally, ethically, with due regard for those involved in evaluation and affected by results Accuracy – technically adequate information about programme features, defendable results Accuracy – technically adequate information about programme features, defendable results

14. Competencies Systematic inquiry – research and evaluation Systematic inquiry – research and evaluation Evaluation competence - info needs of PIUs, situational analysis, organise and manage evaluation project Evaluation competence - info needs of PIUs, situational analysis, organise and manage evaluation project General skills – logical and critical thinking, communication, interpersonal and computer General skills – logical and critical thinking, communication, interpersonal and computer Professionalism – self-knowledge, ethical, standards, professional development Professionalism – self-knowledge, ethical, standards, professional development

15. Resources Websites Websites Models and checklists Models and checklists