1. 4.10.1 Monitoring and Evaluation Systems for NARS organizations in Papua New Guinea Day 4. Session 10. Evaluation

2. 4.10.2 Learning objectives By the end of this session, participants will be able to: Explain the different categories and types of evaluation used in M&E. Present examples of evaluation events in NARS organizations. Explain the issues to consider when planning an evaluation. Develop concept notes for evaluation events at all levels of a NARS organization.

3. 4.10.3 Evaluation – the concept Evaluation is judging, appraising or determining the worth, value or quality of research – whether it is proposed, ongoing or completed – in terms of its relevance, effectiveness, efficiency, impact and sustainability. Granger et al. (1995)

4. 4.10.4 Evaluation – the event..... implemented by an individual or a team of evaluators to answer specific questions on various aspects of an intervention by using approaches and methods to collect data (or assemble existing data), analysing the data, generating information, coming to conclusions and making recommendations in the evaluation report..

5. 4.10.5 Types of Evaluation Timing in relation to intervention a)Ex-ante evaluation: Before; forward looking; planned objectives and expected results (benefits, impact) or context - to guide design b)On-going evaluation: During; performance of an evaluation; quality of the implementation process; achievement of objectives and results - to warn of deviations, take corrective actions c)Ex-post evaluation: After; effects/impact of the intervention on e.g. target group, wider population, region - to learn and plan for the future (new interventions, policies, funding)

6. 4.10.6 Types of Evaluation Internal versus external a)Internal evaluation: Managers or staff of the intervention, e.g. internal reviews; conducted routinely in a NARS organization b)External evaluation: Individuals or teams outside the intervention, e.g. such as experts or peers with specialist skills, knowledge, experience; staff or stakeholders participate

7. 4.10.7 Examples of Evaluations in a NARS organization 1.Context assessments 2.Critical reflection as part of routine monitoring (process evaluation) 3.Reviews 4.Final evaluations 5.Impact assessments 6.Critical reflection 7.Peer review 8.Employee performance assessment

8. 4.10.8 Examples of Evaluations in a NARS organization Timing in relation to intervention Type of evaluation in a NARS organization Level mainly used Examples Ex-ante (Formative Evaluation) Context assessmentprogram, organization needs assessment, stakeholder analysis, sector analysis, ex-ante impact assessment On-going (Process Evaluation: related to evaluation of implementation process and Outcome Evaluation: related to evaluation of results/outcome Critical reflection as part of routine monitoring (related to evaluation of implementation process) project activity, project, program conduct routine monitoring methods and reflect on data Reviews (related to evaluation of implementation process) project, program, organization annual program review, project mid-term review, quarterly project activity meeting, program mid-term review, periodic review of organization, annual project review Reviews (related to evaluation of results) project, program, organization Final Evaluation (related to evaluation of implementation process) project, program end of project evaluation, end of program evaluation Final evaluation (related to evaluation of results) project, program, organization end of project evaluation, end of program evaluation, final project outcome evaluation, final program outcome evaluation (often called ‘Impact assessment’) Ex post (Impact Evaluation) Impact assessmentprogram, organization program impact assessment Source: Authors

9. 4.10.9 Planning Evaluations – Issues 1. Focus 7. Responsibility for conducting 2. Purpose 8. Methods 3. Recipient 9. Resources 4. Key questions10. Data Sources 5. Evaluation criteria11. Data Analysis 6. Responsibility for organization12. Reporting and dissemination

10. 4.10.10 Planning Evaluations – Issues Proposed objectives and outcomes of intervention - during design (ex-ante) Management of the implementation process of the intervention – ongoing Purpose and outputs– during implementation or at completion Outcomes - during implementation or at completion Impact – ex post. 1.Focus: Which aspect of the intervention needs to be evaluated?

11. 4.10.11 Planning Evaluations – Issues 2.Purpose: Why is evaluation done? How will evaluation be used? 3.Recipient: Who is the evaluation being done for? Who needs answers to specific questions?

12. 4.10.12 Planning Evaluations – Issues  Was the intervention implemented as planned (activities, outputs)? If no, what was done differently to the plan? Why? Which short term outcomes were achieved by the intervention? Are they relevant to target population? Which factors external to the intervention facilitated or hindered the achievement of outcomes? What measures have been put in place to promote sustainability of the intervention's outcomes and impact? etcetera 4.Key questions: What specific information needs does the evaluation address? What are the priority questions that need to be answered?

13. 4.10.13 Planning Evaluations – Issues  relevance – outcomes consistent with beneficiaries’ needs? efficiency – input versus output effectiveness – outputs, outcomes achieved impact – total effects sustainability – continuation of benefits 5. Evaluation criteria: What criteria are used to judge the aspects of an intervention?

14. 4.10.14 Planning Evaluations – Issues  meeting set standards in the specific field of the intervention  best or good practice  ideals or social/political values and expectations equity appropriateness potential usability by others degree of dissemination to stakeholders 5. Evaluation criteria … continued:

15. 4.10.15 Planning Evaluations – Issues 6.Responsibility for organization: Who in the NARS organization is responsible for organizing the evaluation? 7.Responsibility for conducting: Who should conduct the evaluation?

16. 4.10.16 Planning Evaluations – Issues 8.Methods : Which forms/approaches/methods might be most suitable? What are the most appropriate/preferred methods of data collection? 9.Resources: What resources are needed for the evaluation?

17. 4.10.17 Planning Evaluations – Issues  Routine monitoring  Periodic data collection  Secondary sources  One off data collection for special data as part of the evaluation 10.Data sources: What data is needed? What are the data sources?

18. 4.10.18 Planning Evaluations – Issues 11.Data Analysis: How will data be analyzed? 12.Reporting and dissemination : Who will receive the report (audience)? What type of information/content will the report contain? How will report be presented? How will report be disseminated? When should the report be presented?

19. 4.10.19 Standard Evaluations in NARS organizations identify a set of standard evaluations for the various organizational levels include the essential evaluations to manage better, achieve, learn all interventions at a particular level will be required to implement guided by information needs of managers and stakeholders; and the financial and personnel resources available in the organization.

20. 4.10.20 Standard Events in NARS organizations develop concept notes and guidelines for managers and staff of the organization coordinated by M&E unit use planning issues presented to prepare the concept note easy access for managers and staff include in the overall organizational M&E plan