1 of 16 Evaluating an Information Project Building Consensus © FAO 2005 IMARK Investing in Information for Development Evaluating an Information Project Building Consensus

2 of 16 Evaluating an Information Project Building Consensus Learning Objectives At the end of this lesson you should be able to: identify a framework for analysis of subjects to be included in the evaluation; be aware of how to carry out a preliminary analysis of risks to the evaluation.

3 of 16 Evaluating an Information Project Building Consensus Introduction In order to consider the issues involved in the planning of an evaluation, we are following a scenario, in which Dr Kumar, the Director of publications at SMAU University, must evaluate a newsletter. So far, Dr Kumar and his colleagues have: defined the focus of the evaluation; developed a list of possible Evaluation Management Committee members; and called the first Evaluation Management Committee meeting.

4 of 16 Evaluating an Information Project Building Consensus Reaching agreements on the subjects Here are the questions for the agenda of the meeting: What will be the main objective in an evaluation of RICE NEWS? What will be the main subjects to be included? What is going to be measured? What is likely to go "right" and what could possibly go "wrong"?

5 of 16 Evaluating an Information Project Building Consensus Reaching agreements on the subjects There are many stakeholders invited to the meeting. The goal will be to encourage everyone to agree on the basis of the evaluation. Jon suggests to use a card technique...

6 of 16 Evaluating an Information Project Building Consensus The NBAO analysis Lets divide the RICE NEWS project into three-four smaller topics... Should we use inputs, activities, outputs and impacts?

7 of 16 Evaluating an Information Project Building Consensus I would rather use simpler categories, like NBAO... N EEDS B ENEFICIARIES A CTIVITIES O UTCOMES The NBAO analysis

8 of 16 Evaluating an Information Project Building Consensus Here is how the technique works: give each member a small pile of cards. start with N (needs). ask everyone to write down the needs RICE NEWS was supposed to address, one need for card. The NBAO analysis

9 of 16 Evaluating an Information Project Building Consensus then, gather up the cards. pin or tape them on a large board, so that everyone can see. Dr Kumar can lead a discussion in which people try to group them, and reach an agreement on the main needs, beneficiaries, activities, and outcomes. The NBAO analysis

10 of 16 Evaluating an Information Project Building Consensus The NBAO analysis Once the meeting has started, Dr. Kumar is ready to use the card technique. He explains to the members of the EMC its goal: This technique will help us: to reach an agreement on the subjects we want to include in the evaluation; later, as a basis to develop specific questions.

11 of 16 Evaluating an Information Project Building Consensus The NBAO analysis At the completion of the card technique", here's what the EMC has come up with:

12 of 16 Evaluating an Information Project Building Consensus The NBAO analysis Now, it is important to be aware of what could go wrong with the evaluation before the whole process starts. Dr. Kumar says: …we should also do a risk analysis right at the beginning, to think of things that might go wrong, and prevent potential problems...

13 of 16 Evaluating an Information Project Building Consensus...I propose a SWOT analysis, to think together about the Strengths, Weaknesses, Opportunities and Threats of the evaluation we are proposing... The SWOT analysis

14 of 16 Evaluating an Information Project Building Consensus The SWOT analysis What are likely to be the biggest STRENGTHS of our evaluation? What are likely to be its biggest WEAKNESSES? If the evaluation goes well, what are likely to be the biggest OPPORTUNITIES that it will bring to us? And what are the biggest THREATS to it not going well?

15 of 16 Evaluating an Information Project Building Consensus The SWOT analysis After the completion of the SWOT analysis, Dr. Kumar concludes the first EMC meeting. He and his colleagues have both an evaluation committee in place and a consensus on major issues. The next step will be to become more specific, and think about what to measure and how to do it.

16 of 16 Evaluating an Information Project Building Consensus Summary When planning an evaluation it is important to find an agreement on the content of the evaluation. During the Evaluation Management Committee meeting, a good starting point can be to examine: Needs, Beneficiaries, Activities, Outcomes. To reach an agreement, it is useful to introduce in the meeting a card technique. The next step will be to use the same technique to do a SWOT analysis. After reaching consensus on these major issues, the Evaluation Committee can move on to the details of what needs to be measured.