E-101 Thursday Section 1 T HURSDAY, 1 ST NOV., – P. M.

Roadmap for today Housekeeping Where are we? What are the goals? Application Feedback – sticky notes 2

Housekeeping 3 Paper II is coming up Paper III – Dec. 20 th – we will start looking at this more closely once Paper II is back

4

Where are we- Assessing the performance of a system Policy and program evaluation Curriculum, Standards & Assessment 5

M&E - 6 One of the most impt. aspects of this course - make sure you walk away with it Resources – International Initiative for Impact Evaluation J-PAL UNESCO IIEP UNICEF Innocenti Research Center UNICEF Evaluation Database World Bank Evaluation Database Other classes – Haiyan Hua’s M & E class Tom Kahne - Program Evaluation

Weiss Reading 7 What does she have to say? Summary? What are the tensions that might arise in the evaluation process?

Monitoring & Evaluation 8 There is a difference between M and E! Monitoring: The gathering of evidence to show what progress has been made in the implementation of programs. Focuses on inputs and outputs. Evaluation: Measuring changes in outcomes and evaluating the impact of specific interventions on those outcomes.

Monitoring & Evaluation 9

10

11

Reflections - 12 What are the tensions b/w an insider & outsider role as an evaluator? If time is money, how does one implement process? What are the ways in which impact might be considered? How might cultural differences impact these? Creating local capacity for communities to monitor their own programs & hold states accountable for the delivery of basic services? Dissemination to the beneficiaries?!

Implementation Issues? 13 Political Economy Policy Context Methods

Political Economy 14 What is the policy purpose? Test innovations & defend budget Allocate budget to certain programs Pressure to demonstrate aid effectiveness and scale up Answer electorate Reaching Targets

Policy Context 15 What questions need answering? Who are the stakeholders who want answers? Do they also have the same amount of clout? What policy questions need answers? How much of a change would determine success? What does the government really need, and what will it use?

Methods 16 Operationalizing the outcome Process/outcome indicators Channels of impact – theory of change Underlying assumptions [log frame] Who has access to the program? Are they the same as the intended beneficiaries? Data collection – time vs. cost Analysis Dissemination

Things to think about 17 What ? Type of information and data to be consolidated How? Procedures and approaches for data collections and analysis Why? Why are we doing this – does it support the program/policy When? Freq. of data collection, reporting Who? Responsibilities and capacities of focal points and resource persons

Sector Analysis: What is it?  Sector Review: summary of the state of the sector  Sector Assessment: success of the sector in meetings certain goals  Sector Analysis: (most comprehensive) includes review and assessment (above), and also describes sector needs, constraints, and opportunities for improvement.  (Kremmerer, 1994)

Sector Analysis: What is it? Tries to answer questions relating to:  External efficiency  Internal efficiency  Access & equity  Administration & supervision  Costs & financing (Pigozzi & Cieutat, 1998)

Sector Analysis: What is it? The contents of a sector analysis may include: 1. Synthesis 2. Economic & Financial Analysis 3. History, Structure & Management Capacity 4. Analysis by Sub-Sectors: 5. Special Studies 6. A list of background documents (Pigozzi & Cieutat,1988)

Sector Analysis: Steps I. Initiation II. Formation of the National Steering Committee III. Determination of the scope of the analysis IV. Formation of the subsector teams V. Formation of the technical analysis team VI. Instrumentation and collection of data VII. Preliminary analysis of data VIII. Discussion of preliminary findings IX. Collection of any additional data X. Formulation of draft recommendations XI. Discussion of draft recommendations XII. Revision and prioritization of recommendations. (Kremmerer, 1994)

Opposition sectorsSupport sectorsOpposition sectors External actors Sector positi on Anti-system Legal Opposition Ideological support Core support Ideological support Legal Opposition Anti- system The Government Social sectors Political parties Pressure groups Political Mapping: An Example

Stakeholder Map: An Example

Program Theory “Program theory…refers to the mechanisms that mediate between the delivery (and receipt) of the program and the emergence of the outcome of interest. The operative mechanism of change isn’t the program activities per se but the response that the activities generate.” - Carol Weiss, Evaluation, p. 57 You will be reading a section of this book for next week.

Activity: Program Theory For each program, there may be multiple pathways to the intended outcome. Source: Weiss, C. (1997). Theory-based evaluation: Past, present, and future. New Directions for Evaluation, (76), 41-55, p. 42.

From Program Theory to Implementation to Evaluation  Logical Frameworks  can help you to organize:  Design  Implementation  Evaluation  Developed by USAID in the 1960s  Vary slightly, but always include: 4 x 4  Narrative Summary, Indicators, Means of Verification, and Assumptions

Log Frames 28 “In terms of relevance, practice could be the development of common understandings gained via an examination of assumptions and creation of a common language for evaluation” - Garaway p.p. 718 Log frame as tool to do examine Relevance: what, where, who Effectiveness: Process & implementation Efficiency: Making the best use of resources Impact: Support the program purpose

How do you develop a logic model? Linear process from beginning to end Move around (iterative) Backwards (identify results then determine the resources/inputs and activities that are required) Can be described as If ___________ then ____________. If we have X resources and do Y activities, then we will have Z Outputs, W Outcomes and T Impact.

Putting It All Together Impact Outcome Output Impact Outcome Output

How to annoy your family with what you learned at HGSE… Your Planned Work Your Intended Results Resources/ Inputs ActivitiesOutputsOutcomes Impact Logic Model of a Family Trip Holiday flight schedules Family schedules Holiday weather Your exam/ paper due dates Create family schedule Get holiday flight info Get tickets Arrange ground transport Tickets for all family members Frequent flyer miles used Money saved Ground transport Family members enjoy vacation Continued good family relations

Logical Frameworks Project Structure Objectively Verifiable Indicators Means of Verification Important Assumptions Goal: Purpose: Outputs: Activities: (Inputs)

Logical Frameworks Project Structure Objectively Verifiable Indicators Means of Verification Important Assumptions Goal Purpose Outputs Activities (Inputs) IF AND THEN

Logical Frameworks Project Structure Objectively Verifiable Indicators Means of Verification Important Assumptions Goal: Purpose: To increase the number of primary school graduates. Outputs: Activities: (Inputs)

Logical Frameworks Project Structure Objectively Verifiable Indicators Means of Verification Important Assumptions Goal: Develop human capacity in XX country. Purpose: To increase the number of primary school graduates. Outputs: Activities: (Inputs)

Logical Frameworks Project Structure Objectively Verifiable Indicators Means of Verification Important Assumptions Goal: Develop human capacity in Somalia. Purpose: To increase the number of primary school graduates. Outputs: By abolishing school fees, expect to see more children enrolling in primary Activities: (Inputs)

Logical Frameworks Project Structure Objectively Verifiable Indicators Means of Verification Important Assumptions Goal: Develop human capacity in Somalia. Purpose: To increase the number of primary school graduates. Outputs: By abolishing school fees, expect to see more children enrolling in primary Activities: Abolishing school fees.

Logical Frameworks Project Structure Objectively Verifiable Indicators Means of Verification Important Assumptions Goal: Develop human capacity in Somalia. Purpose: To increase the number of primary school graduates. Outputs: By abolishing school fees, expect to see more children enrolling in primary Activities: Abolishing school fees. All primary schools no longer requires fees.

Logical Frameworks Project Structure Objectively Verifiable Indicators Means of Verification Important Assumptions Goal: Develop human capacity in Somalia. Purpose: To increase the number of primary school graduates. Outputs: By abolishing school fees, expect to see more children enrolling in primary Activities: Abolishing school fees. All primary schools no longer requires fees. Poll families to ask what fees exist.

Logical Frameworks Project Structure Objectively Verifiable Indicators Means of Verification Important Assumptions Goal: Develop human capacity in Somalia. Purpose: To increase the number of primary school graduates. Outputs: By abolishing school fees, expect to see more children enrolling in primary Activities: Abolishing school fees. All primary schools no longer requires fees. Poll families to ask what fees exist. Primary can be funded without fees.