1. REAL (working with Singizi) Presentation to NSA 3rd/4th August
Impact evaluation
REAL (working with Singizi)
Presentation to NSA 3rd/4th August

2. Defining impact evaluation
The positive and negative, intended and unintended, direct and indirect, primary and secondary effects produced by an intervention.
OECD Development Assistance Committee definition

3. Extending the definition
When understanding impact there is a need to go beyond “what works and for which groups” and answer other equally important and related questions, such as “why do certain programmes work for some groups and not for others”.
Martin and Grubb, What works and for whom:A review of OECD countries’ experiences with active labour market policies, Swedish Economic Policy Review, 8 (2001) 9-56

4. Possible Impact Evaluations Designs
Experimental and Quasi Experimental
Usually has a counterfactual of some kind (an estimate of what would have happened if the intervention had not occurred) or a particular sort of counterfactual (for example, comparisons with a group who did not receive the intervention)
Non-experimental Designs
Are impact evaluation designs that do not include a matched comparison group
Outcomes and impacts assessed without a conventional counterfactual to address the question

5. Approach for this study
Undertaking a meta-review of impact studies undertaken during the period of NSDSIII
Completing a tracer study (where sectors have not undertaken a tracer study)
Will not have a control group but instead is a longitudinal study (though possibility of comparing with administrative data available being explored to provide a point of comparison for key issues such as length of time to transition to the labour market where programmes are offered to unemployed)

6. Rationale for this design selection
Its practical and …
country-led planning and evaluation (often covers whole country)
Has horizontal and vertical partnerships
Aim of long-term changes, but a need for short-term results
Multiple stakeholders with different perspectives
Non linear
Many components or services
Different theories of change evident across projects
Multiple donors and agencies
Context is critical and challenge has multiple and contested causes
Large, complex
Complex interventions
May include a number of projects and wider scope
Often involves several blueprint approaches
Defined objectives but often broader and less precise and harder to measure
Often not time-bound
Context important
multiple donors and national agencies
Complicated programs
Simple projects
Small, simple
“blue print” producing standardized product
relatively linear
Limited number of services
Time-bound
defined and often small target population
Defined objectives
Adapted from Michael Bambeger, June 28, 2011 and Singizi Theory of Change and complex programmes

7. Implications for methodology
Need to consider quantitative and qualitative data -
Participatory, developmental and utilisation approach – these consider change from the perspective of the people/communities/institutions undergoing the intervention
Which can be collected in many various ways, like surveys, focus group discussions, interviews, etc

8. Tracer focus Programme details – field, level, type
Status in programme (if still in – STOP, if dropped out – one set of questions, if completed – another set of questions)
Demographics – ace, gender, age, disability etc
Entry into programme – motivation, selection, employment status when entering the programme (will route from here)

9. Tracer focus (continued)
If unemployed/pre-employed before the programme –
Employment status now and/or education status
Extent to which programme enabled you to access this employment (support measures in place and recognition of programme)
Whether the programme enabled the learner to succeed in this work (quality and relevance of programme and opportunities that the learner have had access to in the workplace)
Possibilities of navigating further pathways of learning or employment

10. Tracer focus (continued)
If employed before the programme
Status now - Whether the individual is still in the same workplace
Promotion? Increase? New responsibilities? Access to training?
Whether the programme enabled this (recognition or new skills – linked to quality or relevance of the programme, etc)
Any changes in the workplace because of their increased skill

11. Tracer focus (continued)
If self-employed before the programme
Status now - whether still self-employed/employed/studying/unemployed
Extent to which the individual has increased skills to run and grow their business (if still self-employed)
If programme linked to other support as relevant (finance, market, etc)

12. Tracer focus (continued)
If unemployed now –
Whether still looking and if so, where -
Whether the programme helped them to look
What they think the programme could have done more of to assist them?

13. Process We are collecting data:
Which programmes and providers and which learners (plus contact details)
This has been a challenge but working hard to get these
Plan to send tracer to all sectors that have not completed a tracer ( s, calls, SMS)
Then do sample of qualitative interviews