REAL (working with Singizi) Presentation to NSA 3rd/4th August Impact evaluation REAL (working with Singizi) Presentation to NSA 3rd/4th August

Defining impact evaluation The positive and negative, intended and unintended, direct and indirect, primary and secondary effects produced by an intervention. OECD Development Assistance Committee definition

Extending the definition When understanding impact there is a need to go beyond “what works and for which groups” and answer other equally important and related questions, such as “why do certain programmes work for some groups and not for others”. Martin and Grubb, What works and for whom:A review of OECD countries’ experiences with active labour market policies, Swedish Economic Policy Review, 8 (2001) 9-56

Possible Impact Evaluations Designs Experimental and Quasi Experimental Usually has a counterfactual of some kind (an estimate of what would have happened if the intervention had not occurred) or a particular sort of counterfactual (for example, comparisons with a group who did not receive the intervention) Non-experimental Designs Are impact evaluation designs that do not include a matched comparison group Outcomes and impacts assessed without a conventional counterfactual to address the question

Approach for this study Undertaking a meta-review of impact studies undertaken during the period of NSDSIII Completing a tracer study (where sectors have not undertaken a tracer study) Will not have a control group but instead is a longitudinal study (though possibility of comparing with administrative data available being explored to provide a point of comparison for key issues such as length of time to transition to the labour market where programmes are offered to unemployed)

Rationale for this design selection Its practical and … country-led planning and evaluation (often covers whole country) Has horizontal and vertical partnerships Aim of long-term changes, but a need for short-term results Multiple stakeholders with different perspectives Non linear Many components or services Different theories of change evident across projects Multiple donors and agencies Context is critical and challenge has multiple and contested causes Large, complex Complex interventions May include a number of projects and wider scope Often involves several blueprint approaches Defined objectives but often broader and less precise and harder to measure Often not time-bound Context important multiple donors and national agencies Complicated programs Simple projects Small, simple “blue print” producing standardized product relatively linear Limited number of services Time-bound defined and often small target population Defined objectives Adapted from Michael Bambeger, June 28, 2011 and Singizi Theory of Change and complex programmes

Implications for methodology Need to consider quantitative and qualitative data - Participatory, developmental and utilisation approach – these consider change from the perspective of the people/communities/institutions undergoing the intervention Which can be collected in many various ways, like surveys, focus group discussions, interviews, etc

Tracer focus Programme details – field, level, type Status in programme (if still in – STOP, if dropped out – one set of questions, if completed – another set of questions) Demographics – ace, gender, age, disability etc Entry into programme – motivation, selection, employment status when entering the programme (will route from here)

Tracer focus (continued) If unemployed/pre-employed before the programme – Employment status now and/or education status Extent to which programme enabled you to access this employment (support measures in place and recognition of programme) Whether the programme enabled the learner to succeed in this work (quality and relevance of programme and opportunities that the learner have had access to in the workplace) Possibilities of navigating further pathways of learning or employment

Tracer focus (continued) If employed before the programme Status now - Whether the individual is still in the same workplace Promotion? Increase? New responsibilities? Access to training? Whether the programme enabled this (recognition or new skills – linked to quality or relevance of the programme, etc) Any changes in the workplace because of their increased skill

Tracer focus (continued) If self-employed before the programme Status now - whether still self-employed/employed/studying/unemployed Extent to which the individual has increased skills to run and grow their business (if still self-employed) If programme linked to other support as relevant (finance, market, etc)

Tracer focus (continued) If unemployed now – Whether still looking and if so, where - Whether the programme helped them to look What they think the programme could have done more of to assist them?

Process We are collecting data: Which programmes and providers and which learners (plus contact details) This has been a challenge but working hard to get these Plan to send tracer to all sectors that have not completed a tracer (emails, calls, SMS) Then do sample of qualitative interviews