Overview of evaluation of SME policy – Why and How

Why evaluate? Evaluation provides – Valuable evidence for developing schemes to make the most of any investment – It provides a validation of the benefits of a scheme and its value – good news – It provides hard evidence which can be used to make a case for continuing or developing a scheme in budget discussions – It enables inter-scheme, inter-regional and international comparisons

Types of SME policy evaluation TypeTitleCommentsData needed 1 Measuring take-up Provides an indication of popularity but no real idea of outcomes or impact Scheme management data 2 Recipient evaluation Popularity and recipients' idea of the usefulness or value of the scheme. Very often informal or unstructured.Recipient data 3 Subjective assessment Subjective assessments of the scheme by recipients. Often categorical, less frequently numericRecipient data 4 Control group Impacts measured relative to a 'typical' control group of potential recipients of the scheme Recipient and control group data 5 Matched control group Impacts measured relative to a 'matched' control group similar to recipients in terms of some characteristics (e.g. size, sector) Recipient and control group data 6 Econometric studies Impacts estimated using multivariate econometric or statistical approaches and allowing for sample selection Survey data - recipients and non- recipients 7 Experimental Approaches Impacts estimated using random allocation to treatment and control groups or alternative treatment groups Control and treatment group data

Type 2 – Quantitative monitoring Aim to profile operational aspects of the scheme and provide firm list for later impact analysis Key questions: – Who applied for the scheme? – How many were funded? Went ahead? – How long did approval take? – Were all the funds allocated? Responsibility of delivering agent to collect and data management should be part of the service contract Key evaluation failure is lack of good administrative data on scheme

Type 3 – Qualitative monitoring Aim to interpret and analyse admin data plus perhaps some ‘key informant’ interviews. Possible questions: – Did projects finish as expected? – Did support go to the expected types of firms? – Where are these firms located? What industries? – Were applications processed fast enough? Might be seen as ‘interim evaluation’ and best if independently done

Types 4 and 5 – Control group comparisons Compares performance of recipients and a group of similar firms/individuals matched on some quantitative dimension, e.g. growth, exporting etc. Difference in performance is said to be due to the effect of the scheme. Data could be survey or business register type data. Advantages – Relatively simple methodology to apply – Provides ‘hard’ data on impacts not self-assessment Disadvantages – Often difficult to construct relevant control group – Requires information on participants and controls – Costly sometimes – Does not control for self selection into recipient group (see example below) Operational issues – Analysis should be undertaken independently commissioned by sponsoring department – Key issue is design of control group and needs careful thought

Type 6 – Econometric approaches Typically based on a survey of recipients and non-recipients. How did performance changes in recipients compare to that of similar firms allowing for firm characteristics and selection bias? Uses regression models to identify policy impact on performance controlling for selection bias Advantages – Seen as ‘best practice’ methodology – Can control for firm/individual characteristics – Can control for selection bias Disadvantages – Costly, complicated and difficult to understand Operational aspects – Analysis should be undertaken independently commissioned by sponsoring department – Requires survey of recipients and non-recipients so costs at least double self-assessment (Type 1) – Telephone interviews can be very cost effective (CATI)

Comments on the LDC schemes 1/6

KORET Impressive scheme, key effects social+economic value for assisted women Level 2 evaluation: self-assessment before and after – appropriate approach, since appropriate control group diffucult to identify and approach – counterfactual is status quo – little added value in control group approach Information from existing evaluation could be used to develop NPV-type evaluation (scheme value) But this would underestimate the programme‘s effect – strong social impact and local multipliers Consider strategic added value

Business Centres 2 alternative treatments: – virtual incubator (networking) – physical incubator Compare relative growth of two groups of entrepreneurs, control for firm capabilities Small samples suggest to analyse all 5 centres together Compare contribution of different types of support activities of LDCs Complex evaluation with external control group not worth doing Outcome: also consider investment

Initiating a Business Current evaluation is a good approach – matched control group: applicants decided not to go forward (level 4 evaluation) – better controlling for selection bias and unobserved characteristics relevant for decision to start a business, and the business‘ success – collect a bit more information on the applicants aspiration qualification/skills family/community background prior employment record – use selection-control model (level 6 evaluation) – outcome variables: survival, profitability, enter employment – impact period: 18 or 24 months

Ex-Post Evaluation of the Consulting Programme? not clear whether it is worth conducting an evaluation difficult to identify a sensible control group because of focus on firms in crisis Option 1: Effects of alternative treatments – exploit existing monitoring data – estimate effects of different types of consulting support while controlling for firms‘ initial status (= result of diagnosis?) Option 2: Test on performance specifities of target group: – identify sector/size/age/regional distribution of firms receiving consulting – compare performance – differentiate by the type of consulting support provided

Evaluating the Consultancy Programme Develop Logic Models for each initiative: – output of funding activities – short-term, medium-term, long-term outcome – contingency factors Etablish a integrated monitoring system – firms approaching MAOF: motivation, source of information – results of diagnosis – surveying firms on attitudes, behaviour, output at three points in time: at diagnosis, post-consultancy, after 2 years (potentially as part of another diagnosis) – put all into a database

Early, formative evaluation – sustainability effects of the new 2-year approach – effects on attitudes and behaviour of different types of consultancy Impact evaluation – establish a control group: match several „twins“ for each funded firm (e.g. from D&B, business register) – short surveys of control group at beginning and end of 2-year period (past performance, information about MAOF) – displacement and multiplier effects (through second control group survey) – net effect: NPV based on employment differences to control group (survival and growth) Evaluating the Consultancy Programme

Now something speculative... Randomisation of allocating consulting support a) firms can choose between – consulting service based on diagnosis (and having to pay a part of consulting costs) – getting a consulting service assigned randomly (and not having to pay) b) some firms get a second consultancy – firms may choose the subject – they will have to pay Evaluating the Consultancy Programme