1. Mixing methods to assess the impact of private sector support: experiences from the Dutch PRIME-programme
Giel Ton
Thursday 19th October
13.00 – 14.30
IDS Convening Space

2. Outline Evaluation of impacts: evaluation questions intervention logic
Mixing methods:
span of direct influence (attribution/contribution)
scan validity threats to likely conclusions from core method
Application in practice and results:
Cohort design (yearly online survey)
Contribution score (treatment intensity)
Comparative case studies
Overall synthesis: an evidence-based contribution narrative

3. Part one – evaluation of impacts

4. Approaches to impact evaluation

6. Approaches to impact evaluation
Anticipate generalisation = the use of the conclusions = replicability / scaling
Implication: look to both the effect (internal validity) and the contextual characteristics (external validity)
Intervention logic: ‘theory in action’ – verify/test to improve the theory in action
TBE – focus on key assumptions

7. THEORY-BASED EVALUATION
THE HOLY GRAIL:
THEORY-BASED EVALUATION

8. Define the intervention logic

9. PSD interventions
CBI (RVO - Dutch Enterprise Agency) - support imports form development countries to the European market through business training, market information, international fairs, sector development studies, etc.
PUM Senior experts - support SMEs in business management, investment planning and business links through experienced former managers (peer-to-peer) through two week on-site visits.

10. Theory of change of CBI

11. Part two – Mixing methods

12. Methods in impact evaluation

13. Methods in impact evaluation
Span of direct influence: where does attribution become an unattainable goal

14. Policy makers’ interest
Policy interest
in effectiveness

15. Implementing agency’s influence
Effect size
Attribution
Contribution

16. Research method limitations
Net-effects impact research
(attribution)

17. Research methods limitations
Monitoring (contribution approach)
Net-effects impact research
(attribution)

18. Dutch government guidelines (DGIS 2017)

19. Methods in impact evaluation
Validity threats: ‘perspective of the critical outsider towards the conclusions/inferences

20. Four types of validity threats to conclusions (Shadish, W. R. , T. D
Four types of validity threats to conclusions (Shadish, W. R., T. D. Cook and D. T. Campbell (2002). Experimental and Quasi-Experimental Designs for Generalized Causal Inference, Houghton Mifflin Co. Boston, MA.)
Statistical conclusion validity: how are inferences drawn from the data-set?
comply with proven methods and correct interpretation of statistical analysis.
Internal validity: how is causality attributed?
What is the mechanism how interventions contribute to the observed change.
Construct validity: how are generalizations made from the categories used in the evaluation to broader units of representation?
Careful and precise definitions and concepts.
External validity: to what extent are the findings generalizable?
Define the ‘generalization domain’.

21. EXPLORING VALIDITY THREATS TO THE CORE METHOD
TYPE OF VALIDITY THREAT
MAIN THREAT
SUGGESTED ADDITIONAL DESIGN ELEMENTS
RESULT/OBSERVATION
STATISTICAL CONCLUSION
INTERNAL
CONSTRUCT
EXTERNAL
Based on the handbook of Shadish Cook and Campbell – Experimental and Quasi experimental methods

22. Core method: cohort study / Rolling baseline

23. Econometric methods will check for selection bias between the cohorts.
TYPE OF VALIDITY THREAT
MAIN THREAT
INCORPORATED ADDITIONAL DESIGN ELEMENTS
STATISTICAL CONCLUSION
Differences between cohorts averages have to controlled for but are not known in advance and not always observable.
Econometric methods will check for selection bias between the cohorts.
INTERNAL
Correlation is not causation
The self-selection of businesses into the programme makes that conclusions only apply to firms similar to the supported ones.
A qualitative comparative case study approach (N=6) to provide information on the processes and factors that explain the effectiveness of the support.
The characteristics of the treated firms in these case studies are compared with the averages in the sector
CONSTRUCT
The treatment is very diverse
A factor/cluster analysis to make a typology of similar interventions (sector, region, type of support provided)
EXTERNAL
Country and sector context matters (e.g. economic crisis/growth periods, political situation, policy developments)
Explorative multivariate analysis of the final data-set to identify heterogeneity of effects.
Careful conclusions (restricted generalisation domain of the causal inferences)

24. Part three – Application in practice and results

25. Cohort study / Rolling baseline

26. Inter-cohort comparison of characteristics
Difference
Performance
Sales, logarithms
12.62
12.46
12.35
-0.24**
-0.08
Employment, logarithms
3.44
3.37
3.38
-0.06
0.01
Type of firm
Cooperative
0.02
0.01**
0.01***
Individual
0.31
0.32
0.30
0.00
-0.02
Joint Stock
0.09
0.11
0.06
-0.03*
-0.05***
Limited Liability
0.54
0.51
0.56
0.03
Other
0.05
0.04
Region
Africa
0.37
0.38
-0.01
Asia
0.29
0.27
Europe
0.18
0.17
0.14
-0.05**
-0.04
Latin America
0.16
Poverty status
Least Developed
0.34
0.06**
Lower Middle Income
0.47
0.42
0.43
-0.03
Other Low Income
Upper Middle Income
0.24
0.20
Sector
Agriculture
Manufacturing
0.50
0.46
-0.05*
Service
0.05*
Female owner
-0.06***
-0.17***
Received one mission
0.12
-0.12***
Slightly smaller firms
Similar profile
Less in Eastern Europe
Less female-headed firms
More intensive support

27. PRIME online survey (Qualtrics)

28. Effects on intermediate outcomes

29. Question 1 - level of practices: Construct validity check
Objective business practices
Subjective business practices
Keeping financial records via other
Financial management
Verifying financial statements through control outside the company or external control by a certified auditor
Written marketing plan
Marketing techniques to increase sales of your product or service
Website
Promotion material
All employees have written labor contracts
Ways to retain, motivate and train employees
Systems and policies in place to monitor and ensure worker safety.
System to learn about the clients' opinions on products and services.
Leading, planning and organizing the business
Documented quality assurance system of the business processes.
Quality requirements of (inter)national buyers
System in place to monitor effects  on the environment.
Effects of the business on the environment

30. Construct validity test
Horizontal and vertical axis are self assessed practices and proxy indicators for business practices levels respectively

31. Effects on intermediate outcomes

32. Question 2 & 3: Change & Influence PRIME Contribution score
An indicator for CBI contribution: How did your company’s knowledge change over the past 12 months? + How did CBI influence the change?
How did your company’s knowledge change over the past 12 months?
Did CBI influence this change?
CBI contribution score (0-8)
CBI Contribution
index (0-100%)
Strong decrease
No effect
Decrease
No change
Increase
Strong increase
Very little 1 13 2 25
Some 3 38 4 50
Quite a bit 5 63 6 75
A lot 7 88 8
100
No contribution
50 contribution
Full contribution

35. Regressions with Contribution Score as intensity of treatment

36. Heterogeneity analysis for CBI - export promotion
(Estimates for) the effect of firm size, region, income group, access to finance and sector on CBI contribution score for firm practices
In practices transfer CBI is more successful in SSA then in Asia and Latin American regions
Less in the upper middle countries
In agriculture sector, the success is lower than in services.
Effect on contribution score

37. Heterogeneity analysis for PUM - management counceling
(Estimates for) the effect of firm size, region, income group, access to finance and sector on PUM contribution score for firm practices
PUM is more successful in in LDC and Low income countries,
In the Services and Trade sector; in the agriculture the success is lower.
and higher in firms with female owners.
Effect on contribution score

38. Part four – Comparative case studies

39. Comparative case studies
Two missions (2014/15 and 2016/17) of 10 days each to interview SMEs, support organisations, sector policy, etc.
Bolivia - Tourism sector
Peru - High value agriculture
Uganda - Coffee
Bangladesh - IT-sector
Myanmar - textile
Indonesia - Fishery & aquaculture

41. Case study analysis plan

42. Part five – overall synthesis

43. Overall synthesis = assembling the evidence behind the contribution narrative

44. Overall synthesis = assembling the evidence behind the contribution narrative
Activities improve knowledge & practices (online survey)
The promoted ‘ good practices’ are considered to contribute to improved performance of the sector (case studies)
Better SME performance is good for sustainable economic development (literature review, secondary data)

45. Takeaways Try the three-step process for mixing methods
refine evaluation questions based on an intervention logic
anticipate validity threats to the expected type of conclusions
maximise the scope for comparative research and benchmarking
Be aware of the ‘ span of direct influence’ and the limits to attribution
Ask evaluative self-assessment questions along observable indicators
Collate evidence in a contribution narrative to report on development impact (accountability)

46. Thanks! www.primepartnership.nl Former PRIME Research Coordinator
Wageningen Economic Research
Giel Ton
Institute of Development Studies
Centre for Development Impact