1. Good Countries or Good Projects? Micro and Macro Correlates of World Bank Project Performance Cevdet Denizer (Bosphorus University) Daniel Kaufmann (Revenue Watch and Brookings Institution) Aart Kraay (World Bank) IEG Evaluation Week Presentation March 18, 2013

2. Motivation Huge literature on aid effectiveness at two levels: – “macro” level – e.g. does total aid raise aggregate GDP growth? – “micro” level – e.g. evaluations (randomized or otherwise) of individual projects Know much less about the relative importance of project-specific versus country-specific factors in determining project outcomes – “macro” literature uninformative about individual projects – “micro” literature (mostly) does not have cross- country dimension

3. This Paper Uses very large sample of 6000+ World Bank projects since 1980s – crude, but credible, outcome measure for each project based on internal evaluation processes (IEG project success ratings) Match these up with two types of potential correlates of project success: – “macro” country-level variables (easy...) – “micro” project-level variables (hard...but interesting)

4. Preview of Main Results Project-level outcomes vary much more within countries than between countries Limited cross-country average variation in project performance is well-explained by standard “macro” variables Look at variety of “micro” project-level correlates of project-level outcomes – basic project characteristics – early-warning indicators – identity of task team leader – much more to be done here since this is where most of the action is!

5. Project Outcome Data Main evaluation criterion: was project satisfactory/unsatisfactory in reaching its development objective? Three sources of this outcome measure – all projects rated by task team leader in ICR – since 1995 all ICR ratings “validated” by Independent Evaluation Group (IEG) – around 25% of projects selected by IEG for full evaluation (Project Performance Audit Report) Use one evaluation per project, taking most detailed evaluation available

6. Alternative Evaluation Types

7. Many Potential Concerns with Outcome Measure very crude (sat/unsat, or 6 point scale after 1995) – definitely not randomized evaluations! projects assessed relative to development objective only, these are not standardized across projects – different standards for DOs across different sectors? include sector dummies – evolving standards for setting DOs and evaluating them? include sector x approval period dummies include sector x evaluation period dummies – “setting bar low” in difficult countries?

8. Potential Concerns, Cont’d significant self-reporting component – incentives of task managers to give poor ratings? independence of IEG? many steps from effective individual World Bank projects to any macro growth effects of aid Despite these concerns, these ratings seem broadly credible and have advantage of huge country-year- project coverage

9. Setup of Empirical Results Start with universe of 7342 completed projects evaluated since 1983, and construct two subsets based on (i) availability of RHS variables and (ii) units of evaluation ratings – 6569 projects evaluated 1983-2011 (binary outcome variable) – 4191 projects evaluated 1995-2011 (6-point outcome variable) All specifications control for: – potential mean differences across three types of evaluations – evaluation lag (time between evaluation and completion), usually significantly negative

10. “Macro” Correlates of Project Outcomes “Standard” set of country-level variables from literature – Good policy (CPIA) – Shocks (GDP growth) – Democracy (Freedom House) Average each one over life of project – non-trivial decision how to do this, because projects last a long time (median=6 years) alternatives might be initial? final? weighting? separately by year of project life?

11. Results: “Macro” Correlates

12. Generally sensible results in full sample – policies/institutions matter a lot validation of CPIA in PBA – growth matters – no strong evidence that political rights/civil liberties matter

13. Results: From “Macro” to “Micro” Correlates Country-level variables by construction will explain only country-level average variation in project outcomes But, country-level average variation in project outcomes is only 20% of the total variation in project outcomes – based on regression of project outcomes on country dummies, by year – average R-squared is about 0.2 – “macro” correlates explain this 20% reasonably well Points to importance of considering project-level factors (which we do next)

14. Project Outcome Ratings and Country Performance

16. “Micro” Correlates of Project Outcomes, 1 dummy for investment lending (vs DPLs, SALs, etc) three proxies for complexity – “concentration” of project in its major sector – dummy for “repeater” projects, e.g. Botswana Education II, III are repeats, Education I is not – ln(size in dollars) project length (years from approval to evaluation) preparation and supervision costs as share of total project size

17. Results: Basic Project Characteristics

18. Investment projects do slightly better Mixed results on complexity – projects more concentrated in one sector do worse?? – “repeater” projects don’t do better? – larger projects do worse Length, preparation (and especially supervision) costs negatively correlated with outcomes – big-time endogeneity problem – e.g. “difficult” projects require more preparation, supervision, take longer – more on this later (and in paper)

19. “Micro” Correlates of Project Outcomes, 2 Effectiveness delay (time in quarters from approval to first disbursement) “Early-warning” indicators of problem projects from end- of-FY Implementation Status Review (ISR) Reports for each year project is active “problem project” flag – raised if task manager thinks progress towards development objective is unsatisfactory “potential problem” flag – raised if three or more of 12 detailed flags are raised dummy for restructuring (very rare) – dummy=1 if these flags observed in first half of project (only for projects lasting at least four years)

20. Results: Early Warning Indicators

21. 3764 Projects Approval First Half of Implementation Second Half of Implementation Evaluation

22. 3764 Projects 943 Problem Projects 25% 2821 Good Projects 75% Approval First Half of Implementation Second Half of Implementation Evaluation

23. 3764 Projects 943 Problem Projects 25% 592 Problem Projects 63% 351 Good Projects 37% 2821 Good Projects 75% 853 Problem Projects 30% 1968 Good Projects 70% Approval First Half of Implementation Second Half of Implementation Evaluation

24. 3764 Projects 943 Problem Projects 25% 592 Problem Projects 63% 351 Good Projects 37% 2821 Good Projects 75% 853 Problem Projects 30% 1968 Good Projects 70% Approval First Half of Implementation Second Half of Implementation Evaluation 41% Success 81% Success 48% Success 87% Success

25. 3764 Projects 943 Problem Projects 25% 592 Problem Projects 63% 351 Good Projects 37% 2821 Good Projects 75% 853 Problem Projects 30% 1968 Good Projects 70% Approval First Half of Implementation Second Half of Implementation Evaluation 41% Success 81% Success 48% Success 87% Success Overall 71% Success Rate 55% 75%

26. Results: Early Warning Indicators Effectiveness delays are associated with slightly better outcomes Problem Project Flag raised in first half of life of project are highly significantly negative not a mechanical correlation with outcome potential problem flags also significant in AFR Restructurings are positively correlated with outcomes (more so in AFR) Again partial correlations are hard to interpret – e.g. a “difficult” project is more likely to be flagged and is more likely to turn out unsuccessful

27. Role of Unobserved (by us) Project Characteristics Many of the project variables respond endogenously to project characteristics, e.g. – “difficult” projects require more supervision, are more likely to be flagged, and also are more likely to be unsuccessful – creates downward bias in OLS estimates of effects of interventions such as supervision Can’t rely on standard solutions like randomized controlled assignment of Bank inputs (infeasible) or instrumental variables (unjustifiable) Paper has details on alternative approach to quantify likely biases – with reasonable assumptions can retrieve intuitively-plausible positive effects of supervision, flags, etc. on project outcomes – but magnitude hard to pin down precisely

28. Assessing Endogeneity Biases Project outcomes depend on some Bank input such as project supervision: Endogeneity problem means that regressor is correlated with error term, leading to standard omitted variable bias, i.e. ρ<0 (maybe ρ<<0?) Size of bias depends on how strong correlation between regressor and error term is

29. Assessing Endogeneity Biases (or, how to be Bayesian in two easy steps) Step 1: Specify prior beliefs about size of endogeneity problem, i.e. size of ρ – if IEG evaluations perfectly reflect project quality, then ρ 2 is the share of variance in project supervision driven by project “difficulty” how big do you think this is? how sure are you? – summarize prior with various Beta distributions Step 2: Average inferences about β over this prior distribution

30. Alternative Priors for ρ

31. Assessing Endogeneity Bias 2.5 th and 97.5 th percentiles of posterior distribution of β are like a standard confidence interval – For supervision, prior centered on ρ=-0.25 is enough to generate a significant positive effect of supervision – For problem project flag need stronger prior

32. Assessing Endogeneity Bias Three important caveats to keep in mind – prior uncertainty about ρ implies greater posterior uncertainty about β – greater “noise” in IEG assessments means we need larger values of |ρ| to generate positive effects on outcomes – does not mean that better supervision can “fix” bad projects. Consider 1 SD “worse” project: project outcome declines by –σ benefit of more supervision on outcomes is -βρ net effect is –(σ + βρ) is still negative in all specifications

33. Role of Task Team Leaders Task team leader (TTL) is important World Bank “input” into projects We have data on the staff ID number of the TTL: – from final ISR, for 3,925 projects in post-1995 sample publicly available in Project Portal – for each ISR, for 3,187 projects in post-1995 sample use to investigate TTL turnover Explore two practical questions: – How important are TTL fixed effects relative to country fixed effects? – How important is TTL “quality” relative to other correlates of project outcomes?

34. Country Effects vs TTL Effects In order to investigate this, need a sample where there is meaningful variation across countries and TTLs – e.g. if each TTL worked in only one country, can’t separately identify country and TTL effects Restrict attention to sample of 2407 projects where TTL has managed (i) more than one project, and (ii) in more than one country – covers 136 countries and 710 TTLs For projects where we have “time series of TTLs” by ISR within projects, also identify “Initial” TTL, as distinct from “Final” TTL at time of final ISR – look at subset of projects where “Initial” and “Final” TTL are different to separately identify “Initial” and “Final” TTL effects

35. Analysis of Variance Essentially a regression of project outcomes on 136 country dummies and 710 TTL dummies Huge increase in R-squared due to TTL dummies, but is misleading because there are so many dummies – Mean sum of squares gives better indication of relative magnitude of TTL and country effect adjusted for number of dummies

36. How Much Does TTL “Quality” Matter? Proxy for quality of TTL on a given project as average IEG rating on all other projects with same TTL – only for projects with TTLs managing two or more projects – variant 1: define quality as average IEG rating over previous projects managed by same TTL – variant 2: define quality as weighted average (by number of ISRs) of all other projects the TTL was ever responsible for (not just at the end of project) TTL “turnover” is average number of TTLs per ISR – median project lasts six years, has 12 ISRs, and 2 TTLs

37. Results: TTL Quality and Project Outcomes

38. TTL quality is highly significant with economically large effects, e.g. consider move from P25 to P75 of: – TTL Quality: 3.5→4.75, IEG score ↑ by 0.23 – CPIA Score: 3.1→3.6, IEG score ↑ by 0.22 – Alternative quality measures have similarly large effects TTL turnover is highly significant – moving from 2/12 TTLs per ISR to 3/12 TTLs per ISR implies IEG score ↑ by 0.10 – but need to be cautious about endogeneity of TTL turnover – much more to be done here, e.g. to better understand costs and benefits of 3-5-7 rule

39. Results: TTL Quality and Project Outcomes So far have focused on TTL effects – but could very well also be evaluator effects – are there “tough” and “easy” evaluators? – how do they match to TTLs? Two data sources on evaluator identity – anonymized data from IEG on staff who do desk reviews of ICRs, for each project since 1995 – manually (!) collected data on TTL for 1150 Project Performance Audit Reports since 1995 Some evidence of evaluator effects, but: – does not undermine significance of TTL effects – does not survive addition of other controls (likely reflects sectoral specialization of reviewers?)

40. Results: TTL Quality and Project Outcomes Evidence suggests there is a quantitatively-important “human factor” in project outcomes But much more needs to be done: – are there common attributes to TTLs who have a track record of successful projects? – are there endogeneity problems in the “assignment” of TTLs to projects? – do higher levels of management matter? – are there other dimensions, such as counterpart quality, that matter as well?

41. Policy Implications Country-level policies and institutions do matter a lot for project outcomes – don’t throw out baby with bathwater! – (one more) piece of support for donor policies targetting aid to countries with better policy – but at most this can help us with 20% of variation in project outcomes that occurs across countries

42. Policy Implications, Cont’d The 80% of variation in project outcomes within countries challenges us to think hard about how to improve project success within countries, e.g. – why are problem projects hard to turn around, or cancel outright once warning signs emerge? – is there scope for project- as well as country-level aid allocation mechanisms to ensure better outcomes? e.g. what if WB were to allocate some resources to “proposals” submitted by TTLs? – analogous to NSF (or KCP) proposals to obtain research grants – criteria for judging proposals could be tailored to reflect country and TTL characteristics – how can we better learn about the effectiveness of Bank inputs into project outcomes?

43. Pipeline Many more interesting questions to be answered using this kind of project data – some preliminary evidence that projects managed by “decentralized” TTLs located in country of project do better – assembling TTL-VPU assignment data to see if “3- 5-7”-induced TTL turnover matters for project outcomes – working with colleagues at AfDB and AsDB to assemble similar data for their projects – and much more....suggestions welcome!