1. The “Tools of the Trade”: An Overview of Diagnostic and Assessment Instruments December 1, 2003 Francesca Recanatini, WBI www.worldbank.org/wbi/governance

2. Outline of the Session A working framework to select among tools Conceptual design Empirical tools Implementation process Sampling and Field work Analysis and use of the data A few country-specific illustrations

3. Objective Which are the key elements for a governance assessment? Which empirical tools and approaches are already available? How can we select among them? How can such assessments be used for policy purpose? How to choose among governance tools?

4. Governance assessment: one or many approaches? The characteristics of a governance assessment are a function of the objective of the assessment

5. Key starting points 1. What is the purpose of the assessment? Research and analysis Awareness raising Policy and Action planning Capacity building Monitoring

6. Key starting points 2. What is the focus of the assessment? Governance as a whole Corruption Performance of a specific agency/sector Quality of a specific public service delivered

7. Suppose we have determined…. The final purpose of the assessment The focus of the assessment What next?

8. An example – Peru 2002 Issue: the government wanted to monitor progress in terms of Transparency of public administration activities Civil society “ participation ” and voice Quality of public services

9. Peru 2002, cont. Purpose of assessment: monitoring Focus of the assessment: Transparency Citizens ’ “ Participation ” and Voice Quality of public services What next?

10. Existing Empirical Tools BEEPS IGR Public Official surveys PET QSDS Score Cards Investment Climate Surveys EC Audits PER CFAA CPAR GAC Case Studies HIPC Exp. Tracking ROSC www.worldbank.org/wbi/governance/assessing

11. Which are the key elements of a Governance Assessment? Four dimensions: Conceptual Empirical Process / Capacity Building Analytical and Policy

12. Conceptual dimension Clear definition of the variable we focus on and its manifestations Translation of the definition into observable and measurable components Selection of methodological approach Understanding of the links between governance and Performance outcomes Development outcomes

13. Linking the Tools to the Blueprint PER HIPC E.T. ROSC CPAR EC Audits CFAAIGR & GAC & Governance Cross- Country Ind. BEEPS & INVEST. CLIMATE SCORE CARDS QSDS Public Official Surveys PETs

14. Examples of diagnostic tools are widespread … and many overlap Public Expenditure Instruments Public Expenditure Reviews (PER) Fiscal ROSC Country Procurement Assm’t Reviews (CPAR) HIPC Expenditure Tracking Assessments European Commission (EC) Audits Governance Surveys (e.g., IGRs, Anti- Corruption, Public Officials, Household Incidence, Poverty, etc.) Country Financial Accountability Assessments (CFAA) Policy & Expenditure Analysis Budget Formulation & Expenditure Programming Monitoring & Reporting Budget Execution Oversight Illustrative

15. Policy framework Govt. program PRSP Sector strategies etc… Budget allocation Outturn Timely disbursements in accordance with budgeted allocations OutputsImpactOutcomes - Poor institutions of budget management - Political economy factors Nontransparent process - Poor reporting on execution - High level of aggregation - Discretion in allocation Weak accountability - poor service delivery Difficult to assess - Household behavior - Social institutions PUBLIC EXPENDITURE TRACKING AND SERVICE DELIVERY SURVEYS Unclear policy framework Weak management information systems - limited coverage - poor data quality - late and scattered reporting New Concerns

16. Conceptual dimension, cont. Finding answers may require single or multiple methods and data forms The methodological approach can be a combination of different methods (for example, qualitative, quantitative or mixed) To each method corresponds a set of empirical tools that we can use Data can also be qualitative and/or quantitative For more information on alternative methods www.worldbank.org/wbi/governance/assessing

17. Examples of Existing Empirical Tools Qualitative method Budget use monitoring Video Observations Judicial Investigations Quantitative method Investment Climate Surveys QSDS Public official surveys PETs Mixed method Governance Diagnostic Surveys Score Card approach PER

18. Empirical dimension Focus on institutions vs. individuals Experiential vs. perception data One vs. many types of respondents Standard vs. customized empirical tools Definition of sample and field work details Open end vs. close end questions

19. Citizens Government Officials Enterprises Civil Society Private Sector The State Linking the Tools to the Respondents PET QSDS PER CFAA CPAR Score cardsGAC IGR BEEPS INV. CL.

20. Examples of variables measured USERS/HOUSEHOLDS Quality of specific public services Cost and time to obtain a service Information available on basic rights Quality of public agencies Experience with inappropriate procedures and behavior

21. Examples of variables measured ENTERPRISES Quality of specific services and procedures Cost and time to comply with permits and licenses Information available on basic rights Quality of public agencies Experience with inappropriate procedures and behavior

22. Examples of variables measured PUBLIC OFFICIALS Quality of the rules and procedures Transparency of budget and employment decisions Information available on procedures Quality of management Experience with inappropriate procedures and behavior within their office

23. Process / Capacity Building dimension To increase impact and sustainability: Consultative and participatory approach to discuss purpose, use and features of the assessment Engage local NGOs and academic institutions to adapt/revise tools Public dissemination of results Joint design of policy recommendations

24. Challenge: poor governance and corruption 1. Establishment of Steering Committee 2. Diagnostic surveys + analysis 3. Draft of the NAS 4. Public dissemination + discussion 5. Revision of the NAS 6. Implementation by Government 7. Monitoring and Evaluation of NAS WBI Technical Assistance Key Partnership: Government + Civil Society Country Implemented A few Illustrations Guatemala Highly fragmented civil society Joint effort to build consensus Surveys to be launched in Dec. Sierra Leone Strong commitment (civil society, state, donors) => surveys + report completed. Results use for draft strategy and reform projects Honduras CNA: report and strategy to newly elected gov (January 2001); integration of strategy in the 2002- 2006 government plan

25. Analytical and Policy dimension Distill key links between manifestations of governance and: Quality of services Growth Specific characteristics of public sector Results could be used as one input for policy purpose

26. Key dimensions for analysis 1.Identify both weak institutions (in need of reform) and strong institutions (example of good governance) 2.Unbundle corruption by type – administrative, capture of the state, bidding, theft of goods and public resources, purchase of licenses and regulations

27. Key dimensions… Cont. 3.Assess the cost of each type of corruption on different groups of stakeholders 4.Identify key determinants of good governance 5.Develop policy recommendations

28. In sum, how to select among instruments? PER? Score cards? BEEPS? IGR? PET? QSDS? CFAA? CPAR? GAC? Case study? …..? Pub. Officials? HIPC Ex.Tr.? EC Audits? ROSC?

29. Governance Assessment Analysis & use Empirical tools & sample Conceptual dimension Implementation process Use a working framework….

30. ..and remember the key dimensions for a ‘tool comparison’ Conceptual framework Approach Objective Subjective Measuring precision Cost effectiveness

31. Key dimensions, cont. Comparability of the data Across countries Over time Across empirical tools Final user of the data Different agencies Different stakeholders Type of respondent

32. Key dimensions, cont. Effectiveness for judicial actions Ability to identify general challenges Ability to identify priorities for reform Quality of information on: Specific institutional dimension Specific subject

33. Peru 2002 Purpose of assessment: monitoring Final users: government and civil society Key feature: Comparability across time Ability to identify progresses Type of information needed: agency- specific Approach: objective, and based on citizen ’ s feedback

34. Peru 2002 Conceptual dimension Transparency in the management of resources Quality of basic health and education services Quality of complaint and feedback mechanisms Empirical Tool Score card/Questionnaire to households Focus on agency-specific information Objective, experiential data Close-end questions

35. Peru 2002 Process/Capacity building: Partnership between WBI and with National Statistical Office on methodological issues Data and results publicly available Analytical dimension Monitoring of indices ’ performance over time Link between indices of performance and measures of poverty

36. Peru 2002 – Decisions taken To develop the following yearly indicators: Index of transparency and civil society participation Index of quality of public services To focus on households/users only To promote a partnership between the National Statistical Agency and citizens

37. Vulnerabilities of Corruption Reporting: Complaint Mechanisms (as reported by public officials; various countries, 1999-2001)

38. Extent of corruption (Various countries, 2001)

39. The thin lines represent margins of error (or 95% confidence intervals) for each Corruption increases inequality

40. Bribes and quality of services (based on public officials’ response, a Latin American country, 2001)

41. Corruption Restricts Access to Medical Services: Discouraged Poor Users Due to Bribes (as reported by public service users, 2001) Note: The figure shows the percentage of discouraged users not using medical service because a bribe is too high.

42. Sierra Leone Roads Transport Authority Corruption imposes barriers to households to access basic social services

43. SouthNorthEastWestWHOLE Prov. AreaCOUNTRY Corruption in budget3543483940 Overall corruption2232353332 Corruption in public contracts1835293330 Corruption in personnel (2)3944555349 Accessibility for poor8574877478 Audit Mechanisms55596658 Enforcement of rules7067807371 Politicization2134223432 Quality of rules62 706163 Resources5451475552 Transparency5155535155 Citizen voice70596566 Meritocracy6665706968 Governance and corruption indicators by province

44. EnforcementCitizenWage Auditof rulesResourcesTransparencyVoiceSatisfaction Presidential Affairs527451555917 Ministry of Finance576852546025 Quasi-independent institutions698541637832 Ministry of Trade and Industry648460596319 Ministry of Internal Affairs60765761760 Ministry of Energy and Power and NPA729262616925 Ministry of Foreign Affairs & International Cooperation50 74545030 Ministry of Development & Economic Planning547269575825 Ministry of Social Welfare, Gender & Children's Affairs597153565813 Ministry of Health & Sanitation547153556521 Ministry of Information and Broadcasting556851 6018 Ministry of Education, Science and Technology546951556726 Ministry of Youth and Sports567650527225 Ministry of Labor and Industrial Relations617148566425 Ministry of Agriculture and Food Security606946586616 Ministry of Local Government & Municipal and district councils587254576315 Ministry of Transport & Communications647746576631 Ministry of Works and Public Maintenance53675253685 Ministry of Lands, Housing & Country Planning 54575613 Ministry of Mineral Resources60676855 21 House of Parliament545341595013 Office of the Attorney General, Minister of Justice and Director of Public Prosecutions 607549556725 Traffic Police and Sierra Leone Road Transport Authority417464546334 Supreme Courts647855567243 Local Courts616841537736 SALWACO/GVWC and SIERRATEL567353575835 SALPOST616646546028 Sierra Leone Ports Authority536656545942 SL National Tourist Board and SL Standards Bureau44 47495163 Police and Prisons596857536625 Post-conflict institutions607147546444 WHOLE COUNTRY587152556625 Governance and Corruption Indicators by Institution

45. WBI Governance on the Web About Governance Diagnostics and Statistical Capacity Building: http://www.worldbank.org/wbi/governance/capacitybuild/about.html http://www.worldbank.org/wbi/governance/capacitybuild/about.html Governance Diagnostic Surveys Country Sites: http://www.worldbank.org/wbi/governance/capacitybuild/d-surveys.html http://www.worldbank.org/wbi/governance/capacitybuild/d-surveys.html Worldwide Governance Indicators 1996-2002: http://www.worldbank.org/wbi/governance/govdata2002/index.html http://www.worldbank.org/wbi/governance/govdata2002/index.html The Business Environment and Enterprise Performance Survey (BEEPS) 1999-2000: http://info.worldbank.org/governance/beeps/http://info.worldbank.org/governance/beeps/ The Business Environment and Enterprise Performance Survey (BEEPS II) 2002: http://info.worldbank.org/governance/beeps2002/http://info.worldbank.org/governance/beeps2002/ Courses and Surveys: Governance Diagnostic Capacity Building: http://www.worldbank.org/wbi/governance/capacitybuild/courses.html http://www.worldbank.org/wbi/governance/capacitybuild/courses.html Step by Step Guide to Governance Diagnostic Empirical Tools Implementation: http://www.worldbank.org/wbi/governance/capacitybuild/diagnostics.html http://www.worldbank.org/wbi/governance/capacitybuild/diagnostics.html