1. Making Services Work: Indicators, Assessments and Benchmarking of Governance in Health and Education Dena Ringold, Senior Economist, The World Bank APPAM - Moscow June 28, 2011

2. Outline What is governance and why should it be measured? A framework for measuring governance policies and performance; Measuring dimensions of service delivery; Examples and recent innovations.

3. HD outcomes Improved governance is needed to make services work Quality of social service delivery Governance: Incentives to providers Accountability for results Demand for good governance 3

4. Why Measure Governance? The Shock Factor: Measurement allows for benchmarking; ◦ Both inter- and intra-national; Does it work? Measurement allows for impact evaluation; ◦ Measuring along the results chain to assess what works; Look inside the “black box” of service delivery: ◦ Intermediate service delivery indicators assess the “missing middle”; ◦ Indicators are needed for results-based projects.

5. Governance Policies and Performance Governance has two measurable dimensions: Governance policies, or “rules of the game,” that influence provider behavior; ◦ E.g. human resource policies that establish hiring, firing, and rewarding of teachers and doctors; Governance performance, or the extent to which governance policies work in practice; ◦ E.g do teachers and doctors show up for work? Governance policies and performance influence the quality of service delivery; ◦ E.g quality at the point of service; quality measurement is elusive but there is progress. 5

6. A Measurement Framework Governance Policies HR policies Budget and financial management Information Procurement rules Rules for provider entry Service Delivery Governance performance Quality Outcomes Completion rates Learning outcomes Morbidity Mortality

7. Measuring Policies and Performance Degree of meritocracy in hiring; Provider absenteeism; purchase of positions. Human resources Existence of clear rules for the budget cycle; Leakage. Budget Existence of complaints mechanisms; Numbers of complaints resolved. Information Existence of an essential medicines list; Drugs: stock out rates. Procurement Extent of restrictions on entry of private providers; Number of permits required. Entry

8. Human Resources: SABER-Teachers What is it? ◦ Assessment tool for documenting 10 dimensions of teacher policies.10 dimensions How does it work? ◦ Expert survey based on an extensive questionnaire. ◦ Currently applied in 30 countries including OECD; also at the state-level in India. What about performance? ◦ Absenteeism surveys can measure provider motivation, retention, etc.

9. Budget: ABI in the Philippines What is it? ◦ Expert survey of organizational performance piloted in the Department of Health, based on PEFA. What does it cover? ◦ 6 areas, including budget preparation; execution and reporting; organizational management and accountability; ◦ Includes policy and performance indicators. What are other instruments? ◦ Public Expenditure Tracking Surveys; Public Expenditure Reviews (can look at the whole sector).

10. Information: Global Integrity Pilot What is it? ◦ An experiment to develop indicators of access to information in health and education, including policies and performance. What does it cover? ◦ 4 broad areas of access to information. 4 broad areas What has been learned so far? ◦ Pilot in Macedonia suggests that it is possible to collect this type of information, now expanding to Ukraine and Kenya; ◦ Big gaps between legislation and implementation.

11. Critical Inputs: WHO pharmaceuticals What is it? ◦ Expert survey for assessing governance policies for transparent pharmaceutical supply and delivery. What does it cover? ◦ 8 areas, including registration, licensing, inspections, controls of trials, procurement and distribution; ◦ Applied in 25 developing countries. What about performance? ◦ Tests for counterfeit drugs (USAID quality database); ◦ Stock-out rates (WHO/HAI database).

12. Provider Entry: Africa Health What is it? ◦ Recently published study of the private sector in health in Africa by the IFC. What does it cover? ◦ Survey includes the inclusion of the private sector in the national health information system and regulations governing the quality of private provision.

13. Looking forward There is progress and space for applying tools more systematically, and developing new approaches; Key questions are who should collect data and how often? Technological innovations in data collection are promising (SMS for reporting stock-outs; Indaba software)

14. THANK YOU www.worldbank.org/hdchiefeconomist

15. Absence rates among teachers and health workers Note: Surveys were all fielded in 2002 or 2003. Sources: Chaudhury et al (2006) except for PNG, World Bank (2004) and Zambia, Das et al (2005).

16. Share of school grants and funds for health facilities that actually reached the intended recipient/facility Education Percent Health Percent Ghana 1997/98 5120 Kenya 2004 2062 Madagascar 2002 88-- Peru 2001 (utilities) 70 / 97-- PNG (2001/2002) 72 / 93-- Tanzania 2002-2003 62 59 Uganda 1991-1995/2001 <20 / 80-- Zambia 2001-2007 (discretion/rule) 24 / 90-- Chad 2003 (regional/local health centers) --27/ <1 Ye and Canagarajah (2002) for Ghana; Republic of Kenya (2005) for Kenya; Francken (2003) for Madagascar; Instituto Apoyo and World Bank (2002) for Peru; World (Bank 2004) for PNG; MOF, Government of Tanzania (2005) for Tanzania; Reinikka and Svensson (2005) for Uganda; Das et al. (2002) for Zambia; Wane for Chad (2004).

17. 10 Teacher Policy Areas 1. Requirements for entering and remaining in the teaching profession 2. Initial teacher preparation 3. Recruitment and employment 4. Teacher workloads and autonomy 5. Professional development 6. Compensation: salary and non-salary benefits 7. Retirement rules and benefits 8. Monitoring and evaluation of teacher quality 9. Teacher representation and voice 10. School leadership

18. 4 ‘Buckets’: Access to Information Existence and usability of information ◦ Is the Ministry of Education’s overall budget available to citizens? Redress mechanisms to enforce accountability ◦ Do formal processes exist to receive citizen complaints? Availability of Fiscal/Budget information ◦ Does information exist on whether school/medical facilities receive funds budgeted from state/local governments? Influence of information availability on citizen participation in local decision making ◦ Are citizens aware of basic rights to participate in formal community groups/forums?