Application and Development Stephanie Trapnell, PRMPS April 29, 2010 AGI Initiative

Overview  What is, and is not, an AGI?  AGIs on different governance dimensions  Application and development of AGIs

What are AGIs? Design Organizational Capacities Governance system performance Exogenous factors

InputsOutputs Design (Institutional Arrangements) Organizational Capacity Governance System Performance Outcomes The Missing Middle Resources invested in projects to deliver its outputs. Examples: Funding, contracts, materials Goods and services produced by the project. Examples: Surveys and Trainings conducted, Laws revised, Agencies established Desired state of well-being -- a set of conditions, experiences or behaviors – that is the goal for change or improvement. Examples: Maternal/infant mortality rates, Standardized test scores of K-8 students Actionable Governance Indicators (AGIs) focus on specific and narrowly-defined aspects of governance.

What are AGIs?  AGIs provide information on the discrete elements of governance reforms. Specific Narrowly-defined  AGIs reflect actionability, i.e., they establish the link between actions and indicator values.

What AGIs cannot do….  AGIs do not provide a comprehensive picture of governance in a country.  AGIs do not provide a general picture of the state of a particular governance system, i.e., business regulation, freedom of information, etc.  Individual AGIs are not used for ranking countries.  Except in cases of aggregation!

Revising targets into AGIs TargetAGI Extension services reaching 10% of commercial farmers % of extension services reaching commercial farmers Base pay is at least 75% of real pay, up from 10% Percentage of base pay that is real pay Follow-up on at least 3 audits where a prima facie case is apparent Number of audits conducted where a prima facie case is apparent

Clearly defined?  Improved participatory budgeting in 30 poor municipalities.  Capacity of authorities in transport management and project implementation strengthened  Greater use of flexibility in bankruptcy law

10 Political Accountability Political competition, broad-based political parties Transparency & regulation of party financing Disclosure of parliamentary votes Checks & Balances Independent, effective judiciary Legislative oversight (PACs, PECs) Independent oversight institutions (SAI) Global initiatives: UN, OECD Convention, anti- money laundering Citizens/Firms Decentralization and Local Participation Decentralization with accountability Community Driven Development (CDD) Oversight by parent-teacher associations & user groups Beneficiary participation in projects Civil Society & Media Freedom of press, FOI Civil society watchdogs Report cards, client surveys Private Sector Interface Streamlined regulation Public-private dialogue Extractive Industry Transparency Corporate governance Collective business associations Effective Public Sector Management Ethical leadership: asset declaration, conflict of interest rules Cross-cutting public management systems: meritocracy, public finance, procurement Service delivery and regulatory agencies in sectors Governance Dimensions

11 Political Accountability Political competition, broad-based political parties Transparency & regulation of party financing Disclosure of parliamentary votes Citizens/Firms Public Accountability Mechanisms Global Integrity Freedom House Afro/Latino/AsiaBarometer Different Dimensions of Good Governance Are Measured Differently (examples)

12 Different Dimensions of Good Governance Are Measured Differently (examples) Doing Business sub-indicators Business Environment and Enterprise Performance Surveys (BEEPS) Investment Climate Assessment (ICA) Business Risk Service Transparency International (TI) index Global Competitiveness index Citizens/Firms Civil Society & Media Freedom of press, FOI Civil society watchdogs Report cards, client surveys Private Sector Interface Streamlined regulation Public-private dialogue Extractive Industry Transparency Corporate governance Collective business associations Public Accountability Mechanisms (PAM) Human rights database Reporters Without Borders Media sustainability index

13 Public Expenditure Review (PER) Public Expenditure Tracking Surveys (PETS) Public Expenditure and Financial Accountability (PEFA) Quality of Service Delivery Surveys (QSDS) Score cards Decentralization and Local Participation Decentralization with accountability Community Driven Development (CDD) Oversight by parent-teacher associations & user groups Beneficiary participation in projects Citizens/Firms Different Dimensions of Good Governance Are Measured Differently (examples)

14 Judicial/Rule of Law indicators PEFA TI index Global Integrity indicators Open Budget Initiative Checks & Balances Independent, effective judiciary Legislative oversight (PACs, PECs) Independent oversight institutions (SAI) Global initiatives: UN, OECD Convention, anti- money laundering Citizens/Firms Different Dimensions of Good Governance Are Measured Differently (examples)

15 Effective Public Sector Management Ethical leadership: asset declaration, conflict of interest rules Cross-cutting public management systems: meritocracy, public finance, procurement Service delivery and regulatory agencies in sectors Public Accountability Mechanisms (PAM) PSG country-specific projects Score cards PEFA Global Integrity index OECD procurement indicators Different Dimensions of Good Governance Are Measured Differently (examples)

AGI Initiative [WB internal] [external]

Instruments (diagnostics, data collection) Guidance materials Data (collection and dissemination) Analyses (papers, graphs, tables, etc.) Tools One stop shop access to existing AGI data (WB and external) Data analysis platform Data AGI Product Lines

Incorporating AGIs into projects a Established Data Sources b Established Data Collection Instruments c Development of New AGIs

Established Data Sources [WB internal] [external]

AGI Sources 20 Content Summary: 6 thematic areas, 1182 indicators 13 sources Note: The total includes the main indicators & sub-indicators.

Data Sources on AGI Data Portal  AfroBarometer – Governance in Social Sectors (AFROBAR)  Business Environment & Enterprise Performance Surveys (BEEPS)  Country Policy and Institutional Assessments (CPIA)  Doing Business (DB)  Enterprise Surveys (ES)  Global Integrity Index (GI)  Human Resource Management (HRM)  International Budget Practices and Procedures Survey (IBPP)  Open Budget Index (OB)  Press Freedom Index by Reporters without Borders (RSF)  Public Accountability Mechanisms (PAM)  Public Expenditure & Financial Accountability (PEFA)  Statistical Capacity (SC)

Thematic groups of AGIs  Public Sector Management Human Resource Management Public Finance Management Procurement Service Delivery Statistical Capacity  Political Accountability Electoral Competition Branches of Government Integrity Systems Budget Oversight  Checks and Balances Branches of Government Oversight Institutions Redress Institutions Other  Civil Society and Media Civil Society Organizations Media Press Freedom Index Public Access to Information Open Budget Public Access to Key Fiscal Information  Private Sector Interface Permits and licenses Regulation of Entry Property Rights Corruption in Business-Government Transactions Infrastructure Trade Taxes  Decentralization and Local Participation Transparency of inter-governmental fiscal relations Oversight of aggregate fiscal risk from other public sector entities

AGI Dataset Snapshots CategoryQuestion being answered DescriptionWhat is the data source? SourceIs the data located on an external website? Number of indicators on the AGI Data Portal How many AGIs are available on the AGI Data Portal? Country coverageWhat is the breakdown of country coverage by geographic region and income? YearsIn which years has the data been collected? PeriodicityHow often is the data collected? Scope of indicatorsAre the AGIs aggregated? Type of dataIs the data based on facts or perceptions? Object of measurementWhat is the data measuring? Reliability of dataHow is the reliability of the data established? Changes in methodology/coverage Has the instrument, data collection process, or coverage of countries changed? LimitationsWhat are the limitations of the data regarding applicability and interpretation?

Guidance on Indicator Selection What is the indicator measuring? If not AGI, what are the characteristics of the underlying indicators? How are the underlying indicators weighted? What is the object of measurement?

Are the indicators based on facts or perception? How was the data collected? By whom? When? How many experts or expert assessments were used for each indicator? How many households or individuals were surveyed? Was the survey representative of the population as a whole? Were the questions direct and easy to understand by respondents? Guidance on Indicator Selection

Are comparisons over time appropriate? Did the composition of countries change? Did the composition of data sources change? Did the methodology change? Did the weighting of the indicators change? Guidance on Indicator Selection

Are cross-country comparisons appropriate? Is country coverage large enough to allow country rankings? Are data sources large enough to be representative? Is the indicator too context-specific to be compared across countries? Guidance on Indicator Selection

Established Data Collection Instruments  Expansion of datasets in established AGI sources  Human Resource Management  Public Accountability Mechanisms  Public Expenditure Tracking Surveys/ Quality of Service Delivery (PETS/QSDS)  Teacher Policies around the World

Considerations when using established instruments Limitations  Quality of data  Reliability of data Consistency  Sampling methodology  Survey instruments Method of Data Collection Schedule Data collection agent Cost Repeated data collection

Developing new AGIs: Considerations AGIs Conceptual clarity Exogenous factors Sustainability Availability of data ReliabilityValidityRelevance

Conceptual Clarity  Narrowly-defined indicators that reflect empirical research Making Services Work: Service Delivery in the Human Development Sectors Forestry Outcomes: An Analytical Framework for Governance Reforms Public Accountability Mechanisms (PAM) Methodology Paper

Relevance and Validity  Relevance Is the indicator capturing data that is relevant to the goals of the project? Does the data being captured help to understand the design, capacity or performance of a system under study?  Validity (Construct) Is the indicator measuring what it claims to be measuring, or is it measuring something else?

Reliability  Inter-rater or Inter-observer ○ Two or more independent experts provide data  Parallel forms & Internal consistency ○ Different questions assess the same construct  Test-Retest ○ Used for items that are stable over time, such as personality  Statistical tests for reliability  Average Inter-item Correlation, Average Item-total Correlation, Split-Half Reliability, Cronbach's Alpha

Other considerations  Availability of data Can the data be captured, given the resources of the project and the context?  Sustainability Should data collection be conducted regularly for the data to be useful?  Exogenous factors Cannot always be accounted for in the design of the indicator

Red flags on Aggregation  Aggregation requires an understanding of the relative importance of each AGI to the concept being measured in the aggregated indicator. ○ What are the underlying weights of the AGIs? Equal weighting (averages) or unequal weighting (assigning more importance to some AGIs but not others). This understanding should be rooted in empirical research.