Presentation on theme: "INNOVATIONS IN GOVERNANCE MEASUREMENT Public Accountability Mechanisms (PAM) Initiative Francesca Recanatini & Stephanie E. Trapnell April 26, 2013 1."— Presentation transcript:
INNOVATIONS IN GOVERNANCE MEASUREMENT Public Accountability Mechanisms (PAM) Initiative Francesca Recanatini & Stephanie E. Trapnell April 26, 2013 1
Outline Introductions Objectives and potential outputs ISPMS, AGI, and behavioral dimensions of measurement Public Accountability Mechanisms De jure and De facto “Big ideas” for today 2
Objectives of the Workshop To discuss innovations in the measurement of “de facto” aspects of governance focusing on: the gaps in existing data coverage, the strengths and weaknesses of recent data collection methodologies, and the sustainability of these efforts. If possible, to identify a “core” set of governance areas and related indicators (both existing and potential) 3
Outputs 4 Menu of governance indicators and efforts Draft note on methodologies Annotated bibliography of sources Technical expert group (TEG)
A proposal and a new way forward ISPMS, AGI, and behavioral impacts 5
The challenge To measure performance of institutions, and in particular, public sector institutions To identify areas for institutional reform To evaluate the impact of change in policies at the country and local level 6
Complementary sets of indicators Measurement of the performance of Public Sector Management (PSM) systems in the middle of the public sector results chain. Measurement of the way in which the government is held to account through political and non- executive institutions and directly by the public. 7 Indicators of the Strength of Public Management Systems (ISPMS) Actionable Governance Indicators (AGIs)
Public sector results chain within a broader governance environment 8 Broader Governance Environment
Moving forward: Utility criteria + feasibility CriterionDefinition 1. Action-worthyWe know (or strongly believe) that they contribute to results 2. ActionableThey are amenable to government action and project interventions 3. BehavioralFocus on function, not form 4. ReplicableGenerated transparently and can be reproduced by others …and 2 “feasibility” criteria. CriterionDefinition 1. Country coverageAvailable for a minimum of 20 countries 2. Time seriesCollected repeatedly
Selected Examples 1. “Does the executive’s budget or any supporting budget documentation present expenditures for the budget year that are classified by administrative units?” (Open Budget Survey) 2. Does a legislative committee hold public hearings on the individual budgets of central government administrative units in which testimony from the public is heard? (Open Budget Survey) 3. Tax payers access to information on tax liabilities and administrative procedures (PEFA) 10
De jure vs De facto An illustration: Public Accountability Mechanisms 11
Public Accountability Mechanisms (PAM) de jure indicators Financial disclosure (interests, assets, income) 217 indicators 2008, 2012 Freedom of information 37 indicators 2010 Conflict of interest restrictions 124 indicators 2011 Immunity protections 56 indicators 2012 12
Outputs of de jure data 13 Library of laws Analytical publications on the design of public accountability mechanisms Data, including qualitative and quantitative datasets, country profiles, and descriptive statistics Country reports on enabling governance environment ….But how do we know what happens in practice?
Freedom of information systems in practice: public sector management functions 1.1 Facilities 1.2 Data/Records 1.3 Human resources 1.4 Financial 1.5 Policy 2.1 Demystification* 2.2 Responsiveness 2.3 Appeals 2.4 Proactive Disclosure 14 1.0 Administrative Functions2.0 Disclosure Functions * clarity about government processes, rules, and decisions
Freedom of information systems: Immediate impacts/Intermediate outcomes 15 3.1 Public engagement: Extent to which the public understands, believes in, and engages with the freedom of information process. 3.2Government commitment: Extent to which the government supports the freedom of information regime, including efforts to establish participatory decision-making 3.3 Administrative culture: Extent to which the bureaucratic culture has shifted from principle of secrecy to openness. 3.4Operational efficiency: Extent of improvement in operations and decision-making within the organization.
How do you measure impacts in a governance “ecosystem” 16 Participation of civil society organizations and citizens Accountability and enforcement by parliament, judiciary, police, supreme audit institution and the ombudsman Political will that enables meaningful change, including protection from retribution, lack of clientilism, and an environment that maintains appropriate incentives for civil servants Foundational support mechanisms found in public sector systems, such as freedom of information frameworks, conflict of interest rules, financial disclosure practices, and human resources and policy management.
Methodologies matter 17 Surveys: validation issues, response problems Expert assessment: time-consuming, with degrees of subjectivity Household surveys: costly, time-consuming, logistical difficulties Economic/social indicators: collected by whom? National statistics agencies, multi-laterals, NGOs, donors…. Outcome mapping/process tracing: lengthy, involved, requires local capacity
So where do we start? 18 MethodologyIndicatorsConcepts
“Big ideas” for today 19 Actionability and action-worthiness Change in behavior and not in function Sustainable measurement practices Fuzzy concepts and methodologies Practical applications of data Challenges in de facto measurement
Some emerging trade-offs (or complementarities?) Global focus versus sub-national focus? Existing data or new data? Focused on public sector or broad governance issues? Perception-based versus objective data? Or both? Using a single tool or multiple tools? Detailed focus on a specific case/country 20
Some emerging issues Areas of focus? Tax administration Human resources Rule of law Looking forward Clarify the focus of the effort and link to a clear research question Integrate the issue of sustainability Benchmarking Understanding strengths of different types of respondents Clear definition of the context Dissemination and wide use of the data (and need for a strategy) New set of tools for certain phenomena (high level corruption) 21
23 InputsOutputs Rules (Institutional Arrangements) Organizational Capacities Governance System Performance Outcomes The Missing Middle Resources invested in projects to deliver its outputs. Examples: Funding, contracts, materials Goods and services produced by the project. Examples: Surveys and Trainings conducted, Laws revised, Agencies established Desired state of well-being -- a set of conditions, experiences or behaviors – that is the goal for change or improvement. Examples: Maternal/infant mortality rates, Standardized test scores of K-8 students Actionable Governance Indicators (AGIs) focus on specific and narrowly-defined aspects of governance.