1. Measurement in the Public Sector Professor Imran Rasul UCL and IFS January 2015 iGovern IE Workshop, Istanbul

2. Measurement Matters No matter how good the IE design, what can be learned from an evaluation depends crucially on the quality of measured variables Outcomes (dependent variable): Which measures of public service delivery (quantity and quality)? What level of aggregation (pyramid: individual to ministry)? Comparability across units of observation (common forms of data collection) Comparability within units through time (short run, long run) Controls (explanatory variables): Individual Project Organization Population/citizenry being served (bottom up monitoring) Bureaucratic-political nexus (top down monitoring) …

3. Measuring Output Example: Public Sector Projects (Across Ministries, 2003-12) Each project, at its outset, has a proscribed budget, and planned progression (by year).

4. Potential Outcome 1: Meeting Progression Targets

5. Potential Outcome 2: Time/Cost Overruns

6. Measurement Challenges in the Public Sector Systemic interlinkages of activities in the public sector non-independence of project outcomes not all projects leading to outputs for citizenry (e.g. procurement, training) Intermediate steps in service delivery drafting plans, task allocation, coordination, implementation Attribution of output to specific individuals teamwork imperfect indicators of inputs (hours worked ≠ effort) Output not always measured by market prices harder to quantify underlying demand for, and welfare implications of, PS projects Varying objectives of public sector organizations equity, social welfare, etc. Establish common data collection methods: freq./measurement error

7. Opportunities Collating secondary data that already exists Engaging in primary data collection bureaucrat characteristics (time use, non-cognitive traits) organizational characteristics (management practices) user characteristics (satisfaction with public services) engage with researchers to help build capacity to collect own data Use of new technologies provides new possibilities to monitor: bureaucratic activity (time inputs) citizen usage Use of new/combined methods to answer research questions: bottom up monitoring quality of public services (mystery shoppers) uncovering illicit behaviors (list randomization, elicitation etc.) lab and field experiments (use games to measure structural/preference parameters)

8. (Continued) Case Studies on Measurement: Role of Non-Pecuniary Incentives Sorting/recruitment into the to public sector What types of non-monetary incentive are likely to be effective in public sector jobs? wage compression in public sector relative to private sector although, growth in use of pay-for-performance management practices and public service delivery

9. 1. Sorting/Recruitment Roy [1951] model: individuals sort into jobs according to their preferences, skills and job attributes XS differences in personality/preferences between those in public and private sectors: preferences aligned with mission of organization/intrinsic motivation preference aligned with work environment/culture [teamwork, less exposure to competition etc.] Research problem: costly to screen applicants based on these kinds of preferences…

10. Financial Incentives and Recruitment Research Question: Do we induce adverse selection of types if we just use posted wages to recruit workers into the public sector? Evidence: Dal Bo et al. [2013] study the impacts of experimentally varying financial incentives on applicant traits Context: Mexico; 3750 vs 5000 pesos/month wage offer (33% increase) Measurement: Ability and personality: Earnings in previous job; Raven’s IQ, Big 5 Motivation: Public Service Motivation [Perry 1996]

11. Result 1: Higher wages attract more applicants Result 2: Higher IQ applicants apply More high IQ applicants Fewer low IQ applicants

12. Result 3: Personality Traits of Applicants Vary with Wage Offers

13. Result 4: No adverse selection on public service motivation Unanswered Measurement Related Question: do such recruits perform better on the job?

14. Ashraf et al. [2014]: Experimentally Vary Recruitment Posters for Health Workers Contrast is over making career incentives (wage progression) more or less prominent in recruitment adverts Job characteristics element of Roy [1951] model Context: community health workers, Zambia Measured outcomes: Applicant pool [ability, motivation] Performance on the job

16. Such information can affect the beliefs individuals hold… Bayesian individuals: [Crawford and Sobel 1982, Milgrom and Roberts 1986, Gentzkow and Kamenica 2011], Non-Bayesians: think categorically [Fryer and Jackson 2008, Mullanaithan et al. 2008] limited memory/attention [Mullanaithan 2002, Eliaz and Spiegler 2011] Non-informative dimensions of communication can affect belief formation through salience, attention and framing [Mullanaithan et al. 2008] …or what they think might be signalled to others [Benabou and Tirole 2003] Measurement of beliefs critical to understand why applicant pool might have changed [Desserano 2014]

18. Measured: Behaviour on the Job

19. 2. Management Rasul and Rogger [2014]: linking the managerial practices that bureaucrats operate under, to public services delivered Context: Nigeria, 63 government departments/federal agencies Includes 13 ministries, and other regional development authorities Measurement: dimensions of management practice (org. level) autonomy provision of incentives/monitoring (collection and use of KPIs) other (e.g. training, culture…)

20. Measuring Management Practices in Civil Service Organizations `double blind' face-to-face group interviews with senior managers, consensus reported management practice topics: roles, flexibility, incentives, monitoring, facilities, skills, staffing, targets and culture rank each answer 1 to 5, increasing in common notion of good practice derive three measures of management practice: autonomy, incentives/monitoring, other dimensions

21. Table A2: Defining Management Practices Management PracticeTopicSpecific Questions Related to this Topic AutonomyRoles Can most staff above SGL 7 in your organization make substantive contributions to the policy formulation and implementation process? Can most staff above SGL 15 in your organization make substantive contributions to the policy formulation and implementation process? To what extent do the employees in this organization have the ability to determine how they carry out the assignments in their daily work? Flexibility Does your organization make efforts to redefine its standard procedures in response to the specific needs and peculiarities of a community? How flexible would you say your organization is in terms of responding to new practices, new techniques, and regulations? At your organization, how efficiently is best practice shared between departments? Given past experience, how effectively would a conflict within your organization be dealt with?

22. Measurement: Public Service Delivery Audits of 4700 public infrastructure projects conducted by the 63 organizations 11 project types (road, borehole, training etc.) [construction and non-construction] projects evaluated by independent teams of engineers outcome: completion rate (0-100%) Checks and balances in audits: centralized team of technocrats monitored the evaluation teams evaluators were asked to provide material, photographic, or video evidence to support their reports random checks on evaluated sites We then hand coded the material from all projects 4721 projects, 63 civil service organizations, aggregate budget of US$800 million one year projects

23. Context: 38% of our sampled projects in Nigeria have completion rate of zero For projects that start: average completion rate of 70% Hypothesis: overworked bureaucrats? Figure A: Project Completion Rates (Nigeria Rural Infrastructure)

24. Figure B: Major Project Types by Implementing Organization Notes: Each histogram bar represents the total number of projects for the given project type on the x-axis. Each colored band represents the projects at a particular organization (different bands imply different organizations irrespective of whether they are the same color). The sample used to construct the histogram is those projects for which proportion completed evaluation variable and management scores are available. The projects are classified here by their primary classification. Distinct lack of specialization in many project types Overstretched organizations?

25. Does Management Matter?

26. Conclusion: Measurement Quality of IE design depends crucially on measurement of variables Cannot start thinking about such issues too early in IE design Getting academics involved: Fresh ideas less constrained by existing practices Collection of new/objective data Collation of existing secondary data Primary data collection Faster dissemination of best practice across organizations

27. Further Reading Amin et al (2008) ‘Are you being served? New tools for measuring service delivery’ Carrera and Dunleavy (2013) ‘Growing the productivity of government services’ Pidd (2012) ‘Measuring the performance of public services: Principles and practice’ Shah et al (2005) ‘Public services delivery’