Presentation is loading. Please wait.

Presentation is loading. Please wait.

The perils and pitfalls of productivity measurement for English local authorities Prof Iain McLean Dr Dirk Haubrich University of Oxford This work is funded.

Similar presentations


Presentation on theme: "The perils and pitfalls of productivity measurement for English local authorities Prof Iain McLean Dr Dirk Haubrich University of Oxford This work is funded."— Presentation transcript:

1 The perils and pitfalls of productivity measurement for English local authorities Prof Iain McLean Dr Dirk Haubrich University of Oxford This work is funded by the ESRC Public Services Programme, grant no. RES

2 McLean & Haubrich: The perils and pitfalls of productivity measurement for English local authorities 2 Outline of talk Productivity in the context of our project The CPA framework to assess English local authorities Pitfall 1: Vicious triangles Pitfall 2: The limits of self-assessment Pitfall 3: Inferring productivity from authorities expenditure & CPA scores

3 McLean & Haubrich: The perils and pitfalls of productivity measurement for English local authorities 3 1.Is CPA a reliable and valid measure to assess service quality in local authorities? Or is service quality influenced by external constraints, such as deprivation, discretionary expenditure, or political control? 2.Assuming that CPA is (or can be made to be) a reliable & valid measure, can it be used to capture productivity? 3.What, if anything, can England learn from Wales and Scotland? Productivity in the context of our project Productivity is only one of many issues addressed by our 12-months research project, which asks 3 questions about the assessment of local authorities:

4 McLean & Haubrich: The perils and pitfalls of productivity measurement for English local authorities 4 The CPA Assessment Framework* ASSESSMENT FRAMEWORK ABILITY TO IMPROVE Self-Assessment and Corporate Assessment OVERALL ASSESSMENT - Excellent - Good - Fair - Weak - Poor PUBLIC REPORTING IMPROVEMENT PLANNING CURRENT PERFORMANCE ON SERVICES Inspection Judgements, auditor judgements, performance indicators, and governmental Assessments of councils plans Benefits Social Care Environment Libraries & Leisure Use of resources Education Housing Audit Commission 2002 County Councils Total Score Category Score Single Tier Councils Total Score Category Score Less than 24 points1Less than 30 points1 24 to to to to 453 More than 364More than 454 Council ScoreCategory Score Less than 24 points1 24 to to 393 More than 404 ServiceMinimum Score Maximum Score Education416 Social Services416 Housing*28 Environment28 Benefits*14 Libraries and Leisure14 Resources14 TOTAL SCORE1560 Councils core services category score 1234 Councils Ability to Improve category score 1Poor WeakN/A 2PoorWeakFairGood 3WeakFairGoodExcellent 4N/AGoodExcellent * as applied between 2002 and 2004; minor changes introduced in With such an elaborate scheme (using hundreds of PIs and auditor/inspectorate judgements), CPA may be said to present a rounded assessment of an authoritys performance, potentially providing a basis for productivity measurement.

5 McLean & Haubrich: The perils and pitfalls of productivity measurement for English local authorities 5 Pitfall 1 – The vicious triangle (I) Indicator used in CPA and IMD/SSA: % of pupils doing well in external assessment Performance on Indicator increases CPA score increases greater SSA autonomy CPA IMD ranking decreases SSA grant decreases IMD/SSA

6 McLean & Haubrich: The perils and pitfalls of productivity measurement for English local authorities 6 Pitfall 1 – The vicious triangle (II) Implication for local authorities: If you improve your GCSE results, your CPA score goes up but your SSA share goes down. Conversely, if GCSE results worsen, SSA share goes up but your CPA score goes down. Either way you gain a (partly) financial bonus and suffer a (partly) financial penalty. You should do whichever carries the larger net bonus. Implication for central government There are two contradictory regimes in place, at least one of which must be abandoned or modified.

7 McLean & Haubrich: The perils and pitfalls of productivity measurement for English local authorities 7 Pitfall 2 – The limits of self-assessment (I) The CPA approach relies on authorities self-assessing their ability to improve service delivery in the future. Yet, the first three CPA rounds show that their self- assessment in year t is a bad predictor for their actual performance in t+1 or t+2.

8 McLean & Haubrich: The perils and pitfalls of productivity measurement for English local authorities 8 Pitfall 2 – The limits of self-assessment (II) CPA ratings and scores provide policy makers with information about an authoritys performance in delivering public services. These might constitute a suitable output/outcome basis for productivity measurements, but the partial reliance on self-assessment to produce these scores and ratings raises doubts about their reliability and validity. This is a problem, because the Audit Commission is currently in the process of re-designing CPA for the rounds 2008 onwards and seeks, with support from the Local Government Association (LGA), to incorporate additional elements of self-assessment. Our analysis holds even when issues regarding ceilings and moving of the goal posts are considered and the model is rerun.

9 McLean & Haubrich: The perils and pitfalls of productivity measurement for English local authorities 9 Pitfall 3 – Inferring productivity from CPA & spend (I) X Y Z 3 Outliers LAs with high productivity? LAs with low productivity?

10 McLean & Haubrich: The perils and pitfalls of productivity measurement for English local authorities 10 With such an elaborate scheme (which uses hundreds of PIs and auditor/inspectorate judgements), CPA may be said to present a rounded assessment of an authoritys performance. Potentially, then, CPA provides a suitable basis for productivity measurement and allows identifying productive (green) and unproductive (red) authorities. However, this is only true if CPA is a valid and reliable measure of public service in local authorities (for doubts, see pitfalls 1 and 2) if the formula by which central government calculates the SSA/FSS accurately assesses needs and (!) resources (see pitfall 3) But what if not? Pitfall 3 – Inferring productivity from CPA & spend (II)

11 McLean & Haubrich: The perils and pitfalls of productivity measurement for English local authorities 11 End of Presentation

12 McLean & Haubrich: The perils and pitfalls of productivity measurement for English local authorities 12 Backup Slides

13 McLean & Haubrich: The perils and pitfalls of productivity measurement for English local authorities 13 Circularity between CPA & IMD: Preliminary Findings CPA IndicatorIMD IndicatorBehaviour of indicatorsConcern for statistical model 1 Soc. Services Adults, A0/A5: Emergency Admissions (Interface) percentage change on previous year in total emergency admissions to hospital per 1,000 population Health Deprivation: Measures of emergency admissions to hospital CPA: (rise) of indicator (emergency admissions) leads to of CPA score, which leads to in financial freedom IMD: of indicator (emergency admissions) leads to of level of deprivation, which leads to in financial resources. MODERATE, because (a) very similar measures are used on both sides, and both indicators move into the same direction 2 Soc. Services Children, CF/A2: Percentage of young children leaving care aged 16 or over with at least 1 GSCE at grade A*-G Education Deprivation: Average Points Score of pupils at key stage 4 (GSCE) CPA: of indicator (educational attainment of care leavers) leads to of CPA score, which leads to in financial freedom IMD: of indicator (educational attainment) leads to of level of deprivation, which leads to in financial resources. LOW, because (a) CPA indicator (with a denominator of children leaving care) is only a small subset of the IMD indicator (with a denominator of all children); (b) IMD indicator measures average score, while CPA indicator has threshold of 1 GSCE. 3 Soc. Services Adults A0/C29,30: Adults with physical/mental/learning disabilities helped to live at home per 1,000 population Health Deprivation: Comparative Illness and Disability Ratio CPA: of indicator (percentage of adults cared at home) leads to of CPA score, which leads to in financial freedom IMD: of indicator (higher ratio of ill or disabled individuals) leads to of level of deprivation, which leads to in financial resources. LOW, because (a) the two indicators do not measure exactly the same phenomenon. (b) CPA Indicator may have to be per 1,000 population in the denominator, rather than all disabled adults? 4 Housing BV 184b: "The percentage change in proportion of non-decent Local Authority homes between 1 April x and 1 April x+1." and definitions of decent. Living Environment: Social and private housing in poor condition CPA: of indicator (percentage change in non- decent homes) leads to of CPA score, which leads to in financial freedom IMD: of indicator (housing in poor condition) leads to of level of deprivation, which leads to in financial resources. MODERATE, because (a) the two indicators do not measure exactly the same phenomenon, but (b) on the IMD side, the indicator has a considerable weight on the IMD composite In the statistical model, the 100+ PIs used in the seven CPA service blocks (as dependent variable) need to be checked for potential circularity with the 27 indicators used in the IMD domains (as explanatory variable). So far, no circularity issue was found for Culture, Environment, & Benefits. Education is still being checked; Social Serv. & Housing has 4 PIs with moderate/low circularity:

14 McLean & Haubrich: The perils and pitfalls of productivity measurement for English local authorities 14 The CPA Assessment Framework ASSESSMENT FRAMEWORK ABILITY TO IMPROVE Self-Assessment and Corporate Assessment OVERALL ASSESSMENT - Excellent - Good - Fair - Weak - Poor PUBLIC REPORTING IMPROVEMENT PLANNING CURRENT PERFORMANCE ON SERVICES Inspection Judgements, auditor judgements, performance indicators, and governmental Assessments of councils plans Benefits Social Care Environment Libraries & Leisure Use of resources Education Housing Audit Commission 2002 The CPA Assessment Framework


Download ppt "The perils and pitfalls of productivity measurement for English local authorities Prof Iain McLean Dr Dirk Haubrich University of Oxford This work is funded."

Similar presentations


Ads by Google