1. Performance Measurement for Social Protection
Franziska Gassmann
Bangkok, June 2012

2. Objectives
Understand the importance of performance measurement as an essential tool for social protection governance.
What is performance measurement?
Where is it applied/can it be applied?
Understand the basic principles underlying performance measurements in relation to social protection by discussing the type and characteristics of indicators and related issues of data collection and data quality.
Link performance measurement to the policy making process and the dialogue with partner countries.
Understand how to select indicators and set targets for EC programme management.

3. Social Protection
Defined as public interventions to support poor and vulnerable households and individuals with the objectives to
Smooth consumption
Provide support during difficult times
Reduce poverty and inequality
Stimulate economic development
The focus is on non-contributory programs financed from general government revenues, such as
Cash transfers
Subsidized goods and services
Direct food support
Public work programmes
Social services
Residential care provision

4. Outline Relevance of performance measurement for social protection
Indicators for performance measurement
Different types of indicators
The characteristics of good indicators
Performance indicators for social protection policies
Measuring poverty
Data requirements
Wrap up

5. Performance measurement…
…monitors at a programme level the cost and quality of services and outcomes for clients in order to account for public expenditures
…is a tool for controlling and managing resources
…requires the definition of outcome objectives for policies and programmes
…provides regular and systematic information whether a program, policy or development strategy is implemented as planned, is achieving its objectives and whether and how it can be improved
…moves the focus from inputs/outputs to outcomes and results

6. Why focus on outcomes? (EC Guidelines 2009)
Results matter
Encourage evidence-based policy making
Protect political space of beneficiary countries
Promote domestic accountability
Stimulate demand for high quality data

7. Performance measurement for social protection
Non-contributory social protection policies are financed from general government revenues
Spending on social protection is heavily scrutinized and subject to constant pressure
It is vital for the government to prove that public spending on social protection achieves its objectives effectively and efficiently
The systematic and regular monitoring and evaluation of social protection policies is therefore crucial for any national government accountable to the public.

8. Regular performance measurement is especially important for social protection
Social protection policies are developed at central government level, but service delivery takes place at local level
Social programs rely on implementing agents
The poor often live in remote areas or socially isolated surroundings
Market forces are less prevalent in social service delivery

9. Guidelines for measuring progress towards intended outcomes in social protection
Use information that is already available
Monitor the extent to which social protection systems are pro-poor
Agree on a common set of indicators
Invest in building national capacity
(EC, 2003)
Performance indicators measuring achievements of objectives should be coherent with national and sector policies
Coherence with MDGs
A programme ensuring the quality of the indicators should be in place
(EC, 2009)

10. Types of indicators

11. A good indicator is… (World Bank, 2004)
A direct and clear measure of progress (specific)
Varies across areas, groups and over time
Sensitive to changes in policies
Not easy to manipulate or blown off by unrelated developments
Relevant for policy making
Can be tracked in a cost-effective way (available)
Consistent with policy-making cycle
Easy to understand
Reliable
Consistent with data availability and data collection capacity

12. …and supported by intelligent benchmarks (Cichon et.al, 2004)
Objective-based benchmarks
Time-based benchmarks
Comparative national benchmarks
Comparative international benchmarks

13. Two dimensions of social protection performance
Horizontal efficiency:
Effectiveness of the programme in reaching the target group
Under-coverage (exclusion) reduces horizontal efficiency
Vertical efficiency:
Efficiency of the programme in reaching the target group
Inclusion errors reduce vertical efficiency

14. Targeting efficiency – evaluation matrix
Target household
Non-target household
Household receives transfer
Success
Inclusion error
Type II error
Household does not receive transfer
Exclusion error
Type I error

15. Measuring exclusion and inclusion errors
Exclusion error (under-coverage) =
Number of poor w/o benefit
Number of all poor
Inclusion error =
Number of non-poor with benefit
Number of beneficiaries
Leakage =
Total amount of benefits received by non-poor
Total amount of benefits paid

16. Distributional analysis – example

17. Measuring targeting effectiveness
Share of transfers going to the poor =
% of transfers received by poor
% of poor in total population
Example:
Poor are 40% of the total population
Benefits received by the poor = 20%
Share received by the poor = 20/40 = 0.5
Interpretation:
Value = 1: targeting is neutral
Value < 1: targeting is regressive
Value > 1: targeting is progressive

18. Measuring adequacy
Share of benefits in total household income (consumption) =
Benefit value
Household income
Change in poverty rates before and after transfer
Absolute change = change in percentage points
Relative change = change as % of pre-transfer poverty

19. Measuring Poverty (a very short intro to…)
Why measure poverty?
Assess living standard of population, identify the poor
Identify needs/areas for policy support
Evaluate effectiveness of national poverty-reduction and redistribution policies
Assess impact and performance of development aid
What do we need (besides data…)?
Concept
Indicators
Measures
Analysis and interpretation
More reasons why we need to measure poverty:
Advocacy purposes (= make the poor visible)
Definitely since introduction of PRSPs is poverty reduction one of the main goal for development programs and projects; most projects have as their ‘impact objective’ or ‘outcome’ a goal such as ‘to reduce poverty’ or to increase the living standard of the population.
The welfare of the population is the result of all national policies. If everybody would benefit equally, there would be no need for poverty and inequality measures, but some benefit more than others. Some policies, esp. social protection policies are explicitly targeted to reduce poverty, and, consequently, we have to measure poverty regularly.

20. Conceptualizing poverty, cq well-being
Welfarist approach:
Individual utility determines well-being
(Lack of) well-being expressed in the level of income, consumption, assets, etc.
Monetary measures
Non-welfarist point of view:
Other aspects determine well-being, such as ownership of commodities, fulfillment of basic needs, access to basic services, being socially included….
Lack of capabilities to function
Rational consumer theory states that everybody makes the choice that provides the highest utility. The welfarist approach assumes that hhs have preferences and they are expressed in what they consume (food or health?) and that this choice gives them the highest utility they can possibly achieve. As such money is a good representation of welfare. No value attached to the choices made by the household. It also assumes that everything can be bought/consumed. Shows immediately shortcomings of purely welfarist approach. – choices made by household may not be wise; intra-household distribution? Accessibility of services? Only works if all choices are available.
But also in non-welfarist approach and capability approach, command over commodities as an instrument to increase well-being still important.

21. Most common approaches
Monetary poverty (income, consumption, expenditure, assets, …)
Capability approach (coined by A. Sen)
Social exclusion (think ‘EU’)
Vulnerability (origin in disaster-risk-reduction)
Monetary poverty: one-dimensional (although, implicitly multi-dimensional)
Other approaches: multi-dimensional
Monetary poverty: Still most common approach used for policy evaluation and poverty assessments
Development away from simple income or consumption based concept to more sophisticated and complex concepts.
Income may not enough to describe poverty. Poverty can materialize in other aspects, such as basic needs not covered, lack of access to education, health and other public services, relative deprivation, etc. Non-monetary measures gained ground over time.
Why then, after all, are still most poverty assessments based on a monetary measure? All agree that poverty is multidimensional phenomenon, but many of the non-monetary concepts are difficult to be measured and valued.
Monetary poverty:
Concept/idea: welfarist approach where monetary status reflects a household’s utility level and preferences
Economic welfare: goods and services consumed by household
Straightforward, easy to interpret, may not capture well-being when markets do not function properly
Capability approach:
Term coined by Amartya Sen
The capability approach argues that individuals are poor if they do not have the freedom to achieve their goals of well being and they are vulnerable when this lack of freedom persists over time (Sen 1999).
‘Utility’ and other commodity based formulations as a metric of welfare are rejected
‘Well-being’ means ‘being well’: ‘… value of living standards lies in the living and not in the possession of commodities…’ (Sen, 1987)
Poverty is seen as a failure to achieve certain basic capabilities
Basic capabilities: ‘the ability to satisfy certain crucially important functionings’ (Sen, 1993)
An individual’s state of being can be viewed as a vector of functionings; i.e. achieved outcomes that determine life and living.
These functionings are attained by utilising resources over which individuals have legal ownership rights, what Sen (1981) called entitlements.
Measuring capabilities is difficult, since capabilities represent a set of potential outcomes
In practice: focus on functionings (e.g. life expectancy, literacy, nutrition levels…)  very close basic needs approach

22. Poor ≠ Poor
Each approach has different perspective on what constitutes a good life
Each requires methodological assumptions
Each has different policy implications
MA: ensuring higher income solves problem
CA: wider range of action needed (social provision of goods, improved allocation within family, efficient use of goods to achieve outcomes)
SE: relative aspect important; just raising the level does not do the trick; redistribute policies
Social exclusion
1980s and 90s: social exclusion emerged as concept in EU
Marginalization and deprivation can exist even in rich societies with well developed welfare systems
SE is ‘…a process whereby certain individuals are pushed to the edge of society and prevented from participating fully by virtue of their poverty, or lack of basic competencies and lifelong learning opportunities, or as a result of discrimination.’(EC, 2004)
Goes beyond income poverty and includes social, cultural and political dimensions of exclusion  non-welfarist aspects of deprivation
Close to Townsend’s concept of deprivation
Acknowledges the fact that deprivation can have multiple faces
Open method of coordination, Lisbon Agenda (2000)
Laeken indicators
Main characteristics of social exclusion approach:
Relativity: Relative to a particular society
Agency: SE as result of action of an agent
Dynamics: Future is relevant as well, one disadvantage may lead to more disadvantages
Multidimensional: Raises aggregation issues
Problems with SE approach:
Application to developing countries problematic as characteristics of SE differ
Aggregation, causality between different dimensions
What is ‘normal’? What are appropriate SE characteristics?
Overall, least-defined and most difficult approach to poverty
Vulnerability:
No consensus on concept, how to define and measure it
Originates in Disaster Risk Reduction
General agreement that it differs from poverty and is a forward looking concept
Most often: vulnerability as risk of falling into poverty, measured as expected poverty (Calvo&Dercon, 2005; Pritchett et.al. 2002; Haughton&Khandker 2009, and more)

23. In order to measure poverty, we need…
Welfare indicator
What indicator is best suited to determine the living standard of a household or individual in a given society?
What indicator allows ranking households from poorest to richest?
Different welfare concepts imply different indicators, and my identify different groups of poor
Define a cut-off threshold, cq. poverty line
Absolute versus relative poverty
What constitutes a minimum living standard?

24. Proportion of health deprived children by income quintile, Vietnam 200624

25. Relative vs. absolute Relative poverty line Absolute poverty line
Set as a share of average or median income or expenditures
Moving concept; there will ‘always’ remain a class of poor people
Closely linked with income distribution
Examples: EU social indicator (60% of median income)
Absolute poverty line
Based on minimum consumption basket (food plus?)
Survey-based or derived from other standards (e.g. national legislation, WHO)
Examples: 1$ per day line, national poverty lines
Issue:
Relative poverty line
allows comparison between countries
Median or mean? What share?
absolute poverty line has also relative aspects
who determines what needs to be covered?
Goods/services in minimum basket differ per country, ev. Even per region- think for example of Russia with very warm and very cold regions – needs for heating )
Choice of poverty line also depends on purpose of poverty analysis
international lines not useful for national policy making
International lines can be close or far off national lines

26. Example: Poverty in Country X
Absolute poverty is defined regardless of the general welfare level or distribution in a given society. An individual is considered poor if he/she is not able to obtain the minimum necessities to maintain a purely physical existence .
Relative poverty identifies those people that are poorest within the overall distribution in a given society. The concept changes with the ideas and values prevailing in a society.

27. How to determine the minimum living standard?
In monetary terms:
Objectively defined minimum, e.g. based on a basket of goods and services – absolute level
As a percentage of the average living standard – relative level
In non-monetary terms:
Minimum years of schooling
Access to primary health care
Possession of x household appliances
Having one hot meal per day …
Many different ways to define minimum standards. Think of MDGs, or CRC. Regarding the latter: formulated rather generally, minimum still has to be defined; varies per society.

28. Examples of poverty lines
International absolute poverty lines:
1.25 USD PPP per capita per day
Useful for international comparison, but not necessarily for national policy making
International relative poverty lines:
60% of median income per adult equivalent
World Bank methodology based on actual consumption for national poverty lines:
Extreme poverty: costs to obtain a minimum food basket
Absolute poverty: Minimum food basket plus allowance for non-food goods and services
Subjective poverty lines
Derived from minimum income question
Frequently higher than ‘objective’ poverty lines
Guaranteed minimum income (budget based)
Other methods: e.g. food share (show engel curve); this can also be applied to other expenditure items, e.g. housing (to determine housing allowance), but has its own problems in latter case – preference of hh where and how to live. No general acceptable standard.
Minimum subsistence level – very popular in former SU countries! Not based on hbs, but on scientifically determined consumption standards.

29. y
Poverty measures HC z PG q n

30. Poverty incidence (headcount)
The 3 most common poverty indicators n=total population; q=poor; yi=welfare level; z=poverty line
Poverty incidence (headcount)
P0 = q/n = number of poor
total population
Poverty gap = average gap as a percentage of the poverty line
P1 = 1/n ∑ q [(z – yi) / z];
non-poor: gap is zero!
Poverty severity = weighted average gap
P2= 1/n ∑q [(z – yi) / z]2 INDEX!

31. Example: Kyrgyz Republic 2005 Source: WB (2010)
Poverty indicators by region (absolute poverty line)
Total country: % %

32. Scorecard Kyrgyz Republic (WB, 2009)

33. Data requirements – Types of data
Administrative data
Provide information on inputs and outputs
Collected by line ministries or special agencies
Less suitable for analysis of policy outcomes
Household survey data
Representative sample of population
Essential for poverty analysis
Essential for assessing policy outcomes
Population census
Basic information on all inhabitants (limited indicators)
Very costly, at most every 10 years
Needed for poverty maps
Usually not suitable for assessing policy outcomes

34. Data requirements – household surveys
Essential for analysis of policy outcomes
Availability improved considerably over the past decade
Many variations, but not all equally suitable
HBS, LSMS, MICS, DHS, etc.
Have their own shortcomings
Reliability of information (e.g. income from transfers)
Questionnaires not always in line with latest policies
Target groups cannot always be identified (e.g. disabled, war veterans)
How frequent should they be available?
No general rule
At least every 2-3 years, if financially affordable

35. Data requirements – qualitative data
Provides information not provided in surveys
Subjective dimensions of well-being
Intra-household inequality
Barriers to access (benefits, services)
Cultural, political or social factors that determine policy outcomes and impacts
In-depth interviews, focus group discussions
To inform survey design or to validate findings from survey

36. Wrap up
Performance measurement ensures the accountability of the policy maker and the transparency of the decisions taken
Availability and measurement of performance indicators is essential for social protection policy making
Ensure regular and systematic collection of reliable information
Analyse the collected information from various perspectives
Data requirements need to be carefully assessed as resources are often scarce
Better have a limited set of core indicators which are regularly collected and analysed
Performance measurement is part of the policy process and the interpretation of indicators needs to be placed in the political and economic context