Presentation on theme: "Why composite should be non compensatory"— Presentation transcript:
1 Why composite should be non compensatory Giuseppe MundaUniversitat Autonoma de BarcelonaDept.of Economics and Economic History
2 The JRC group on applied statistics and econometrics in Ispra works in cooperation with the Commission services on macro-economic modelling, short term analysis, financial econometrics, and data analysis for antifraud.Several ongoing projects on composite indicators and one on knowledge economy indicatorsFinancial econometrics (Capital Adequacy Directives, Clearing & Settlement), with MARKT
3 Index construction- Steps [OECD-JRC Handbook]Step 1. Developing a theoretical frameworkStep 2. Selecting indicatorsStep 3. Multivariate analysisStep 4. Imputation of missing dataStep 5. Normalisation of dataStep 6. Weighting and aggregationStep 7. Robustness and sensitivityStep 8. Association with other variablesStep 9. Back to the details (indicators)Step 10. Presentation and disseminationThis reviews the steps involved in a composite indicator’s construction process and discusses the common pitfalls to be avoided. We stress the need for multivariate analysis prior to the aggregation of the individual indicators. We deal with the problem of missing data and with the techniques used to bring into a common unit the indicators that are of very different nature. We explore different methodologies for weighting and aggregating indicators into a composite and test the robustness of the composite using uncertainty and sensitivity analysis. Finally we show how the same information that is communicated by the composite indicator can be presented in very different ways and how this can influence the policy message.Steps for the construction of composite indicators (Source: Nardo et al., 2005a)A Theoretical framework - should be developed to provide the basis for the selection and combination of variables into a meaningful composite indicator under a fitness-for-purpose principle.Data selection - should be based on variables analytical soundness, measurability, country coverage, relevance to the phenomenon being measured and relationship to each other. The use of proxy variables should be considered when data are scarce.Multivariate analysis – should be used to study the overall structure of the dataset, and to assess its suitability, and explain the methodological choices, e.g., weighting, aggregation.Imputation of missing data – using different approaches should be done. Extreme values should be examined as they can become unintended benchmarks.Normalisation - should be carried out to render the variables comparable.Weighting and aggregation – should be done along the lines of the underlying theoretical framework.Uncertainty and sensitivity – analysis should be undertaken to assess the robustness of the composite indicator in terms of e.g., the mechanism for including or excluding a variable, the normalisation scheme, the imputation of missing data, the choice of weights, or the aggregation method.Links to other variables – should be made to correlate the composite indicator with other published indicators as well as to identify linkages through regressions.Visualisation – Composite indicators can be visualised or presented in a number of different ways, which can influence their interpretation.Back to the real data – The development of a composite indicator needs to be detailed, and decomposition into the underlying variables must be provided to reveal the main drivers for good/bad performance. Transparency is primordial to good analysis and policymaking.
4 Based on:The JRC group on applied statistics and econometrics in Ispra works in cooperation with the Commission services on macro-economic modelling, short term analysis, financial econometrics, and data analysis for antifraud.Several ongoing projects on composite indicators and one on knowledge economy indicatorsFinancial econometrics (Capital Adequacy Directives, Clearing & Settlement), with MARKT
5 Structure of the presentation Linear Aggregation RuleSocial Choice and Aggregation RulesNon-Compensatory Aggregation RulesConclusions
7 Linear aggregation rules and preference independence The variables X1, X2,..., Xn are mutually preferentially independentif every subset Y of these variables is preferentially independentof its complementary set of evaluators.The following theorem holds:given the variables X1, X2, ..., Xn,an additive aggregation function exists if and only ifthese variables are mutually preferentially independent.
8 Preferential independence implies that the trade-off ratio between two variablesis independent of the values of the n-2 other variables, i.e.Preferential independence is a very strong conditionfrom boththe operational andepistemologicalpoints of view.
9 The meaning of weights in linear aggregation rules “Greater weight should be given to components which are considered to be more significant in the context of the particular composite indicator”. (OECD, 2003, p. 10).Weights as symmetrical importance, that is "… if we have two non-equal numbers to construct a vector in R2, then it is preferable to place the greatest number in the position corresponding to the most important criterion." (Podinovskii, 1994, p. 241).
11 The aggregation of several criteria implies taking a position on the fundamental issue of compensability.Compensability refers to the existence of trade-offs, i.e. the possibility of offsetting a disadvantage on some criteria by an advantage on another criterion.E.g. in the construction of a composite indicator of environmental sustainability a compensatory logic (using equal weighting) would imply that one is willing to accept, let’s say, 10% more CO2 emissions in exchange of a 3% increase in GDP.
12 The implication is the existence of a theoretical inconsistency in the way weights are actually used and their real theoretical meaning.For the weights to be interpreted as “importance coefficients ” (the greatest weight is placed on the most important “dimension”) non-compensatory aggregation procedures must be used to construct composite indicators.
13 Linear Aggregation is a correect rule iff Mutual preferential independence appliesCompensability is desirableWeights are derived as trade-offs
16 Example (rearranged from Moulin, 1988, p Example (rearranged from Moulin, 1988, p. 228; 21 criteria and 4 alternatives )Objective: find best countryA first possibility:apply the plurality rule the country which is more often ranked in the first position is the winning one.Country a is the best.BUT Country a is also the one with the strongest opposition since 13 indicators put it into the last position!# of indicators35761st positionabc2nd positiond3rd position4th positionThis paradox was the starting step of Borda’s and Condorcet’s research at the end of the 18th century, but the plurality rule corresponds to the most common electoral system in the 21st century!!!
17 Borda approach Frequency matrix (21 criteria 4 alternatives) Rank a b Points1st87632nd9523rd1014th13Jean-Charles, chevalier de Borda
18 Borda approach Frequency matrix (21 criteria 4 alternatives) Rank a b Borda score:RankabcdPoints1st87632nd9523rd1014th13Borda solution:bcadNow: country b is the best, not a (as in the case of the plurality rule)The plurality rule paradox has been solved.
19 Condorcet approach Outranking matrix (21 criteria 4 alternatives) Jean-Antoine-Nicolas de Caritat, Marquis de Condorcet
20 Condorcet approach Outranking matrix (21 criteria 4 alternatives) For each pair of countries a concordance index is computed by counting how many individual indicators are in favour of each country (e.g. b better than a 13 times).“constant sum property” in the outranking matrix (13+8=21)
21 Condorcet approach Outranking matrix (21 criteria 4 alternatives) Pairs with concordance index > 50% of the indicators are selected.Majority threshold = 11 (i.e. a number of individual indicators bigger than the 50% of the indicators considered)Pairs with a concordance index ≥ 11: bPa= 13, bPd=21(=always), cPa=13, cPb=11, cPd=14, dPa=13.Condorcet solution: c b d aYet, the derivation of a Condorcet ranking may sometimes be a long and complex computation process.
22 Which approach should one prefer? Both Borda and Condorcet approaches solve the plurality rule paradox. However, the solutions offered are different.Borda solution: b c a dCondorcet solution: c b d aIn the framework of composite indicators, can we choose between Borda and Condorcet on some theoretical and/or practical grounds?
24 Number of indicators231721081stabc2nd3rdExample with 60 indicators[Condorcet, 1785]Borda approachCondorcet approachFrequency matrixOutranking matrixRankabcPoints1st23191822nd12311713rd2510concordance threshold =31aPb, bPc and cPa (cycle)???
25 From this example we might conclude that the Borda rule (or any scoring rule) is more effective since a country is always selected while the Condorcet one sometimes leads to an irreducible indecisiveness.However Borda rules have other drawbacks, too
26 Both social choice literature and multi-criteria decision theory agree that whenever the majority rule can be operationalized, it should be applied. However, the majority rule often produces undesirable intransitivities, thus “more limited ambitions are compulsory. The next highest ambition for an aggregation algorithm is to be Condorcet” (Arrow and Raynaud, 1986, p. 77).
28 Condorcet approachBasic problem: presence of cycles, i.e. aPb, bPc and cPaThe probability of obtaining a cycle increases withboth N (indicators) and M (countries)[Fishburn (1973, p. 95)]
29 Maximilien de Robespierre Condorcet approachCondorcet himself was aware of this problem (he built examples to explain it) and he was even close to find a consistent rule able to rank any number of alternatives when cycles are present…… butMaximilien de Robespierre
30 Condorcet approachBasic problem: presence of cycles, i.e. aPb, bPc and cPaFurter attempts made by Kemeny (1959) and by Young and Levenglick (1978) … led to:Condorcet-Kemeny-Young-Levenglick (C-K-Y-L) ranking procedure
31 C-K-Y-L ranking procedure Main methodological foundation: maximum likelihood concept.The maximum likelihood principle selects as a final ranking the one with the maximum pair-wise support.What does this mean and how does it work?
35 The Computational problem C-K-Y-L ranking procedureThe Computational problemThe only drawback of this aggregation method is the difficulty in computing it when the number of candidates grows.With only 10 countries 10! = 3,628,800 permutations (instead of 3!=6 of the example)To solve this problem of course there is a need to use numerical algorithms
42 The frequency matrix (Borda type approach) of a country’s rank in each of the seven dimensions and the overall KEIwas calculated across the ~2,000 scenarios.
43 A first consideration is that the overall ranking is very stable; in fact considering the whole 2,000 simulations, all countries are clustered unambiguously. No doubt the top performing countries are Sweden, Denmark Luxembourg, Finland and the USA. Then it follows the group Japan, United Kingdom, Netherlands and Ireland (where Japan and UK are slightly better than the other two). Austria, Belgium, France and Germany form the next group (where Germany is slightly worst than all the other three). All the rest of countries can be considered with a bad performance with respect to a knowledge based economy. However, we could still split this class into two subsets: a first one including Slovenia, Estonia, Malta, Cyprus, Spain, the Czech Republic, Latvia, Italy, Greece and Lithuania is a bit better than the worst performing group including Hungary, Portugal, Slovakia and Poland. An interesting result is also that overall both USA and Japan have a better performance than EU 15 and EU 25.
44 REFERENCESBorda J.C. de (1784) – Mémoire sur les élections au scrutin, in Histoire de l’ Académie Royale des Sciences, Paris.Brand, D.A., Saisana, M., Rynn, L.A., Pennoni, F., Lowenfels, A.B. (2007) Comparative Analysis of Alcohol Control Policies in 30 Countries, PLoS Medicine, 0759 April 2007, Volume 4, Issue 4, e151, p , www.plosmedicine.orgCondorcet, Marquis de (1785) – Essai sur l’application de l’analyse à la probabilité des décisions rendues à la probabilité des voix, De l’ Imprimerie Royale, Paris.Fishburn P.C. (1973) – The theory of social choice, Princeton University Press, Princeton.Fishburn P.C. (1984) – Discrete mathematics in voting and group choice, SIAM Journal of Algebraic and Discrete Methods, 5, pp
45 Munda G. (1995), Multicriteria evaluation in a fuzzy environment, Physica-Verlag, Contributions to Economics Series, Heidelberg.Munda G. (2004) – “Social multi-criteria evaluation (SMCE)”: methodological foundations and operational consequences, European Journal of Operational Research, vol. 158/3, ppMunda G. (2005a) – Multi-Criteria Decision Analysis and Sustainable Development, in J. Figueira, S. Greco and M. Ehrgott (eds.) –Multiple-criteria decision analysis. State of the art surveys, Springer International Series in Operations Research and Management Science, New York, ppMunda G. (2005b) – “Measuring sustainability”: a multi-criterion framework, Environment, Development and Sustainability Vol 7, No. 1, ppMunda G. and Nardo M. (2005) - Constructing Consistent Composite Indicators: the Issue of Weights, EUR EN, Joint Research Centre, Ispra.Munda G. and Nardo M. (2007) - Non-compensatory/Non-Linear composite indicators for ranking countries: a defensible setting, Forthcoming in Applied Economics.Munda G. (2008) - Social multi-criteria evaluation for a sustainable economy, Springer-Verlag, Heidelberg, New York.