Procedures for the comparison of policy options: Scryer The ex-ante evaluation of policies: The case of food safety regulations Corso per dottorandi Economia e Statistica Agro-alimentare Maddalena Ragona Dipartimento di Scienze Statistiche, Università di Bologna Bologna, febbraio-marzo 2012

Contents Fuzzy sets Scryer

Fuzzy logic Classifying statements in «true» and «false» may be too restrictive… Any statement may have a certain degree of truth E.g. is your coffee bitter or sweet? 0.8 sweet 0.2 bitter When linguistic variables are exploited, there are specific functions to manage different degrees of truth

Fuzzy logic - Coffee example Truth Quantity of sugar 1 0 Bitter Sweet Very sweet Note that the fuzzy membership functions may have very different shapes, which also depend on how large they are (how uncertain is the judgement)

Fuzzy vs. Probabilistic Logic The distinction is philosophical Fuzzyness as «degree of belonging» to different sets (Subjective) probability: how much it is probable that the element belongs to that set (it belongs to one set only, but there are different degrees of perception) Probability as a sub-set of fuzzy logic? Fuzzy probability?

Scryer (MoniQA socio-economic evaluation tool) Fuzzy multi-criteria tool to support decision-making Steps 1.Qualitative assessment of each impact for each policy option (coding/scoring procedure based on expert(s) judgement) 2.Feasibility filter (data availability, time, costs) to evaluate the possibility of quantitative assessment 3.Quantitative assessment of some impacts accounting for statistical error 4.Fuzzy multi-criteria comparison of options Computer-based Currently Excel spreadsheet, to be implemented into web-application

Characteristics Analysis of impacts based on the directions of the EC Impact Assessment Guidelines (2009) Ranking of policy options based on NAIADE software, developed at JRC- EC for environmental impact assessment It allows for both synthesis of quantitative (model-based) and qualitative assessments without the need for monetisation It may take into account public sensitiveness It accounts for uncertainty in outcomes evaluations (including lack of data / external uncertainty like weather / expert internal uncertainty) Weighting of impacts is allowed for Sensitivity analysis of the policy ranking Advantages of a fuzzy multicriteria approach (Scryer)

qualitative assessment – data entering

qualitative indicator X on a 1-9 scale uncertainty indicator U on a 1-5 scale qualitative assessment – coding procedure 15 Very good information Very low or no uncertainty No information High uncertainty

case study – qualitative assessment

case study – feasibility filter Is quantitative evaluation needed / feasible?

Step 2 - Feasibility filter

Step 3: quantitative assessment For each policy option, insert: Estimated impact Standard error of estimate

user weight

The scoring system tends to privilege impacts with high probability of occurrence and high level of information certainty. One prominent impact on the benefit side & several important impacts on the cost side case study – final ranking

Fuzziness in Scryer What’s the impact of a specific regulation on public healt? Negative and weak? Neutral? -Positive and weak? -Strong and positive? -The qualitative evaluation may belong to a single statement or to several ones with different degree of membership -It depends on uncertainty -Qualitative fuzzy evaluations may be aggregated with quantitative statistical (probabilistic, model-based) evaluations

The starting impact matrix for fuzzy multi-criteria calculations dimension 14*(2n) X  14  n matrix whose elements x ij : ordinal values (between 1 and 9) which measure the impact of policy j for the impact i U  a 14  n matrix whose elements u ij : corresponding uncertainty assessments (values between 1 and 5) policy j (j: 1,…,n) impact category i (i: 1,…,14)

Steps 1)Transform qualitative variables into Gaussian fuzzy sets 2)Compute distances between pairs of policy options for each specific impact category (distance between two fuzzy sets or stochastic variables) 3)Produce a pairwise comparison between policy options based on the above distances and the weights assigned to impact categories 4)Rank the policies based on their performance in pairwise comparisons

Gaussian fuzzy sets If element x ij is a qualitative score, it needs to be transformed into a fuzzy set Gaussian fuzzy sets Fuzzy sets defined through a membership function for each of its elements A degree of membership is needed for each of the 9 values of X Fuzzy set S k where k: 1,…,9 are the potential values that x ij may assume q actual assessment of x ij, where q is a single value between 1 and 9 The membership function is defined as follows: K  centre of the fuzzy set S k  k  width of the fuzzy set S k (i.e. a measure of dispersion around the centre) The Gaussian membership functions return a value between 0 and 1, where when q=k

The «variance» (uncertainty) function of the centre k of each fuzzy set and of the stated uncertainty level u assumption  dispersion is larger for assessments around 5 and for smaller values of u ij standard deviation for a continuous uniform distribution ranging from 1 to 9 is 2.58  we adopt this value as the maximum variability level with k=1,…,9

Example x ij =3  score for a given impact u ij =4  level of uncertainty The membership function is computed for all sets S k with k ranging from 1 to 9, considering the relative dispersion value  k. Consider the first fuzzy set S 1, for which k=1

Distance between two fuzzy sets ox i1 qualitative impact of the first policy for the i-th category of impact ox i2 impact of the second policy for same category oThe comparison depends on two fuzzy sets  S (x i1 =q) and  S (x i2 =h) 1)Rescale the membership functions through a constant c so that their integral equals to 1, for example, for  S (x i1 =q) 2)Compute the distance  weighted average of all potential distances between the linguistic values, weighted by their membership functions

Distance - quantitative when the impact is quantitative distance between two impacts  assuming a normal distribution and exploiting the Hellinger distance s 1 and s 2  standard errors of the estimated impacts x i1 and x i2

Pairwise comparison (by impact) Credibility values are computed for a set of preference relations between 2 options for each impact category 2 policy options P 1 and P 2 6 statements: P 1 is much better than P 2 (according to criterion i) P 1 is better than P 2 P 1 is more or less like P 2 P 1 is identical to P 2 P 1 is worse than P 2 P 1 is much worse than P 2 range between 0 (not credible at all) and 1 (maximum confidence)

Computation of credibility values (1) elements needed (a)semantic distances (also considering the “sign” of the relationship); (b)cross-over values parameter which indicates the distance for which credibility is set at 0.5 (i.e. the confidence that the statement is credible equals the confidence that it is not credible) must be fixed (or left to the user)

Computation of credibility values (2)

Pairwise comparison (across impacts – aggregation) w i [0,1] (with i:1,…,c)  weights assigned to each criterion Aggregate preference intensity index for each of the 6 preference statements

entropy 29 preference intensity indices may hide very heterogenous situations, in terms of consistency across the credibility indices for the various criteria  entropy measure, to ‘weigh’ the preference intensity indices in the final policy ranking step Adjusted membership function for each policy comparison, considering a threshold to rule out very small preference intensities: increases as the basic credibility values concentrate around 0.5 (i.e. uncertainty) tends to 0 when most of the basic credibility values are 0 or 1 (i.e. certainty) extremes : H=0 when all basic credibility values are 0 or 1, H=1 when all basic credibility values are 0.5

Ranking of policy options entropy can be considered distance between two impacts  assuming a normal distribution and exploiting the Hellinger distance without entropy  omit terms in square brackets and uss 2(p-1) as denominator. For each policy option, the equations aggregate the much better (much worse) and better (worse) preference intensity indexes, to generate an aggregate preference index for the best/worst policy option. degree of membership to the statements that ‘Policy alternative i is the best policy option’ and ‘Policy alternative j is the worst policy option’ range between 0 and 1

Multi-Criteria Analysis vs. Cost-Benefit Analysis MCACBA more comprehensive approachless comprehensive approach (only monetary values) based on experts’ preferences (subjectivity) measures individual preferences (objectivity), even though biased by income objectives and criteria are more clearly stated objectives and criteria are often implicitly assumed has not a rigorous approach to include time discounting has a rigorous approach to include time discounting (but difficult to choose appropriate discount factor) distributional impacts are more clearly considered distributional impacts are less clearly considered

Final considerations There is no optimal procedure: Scale of measurement of impacts Decision aim

References Figueira, J., Greco, S., and Ehrgott, M., Multiple criteria decision analysis: state of the art surveys. International Series in Operations Research and Management Science. Springer Munda, G., Nijkamp, P., and Rietveld, P., Comparison of fuzzy sets: A new semantic distance. Serie Research Memoranda. Free University, Amsterdam -----, Qualitative multicriteria methods for fuzzy evaluation problems: An illustration of economic-ecological evaluation. European Journal of Operational Research 82, Ragona, M., Mazzocchi, M., Zanoli, A., Alldrick, A.J., Solfrizzo, M., and van Egmond, H.P. (2011). Testing a toolbox for impact assessment of food safety regulations: Maximum levels for T-2 and HT-2 toxins in the EU. Quality Assurance and Safety of Crops & Foods, 3(1):12-23 Zadeh, L.A., Fuzzy sets. Information and Control 8,