1. Taking seriously the political dimension of Policy evaluation Vincent Spenlehauer École nationale des ponts et chaussées / IFRIS Séminaire ASIRPA 13 juin 2012 Inra Paris

2. Introduction Policy evaluation stems from the encounter of 1) the policy- something movement in US social sciences and 2) the multiplication of federal public interventions in the 1960s The history of PE is marked by great hopes, few practical successes and plenty of practical failures One main reason of this disappointing balance-sheet is knowledge/power clutching is not an easy and a normal game to play for social scientists and that most of them usually prefer the Leon Fetzingers solution (reduction of cognitive dissonance, 1957, 1993) to the Chris Argyris solution (double-loop learning, 1978) My message here is simple : sorry my friends PE presents a political dimension and youd better cope with it !

3. Knowledge/Power Clutching (I) A common definition of policy evaluation (D1) : an official process launched by public authorities and by which sound knowledge is produced about the evaluated policys general quality, implementation scheme, goal attainment, costs, outputs and outcomes To be really complete one should add (A2): « in order to ameliorate the policy under evaluation » Obviously, this poses the question of Knowledge/Power Clutching, because PE constitutes one aspect of policy-making which is one aspect of political action But why D1 more frequent than D1+A2 ? H1 : Enlightenment ideology or faith saying Good Knowledge will necessarily have positive impact. H2 : in big modern democratic societies politics and knowledge production are professionalized, hence quite separated As a result the history of policy evaluation practice is paved with cases of non-influential evaluations. Because their actors globally decided to comply with D1 and ignore A2. A good and politically useful policy evaluation is necessarily hybrid A good and politically useful policy evaluation is necessarily hybrid

4. Knowledge/Power Clutching (II) Illustration from Road Safety evaluation Context: Prime Minister service, Transport Minister, Road Safety agency director… (Spenlehauer 2011) Policy folds Evaluanda envisaged Evaluanda chosen Evaluanda evaluated vehicles Car, truck, motorcycle, bicycle roads Urban, rural Governtal campaigns Gore, informative, TV, radio, press education Schools, driving schools Involvement and empowerment Infra-nal tiers of gvt, corporate world, associations Law enforcement Speed, alcohol, safety belt, helmet Mostly speed

5. Knowledge/Power Clutching (III) evaluanda setting Complex evaluanda = dispersion of evaluative energy = weakening of evaluation impact on policy Thinking pol. Rate of return = eval. Impact/difficulty The hybrid exercise of good evaluanda setting Policy analysis as a main preliminary step. The evaluanda is not a given object. Beyond measurement methodology : what could and should be measured, and related to what values, objectives, norms. Political considerations. The inescapable political game to play, even for professional scientists (if they want to have a rationalization impact) Choose Rather than PKPP P P P

6. The knowledge production issue Evaluative KP = capturing, probing, gathering, assembling information about the evaluanda to answer in a convincing manner three basic questions : how (much) the evaluanda works, why does it work like this, is it possible to improve the way it works ? As a professional group academic social scientists can (should) play a crucial role because of their wonderful tool box and their institutionalized freedom of thought and speech > super- skill to design KP global scheme BUT The evaluandas stakeholders own a large part of info needed, dont share it easily, in as much informal and vernacular knowledge can be very relevant for the PE A good KP system is then multi-actors. Its construction and management shows also a political dimension: use of authority, alliance building, bargaining (participation against info). A balance must be found between 1) the access to, reliability and pertinence of the info processed and 2) the scientific robustness of the process

7. Combining earth, wind and fire ! Traditional input/output & outcome attributive measurement (renaissance with evidence-based policy and randomized experimental design) (Campbell & Julian 1966 ; Pawson 2006) Utilization-focused evaluation (the governance of the evaluation relies on an assembly of good-will stakeholders who transform the policy on the basis of the evaluation (Patton 1978) Theory-driven evaluation recommending focus on the opening of the evaluanda black-box to understand the action theories behind (what policy actors have in mind?) (Chen & Rossi 1989 ; Stame 2004) The question is… Instead of doing thiswhy dont we do this ?!?

8. Biblio Argyris Ch. 1993, Knowledge for Action. A guide to overcoming barriers to organizational change, Jossey Bass Baumgartner F. & Jones B. 2005, The Politics of Attention. How Government Prioritizes Problems, University of Chicago Press Campbell, D.T., and Julian C.S. 1966. Experimental and quasi- experimental designs for research. Rand McNally. Chen H., Rossi P., 1989, Issues in the Theory-Driven Perspective, Evaluation and Program Planning, (12)4:299-306 Festinger L. 1957. A theory of cognitive dissonance. Row, Peterson Lindblom C.E. 1958. 'Policy Analysis'. American Economic Review 48(3):298–312 Lowi T 1964 "American Business, Public Policy, Case-Studies, and Political Theory", World Politics, 16(4):677-715 Palumbo D. Eds 1987, The Politics of Program Evaluation, Sage Patton M.Q.1978, Utilization-Focused Evaluation, Sage Publications Pawson R. 2006, Evidence-Based Policy. A Realist Perspective, Sage Spenlehauer V. 2011, Des sciences sociales engagées : lévaluation des politiques publiques, (mémoire dHDR consultable sur demande) Spenlehauer V.1995, Lévaluation de politique, usages sociaux. Trois études de cas dévaluation, LHarmattan Stame N. 2004, Theory-Based Evaluation and Types of Complexity, Evaluation (10)1:58-76