The Separate but Related Cultures of Evaluation & Policy Analysis AES 2011 Conference David Turner

Overview This paper started with the observation that evaluation and policy practitioners read different journals, cite different authors, used different terminology but cover common ground I acknowledge that policy and evaluation have wide and diverse literatures; more than one person will usually follow closely

Common Ground Evaluation and policy analysis have a common focus on how policies are developed and implemented, and a general goal of public interest Evaluation and policy analysis have a common focus on how policies are developed and implemented, and a general goal of public interest Evaluation practitioners have to consider policy needs and stakeholders for evaluation work Evaluation practitioners have to consider policy needs and stakeholders for evaluation work

Different Outlooks Evaluation practitioners may treat policy goals and designs as fixed Evaluation practitioners may treat policy goals and designs as fixed Evaluators may not follow debates within policy community on policy & implementation Evaluators may not follow debates within policy community on policy & implementation

Separate Literatures Analysis of journal publications shows differences in sources Reviewed 2010 publications in Journal of Policy Analysis and Management (JPAM) and American Journal of Evaluation Reviewed 2010 publications in Journal of Policy Analysis and Management (JPAM) and American Journal of Evaluation Looked at content of the articles and the sources cited Looked at content of the articles and the sources cited

JPAM Articles 50 research articles in 4 issues, on range of policy areas such as education, tax policies, health, employment, and housing 50 research articles in 4 issues, on range of policy areas such as education, tax policies, health, employment, and housing Focus often on evaluative questions concerning policy implementation and impacts Focus often on evaluative questions concerning policy implementation and impacts Public administration, economic journals cited (rarely evaluation) Public administration, economic journals cited (rarely evaluation)

AJE Articles 18 research articles in 4 issues, most often focused on evaluation methods or approaches (less focus on policy substance) 18 research articles in 4 issues, most often focused on evaluation methods or approaches (less focus on policy substance) A few references to public administration or management journals, or journals in policy areas such as criminal justice or education A few references to public administration or management journals, or journals in policy areas such as criminal justice or education

Book Buying Patterns Looked for reading patterns among policy and evaluation books by using Amazon data Looked for reading patterns among policy and evaluation books by using Amazon data Identified lists of evaluation and policy texts Identified lists of evaluation and policy texts Looked for links under “customers who bought this book also bought…” Looked for links under “customers who bought this book also bought…” Mapped linkages between books (pointing from one to another bought by same people) Mapped linkages between books (pointing from one to another bought by same people)

Evaluation Books Patton, Utilization-Focused Evaluation Patton, Utilization-Focused Evaluation Morris, Evaluation Ethics for Best Practice Morris, Evaluation Ethics for Best Practice Chelimsky and Shadish, Evaluation for the 21 st Century Chelimsky and Shadish, Evaluation for the 21 st Century Rossi, Lipsey & Freeman, Evaluation Rossi, Lipsey & Freeman, Evaluation Scriven, Evaluation Thesaurus Scriven, Evaluation Thesaurus Posavac, Program Evaluation Posavac, Program Evaluation Stufflebeam, Evaluation Theory, Models, & Applications Stufflebeam, Evaluation Theory, Models, & Applications Weiss, Evaluation Weiss, Evaluation

More Evaluation Books Shadish, Foundations of Program Evaluation Shadish, Foundations of Program Evaluation Mathison, Encyclopedia of Evaluation Mathison, Encyclopedia of Evaluation Wholey, Handbook of Practical Program Evaluation Wholey, Handbook of Practical Program Evaluation Owen, Program Evaluation Owen, Program Evaluation Davidson, Evaluation Methodology Basics Davidson, Evaluation Methodology Basics Yarbrough, Program Evaluation Standards Yarbrough, Program Evaluation Standards Fitzpatrick, Program Evaluation: Alternative Approaches Fitzpatrick, Program Evaluation: Alternative Approaches Bamberger, Real World Evaluation Bamberger, Real World Evaluation

Policy Books Bardach, A Practical Guide for Policy Analysis Bardach, A Practical Guide for Policy Analysis Weimer & Vining, Policy Analysis Weimer & Vining, Policy Analysis Birkland, An Introduction to the Policy Process Birkland, An Introduction to the Policy Process Stone, Policy Paradox: The Art of Political Decision Making Stone, Policy Paradox: The Art of Political Decision Making Kraft & Furlong, Public Policy Kraft & Furlong, Public Policy Sabatier, Theories of the Policy Process Sabatier, Theories of the Policy Process Kingdon, Agendas, Alternatives, and Public Policies Kingdon, Agendas, Alternatives, and Public Policies

More Policy Books Stokey & Zeckhauser, A Primer for Policy Analysis Stokey & Zeckhauser, A Primer for Policy Analysis Pressman & Wildavsky, Implementation Pressman & Wildavsky, Implementation Hill & Hupe, Implementing Public Policy Hill & Hupe, Implementing Public Policy Smith, Writing Public Policy Smith, Writing Public Policy Dye, Understanding Public Policy Dye, Understanding Public Policy Wilson, Bureaucracy Wilson, Bureaucracy

E8E2 E7 E1 E4 E5 E6E3 E10 E15E9E16 E11 E12E13 E14 P13 P6P1 P7 P4P3 P2P5P11 P12 P1 0 P9 P8 Book Buying Networks (all links)

E8E2 E7 E1 E4E5 E6E3E10 E15E9 E16 E11 E12 E13 E14 P13 P6 P1P7 P4P3P2 P5 P11 P12 P1 0 P9P8 Book Buying Networks (reciprocal)

Learning from Policy Evaluation practice could draw on developments in policy and public management Evaluation practice could draw on developments in policy and public management How the policy process works How the policy process works What makes for successful policy implementation What makes for successful policy implementation

Policy Process Models Descriptions of how the policy process works have advanced from the “stages heuristic” over time Descriptions of how the policy process works have advanced from the “stages heuristic” over time Programs emerge over time, decisions are part of a continuous process Programs emerge over time, decisions are part of a continuous process Different models of policy provide a range of insights Different models of policy provide a range of insights

Policy Model: Example 1 Richard Matland described policy implementation as a function of ambiguity and conflict Richard Matland described policy implementation as a function of ambiguity and conflict A policy can be ambiguous in either means or ends A policy can be ambiguous in either means or ends A policy is subject to conflict when parties with a stake in an issue have different views over its means or ends A policy is subject to conflict when parties with a stake in an issue have different views over its means or ends

Ambiguity-Conflict Model Conflict Ambiguity LowHigh Low High Administrative implementation Issue: resources Example: smallpox eradication wide range of policy areas such as education, tax policies, health, employment, and housing Political implementation Issue: power Example: busing Experimental implementation Issue: Context Example: Headstart for children Symbolic implementation Issue: coalition strength Example: community action agencies

Implications for Evaluation The model describes a policy in ways that suggest what evaluation can achieve The model describes a policy in ways that suggest what evaluation can achieve Administrative: focus on technical questions Administrative: focus on technical questions Political: consider whose values and needs are being met by evaluation Political: consider whose values and needs are being met by evaluation Experimental: scope for contribution to new ideas and approaches Experimental: scope for contribution to new ideas and approaches Symbolic: people may need to be seen doing something, may not expect impacts! Symbolic: people may need to be seen doing something, may not expect impacts!

Another Policy Model Source: Institute of Policy Studies Working Paper 11/04, “Experimentation and Learning in Policy Implementation”, Eppel, Turner, Wolf

Implications for Evaluation Program theory: need to adapt to emerging issues, anomalies; may not have a stable theory of action Program theory: need to adapt to emerging issues, anomalies; may not have a stable theory of action Outcome evaluation: question of when (if ever) program goals stabilise for measurable results Outcome evaluation: question of when (if ever) program goals stabilise for measurable results

Grounding Formative Evaluation Implementation has been widely studied since Pressman & Wildavsky, but may not be considered in formative evaluation Implementation has been widely studied since Pressman & Wildavsky, but may not be considered in formative evaluation Wide range of endogenous and exogenous factors, significance of key actors Wide range of endogenous and exogenous factors, significance of key actors Substantial and varied literature Substantial and varied literature

Learning from Evaluation It’s not just evaluators who should look outside their own boundaries; policy analysts can learn from evaluation: It’s not just evaluators who should look outside their own boundaries; policy analysts can learn from evaluation: Alternative approaches for gaining insight (not just “gold standard”) Alternative approaches for gaining insight (not just “gold standard”) Need to define goals, indicators, logic more precisely where possible Need to define goals, indicators, logic more precisely where possible

Evidence of Change Recent attention to issues of complexity reflects awareness of need to adapt evaluation approaches Recent attention to issues of complexity reflects awareness of need to adapt evaluation approaches Developmental evaluation one example of a response to complex policy environments and emerging policy solutions Developmental evaluation one example of a response to complex policy environments and emerging policy solutions Attention also given to need for knowledge exchange across boundaries Attention also given to need for knowledge exchange across boundaries

Discussion What do you think? Is there a cultural divide with policy analysis, and if so how significant is it? What do you think? Is there a cultural divide with policy analysis, and if so how significant is it?