EVAL 6000: Foundations of Evaluation Dr. Chris L. S. Coryn Kristin A. Hobson Fall 2012

Agenda Stage One theories – General characteristics – Michael Scriven Questions and discussion Encyclopedia of Evaluation entries

General Characteristics of Stage One Theories Stage One theories can be characterized as a period of social problem solving Emphasis was on scientific rigor and valid causal knowledge of the effects of social programs Reflects the tenor of the 1960s, a period of rapid development in the social sciences

Stage One Theory of Social Programming Programs largely viewed as immediately implementable solutions to social problems Rested on assumptions that A.Implementation and evaluation would be unambiguous B.“Successes” would be adopted C.A and B combined, would lead to ameliorating social problems Concerned primarily with innovations/pilots/demonstrations

Stage One Theory of Knowledge Construction Priority was given to “truth” The dominant view was that an external reality probably exists, though it may never be perfectly known Valid knowledge could be constructed (more or less) Emphasis on sources of bias and the best methods for reducing them

Stage One Theory of Valuing Theorists at this stage agree that valuable social programs solve important social problems They disagree about how value claims are made and whose values should be given priority – Prescriptive versus descriptive valuing

Stage One Theory of Knowledge Use Dominant assumption was “naïve instrumentalism” Effective programs would be maintained or scaled-up and ineffective ones would be eliminated (i.e., a rationalist view) Use would occur almost naturally – Evaluation findings, being scientifically rigorous, would inform decision making

Stage One Theory of Evaluation Practice Emphasis on assessing a program’s effectiveness at solving important social problems Primary task is summative claims or judgments – Methods of arriving at summative statements were given priority Distance is crucial to scientific rigor and integrity

“Bad is bad and good is good and it is the job of evaluators to decide which is which” — Michael Scriven

Biographical Sketch Born in 1928, in Bealieu, Hampshire, England Ph.D. in Philosophy, Oxford University M.A. in Mathematics and Philosophy, Melbourne University Former President of AERA and AEA Author or more than 300 publications Recipient of numerous awards and prizes

Scriven’s View of Evaluation Scriven argues that a singular statement of value is the sine qua non of evaluation and is the primary task of evaluators Evaluation is an autonomous discipline, and a vital component of all other disciplines

Scriven’s Influence Scriven is widely viewed as the first major evaluation theorist Whether one agrees or disagrees with his views, Scriven has directly and indirectly influenced nearly every other theorist and theory

“Not to know Scriven is to be illiterate in evaluation” — Thomas D. Cook

Scriven’s Major Contributions Definition of evaluation Valuing theory Formative and summative distinction Goal-free evaluation Causal logic and cause-probing methods Logic of evaluation Theory-free evaluation Metaevaluation Key Evaluation Checklist

Goal-Free Evaluation Premised on examining what effects a program is actually having rather than intended effects Aimed at bias reduction through avoiding knowledge of program goals Relies on needs assessment to determine the extent to which a program meet needs

Causal Logic and Reasoning Modus operandi method (MOM) and general elimination method (GEM) for causal analysis Causal claims ‘beyond a reasonable doubt’ should be the standard within and outside of science, as opposed to those supported by ‘the balance of the evidence’ Overdetemination problem

Theory-Free Evaluation The role of evaluators is to determine only whether programs work, not to explain how they work To attempt explanations is beyond the capability of most evaluators Requires the correct (not just the believed) logic or theory of the program

Metaevaluation An evaluation of an evaluation Internal and external quality control Identify strengths, weaknesses, and uses Appropriate criteria – Validity – Credibility – Utility – Cost-effectiveness – Ethicality

Key Evaluation Checklist Organized into four parts with 15 checkpoints – Part A: Preliminaries – Part B: Foundations – Part C: Subevaluations – Part D: Conclusions & Implications

Scriven’s Theory of Social Programming Social programs exist to meet important societal needs – Consumer needs should be given the same primacy in an evaluation Good programs should be expanded and poor programs should be abandoned

“If I had asked people what they wanted, they would have said faster horses” — Henry Ford

Scriven’s Theory of Knowledge Construction Realist philosophy and ideology Valid knowledge is possible, but nearly always incomplete Practical inference rather than the development of scientific theories about social programs Socio-organizational controls to reduce bias in knowledge claims

Scriven’s Theory of Valuing Prescriptive theory premised on some values having higher priority than others Valuing is no less scientific in its validity than “value-free” science Relative valuing, rather than absolute valuing, is preferable

Scriven’s Theory of Knowledge Use Use is “self-evident” “Naïve instrumentalism” Primarily concerned with summative statements emphasizing short-term instrumental use

“Evaluation is what it is, the determination of merit and worth, and what it is used for is another matter” — Michael Scriven

Scriven’s Theory of Evaluation Practice Methodologically pluralistic Audience of evaluation is predominately “consumers” Evaluations should serve the “public good” Emphasizes bias minimization through socio-organizational controls

Encyclopedia Entries Bias Causation Checklists Chelimsky, Eleanor Conflict of Interest Countenance Model of Evaluation Critical Theory Evaluation Effectiveness Efficiency Empiricism Independence Evaluability Assessment Evaluation Use Fournier, Deborah Positivism Relativism Responsive evaluation Stake, Robert Thick Description Utilization of Evaluation Weiss, Carol Wholey, Joseph