1. EVAL 6000: Foundations of Evaluation Dr. Chris L. S. Coryn & Carl D. Westine October 14, 2010

2. Stage 3 Theories Stage 3 theories represent an amalgamation of earlier theories (but still incomplete) –During this period theorists recognized the need for contingency-based approaches to evaluation –Emphasis was on specifying the conditions under which and for which purposes different practices are sensible –Less emphasis on describing programs, more on explanation –Recognition (mostly) that evaluation (mostly) can contribute to incremental changes

3. Stage 3 Theories Criterion 1: Theory of Social Programming –Stage 1 theorists operated on “naive” assumptions (i.e., that radical changes would be made as a result of evaluation) –Stage 2 theories and theorists recognized the “political” aspects of evaluation and directed efforts toward “incremental change” (Weiss, in her later work, reverted to more radical ideologies) –Stage 3 theories recognized all of the same problems as Stage 1 and 2, but with an eye toward using evaluation as a means for making generalizations about programs

4. Stage 3 Theories Criterion 2: Theory of Use –In Stage 1, use was largely ignored (i.e., it would occur naturally with truth) –In Stage 2, theories of use (by whom? For what?) became the dominant concern of theories and theorists –In Stage 3, while still emphasizing instrumental use, also promoted enlightenment as a means for summarizing what is known about a problem and potential solutions for policy making Enlightenment was largely the domain of experts (i.e., academics)

5. Stage 3 Theories Criterion 3: Theory of Knowledge Construction –Stage 1 theories prioritized truth, above all else –Stage 2 theories from (just) truth to a pragmatic position including knowledge that is both truthful and useful –Stage 3 theories assert that no single paradigm for knowledge construction has sufficient empirical or theoretical support to dominate Priority includes program description, causation, explanation, and generalization As with Stage 2, methodological pluralism was advocated (though theorists of this period were critical of some methods of inquiry/discovery)

6. Stage 3 Theories Criterion 4: Theory of Valuing –Stage 1 theories included both prescriptive and descriptive valuing –Stage 2 theories predominately advocated descriptive valuing Valuing is not a task for evaluators given the complexities involved in valuing as well as the complexities involved in decisions about social programs –Stage 3 theories advocated a “middle ground” that integrated both prescriptive and descriptive valuing and sources of values Even so, single, summative statements of value are still not seen as the task of evaluation given competing stakeholder interests and value positions

7. Stage 3 Theories Criterion 5: Theory of Practice –Stage 1 theories emphasized assessing programs effectiveness at solving important social problems Primary task is summative claims or judgments Eliminating bias was paramount –Stage 2 theories emphasized methodological pluralism, although Wholey and Stake assigned central methods to practice –Stage 3 emphasized contingencies under which certain methods would produce certain results Still a strong emphasis on causal knowledge and generalization Also an emphasis on the mechanisms by which programs produce results

8. Lee Cronbach & Peter Rossi Cronbach and Rossi advocated powerful theoretical positions (if accurate) that, to this day, have not been fully realized (and which are often misunderstood/misinterpreted) Independently and combined, their contributions represent radical departures from the positions and orientations of earlier theorists Although each drew extensively on the work of those who preceded them, both developed more extensive and sophisticated “theories” of program evaluation

9. Lee Cronbach Cronbach’s theory is complex, comprehensive, multifacteded, and more empirically-grounded than those before (and since) His theory is critical (and cynical) of social science theory, it methods, and how information derived through such methods should be used He seeks complex answers to complex problems, but with the intent of producing generalizable knowledge claims—disputed classic notions of validity Major interest was in individual differences (i.e., psychometrics)

10. Lee Cronbach Cronbach’s major contributions included, but are not limited to: –Measurement theory, in general, and reliability, in particular –Generalizability theory –Generalization through explanation –Bandwidth and fidelity –Aptitude-treatment interactions –Generalizations from and to utos UTOS *UTOS sub-utos

11. Peter Rossi Rossi’s theory emphasizes comprehensive and /or tailored evaluations with a focus on: –Program theories –Program life cycles or stages Eclectic integration or prior theorists work, but lacking detail about how and when to make decisions about tradeoffs (e.g., costs, time, expertise) Recognizes the legitimacy of different/equal values and needs –Emphasis is placed on needs as a primary source of criteria/values

12. Peter Rossi Rossi’s major contributions included, but are not limited to: –Comprehensive, tailored, and theory-driven evaluation –The “good enough” rule –The metallic and plastic laws of evaluation The “iron law”: The net impact of any program is 0 The “stainless steel law”: The better the evaluation, the greater the likelihood that the net impact is 0 The “copper law”: The more programs are designed to change individual, the more likely that the net impact is 0 The “plastic law”: Only those programs that fail are likely to be evaluated

13. Further Discussion of the Readings What specific questions were raised as you read the assigned readings for this week (or past weeks)? Is there something specific that you didn’t understand? What problems remain unresolved regarding Stage 3 theories? What other issues emerged from the readings that need further exploration?

14. Role Playing as Major Theorists For next week –Each of you has been assigned the role of a particular theorist (so, be certain to have a sound knowledge of your assigned theorist) –In playing that role you will be asked to discuss the following from the point of view of your assigned theorist: What is the purpose of evaluation? What is the role of the evaluator? Is it the job of evaluators to make sure that evaluations are used? If so, why and how? What is the relevance, if any, of a theory of social programming to evaluation? What type of knowledge claims are central to evaluation, and why?

15. Encyclopedia Entries for this Week Appreciative inquiry CIPP model Connoisseurship Deliberative democratic evaluation Empowerment evaluation Goal-free evaluation