EVAL 6000: Foundations of Evaluation Carl D. Westine & Dr. Chris L. S. Coryn December 2, 2010

Agenda The Program Evaluation Standards (45 min) –Overview of JCSEE and the standards –The “new” 30 standards –2 nd Edition vs. 3 rd Edition –Research on overlaps across standards –Activity on 3 rd Edition Sufflebeam and Coryn 2010 (10 min) Stufflebeam 2001 (10 min) Break (15 min) Activity: Hard Won Lessons (60 min)

The JCSEE The Joint Committee for Standards on Educational Evaluation was created in 1975, and currently oversees the maintenance and updates to the Program Evaluation Standards Made up of representatives from professional societies WMU folks involved in the JCSEE: Daniel Stufflebeam, James Sanders, Arlen Gullickson Currently Daniela Schröter is becoming more involved with the revision process (our point of contact)

What are Standards JCSEE defined an evaluation standard as a “principle commonly agreed to by experts in the conduct and use of evaluation, that when implemented will lead to greater evaluation quality” (JCSEE, 2010, p. 292) Focus on North America but other countries have been ada/opting these Other standards exist (GAO, Guiding Principles, etc.) For educational programs but… education is relied upon by everything

Early Standards The first edition was published in –Contained 30 standards –Defined the groups (Utility, Feasibility, Propriety, and Accuracy) Spearheaded by Dan Stufflebeam (WMU) and a collection of education and evaluation experts

Second Edition Standards The second edition was published in 1994 –Contained 30 standards –Maintained same group structure Achieved status of being accredited as official standards though the American National Standards Institute (ANSI) Update led by James Sanders (WMU)

Current Happenings Transition from the 2 nd Edition to the 3 rd Edition Third Edition is being published, and copies can be ordered through the JCSEE website: Don Yarrborough at the University of Iowa oversees the JCSEE and the development of the newest standards

The Latest Edition 3 rd Edition Took more than 5 years to develop and finalize (several delays in publishing) The next update has already begun –JCSEE continuously takes suggestions and feedback from users –Goal to update standards more frequently

Organization of the PES Book Applying the Standards Functional Table of Standards –Shows which standards are relevant at points along the evaluation continuum Standards –Group Overview/Scenario –Standard Statements –Rationale/Clarification (Overview) –Implementing (Guidelines) –Hazards (Common Errors) –Application(s) –Documentation

The Program Evaluation Standards The Program Evaluation Standards for educational evaluations Should be used for: –Guiding an evaluation effort (formative) –Assessing the quality of educational programs (summative) –Assessing the quality of an evaluation of an educational program (metaevaluation) –A tool to help policy makers understand evaluations –[Research on evaluation] (in my own opinion)

The Program Evaluation Standards There are now 5 categories totaling 30 individual standards Designed to ensure that an evaluation will… –Utility (7 -> 8): “… serve the information needs of intended users.” –Feasibility (3 -> 4): “… be realistic, prudent, diplomatic, and frugal.” –Propriety (8 -> 7): “… be conducted legally, ethically, and with due regard for the welfare of those involved in the evaluation, as well as those affected by its results.” –Accuracy (12 -> 8): “… reveal and convey technically adequate information about the features that determine worth or merit of the program being evaluated.” –Evaluation Accountability (0 -> 3): be well documented and held subject to internal and external evaluation (JCSEE, 2010).

Utility Standards U1 Evaluator Credibility Evaluations should be conducted by qualified people who establish and maintain credibility in the evaluation context. U2 Attention to Stakeholders Evaluations should devote attention to the full range of individuals and groups invested in the program and affected by its evaluation. U3 Negotiated Purposes Evaluation purposes should be identified and continually negotiated based on the needs of stakeholders. U4 Explicit Values Evaluations should clarify and specify the individual and cultural values underpinning purposes, processes, and judgments.

Utility Standards (cont.) U5 Relevant Information Evaluation information should serve the identified and emergent needs of stakeholders. U6 Meaningful Processes and Products Evaluations should construct activities, descriptions, and judgments in ways that encourage participants to rediscover, reinterpret, or revise their understandings and behaviors. U7 Timely and Appropriate Communicating and Reporting Evaluations should attend to the continuing information needs of their multiple audiences. U8 Concern for Consequences and Influence Evaluations should promote responsible and adaptive use while guarding against unintended negative consequences and misuse.

Feasibility Standards F1 Project Management Evaluations should use effective project management strategies. F2 Practical Procedures Evaluation procedures should be practical and responsive to the way the program operates. F3 Contextual Viability Evaluations should recognize, monitor, and balance the cultural and political interests and needs of individuals and groups. F4 Resource Use Evaluations should use resources effectively and efficiently.

Propriety Standards P1 Responsive and Inclusive Orientation Evaluations should be responsive to stakeholders and their communities. P2 Formal Agreements Evaluation agreements should be negotiated to make obligations explicit and take into account the needs, expectations, and cultural contexts of clients and other stakeholders. P3 Human Rights and Respect Evaluations should be designed and conducted to protect human and legal rights and maintain the dignity of participants and other stakeholders. P4 Clarity and Fairness Evaluations should be understandable and fair in addressing stakeholder needs and purposes.

Propriety Standards (cont.) P5 Transparency and Disclosure Evaluations should provide complete descriptions of findings, limitations, and conclusions to all stakeholders, unless doing so would violate legal and propriety obligations. P6 Conflicts of Interests Evaluations should openly and honestly identify and address real or perceived conflicts of interests that may compromise the evaluation. P7 Fiscal Responsibility Evaluations should account for all expended resources and comply with sound fiscal procedures and processes.

Accuracy Standards A1 Justified Conclusions and Decisions Evaluation conclusions and decisions should be explicitly justified in the cultures and contexts where they have consequences. A2 Valid Information Evaluation information should serve the intended purposes and support valid interpretations. A3 Reliable Information Evaluation procedures should yield sufficiently dependable and consistent information for the intended uses. A4 Explicit Program and Context Descriptions Evaluations should document programs and their contexts with appropriate detail and scope for the evaluation purposes.

Accuracy Standards (cont.) A5 Information Management Evaluations should employ systematic information collection, review, verification, and storage methods. A6 Sound Designs and Analyses Evaluations should employ technically adequate designs and analyses that are appropriate for the evaluation purposes. A7 Explicit Evaluation Reasoning Evaluation reasoning leading from information and analyses to findings, interpretations, conclusions, and judgments should be clearly and completely documented. A8 Communication and Reporting Evaluation communications should have adequate scope and guard against misconceptions, biases, distortions, and errors.

Evaluation Accountability Standards E1 Evaluation Documentation Evaluations should fully document their negotiated purposes and implemented designs, procedures, data, and outcomes. E2 Internal Metaevaluation Evaluators should use these and other applicable standards to examine the accountability of the evaluation design, procedures employed, information collected, and outcomes. E3 External Metaevaluation Program evaluation sponsors, clients, evaluators, and other stakeholders should encourage the conduct of external metaevaluations using these and other applicable standards.

Utility (Old vs. New) 2 nd Edition (OLD)3 rd Edition (NEW) U1Stakeholder IdentificationU1Evaluator Credibility U2Evaluator CredibilityU2Attention to Stakeholders U3Information Scope and SelectionU3Negotiated Purposes U4Values IdentificationU4Explicit Values U5Report ClarityU5Relevant Information U6Report Timeliness and Dissemination U6Meaningful Processes and Products U7Evaluation ImpactU7Timely and Appropriate Communicating and Reporting U8Concern for Consequences and Influences

Feasibility (Old vs. New) 2 nd Edition (OLD)3 rd Edition (NEW) F1Practical ProceduresF1Project Management F2Political ViabilityF2Practical Procedures F3Cost EffectivenessF3Contextual Viability F4Resource Use

Propriety (Old vs. New) 2 nd Edition (OLD)3 rd Edition (NEW) P1Service OrientationP1Responsive and Inclusive Orientation P2Formal AgreementsP2Formal Agreements P3Rights of Human SubjectsP3Human Rights and Respect P4Human InteractionsP4Clarity and Fairness P5Complete and Fair AssessmentP5Transparency and Disclosure P6Disclosure of FindingsP6Conflicts of Interest P7Conflict of InterestP7Fiscal Responsibility P8Fiscal Responsibility

Accuracy (Old vs. New) 2 nd Edition (OLD)3 rd Edition (NEW) A1Program DocumentationA1Justified Conclusions and Decisions A2Context AnalysisA2Valid Information A3Described Purposes and ProceduresA3Reliable Information A4Defensible Information SourcesA4Explicit Program and Context Descriptions A5Valid InformationA5Information Management A6Reliable InformationA6Sound Designs and Analysis A7Systematic InformationA7Explicit Evaluation Reasoning A8Analysis of Quantitative InformationA8Communication and Reporting A9Analysis of Qualitative Information A10Justified Conclusions A11Impartial Reporting A12Metaevaluation

Evaluation Accountability (Old vs. New) 2 nd Edition (OLD)3 rd Edition (NEW) A3Described Purposes and Procedures EA1Evaluation Documentation A12MetaevaluationEA2Internal Metaevaluation EA3External Metaevaluation

Significant Changes Managing focus (Process Use) –Project –Information Less emphasis on “actionable” statements Evaluation Accountability –New standard group –Transparency/documentation –Metaevaluation

Significant Changes (cont.) Combining of Standards –Quant/Qual into Design –Human Rights/Interactions into Respect –Clarity/Timeliness/Dissemination into Appropriate Communication Expansion of the drawn out examples/scenarios

Using the Standards Three principles to guide the use of the Program Evaluation Standards 1.Standards require adaptive, responsive, and mindful use –The user must discover how to apply them in each specific situation 2.Order of the Standard groups does not matter (contrary to what others say: Utilization-Focused, Accuracy) 3.In depth knowledge is required –Don’t just read the individual standards

Overlaps Across the Standards Functional Table of Standards begins the process of identifying standards that are related to each other a specific points in the evaluation process –How should we identify overlaps (dependency relationships) among the standards? –Could this enhance metaevaluation efficiency? –Do specific standards have more significance than others?

Activity Four Groups: Utility, Feasibility/Evaluation Accountablity, Propriety, and Accuracy. Identify the keywords in the standard statements. Example: U1 Evaluator Credibility “Evaluations should be conducted by qualified people who establish and maintain credibility in the evaluation context.”