1. EVAL 6000: Foundations of Evaluation Dr. Chris L. S. Coryn Nick Saxton Fall 2014

2. Agenda American Evaluation Association (AEA) conference experience(s) Improvement- and accountability- oriented evaluation approaches Social agenda and advocacy evaluation approaches Eclectic approaches

3. NOTE: If we do not get through all of this material tonight we will continue next week…

4. AEA experience

5. What was your most ‘profound’ experience at the conference? What did you learn (or not learn that you expected to)? What did you like most? What did you like least?

6. Improvement- and accountability-oriented approaches

7. (Ideally) are expansive and cover all aspects related to merit, worth, and significance of a program or other type of evaluand Intended to inform program improvement and/or decision making (i.e., accountability) Extremely difficult to implement in practice – Stufflebeam – Scriven

8. Approach 15: Decision- and accountability-oriented studies Advance organizers – Relevant decision makers, decisions to be made, accountability requirements, etc.. Purposes – Decision making—whether formative or summative Sources of questions – Stakeholders (typically limited) Questions – How can the evaluand be improved? – Does the evaluand meet accountability requirements?

9. Methods – Any relevant method for determining how to improve a program or provide information regarding accountability Pioneers – Lee Cronbach, Daniel Stufflebeam Use considerations – Generally directed toward program management and staff Strengths – Scope Weaknesses – Emphasis on program management and staff

10. Approach 16: Consumer- oriented studies Advance organizers – Complex conceptions of ‘values’ (i.e., criteria) Purposes – Typically, but not always, summative – Extent to which consumers’ needs are met Sources of questions – Relevant sources of values (see Scriven’s Key Evaluation Checklist [KEC]) Questions – How well does the evaluand meet consumers’ needs?

11. Methods – Any relevant method for determining how well consumers’ needs have been met Pioneers – Michael Scriven Use considerations – Emphasis is on instrumental use Strengths – Independence Weaknesses – Independence

12. Approach 17: Accreditation and certification Advance organizers – Accreditation or certification requirements Purposes – Whether accreditation or certification requirements are met Sources of questions – Accreditation or certification requirements Questions – Have accreditation or certification requirements been met?

13. Methods – Any relevant method for determining how well consumers’ needs have been met Pioneers – Various Use considerations – Accordance to accepted standards Strengths – For making informed judgments Weaknesses – Emphasis on inputs and processes

14. Social agenda and advocacy approaches

15. Predominately aimed at increasing social justice Giving power to marginalized/disenfranchised groups Tend to favor ‘qualitative’ methods and a constructivist/constructionist perspective Stress cultural pluralism, relativism, and multiple realities Only a small emphasis on determining merit, worth, or significance

16. Approach 18: Responsive or stakeholder-centered evaluation Advance organizers – Stakeholders’ concerns Purposes – Varies widely, depending on stakeholders’ needs Sources of questions – Stakeholders Questions – What has the program achieved? – How well has the program been implemented? – What do experts ‘think’ about the program?

17. Methods – Most often qualitative Document analysis Observations Interviews Pioneers – Robert Stake Use considerations – Primary stakeholders Strengths – Intended to address stakeholders’ concerns Weaknesses – External credibility

18. Approach 19: Constructivist evaluation Advance organizers – Rejects the notion that merit, worth, and significance can be ‘objectively’ determined Purposes – To ‘construct’ stakeholders’ experiences Sources of questions – Pluralistic values of stakeholders Questions – Emergent

19. Methods – Predominately qualitative Pioneers – Evon Guba, Yvonne Lincoln Use considerations – Often conflicting accounts and no definitive judgments Strengths – Stakeholder involvement Weaknesses – Time, costs, and—most importantly— relativist results

20. Approach 20: Deliberative democratic evaluation Advance organizers – Democratic participation – Dialogue – Deliberation Purposes – Democratic participation Sources of questions – Stakeholders/evaluator(s) Questions – What is the merit and/or worth of the program?

21. Methods – Any relevant methods for gathering evidence Pioneers – Ernie House Use considerations – Should represent interests of all relevant stakeholders Strengths – Democratic participation Weaknesses – Ambitious demands required to execute the approach

22. Approach 21: Transformative evaluation Advance organizers – Social justice and equity Purposes – Recognizing ‘situational’ nature of knowledge claims Sources of questions – Least advantaged groups of stakeholders Questions – None that can be identified in advance

23. Methods – Generally mixed-method Pioneers – Donna Mertens Use considerations – Inclusion of marginalized stakeholders Strengths – Emphasis on social justice Weaknesses – External credibility

24. Eclectic evaluation approaches

25. No particular philosophical or methodological orientation Draw from various evaluation concepts and methods to secure ‘useful’ evaluation findings Tend to invoke the Program Evaluation Standards

26. Approach 22: Utilization- focused evaluation Advance organizers – Potential users and uses Purposes – Information necessary for intended uses by intended users Sources of questions – Intended users, often determined through situational analysis Questions – Specific questions articulated by intended users

27. Methods – Any relevant method for addressing intended users questions Pioneers – Michael Patton Use considerations – Situational, and tailored to intended users intended uses Strengths – Maximizes evaluation impact Weaknesses – Too much user input and involvement

28. Approach 23: Participatory evaluation Advance organizers – Promoting buy-in and use Purposes – Tends to be directed toward program improvement Sources of questions – Intended users and other stakeholders Questions – Specific questions articulated by intended users and other stakeholders

29. Methods – Any relevant method for addressing intended user and stakeholder questions Pioneers – Brad Cousins Use considerations – Situational, and tailored to intended users intended uses Strengths – User-friendly Weaknesses – Sometimes, poor technical quality