1. Transformative Mixed Methods in Evaluation Donna M. Mertens Gallaudet University Katrina L. Bledsoe Walter R. McDonald & Associates, Inc. AEA/CDC 2010 Summer Evaluation Institute Atlanta GA, June 15 & 16, 2010 © No part of this presentation can be duplicated without consent from the author.

2. Advanced Organizer Role of the evaluation in an unequal world Role of the evaluation in an unequal world Transformative paradigm places priority on social justice & human rights Transformative paradigm places priority on social justice & human rights Rationale Rationale Paradigm as driving force Paradigm as driving force Transformative Paradigm Transformative Paradigm Assumptions Assumptions Dimensions of diversity Dimensions of diversity Methodological framework Methodological framework Criteria for rigor Criteria for rigor

3. Introduction/Expectations- Name, position, organization (no acronyms please) Name, position, organization (no acronyms please) Where do you come from? Where do you come from? Where are you now with regard to evaluation in communities as multicultural settings? Where are you now with regard to evaluation in communities as multicultural settings? What do you want to get out of this workshop? What do you want to get out of this workshop? Why did you decide to come here? Why did you decide to come here? Describe your experiences in research with communities Describe your experiences in research with communities

4. Exercise-What is the role of the Evaluator You know the process called “free association”? Imagine someone asks you – what is the role of the evaluator? What would be the first few thoughts that would come to mind?

5. Discussion of Some Possible Evaluator Roles Advocate Advocate Social Change Agent Social Change Agent Representative of Community Values Representative of Community Values Critical Friend Critical Friend Resident Truth-teller Resident Truth-teller

6. Steps in Transformative Paradigm 1. Identify your worldview (paradigm) 2. If transformative, identify community members to involve 3. Develop mechanisms for working together to identify research focus & research questions 4. Develop a rationale and write a mixed methods purpose statement 5. Develop a design for mixed methods 6. List your qual and quant data to be collected 7. Review your qual and quant data analysis 8. Draw a diagram of procedures in the design

7. Cyclical model for Transformative Research (Mertens, in press) Community Participation

8. Paradigms: Your world view Basic beliefs about ethical action (axiology) ethical action (axiology) reality (ontology) reality (ontology) relation between knower and would-be- known (epistemology) relation between knower and would-be- known (epistemology) the appropriate approach to systematic inquiry (methodology) the appropriate approach to systematic inquiry (methodology) With acknowledgement to the ground breaking work of Guba & Lincoln

9. Transformative Paradigm Axiology: Respect for cultural norms; support for human rights and social justice Axiology: Respect for cultural norms; support for human rights and social justice Ontology: Ontology: Rejects cultural relativism. Recognizes privilege given to what is perceived to be real based on: social, political, cultural, economic, ethnic, gender, and disability positionality Epistemology: Epistemology: Interactive link; knowledge is socially and historically located; trusting relationship historically located; trusting relationship Methodology: Methodology: Qualitative/dialogic; Quantitative mix; Context

10. Ethics (Axiology): Ethnic Categorization Prenatal Program in New Jersey Prenatal Program in New Jersey Groups collapsed into one, although it was noted that there were several distinct cultural groups. Groups collapsed into one, although it was noted that there were several distinct cultural groups. Refusal to speak with groups to determine what was needed for program Refusal to speak with groups to determine what was needed for program Services then discontinued when not used Services then discontinued when not used

11. Understanding Privilege Knowing one’s self (sense of self) Knowing one’s self (sense of self) Knowing one’s status in the world Knowing one’s status in the world Acknowledging one’s privilege Acknowledging one’s privilege

12. Social Justice & Dimensions of Diversity Gertrude Stein: Rose is a rose is a rose… Gertrude Stein: Rose is a rose is a rose… Myth of homogeneity (Mertens, 2005) Myth of homogeneity (Mertens, 2005) Sampling with dimensions of diversity in mind: Who needs to be included? How can people be included in the most appropriate way? Sampling with dimensions of diversity in mind: Who needs to be included? How can people be included in the most appropriate way? But, is a pregnant African American woman a pregnant African American woman? But, is a pregnant African American woman a pregnant African American woman? Is a person with a disability a person with a disability a person with a disability? (Mertens & McLaughlin, 2004) Is a person with a disability a person with a disability a person with a disability? (Mertens & McLaughlin, 2004)

13. Diversity and Sampling Strategies Example: Deaf/HH court access Communication - languages and modes: Communication - languages and modes: American Sign Language; highly educated American Sign Language; highly educated ASL; limited education ASL; limited education Gesture/pantomime/limited signing/low literacy Gesture/pantomime/limited signing/low literacy Deaf/blind Deaf/blind Hard of hearing people with assistive listening devices Hard of hearing people with assistive listening devices Oral deaf adults Oral deaf adults Mexican sign language Mexican sign language Other dimensions of diversity: gender, race/ethnicity; status with court Other dimensions of diversity: gender, race/ethnicity; status with court

14. Case study What dimensions of diversity are relevant in the case study? What dimensions of diversity are relevant in the case study? How would you know? How would you know? How would you invite people to participate? How would you invite people to participate? How would you support them in culturally appropriate ways? How would you support them in culturally appropriate ways?

15. We…need enhanced understandings of related systemic processes of asymmetric power relations and privilege, not simply awareness and knowledge of difference and diversity… Hazel Symonette (2004). How and to what extent is sociocultural diversity associated with patterned differences in access, resource opportunities, and life chances?” (p. 108)

16. Example of Mixed Method Transformative Approach Trenton Obesity Prevention Study Trenton Obesity Prevention Study 1. Identified world view of evaluators and partners 2. Identified community members to involve 3. Developed process to work with one another 4. Redesigned original research proposal to reflect mixed-methods approach. 5. Provided new model of research design and program

17. Power issues & Methodological Assumptions Determination of the research focus, planning, implementation and use from a transformative ethical stance Determination of the research focus, planning, implementation and use from a transformative ethical stance Power can be reframed in transformative research Power can be reframed in transformative research

18. Power Issues in the TOPS Project Articulated power structure Articulated power structure Articulated standards of that power structure Articulated standards of that power structure Articulated the assumptions of what quality research would look like within that power structure Articulated the assumptions of what quality research would look like within that power structure Versus the reality of what would be most appropriate for the situation Versus the reality of what would be most appropriate for the situation

19. Methodology Applied to Case Study What kind of methodological approach would you use in conducting a transformative evaluation? What kind of methodological approach would you use in conducting a transformative evaluation? How would you address issues of power in the evaluation? How would you address issues of power in the evaluation?

20. Transformative Criteria Experiential Experiential Consequential Consequential Interpersonal Interpersonal Ontological Ontological Catalytic Catalytic Critical reflexivity Critical reflexivity Reciprocity Reciprocity (first three criteria are elaborated on in Kirkhart, 2005; others are found in Mertens, 2005; and Lincoln, 2009)

21. Agenda for Action How will you modify your own evaluation to incorporate ideas from this workshop?

22. Resources Mertens, D. M. (2010). Research and evaluation in education and psychology: Integrating diversity with qual, quant and mixed methods. 3 rd ed. Thousand Oaks, CA: Sage. Mertens, D. M. (2010). Research and evaluation in education and psychology: Integrating diversity with qual, quant and mixed methods. 3 rd ed. Thousand Oaks, CA: Sage. Mertens, D. M. (2009). Transformative research & evaluation. NY: Guilford Mertens, D. M. (2009). Transformative research & evaluation. NY: Guilford Mertens, D. M. & Ginsberg, P. (2009).(Eds.) Handbook of Social Research Ethics. Thousand Oaks, CA: Sage. Mertens, D. M. & Ginsberg, P. (2009).(Eds.) Handbook of Social Research Ethics. Thousand Oaks, CA: Sage.

23. Contact information Donna M. Mertens, Gallaudet University Donna M. Mertens, Gallaudet University Donna.Mertens@Gallaudet.edu Donna.Mertens@Gallaudet.edu Donna.Mertens@Gallaudet.edu Katrina L. Bledsoe, Walter R. McDonald & Associates, Inc. Katrina L. Bledsoe, Walter R. McDonald & Associates, Inc. KBledsoe@wrma.com KBledsoe@wrma.com KBledsoe@wrma.com