Presentation on theme: "UNWOMEN/IPEN TRANSFORMATIVE MIXED METHODS EVALUATION: DAY 3 MIXED METHODS Prof. Donna M. Mertens Gallaudet University Almaty, Kazakhstan July 2011."— Presentation transcript:
UNWOMEN/IPEN TRANSFORMATIVE MIXED METHODS EVALUATION: DAY 3 MIXED METHODS Prof. Donna M. Mertens Gallaudet University Almaty, Kazakhstan July 2011
Three days together IPEN Almaty Kazakhstan July 2011 Mertens Mixed Methods Day 1: What is the role of the evaluator? Overview of evaluation (Terms of Reference) Transformative paradigm Diversity & social justice; privilege Cultural competence Day 2: Quantitative & qualitative approaches Day 3: Mixed methods approaches
Methods Options Quantitative approaches such as experimental, quasi- experimental, causal-comparative, correlational, survey, and single-case designs Qualitative approaches such as group processes (e.g., focus groups or some indigenous methods), case studies, ethnographic research, phenomenological research, and PAR Gender analysis is a mixed-methods approach that provides a framework for transformative research and evaluation that has potential for transfer to other groups that experience discrimination. Mixed methods are most likely to be the approach of choice because of the need to integrate community perspectives into the inquiry process, thus necessitating collection of qualitative data during the research or evaluation process. (Mertens, 2009, TRE, p. 165). IPEN Almaty Kazakhstan July 2011 Mertens Mixed Methods
Mixed Methods Decisions Compatibility: Match the purpose, focus, questions, and design. Timing: Determine the temporal relationship between the quantitative and qualitative data collection. Weight: Establish priority or emphasis of the qualitative and quantitative approaches in the study. Mixing: Determine when quantitative and qualitative strategies/data will be mixed in the process of evaluation. IPEN Almaty Kazakhstan July 2011 Mertens Mixed Methods
Concurrent Design Quantitative and Qualitative occur more or less simultaneously QualitativeQuantitative Sequential Design: Quantitative Followed by Qualitative OR Sequential Design: Qualitative Followed by Quantitative Quantitative Qualitative Mertens, 2009, TRE, p. 167 IPEN Almaty Kazakhstan July 2011 Mertens Mixed Methods
Example: Making Visible Botswana youth: addressing power inequities in the fight against HIV/AIDS using a transformative lens IPEN Almaty Kazakhstan July 2011 Mertens Mixed Methods
Transformative Mixed Methods Design IPEN Almaty Kazakhstan July 2011 Mertens Mixed Methods Stage 4 Concurrent Stage 3 Sequential Stage 2 Concurrent Stage 1 Qual Assemble team; read documents; engage in dialogues Preliminary studies: youth, older men Process eval Pilot intervention: Observations, Interviews, Surveys Demographic information; Surveys; Incidence data Pretest: Knowledge, Attitude, Behavior; Post tests: Quant Qual; Behavior & Policy Change; Transfer To other contexts
Transformative Design Components: Post-Colonial Critical Ethnography A statement of your evaluation problem or questions (focus). A description of your data-collection methods, including interviewing, journaling, and coding processes, and how these will be accomplished with the evaluator as a co-performer in the field or participant observer. An explanation of your ethical methods and how the welfare of the participants will be put first by protecting their rights, interests, privacy, sensibility, and offering reports at key stages to them, including the final report. A description of the participants in terms of population, geographic location, norms and rules, significant historical and cultural context, and expectations for key informants. A time frame for entering the field, collecting the data, departing from the field, coding and analysis and completion of the written report, and/or public performance. Use of a critical theoretical framework in the design, implementation, and dissemination of the study. Madison (2005) IPEN Almaty Kazakhstan July 2011 Mertens Mixed Methods
PAR and Transformative Design 1. The group decides on the focus and questions for the evaluation. 2. Evaluators and participants observe, engage in action, observe and record. 3. Evaluators and participants immerse themselves in action and elaborate and deepen their understandings. 4. Group members reassemble and share their knowledge, using this iteration as an opportunity to revise their plans for the next cycle ofevaluation. 5. This cycle might be repeated between 6 and 10 times depending on the complexity of the evaluation context. (Heron & Reason, 2006). IPEN Almaty Kazakhstan July 2011 Mertens Mixed Methods
Transformative MM and PAR Greenwood and Levine (2007) summarize: “Surveys, statistical analyses, interviews, focus groups, ethnographies, and life histories are all acceptable, if the reason for deploying them has been agreed upon by the AR collaborators and if they are used in a way that does not oppress the participants” (p. 6). IPEN Almaty Kazakhstan July 2011 Mertens Mixed Methods
Questions for Thought: Design What is your reaction to the idea that one group will receive an intervention and another group will not? Under what circumstances would you accept that format as an ethical course of action? What other alternatives are possible? (Mertens, 2009, TRE, p. 197) IPEN Almaty Kazakhstan July 2011 Mertens Mixed Methods
Gender Analysis and MM IPEN Almaty Kazakhstan July 2011 Mertens Mixed Methods
Outcomes, Outputs & Impacts Inputs —> Activities —> Outputs —> Outcomes —> ImpactsOutputsOutcomes Impacts Outputs are the products, services, and capacities that result from the completion of activities Outcomes are the intended or achieved short-term and medium-term effects of an intervention’s outputs, usually requiring the collective effort of partners. Impacts are the long term effects or change to which the programme, through collective effort with partners, will contribute. GE/HR evaluation: how did the programme contribute to GE and HR?
INDICATORS Indicators are the means by which you determine progress towards a result or whether an expected result has been achieved. Developed for ALL levels of results – outputs, outcomes & impact Indicators measure (quantitatively or qualitatively) the status of an expected result 15 Module 2-1 Source: UNIFEM RBM training IPEN Almaty Kazakhstan July 2011 Mertens Mixed Methods
INDICATORS Quantitative: focus on numbers & counting (percentage of women and men in parliament, male and female wage rates, school enrolment for girls and boys) Qualitative: capture opinions, attitudes and feelings (nature of dissatisfaction, extent of increased awareness) 16Module 2-1 IPEN Almaty Kazakhstan July 2011 Mertens Mixed Methods
EXAMPLES Quantitative - Can be directly counted & expressed as a number % of … # of … Frequency of … Ratio of … Amount of … Timeliness of … Qualitative - Involves perception (can be expressed quantitatively or as narrative) Level of Satisfaction with … Knowledge of … Ability to … Appropriateness of... Importance of … Use and usefulness of … 17Module 2-1 Source: UNIFEM RBM Training. IPEN Almaty Kazakhstan July 2011 Mertens Mixed Methods
GENDER SENSITIVE INDICATORS Measures gender-related changes in society over time Includes sex-disaggregated indicators that provide separate measures for women and men Indicators that are gender-specific to either women or men 18Module 2-1 Source: A. Moser Gender and Indicators IPEN Almaty Kazakhstan July 2011 Mertens Mixed Methods
HUMAN RIGHTS INDICATORS Assess both state progress in guaranteeing human rights and individual programme adherence and promotion of rights. HR indicators should measure: Improvements in the capacities of rights holders and duty bearers to realize rights Improvements in the enjoyments of rights 19Module 2-1 IPEN Almaty Kazakhstan July 2011 Mertens Mixed Methods
GE & HR RESPONSIVE EVALUATION IPEN Almaty Kazakhstan July 2011 Mertens Mixed Methods Follow-Up and Use : Dissemination strategies should make evaluation findings accessible and barrier-free to women, including both RHs and DBs Targeting women’s organizations/networks and knowledge networks User-friendly language Stakeholder workshops that include women and other groups subject to discrimination (RHs &DBs) Management Response should be issued to ensure follow-up on key gender and human rights issues
IPEN Almaty Kazakhstan July 2011 Mertens Mixed Methods EXAMPLE INDICATORS Outcome indicator: Evidence of changes in the capability of ministries of xxx to formulate and implement policies responsive to indigenous women Output indicator: Capacity assessments (focusing on individual competencies within ministries of xxx) indicating increase in relevant knowledge, skills and attitudes within key public institutions
Examples: GE/HR Indicators proportion of target group by sex aware of the benefits of birth registration, employment to population ratio by age and sex, gross primary graduation ratio by sex, percentage of women in parliament.
“REALITY CHECK” IPEN Almaty Kazakhstan July 2011 Mertens Mixed Methods Expense and making use of available data; are there too many indicators? Unrelenting pull to quantitative measures - need a balance of qualitative and quantitative data Proxy indicators, where measurement takes an more indirect path Indicators should be disaggregated, as much as possible, by sex, race, ethnicity, age, geographic area Indicators do not exist in a vacuum and must be tied to a result.
EXERCISE: DEVELOP INDICATORS IPEN Almaty Kazakhstan July 2011 Mertens Mixed Methods two indicators Develop two indicators (one qualitative and one quantitative) for the course example.
Sample Relevance Evaluation ?’s How well do the programme objectives target the identified rights and needs of male and female beneficiaries? What rights does the programme advance under CEDAW, the Millennium Development Goals and other international development commitments?Goals
Sample Evaluation Effectiveness ?’s To what extent have the objectives been achieved, and do the intended and unintended benefits meet fairly the needs of disadvantaged women? To what extent have the capacities of duty-bearers and rights-holders been strengthened?
Sample Evaluation Efficiency ?’s Could the activities and outputs have been delivered with fewer resources to the target populations without reducing their quality and quantity? outputs How has the programme maximized partnerships in the delivery of the programme? Have UN Woman’s organisational structure, managerial support and coordination mechanisms effectively supported the delivery of the programme?
Sample Evaluation Impact ?’s To what extent have efforts been successful in stopping harmful and discriminatory practices against women? What is the evidence that the programme enabled the rights-holders to claim their rights more successfully and the duty-holders to perform their duties more efficiently?
Sample Evaluation Sustainability ?’s Is the programme supported by national and local women’s organisations? Do these organisations demonstrate leadership commitment and technical capacity to continue to work with the programme or advocate for change?
Additional GE/HR ?’s Equality and non-discrimination: Did the programme benefits affect equally men and women? Empowerment: Did the budget designate sufficient resources and level of effort to address the inclusion of disadvantaged or marginalised groups? Accountability: Were monitoring data (disaggregated according to relevant criteria such as gender, age, ethnicity, location, and income) collected and used to adjust implementation? Social transformation: Does the intervention’s theory of change include attention to GE and HR? Participation and inclusion: Did the implementation make systematic and appropriate efforts to include women and men, and/or reach out to disadvantaged groups? UN Women 2011
GE/HR Evaluation Methods Avoid bias: selection of data collection methods & in sampling methods Potential biases: Gender & Power (sources able to contribute more easily because of privacy & confidentiality issues ), Class or distance (favoring the more accessible) Plans for how to include marginalized groups Mixed methods: Quan & Qual
Rigor applied to methods choices IPEN Almaty Kazakhstan July 2011 Mertens Mixed Methods
In the positivist tradition, there are two important tests of knowledge claims: 1. Is the knowledge claim true in this situation? Are the changes observed in the dependent variable due to the effect of the independent variable? (Internal Validity) 2. Is the knowledge true in other situations? Generalizability? (External Validity) IPEN Almaty Kazakhstan July 2011 Mertens Mixed Methods
Comparative Studies: Rigor Causal relationship assumed? Competing explanations? Comparable groups in causal comparative? Third variable cause both predictor and criterion variables? Sub groups analysis? Correlational: ordering of variables? Predictive studies: Other screening variables? Level of.8 or better? Reliability and range of variables IPEN Almaty Kazakhstan July 2011 Mertens Mixed Methods
Criteria that establish rigor in qualitative methods Credibility: just like validity in qualitative research – asks if there is a correspondence between the way the respondents actually perceive social constructs and the way the evaluators portray their viewpoints. IPEN Almaty Kazakhstan July 2011 Mertens Mixed Methods
(a) persistent observation (b) peer debriefing (c) progressive subjectivity (d) member checks (e) triangulation (f) transferability (g) dependability (h) confirmability (i) authenticity and fairness Criteria that establish rigor in qualitative methods IPEN Almaty Kazakhstan July 2011 Mertens Mixed Methods
TRANSFORMATIVE EVALUATION: Rigor IPEN Almaty Kazakhstan July 2011 Mertens Mixed Methods Emphasizes Human Rights and Social justice Analyses asymmetric power relations Advocates culturally competent relations between the evaluator and community members Employs culturally appropriate mixed methods tied to social action Applies feminist theory, critical race theory, postcolonial and indigenous theories Mertens (2009) Transformative Research and Evaluation. The Guilford Press.
Authenticity (Transformative criteria) Fairness: The evaluator presents all value differences, views, and conflicts. Ontological Authenticity: An individual’s or groups’ conscious experience of the world became more informed and sophisticated. Catalytic Authenticity: Action was stimulated by the inquiry process. IPEN Almaty Kazakhstan July 2011 Mertens Mixed Methods
Power and Privilege IPEN Almaty Kazakhstan July 2011 Mertens Mixed Methods How do we understand the dynamics of power when participatory methods are employed by the powerful? Whose voices are raised and whose are heard? How are these voices mediated as issues of representation become more complex with the use of participatory methods in larger-scale planning and consultation exercises? (Mertens, 2009, P. 85) What if I am a member of the community? How does that prepare me to work in that community? What if I am not a member of a community? To what extent is it necessary to share salient characteristics of a community? How does cultural competency come into the discussion of interactions in evaluation contexts? (Mertens & Wilson, in press)
Criteria for Rigor: Utilization IPEN Almaty Kazakhstan July 2011 Mertens Mixed Methods Dissemination Use Management Response Engagement with Stakeholders
Dissemination Purposes Transparency and accountability Informing and improving an organisation’s work Sharing good practices and ‘how to’ to advance women’s human rights Sharing lessons with partners on the ground and building their capacity, and Generating knowledge on how gender equality, women’s empowerment and human rights can be advanced
Dissemination Language and presentation of the report: use graphs or pictures, and be written in a manner that is gender and culturally sensitive. Translation: anticipate the need for translation when planning the evaluation so time and budget can be included) Making the report public: Within the UN, there is requirement to make all evaluations public (UNEG Norms and Standards) and in UN Women, this is accomplished by having the Evaluation Unit post the evaluation report in the Evaluation Resource Center (which is publicly accessible). Reaching target audiences in a user-friendly way: lower rates of literacy? The manager needs to be creative in using other dissemination channels, suited to different audiences.
Dissemination Strategies Brochures outlining key evaluation lessons and recommendations Annual reports Articles in technical or organisational newsletters News releases Press conferences Media appearances Public meetings, public debates or town halls Seminars, workshops, and informal group discussions Electronic media ( , websites, blogs etc.) Meeting with community leaders, one on one.
Challenges to Use Lack of consensus on recommendations & required action Those who don’t like the results attack the process Dissemination is minimal: no funds, interest or time No follow-up process
Formal Management Response presentation of the response, action or non action to evaluation recommendations and lessons learned and follow-up or tracking mechanisms. engage with stakeholders to reflect on the evaluation process, findings, recommendations and lessons learned. Involve reference group in developing the management response, with the manager.
Agenda for Action How does the information you learned here apply to the work that you are/will be doing in evaluation? IPEN Almaty Kazakhstan July 2011 Mertens Mixed Methods
Resources Mertens, D. M. & Wilson, A. (in press). Program Evaluation. NY: Guilford. Mertens, D. M. (2010). Research and evaluation in education and psychology: Integrating diversity with qual, quant and mixed methods. 3 rd ed. Thousand Oaks, CA: Sage. Mertens, D. M. (2009). Transformative research & evaluation. NY: Guilford Mertens, D. M. & Ginsberg, P. (2009).(Eds.) Handbook of Social Research Ethics. Thousand Oaks, CA: Sage. IPEN Almaty Kazakhstan July 2011 Mertens Mixed Methods
Contact information IPEN Almaty Kazakhstan July 2011 Mertens Mixed Methods Donna M. Mertens, Gallaudet University