1. Chapter 7: Evaluating Quality in Mixed Methods Publications
Purpose: Identify and illustrate different ways to evaluate the quality of a mixed methods research publication.

2. An Introduction to Fully Integrated Mixed Methods Research
Goals of the Chapter
Review controversies regarding ways to evaluate mixed method publications.
Identify key design features of the chapter exemplar.
Define the criteria in the Mixed Method Evaluation Rubric (MMER).
Illustrate how to use the MMER to evaluate the exemplary publication.
Review evidence about the link between design features and overall quality.
Consider challenges to using an evaluation rubric.
An Introduction to Fully Integrated Mixed Methods Research

3. Replicability: A Central Principle that Guides Research of All Kinds
The expectation that others using the same data and methods can duplicate the same results of a study and reach the same conclusions.
Requires a high degree of methodological transparency- Promotes replication by providing explicit details about the steps taken to complete data collection and data analysis, as well as in specifying which results came from the qualitative analysis, which came from the quantitative analysis, and which come from mixing.
An Introduction to Fully Integrated Mixed Methods Research

4. Controversies Regarding Quality in Mixed Method Research
Other than that mixing must be present, there is very little agreement about the features that distinguish high-quality MMR. publications.
The conventional assumption is that an evaluation framework requires three parts:
A part to evaluate the QUANT phase.
A part to evaluate the QUAL phase.
A part to evaluate mixing.
An Introduction to Fully Integrated Mixed Methods Research

5. An Introduction to Fully Integrated Mixed Methods Research
Examples of How Paradigmatic Assumptions Influence the Choice of Evaluation Criteria
Criteria Emphasized in a Realist Stance
Criteria Emphasized in a Transformative/Emancipatory Stance
Replicability
How systematically the research was executed.
How systematically the research is documented.
Consequential validity – the power of the research to influence change in policy or practice.
An Introduction to Fully Integrated Mixed Methods Research

6. Design Features of the Chapter Exemplar by McMahon, 2007
Rationale/Purpose
Complementarity
Priority
Qualitative
Timing of Data Collection
quant to QUAL to QUAL
Timing of Data Analysis
quant + QUAL, QUAL
Mixing
Mixing at sampling; QUAL QUAL mixing
McMahon, S. (2007). Understanding community-specific rape myths. Affilia: Journal of Women and Social Work, 22 (4),
An Introduction to Fully Integrated Mixed Method Research

7. Types of Mixing in the Chapter Exemplar by McMahon, 2007
Phase of the Research Process
Type of Mixing
Design/Research Question
Sampling
QUAL-QUANT samples overlap
Data Collection
QUANT to QUAL to QUAL
Data Analysis
No QUANT-QUAL mixing.
Drawing Conclusions/Inferences
A meta-inference was produced, but it is primarily from the QUAL.
An Introduction to Fully Integrated Mixed Method Research

8. An Introduction to Fully Integrated Mixed Methods Research
Mixing at the Inference Stage: Meta Inference in the Chapter Exemplar by McMahon, 2007
META INFERENCE: Both male and female athletes unconsciously avoided assigning blame to perpetrators of sexual violence for purposes of self-protection.
QUAL Conclusion: Both male and female athletes expressed views associated with victim blaming in subtle ways. For males, the concern was about false charges. For females, victim blaming provided a form of self-protection against feelings of vulnerability.
QUANT Conclusion: Survey results showed low levels of agreement with victim-blaming beliefs.
An Introduction to Fully Integrated Mixed Methods Research

9. Mixed Method Evaluation Rubric (MMER)
Insert Table 8.1.
An Introduction to Fully Integrated Mixed Methods Research

10. Scoring of the Chapter Exemplar on the MMER
Insert Table 8.3 About Here
An Introduction to Fully Integrated Mixed Methods Research

11. Scoring of the Exemplar on the MMER Criterion # 1 -TRANSPARENCY
LANGUAGE FROM THE EXEMPLAR THAT REFLECTS TRANSPARENCY
Clarity in Purpose Statement: “… to use a multiple-method approach to understand the definition, function, and salience of rape myths within the athletic community at one university” (McMahon, 2007, p. 358).
Clarity about Value-Added and Mixing: “Three methods of data collection … were used to provide a rich, comprehensive look at student athlete culture and to dig deeper into the issue of rape myths ... The integration of the findings from all three sources provides the riches and most telling analysis” (McMahon, 2007, p. 366)
Scored 3 of Possible 3:
Explicitly states one or more reasons why mixed methods were used in the study or about what was gained by using mixed methods.
An Introduction to Fully Integrated Mixed Methods Research

12. Scoring of the Exemplar on the MMER Criterion # 2-MIXING
Insert Table 8.5
Scored 3 of Possible 4:
Mixing occurred at sampling, during data analysis, and in the creation of a strong meta-inference. The QUAL and QUANT were not mixed during analysis.
An Introduction to Fully Integrated Mixed Methods Research

13. An Introduction to Fully Integrated Mixed Methods Research
Scoring of the Exemplar on the MMER Criterion # 3–INTERPRETIVE COMPREHENSIVENESS
Insert Table 8.6
Scored 1 of Possible 1
An Introduction to Fully Integrated Mixed Methods Research

14. Scoring of the Exemplar on the MMER Criterion # 4: METHODOLOGICAL FOUNDATION
Insert Table 8.7
Scored Low on this Criteria: 1 of 4
No references supplied to the methodological literature.
An Introduction to Fully Integrated Mixed Methods Research

15. Challenges in the Use of an Evaluation Rubric in Mixed Methods
There are many different views about what constitutes quality.
To date, most frameworks only evaluate the QUAL and QUANT strands and overlook the mixing strand.
To create a streamlined set of criteria that can be readily applied with a strong level of agreement between co-raters.
A lot of mixed methods research is reported in separate QUANT and QUAL publications.
To create a set of criteria that spans the qualitative, quantitative, and mixing strands.
An Introduction to Fully Integrated Mixed Methods Research

16. An Introduction to Fully Integrated Mixed Methods Research
Review Questions
Why are there so many different views about how to evaluate quality in a mixed methods publication?
What is the conventional view about how mixed method research should be evaluated?
What is the one topic where there is agreement that it must be included in an evaluation framework for mixed method research?
Why is methodological transparency so central to most evaluation frameworks?
ANSWERS:
QUESTION 1. Differences in paradigmatic views is one reason there are so many different views about how to evaluate quality in MMR.
QUESTION 2: That the quality of the QUAL and QUANT strands must be evaluated first.
QUESTION 3: That mixing must occur in some form is the one area where there is agreement.
QUESTION 4: Methodological transparency encourages replication.
An Introduction to Fully Integrated Mixed Methods Research

17. An Introduction to Fully Integrated Mixed Methods Research
Next Chapter
The next chapter is designed to help a graduate student plan a realistic dissertation that utilizes mixed methods.
An Introduction to Fully Integrated Mixed Methods Research