1. Kevin Krost, M.A. & Elizabeth Creamer, Ed.D.
Mixed Methods Systematic Literature Review of Mixed Methods Instrument Development and Validation
Kevin Krost, M.A. & Elizabeth Creamer, Ed.D.
Educational Research and Evaluation (EDRE) Ph.D. program, Virginia Tech, Blacksburg, VA

2. Background
When developing and validating instruments across content areas, the emphasis has traditionally been on quantitative methods as the sole provider of evidence (Daigneault & Jacob, 2014)
There has been an increase in the amount of mixed methods research conducted on instrument development and validation, requiring synthesis

3. Background Validity is largely considered a quantitative concept
Different terms used in qualitative research to describe similar phenomena
Credibility
Transferability
Dependability
Confirmability

4. Research Purpose
Provide a comprehensive and cohesive analysis of the previous research conducted on instrument development and validation using mixed methods
Fill the gap within MMIDV by providing a comprehensive, systematic literature review to describe it

5. Research Questions
Research Question 1 (Mixed): What are mixed methods design characteristics of mixed methods instrument development and validation articles?
Research Question 2 (Qualitative): Do the authors identify the value-added of a mixed method? If so, what characteristics are present?
Research Question 3 (Quantitative): What design features of mixed methods studies are related to mixed methods evaluation rubric score?

6. Methods
RQ 1 (Mixed): Inductive, emergent open- and focused-coding to derive quality criteria score level meanings. Descriptive statistics and correlations to describe each criterion and their relationships.
RQ 2 (Qualitative): Inductive, open-coding to evaluate presence of value-added statement. Focused coding to derive characteristics, or themes.
Research Question 3 (Quantitative): Linear and curvilinear regression of criterion scores on overall quality score to evaluate contributions of each.

7. Mixing Table Article Criteria and Inductively-Derived Score Levels
Article Critique Criteria
Amount of Mixing
Appropriateness of Analyses
Mixed Methods Rationale 1
Mixing During Discussion
One Successful Method
No Rationale 2
Mixing During Analysis
One Adequate and Inadequate Method
Unclear Rationale 3
Mixing During Methods
Both Methods Adequate
Clear Rationale 4
Mixing During Literature
Both Methods Exceptional
Value-Added Statement
Example
"a mixed method design was developed that, as described by Tashakkori and Teddlie (1998), includes...” (4)
"Specifically, the data were analyzed using principle components analysis (PCA) with varimax rotation” (3)
“Notably, although the research was aimed at exploring the supportive care needs of the men and their partners, this paper focuses on…“ (1)

8. Mixing Table Descriptive Statistics and Correlations (Quant)
between each Criterion (Qual) and Total Score
Amount of Mixing
Appropriateness of Analyses
Mixed Methods Rationale
Criteria Total Score
Mean
3.000
3.0667
2.7333
8.8000
Median 3 8
Mode 4 12
Standard Deviation
.8452
.8837
1.1629
2.624 1
0.6695
0.7268
0.7831
Criteria Total Score
0.8696
0.8994
0.9410

9. Qualitative Table Value-Added Codes with Illustrative Statements and
Corresponding Article Numbers
Primary Code
Illustrative Statement
Article Numbers
Deeper understanding of phenomenon
"The MR has proven to be a suitable paradigm for studying the causes of DIF and also offering a more integrated view of the phenomenon; it has led to conclusions that could not be obtained by applying a single method"
1, 7, 10
Unique contributions of both methods
"The quantitative and qualitative methods in the current study were highly integrated – that is, the whole of the findings exceed the sum of the individual quantitative and qualitative parts"
4, 15

10. Quantitative Table Category Scores and Composite Score by Article
Number
Amount of Mixing
Appropriateness of Analyses
Mixed Methods Rationale
Criteria Total Score 1 4 12 2 5 3 7 6 8 9 10 11 13 14 15

11. Quality Score Variability

12. Prediction of Quality Score by Criteria

13. Mixed RQ Results Each criterion was fairly consistently high
Sufficient variability indicating distinct, effective score meanings
Score meanings based on quantity and adequacy of criteria
Rationale lowest and most variable criterion
Highest correlation between rationale and appropriateness of analyses
Perfect score most common among appropriateness and rationale (Mode = 4)

14. Qualitative RQ Results
Several articles discussed the value-added of using mixed methods
Highly related to quality based on mixed RQ
Some explicitly stated value-added, others implied it and were analyzed
Both codes applied to a couple of articles
Each focused-code fairly saturated given n
Moderate number of articles given N = 15
Maximum points for rationale, potential for perfect quality score

15. Quantitative RQ Results
Overall quality score quite variable
Mode = 12, SD = 2.624
4 out of 5 value-added articles had max score
MM rationale contributed most to quality (R2)
Indicated the importance of value-added statements to overall quality
Could be related to journals in question
Average quality score high (M = 8.8, Med =8)
Amount of mixing least-related to quality
Few “Low” quality articles, more “Medium”

16. Conclusions
There were clear, discrete themes to describe the criterion score levels
Quantitative analysis supported this and the importance of rationale and value-added
The value-added of conducting MMIDV was captured by two codes both which made sense
High R2 values from prediction of overall quality score gave more evidence to support clear MM rationales and value-added

17. Questions?
Thank you!