An Introduction to Mixed Methods Research by John W. Creswell, Ph.D. University of Nebraska-Lincoln SSP, University of Nebraska-Lincoln March 9, 2007 © Please do not duplicate or use these slides without the express permission of the author.

Office of Qualitative and Mixed Methods Research (OQMMR), Educational Psychology, UNL Purpose: To provide consultation on qualitative and mixed methods research, help scholars develop proposals for funding, and help conduct and evaluate funded projects. 5 Ph.D.s: Creswell, Plano Clark, Lu, Green, Shope; 2 RA’s Began 5 years ago Current projects in health sciences, physics, language arts, family/child research, educational assessment, veterans affairs Funding sources: NIH, NSF, Dept of Veterans Affairs, National Department of Education, Kellogg Foundation, Esperance Family Foundation, NE Dept of Education 23 funded projects since opening; 28 journal publications and 5 books or book chapters Editorial Office of the Journal of Mixed Methods Research Self-supporting research office

Topics that I address in my mixed methods classes and workshops: Core idea of mixed methods research Assumptions about research needed When to use mixed methods research Basic components Mixed methods literature Philosophical assumptions Mixed methods designs Criteria for choosing a design Procedural challenges in using designs Standards for evaluation Future research About 1989, researchers began talking about how to best use quantitative and qualitative data together to address research problems. The mixed methods movement was born. One development has been four primary designs for mixed methods studies. This presentation tells the story of mixed methods research, how it can be defined, and the procedures for conducting it.

Understanding the core idea of mixed methods research Quantitative Data Qualitative Data Methodology (called Mixed Methods Research)

Learning the assumptions about research needed to conduct this form of inquiry Qualitative is legitimate Knowledge of qualitative research. Methodology evolves Diversity in methodology Times when quantitative and qualitative inadequate

Identifying situations in which mixed methods research is needed You are measuring a concept on an instrument. You have a sense that scores are not telling you the entire story. If you just asked a few people about the concept you might obtain a better understanding…mixed methods research provides a more complete understanding of the research problem than either quantitative or qualitative alone.

Identifying situations in which mixed methods research is needed You look over the instruments available to study a concept. They were developed from a different sample/population than the one you are studying. You consider that you will need to develop an instrument before you can administer it to your sample… Mixed methods is a methodology for developing better, more context specific instruments. You have gathered data about a factors that predict a concept on several instruments. Although you have general information about the importance of predictors, you can only guess as to what explains why the results occurred…Mixed methods helps to explain results (or how mechanisms work) in causal models.

Identifying situations in which mixed methods research is needed You are conducting an intervention study. You have an intervention that was developed by other researchers. You are not certain that it will work with the sample you are studying…Mixed methods is a way to explore first to determine if an intervention will work. We want to evaluate the performance of an organization. This calls for understanding the expected outcomes of the organization (needs assessment), designing some instruments to measure those outcomes, and then helping to explain why the outcomes occurred…Mixed methods is an approach to tie together several steps in an evaluation process.

Learning the basic components involved in mixed methods research Mixed methods research is a methodology for conducting research that involves collecting, analyzing, and integrating quantitative and qualitative research in a single study or a longitudinal program of inquiry. The purpose of this form of research is that both qualitative and quantitative research, in combination, provide a better understanding of a research problem or issue than either research approach alone.

It is a research methodology The entire process of research Philosophical assumptions of inquiry Research questions Data collection Data analysis Data interpretation Data reporting

It involves quantitative research A research approach that involves: Variables, hypotheses, questions Instruments, closed-ended questions, reliability, validity Statistical analysis Generalizability, replicability, control, and lack of bias A research approach that involves: Posing variable-oriented research questions/hypotheses Using close-ended questions on instruments with scores that are reliable and valid Employing statistical analysis – descriptive/inferential Having scientific rigor through generalizability, replicability, validity, control, and lack of bias

It involves qualitative research An inquiry approach which includes: Central phenomenon Broad, general questions Views of participants Reciprocity and respect Description and themes Interpretation Personal reflexivity Flexible structure Meaning or advocate for groups/individuals An inquiry approach in which the researcher: The central phenomenon is one key concept the researcher wants to explore Asks participants broad, general questions which are open-ended and allow for in-depth responses Collects detailed views of participants in the form of words or images Collects data with reciprocity and respect in mind Analyzes and codes the data for description and themes Interprets the meaning of the information drawing on personal reflections and past research Writes a report that includes personal reflexivity and a flexible structure Emphasizes ideas such as the social construction of meaning, advocacy, and collaboration (adapted from Creswell, 2002, p. 58)

It involves collecting both quantitative and qualitative data Quantitative data Instruments Checklists Records Qualitative data Interviews Observations Documents Audio-visual materials Quantitative data Close-ended scales Attitudinal/behavioral scales Behavioral checklists Census, attendance records Qualitative data Open-ended responses Semi-structured interviews Semi-structured observations Records/documents Videotapes Images such as pictures or drawings

It involves quantitative and qualitative data analysis Quantitative analysis Use statistical analysis, For description For comparing groups For relating variables Qualitative analysis Use text and images, For coding For theme development For relating themes Quantitative analysis Numeric data Descriptive trend analysis Hypothesis testing, effect size, interval estimates Qualitative analysis: Text/image data Coding Themes Description Interrelated themes to build a picture of the phenomenon.

It involves mixing the data Converge data: Qual Results Quan Connect data: Qual Quan Results Embed the data: Quan data Qual data

It may consist of a single study or multiple studies Qual Quan Results Multiple Studies: Quan Qual Qual Quan Study 1 Study 2 Study 3 Study 4

Recognizing the body of mixed methods literature 1988, 1989 – books, articles by sociologists, evaluators Accelerated interest in 1990s 14 books Journals Conference papers Methodological journal article publications Empirical journal article publications US and international interest

Editors: John W. Creswell and Abbas Tashakkori Managing Editor: Vicki L. Plano Clark Email: vpc@unlserve.unl.edu

Topics addressed in this body of literature Paradigm use Research designs Threats to rigor of the designs Sampling procedures Data analysis approaches Writing study aims, research questions Notation system Interdisciplinary team research Writing mixed methods studies Evaluating mixed methods studies

Examining the philosophical assumptions behind mixed methods research Worldview or philosophy (e.g., attitudes and beliefs about knowledge, such as constructivism, post-positivism) Theoretical lens (e.g., feminist, racial) Methodological approach (e.g., experiemental, survey, ethnography, mixed methods) Methods of data collection (e.g. interviews, focus groups) Worldview or philosophy – these refer to beliefs or issues regarding epistemology These are theories such as ones from the social sciences, feminist theory , racial theory This is the research approach that is used to conduct the study such as experimental research, survey, ethnography, grounded theory Examples of methods of data collection include interviews, checklists, instruments, interviews, mixed methods Adapted from Crotty M. The foundations of social research: Meaning and perspective in the research process. London: Sage, 1998.

Learning about aspects of a worldview or paradigm How we see reality in this world (ontology) How we know what we know (epistemology) How we view the role of values (axiology) How we conduct the procedures of research (methodology) How we view the use of language in research (rhetorical)

Understanding four typical worldviews Postpositivism Determination Reductionism Empirical observation and measurement Theory verification Constructivism Understanding Multiple participant meanings Social and historical construction Theory generation Advocacy/Participatory Political Empowerment issue-oriented Collaborative Change-oriented Pragmatism Consequences of actions Problem-centered Pluralistic Real-world practice oriented

Understanding the different worldview stances in mixed methods research: Different stances: Single worldview – pragmatism, transformative-emancipatory (advocacy/participatory) Multiple worldviews made explicit Worldviews change depending on type of design There are many worldviews that can be used Worldview are important, and we need to make them explicit and honor them (Greene & Caracelli, 1997) In research you can use multiple worldviews (Creswell, 2003) If you want to use only one worldview, then it is pragmatism for mixed methods (Tashakkori & Teddlie, 2003)

Learning about the types of mixed methods designs typically used in research I. Triangulation Mixed Methods Design II. Embedded Mixed Methods Design QUAN Data and Results Pre-test + QUAL Data and Results Post-test Interpretation Qual Process

III. Explanatory Mixed Methods Design IV. Exploratory Mixed Methods Design QUAN Data and Results QUAL qual Data and Results quan Follow-up Building

Learning about the concurrent Triangulation Design One-phase project Concurrent data collection Combine results Intent is to merge two separate “strands” of data Collecting both quantitative and qualitative data Collecting these data at the same time in the research procedure Analyzing the quantitative and qualitative data separately Comparing or combining the results of the quantitative and qualitative analyses Example: collect survey data (quantitative) and collect individual interviews (qualitative) and then compare the results

Learning about the Embedded Design Supportive role of one type of data or both forms of data (mixed methods within an experiment, case study, narrative study, correlational design) Concurrent or sequential data collection One phase or two phases Intent of enhancing the design Collecting both quantitative and qualitative data Collecting these data at the same time in the research procedure Analyzing the quantitative and qualitative data separately Comparing or combining the results of the quantitative and qualitative analyses Example: collect survey data (quantitative) and collect individual interviews (qualitative) and then compare the results

and analysis of qualitative (before, during, after trial) Embedded Research Design Experiment Intervention Quan Data collection Pre-test Quan Data collection Post-test Process – collection and analysis of qualitative data (before, during, after trial)

Experimental Methodology Embedded Design within an Experiment Flow of the experiment Experimental Methodology QUAN QUAN Pre- Intervention Post- & Follow-up Follow-up Measure Measures Overall results and interpretation qual before intervention qual after intervention Procedures: One-on- one semi-structured interviews Thematic analysis Products: Transcripts Developed intervention treatment Procedures: Three groups: control group, compliance Intervention group, alliance intervention group – group comparisons Outcome measures: 1) attitudes toward medication 2) adherence to treatment 3) avoidance of relapse DAI measure completed 3 times (pre, post, & follow up) Products: Numerical item scores Change scores Test statistics Procedures: One-on-one semi- structured interviews – exiting the trial, participants from two experimental conditions; Thematic analysis Products: Transcripts Themes and quotes Procedures: Discuss treatment effectiveness Discuss themes in context of interventions and outcomes Products: Discussion

Learning about the Explanatory Sequential Design? Sequential data collection Two-phase project Quantitative phase first; qualitative, second One phase builds on other phase Intent: to explain results or to select participants to better understand results Viewing the study as a two-phase project Collecting quantitative data first followed by collecting qualitative data second Typically, a greater emphasis is placed on the quantitative data in the study Example: You first conduct a survey and then follow up with a few individuals through interviews

An Explanatory Sequential Design Quant itative Quantitative Case Selection Qualitative Interpretation – based on quan and QUAL results Data Collection (quan) Data Analysis (quan) Data Analysis (QUAL) + Qualitative Data Collection Quantitative Data* Quantitative Analysis Case Selection Qualitative Analysis Interpretation Selected 5 cases maximally varying Identified critical months in which smoking varied Number of cigarettes Graphic plot of CES - Descri ption of each Why did changes in CES - D6 D6 scores over time case smoking occur? for each participant Identification of life Qualitative Data* Graphic plot of events occurring Semi - structured cigarettes/day values during critical over time for each months where interviews, audio recorded and participant smoking increased or decreased transcribed Thematic analysis of life events for each * Data collected 10 times case over the course of a calendar year for 40 Cross - case thematic participants analysis Source: Creswell, Plano Clark, Shope, McVea. (in progress)

Learning about the Exploratory Sequential Design Sequential data collection Two-phase project Qualitative phase first; quantitative, second One phase builds on other phase Intent: to first explore in order to develop an instrument, to identify categories, taxonomy for follow up Viewing the study as a two-phase project Qualitative data collection precedes quantitative data collection Typically, greater emphasis is placed on the qualitative data in the study Example: You collect qualitative diary entries, analyze the data for themes, and then develop an instrument based on the themes to measure attitudes on a quantitative survey administered to a large sample.

Exploratory Sequential Design Phase I Qualitative Research - Year 1 Unstructured Interviews - 50 participants 8 observations at the site 16 documents Qualitative Data Collection Qualitative Data Analysis Text Analysis: Using QSR N6 Development of codes and themes for each site Qualitative Findings Phase II Quantitative Research - Year 2 Create approximately a 80-item instrument plus demographics Quantitative Instrument Development Administer survey to 500 individuals Determine factor structure of items and conduct reliability analysis for scales Quantitative Test of the Instrument Quantitative Results Determine how groups differ using ANOVA test Exploratory Sequential Design

Identifying criteria for choosing a design Intent for conducting mixed methods research Concurrent (each strand stands alone then brought together) or sequential (one strand builds on the other) Emphasis or priority given to one strand Nature of research questions Resources available Stakeholders in field Timing or Order of Use. Do you plan to collect your quantitative and qualitative data at the same time or in sequence over time? Priority. What emphasis will be given to the qualitative and quantitative arms of your study? Mixing. When do you plan to mix the data? Research objective. What design best addresses your research problem? Resources. What design is most workable given your time and resources?

Understanding the procedural challenges in using the designs Contradictory findings Data integration Sample selection Sample size Introducing bias Time IRB support Study same/different participants in phases Sample sizes in phases Contradictory findings How to analyze the data How to merge quantitative/qualitative data What criteria to use to select participants (second phase) How to treat validity Practical factors: Time Resources Skills Convincing audiences

Recognizing standards for evaluating the “quality” of a mixed methods study Collection of both quantitative and qualitative data (in response to quantitative and qualitative questions) Mixing of the two forms of data Rigor of quantitative and qualitative approaches Contribution to the mixed methods literature Use of mixed methods terms Did study use appropriate terminology for the title and design? Did it provide clear research objective for using mixed methods research? Does it contain a mixed methods purpose statement? Does it identify types of qual and quan data to be collected and qual and quan data analysis steps? Does it include a visual/procedural diagram of methods with timeline? Does it use rigorous procedures for the quantitative data collection and analysis Does it address validity considerations

Needed research on mixed methods Visual diagrams and notation system Understanding types of designs, procedures (e.g., mixed methods questions), and reasons for mixed methods Understanding issues (threats to rigor) arising in designs Emerging language of methodology Adaptation to different fields of study Worldview perspectives Inferences in mixed methods research Skills needed Funding source requirements and guidelines International perspectives Writing mixed methods studies Collaborative research and team research

Additional Readings Books: Creswell, J. W., & Plano Clark, V. L. (2007). Designing and conducting mixed methods research. Thousand Oaks, CA: Sage. Creswell, J. W. (2003). Research design: Qualitative, quantitative, and mixed methods approaches (2nd ed.). Thousand Oaks, CA: Sage Publications. Greene, J. C., & Caracelli, V. J. (Eds.) (1997). Advances in mixed-method evaluation: The challenges and benefits of integrating diverse paradigms. New Directions for Evaluation, Vol. 74. San Francisco, CA: Jossey-Bass Publishers. Mertens, D. M. (2004). Research methods in education and psychology: Integrating diversity with quantitative and qualitative approaches (2nd ed.). Thousand Oaks, CA: Sage Publications. Tashakkori, A. & Teddlie, C. (1998). Mixed methodology: Combining qualitative and quantitative approaches. Thousand Oaks, CA: Sage Publications. Tashakkori, A. & Teddlie, C. (Eds.) (2003). Handbook of mixed methods in social and behavioral research. Thousand Oaks, CA: Sage Publications.

Additional Readings Articles and Chapters: Caracelli, V. J., & Greene, J. C. (1993). Data analysis strategies for mixed-method evaluation designs. Educational Evaluation and Policy Analysis, 15 (2), 195-207. Creswell, J. W., Plano Clark, V. L., Gutmann, M., & Hanson, W. (2003). Advanced mixed methods research designs. In: A. Tashakkori & C. Teddlie (Eds.), Handbook of mixed methods in social and behavioral research (pp. 209-240). Thousand Oaks, CA: Sage. Greene, J. C., Caracelli, V. J., & Graham, W. F. (1989). Toward a conceptual framework for mixed-method evaluation designs. Educational Evaluation and Policy Analysis, 11 (3), 255-274. Morgan, D. L. (1998). Practical strategies for combining qualitative and quantitative methods: Applications to health research. Qualitative Health Research, 8 (3), 362-376. Morse, J. M. (1991). Approaches to qualitative-quantitative methodological triangulation. Nursing Research, 40, 120-123.

An Introduction to Mixed Methods Research by John W. Creswell, Ph.D. University of Nebraska-Lincoln SSP, University of Nebraska-Lincoln March 9, 2007 © Please do not duplicate or use these slides without the express permission of the author.