3 How would you combine two types of data? Qualitative Text DataThis is a sample of a text file of words that might be collected on transcripts through interviews, fieldnotes from observations, or from optically-scanned documents.Quantitative Numeric Data
4 Objectives of the workshop: Let’s design a mixed methods studyLet’s study how people learn mixed methods research in this room? (or you can work on your own project and follow along at each step)Let’s start with a title. Write a title.What data will we collect?
5 What are types of quantitative and qualitative data? Quantitative dataClose-ended scalesAttitudinal/behavioral scalesBehavioral checklistsCensus, attendance recordsQualitative dataOpen-ended responsesSemi-structured interviewsSemi-structured observationsRecords/documentsVideotapes
6 Let’s identify our quantitative and qualitative data collection
7 Now let’s consider some reasons for why we are collecting (and mixing) both forms of data Together quantitative and qualitative data provide both precise measurement and generalizability of quantitative research and the in-depth, complex picture of qualitative researchTo validate quantitative results with qualitative dataWe do not have an adequate instrument. Thus, we need to explore views and develop an instrumentOur quantitative data provide a general explanation and we need to follow-up with participants and have them explain the quantitative resultsIn our experiment, outcomes to be measured are not enough; they need to be complemented by understanding the process of participants
9 So… There are good reasons for gathering both forms of data But…there are certain requirements for this to work best
10 Requirement #1: Now let’s consider whether we have the skills, time, and resources? We need minimum skills in both qualitative and quantitative data collection. What do we need?We need time and resources for extensive data collection and analysis. How much time and resources do we need?
11 Write down the skills, time, and resources we will need
12 Requirement #2: The audience(s) Does our audience appreciate both numbers and stories?Are they familiar with this design?Do they need to be educated?Are examples of published studies available in our content area?
14 But audiences may not recognize it yet because it is so new Increased use and acceptance of qualitative research from 1990’s to presentThe complexity of our research problems today requires understanding trends, differences, as well as individual stories, settingIndividuals advocating for and writing about mixed methods research as a distinct, new procedure (e.g., books)
15 They may think that it is analyzing data separately Quantitative DataQualitative DataMixing: converging the data or connectingthe data
16 But how do we mix? Converge data: Connect data: Qual Results Quan Qual
17 Why our audience may recognize it The evidenceBooksMethodological articlesMany published research studies using itFederal agenciesPrivate foundations
19 Other writings, initiatives on mixed methods research: Research studies reported in journalsMethodological articles exploring issues and proceduresWebsite for bringing mixed methods writers togetherConference sessionsHandbook of Mixed Methods in Social and Behavioral ResearchPrivate foundation interest; federal agency interest
20 NIH Guidelines - Mentioned several approaches for combining qualitative andquantitative research- Considerations for decidingwhat model to use (literatureavailable, prior studies,realistic design, expertise)- Need to describe eachmethod thoroughly
21 Quotes:“Combining qualitative and quantitative methods has gained broad appeal in public health research. The key question has become not whether it is acceptable or legitimate to combine methods, but rather how they will be combined to be mutually supportive and how findings achieved through different methods will be integrated.” (NIH, Office of Behavioral and Social Science Research, 1999).
22 National Academy of Sciences Three major research questions inquality educational research:What is happening? (qualitative designs)Is there a systematic effect?(a quantitative experiment)Why or how it is happening?(a qualitative followup)
23 But even if they recognize it, they may not appreciate or understand how to design a mixed methods study“We are interested in a randomized control trial with a non-experimental approach embedded within it.” (a private foundation officer)“We accept multi-method studies, but investigators mostly do not sort out the complexity of these projects so that we can understand them.” (a federal projects officer).
24 We need to define mixed methods research for our audiences Mixed methods research is a design for collecting, analyzing, and mixing both quantitative and qualitative data in a single study or series of studies to understand a research problem.The purpose of this form of research is that both qualitative and quantitative methods, in combination, provide a better understanding of a research problem or issue than either method alone.
25 Now we could mix within single studies or multiple studies Single Study:QuanQualResultsMultiple Studies (called multimethod research):Quan Qual Qual QuanStudy Study Study Study 4
26 So how do we design a mixed methods study? The model Worldviews, theoretical frameworks, problem and researchquestion, skills, resourcesType of mixed methods designProcedures for:designing the titlewriting the introduction to a studywriting the purpose statement and researchquestions/hypothesesdata collectiondata analysiswriting the mixed methods reportevaluating the mixed methods research
27 What is a worldview?Philosophy about your preferences for how you learn about something through researchYou prefer the quantitative worldview: you are the expert, you decide what needs to be learned, you build in objectivityYou prefer the qualitative worldview: participant is the expert, participant helps you build knowledge, you bring personal bias inYou prefer both the quantitative and qualitative worldview
28 The next steps in planning our study Let’s write the overall research question for our study
29 Then let’s choose a type of mixed methods study to conduct What designs are possible?
30 Types of mixed methods designs I. Triangulation Mixed Methods Design+QUANData and ResultsQUALData and ResultsInterpretationII. Nested Mixed Methods DesignQUANPre-testData and ResultsQUANPost-testData and ResultsQual Process
31 Types of mixed methods designs III. Explanatory Mixed Methods DesignqualData and ResultsQUANData and ResultsFollow-upIV. Exploratory Mixed Methods DesignQUALData and ResultsquanData and ResultsBuilding
32 Triangulation Design: Characteristics Collecting both quantitative and qualitative dataCollecting these data at the same time in the research procedureAnalyzing the quantitative and qualitative data separatelyComparing or combining the results of the quantitative and qualitative analysisExample: collect survey data (quantitative) and collect individual interviews (qualitative) and then compare the results
33 Triangulation Design: When is it used? When you want to combine the advantages of quantitative (trends, large numbers, generalization) with qualitative (detail, small numbers, in-depth)When you want to validate your quantitative findings with qualitative dataWhen you want to expand your quantitative findings with some open-ended qualitative data (e.g., survey with closed- and open-ended data)
35 Nested Design: Characteristics Collecting both quantitative and qualitative dataCollecting both types of data at the same timeHaving ONE form of data play a smaller role in the study than the other form of dataAlso,Using one form of data to answer one question; the other form another questionCollecting one form of data at one level of analysis and another at another level of analysisExample: You conduct an experiment and during the experiment you gather qualitative interview data. The outcomes of the experiment assessed quantitatively address different questions than the process of the experiment explored qualitatively.
36 Nested Design: When is it used? When you do not have time or resources to commit to extensive quantitative and qualitative data collectionWhen you want to study the process of an experiment as well as the outcomesWhen you want to examine different levels in an organization
37 and analysis of qualitative Nested Research DesignExperimentQuanData collectionPre-testInterventionQuanData collectionPost-testProcess – collectionand analysis of qualitativedata
38 Explanatory Sequential Design: Characteristics Viewing the study as a two-phase projectCollecting quantitative data first followed by collecting qualitative data secondTypically, a greater emphasis is placed on the quantitative data in the studyExample: You first conduct a survey and then follow up with a few individuals who answered positively to the questions through interviews
39 Explanatory Sequential Design: When do you use it? When you want to explain the quantitative results in more depth with qualitative data (e.g., statistical differences among groups, individuals who scored at extreme levels)When you want to identify appropriate participants to study in more depth qualitatively
40 Here is an example of an explanatory design: QuantitativeQuantitativeCase SelectionQualitativeInterpretation –based on quanand QUAL resultsData Collection(quan)Data Analysis(quan)Data Analysis(QUAL)+QualitativeData CollectionQuantitative Data*Quantitative AnalysisCase SelectionQualitative AnalysisInterpretationSelected 5 casesmaximally varyingIdentified criticalmonths in whichsmoking variedNumber of cigarettesGraphic plot of CES-Description of eachWhy did changes inCES-D6D6 scores over timecasesmoking occur?for each participantIdentification of lifeQualitative Data*Graphic plot ofevents occurringSemi-structuredcigarettes/day valuesduring criticalover time for eachmonths whereinterviews, audiorecorded andparticipantsmoking increased ordecreasedtranscribedThematic analysis oflife events for each* Data collected 10 timescaseover the course of acalendar year for 40Cross-case thematicparticipantsanalysisCreswell et al. (in progress)
41 Exploratory Sequential Design: Characteristics Viewing the study as a two-phase projectQualitative data collection precedes quantitative data collectionTypically, greater emphasis is placed on the qualitative data in the studyExample: You collect qualitative diary entries, analyze the data for themes, and then develop an instrument based on the themes to measure attitudes on a quantitative survey administered to a large sample.
42 Exploratory Sequential Design: When do you use it? To develop an instrument when one is not available (first explore, then develop instrument)To develop a classification or typology for testingTo identify the most important variables to study quantitatively when these variable are not known
43 Sequential Exploratory Mixed Methods Design Phase I Qualitative Research - Year 1Unstructured Interviews -50 participants8 observations at the site16 documentsQualitative Data CollectionQualitative Data AnalysisText Analysis: Using QSR N6Development of codes and themesfor each siteQualitative FindingsPhase II Quantitative Research - Year 2Create approximately a 80-iteminstrument plus demographicsQuantitative Instrument DevelopmentAdminister survey to 500 individualsDetermine factor structure of items andconduct reliability analysis for scalesQuantitative Test of the InstrumentQuantitative ResultsDetermine how groups differusing ANOVA testSequential Exploratory Mixed Methods Design
44 How will we analyze the quantitative and qualitative data (within the design types)? Qualitative analysisText/image dataCodingThemesDescriptionInterrelated themesTypes of analysis:Quantitative analysisNumeric dataDescriptive trend analysisHypothesis testing, effect size, interval estimates
45 Triangulation data analysis QUANdata collectionQUALdata collectionSeparate QUANand QUAL dataanalysisQUANdata analysisQUALdata analysisTwo optionsData transformation (changeQUAL to QUAN or QUAN to QUAL)Comparison (keep separate andcompare/contrast)Results
46 Table. Example of Data Transformation of Text Units into Numeric Data CountAdj.Count**Row PctColumn PctPatientsN=2PhysiciansN=4Medical AssistantsThemesFamiliarityWith the Form136.541.95.8174.2527.43.0194.7530.75.34915.5100.0Reactions to the Form2311.522.210.210025.048.317.46115.329.516.918451.75Use for Managing Depression6733.538.629.817744.2551.0369.010.410.028086.75Changes to the Form11557.537.551.119649.032.034.018746.830.551.7498153.3Situational Use of the Form18.104.22.16821.554.414.95814.536.716.115139.5225112.5576144.036190.3
47 Nested data analysis Quantitative Experiment Intervention Quan Data collectionPre-testQuanData collectionPost-testQualitative ProcessDataAnalysisPre-testscoresThemes/Codes/Interrelated ThemesPost-test scoresor gain scoresCompare/Describe Results
48 Explanatory sequential data analysis Qualdata collection(purposeful sampling)Select cases based on s.d. variablesSelect cases to represent outliersSelect cases to represent extreme casesSelect cases to make group comparisonsQUANdata analysisStatistical resultsOutlier casesExtreme casesQualanalysiscodesthemescases
49 Exploratory sequential data analysis QUAL data analysisQuotesCodesThemesQuan data analysisinstrument developmentItems on a surveyVariables on a surveyScales on a survey
51 Drawing our Design Let’s draw a picture of our design Identify the type of designAdd in data collectionAdd in data analysisShow the flow of activitiesAdd in “products” for our audiences
52 Helpful tips for creating this visual: Give a title to the visual model.Choose either horizontal or vertical layout for the model.Draw boxes for quantitative and qualitative stages of data collection, data analysis and interpretation of the study results.Use capitalized (QUAN) or small letters (quan) to designate priority of quantitative and qualitative data collection and analysis.Use single-headed arrows to show the flow of procedures in the design.Specify procedures for each quantitative and qualitative data collection and analysis stage.Specify expected products or outcomes of each quantitative and qualitative data collection and analysis procedure.Make your model simple.Size your model to one page.
53 Now let’s rework our purpose statement using some scripts
54 Sample Script for a Concurrent Design (Triangulation or Nested) “The purpose of this concurrent mixed methods study is to better understand a research problem by converging both quantitative (numeric) and qualitative (text or image) data. In this approach, ___________ (quantitative instruments) will be used to measure the relationship between the ________ (independent variables) and __________ (dependent variables). At the same time in the study, the __________ (central phenomenon) will be explored using _____________ (qualitative interviews, documents, observations, visual materials) with _________ (participants) at ____________ (the research site).”
55 Sample Script for a Sequential Exploratory Design “The purpose of this two-phase, exploratory mixed methods study will be to explore participant views with the intent of using this information to develop and test an instrument with a sample from a population. The first phase will be a qualitative exploration of a _______(central phenomenon) by collecting ___________(data) from ____________ (participants) at _______ (research site). Themes from this qualitative data will then be developed into an instrument (or survey) so that the __________ (theory and research questions/hypotheses) can be tested that ________ (relate, compare) ____________ (independent variable) with __________ (dependent variable) for _________(sample of a population) at _________ (research site).”
56 Sample Script for a Sequential Explanatory Design “The purpose of this two-phase, explanatory mixed methods study will be to obtain statistical, quantitative results from a sample and then follow-up with a few individuals to probe or explore those results in more depth. In the first phase, quantitative research questions or hypotheses will address the relationship or comparison of __________ (independent) and ________ (dependent) variables with ___________ (participants) at ___________(the research site). In the second phase, qualitative interviews or observations will be used to problem significant _______(quantitative results) by exploring aspects of the ________ (central phenomenon) with_______ (a few participants) at ____________ (research site).”
57 Criteria for evaluating our plan: Use appropriate terminology for title and designProvide a rationale for mixing and include it early in the study (“when you use…”)Create a mixed methods purpose statementIdentify types of qual and quan data to be collected and qual and quan data analysis stepsInclude a visual/procedural diagram of methods with timelineUse rigorous procedures for the quantitative data collection and analysis
58 Let’s share our drawings of our mixed methods procedures
59 How to Design a Mixed Methods Study by John W. Creswell, Ph.D. and Vicki L. Plano Clark, M.S.University of Nebraska-LincolnAndrews University, July, 2004