1. Complementary Research Methods
Michele Jacobsen, PhD
Postdoctoral Fellow, SERN
University of Calgary
Jacobsen, D. M.

2. Overview
Outline relative merits of Triangulation - the integration of both qualitative and quantitative research methodologies
Review a variety of research methods:
Experimental design, ethnography, case study, survey methods.
Discuss Nowaczyk and Underwood’s (1995) paper.
Jacobsen, D. M.

3. Fundamental Goals... Of Science: Of Scientists:
To Understand, To Predict, To Control
Of Scientists:
To communicate discoveries and findings to a community of peers
Jacobsen, D. M.

4. Designing Research Dimensions of Analysis
Research Purposes - theoretical or applied?
Research Problems - what questions are asked?
Research Settings - simulated or natural?
Research Investigators - background and training
Research Methods - a continuum
Experimental, Ethnography, Case study, Survey
Jacobsen, D. M.

5. Evaluating Research Validity Reliability
A concern for most social scientists is the complex nature of the phenomena under study: human behavior.
Multiple perspectives are required in order to adequately reflect the richness of these complexities.
Reliability
Consistency, Replicability
Usefulness or Value of Investigation
Contribution to knowledge
Advance THEORY and PRACTICE in discipline
Jacobsen, D. M.

6. Research Methodologies A continuum rather than “either/or”
Qualitative
Goal: To Understand, Predict
Descriptive accounts
Similarities and Contrasts
Applied and Theoretical
Research Questions
Field study
Natural conditions
Quantitative
Goal: To Predict and Control
Measure and Evaluate
Generalize to population, reproduction
Basic and Theoretical
Hypothesis testing
Lab study
Controlled, contrived
Jacobsen, D. M.

7. Validity and Reliability
Both Quantitative and Qualitative research designs seek reliable and valid results. For example:
Quantitative Reliability: Data that are consistent or stable as indicated by the researcher's ability to replicate the findings.
Qualitative: Validity of findings are paramount so that data are representative of a true and full picture of constructs under investigation.
Jacobsen, D. M.

8. Part Versus Whole “Whole” is often greater than “Parts”
It is a non-trivial matter to infer the behavior of the whole from the behavior of its parts
Quantitative research designs strive to identify and isolate specific variables within the context (seeking correlation, relationships, causality) of the study.
Qualitative design focuses on a holistic view of what is being studied (via documents, case histories, observations and interviews).
Jacobsen, D. M.

9. Data Collection Quantitative Qualitative
Emphasis on numerical data, measurable variables
Data is collected under controlled conditions in order to rule out the possibility that variables other than the one under study can account for the relationships identified
Qualitative
Emphasis on observation and interpretation.
Data are collected within the context of their natural occurrence.
Jacobsen, D. M.

10. Static and Dynamic Quantitative Qualitative
The accumulation of facts and causes of behavior through careful isolation, measurement and evaluation of variables.
Predictability and Control over time.
Qualitative
Concerned with the changing and dynamic nature of reality.
Understanding a Point in time
Jacobsen, D. M.

11. Triangulation Combines independent yet complementary research methods.
Simultaneous triangulation:
Use of both qualitative and quantitative methods at the same time
e.g., Survey methods and Case study
Sequential triangulation:
Results of one method are essential for planning the next method
e.g., Exploratory Pilot study precedes Experimental design
Also known as “mixed methods”
Jacobsen, D. M.

12. Benefits of Triangulation
Advantages of each complement the other
resulting in a stronger research design, and
more valid and reliable findings.
Inadequacies of individual methods are minimized
threats to Internal Validity are realized and addressed
Example
Quantitative design strives to control for bias so that facts, instances, phenomena can be understood in an objective way.
Qualitative approach strives to understand the perspective of participants or a situation by looking at firsthand experience to provide meaningful data.
Jacobsen, D. M.

13. Additional Benefits
Triangulation offers a balance between logic and stories.
Qualititative research, which emphasizes exploration, understanding, contextualizing, introspection, and theory construction, provides a strong base for wider quantitative measures, scaling, and generalization.
Quantitative research, which emphasizes large samples, can provide an overview of an area that can reveal patterns, inconsistencies, and so forth, that can be further investigated with qualitative methods.
Jacobsen, D. M.

14. Effectiveness of Triangulation
In order for Triangulation to be used effectively, four principles must be adhered to:
1. research question(s) must be clearly focused;
2. strengths and weaknesses of each chosen method must complement each other;
3. data collection methods should be selected according to their relevance to the nature of the phenomenon being studied;
4. a continual evaluation of the approach should be under-taken during the study Corner (1990)
Jacobsen, D. M.

15. Range of Research Methods
Experimental design
Ethnography
Case study
Survey
Jacobsen, D. M.

16. Experimental Design Hypothesis testing
Independent and Dependent Variables
For example - Predictor: method of instruction, Resulting differences: math performance
Sampling of Population
Experimental and Controlled Conditions
Random assignment
Jacobsen, D. M.

17. Experimental Research
The researcher does something to the subjects or objects or research, and then attempts to determine the effects of these actions
Reporting
Careful description of sampling procedure
Inferential statistics, effect size, and so on.
Jacobsen, D. M.

18. Ethnography
Defined: a picture of the “way of life” of some identifiable group of people
Anthropology - “doing fieldwork”, “going native”
Preoccupied with culture, and how people interact with each other
Qualitative Methodology - Both a research process and a product
Outcome: an ethnographic account
Jacobsen, D. M.

19. Ethnographic Process
The ethnographer is the primary research instrument
One year or more in the field setting
long enough to see a full cycle of activity
For example, a full school year
Tension and balance between involvement and detachment
Outsider’s broad and analytical perspective on group studied
Insider view, familiarity, empathy, identification with group
Jacobsen, D. M.

20. Field Research Techniques
An Inquiry Process of multiple methods:
Participant observation
privileged, active participant
passive observer
Interviewing
key informants, structured, unstructured
groups, surveys and questionnaires
Making and using records
historical documents, archives, written records
Jacobsen, D. M.

21. Validity and Reliability of the Ethnographic Account
“The satisfactoriness of the explanation is what counts, not the power of the method for deriving it”.
Significance is derived socially, not statistically
Jacobsen, D. M.

22. Case Study
Understanding the intricate complexity, idiosyncrasy of one particular case
investigation of a “bounded system”
Some entity deemed worthy of close watch
a single child, a single classroom, a single school, a single national program…
Goals
Understand and report the uniqueness of individual cases (both commonalities and differences)
Usually no attempt to represent case by single or multiple “scores”
Jacobsen, D. M.

23. Case Study Methods Similar to ethnographic field methods Reporting
ASKING - Interviews
Gather narrative and testimony
WATCHING - Observations
SEARCHING - Written records and artifacts
Reporting
Develop a conceptual structure, look for patterns, consistencies, repetitions, and manifestations pertinent to your research question(s)
Jacobsen, D. M.

24. Validity and Reliability
There are many different stories to be told
Different researchers have different questions to answer, different conceptualizations of the situation, and set different boundaries for the case
Generalizability: What is true of one case is often true about other cases
Consistencies can be found - predictability
How many cases are needed before patterns emerge? It depends...
Jacobsen, D. M.

25. Survey Research Methods
Purpose and Goal
Describe specific characteristics of a large group of persons, objects, or institutions
Understand present conditions, rather than the effects of particular intervention (as in experimental research)
Sample of Population
Groups of interest are well defined and chosen using well defined rules
Representativeness
Jacobsen, D. M.

26. Survey Methods Mail Telephone Face-to-Face Web-based
postage and printing costs, participation rate
Telephone
sampling, wage and time costs, participation rates
Face-to-Face
wage and time costs, participation rates, like structured interview
Web-based
anytime, anywhere, cost effective
Jacobsen, D. M.

27. Issues in Survey Construction
Item (question) and scale construction
Pilot Testing and revision
Sampling procedures
Analysis and reporting of results
Generalizability
Drawing conclusions about the conditions, attitudes, opinions, or status of a population of persons, objects, institutions, or other entities.
Jacobsen, D. M.

28. Nowaczyk and Underwood (1995) http://epaa.asu.edu/epaa/v3n20.html
"Possible Indicators of Research Quality for Colleges and Universities"
Used qualitative method, “focus group”, to investigate research quality in higher education
Jacobsen, D. M.

29. Results
Academics promoted the use of both quantitative and qualitative measures to report on “quality”
QUANTITY OF:
Journal publications, conference presentations, books and book chapters, awards, grants, budget, and so on…
QUALITY OF:
Reputation of publication, reputation of granting agency, quality of conference, peer reviews of research programs,…
Quality of institutions that hire graduate students
Societal benefit of research
Jacobsen, D. M.

30. Resources
Jaeger, R. M. (1997). Complementary Research Methods for Research in Education, (2nd ed). American Educational Research Association: Washington, DC.
Edyburn, D. L. (1998). The Electronic Scholar: Enhancing Research Productivity with Technology. Prentice-Hall: Columbus, OH.
Nowaczyk, R. H., & Underwood, D. G. (1995). Possible indicators of research quality for colleges and universities. Education Policy Analysis Archives, 3(20). [On-line]. Available:
Bowen, K. A. (1996). The Sin of Omission -Punishable by Death to Internal Validity: An Argument for Integration of Qualitative and Quantitative Research Methods to Strengthen Internal Validity. [On-line]. Available:
Jacobsen, D. M.