1. Introduction to Research Design
Module 6
Darcy Freedman, MPH, PhD
June 18, 2014

2. Assumptions in Scientific Research
Nature is orderly and regular
To some extent, events are consistent and predictable
Events or conditions have one or more causes that can be discovered
This enables establishing cause and effect relationships

3. The Scientific Method
The scientific method has been defined as a systematic, empirical, controlled and critical examination of hypothetical propositions about the association among natural phenomena.1,2
Kerlinger FN. Foundations of Behavioral Research. New York: Holt, Rinehart & Winston, 1973.
Portney LG, Watkins MP Foundations of Clinical Research: Applications to Practice. 2nd Ed. Upper Saddle River, NJ: Prentice Hall Health.

4. Properties of scientific method
Systematic
Use of orderly procedures to ensure reliability
Logical sequence is used from problem identification, through data collection, analysis, & interpretation
Empirical
Documentation of objective data through direct observation (or other systematic methods)
Findings are grounded in the objective observation of phenomena rather than the personal bias or subjective belief of the researcher
Control
In order to understand how one phenomenon relates to another, factors are controlled that are not directly related to the variables in question
Investigators have confidence in their research outcomes to the extent that they control extraneous influences

5. Limitations
Science is imperfect, especially when it is applied to human behavior and performance

6. Limitations
Science is imperfect, especially when it is applied to human behavior and performance
Sources of uncertainty:
Complexity and variability within nature
The unique psychosocial and physiological capacities of individuals

7. Limitations
Science is imperfect, especially when it is applied to human behavior and performance
Sources of uncertainty:
Complexity and variability within nature
The unique psychosocial and physiological capacities of individuals
Social science researchers must be acutely aware of extraneous influences in order to interpret findings in a meaningful way

8. Types of Research Descriptive Exploratory Experimental Case study
Cross-sectional study
Qualitative study
Exploratory
Cohort study
Case control study
Experimental
True experimental designs
Quasi-experimental designs

9. Descriptive Research
Descriptive: investigator attempts to describe a group of individuals on a set of variables or characteristics.
Enables classification and understanding
Methods: survey research, case study, qualitative, developmental (natural history of something, patterns of growth and change), normative, evaluation

10. Quality Composite Score (mean)
Store Type
(33 stores)
Quality Composite Score (mean)
Example
Convenience Stores (70%)
-0.74
Local Markets (24%)
-0.38
Supermarkets (6%)
6.5
Composite score = sum of scores for access to fresh fruit,
fresh vegetables, lean meats, low-fat milk,
tobacco products, alcohol. Chronbach’s alpha = .76
Source: Freedman & Bell, 2009

11. Exploratory Research
Investigator examines a phenomenon of interest and explores its dimensions, including how it relates to other factors.
Proven relationships between the phenomenon and other factors can lead to predictive models
Correlational studies, cohort and case control, secondary analysis, historical research

12. Freedman, Blake, & Liese, 2013

13. Figure 2. Simplified Path Analytic Model 1 of Environmental Influence on FV Intake
Source: Liese et al., 2013.
*p<.05

14. Freedman et al., under review

15. Experimental Research
Provides a basis for comparing 2 or more conditions
Controls or accounts for the effects of extraneous factors, providing the highest degree of confidence in the validity of outcomes
Enables the researcher to draw meaningful conclusions about observed differences
Randomized controlled trials, single subject designs, sequential clinical trials, evaluation research, quasi-experimental research, meta-analysis

16. Individual-level Change in Fruit and Vegetable Consumption
Design: Longitudinal; no comparison group
Sample: 45 diabetic patients at FQHC
Intervention: FQHC-based farmers’ market + financial incentive (up to $50)
Outcome measure: F/V consumption measured with NCI screener
Results:
Dose-response relationship between improvement in F/V consumption and use of market
Improvers more likely to rely on financial incentive to purchase foods at market
Source: Freedman et al., 2013

17. Continuum of Research Descriptive Exploratory Experimental
Describe populations
Find relationships
Cause and Effect
Case Study
Experimental randomized controlled trial (RCT)
Quasi-experimental designs
Survey research
Qualitative research
Sequential clinical trial
Correlational research
Single subject designs
Evaluation research
Secondary analysis
Meta-analysis
-----Cohort/Case-Control Study-----
Historical research
Based on: Portney LG, Watkins MP Foundations of Clinical Research: Applications to Practice. 2nd Ed. Upper Saddle River, NJ: Prentice Hall Health., p. 13.

18. Community-engaged research
Philosophy versus method
Who are the “knowers” of phenomenon?
Participatory processes during some or all stages of research
Knowledge for action/change
Can be used with any research approach
Example: Community Visions Photovoice Project
(~9 min)

19. Quantitative/Qualitative
Quantitative research involves measurement of outcomes using numerical data under standardized conditions
May be used along the continuum of research
Qualitative research is concerned with narrative information under less structured conditions that often takes the research context into account
Descriptive and exploratory research
Purposes: describing conditions, exploring associations, formulating theory, generating hypotheses

20. Choosing evaluation methods

21. Descriptive Research Case study Cross-sectional study
Qualitative study

22. Case Study Design
Often a description of a individual case’s condition or response to an intervention
can focus on a group, institution, school, community, family, etc.
data may be qualitative, quantitative, or both
Case series: observations of several similar cases are reported

23. Case Study
Example
In 1848, young railroad worker, Phineas Gage, was forcing gun powder into a rock with a long iron rod when the gun powder exploded. The iron rod shot through his cheek and out the top of his head, resulting in substantial damage to the frontal lobe of his brain. Incredibly, he did not appear to be seriously injured. His memory and mental abilities were intact, and he could speak and work. However, his personality was markedly changed. Before the accident, he had been a kind and friendly person, but afterward he became ill-tempered and dishonest.
Phineas Gage’s injury served as a case study for the effects of frontal lobe damage. He did not lose a specific mental ability, such as the ability to speak or follow directions. However, his personality and moral sense were altered. It is now known that parts of the cortex (called the association areas) are involved in general mental processes, and damage to those areas can greatly change a person’s personality.

24. Case Study Design Strengths Weaknesses
Enables understanding of the totality of an individual’s (or organization, community) experience
The in-depth examination of a situation or ‘case’ can lead to discovery of relationships that were not obvious before
Useful for generating new hypotheses or for describing new phenomena
Weaknesses
No control group
Prone to selection bias and confounding
The interaction of environmental and personal characteristics make it weak in internal validity
Limited generalizability

25. Cross-sectional Study
Researcher studies a stratified group of subjects at one point in time
Draws conclusions by comparing the characteristics of the stratified groups
Well-suited to describing variables and their distribution patterns
Can be used for examining associations; determination of which variables are predictors and which are outcomes depends on the hypothesis
eg. Does lead paint ingestion cause hyperactivity or does hyperactivity lead to lead paint ingestion?

26. Cross-sectional Study
Example:
What is the prevalence of chlamydia in women age in Cleveland, and is it associated with the use of oral contraceptives?
Select a sample of 100 women attending an STD clinic in the city of Cleveland
Measure the predictor and outcome variables by taking a history of oral contraceptive use and sending a cervical swab to the lab for chlamydia culture
A questionnaire may be used to gather information abut oral contraceptive history

27. Cross-sectional Study
Strengths
Fast and inexpensive
No loss to follow-up (no follow-up)
Ideal for studying prevalence
Convenient for examining potential networks of causal links
e.g., in analysis, examine age as a predictor of oral contraceptive use, and then examine oral contraceptive use as a predictor for chlamydia infection
Weaknesses:
Difficult to establish a causal relationship from data collected in a cross-sectional time-frame (Lack of a temporal relationship between predictor variables and outcome variables - Does not establish sequence of events)
Not practical for studying rare phenomena

28. Qualitative Study
Seeks to describe how individuals perceive their own experiences within a social context
Emphasizes in-depth, nuanced understanding of human experience and interactions
Methods include in-depth interviews, direct observations, examining documents, focus groups
Data are often participants’ own words and narrative summaries of observed behavior

29. Qualitative Study Example
A researcher wants to understand how provision of healthcare to undocumented persons affects the people and institutions involved
In 3 communities, information is gathered from undocumented patients, FQHC primary care clinicians, specialists, and hospital administrators
Methods: in-depth interviews, key informant interviews, participant observations, case studies, focus groups

30. Qualitative Study Strengths
Data based on the participants’ own categories of meaning
Useful for studying a limited number of cases in depth or describing complex phenomena
Provides understanding and description of people’s personal experiences of phenomena
Can describe in rich detail phenomena as they are embedded in local contexts
The researcher can study dynamic processes (i.e., document sequential patterns/change)
Weaknesses
Knowledge produced might not generalize to other people or other settings
It is difficult to make quantitative predictions
It might have lower credibility with some administrators and commissioners of programs
Takes more time to collect and analyze the data when compared to quantitative research
The results are more easily influenced by the researcher’s personal biases and idiosyncrasies

31. Exploratory Research
Cohort study
Case control study

32. Cohort Study
A group of individuals who do not yet have the outcome of interest are followed together over time to see who develops the condition
Participants are interviewed or observed to determine the presence or absence of certain exposures, risks, or characteristics
May be simply descriptive
May identify risk by comparing the incidence of specific outcomes in exposed and not exposed participants

33. Cohort Study
Example
To determine whether exercise protects against coronary heart disease (CHD).
Assemble the cohort: 16,936 Harvard alumni were enrolled
Measure predictor variables: Administer a questionnaire about activity and other potential risk factors , collected data from college records
10 years later, sent a follow-up questionnaire about CHD and collected data about CHD from death certificates

34. Cohort Study Strengths Weaknesses
Powerful strategy for defining incidence and investigating potential causes of an outcome before it occurs
Time sequence strengthens inference that the factor may cause the outcome
Weaknesses
Expensive – many subjects must be studied to observe outcome of interest
Potential confounders: eg, cigarette smoking might confound the association between exercise and CHD

35. Case-Control Study Generally retrospective
Identify groups with or without the condition
Look backward in time to find differences in predictor variables that may explain why the cases got the condition and the controls did not
Assumption is that differences in exposure histories should explain why the cases have the condition
Data collection via direct interview, mailed questionnaire, chart review

36. Case-Control Study Strengths Weaknesses
Useful for studying rare conditions
Short duration & relatively inexpensive
High yield of information from relatively few participants
Useful for generating hypotheses
Weaknesses
Increased susceptibility to bias:
Separate sampling of cases and controls
Retrospective measurement of predictor variables
No way to estimate the excess risk of exposure
Only one outcome can be studied

37. Case-Control Study Example
Purpose: To determine whether there is an association between the use of aspirin and the development of Reye’s syndrome in children.
Draw the sample of cases – 30 patients who have had Reye’s syndrome
Draw the sample of controls – 60 patients from the much larger population who have had minor viral illnesses without Reye’s syndrome
Measure the predictor variable: ask patients in both groups about their use of aspirin

38. Experimental Research
True experimental designs
Quasi-experimental designs

39. Efficacy vs. Effectiveness
Efficacy: the benefit of an intervention compared to a control or standard program under controlled, randomized conditions
Randomized controlled trial (RCT) design often used
Effectiveness: the benefit of an intervention under less controlled ‘real world’ conditions
Quasi-experimental design often used

40. Types of designs 1. One group posttest only design P T2
P = Program or intervention
T2 = Posttest

41. Types of designs 2. Before and After Design T1 P T2
One group pretest-post-test design
T1 P T2
T1 = Pretest (treatment group)
T2 = Posttest (treatment group)
P = Program or intervention

42. How much of the effect is due to the program?
Desired Outcome
(Y) T
Net
Effect C
Gross
Effect
Pre
Post
Time (X)

43. Types of designs 2. Comparison Group Design T1 P T2 C1 C2
T1 = Pretest (treatment group)
T2 = Posttest (treatment group)
P = Program or intervention
C1 = Pretest (comparison group)
C2 = Posttest (comparison group)

44. Experimental Design
True experimental design: Subjects are randomly assigned to at least 2 comparison groups
Purpose is to compare 2 or more groups that are formed by random assignment
The groups differ solely on the basis of what occurs between measurements (ie, intervention)
Changes from pretest to posttest can be reasonably attributed to the intervention
Most basic is the pretest-posttest control group design (randomized controlled trial, RCT)

45. Experimental Design Example:
Researchers conducted an RCT to study the effect of progressive resistance exercises in depressed elders. They studied 35 volunteers who had depression.
Participants were randomly assigned to an exercise group, which met three times per week for 10 weeks, or a control group which met 2 times per week for an interactive health education program.
The outcome variables were: level of depression, functional status, and quality of life, using standardized instruments.
Pretest and posttest measures were taken for both groups and differences were compared.

46. Experimental Design Strengths
Controls the influence of confounding variables, providing more conclusive answers
Randomization eliminates bias due to pre-randomization confounding variables
Blinding the interventions eliminates bias due to unintended interventions
Weaknesses
Costly in time and money
Many research questions are not suitable for experimental designs
Usually reserved for more mature research questions that have already been examined by descriptive studies
Experiments tend to restrict the scope and narrow the study question

47. Quasi-Experimental Design
Quasi-Experimental designs do not use randomized assignments for comparisons

48. Quasi-Experimental Design
Example:
A study was designed to examine the effect of electrical stimulation on passive range of motion of wrist extension in 16 patients who suffered a stroke.
Outcomes: effects of treatment on sensation, range of motion, & hand strength.
Patients were given pretest and posttest measurements before and after a 4-week intervention program.
Note: No randomization, and no comparison group

49. Quasi-Experimental Design
Strengths
Q-E designs are a reasonable alternative to RCT
Useful where pre-selection and randomization of groups is difficult
Saves time and resources vs. experimental designs
Weaknesses
Nonequivalent groups may differ in many ways -- in addition to the differences between treatment conditions, introducing bias
Non-blinding allows the possibility of unintended interventions; blinding can be used in some Q-E studies
Must document participant characteristics extensively
Potential biases of the sample must be acknowledged when reporting findings
Causal inferences are weakened by the potential for biases vs. experimental designs

50. Compared to what? Over time Between groups Pre to post Longitudinal
4/14/2017
Compared to what?
Over time
Pre to post
Longitudinal
Between groups
Randomly composed
Naturally occurring (waitlist, other programs)
National norms/standards
Low Ability to
Attribute Effect
High Ability to
Attribute Effect
Post-test
only
Pre & Post
test
Nonequivalent
comparison
group
Quasi-experiment
(matched groups,
regression discontinuity)
Randomized
experiment

52. Major sources
Portney LG, Watkins MP Foundations of Clinical Research: applications to practice. 2nd Ed. Upper Saddle River, NJ: Prentice Hall Health
Hulley SB, Cummings SR Designing Clinical Research: an epidemiologic approach. Baltimore, MD: Williams and Wilkins
Cook TD, Campbell DT Quasi-Experimentation: design & analysis issues for field settings. Boston, MA: Houghton Mifflin Company