Statistical Issues in Data Collection and Study Design For Community Programs and Research October 11, 2001 Elizabeth Garrett Division of Biostatistics Department of Oncology

Overview Goals of Data Collection and Study Design Key concepts –Reliability –Validity “Latent” Constructs Study Designs Potential Biases

Goals of Data Collection Two broad goals* –evaluation of intervention controlled uncontrolled –summary of population demographics attitudes

The Goal of Study Design To devise a model for some complex etiologic or clinical process that gives valid and precise inference.

Issue Specific to Interventions Outcomes tend to be “soft” Not always an easily quantified response We often use one or more “surrogates” to measure outcomes.

Key Concepts Reliability: Is the data that you are collecting a reliable or reproducible measure? Has to do with how closely your measure correlates with the underlying construct you want to measure. Truth = Observed Data + Error –If you collected the same data tomorrow, would you get the same answer? –If you ask a related question, will the two questions have correlated answers? – If two different “raters” (i.e. collectors) evaluate the same individual, do they get the same data?

Validity Validity: Is what you are collecting measuring what you want to measure? –Face validity –Construct validity –Criterion validity –Etc.

Valid

Invalid

Reliable (but still invalid!)

Valid and Reliable

Validity Internal Validity: –Valid for the population from which you sampled External Validity –Generalizable to a broader population

Latent Constructs Definition: A latent variable is a variable that cannot be directly measured. Examples: –Quality of Life –Socio-economic status –Distress –Depression

Latent Constructs Need to be measured using multiple variables Variables, taken together, should “define” the construct Methods should be decided upon ahead of time and data collection needs to be considered. Examples: latent class analysis, factor analysis Coding is important –likert scale: “On a scale of 1 to 7…..” –binary: yes/no, present/absent –continuous: age, income

Some Study Design Types Cross-sectional, No Intervention –Attributes quantify community “summarize” attitudes, demographics, etc. descriptive statistics –means, medians, standard deviations –“pictures” of the sample: histograms, boxplots “hypothesis generating”, and NOT “hypothesis testing” simplest conceptually

Cross-sectional, No Intervention (cont.) –Issues to think about sampling –Who? –When? –Where? Data (this is general to ALL study designs) –format? –Binary versus continuous versus ordinal versus categorical? –open-ended?

Pre-post Design, One group (uncontrolled) –Was intervention successful? –Attributes: Compare baseline to follow-up simplest when only two time points are collected. Convenient that each individual serves as his/her own control Hypothesis testing: –Ho: intervention worked –Ha: intervention did not work Some methods: binomial tests, signed rank test, paired t-test, regression methods

–Issues to think about when should “success” be measured? –1 week? 1 month? Both? –What if effect at 1 month but “washed out” by 6 months? How is success measured? –Yes/no? Continuous change in score? Learning effect –bias of questionnaires –is this the most appropriate design if there is a potential learning effect? “ Placebo” effect could play a role. Adherence! –Is the effect of intervention smaller than anticipated because some study participants did not adhere? Confounders and effect modifiers! –Are there some individuals that respond more strongly to the intervention than others in such a way that is predictable (e.g. age? weight? political views?)

Pre-Post, Two Groups (Controlled) –Does intervention group improve more than the control group? –Attributes similar to pre-post, one group can quantify placebo and learning effects (caveat) hypothesis testing: –Ho: effect in control group = effect in treatment group –Ha: effect in control group  effect in treatment group Some methods: 2 sample t-test, rank sum test, fisher’s exact test, regression methods

–Issues to think about We have a measure of placebo effect blinding or masking? Is it possible? Randomization –Balance? –Stratification necessary? –Matched? ITT versus Treatment received? Drop out Adherence

Other Study Designs Case-Control Studies Cohort Studies (aka Prospective Study) Ecologic Study

Potential Biases to Keep in Mind Selection Bias (IV) –individuals who join the study are not representative of the population in a way that affects the outcome. Information Bias (IV) –measures tend to be biased in one direction Confounding (IV) –Mixing of effects leads to wrong inference Effect Modification (IV) –effect of treatment depends on another factor (e.g. age)