Research Design and Validity Threats

Evaluation Designs Quantitative versus Qualitative Combination Quantitative – deductive – apply general principle to a specific case, uses hard data Qualitative – inductive – individual cases are studied to formulate a general principle, narrative data Combination – You can use qualitative data to help support quantitative data, or you can start with qualitative information and then use quantitative methods (for example, instrument development – content validity. . .send around to a panel of experts, get feedback, modify the instrument, and then collect quantitative data to validate in another way

Categories of Research Designs Nonexperimental or Pre-experimental designs One group, little validity control Quasi-experimental Experimental and comparison group, but no random assignment or selection Experimental Random assignment of experimental and control groups Experimental exerts the greatest control over internal validity. . .many exercise science studies are conducted this way. . .but very few health promotion studies are(larger groups, sometimes in schools, etc)

Terminology Internal validity External validity Extent to which an observed outcome can be attributed to a planned intervention External validity Extent to which an observed outcome can be attributed to a replicable intervention and generalized to other settings and populations Extraneous variables – did you control for all other variables that might have an impact on the outcome?

Internal validity threats History An event that occurs during the intervention that could have an impact on the results Maturation Bias from biological, natural, or social events that can bias results Internal validity threats can bias results or interpretation History – ex. Doing a smoking cessation program and having the state impose a tax increase on tobacco Maturation – ex. Trying to decrease weight of adolescents. . .they are maturing and changing rapidly. . .they will naturally gain weight as they age, or people can become more skilled or educated about topics through their health program at school or work

Internal validity threats Testing Testing might cue a person in to change behavior, regardless of the program Instrumentation Bias in data collection instruments Testing – people could also be cued in on your focus from a pre-test, might react in a certain way Instrumentation – make sure you use the same instrument from pretest to post-test, make sure you use valid and reliable instruments.

Internal validity threats Statistical Regression Bias from selecting a group with unusually high or low scores on something Selection Comparison groups are unequal Statistical regression – extreme scores (high or low) on pretest closer to mean on post-test. Think about educational settings – say you use a test to “diagnose” a learning disability. You find that 10 students scored really low on the test. You work with these students, and their mean score seems to increase on the post-test. . .can you think of something that might have occurred? Some of the students might not have put forth effort on the first exam, were misdiagnosed, and then put forth effort on the second, thus increasing the mean. This was not a function of the program, but a function of the inaccuracy of the results. Selection – nonequivalent groups

Internal validity threats Attrition/subject mortality Dropouts of subjects; if there is more than one group, then unequal dropouts between groups Interactive effects Combinations of the above Attrition – why did people drop out? Are respondents different than non-respondents? Did you lose more people in your intervention group than in your control group. Interactive effects – history and maturation, etc

Other internal validity issues Diffusion Contamination of comparison condition or intervention condition Demoralization Subjects upset they are not receiving the other condition If you have two classrooms of students – one is an intervention group, one is a control group, and they are right next to each other. . .chances are, the students will talk between the two conditions, and people in the intervention group might share what they are learning, doing, etc. You are then contaminating your control group.

External Validity Threats Social desirability Expectancy effect Hawthorne effect Placebo effect Novelty effect External validity is important for health promotion – we are ultimately trying to develop programs that work in different populations and settings Threats: Social desirability – person or group tries to please the researcher and answers how they think they should answer Expectancy effect – attitudes projected by the researcher can influence participants Hawthorne effect – people react to attention they are getting – if they are getting increased attention, they may act more favorably, regardless of the condition Placebo – change in behavior because people believe in the treatment (placebo means there was no treatment) Novelty effect – people may initially react favorably to an innovation. . .this may wear off, and the program may not be as effective as originally thought Control for these – blind participants (they don’t know which group they are in), blind the researcher (they don’t know which group the people are in)

Research Designs Key to abbreviations: O = data collection X = treatment/intervention R = random assignment Solid line separating groups – equal groups Dashed line separating groups – unequal groups This is all taken from reading 8b in your packet.

Pre-experimental designs One group, pretest, post-test O X O Good for pilot testing Does not control for IV threats Be certain to use valid and reliable instruments

Pre-experimental designs One shot case study X O No control for validity threats, no pretest measures Perhaps the weakest of all designs.

Quasi-experimental designs Nonequivalent comparison group O X O - - - - - O O A comparison group is added, but they are not equal Not equal according to size, demographics, other variables. . .you could be comparing two very different groups Many times researchers try to control for as many factors as possible when selecting comparison groups. . .age, gender, SES, variables of interest

Quasi-experimental designs Time series O O O O X O O O O Several measures to assess if there is a trend No comparison group Sometimes you might want to know if there is a trend. . .for example, you might want to see if smoking rates are already decreasing before your program. . .then it is difficult to say that your program was responsible for a decline (if it was already part of a trend)

Quasi-experimental designs Multiple Time series O O O O X O O O O - - - - - - - - - - - - - - - - - - O O O O O O O O Added a comparison group

Experimental Designs Pre-test, post-test, control group design R O X O R O O Randomly assigned to groups Randomly assigned subjects to groups – this makes the groups equal in the eyes of probability theory. . .everyone has an equal chance of being assigned to either group

Experimental Designs post-test only, control group design R X O R O Randomly assigned to groups Post-test only – how do you know the groups are equal at pre-test? If you use random assignment, then you can say that they are equal. . .this controls for the testing threat

Experimental Designs Solomon Four Group Design R O X O R O O R X O R O Here you combine the two main experimental designs. This is one of the most rigorous designs in controlling for internal validity. External validity, though, is another story.

Why do you care? Having this knowledge will help you determine the quality of research studies, which will impact your conclusions regarding the results. Resources – you only have a small amount of money allocated to the program. . .you might not be able to have two groups. . .or you might not have enough personnel to collect all the data needed for a time-series design Intervention – you might be doing a school-based intervention. . .you might not have the option of random assignment! Statistical control – there are statistical tests that can control for pre-test differences (ANCOVA – use pre-test scores as a covariate) Also realize that not all comparison groups get nothing. . .sometimes they get something, or they are offered the intervention after the initial study period