1. Basic Principles of Research Design
Variables and Control
Basic Principles of Research Design

2. Overview of Tonight’s Lecture
Variables and Control
2 variables we can control (subject + situation)
Research designs are rules about controls and variables
evaluated in terms of Validity (answers the question set to address)
Internal Validity (relevant variables are controlled to answer question); 8 internal validity factors
External Validity (findings can be generalized to other settings and populations); 4 external validity factors
Observations are evaluated in terms of Reliability

3. Basic Principles of Research Design
One of the most important aspects of research in CDIS is the careful selection of the variables that are to be controlled or systematically varied.
After a problem has been identified and the purpose specified, there must be careful consideration of which variables are most relevant to the problem and the purpose.

4. 2 Purposes of Research Designs
Answer the research question
Reduce or eliminate contamination of extraneous variables

5. In experimental designs, this means ….
Manipulate the IV in order to answer the question
Arrange the experiment so that extraneous variables are controlled

6. In descriptive designs, this means ….
Select the variables for observation in order to answer the question
make the observations in a systematic and unobtrusive manner so that the criterion variables are not confounded by the extraneous variables

7. RESEARCH DESIGNS are the rules for observation
RESEARCH DESIGNS are the rules for observation - research designs are rules about controls and variables and controlled variation

8. Research design is described in terms of:
systematic variation of the IV
observation of the effects of the IV on the DV
systematic control of other variables that might produce variation in the DV

9. Researchers systematically vary the IV by controlling its variation
a detailed explanation of controls and variables is essential to understanding the basic principles or rules of research design

10. Control and Variability
Control is a restriction of natural variation
represent opposite extremes of ORDER and DISORDER
both are essential to research
IV is controlled variation

11. 2 Controlled Variables in Human Communication Research
Subject (participant) variables
characteristics of subjects
Situation variables
characteristics of the situation in which the observations are made

12. In carefully controlled studies, subject and situation variables are controlled. - only the phenomena of interest are allowed to vary. - Control of the subject and situation variables permits researchers to study the effects of the variables in which they are interested.

13. Controls are essential for achieving the purposes of research
Controls are essential for achieving the purposes of research. - if the variables under study are not isolated by controlling the relevant variables, alternative explanations of the findings cannot be ruled out. - the extent to which each kind of variation is controlled depends on the research design.

14. Control of Subject Variables
The type of communication disorder to be studied is controlled by the selection of subjects who have that type of disorder
Subject variables (e.g., age, gender, history of tx, severity) are controlled by holding them constant
If these variables were not controlled, the effects of the disorder might be confounded by the effects of the uncontrolled variables

15. Control of Situation Variables
The research design also controls the situation variables
These include variables such as time of day and instructions given to subjects
The need for situational controls varies with the type of design
EX: If you’re interested in determining how sounds are detected in real-life situations, you will not control background noise.

16. Criteria for Evaluating Research Designs
A research design is evaluated in its ability to achieve its 2 purposes (answer the question AND reduce/eliminate contamination of extraneous variables) in terms of INTERNAL and EXTERNAL VALIDITY
Therefore, research designs are evaluated in terms of VALIDITY (internal and external)

17. Internal Validity Ability to isolate effects of the IV
means that we can believe that the manipulation of the IV did cause the observed change in the DV
a study is internally valid when relevant variables have been controlled and the only variables that affect the DV’s are the IV’s

18. Internal Validity (con’t)
When uncontrolled variables can affect the DV, the effects of the IV are confounded with the effects of the uncontrolled variables
internal validity indicates that the research design accomplished what it was suppose to accomplish, i.e., answer the question.

19. External Validity
Means that the findings from the study can be generalized to other populations and settings
external validity is the extent to which the effects of the IVs on the DVs in the research situation apply to the natural settings.
Indicates the degree to which the findings from the study can be generalized
if findings from a study do not external validity, they may not be of any practical use or value.

20. The primary thing in evaluating experimental research is whether you can believe with enough confidence that the change in the DV was caused by manipulation of the IV - the fewer the alternative explanations for the observed change in the DV, the greater the internal validity of the study.

21. 8 Factors that Affect Internal Validity
history
maturation
test-practice effects
instrumentation
statistical regression
differential selection of subjects
mortality
Hawthorne effect

22. History
Events that occur between the first and subsequent measurements in addition to the experimental variable (“in the gap”)
these are the events that occur outside the experimental setting and therefore are outside the control of the experimenter
EX: prior treatment received in another setting; history of ear infections

23. History (con’t)
Descriptive studies: longitudinal studies more susceptible to history effects than cross-sectional studies
Experimental studies: one group pretest-posttest design is susceptible to history effects

24. Maturation
Changes in the subjects that cannot be controlled by the experimenter (“time-tied”)
these changes may cause effects that are attributed incorrectly to the experimental variable
EX: age changes, changes in biological or psychological processes that take place over time; even short-term changes, such as changes in motivation, interest, attention

25. Solution to maturational effects in studies:
Keep experimental sessions relatively short
randomize or counterbalance conditions
randomly assign subjects to experimental and control groups

26. Test-Practice Effects
The effect of taking a test may influence the subjects’ performance when they retake the test
practice with the test, familiarity with the test items, reduction of test anxiety, etc.
pretest-posttest designs are especially vulnerable to this problem
must be aware of the effect of testing on behavior when evaluating findings from research

27. Instrumentation
Changes in the calibration of an instrument or changes in the observers or scorers may produce changes in the observed measurements.
EX: (a) faulty or changing calibration of equipment; (b) rating scales, questionnaires, attitude inventories that have not been standardized or are not reliable; (c) biases of judges or observers used to collect data for the study; (d) poor test environment which may contaminate observations due to distractions, noise, interruptions, etc.

28. Statistical Regression
The migration of extreme scores toward the mean
atypical scores become more typical (elastic band)
did change in performance occur because of tx or because of regression to the mean?
occurs because subjects elected on basis of their extreme scores.
Solution: random selection of subjects and random assignment of subjects to groups

29. Differential Selection of Subjects
Inherent differences between subjects assigned to groups may account for the treatment effects rather than the treatment itself.
EX: SES, IQ
Solution: subjects should be equal on important dimensions prior to the study and systematic differences between comparison groups must be minimized with the exception of the criterion variable under study. Further, there should be no overlap between groups on the variable that presumably distinguishes the groups from one another.

30. Mortality
Differential loss of subjects between experimental and control groups or between comparison groups
EX: subjects who drop out may be different in important respects from those subjects who continue to participate in the study.

31. Hawthorne Effect
Changes in a subject’s behavior that occur because the subject knows he/she is participating in a study (“on stage”)
Solution: apply Hawthorne effect to control group; begin sessions with a brief period of acclimation to establish rapport.

32. Finally, you can have an interaction of any one or combination of the above factors that affect internal validity.

33. External Validity (Generalizability)
External validity is qualitatively different from internal validity.
If a study is internally invalid, then the results are meaningless.
If, however, the study is externally invalid, the results are only limited to the degree to which the internally valid results can be generalized
A study can be internally valid, but externally invalid. However, a study cannot be externally valid, but internally invalid

34. 4 Factors that Affect External Validity
Subject selection
reactive or interactive effects of pretesting
reactive arrangements
multiple treatment interference

35. Subject Selection
The representativeness of subjects selected from the population to which the researcher hopes to generalize the findings.
The relevant characteristics of the subjects that determine their degree of representativeness should be specified in the article to allow readers to evaluate the generality of the results to other subjects

36. Subject Selection (con’t)
Investigators should report what the relevant characteristics were in selecting the subjects AND what the characteristics were of the subjects on those relevant characteristics (e.g., age, gender, severity, etc.)
this factor affects the generalizability of findings to other subjects

37. Reactive or Interactive Effects of Pretesting
The degree to which a reactive pretest may interact with an IV in determining the subjects’ performance on the DV
that is, subjects who take a reactive pretest may react to an experimental treatment in a different way from people who have not been exposed to the pretest.
The test itself may change the subjects’ responses.

38. Reactive or Interactive Effects of Pretesting (con’t)
Therefore, the effect of pretesting may limit the generalization that can be made to people who have not been pretested
this factor affects the generalizability of findings to other measures and subjects

39. Reactive Arrangements
The degree to which the setting of the research is reactive or interacts with the IV in determining the subjects’ performance on the DV.
For example, the child may respond in a certain way with the clinician in the therapy room, but not in their classroom
this factor affects the generalizability of findings to other settings.

40. Multiple Treatment Interference
The degree to which various parts of multiple treatment interact with each other in determining the subjects’ performance on the DV.
This is likely to occur when more than one treatment is administered to the same subjects
this factor affects the generalizability of findings to other treatments.
The results of a multiple treatment study can only be generalized to people who would receive the same sequence and number of treatments.

41. The 4 factors that affect external validity can affect generalization in 1 of 4 ways:
Generalizability to other subjects
generalizability to other settings
generalizability to other measurements
generalizability to other treatments

42. In sum….
Research designs are evaluated in terms of validity (internal and external)
Observations are evaluated in terms of reliability
a measure is valid to the extent that it measures what it claims to measure, and reliable to the extent that it is consistent and repeatable.

43. Reliability
Refers to the precision of measurement and can be assessed by examining the consistency or stability of a test or measure.
Measurements of DVs are reliable to the extent that the same measurements would be obtained if the study were repeated.

44. Reliability of measurements can be estimated in several ways
Test-retest reliability
split-half reliability
alternate form reliability
observer reliability

45. Test-Retest Reliability
Assessed by obtaining the same measurement twice and comparing the two measurements

46. Split-Half Reliability
Assessed when measurements consist of a series of items
the items are split in half and the resulting measurements are compared.

47. Alternate Form Reliability
Assessed when alternate forms of measurement are compared

48. Observer reliability
Assessed when the judgement of two or more observers are compared.

49. The design is evaluated in terms of validity and the observations are evaluated in terms of reliability.