CHAPTER 4 ISSUES IN SINGLE SUBJECT RESEARCH Pelaez, EDP 7058

THE CONCEPTS OF PREDICTION, VERIFICATION, AND REPLICATION The concepts of prediction, verification, and replication relate to the issues of reliability and validity as specifically applied to single subject research. When these concepts can be demonstrated within a single subject research design, the functional relationship between the independent and dependent variables is evident. The reliability and validity of the study are verified assuming the extraneous variables and systematic bias cannot be reasonably perceived to account for the changes that have occurred. 2

Prediction Prediction refers to the idea that if there is no effect attributable to the independent variable, the dependent variable’s data path will remain unchanged; and one could predict that it will remain unchanged. – EX: A researcher has collected baseline data on an individual’s target behavior. After stability has been achieved during the baseline phase, the intervention phase would be introduced. – If the intervention had no effect on the dependent variable, one could logically assume the data path from baseline to intervention phases would depict no appreciable change. Therefore, one could predict that the data path will remain unchanged despite a phase change. – Should the data path change and that change be maintained one could reject the prior notion and possibly verify that the intervention has an effect on the dependent variable or target behavior. 3

Verification Verification is the confirmation that the dependent variable is changing in a predictable fashion as the independent variable is systematically applied. – EX: A baseline phase is completed and an intervention is introduced. If the independent variable will have no effect, we could predict there would be no change in the data path with that phase change. – We may predict conversely that the data are likely to reveal a change in the desired direction as a result of the implementation of the independent variable. – When this happens, we have verified our hypothesis that the dependent variable will change predictably with the introduction of the independent variable. – Replication is needed also to complete the demonstration of a functional relationship. 4

Replication Replication refers to the repeating of the observed predictions and verifications within the same study. – EX: A practitioner gathers baseline data on math accuracy and implements a reinforcement program, and the desired changes are observed. One could state that both a prediction and a verification are present, but replication is not present. – For replication to occur, the researcher might return to a baseline condition and measure the outcome. Should the target behavior change during this phase prediction has been verified. – The researcher may then reintroduce the intervention, which allows for replication of the original prediction and verification. This makes a stronger case for the demonstration of a functional relationship (ABAB design). 5

REPLICATION CONT. The more replications included within a study, the less changes in the dependent variable are attributable to extraneous or confounding variables. Also, the case for the power (robustness) of an independent variable to influence a dependent variable is strengthened when other researchers seek to replicate effects with other individuals, with similar behaviors, in different settings, and so on. 6

RELIABILITY AND VALIDITY There are three major areas of concern relating to reliability and validity in single subject research.  Interobserver reliability (sometimes referred to as interrater reliability or interobserver agreement) is important to establish confidence in the measurement of the dependent variable.  Internal validity is important to establish the believability of the functional relationship and, therefore, establish confidence in the results and conclusions drawn.  External validity is related to the confidence others may have that the same independent variable will yield similar results in similar studies. 7

Interobserver Reliability The researcher must identify and define the target behavior in such a way that there is confidence that at least two people can observe the individual and agree whether or not the behavior has occurred, or to what extent, or for how long, and so on. The researcher identifies at least two people to serve as observers (one can be the researcher). Then the observers are trained and practice observing the individual and scoring the dependent variable. Following practice sessions, the observers should compare results for each and every occurrence or measurement of the target behavior (dependent variable). 8

Interobserver Reliability cont. They should reach an understanding of both why and why not the target behavior has occurred. When disagreements occur, it is particularly important to understand and reach agreement as to what the correct measurement was and why. Interobserver reliability should be calculated among and between observers. The coefficients should be calculated in a rigorous manner to ensure adequate reliability. Some experts (e.g., Barlow & Hensen, 1984) argue that observers should be naïve to the intervention and purpose of the study, should not be involved in calculating interobserver agreement, and should remain unaware of when reliability checks are scheduled. However, since the researcher is often involved in data collection such precautions are not always possible. 9

Reactivity Reactivity refers to the individual being observed altering his or her behavior (i.e. target behavior) as a response to being observed. Reactivity generally diminishes as the number of observations increases. Practice observations in the experimental setting can reduce the possibility of reactivity when the official data collection begins, although the presence of observers regularly before the introduction of the independent variable may have an unknown and confounding result. Reactivity may also be overcome by extending baseline measurements until there is stability, but also a reasonable assumption that the performance on the dependent variable truly represents previous levels unaffected by observer presence. 10

Observer Drift Observer drift refers to a change in interpretation of when the target behavior occurs or not (or at what level, or how intensely, or for how long, etc.) from the original operational definitions. Observer drift occurs generally when a number of observations have been made, particularly if the communication and comparison of scores among raters has not been regularly reviewed and discussed. – EX: Two observers are recording “sitting in the seat” behavior, and one observer begins to score occurrences even if the individual has begun to slump over and props herself with her hand. The other observer, simultaneously scoring the individual’s performance, may disagree on what the actual performance was and therefore threaten interobserver reliability. 11

VALIDITY There are several types of validity associated with measurement in particular (e.g., face validity, content validity, and predictive validity). These are used less frequently in single subject research then in group studies, but understanding them is important. Internal and external validity are concepts that apply to both group and single subject research. 12

Internal Validity Internal validity refers to the degree to which the researcher has adequately controlled the independent, dependent, and extraneous variables so that changes in the dependent variable are directly attributable to the presence or absence or intensity of the independent variable. – EX 1: A researcher is attempting to help an individual improve his articulation. The researcher implements a treatment that is intended to accomplish that objective. After several weeks, the researcher’s measurements verify that the individual’s articulation improves, and there is no reason to suspect that that any other factor or variable accounted for the improved articulation. The researcher has convincingly demonstrated the functional relationship between the independent variable and the dependent variable. 13

Internal Validity cont. – EX 2: As an example of diminished internal validity, the same change in the target behavior occurs. However, the individual reports that he has been reading materials that tell how other people overcame their articulation difficulties. Also, he and his mother have been praying for divine assistance. The researcher must now acknowledge that the treatment alone may not account for the changes in the target behavior. 14

Extraneous Variables Extraneous variables refer to virtually anything that may affect the demonstration of the functional relationship between the independent and dependent variables. – History is used to refer to the passage of time and both foreseen and unforeseen events that arise. – Maturation refers to the natural development of an individual that occurs over time. – Attrition is the loss of subjects during the course of the study. 15

Extraneous Variables cont. – Multiple treatment interference is a very real threat in many single subject studies. Interference may occur when more than one independent variable (e.g., positive reinforcement followed by negative reinforcement) is used.  EX: A package treatment is used (e.g., positive reinforcement plus verbal prompting plus response interruption). Because more than one intervention is being used, it may be difficult if not impossible for the researcher to accurately determine which of the components of the independent variable or variables actually accounted for changes in the dependent variable. 16

Extraneous Variables cont. Treatment drift refers to individuals involved in administering the independent variable producing personal modifications (consciously or unconsciously) that may influence the impact of the independent variable on the dependent variable. Systematic bias refers to the researcher introducing some element to the experimental conditions that is likely to influence changes in the dependent variable regardless of the manipulation of the independent variable. 17

External Validity External validity refers to the degree to which the researcher may have confidence that she or he or other researchers will obtain the same or similar results if they use the same or very similar experimental procedures with other individuals, with other target behaviors, or in other settings. The more an experimental effect is replicated by the same or other researchers, the greater the external validity. 18

Types of Replication Direct replication: The researcher attempts to duplicate the procedures as precisely as possible. Intrasubject replication: The same subject is used in a subsequent study. Intersubject replication: The researcher maintains every aspect of an earlier study, but with different although similar subjects. Systematic replication: The conditions from a previous study are varied, but still obtaining similar results. 19

Educational Significance Educational significance refers to the concern that, although statistically significant results may be achieved the results should merit conclusions that the interventions used also translated into real world significance. The question is asked, “Did the interventions result in outcomes that are meaningful to practitioners and to the lives of the participants in the study?” 20

ETHICS There are many considerations in the use of applied behavior analysis and single subject research; empirical and social validity are equally important. Social validity refers to the degree to which other people think that the targeted changes in behavior are important and that the methods used to encourage behavior change are acceptable. Empirical validity refers to the measurements that actually demonstrate that the proposed behavioral changes will indeed positively affect the individual’s life. 21