1. IES Single-Case Research Institute: Training Visual Analysis Rob Horner University of Oregon www.uoecs.org

2. Specific Goals Define protocol for teaching students Discriminate Acceptable Designs Visual Analysis of Data Present www.singlecase.orgwww.singlecase.org For training For research

3. Evaluate the Design Meets Design StandardsMeets with ReservationsDoes Not Meet Design Standards Evaluate the Evidence Strong Evidence Moderate Evidence No Evidence Effect-Size Estimation Social Validity Assessment

4. Basic effect versus Experimental Control Basic Effect: Change in the pattern of responding after manipulation of the independent variable. (level, trend, variability) Experimental Control: At least three demonstrations of basic effect, each at a different point in time.

5. When Assessing Design Standard Does the design allow for the opportunity to assess experimental control? Baseline At least five data points per phase (3 w/reservation) Opportunity to document at least 3 basic effects, each at a different point in time.

6. First Demonstration of Basic Effect Second Demonstration of Basic Effect Third Demonstration of Basic Effect Intervention X 1. Baseline 2. Each phase has at least 5 data points (3 w/reservation) 3. Design allows for assessment of “basic effect” at three different points in time.

7. Intervention X Does Not Meet Standard

8. Intervention XIntervention Y Does Not Meet Standard

9. Intervention X Does Not Meet Standard

10. Intervention X Meets Standard With Reservation

11. First Demonstration of Basic Effect Second Demonstration of Basic Effect Third Demonstration of Basic Effect

12. Does Not Meet Standard

13. Meets Standard

15. Alternating Treatment (Multi-element) Designs Research Question: Is there a DIFFERENCE between the effects of two or more treatment conditions on the dependent variable. Methodological Issues: How many data points to show a functional relation? Five data points per condition(meets) Four (meets with reservation) The lower the separation, or higher the overlap, the more data points are needed to document experimental control.

16. Attn Escape PlayFood

17. Tangible Escape Attention Control * **

18. Attn Escape PlayFood Escape Meets Standard With Reservation

19. Attn Escape PlayFood Escape Does Not Meet Standard

20. Activity Review Fisher studies: Review Fisher studies: To what extent does each graph allow documentation of experimental control?

21. Training Evidence Standards

22. Evaluate the Design Meets Design StandardsMeets with ReservationsDoes Not Meet Design Standards Evaluate the Evidence Strong EvidenceModerate EvidenceNo Evidence Effect-Size Estimation Social Validity Assessment

23. Visual Analysis within Single-case design Documenting Experimental Control Three demonstrations of a “basic effect” at three different points in time. A “basic effect” is a predicted change in the dependent variable when the independent variable is actively manipulated. To assess a “basic effect” Visual Analysis includes simultaneous assessment of: Level, Trend, Variability, Immediacy of Effect, Overlap across Adjacent Phases, Consistency of Data Pattern in Similar Phases. (Parsonson & Baer, 1978; Kratochwill & Levin, 1992) Use www.singlecase.org to assess student performancewww.singlecase.org

24. Assessing within phase “pattern” and Between phase “basic effect” Level Trend Variability + Overlap Immediacy of Effect Consistency across similar phases _________________________________________ Other: vertical analysis; intercept gap Within Phase Between Phases

25. Six Variables Considered in Visual Analysis Level The mean of the data within a phase Also can be used to assess the level of the last 3-5 data points within a phase. Trend The slope of the best-fit straight line describing data within a phase Variability The level deviation of data around the slope of the best fit straight line (range, standard deviation)

26. Level

30. Trend

34. Variability NO Not, the way to assess variability

35. Variability YES The way to assess variability

36. Variability

38. Variables Considered in Visual Analysis Overlap  The percentage of data from one phase (typically the intervention phase) that overlaps with the range of data from the previous phase (typically the baseline phase)

39. Overlap 100% overlap 0% non-Overlap

41. Overlap

42. Overlap(ceiling) Graph A Graph B Calculation of PAND 100% Zero Overlap

43. Variables Considered in Visual Analysis Immediacy of Effect  The magnitude of change (in level, trend or variability) between the last 3- 5 data points in one phase and the first 3-5 data points in the next phase. Consistency of Data Pattern in Similar Phases  The extent to which phases with similar conditions are associated with data similar data patterns.

44. Immediacy of Effect

46. Consistency across Similar Phases Level, Trend, Variability define “similar” pattern across phases with “similar” status of the Independent Variable. “A” phases are more similar to each other than “B” phases… and vice versa.

47. Visual Analysis: The Interactions within Visual Analysis Assessing change in Level  The greater the variability the more data points needed to build a confident index of “level”  The greater the trend the less useful “level” becomes for predicting future performance, or assessing overlap Assessing change in Trend  Within phase trends may be linear or non-linear.  The greater the trend, the more data points needed to document a predictable pattern.  The shift in trend at the end of a phase should not be moving in the direction of the anticipated IV effect.

48. Trend

49. Visual Analysis: The Interaction of the Visual Analysis Variables Assessing change in Variability  The greater the variability the more data points are needed within a phase to document a predictable pattern  Within phase shift in variability requires more data points to establish a stable pattern  Outliers require more data points within a phase, and a phase should not end with an outlier. Assessing Overlap  Overlap (using range) becomes less relevant with increased trend.  Instead use overlap projected by confidence intervals around best fit straight line.

50. Visual Analysis: The Interaction of the Visual Analysis Variables Immediacy of Effect Variability at end of first phase and beginning of second phase reduce the impact of “immediacy of effect.”

51. Visual Analysis: The core discriminations once the Design is selected as meeting standards Is the Baseline adequate? Are there sufficient data within a phase to document a “pattern?” Is there a “basic effect” between two phases? As this for each pair of phases. Is there a functional relationship documented by the full data set within a study?

52. Random Data Add John Ferron’s randomized data stream…. For analysis

53. Reversal/ Withdrawal Designs

54. First Demonstration of Effect Second Demonstration of Effect Third Demonstration of Effect Visual Analysis: 1. Change in Level 2. Change in Trend 3. Change in Variability 4. Immediacy of Effect 5. Overlap 6. Consistency of Data Patterns across similar Phases Parsonson & Baer, 1978; Kratochwill & Levin, 1992

55. First Demonstration of Effect Second Demonstration of Effect Third Demonstration of Effect Comparison of actual against projected data (Analysis of Transition States versus Analysis of Steady States)

56. First Demonstration of Effect Second Demonstration of Effect Third Demonstration of Effect Consistency in data pattern across similar phases

57. Multiple Baseline Design Level, Trend, Variability, Overlap Immediacy of Effect Consistency across similar phases Stability in non-intervened series when effect demonstrated in one series

59. Use the Intercept Gap to assess a “basic effect” The magnitude of the intercept gap between the best-fit straight lines associated with two phases at each point of intervention.

60. Assess the magnitude of the intercept gap (Point X, Point Y, Point Z) X Y Z

61. X Z

62. Alternating Treatment/ Multi-element designs Magnitude of separation Greater the difference between two conditions, larger the demonstration of a functional relation Consistency of separation Greater consistency of separation between two conditions (no overlap) larger the demonstration of a functional relation Number of data points used to establish experimental control At least 4 comparisons. The more points documenting separation the stronger the demonstration of experimental control.

63. Tangible Escape Attention Control * **

64. Magnitude of separation Number of comparisons (data points) Consistency of separation across comparisons JABA 1994

65. Social Attention

66. Implications for Construction of Research Questions in Single-case Research Increase the precision of Research Questions Define conceptual logic for research question Define research question with greater precision IV related to change in level, trend, variability? “Is there is a functional relation between Functional Communication Training and reduction in the level and variability of problem behavior?”

67. Visual Analysis Activity Development of visual analysis graphs Bruce Wampold, Richard Freund, Rick Albin, Tom Kratochwill, Chris Swoboda Level, trend, variability, overlap Purpose is to emphasize interaction Examine and score each graph Use ALL data in the graph Combine assessment of: Level, trend, variability, overlap, immediacy of effect, consistency of data patterns in similar phases. Use scoring metric: 0= no functional relation 5 = publishable 7 = strong functional relation Compare scores with Excel file Median scores from five “expert” Single-case Researchers ABCB MBL Excelwww.singlecase.org

68. Visual Analysis Activity 1234 ABAB 1ATD 1 ABAB 2ATD 2 ABAB 3ATD 3 ABAB 4ATD 4 ABAB 5ATD 5 ABAB 6ATD 6 ABAB 7ATD 7 MBL 1 MBL 2 MBL 3 MBL 4 MBL 5 MBL 6 MBL 7

69. Reversal Design Is there a functional relation between the Intervention and reduction in the level of the problem behavior? Hold Design and amount of data constant… only vary data content. No Exp Control Publishable Strong Exp Control 1 2 3 4 5 6 7 Meets criterion for experimental control: but with some reservation(s)

70. ABAB 1 Level of Experimental Control No Exp Control Publishable Strong Exp Control 1 2 3 4 5 6 7 6

71. ABAB 2 Level of Experimental Control No Exp Control Publishable Strong Exp Control 1 2 3 4 5 6 7 1

72. ABAB 3 Level of Experimental Control No Exp Control Publishable Strong Exp Control 1 2 3 4 5 6 7

73. ABAB 4 Level of Experimental Control No Exp Control Publishable Strong Exp Control 1 2 3 4 5 6 7

74. ABAB 5 Level of Experimental Control No Exp Control Publishable Strong Exp Control 1 2 3 4 5 6 7

75. ABAB 6 Level of Experimental Control No Exp Control Publishable Strong Exp Control 1 2 3 4 5 6 7

76. ABAB 7 Level of Experimental Control No Exp Control Publishable Strong Exp Control 1 2 3 4 5 6 7 Evaluation for LEVEL Evaluate for TREND

77. Multiple Baseline Is there a functional relation between introducing the intervention and reduction in the level of the problem behavior?

78. MBL 1 6 DV: Proportion of intervals with social disruption

79. MBL 2 2 – level 5 - variability

80. MBL 3

81. MBL 4 3

82. MBL 5

83. MBL 6

84. MBL 7

85. Goals Finalize visual analysis for ATD Extend visual analysis to complex designs Build fluency

86. Alternating Treatment Designs Is there a difference in the percentage of 10 s intervals with social initiation under Condition B compared with Condition C?

87. ATD 1 Level of Experimental Control No Exp Control Publishable Strong Exp Control 1 2 3 4 5 6 7

88. ATD 2 Level of Experimental Control No Exp Control Publishable Strong Exp Control 1 2 3 4 5 6 7

89. ATD 3 Level of Experimental Control No Exp Control Publishable Strong Exp Control 1 2 3 4 5 6 7

90. ATD 4 Level of Experimental Control No Exp Control Publishable Strong Exp Control 1 2 3 4 5 6 7

91. ATD 5 Level of Experimental Control No Exp Control Publishable Strong Exp Control 1 2 3 4 5 6 7

92. ATD 6 Level of Experimental Control No Exp Control Publishable Strong Exp Control 1 2 3 4 5 6 7

93. ATD 7 Level of Experimental Control No Exp Control Publishable Strong Exp Control 1 2 3 4 5 6 7

94. 1234 ABAB 1 6 ATD 1 6 ABAB 2 1 ATD 2 5 ABAB 3 2 ATD 3 2 ABAB 4 6 ATD 4 1 ABAB 5 2 ATD 5 3 ABAB 6 7 ATD 6 5 ABAB 7 MBL 1 1 level, 6 Trend 6 ATD 7 7 MBL 2 2-level, 5-variability MBL 3 7 MBL 4 3 MBL 5 1 MBL 6 5 MBL 7 6

95. Using www.singlecase.org Anyone, any time may use the website to assess their visual analysis acumen. For Instructors Contact Rob with course code, and dates needed Set up “coordinator” Students enter code and name You can download their scores, number of sessions, time For Research Inter-rater agreement with visual analysis Training protocols for teaching visual analysis.

96. Summary Visual Analysis of Single Case Designs Determine viability of design first Assess Baseline Assess the within phase pattern of each phase Assess the basic effects with each phase comparison Assess the extent to which the overall design documents experimental control E.g. Three demonstrations of basic effect, each at a different point in time Assess effect size Assess social validity