Presentation is loading. Please wait.

Presentation is loading. Please wait.

What Works Regarding Social Skills Interventions Using Single Subject Design Jeffrey Chenier, M.A., Aaron J. Fischer, Katherine Hunter, Emily Patty, Lisa.

Similar presentations

Presentation on theme: "What Works Regarding Social Skills Interventions Using Single Subject Design Jeffrey Chenier, M.A., Aaron J. Fischer, Katherine Hunter, Emily Patty, Lisa."— Presentation transcript:

1 What Works Regarding Social Skills Interventions Using Single Subject Design Jeffrey Chenier, M.A., Aaron J. Fischer, Katherine Hunter, Emily Patty, Lisa Libster, M.A., Kristen OLeary, Haley York, Natalie Robichaux and Frank Gresham, Ph.D.

2 Introduction Scientifically Based Research –Section 9101(37) NCLB: Research that involves the application of rigorous, systematic, and objective procedures to obtain reliable and valid knowledge relevant to education activities and programs (No Child Left Behind Act, 2001)

3 Introduction What Works Clearinghouse (2006): Group Design –Meet Evidence Standards: well designed and implemented randomized controlled trials –Meet Evidence Standards with reservations: quasi-experiments with equating and no severe design or implementation problems or randomized clinical trials with severe design or implementation problems

4 Introduction What Works Clearinghouse (2010): Single Case (SC) –Meet Evidence Standards IV must be systematically manipulated, with the researcher determining when and how the IV conditions change Each outcome variable must be measured systematically over time by more than one assessor, and the study needs to have IOA calculated 20% of the time in each condition, and IOA percentage must meet minimum thresholds –0.80 IOA or 0.60 Cohens Kappa Study must include at least three attempts to demonstrate an intervention effect at three different points in time or with three different phase repetitions –Phase must have a minimum of three data points –Effect size estimation follows if a study has either Strong Evidence or Moderate Evidence

5 Introduction Meta-Analysis –Strube & Hartmann (1982) Objective method for summarizing a body of empirical findings –Emphasizes the direction and magnitude of effects across studies for a particular intervention

6 What Works Clearinghouse (2010) No agreed upon method or gold standard to calculate effect sizes from single-case design research –Problems How to quantify the effect? –How accurate is the effect? How comparable are the effects across other SC designs? How comparable are the effects compared to group design effect sizes?

7 Current Effect Size Estimators (WWC, 2010) Nonparametric Methods –Percentage of Nonoverlapping Data (PND), Percentage of All Nonoverlapping Data (PAND), Percent Exceeding the Median (PEM) Distributional properties of these measures are unknown, so standard errors and statistical tests are not formally justified. –Additionally, trend is not addressed Because of the lack of statistical justification, only use if an approximate size of the effect is desired. Wolery et al. (2010) compared four overlapping methods to visual inspection of effect and each method had its own host of issues, so much that they called for their abandonment –Visual analysis only agreed 121/160 on whether the treatment was effective or not

8 Current Effect Size Estimators (WWC, 2010) Parametric Methods –Regression Estimates Advantages –Familiarity –Ability to model trends –Ability to attain an Effect Size from a single case Disadvantages –Inability to deal with complex structures present in single case design

9 Current Effect Size Estimators (WWC, 2010) Parametric Methods –Multilevel Modeling Advantages –Ability to account for complexity of design Disadvantages –Unfamiliarity –Technically challenging and time consuming –Different metric from group design Effect Sizes, therefore the estimate is not comparable

10 Current Effect Size Estimators (WWC, 2010) Quantitative Methods –Differing methods to calculate a Standardized Mean Difference statistic (current study) Advantages –Encourages inclusion of SC designs in evaluating effects of interventions –Potentially gives another method in which to rank order interventions Disadvantages –Not completely comparable to group design research »Pooled within-group variance not comparable to pooled within phase variance –Small n leads to imprecise estimates –Trend is not assessed

11 Summary of Effect Size Estimators for SC Design (WWC, 2010) Simply put, science is not there yet Nonparametric estimators should be reported with a parametric estimator (regression) –Multilevel methods are not ready Quantitative methods are not as statistically sound as they should be, but the base from which to build is present

12 Social Skills Learned behaviors that enable positive interactions and allow for escape/avoidance of negative interactions Academic Enablers (DiPerna & Elliott, 2002) –Better predictor of academic achievement in 8 th grade than 3 rd grade academic achievement (Caprara et. al, 2000) Myriad of problems co-occurring with social skills deficits –Both externalizing and internalizing

13 Does social skills training work? –Gresham, Cook, Crews, and Kern, 2004 Meta-Analysisn studies ES gES r BESD Control BESD Treatment Ang & Hughes (2001) %65% Beelman et al. (1994) %62% Losel & Beelmn (2003) %60% Schneider (1992) %70% Schneider & Byrne (1985) %66% Quinn et al. (1999) %55% Means M =.60M =.29M = 35%M = 65% Introduction

14 Does social skills training work? –Godbold et. al, group design studies –Random Assignment with Equivalent Starting Groups »g=0.67, p<0.05 ; BESD treatment = 82% »Significantly higher than quasi-experimental designs

15 Introduction Godbold et. al, 2010 –Contrast Analyses

16 Introduction Research question –Evidence is there for Primary Programs; is there evidence for Secondary Programs?

17 Other Meta Analyses Since 2000 Meta AnalysisDisability Typen studiesIntervention TypeES statisticESDegree of Effect Bellini & Akullian, 2007Autism Spectrum15Video ModelingPND81%Effective Bellini & Akullian, 2008Autism Spectrum7Video Self MonitoringPND77%Questionable/Effective Bellini et al., 2007Autism15Child SpecificPND71%Questionable Bellini et al., 2007Autism7Collateral SkillsPND75%Questionable Bellini et al., 2007Autism20ComprehensivePND72%Questionable Bellini et al., 2007Autism10Peer MediatedPND62%Questionable Bellini et al., 2007Autism55TotalPND70%Questionable Kokina & Kern, 2010Autism Spectrum18Social StoriesPND60%Questionable Wang & Spillane, 2009Autism2CBTd , Medium - Large, Small - Medium Wang & Spillane, 2009Autism1CBTPND100%Very Promising Wang & Spillane, 2009Autism9OthersPND80.77%Effective Wang & Spillane, 2009Autism9Peer MediatedPND60.69%Questionable Wang & Spillane, 2009Autism6Social StoriesPND67.21%Questionable Wang & Spillane, 2009Autism11Video ModelingPND84.25%Effective Durlak & Weissberg, 2010 Typical68 After School Programs (Targeting Positive Social Behaviors) g0.19 Small (Significantly different from zero) Schneider et al, 2008Autism19 Social Behaviorphi0.72Large

18 Summary 7 total studies –6 with Autism Spectrum, 1 with Typically Developing Multiple interventions available Effectiveness –3 very effective –3 moderately effective –1 not as effective (but still statistically significant)

19 Method Literature Search, –Keyword Search in PsycINFO –5940 Articles Total Social Skills + Competence Intervention Training

20 Method Coding 1 Primary Inclusionary Criteria –No Books, Reviews, Meta-Analyses, Group Designs, Dissertations Coding 1 Secondary Inclusionary Criteria –Is study a social skills intervention or does it target a social skill? (YES) –Does study focus on ages 3-21, or through high school? (YES) –Does study target drugs, alcohol, or sexual offenders? (NO) 296 studies remained, 100% IOA in coding 1 (approximately 22% of articles)

21 Method Coding 2 Primary Inclusionary Criteria –Is the full article in English? –Does article include single subject graphs? No AB design Coding 2 Secondary Inclusionary Criteria –Does study fit our Social Skills definition? Gresham, Van, and Cook, 2006: –Facilitates initiating and maintain positive social relationships –Contributes to peer acceptance and friendship development –Results in satisfactory school adjustment –Allows individuals to cope with and adapt to the demands of the social environment –Is the study not part of a larger treatment package? Coded 190 unique studies –64 studies on to Coding 3 (IOA = 92% for 38% of studies)

22 Method Coding 3 Primary Inclusion Criteria –If one participant, more than 1 replication across setting or behavior –Presence of variability in baseline and treatment conditions across at least 2 participants, settings, or behaviors –Graphs in which UnGraph was able to score –40 studies eligible for analysis (IOA 100% on 30% of studies)

23 Method –Design Type and Subtype –Research Question –Main Unit of Comparison –Participant Info –Phase Info –Dataset Info Measurement Strategy DV information IOA Treatment Efficacy –Study Quality Three replications across or within? Treatment Integrity? IV Operationally Defined? DV Operationally Defined? IOA Coding 3

24 Method Data Extraction –UnGraph (Biosoft, 2004) Extracts numerical data from graphs and puts it into Microsoft Excel –High reliability and validity in collecting data from single subject graphs (Shadish et al., 2009)

25 Method Effect Size Calculation (Shadish, 2007) –G = (M t – M b ) / s p yields a standardized mean difference statistic –Currently the best quantitative method available, but not absolutely accurate

26 Method 1) Calculate Mean of Baseline and Tx Panels One, Two, and Three 2) Calculate BL and Tx Mean of Means across the three panels 3) Calculate Standard Deviation of BL and Tx from the Mean of Means 4) Calculate Effect Size for Positive Social Interactions

27 Results Participants# Disability Type #Ethnicity# Total #148ADHD24% Reported of n41.10% Age 2-528ODD9Caucasian28 Age 6-88Asperger's Disorder9African American21 Age Autism37Hispanic6 Age BD, EBD, ED19Asian1 Unspecified8Language Delay2Bi-Racial1 Male123Intellectual Disability9Vietnamese1 Female23Typically Developing44Latino1 Classrooms2PDD1Native American1 Williams1

28 Journals Journal Name# of Articles Education & Treatment of Children5 Behavioral Disorders5 Focus on Autism and Other Developmental Disabilities5 Journal of Positive Behavior Interventions4 Psychology in the Schools3 Behavior Modification2 Therapeutic Recreation Journal2 Journal of Autism and Developmental Disorders2 Education and Training2 Journal of Applied Behavior Analysis2 Behavior Therapy1 School Psychology Quarterly1 Remedial and Special Education1 Topics in Early Childhood Special Education1 Journal of Early Intervention1 Journal of the Association for Persons with Severe Handicaps1 International Journal of Play Therapy1 Education Psychology1 Educational Psychology in Practice1

29 Results Are social skills interventions evaluated by single subject methods effective? Yes g=3.06

30 Results Independent Variable IVnES Direct Instruction Computer-Based32.02* Peer Mediated52.67 Self Management63.53 Social Stories43.94 SODA25.54* Reinforcement Based32.17 *p<.05

31 Results Independent Variable –Largest effects occurred with interventions that target self-awareness / self-monitoring of behavior as the primary independent variable

32 Results Dependent Variable DVnES AET/On Task Bx51.57 Total Destructive Behaviors (TBD)/ Problem Behaviors / Negativistic Bx71.30 Prosocial Behaviors Cooperative Play25.54* Positive Statements, Terminations42.12 Prosocial Behavior Sportsmanship32.01 Social Play Skills - Recess33.18 SILA21.21* Seeking Help when lost16.92* *p<.05

33 Results Dependent Variable –Larger effects are seen when targeting prosocial behavior over problem behavior

34 Results RecipientDirector RecipientnES Student373.15* Classroom12.68 Teacher12.7 Peer12.34 DirectornES Experimenter Teacher Student42.48 Computer32.02* Coach/Caregiver32.04* Interventionist43.62* Peer23.81* *p<.05

35 Results Recipient and Director –Largest effects are seen when the student is the target of the intervention and when either an independent experimenter/interventionist or peer is the director of the intervention

36 Results Disability nES Autism134.04* Asperger's34.00* ADHD/ODD42.47 ED/BD/EBD42.31 Language Impaired12.68 Intellectual Disability21.82* Typically Developing *p<.05

37 Results Disability* –Interventions are most effective with children who have disabilities along the Autism spectrum

38 Results Setting nES School / Classroom233.29* School settings not classroom Recess52.43* College Campus22.60 Other43.25*

39 Results Setting –The largest effects were seen when interventions were implemented in schools

40 Results Number of Components # ComponentsnES * * *

41 Results Components** –Increasing the number of components in an intervention did not increase the magnitude of effect Size may not matter, quality matters

42 Results Treatment Integrity IntegritynES Monitored32.55 No Yes183.46*

43 Results Treatment Integrity* –Studies that report percentage of integrity had largest effects

44 Results Reinforcement Reinforcement ProvidednES Yes No173.29

45 Results Reinforcement** –No difference in these studies in regards to reinforcement Could be a definitional issue. Coded articles that specified reinforcement given, not necessarily lack of reinforcement –Reinforcement should be provided if necessary to acquire behavior change Quality of intervention may be more important for some

46 Results Study Quality QualitynES

47 Results Study Quality* –Highest effects were found in studies that had highest quality ratings, although not statistically significant –Studies/interventions should be implemented with highest quality possible

48 Summary of Results ContrastingSignificantly HigherSignificantly Lower IVSelf Awareness (1) Computer-Based IVs DVProsocial BehaviorProblem Behavior RecipientStudent DirectorPeer, InterventionistComputer, Coach/Caregiver Disability (2) Autism SpectrumIntellectual Disability SettingSchoolRecess Components (3) LessMore Integrity Taken (2) YesNo Reinforcement (3) Neither Study QualityHigher (4) Lower 1)Mixture of Ivs, all having self awareness qualities 2)Matches effect of group design study 3)Opposite of effect of group design study 4)Not significantly higher, but higher, and similar to our group design finding

49 Limitations Effect size estimator not entirely accurate No correction for small sample size Stringent selection criteria / data analytic method responsible for abandonment of nearly 40% of single subject studies –Both parametric and nonparametric methods would have had a larger n

50 Discussion and Future Directions All interventions were effective –If assessment leads to a social skills deficit, with almost any kind of student, there are interventions that work Single subject meta-analyses are not yet as informative/definitive as they could be, but current best practice is still to aggregate magnitude and direction of effects across studies –Calculate results using other available methods and comparing effects (NASP in Philly 2012????)

51 Questions and Comments? For additional copies of this presentation, either check the NASP website or contact Jeffrey Chenier at

52 Selected Citations Kratochwill, T. R., Hitchcock, J., Horner, R. H., Levin, J. R., Odom, S. L., Rindskopf, D. M & Shadish, W. R. (2010). Single­case designs technical documentation. Retrieved from What Works Clearinghouse website: Wolery, M., Busick, M., Reichow, B., & Barton, E.E. (2010). Comparison of overlap methods for quantitatively synthesizing single-subject data. The Journal of Special Education, 44,

Download ppt "What Works Regarding Social Skills Interventions Using Single Subject Design Jeffrey Chenier, M.A., Aaron J. Fischer, Katherine Hunter, Emily Patty, Lisa."

Similar presentations

Ads by Google