1. Evidence-based Practice for Applied Behavior Analysts: Necessary or Redundant Ronnie Detrich, Wing Institute Tim Slocum, Utah State University Teri Lewis, Oregon State University Trina Spencer, Northern Arizona University Susan Wilczynski, Ball State University

2. Goals for Today  Describe basic concepts of evidence- based practice.  Describe integrated decision making framework.

3. Two Ways of Thinking about EBP  An intervention found to have strong research support. (Cook, Tankersley, & Landrum, 2009)  Decision making process that informs all professional decisions. (Sakett, Straus, Richardson, Rosenberg, & Haynes, 2000)

4. Two Ways of Thinking about EBP  Using same term (EBP) to describe two different constructs creates confusion: Empirically supported treatments (EST)- interventions that meet defined standards of quality and quantity. Evidence-based practice-decision making process.

5. Evidence-based Practice and Applied Behavior Analysis  EBP is framework for guiding decisions of practitioners.  Decisions are based on the integration of: Best available evidence Client values and context Professional judgment  Consistent with foundational principles of applied behavior analysis: Data-based decision making Consideration of client values Considerations of contextual fit Commitment to research-based treatment

6. Evidence-based Practice and the Research-Practice Gap  Across disciplines, great concern about the discrepancy between what is known from research about effective treatments and the interventions practitioners routinely employ.  EBP is basis for closing the gap.

7. How Does EBP Narrow the Research to Practice Gap?  Practitioners must make decisions “now” even when research evidence is absent or incomplete. What is to be the basis for decisions?  Decisions informed by best available evidence allows practitioners to: Select Adapt to fit local circumstances Modify on the basis of progress monitoring data  These decisions require professional judgment.

8. Dilemma for Practitioners  Practitioners must make many decisions daily.  EBP assumes there is evidence for all decisions and that the relevant evidence is accessible.  How do practitioners incorporate evidence into all decisions?

9. The challenge of Best available Evidence

10. Goal  The best available evidence should pervade the practice of Applied Behavior Analysis. What is the “Best Available Evidence” for ABA practice? How do we systematically identify it?

11. Best Available Evidence  What do we mean by “ best ”? Quality : research methods and outcomes Relevance : close match with our applied question in terms of: Participants Treatment Outcomes Context Amount : number of participants, studies, investigators

12. Best available evidence Quality High Low High Better evidence Relevance (P, T, O, C) Empirically Supported Treatments

13. Best Available Evidence  What do we mean by “best available ”? We should use the best of what is available, This may mean using extremely high quality evidence Or it may mean using evidence that is less certain. “Unlimited skepticism is equally the child of imbecility as implicit credulity.” Dugald Stewart “Unlimited skepticism is equally the child of imbecility as implicit credulity.” Dugald Stewart

14. Best available evidence Quality High Low High Relevance (P, T, O, C)

15. Best available evidence Quality High Low High Empirically- Supported Treatment Relevance (P, T, O, C) Need to Generalize Uncertainty Lower Quality Evidence Uncertainty

16. The practical problem  Practitioners must often make decisions with insufficient empirical support.  What are they to do? Make the best possible inferences from imperfect evidence? Make decisions without using systematic evidence?

17. The practical problem  If Evidence-Based Practice of ABA is to be a pervasive model for professional decision-making… then we need ways to identify the best available evidence when the evidence is imperfect.

18. Accessing the Best Available Evidence 1.Systematic reviews – The foundation Identifies empirically supported treatments 2.Alternative types of review Improve our ability to glean recommendations from imperfect literature 3.Other units of practice - beyond the package Using what we know about intervention elements and principles 4.Progress Monitoring The best evidence about what works for this particular case

19. 1. Systematic reviews

20. Systematic Reviews Systematic EBP review (e.g., WWC, BEE, NAC)  Establish standards for: Identifying research base o Participants o Interventions o Comparisons o Outcomes o Settings Quality of evidence Quantity of evidence  Unit typically limited to “programs” (treatment packages)

21. Systematic Reviews  Reduced bias Transparency Objectivity Rigorous methods  Reduced risk of false positives Exclusive reliance on high quality evidence  Often fail to identify sufficient evidence  Not informed by lower quality evidence  Higher risk of false negative. Strengths Limitations

22. 2. Alternative types of reviews

23. a. What Works Practice Guides  Practice guide system:  Provide:  expert recommendations and  specific level of supporting evidence.  Allows broader generalization from specific studies  Allows for expert recommendations on topics where evidence is spare

24. a. What Works Practice Guides  Expert interpretation of research  Includes “level of evidence” ratings – based on systematic review.  Expert interpretation may introduce bias & increase uncertainty Strengths Limitations

25. b. Best Practice Panels Best practice panel  Panel selection Organization selects “expert” panelists Panel can be broadly or narrowly constructed Key to validity o Face validity o Functional aspects of validity  Recommendations based on group consensus

26. b. Best Practice Panels  May include diverse perspectives: Researcher Practitioner Consumer  Allows for interpretation of research  Tend to include factors other than research social validity? contextual fit?  May lack transparency: Selection of panel Criteria for “best practice”: politics or science?  Tend to include factors other than research Bias? Strengths Limitations

27. c. Narrative Reviews Narrative review E.g., APA Handbook of Behavior Analysis; NASP Best Practices; Experts review research base to establish recommendations Allows experts to draw on wide range of evidence Allows for expert interpretation of patterns of relevant findings

28. c. Narrative Reviews  Allows for broad generalization from specific studies to implications for practice.  Can incorporate many sources of evidence and logic/theory  No methodology to reduce bias. Selecting experts Relevant research base Criteria for “best practice” Strength of evidence rating Strengths Limitations

29. 3. Other units of practice

30. What is a “treatment”?  The best available evidence can validate: ←Comprehensive schoolwide systems ← Multi-component instructional or behavior packages ←Specific components/elements/kernels ←Principles of behavior and learning

31. a. Practice Elements and Kernels  Examples Differential praise, self-monitoring, frequent student responses with corrections  How they are validated Kernels Kernels are implemented, outcomes are measured. Practice elements Effective treatment packages are analyzed. Practice elements are components included in most effective packages.

32. a. Practice Elements & Kernels  More flexible than multi-component programs.  May be used to assemble custom interventions directed at specific problems  Can supplement information on multi- component packages  A set of effective components may not produce an effective package.  Assembling package is time-consuming and requires high level of skill. Strengths Limitations

33. b. Principles of Learning and Behavior  Examples Differential reinforcement, extinction Principles of using examples and non- examples  How they are validated Numerous studies across wide variety of populations and contexts.

34. b. Principles of Learning and Behavior  The most basic and flexible guides to intervention.  Apply to huge range of educational problems including modifications & adaptations.  Can supplement evidence on packages and components  Do not provide specific interventions. Principles must be applied – this process is uncertain.  Application requires extremely high level of expertise. Strengths Limitations

35. 4. Progress monitoring

36. 4. Progress Monitoring  Progress monitoring can validate: The specific treatment (as modified) with the specific client on the specific outcomes in the specific context  No other evidence can substitute.

37. 4. Progress Monitoring Strengths  The best evidence on whether this particular program is working.  Provides basis for data based decision making Limitations  Does not help: initial selection of treatments select modifications

38. Conclusion  The best available evidence to support pervasive evidence-based practice can be derived from: Empirically supported treatments Evidence from alternative types of reviews Evidence on alternative units of practice  And should always include progress monitoring.

39. Client Values and Context Teri Lewis Oregon State University

40. Beyond EST  Even when there are EST, practitioners may not choose these interventions and/or implement them when they are recommended.  In education innovations come and go in 18-48 months (Latham, 1988).

41. So, if we have the right answers why aren’t EST adopted and sustained?  Face Validity  Client-values and context need to be included and respected

42. Historical Perspective  Baer, Wolf & Risley (1968) Immediate and important change in behavior that has practical value as determined “…by the interest which society shows in the problems.” (p. 92)  Wolf (1978) Social significance of the goals. Societal value Social appropriateness of the procedures. Treatment acceptability Social importance of the effects. Consumer satisfaction

43. “Client”  Individual(s) who are invested in outcomes and/or are critical to the behavior change process (e.g., Baer et al, 1968; Heward et al, 2005) Individual who is the focus of the behavior change Parents and family members, teachers, mentors, colleagues, employers Organization, society

44. In contrast to EBP, BACB (2010) relies on a narrow definition Individual receiving services from a behavior analyst Decision about who is a client based on: Ethics Acceptability & Fidelity Effectiveness Maintenance and sustainability

45. Context  Just as we focus on behavior within an environmental context, we need to consider the context for selection and implementation of interventions  Contextual-fit (e.g., Albin et al 1996) Values, skills, goals and stressors of the implementors and those impacted by the target behavior

46. Implementation and Acceptability  Motivating operations such as values, goals and stressors  Compatibility with other aspects of the context (e.g., routines, resources, policies)  Reinforcement and punishment associated with implementation  Clients ultimately decide whether the intervention effects are judged successful (Wolf, 1978)

47.  Strain, Barton & Dunlap (2012) assert that incorporating client-values successfully informs decision-making: Design of service delivery Technical support to key implementors When to fade intervention components Identification of unanticipated events Focus of future research needs

48. Ethical Perspective  Three basic and fundamental questions (Cooper, Heron & Heward, 2007) What does it mean to be a behavior analyst? What is the right thing to do? What is worth doing? Social Validity Client Values

49. Summary  Including client values into assessment, intervention selection and implementation Respects all individuals perspectives Increases acceptability Improves decision-making Increases fidelity and sustainability

50.  Specifically, client values inform selection of treatment and the methods of treatment  Art of behavioral interventions

51.  But, most importantly, it allows behavior analysts the opportunity to implement our EST in applied settings and to produce important changes in behavior that have practical value

52. Professional judgment

53. Professional Judgment  Process by which the best available evidence is applied to individual cases in specific contexts.  An evidence-based practice framework provides an opportunity for behavior analysts to understand the role of professional judgment so that we can ensure all behavior analysts are wise decision makers and ultimate improve our effectiveness.

54. Evidence-based Practice Framework Spencer, Detrich, & Slocum, 2012 Client and Context

55. Behavior analysts use professional judgment…  …when formulating a practical question based on the presenting needs of the client and context.  Hudson engages dangerous behaviors such as climbing houses to get attention and because climbing is fun. What treatment should I use? Client and Context

56.  …to determine what evidence to look for, where to look for it, and to make sense of it in relation to the practical question and client.  Conduct a search for a systematic review or studies on treatments for dangerous behaviors that involve attention and automatic reinforcement functions. Client and Context Behavior analysts use professional judgment…

57.  …to weigh the quality and relevance of the currently available scientific knowledge in relation to the client and context to select a treatment.  Differential reinforcement of other behaviors and a regular routine of appropriate replacement activities. Behavior analysts use professional judgment… Client and Context

58.  …to adapt a selected treatment to be suitable for the specific client and context based on the best available evidence.  For every day that Hudson does not engage in dangerous behaviors, he earns 30 minutes of a special activity. In addition, he participates in a rock climbing club for children every week. Behavior analysts use professional judgment… Client and Context

59.  …to implement and monitor a selected treatment that was adapted for a specific client and context, based on the best available evidence.  Hudson can monitor his dangerous behaviors and record their absence on a calendar. He can report to his mother every evening and request 30 minutes of a special activity. Behavior analysts use professional judgment… Client and Context

60. Behavior analysts use professional judgment…  …to determine when a socially meaningful outcome has been achieved, based on the specific client and the best available evidence. Spencer, Detrich, & Slocum, 2012

61. Professional Judgment  Behavior analysts use professional judgment in their practice every day.  Importantly, these judgments are a necessary part of Applied Behavior Analysis.  The pervasiveness of professional judgment suggests that we should be very skilled at this activity.

62. Bases of Professional Judgment  Behavior analysts can draw from the available scientific bases to strengthen professional judgments. Principles of learning and behavior Ongoing progress monitoring

63. Bases of Professional Judgment  Principles of behavior, grounded in decades of experimental and applied research, serve as a foundation for judgments made by behavior analysts.  Behavior analysts can identify the principles that contribute to specific decisions (e.g., discrimination, differential reinforcement, extinction, schedules of reinforcement).

64. Bases of Professional Judgment  Most of our decisions can and should be informed by close and continual contact with relevant outcome data. This does not mean that we do not use professional judgment. It means our judgments are data-based.  Ongoing progress monitoring is THE best evidence of what works with specific individuals in specific contexts regardless of the strength of the empirical research.

65. Professional Judgment  Given professional judgment is ubiquitous, necessary and it has a scientific basis, it serves our profession to embrace it, develop it, and promote it.  Doing so, has implications for training and professional development.

66. Training Professionals  A great deal of training is necessary to build the expertise required to be effective behavior analysts.  The BACB training sequence is designed to ensure that behavior analytic practitioners are prepared to make good judgments.

67. Training Professionals  Coursework is important, but professional judgment is primarily shaped through supervised field experiences.  Guided or mentored experiences are necessary to teach the subtle discriminations required to be an effective behavior analytic practitioner.

68. Take Home Points  Professional judgment is pervasive in our practice and necessary for behavior analysts to be effective.  Professional judgment can be strengthened through an explicit recognition of its role, regular contact with empirical research, careful integration of principles and tactics, and ongoing measurement of relevant outcomes.  Professional judgment can be developed and enhanced via quality preservice training, supervised field experiences, and real world contingencies.

69. Recommendations  Adopt a decision-making framework to help guide practitioners to make judgments based on the best available evidence and client and context.  Organize our scientific knowledge so that principles and tactics are more accessible to practitioners who rely on them as a basis for professional judgment.  Infuse behavior analytic training programs with professional decision making.

70. Conclusions & Contingencies for EBP of ABA Formerly named: Contingencies and Evidence-Based Practice: Are we seeking punishers for our scientists and practitioners?

71. Two Ways of Thinking about EBP  An intervention found to have strong research support. (Cook, Tankersley, & Landrum, 2009)  Decision making process that informs all professional decisions. (Sakett, Straus, Richardson, Rosenberg, & Haynes, 2000)

72. EBP of ABA  EBP is process for guiding decisions of practitioners.  Decisions are based on: Best available evidence Client values and context Professional judgment  An EBP to ABA framework promotes translational methods at the practitioner level.

73. Translational Methods

75. EBP of ABA  Consistent with foundational principles of ABA Data-based decision making Consideration of client values Considerations of contextual fit

76. Consonance of Behavior Analysis and EBP Behavior AnalysisEvidence-Based Practice Analytic BAE* – goal: establish a functional relationship based on local data collection and progress monitoring Technological BAE* – methods that are sufficiently well defined to allow for treatment fidelity & replication Conceptually Systematic BAE* – well documented interventions are understood as examples of a unifying base Professional Judgment – selecting/adapting tx based on an integrated system in lieu of ‘bag of tricks’ Applied Client values and context* – improve quality of life targeting socially significant behavior Effective Client values and context* – effective change of practical and social significance Generality BAE & Client values and context* – programming across time, settings, people Behavioral BAE* – must be measureable, precise, reliable Baer, Wolf, & Risley (1968) * BAE & Client values/Context always involve professional judgment

77. Consonance of ABA and EBP  Behavior Analysis Certification Board Best Available Evidence Identify interventions based on “best available scientific evidence” Client Values & Context Select treatments based on a range of variables including but not restricted to: client preferences; client’s current repertoires; supporting environments including the constraints on those environments; social validity Professional Judgment Making determinations about treatment effectiveness based on graphical display of data (in various formats)

78. Implications of EBP of ABA: Practice  Help provide an explicit rationale for decision-making that is consistent with the rationale forwarded by virtually every other discipline.  Decision-making framework is: Transparent about how information regarding evidence is evaluated Clear about why evidence is most appropriate for a given case Cogent regarding how client values and contextual factors influence treatment selection (and the decision to retain/alter a treatment) Highlights critical analytic thinking and why this analytic process should be applied to all treatments considered

79. Implications of EBP of ABA: Practice  Demonstrates the strengths of ABA in a way that is accessible to clients and other stakeholders Client and contextual factors Define the goals for treatment and acceptable methods for intervention Serves to prompt us to focus on these factors that have always been held important to our field To meet this goal: practitioners must be prepared to discuss the goodness of contextual fit By referring to these factors in practice, we undermine the fallacious argument that we are ‘cold’ in our decision-making

80. Implications of EBP of ABA: Practice  Demonstrates the strengths of ABA in a way that is accessible to clients and other stakeholders Professional judgment An EBP of ABA framework gives us a way to discuss the expertise we bring to the table Common framework and language adopted by other fields but allows us to highlight our unique strengths in treatment evaluation on an individual basis Allows us to reject practitioners who argue EBP involves following a ‘list’

81. Implications of Ignoring EBP of ABA: Practice  Portrayed as ‘ignorant’ of the most pervasively adopted framework across disciplines  Our professional practice may be undermined with: Practitioners in other disciplines Clients & Families Administrators  Insurance Reimbursement

82. Implications of Ignoring EBP of ABA: Science  Federal agencies and funding Institute for Educational Sciences NIMH DHHS

83. Future Directions  By highlighting ‘best available evidence’ we identify that further refinements are necessary  As highlighted by Smith (2013), we must continue to validate practice elements or kernels

84. Future Directions  Dissemination remains the great challenge of all fields How do we get our information about effective treatments organized well enough that it is accessible to practitioners? How do we make decisions when the best available evidence is not ‘good enough?’ How do we set guides for optimal care for behavior analysts that will improve the overall quality of the services we provide?

85. Future Directions  Training of new behavior analysts The EBP of ABA framework suggests the decision-making process is the cornerstone of good practice – not a list of treatments Shifting greater emphasis on client values and contextual variables Requiring a rationale for how decisions are made by students in all training activities and providing feedback will enhance professional judgment (and thus practice)