1 A Description of Performance Management & Data Based Decision Making within a Center Based Program for Children with Autism Presented by: Vincent J. Carbone, Ed.D., BCBA Margaret M. Hagerty, BCABA Emily J. Sweeney-Kerwin, BCABA Carbone Clinic Valley Cottage, NY NYS Association for Behavior Analysis 17th Annual Conference Verona, NY

2 Introduction The purpose of this workshop is to provide a description of the application of evidence based procedures to manage staff performance and learner instruction within a center based educational program for children with autism. SETTING The setting is a clinic for children with autism in Rockland County, NY. The clinic serves children with autism between the ages of 2 years and 14 years.

3 The mission of the clinic is to assess learner needs, provide instructional services and then train parents and community based educational providers to implement effective procedures so that the child may transition back to the home school environment. The methodology is exclusively behavior analytic with an emphasis upon teaching learner cooperation, reducing problem behavior, and teaching language using B.F. Skinner’s analysis of verbal behavior as a conceptual guide. Ten (10) instructors provide 1-1 instructional services. Most of the instructors are board certified behavior analysts.

4 About 30 children are served on a weekly basis in the center with no child attending for more than 15 hours per week. All of the children receive services in other educational settings or their homes during the other instructional hours of the day. The setting is divided into two classrooms with five instructors and five children in each classroom. A full time board certified behavior analyst supervisor is assigned to each classroom to manage the instructional environment.

5 Director Evaluation Services Outreach Services Assistant Office Manager Supervisor of Education Assistant to the Director Lead Instructor Lead Instructor Carbone Clinic Organizational Chart Instructor

6 PERFORMANCE MANAGEMENT The center staff have adopted a radical behavioral approach. As a result, performance of staff behavior is analyzed according to the same principles that are implicated in the control of the learner’s behavior. Since learner outcomes are directly related to the management of contingencies of reinforcement by the staff in an educational setting, precise management of staff performance may result in the best outcome for learners.

7 Performance management has been defined by Aubrey and James Daniels (2004) as “ …a technology for creating a workplace that brings out the best in people while generating the highest value for the organization.” In an educational setting the “highest value” is the learner’s outcomes. In the past 40 years the principles of behavior analysis have been applied to the management of the behavior of persons within business and industry.

8 This extension of the principles of behavior analysis to the behavior of persons within organizations has been called Organizational Behavior Management (OBM). The research published in this area has led us to an understanding of the empirically verified methods and procedures of performance management. It is this body of literature that we have relied upon to develop a performance management system to improve the quality of instruction within a center based program for children with autism.

9 EMPIRICAL SUPPPORT FOR PROCEDURES The management of staff performance in settings designed to provide education and treatment to persons with developmental disabilities has been the focus of many research studies in the Organizational Behavior Management literature. (see Babcock, Fleming & Oliver, 1998 and Sturmey, 1998, for a review of these studies) Reid, Parsons, Green, & Shepis (1991) state “The literature in organizational behavior management (OBM) offers a number of proven procedures for improving the quality of services for persons with developmental disabilities” (p.33)

10 Organizational Behavior Management has the strongest scientific support for methods that are designed to influence staff performance (Sturmey, 1998). Moreover, this field of study offers “a wide range of methods for changing staff behaviors” (Sturmey, 1998). Reid and Parsons (1995) suggest that there are seven steps to effective training and management of the quality of staff performance within a human services delivery system.

11 Sturmey (1998) refers to these steps as “prototypical OBM” methods. The seven steps to effective performance management (Reid & Parsons, 1995) are : 1. Specify work skills 2. Provide staff with a checklist description of work skills 3. Describe the work skills 4. Model the work skill behaviors. 5. Observe staff practice the work skills 6. Provide feedback 7. Continue process until staff person is competent.

12 This is the model of performance management that has been implemented in our center based program. In addition monthly monetary incentives are provided for exemplary performance as measured by supervisory staff.

13 The specific steps of our performance management approach include the manipulation of antecedent and consequent variables: 1.Ten (10) days of staff training that explicitly describes the behaviors of instructing children with autism using the methods of applied behavior analysis (ABA). See an outline of the 10 days of training in Appendix A of this document. STEPS OF THE PERFORMANCE MANAGEMENT SYSTEM

14 2.Measurable competency checklists explicitly describing most of the relevant instructional skills to include, Discrete Trial Instruction, Natural Environment Teaching (NET), teaching self- care skills, etc. 3.Demonstration of the skills 4.Required practice of the skills and measurement of skill competency on the checklists. 5.Unannounced monthly competency checklist skill assessment for the term of employment. 6.Contingency management of staff behavior through verbal feedback immediately following assessment, public posting of performance, monetary incentives for exemplary performance and remedial checks for all skills targeted for improvement.

15 IMPLEMENTATION OF THE PERFORMANCE MANAGEMENT SYSTEM Following training and assignment to several learners, monthly assessments of performance are conducted by a supervisor. The supervisor watches an instructional session with three to four relevant competency checklists on a clipboard and measures the quality of instruction across a range of instructional settings by recording the occurrence or non-occurrence of the list skills on the checklist.

16 The competency checklists were developed to measure the accuracy of instructor implementation of various instructional practices and are derived from behavior analytic literature related to the effective treatment of children with autism and other developmental disabilities. Competencies were developed to measure instructor performance related to: – Natural environment teaching – Discrete trial teaching – Teaching adaptive living skills – Implementation of behavior reduction protocols

17 The following slides provide examples of competency checklists developed for Intensive Teaching, Natural Environment Teaching, and Teaching Vocal Manding. Copies of these competency checklists are provided in the handouts.

18 When competency checklist sessions are complete the supervisor provides verbal feedback to the instructor regarding the quality of instruction. While feedback alone can be effective, its effectiveness can be increased by adding other methods such as programmed consequences for performance (Balcazar, Hopkins, & Suarez, 1985; Alvero, Bucklin & Austin, 2001). A score of 90 percent on each checklist is required to receive a “passing” score”.

19

20

21

22

23

24

25

26

27

28

29

30

31

32

33

34 Any areas scored as non-occurrence or of poor quality are listed as targets to be re-checked and remediated within a week. In addition, on each competency a number of skills have been identified as “critical” to effective instruction. Therefore failure to emit these “critical” responses or incorrect responses are considered “critical errors” and result in a failing score regardless of performance across other skill areas. When the instructor receives a score of 90 percent or greater on all the competencies scored for that session their name is publicly posted in the staff dinning room indicating their exemplary performance (See Norstrom, Lorenzi, & Hall, 1990, for a review of the research on this topic).

35 For those who do not meet the 90 percent criterion their names are omitted from the list. Instructors whose names appear on the list also receive a check for $ This check is delivered by the instructor’s supervisor at the next Friday morning staff meeting. Congratulatory applause by the entire staff occurs each time a check is distributed.

36 MONETARY INCENTIVE SYSTEM The monetary incentive system at the center conforms to the OBM research findings (See Bucklin & Dickinson, 2001) that suggests the most effective delivery of money as a consequence includes the following: Incentives are based only on the employee’s performance. Incentives are based upon clearly specified behaviors. Incentives are certain; if the target behaviors occur they will receive the money. Incentives are paid as soon as possible following the performance rating.

37 In addition, the monthly performance scores for each instructor is referred to in the employee’s annual review and contributes largely in determining the size of the annual raise and future promotions. Finally, a similar competency checklist system is used monthly to evaluate each child’s program/data book for accuracy. These competencies are referred to as book audits. An additional $ amount is awarded for exemplary performance across this measure. A copy of the book audit checklist is provided in the supplementary packet.