1. PROGRAM EVALUATION OF POSITIVE BEHAVIOR SUPPORT IN AN ALTERNATIVE SCHOOL National Association of School Psychologists Annual Convention February 27th, 2009 Boston, MA
Elana R. Weinberger, M.Ed.
Melissa L. Greene, Ph.D.
William J. Matthews, Ph.D.
John M. Hintze, Ph.D.

2. Positive Behavior Support (PBS)
Rooted in the field of behaviorism and applied behavior analysis (ABA)
Defined as the “application of positive behavioral interventions and systems to achieve socially important behavior change” (Sugai et al., 2000)
Systems approach to promoting positive behaviors of students and preventing and responding to school and classroom discipline problems
By using positive reinforcement primarily, PBS creates and maintains safe learning environments where teachers can teach and students can learn
Use of PBS has become more common and necessary in recent years as a method of preventing school violence
PBIS develops school-wide systems that support staff to teach and promote positive behavior in all students.

3. Purpose of Evaluation
To evaluate the strengths and weaknesses of a PBS program as implemented in an alternative school, and to implement changes to the program in an effort to improve program effectiveness and positive outcomes for students

4. Design Program Evaluation using an exploratory case study design
Three phase model
Phase 1: initial evaluation (11/07-12/07)
Phase 2: intervention (1/08-3/08)
Phase 3: follow-up (4/08-5/08)
All data collected in phase 1 were collected again in phase 3
Exploratory case study design will be used to conduct the proposed formative evaluation. This design was selected due to its focus on achieving an in-depth understanding of a single case or unit through many different forms of data collection, both quantitative and qualitative (Fitzpatrick, Sanders & Worthen, 2004). The proposed evaluation will have three phases. Phase 1 will be the evaluation phase, in which quantitative and qualitative data will be collected in order to answer each of the evaluation questions based on the current program implementation. Findings from Phase 1 will lead to suggestions and changes for program improvement. These suggestions will be made to program leaders, and the evaluator will oversee Phase 2, an implementation phase, in which suggested changes and/or improvements will be implemented. Phase 3 will serve as a follow-up phase, in order to explore whether or not the changes in program delivery were associated with positive outcomes. Phase 1 will last approximately two months. Phase 2 will begin immediately following reporting of Phase 1 findings, and will last at least two months prior to the start of Phase 3.

5. Evaluation Questions
Are desired behaviors defined in observable terms?
Do students understand behavioral expectations?
Do staff understand behavioral expectations?
Is there inter-rater reliability among staff in terms of point allocations for behaviors?
Is the school-wide positive behavior support system implemented consistently?
Proposed research questions were chosen by the principal evaluator and program leaders in an effort to obtain information about the most important aspects of the program, include participant perceptions and experience, program integrity and outcomes.

6. Evaluation Questions (continued)
Do staff buy-in to the importance of the positive behavior support program?
Do staff find the positive behavior support program effective?
Does the school-wide positive behavior support system lead to positive student behaviors?
Does the school-wide positive behavior support system foster a safe school environment?

7. Setting
12-months per year, five-days per week, comprehensive interdisciplinary, psychiatric day treatment program
Collaboration between a private hospital and local school district
4 classrooms, referred to as teams in which students are grouped by grade and ability, 8:1:1 ratio
Program accommodates 25 students, ages 5-12
Diverse staff that includes professionals from a variety of disciplines
Students receive milieu, individual and group therapy, as well as medication management and educational services

8. Challenges before PBS
Two separate systems under one roof, trying to provide different services to the children
Different point of reference for educational and clinical staffs
Each system uses its own “language”
Not enough focus on the positives
Inconsistent behavioral expectations throughout the program
Poor communication among staff

9. School-Wide PBS Joined New York State PBIS initiative in 2004
Required commitment of hospital and education administration
Received support from PBIS coaches
Created multidisciplinary committee
Created matrix of behavioral expectations in collaboration with educational and clinical staff
The positive behavior support system in place utilizes a points system, through which students earn points for half-hour blocks throughout the school day in each of four behavioral areas: respect, responsibility, safety and citizenship. Students may earn 0, 1 or 2 points in each area for each ½ hour time period. Points are allocated and sheets filled out by a member of the educational or clinical staff based in the student’s classroom. Levels are awarded to students based on total points earned each day. Students who achieve Levels 4 and 5 get a special reward at the end of the day. In addition to rewarding students for positive behaviors, students receive consequences for aggressive or unsafe behaviors in the form of a level 2 drop, in which a student is put on Level 2 (the lowest level that they can earn) for the remainder of the day and the following day, until a higher level is earned. In instances where students are exhibiting behaviors that are too disruptive to be managed in the classroom or seem to be a precipitant to an aggressive or unsafe episode, students are asked to take a cool down or a time-out, depending on the severity of the behavior. Time-outs are instances of time spent outside of the classroom following inappropriate behavior, and cool-downs are instances of time spent outside of the classroom in an effort to regain control over one’s behavior. Students complete time-outs for the number of minutes that is equal to their age (e.g. a 9-year-old must complete a 9 minute time-out), and students cannot earn positive points during time-outs. Additionally, since the behavior support program is based on earning point for positive behaviors, students cannot earn points when they are absent or uninvolved in program or educational activities (e.g. sleeping).

10. School-Wide PBS Daily Level System
Students are given 0, 1 or 2 points every 30 minutes in four areas
Respect, Responsibility, Safety, & Citizenship
Points are allocated by teacher, TA and MHW in student’s classroom
Daily reward system based on accumulation of points and corresponding level earned (2-5)
Students can automatically be dropped to Level 2 for unsafe behaviors
Staff is encouraged to use positive statements when interacting with children at all times
Students are encouraged to take Cool Downs to manage their behavior
Staff instruct students to take Time Out when behavior cannot be managed in classroom
Weekly rewards given based on accumulated daily levels

11. Point Sheet

12. Participants Students and staff N ~ 50
Staff consent, parental verbal consent and student assent obtained prior to enrolling
N ~ 50
n = ~25 students (ages 7-12)
n = ~25 staff

13. Data Collection Procedures: Quantitative
Frequencies
Daily levels earned by students
Level 2 drops (instances of unsafe behavior)
Inter-rater reliability
25% of point sheets completed by two staff members independently for a period of 2 weeks, for each classroom during phases 1 and 3

14. Data Collection Procedures: Quantitative/Qualitative
Staff Survey assessed 1) Buy-in to the PBS program, 2) understanding of the PBS program, and 3) implementation of the PBS program
Combination of 20 Likert-scale items and 8 open-ended questions
Likert scale items regarding buy-in were rated on a scale of 1-3 (not important to very important)
Likert scale items regarding understanding and implementation of program were rated on a scale of 1-4 (strongly disagree to strongly agree)
Surveys were completed anonymously

15. Data Collection Procedures: Qualitative
Staff focus groups were completed separately for educational and hospital staff
Facilitated by one of the evaluators
Two independent note takers recorded staff comments
Thematic elements were summarized by the evaluators after each focus group

16. Focus Group Protocol
In your opinion, what is the purpose of the PBS program?
Does the current implementation of the PBS program lead to positive student behaviors, and a safe school environment? Please explain?
Do you feel that you have received adequate training and support to reliably score point sheets and deliver time-outs and level 2 drops?
What elements of the PBS program would you change in order to make it more effective and supportive of staff and students?

17. Data Collection Procedures: Qualitative
Student Focus Groups
What do respect, responsibility, safety and citizenship mean?
What behaviors do you have to do to earn Level 4 or 5?
What behaviors would cause you a get Level 2 drop?

18. Results: Phase 1 Assessment (11/07-12/07)
Overall reliability of point sheets is good (.86)
Reliability by behavioral category ranged from .81 to .92
Areas for improvement: Overall reliability can be better, and better agreement on “Responsibility” (.82) and “Respect” (.81)
Survey revealed that staff as a whole is invested in PBS program (average score = 2.6/3.0)
Understanding of program (average score = 2.5/4.0) and implementation of program (average score = 2.6/4.0) need improvement
Level 2 drops account for 14% of total number of levels earned
Levels 4 or 5 account for 70% of total number of levels earned

19. Feedback from Focus Groups
Staff was in agreement on understanding purpose of PBS program
General agreement on modeling and teaching positive behavior
Increase the focus on education
Mixed responses regarding the success of the current implementation and creating safe learning environment
Some felt that it works, but we were not fully implementing the program, while others felt that it was not effective
Strong sentiment that program was not as effective as it had been in past and we were not implementing it consistently

20. Feedback from Focus Groups
Across the board, staff felt that they had not received sufficient training
Most staff felt that they had not been sufficiently trained, but learned on the job
This led to discrepancies in implementation
Most staff felt they personally could benefit from ongoing training

21. Feedback from Focus Groups
When asked what they would change, staff discussed need for additional staff and training
In addition, staff identified need for modifications to reward system, and use of more visual reminders and tools for classroom and milieu

22. Feedback from Student Focus Groups
Children were aware of rules of conduct, but unclear as to distinctions between four categories of behavior
Safety was most clearly understood
Younger children unclear about distinctions between use of cool down, time out, and level drop
Lack of clarity between behavior and consequence

23. Phase 2: Interventions (1/08-3/08)
Met with PBS committee and staff regarding results of assessment
Conducted 5 staff trainings
Worked with teachers to provide “refreshers” to students around behavioral expectations
Created new posters to hang in classrooms and milieu to remind students about expected behaviors
Updated matrix of behaviors
Updated staff manual and created staff training protocol for new staff

24. Results: Phase 3 Follow up (4/08-5/08)
Overall reliability of point sheets remains the same (.87)
Reliability by behavioral category ranged from .82 To .92
Reliability improved on “Responsibility” (.85) and “Respect” (.83), indicating better understanding of these categories
Survey revealed that staff investment in PBS program diminished slightly (average score = 2.4/3.0)
Understanding of program (average score = 3.0/4.0) and implementation of program improved (average score = 3.1/4.0)
Level 2 drops account for 17% of total number of levels earned, which is a slight increase from phase 1
Levels 4 or 5 account for 66% of total number of levels earned, which is a slight decrease from phase 1
Non-parametric (Wilcoxon) test revealed that these differences were not statistically significant
Also, we plan on doing item analysis with items from the survey.

25. Feedback from Phase 3 Focus Groups
Staff continues to be in agreement on the purpose of the PBS program
General agreement on modeling and teaching positive behaviors
Promote a safe environment for staff and students
Increase focus on learning
General consensus that the current PBS program may not be an adequate approach for some of the students at Bard House, but is working well for others
PBS would be more effective if it were reinforced at home
Some feel that PBS promotes a safe environment, others feel that the program is not a safe environment despite PBS
It is not that PBIS isn’t a good program, but that these kids need a more intensive intervention in order to do well.

26. Feedback from Phase 3 Focus Groups
Active supervision on PBS is needed
Some issues were indirectly related to PBS effectiveness
need for additional staff
staff burnout
poor communication between educational and hospital staff
Across the board, staff felt that the trainings that were given were helpful and adequate
Trainings helped improve confidence, understanding and implementation of program
Some staff still feel that they are inconsistencies regarding what constitutes a level drop
Ongoing training is necessary

27. Feedback from Phase 3 Focus Groups
When asked what they would change about the program at this point, staff discussed:
Need for additional staff
More effective and consistent rewards for students
Better communication among staff
Positive reinforcement for students at home

28. Discussion
The evaluation was successful in that we were able to identify strengths and weaknesses, and identify areas of concern to be addressed through interventions
We were able to implement a variety of interventions, and received feedback that interventions were successful
For example, interventions increased staff confidence, understanding, and implementation of the program
We also increased accuracy of utilizing point sheets
Through this evaluation, we were not able to effectively change overall safety of program or student behaviors
We were able to develop plan for future trainings and program modifications that we were not able to implement during our limited time frame
Strength: staff were in agreement on purpose of program and seemed to understand the program
Weakness: staff were not confident in their ability to implement, and there were inconsistencies among staff

29. Limitations Incomplete data collection due to variety of factors.
Fewer surveys were returned in phase 3 than in phase 1
Student focus group could not be completed in phase 3 due to scheduling
Internal evaluation, inherent biases and censoring of participants
Ongoing staff and program changes
Timeline may not have been sufficient to see an effect on student outcomes
Evaluation was affected by the larger systems within which the program exists (hospital, public school)
Lack of staff motivation, changes in staffing, voluntary participation may have played a role.
Student focus group could not be repeated in phase 3 due to scheduling
Fluctuations in school year also may have played a role.

30. Recommendations Ongoing evaluation of the program
Ongoing staff trainings to increase reliability of point allocations and level drops
Continue to tailor PBS to meet the needs of program
Increase communication among staff
Increase involvement of parents and caregivers with the PBS program
Continue to implement program changes over summer and next academic year

31. Contact: Elana Weinberger elana@educ.umass.edu
Questions?
Contact:
Elana Weinberger