Hopkins PSMG Oct An Incomplete Model for Trial Designs for Moving to Scale: The Whole Day Program C Hendricks Brown Sheppard Kellam Jeanne Poduska Amy Windham John Reid Carla Ford Natalie Keegan Supported by R01 DA PI: Kellam Ro1 MH PI: Brown Baltimore City Public School System

Hopkins PSMG Oct Premise The Whole Day Program is where the school would want to be in 5 years, systemwide. What specific steps should be examined along the way?

Hopkins PSMG Oct Primary Question Effective? Who benefits, for how long? Who doesn’t benefit, who is harmed? Efficacy Trial 12 Schools 24 Classrooms 600 Students 3 Time Periods

Hopkins PSMG Oct Differential impact on spring of first grade reading achievement by baseline reading (fall of first grade): control against two interventions Control Classroom / Parent Interventions Brown: Statistical Power

Hopkins PSMG Oct Presuming Effectiveness Sustainabililty Do WD teachers continue to sustain the same effects on children that they have in following years? 12 WD Teachers vs Themselves & 12 Wait- Listed Controls Scalability Can program be successfully implemented outside the original 12 schools throughout the school system?

Hopkins PSMG Oct Scale to What? From 2 To all 4 of the School Areas From 12 To 20 To 135 Schools From 24 To 40 To 400 First-Grade Classrooms

Hopkins PSMG Oct An Evidence-Based Decision Tree Efficacious especially w/ High-Risk Youth? Sustainable? Feasible to Scale? Y Y Go to Scale! Y N N N

Hopkins PSMG Oct The Real Decision Tree for BCPSS Something Different

Hopkins PSMG Oct Person, Place, and Time Child Classroom / Teacher (1 st Grade) School Area (Region of Baltimore) Time – Scale(s): 10 Secs, Min, Quarter, Year Child : Track from 1 st through 3 rd Grade Teacher: Track over 3 years (cohorts)

Hopkins PSMG Oct Randomization 20 Schools in 2 Areas randomly assigned to One of 12 schools where trial begins One of 8 schools that are “wait-listed” Children randomly assigned to classes within school Teachers/Classes randomly assigned to WD or wait-list

Hopkins PSMG Oct Baltimore Whole Day Trial Assignment of Classrooms/Teachers in First Year Half the Teachers are Wait-Listed for WD to 3 rd Year Replicate Same Design w/ SC and and WD BOTH in 12 Different Schools Brown: Statistical Power

Hopkins PSMG Oct Effectiveness Evaluation Compare 3 years of Outcomes for Children in the 12 WD classes to those in 12 Standard classes -- 1 st Cohort Examine Level of Program Implementation in the 12 WD and 12 Standard Classrooms in 1 st Cohort

Hopkins PSMG Oct Classroom Observations by Independent Observers 3 times per year WD and SC classrooms AM-Reading Instruction PM-Other subjects Teacher Practices Timed Observations Checklist Global Ratings of Quality Student Behavior On-Task/Off-task Ratings of Disruptive, Aggressive, and Shy Behavior

Hopkins PSMG Oct Classroom Observation Data Collection Form Teacher Practices and Student Behavior Actual Form

Hopkins PSMG Oct Domains of Reading Instruction Word Work – phonemic awareness, alphabetic instruction, letter recognition, work work Reading Comprehension – reading comprehension, previewing a book, reading books Language – oral language, vocabulary, reading own writing, writing, grammar Non-reading Instruction – interruptions, teacher out of classroom Giving instructionally related reading directions Feedback in response to behavior Feedback in response to a reading relevant comment or event

Hopkins PSMG Oct Classroom Observations Baseline Data Overview 12 schools 24 teachers (12 Whole Day, 12 Standard setting) Each teacher observed 3 times per year, at each time point: –1 reading instruction (am observation) –1 non-reading instruction (pm observation) Length of observations at each time point: Teacher: mean = 164 minutes Each student: mean = 9 min/student

Hopkins PSMG Oct Given Teacher Instructional Practices, Classroom Behavior (Standard Practice Classrooms during Reading Instruction at Baseline) Well behaved v Fairly well behaved: X 2 =36.83 p<10 -7 Word WorkReading Comprehension Language Well behaved % % % Fairly well behaved % % % Poorly behaved 1.3% 7 2.3% % Chaotic5 1.6% 00

Hopkins PSMG Oct Student behavior: Reading v Other subjects (Standard Practice Classrooms at Baseline) Never off-task v Ever off-task: X 2 =29.80 p<10 -8 AM-ReadingPM-Other subjects Never off task % % Off task-1 interval13 1.1% % Off task-2 intervals5.4% % Off task-3 intervals42 3.5% %

Hopkins PSMG Oct Effectiveness Results for Cohort 1 (1 of 2) At Baseline Level of Off-Task Behavior is Well Balanced across Intervention Condition Reduction in Off-task Behavior Control 21% WD 9% 50% Reduction in Aggressive Behavior for Males in Less Structured Settings

Hopkins PSMG Oct Graphical Comparison for Off-Task Behavior

Hopkins PSMG Oct Comparison of Teachers

Hopkins PSMG Oct Unsolved Multilevel Modeling Issues Child Gender, Baseline Off-Task Classroom Intervention, Classroom Baseline Aggr School Time-Varying Classroom Context Binary Outcomes

Hopkins PSMG Oct Sustainability Evaluation (Funded) Multilevel Support Compare Teacher Implementation (and Child Outcomes) for 12 WD Teachers in 1 st through 3 rd Year for First Grade Classes

Hopkins PSMG Oct Sustainability Hypotheses and N’s & Years Teacher practices over time will be related to the quality of WD integration at the level of the School Building Team. 12  20  Teacher practices over time will be related to the quality of WD integration at the level of the School Area over time. 2  4

Hopkins PSMG Oct Scalability Design Change in fidelity (Teacher practices) and impact (Child Outcomes) in succeeding years after initial training 12  24  135

Hopkins PSMG Oct Combined RCT and Implementation Trial Trial After a Trial Design

Hopkins PSMG Oct How to Go From 24 to 400 Classes? A Dynamic Wait-Listed type of design Randomly Assign WHEN schools and classes obtain training, coaching, and full implementation Continue to assess intervention fidelity Paired-down measure of Teacher Practices Random Assign Children to Classes?

Hopkins PSMG Oct Classes Schools Waiting Active School Trained Wait Listed Sustained TrainedSustained

Hopkins PSMG Oct Plausible Time Scale # Classes (Schools) Time Period/Cohort New WDSustainabilityComparisonWait-Listed Effectiveness12 (12)012 (Same 12)376 (123) Sustain012 (12)12 (Same 12)376 (123) Scalability12 (12)12 (Same 12)20 (8)356 (115) Full Scalability Yr 1 90 (30)44 (20)40 (30)230 (85) Yr 290 (25)130 (50)40 (25)140 (60) Yr 390 (30)220 (70)40 (30)50 (35) Yr 490 (35)310 (100)0 (0)