Karen Cheung, MPH, Pamela Luna, DrPH, MST, Sarah Merkle, MPH American Evaluation Association Annual Meeting November 11, 2009 The findings and conclusions in this report are those of the authors and do not necessarily represent the official position of the Centers for Disease Control and Prevention

 Background  Process and roles of rapid evaluation  Evaluation focus  Evaluation design and methods  Findings 2

3 Background

 Urban district with 167 schools and 132,000 students  Demographics: ◦ 42% African American, 35% White, 15% Hispanic ◦ 47% of students qualify for free or reduced-price lunch  Nurse to student ratio of 1:1100 4

 Key components of Asthma Program 1.Staff development and education 2.School health teams with an asthma focus 3.Case management (CM) 4.Respiratory therapy services (RTS) 5.Open Airways for Schools (OAS) 5 Student-level services

6 Process and Roles of Rapid Evaluation

 Participatory ◦ Determining what aspects of the program to evaluate  Team-based ◦ Directors of school health, nurse manager ◦ ICF Macro and CDC  Iterative ◦ Refining evaluation questions ◦ Interpreting results 7

 Participate in evaluation during the school year  Review and approve all instruments, including parent consent letters  Facilitate process of school district’s IRB  Assist with recruitment (principals, nurses)  Provide context for results  Own the data 8

 Has previous experience with school sites  Recruit sites  Track parental consent forms  Coordinate logistics  Gather school administrative data  Receive training in data collection  Schedule and collect data  Distribute appropriate incentives 9

 Design the evaluation, with input from school district and CDC  Oversee local site coordinator  Troubleshoot  Conduct evaluation ◦ data collection ◦ data analysis  Prepare final report, presentation, and manuscript for dissemination 10

 Select the school district for evaluation  Participate in determining evaluation design  Provide asthma and programmatic expertise  Provide technical support  Review and approve all aspects of the evaluation  Prepare final report, presentation, and manuscript for dissemination 11

 Monthly teleconferences with school district  Weekly teleconferences between CDC and ICF Macro  Regular updates to all team members  Debriefings during data collection  Interim data analysis reporting  Feedback and revisions on all products 12

13 Evaluation Focus

 Relationship between exposure to service(s) and student outcomes  Scope and depth of Asthma Program components 14

15 Evaluation Design and Methods

 Mixed-methods, longitudinal design  school year  12 elementary schools  10.5% average asthma prevalence  >75% of students in free or reduced lunch program  Quantitative and qualitative data collected through:  Document review  Interviews & focus groups  Student questionnaires 16

 Asthma Management Difficulties (AMD)  Student questionnaires  Hospital/ER visit, overnight hospitalization, waking at night, interference with physical activity 17

 Nurse ratings of level of need for asthma services  Rated as low, medium, or high need based on clinical indicators and symptom frequency  Exposure to student-level services  Obtained from administrative records  Enrollment in CM, RTS, or OAS was considered exposure to direct student services (exposure to any one of the three = yes) 18

19 Pretest Posttest October students eligible; 304 completed the pretest (94%) 286 responded “yes” or “I don’t know” to question about having asthma May students completed the posttest (81.8%) (of those who responded “yes” or “I don’t know” to question about having asthma on pretest)

20 Findings

21  Significant improvement in AMD from pretest to posttest (F=4.447; p=.036)  However, no overall difference in AMD between students with and without exposure to services (F=1.254; p=.264)

22  Despite improvement in AMD among medium and high need students receiving CM, RTS, or OAS, the improvement was not any greater than for students not receiving these services  No evidence that improvement was due to the student-level services of the Asthma Program

23  What might explain these findings? ◦ Previous exposure to services ◦ Variation in program implementation ◦ Possibility of incomplete service records  Analyses did not take into account dosage ◦ AMD may be difficult to change within one school year ◦ Seasonal effects of asthma

 Continue to evaluate the asthma program ◦ Work to increase fidelity of implementation ◦ Examine more proximal outcomes  More students using controller medications  Greater awareness of asthma symptoms and triggers  Maintain records on student exposure to services 24

 School district is using results from this evaluation to improve their current program Collaborating with community partners to reach students seen in hospitals and ERs Offering parent asthma classes through local asthma coalition Implementing OAS as intended  CDC is in the process of clarifying their national guidance, particularly for “case management.” 25

 Improves programs ◦ Provides feedback to schools ◦ Advances school-based health programming efforts  Engages stakeholders  Builds capacity  Is timely  Informs CDC’s work 26

“We appreciate the wonderful opportunity to participate in this evaluation and for everyone's input, support and patience. It has provided us with meaningful data and identified specific recommendations to improve our asthma program as we move forward.” - Director of School Health 27

28 Conducting a Rapid Evaluation in a School District in the Southeast Karen Cheung, MPH