Presentation is loading. Please wait.

Presentation is loading. Please wait.

Healthy Schools Leadership Program (HSLP) Evaluation Plan February 2006.

Similar presentations


Presentation on theme: "Healthy Schools Leadership Program (HSLP) Evaluation Plan February 2006."— Presentation transcript:

1 Healthy Schools Leadership Program (HSLP) Evaluation Plan February 2006

2 What Im going to talk about Present & get your feedback on Evaluation Plan for HSLP cohort Present & get your feedback on Evaluation Plan for HSLP cohort Generate ideas for your own assessment/evaluation work Generate ideas for your own assessment/evaluation work Ideas for using the Healthy Youth Survey Ideas for using the Healthy Youth Survey Please consider this a discussion! Please consider this a discussion!

3 George tried to be a good monkey…

4 The Primary Evaluation Question Is the Healthy Schools Leadership Program (HSLP) effective for making schools healthier? Is the Healthy Schools Leadership Program (HSLP) effective for making schools healthier? Note that this is a state-level evaluation question Note that this is a state-level evaluation question I will present an ambitious scenario for evaluation I will present an ambitious scenario for evaluation We will attempt to be as rigorous as possible We will attempt to be as rigorous as possible Im not a schools expert… you are the schools experts Im not a schools expert… you are the schools experts

5 Caveats to Evaluation Model Small numbers of schools (n=15) Small numbers of schools (n=15) Change takes time – especially for long-term outcomes Change takes time – especially for long-term outcomes Our comparison group is the rest of the state Our comparison group is the rest of the state If findings are positive, some will argue that this is the results of your existing capacity/readiness If findings are positive, some will argue that this is the results of your existing capacity/readiness The plan is ambitious – looking for additional resources to support evaluation The plan is ambitious – looking for additional resources to support evaluation

6 What we can say right now with certainty is that if you spend resources implementing coordinated school health it doesnt appear to make your academic achievement any worse…. What we can say right now with certainty is that if you spend resources implementing coordinated school health it doesnt appear to make your academic achievement any worse…. Id like to say something stronger

7 Describe the Program

8 Coordinated School Health Goals Health Education Health Education Physical Education Physical Education Counseling/Support Services Counseling/Support Services Food/Nutrition Services Food/Nutrition Services Healthy Environment Healthy Environment Health Services Health Services Family/Community Involvement Family/Community Involvement Staff Wellness Staff Wellness

9 Simplified Logic Model B. Activities Asst Plan Evaluate A. Inputs Teams HSLP training C. Outputs Asst results Action Plan E. Intermediate Outcomes Policy, procedure, environment changes Sustainability D. Short-term Outcomes Results from action plan activity F. Long-term Outcomes Staff/studnt behavior change G. Impacts Health improvemt Academic achievemt Catalysts Barriers Non-HSLP External Factors

10 Getting the Right Perspective: Evaluation Questions

11 Specific Aim 1. Describe the extent to which the inputs and activities are sufficient to generate outputs – assessment results, action plans (A/B C, Descriptive) Were teams able to assemble wellness committees? How did they function? Were they able to complete assessment and planning? What priorities were identified? What were the barriers and motivating factors related to output completion? Were teams able to assemble wellness committees? How did they function? Were they able to complete assessment and planning? What priorities were identified? What were the barriers and motivating factors related to output completion? Are there differences among HSLP teams in satisfaction with outputs? What are the factors (HSLP or other) associated with high/low satisfaction? Was HSLP a critical factor in achievements of high-quality outputs? Are there differences among HSLP teams in satisfaction with outputs? What are the factors (HSLP or other) associated with high/low satisfaction? Was HSLP a critical factor in achievements of high-quality outputs? Data Sources: Team interviews, action plans, Healthy Schools Report Card

12 Hypothetical Findings All schools assembled teams; X% included community members All schools assembled teams; X% included community members 12 of 15 teams created formal action plans 12 of 15 teams created formal action plans All schools identified at least one nutrition priority All schools identified at least one nutrition priority School teams that were very satisfied with their plans were more likely to have involved community members and had a clear process for translating assessment results into an action plan School teams that were very satisfied with their plans were more likely to have involved community members and had a clear process for translating assessment results into an action plan

13 Specific Aim 2. Describe the extent to which participation in the HSLP is associated with short-term outcomes – early results of action plans (A/B D, Descriptive) Were school wellness teams able to use plans to initiate change? How did implementation activity begin? Which priorities did schools take action on first? How were responsibilities identified? What factors were associated with transition from planning to action? Were school wellness teams able to use plans to initiate change? How did implementation activity begin? Which priorities did schools take action on first? How were responsibilities identified? What factors were associated with transition from planning to action? Are there differences among HSLP schools in their satisfaction with/quality of early implementation? Were some kinds of activities more easily translated to quality action? What are the factors (HSLP or other) associated with high/low satisfaction? Was HSLP a critical factor in initiating high-quality work? Are there differences among HSLP schools in their satisfaction with/quality of early implementation? Were some kinds of activities more easily translated to quality action? What are the factors (HSLP or other) associated with high/low satisfaction? Was HSLP a critical factor in initiating high-quality work? Data Sources: Team interviews, action plan reports, Healthy Schools Report Card

14 Hypothetical Findings 10 of 15 schools increased the overall number of maintenance items on their Healthy Schools Report Card 10 of 15 schools increased the overall number of maintenance items on their Healthy Schools Report Card Teams reported highest satisfaction with their work on XXX priorities Teams reported highest satisfaction with their work on XXX priorities XX Teams (X%) said they strongly agreed that change would not have occurred without the HSLP training and technical assistance XX Teams (X%) said they strongly agreed that change would not have occurred without the HSLP training and technical assistance

15 Specific Aim 3. Describe the extent to which schools that participated in HSLP change their environments (A/B E, Pre/post comparison to non-HSLP schools) Did HSLP school environments change, and were changes different than those of non-participating schools? Which changes were achieved? Did HSLP school environments change, and were changes different than those of non-participating schools? Which changes were achieved? Are there differences among HSLP schools that achieved policy/procedure/environment change vs. those that did not? Are there differences among HSLP schools that achieved policy/procedure/environment change vs. those that did not? How has sustainability been addressed? What additional resources have been attracted for school health? Has institutionalization of school health work been achieved? How has sustainability been addressed? What additional resources have been attracted for school health? Has institutionalization of school health work been achieved? Data Sources: SHEP, Healthy Schools Report Card, HYS (perceived school environments), Staff wellness survey (if perceived environment indicators included)

16 Sample Findings Source: SHEP 2004

17 Specific Aim 4. Describe the extent to which schools that participated in HSLP change their student/staff behaviors (A/B F, Pre/post comparison to non-HSLP schools for students, self-comparison for staff) Did HSLP schools student behaviors change, and were changes different than those of non-participating schools? Did HSLP schools student behaviors change, and were changes different than those of non-participating schools? Did HSLP schools staff behaviors change? Did HSLP schools staff behaviors change? Are there differences among HSLP schools that achieved behavior change vs. those that did not? Are there differences among HSLP schools that achieved behavior change vs. those that did not? Data Sources: HYS, Staff Wellness Survey

18 Sample Findings Source: Healthy Youth Survey

19 Specific Aim 5. Describe the extent to which schools that participated in HSLP have had health impact (A/B G, Pre/post comparison to non-HSLP schools for WASL, self-comparison for absenteeism) Did HSLP schools student academic achievement change, and were changes different than those of non-participating schools? Did HSLP schools student academic achievement change, and were changes different than those of non-participating schools? Did HSLP schools student and/or staff absenteeism rates change? Did HSLP schools student and/or staff absenteeism rates change? Are there differences among HSLP schools that achieved impact vs. those that did not? Are there differences among HSLP schools that achieved impact vs. those that did not? Data Sources: WASL, Student absenteeism measures, Staff absenteeism measures

20 Sample Findings

21 Data Sources: Gathering Credible Evidence Not everything that counts can be counted; and not everything that can be counted counts. Albert Einstein

22 Data Sources Proposing to use data you already collect, or new data collection related to your HSLP implementation Proposing to use data you already collect, or new data collection related to your HSLP implementation Exception: interviews of team/administration Exception: interviews of team/administration For anything where we want to look at everyone together, methods have to be standard… For anything where we want to look at everyone together, methods have to be standard… For some things it would be fine to have sentinel schools For some things it would be fine to have sentinel schools See school-specific data suggestions later See school-specific data suggestions later

23 Team Interviews During Spring 06 develop interview protocols for assessing function and satisfaction among HSLP participants (see evaluation questions) During Spring 06 develop interview protocols for assessing function and satisfaction among HSLP participants (see evaluation questions) Repeat in Spring 08 Repeat in Spring 08 Augment with interviews with other wellness committee members (or lead administrators?) Augment with interviews with other wellness committee members (or lead administrators?) Your role: participate in interviews Your role: participate in interviews

24 Healthy Schools Report Card Conduct assessment to identify priorities for action planning Conduct assessment to identify priorities for action planning ASCD (Association for Supervision and Curriculum Development) has a clear process/instrument ASCD (Association for Supervision and Curriculum Development) has a clear process/instrument Your role: use good assessment to develop plans, provide a copy of whatever baseline (spring 06) and follow-up (spring 08) assessment results Your role: use good assessment to develop plans, provide a copy of whatever baseline (spring 06) and follow-up (spring 08) assessment results

25 HSRC Component Areas 1. School health program policy & strategic planning 2. Coordination of school health programs 3. Social and emotional climate 4. Family and community involvement 5. School facilities and transportation 6. Health education 7. Physical education and physical activity 8. Food and nutrition services 9. School health services 10. Counseling, psychological and social work services 11. School-site health promotion for staff

26 Green = short-term high priority Green = short-term high priority Red = short-term lower priority Red = short-term lower priority Lime = long-term, highest priority Lime = long-term, highest priority Blue = long-term, high priority Blue = long-term, high priority Orange = long-term, lower priority Orange = long-term, lower priority No color = Maintenance No color = Maintenance HSRC Indicator Priority Rankings Multiple specific topics within components Multiple specific topics within components Rate with your team for each: current status, perceived benefit, perceived effort Rate with your team for each: current status, perceived benefit, perceived effort Color-coded result report for combined scores Color-coded result report for combined scores

27 HSRC Sample Component-Topic School health program policy & strategic planning School health program policy & strategic planning All staff members are provided with time/resources to comply with health program policies All staff members are provided with time/resources to comply with health program policies Coordination of school health programs Coordination of school health programs School health program members regularly inform the principal, district superintendent, or school board of current developments in the school health program. School health program members regularly inform the principal, district superintendent, or school board of current developments in the school health program.

28 Annual Action Plans & Reports Design your action plan based on assessment results Design your action plan based on assessment results Your role: provide action plan annually, report of progress annually (format?) Your role: provide action plan annually, report of progress annually (format?)

29 School Health Education Profile (SHEP) Survey of Secondary School Principals and Lead Health Teachers Survey of Secondary School Principals and Lead Health Teachers Sponsored by CDC Sponsored by CDC Will be sent in Spring 06 and Spring 08 Will be sent in Spring 06 and Spring 08 Your role: encourage completion of the survey Your role: encourage completion of the survey

30 SHEP/Profiles Data Examples: School Health Advisory Groups Have a School Health Advisory Group Have a School Health Advisory Group 59% of high schools 59% of high schools 56% of middle schools 56% of middle schools 46% of combined secondary schools 46% of combined secondary schools 55% overall 55% overall Similar to 52% total in 2002 Similar to 52% total in 2002

31 SHEP/Profiles Data Examples: Tobacco A designated individual has primary responsibility for seeing that the tobacco prevention policy is enforced A designated individual has primary responsibility for seeing that the tobacco prevention policy is enforced 64% high school 64% high school 68% middle school 68% middle school 59% secondary combined 59% secondary combined 65% overall 65% overall Similar to 68% overall in 2002 Similar to 68% overall in 2002 No association with school advisory presence No association with school advisory presence

32 SHEP/Profiles Data Examples: Nutrition Students can purchase fresh fruits/vegetables from vending machines or at the school store, canteen, or snack bar Students can purchase fresh fruits/vegetables from vending machines or at the school store, canteen, or snack bar 49% high school 49% high school 45% middle school 45% middle school 39% secondary combined 39% secondary combined 46% overall 46% overall Slight improvement from 39% overall in 2002 Slight improvement from 39% overall in 2002 No association with school advisory presence No association with school advisory presence

33 SHEP/Profiles Data Examples: Asthma Obtain and use an Asthma Action Plan (or Individualized Health Plan) for all students with asthma Obtain and use an Asthma Action Plan (or Individualized Health Plan) for all students with asthma 77% high school 77% high school 77% middle school 77% middle school 74% secondary combined 74% secondary combined 76% overall 76% overall Improvement from 62% overall in 2002 Improvement from 62% overall in 2002 Schools with advisories are twice as likely to obtain asthma plans as schools without advisories Schools with advisories are twice as likely to obtain asthma plans as schools without advisories

34 Staff Wellness Survey Designed by CDC – available in March Designed by CDC – available in March Assuming we can modify (shorten) if needed Assuming we can modify (shorten) if needed Assuming we (DOH/OSPI) can provide as online survey and give data back to school teams Assuming we (DOH/OSPI) can provide as online survey and give data back to school teams Are there other ways to measure staff absences? Are there other ways to measure staff absences? Your role: engage staff to participate Your role: engage staff to participate Caution: poor participation will not yield useful data! Caution: poor participation will not yield useful data!

35 Healthy Youth Survey (HYS) Administered Fall of even years Administered Fall of even years Grades 6,8,10,12 Grades 6,8,10,12 How many HSLP schools participated in 2002, 2004? How many HSLP schools participated in 2002, 2004? Most powerful comparison will be Most powerful comparison will be Your role: register to participate in 2006 (soon!) Your role: register to participate in 2006 (soon!)

36 Student Absenteeism NCLB requires measurement of excused/unexcused absences NCLB requires measurement of excused/unexcused absences Assuming no change in non-illness excused absences, we can monitor changes in this rate over time Assuming no change in non-illness excused absences, we can monitor changes in this rate over time Your role: provide annual absence rates Your role: provide annual absence rates

37 Academic Achievement/WASL Annual, building-level Annual, building-level Grades 4,7,10 Grades 4,7,10 Math, Reading, Writing Math, Reading, Writing Your role: none needed Your role: none needed

38 Other School Factors Free & reduced lunch enrollment [this may be an outcome, consider different measure of SES] Free & reduced lunch enrollment [this may be an outcome, consider different measure of SES] Percent minority enrollment Percent minority enrollment School/district size/enrollment School/district size/enrollment School level (elementary, middle, high, combination) School level (elementary, middle, high, combination) Urban/rural location Urban/rural location Baseline levels of outcomes Baseline levels of outcomes Other (baseline) funding sources for health improvement Other (baseline) funding sources for health improvement Your role: none – we can obtain Your role: none – we can obtain

39 Too much information?

40 Timeline

41 School-level Evaluation We wont evaluate you as individual teams We wont evaluate you as individual teams Evaluating yourself on a few points would be helpful in telling your story later Evaluating yourself on a few points would be helpful in telling your story later Suggest designating a lead for local evaluation (just like any implementation piece) Suggest designating a lead for local evaluation (just like any implementation piece)

42 Sample School-level Evaluation Data vending revenue data vending revenue data key informant interviews with students or staff key informant interviews with students or staff tracking of policy enforcement activities tracking of policy enforcement activities other data gathering from communities other data gathering from communities health room utilization data health room utilization data student absences – use more advanced tracking system (Public Health Tracking System grant available from DOH environmental health) student absences – use more advanced tracking system (Public Health Tracking System grant available from DOH environmental health) lots of (digital) pictures of your environment – now and later lots of (digital) pictures of your environment – now and later

43 Still Awake??

44 Healthy Youth Survey

45 Why do I need HYS now? Contribute to assessment, planning Contribute to assessment, planning Identification of your priorities Identification of your priorities Describing affected students Describing affected students Gain attention for student health as a priority Gain attention for student health as a priority

46 HYS 2004: Collaborative Effort of OSPI, DOH, DSHS, CTED, FPC, and RMC Research State-level simple random sample of schools State-level simple random sample of schools County-level samples drawn as appropriate County-level samples drawn as appropriate Non-sampled schools also invited to participate Non-sampled schools also invited to participate Survey consistently administered in the Fall of even years Survey consistently administered in the Fall of even years Survey given to 6 th, 8 th, 10 th, and 12 th graders Survey given to 6 th, 8 th, 10 th, and 12 th graders Survey booklets have one-page tear-off answer sheet Survey booklets have one-page tear-off answer sheet Survey for 8 th, 10 th, and 12 th graders uses 2 form interleaved administration Survey for 8 th, 10 th, and 12 th graders uses 2 form interleaved administration

47 The Healthy Youth Survey 2004 was completed… The Healthy Youth Survey 2004 was completed… HYS 2004: Participation By 185,095 students… In 1,013 schools… In 235 school districts… In all 39 counties.

48 Risk & Protective Factor Data Consistent with CTC & MTF Surveys Development led by DSHS/DASA & OSPI WSSAHB Questionnaire Demographics Alcohol, Tobacco, other drugs School risk/protective factors Community risk/protective factors Peer-Individual-Family risk/ protective factors

49 Youth Health Behavior Data Consistent with National YRBS & YTS Development led by DOH YRBS Questionnaire Demographics Alcohol, Tobacco, Other drugs Nutrition Physical Activity Safety Behaviors (Helmets, Seatbelts) Mental Health/Depression/Suicide Additional Tobacco Indicators

50 Getting the Best of Both Form A WSSAHB- like Form B YRBS-like Survey Core

51 6 th Grade Survey One Form Survey Core + Mutually Agreed Items

52 Core Survey Items Demographics Demographics Alcohol, Tobacco and Other Drug Use Items Alcohol, Tobacco and Other Drug Use Items Key Violence-Related Items Key Violence-Related Items School-Specific Asset Items School-Specific Asset Items Depression Depression

53 Administration procedures Safe and confidential environment Safe and confidential environment Standardized administration procedures (e.g., coordinator training, teacher training, written instructions, teacher stays in room but at desk, single class period to avoid discussion, absent students do not make up) Standardized administration procedures (e.g., coordinator training, teacher training, written instructions, teacher stays in room but at desk, single class period to avoid discussion, absent students do not make up) Students informed of importance of survey Students informed of importance of survey No student name or other identifying information No student name or other identifying information Students place own answer sheet in envelope Students place own answer sheet in envelope Students provided resource list Students provided resource list

54 Data Cleaning Procedures Inconsistent answers Inconsistent answers Evidence of faking high level of substance use Evidence of faking high level of substance use Dishonesty Dishonesty Wrong grade Wrong grade

55 What is a Confidence Interval? The reported value is probably a little different than the true value for all your students We are 95% confident that the true value is within the +/- range, called the confidence interval

56 Interpreting CIs Graphically

57 Local Report Contents Introduction and Overview Highlights of Local Results Selected Results by Gender Responses to Items Risk and Protective Factors List of Core Items & Item index

58 For your continued exploration… HYS06/index.htm

59 Source: 2004 Healthy Youth Survey, state sample only. Graph uses unadjusted estimates; odds ratios are for estimates adjusted for grade ( ) and maternal education (3-level). Students with severe persistent asthma symptoms had increased odds for academic risk

60 Source: 2004 Healthy Youth Survey, state sample only. Graph uses unadjusted estimates; odds ratios are for estimates adjusted for grade ( ) and maternal education (3-level). Students with diabetes had increased odds for academic risk

61 Source: 2004 Healthy Youth Survey, state sample only. Graph uses unadjusted estimates; odds ratios are for estimates adjusted for grade ( ) and maternal education (3-level). Students who smoked any cigarettes during the past month had increased odds for academic risk

62 Healthy Youth Survey Website Some quick crosstabs examples you could run… Some quick crosstabs examples you could run… Kids who smoke are less likely to report having access to an ATOD counselor (61% among smokers v. 71% among non-smokers, 10 th grade state sample) Kids who smoke are less likely to report having access to an ATOD counselor (61% among smokers v. 71% among non-smokers, 10 th grade state sample) Kids who report being bullied are also more likely to report depression (46% depression among bullied v. 29% among non-bullied, 10 th grade) Kids who report being bullied are also more likely to report depression (46% depression among bullied v. 29% among non-bullied, 10 th grade) Kids who drank more than 2 soda pops per day were more likely to report getting Ds and Fs (7% vs. 15% among heavy pop-drinkers, 10 th grade) Kids who drank more than 2 soda pops per day were more likely to report getting Ds and Fs (7% vs. 15% among heavy pop-drinkers, 10 th grade)

63 Healthy Youth Survey 2006 – HSLP related measures Physical Education Physical Education Number of days of PE per week Number of days of PE per week How long doing actual activity in PE class How long doing actual activity in PE class Counseling/Student Support Services Counseling/Student Support Services Is there a school counselor to talk about ATOD Is there a school counselor to talk about ATOD Is there someone you could go to for help Is there someone you could go to for help Food/Nutrition Services Food/Nutrition Services New junk food module in 2006 asks about soda pop and junk/fried food consumption at school, also breakfast New junk food module in 2006 asks about soda pop and junk/fried food consumption at school, also breakfast Fruit & vegetable 5-a-day consumption Fruit & vegetable 5-a-day consumption Soda pops per day (previous day) Soda pops per day (previous day)

64 HYS 06 – HSLP-related measures Healthy Environment Healthy Environment Feeling safe at school Feeling safe at school Enjoy being in school/hate being in school Enjoy being in school/hate being in school Carry weapon on school property Carry weapon on school property Bullying Bullying Teachers care about me Teachers care about me Using tobacco on school property Using tobacco on school property Been drunk or high at school Been drunk or high at school Tobacco rules are usually enforced Tobacco rules are usually enforced Health Education Health Education Number of times received tobacco education Number of times received tobacco education Number of times practices ways to say no Number of times practices ways to say no How good in school at educating about HIV How good in school at educating about HIV

65 HYS 06 – HSLP-Related Measures Family/Community Involvement Family/Community Involvement Dinner with family at home Dinner with family at home Parent/guardian discussed tobacco at home Parent/guardian discussed tobacco at home How easy to get tobacco/alcohol if you wanted some How easy to get tobacco/alcohol if you wanted some Student Support Services Student Support Services Ever had an asthma action plan Ever had an asthma action plan

66 Your Questions… What if I cant find my school or district report? What if I cant find my school or district report? What if I just need some general help getting started with HYS data or the website? What if I just need some general help getting started with HYS data or the website? What if I want more analysis done with our data? What if I want more analysis done with our data? What if I have specific questions about HYS data and our Health Department staff cant help me? What if I have specific questions about HYS data and our Health Department staff cant help me?

67 You made it!

68 What we talked about… Present & get your feedback on Evaluation Plan for HSLP cohort Present & get your feedback on Evaluation Plan for HSLP cohort Generate ideas for your own assessment/evaluation work Generate ideas for your own assessment/evaluation work Ideas for using the Healthy Youth Survey Ideas for using the Healthy Youth Survey

69 Here to help! General Questions Lori, Martin, Pam HSLP Evaluation Detail Julia Dilley Healthy Youth Survey Diane Pilkey


Download ppt "Healthy Schools Leadership Program (HSLP) Evaluation Plan February 2006."

Similar presentations


Ads by Google