1. Results from the
San Francisco
California Healthy
Kids Survey
Is there an association between school-based health and wellness program utilization & youth developmental assets?
Literature Review
Theoretical framew ork
Study Site
Methods
Results
Implications
Kelly Whitaker, UC Berkeley & ETR Associates
Susan Stone, UC Berkeley
Yolanda Anyon, UC Berkeley
John Shields, ETR Associates

2. What we know. . .
Addressing school-based assets (caring relationships with an adult at school, opportunities for meaningful participation, and high expectations) may enhance student academic performance (Brooks, 2006; Jennings, 2003)
Associations between adolescent health and mental health indicators, developmental assets, and academic performance (Durlak, 2007; Hanson &Austin, 2003; Greenberg, 2003)
Few school-based health and wellness programs have been rigorously evaluated relative to their impact on these academically salient outcome domains (Hoagwood, Olin, et al., 2007; Murray, Low, et al., 2007).
Well-controlled intervention studies almost exclusively assess health and mental health indicators.
Most intervention studies conducted on elementary-aged student populations
Population-based studies show associations between adolescent health risk behaviors, developmental assets, and academic performance (Hanson &Austin, 2003). Research indicates that addressing what are conceived as malleable, school-based assets (caring relationships with an adult at school, opportunities for meaningful participation, and high expectations) may enhance student academic performance (Brooks, 2006; Jennings, 2003). Few school-based health and wellness programs have been rigorously evaluated relative to their impact on these academically salient outcome domains (Hoagwood, Olin, et al., 2007; Murray, Low, et al., 2007). Prior intervention research largely focuses on elementary aged students in highly targeted health and mental health domains.

3. Gap in the Literature. . . Academic Performance?
Hanson & Austin, 2003
Academic Performance
School-based heath programs
Youth development assets
Walker et al., 2010

4. Study Context
The San Francisco Unified School District (SFUSD) has a longstanding commitment to wellness programming and evaluation
The San Francisco Wellness Initiative (SFWI) is a partnership between SFUSD and key city stakeholders:
Department of Children Youth and their Families (DCYF)
Department of Public Health
SWFI programming serves 15 of the 16 traditional and alternative high schools in SFUSD
ETR Associates has evaluated the SFWI since 2001
Overall, the SFUSD student population is racially/ethnically and socio- economically diverse (SFUSD, 2009)
Chinese (37%), Latino (21%), African-American (12%), and Caucasian (9%) students; 20% are English Language Learners (ELL)
43% receive free or reduced lunch
Make it clear that SFUSD student population as a whole.
Don’t say predominant free and reduced.

5. San Francisco Wellness Initiative (SFWI)
School-Based Health & Social Services
in Public High Schools
Prevention, Education &
Youth Development
Medical Assistance & Referrals
Counseling & Case Management
The San Francisco Wellness Initiative, a partnership between The San Francisco Unified School District (SFUSD), The Department of Children Youth and their Families and the Department of Public Health, has been offering coordinated HWS to high schools since Services are intentionally designed to address a wide range of student needs using a public health model (i.e., offering a continuum of primary prevention to indicated intervention strategies).

6. SFWI service utilization (2005-2010)
School Year
Number of Students Served†
Proportion of Total Population
4,339
39% (7 schools)
4,825
38% (11 schools)
6,072
38% (15 schools)
6,609
42% (15 schools)
6,988
45% (15 schools)
7,048
46% (15 schools)
†Unduplicated count of individual students served based on DCYF CMS annual data.

7. SFWI Theory OF CHANGE Wellness Staff Community partners
Resources
School-based service
Service utilization
Hypothesized mediators
Educational outcomes
Wellness Staff
Community partners
Counseling and case management
Medical assistance & referrals
Prevention, education, & youth development
Use of SFWI services
Health risk factors
Caring relationships
High expectations
Opportunities for meaningful participation
Total school assets
Internal youth assets
Improved academic performance
Improved attendance
Reduced disciplinary actions
(Adelman & Taylor, 2006; Scales & Leffert, 2004)

8. Hypothesis
Use of school-based health and wellness services is positively related to school-based youth development assets, after rigorously controlling for potential confounding student risk factors and demographic characteristics.
The study goal was to explore the association between utilization of school-based health and wellness programming and youths’ school-based developmental assets.

9. Methods

10. Data Source
The study is a secondary data analysis of the SFUSD California Healthy Kids Survey (CHKS).
The overall survey response rate was 70%
The CHKS is the largest statewide survey of risk and protection factors among California school children
Extensive efforts made to assess the validity and reliability of the survey instrument
However, prior school performance variables are limited
Since survey is administered anonymously, it is not linkable to actual student academic records

11. SAMPLE CHARACTERISTICS
Comprised of 8,466 students from 15 public high schools in San Francisco Unified School District (SFUSD) with Wellness programs
Representative of SFUSD population
42% reported using Wellness services
Sample was demographically similar in terms of race/ethnicity and grade
The sample is comprised of about 8500 students. CHKS survey are representative of the SFUSD population in key ways, including percent reporting Wellness, background and grade
42% of 7314
There was a 14% missing on this question 11

12. Measures Dependent variables Key Independent variable
Composite variable of youth school-based assets, reflecting caring relationships, meaningful participation, and high expectations (5 items, α= , Austin & Kim, 2007).
Composite variable of youth internal assets, including cooperation and communication, self-efficacy, empathy, problem-solving, self awareness, and goals and aspirations (3 items, α= , Wested, 2007).
Key Independent variable
Dichotomous (0,1) variable, indicating whether a student reported using school-based wellness services.
Control variables
Student and school demographic background
Student risk behaviors
We have control variables from the following domains, student and school demographics as well as student risk behaviors.
“During the past school year, how often have you visited your school’s Wellness Program for information or services?” 12

13. Thus, estimates of effects of programming are prone to bias
Limitations of data
Most research on school-based programming is observational/ correlational
Thus, estimates of effects of programming are prone to bias
Users may be conceptualized as either more or less vulnerable than non-users
With some exceptions (e.g. Kerns et al., 2011; Walker et al., 2010), prior research has not attempted to adjust for bias
Before we get into the details of analysis, important point, is that most studies of school-based programming is observational. Because of that, have to be worried about bias. Only recently been able to grapple with.
We can conceptualize users as either more or less vulnerable

14. Analytic strategy: Propensity Score Methods (PSM)
Aaid in estimates of causal effects from observational data, by correcting for potential selection bias (Guo & Fraser, 2010)
PSM methods initially generate estimates of “true” propensity scores, which denote the probability that a particular subject will receive a particular treatment (in our case, whether or not a student uses Wellness services), based on careful selection of pre-treatment covariates
Like Walker and Kearns, going to use PSM. Aid in estimating causal effects…

15. Analytic Steps PSM have several steps (Guo & Fraser, 2010)
Estimate logistic regression predicting use of wellness services using key pre-treatment covariates to obtain propensity scores
Conduct matching procedures using propensity scores(e.g., nearest neighbor) and assess balance
Analysis (including multiple regression) using propensity scores
Propensity scores have several analytic
Assess balance – not different on pre-treatment covariates. On average, the users and non-users are similar in the pretreatment characteristics.

16. Selection of pre-treatment covariates: Prior Research
Key role of propensity score analysis is careful selection of pre-treatment covariates (Guo & Fraser, 2010; Hong, 2008).
Prior research on school-based health services utilization and academic outcomes using propensity scoring methods has utilized covariates based on student background characteristics including:
Student age/grade, gender, free-lunch status, race/ethnicity, English proficiency, and special education placement (Kerns et al., 2011; Walker, 2010).
Program evaluation and research conducted by ETR Associates points to other key variables potentially selecting students into Wellness programming, including:
Prior student health risk (e.g., early onset of alcohol and other substance use) and school-level compositional and SFWI-related characteristics (i.e., percent of teachers referring to SFWI services and overall utilization rates) (Anyon & Whitaker, 2011).
These methods really hinge on careful selection of pre-treatment covariates. Used literature and our own research to identify.

17. Available Pre-treatment Covariates
Student demographics
Dichotomous variables indicating student sex, grade, race/ethnicity, family structure, and student age
School-level demographics
School racial composition, suspension rate, percent truant, percent participating in federal free and reduced lunch programs, and school size
Correlates of service utilization (Ilgen et al., 2011)
Early substance use including an alcoholic beverage, smoked a cigarette, used other tobacco products, used marijuana, or used another illicit drug
age at which they first used a variety of substances occurred prior to SFWI utilization in combination with a bank of items that asked “how old were you the first time that you did” each of the following: had an alcoholic beverage, smoked a cigarette, used other tobacco products, used marijuana, or used another illicit drug to get high 17

18. Analytic Caveats
Multiple specifications of the propensity model reduced, but did not eliminate pre-treatment differences between Wellness users and non-users
Constrains confidence in results and suggests multiple analytic strategies
age at which they first used a variety of
These methods work best when we have balance between pre-treatmet covariates. Need to assess for the influence of some of these variables. 18

19. RESULTs

20. Results: Estimated Effects of Wellness Utilization1 (ntreated=1393; nuntreated=1828)
Unmatched
Matched1
Regression-- unweighted
Regression-- weighted2
School-based assets
.10 (.02)***
.09 (.02) ****
.11 (.02) ****
.14 (.04)***
Internal assets
.15 (.02)***
.09 (.03) ****
.12 (.02) ****
.14 (.02) ***
1Results presented are generated from a nearest neighbor matching procedure, but are nearly identical across multiple matching procedures
2 Regressions include all control variables
***p<.000
Following Fraser and As you can see from this table, across all these ways of presenting results, students who use Wellness are showing even when we use propensity score methods, we see small but consistent positive effects of wellness utilization on key school-based and internal assets.
First column – difference between users and non-users, without propensity score adjustment
Second column - difference between users and non-users, WITH propensity score match
Third column – matched sample. put in regression results, latter two columns are the regression coefficient on wellness utilization. Put a footnote number 2 – school and student demographics and risk taking behaviors. Don’t use propensity weights.
Fourth column – Weighting the sample higher for students who have stronger tendencies to be in Wellness. Trust the results of the sample that have propensity to use service.

21. Summary of results
Students who used Wellness services reported higher levels of youth development assets in the school environment (caring relationships, meaningful participation and high expectations) after controlling for important confounding variables.
Students who used Wellness services also reported higher levels of internal assets (cooperation and communication, self-efficacy, empathy, problem-solving, self awareness, and goals and aspirations) after controlling for important confounding variables.

22. Summary of the Results Academic Performance
School-based heath programs
Youth development assets
Whitaker, Stone, Anyon & Shields, 2012
Hanson & Austin, 2003
Academic Performance
Given evidence to show to the link school-based heath programs and acacic perf.
Walker et al., 2010

23. Limitations
Propensity scores from models using available covariates did not fully “balance” users and non-users
Missing some key pre-treatment covariates (e.g., ELL status, special education participation, prior school performance)
Reliance on student reports only
Analyses don’t control for type and dose of services
Academic performance outcomes not available
Results should be considerded with limitation s in mind

24. Implications Research Practice
Suggests the potential utility creating data sets that contain sufficient data to analyze the full conceptual model
Points to more work on understanding factors that select students into school based programming
Practice
Supports advocacy-related calls for school-based health and mental health practitioners to develop of youth assets
Suggests that school programs monitor both academic performance and assets simultaneously
Despite these limitations two sets of important implications we provided evidence that we should be analyzing effects of academic performance

25. Questions?
Contact information:
Kelly Whitaker; UC Berkeley & ETR Associates
Susan Stone; UC Berkeley
Yolanda Anyon; UC Berkeley
John Shields; ETR Associates