Evaluation of a Multi-Site, Out-of-School-Time Program: Contextual, Individual, and Combined Influences on Outcomes Amy Corron, United Way of Greater Houston Roger Durand, University of Houston-Clear Lake* Emily Gesing, United Way of Greater Houston Julie Johnson, Communities in Schools of Greater Houston Kevin Kebede, Alief YMCA Jennifer Key, Alief Independent School District Linda Lykos, YMCA of Greater Houston Cheryl McCallum, Children’s Museum of Houston *presenter for the group

Abstract In this paper the results of an evaluation of a multi-site, out-of-school-time program are presented. A particular focus will be on evaluation findings concerning the separate and combined impacts of site (or contextual) characteristics and individual participant differences in determining program outcomes. In the aftermath of Hurricanes Katrina and Rita, an out-of-school- time, collaborative program, known as "Houston's Kids," was implemented with the intention of addressing the needs of displaced and other at-risk youth in a single community. An outcomes evaluation was conducted to assess the program's effects on developmental assets as well as on academic achievement among the approximately 375 individual kindergarten through intermediate school participants. True panels of data that tracked changes in the same individual participants over a school year were employed as were data on a sample of control or “comparison” subjects. Hierarchical and multiplicative statistical models were used to analyze the data.

About Houston’s Kids In the aftermath of Hurricanes Katrina and Rita, an out-of-school-time, collaborative program, known as "Houston's Kids," was implemented with the intention of addressing the needs of displaced and other at-risk youth in a single school district in the greater Houston area. Collaborating partners: The United Way of Greater Houston; the greater Houston YMCA; the Alief Independent School District; Communities in Schools of Houston; The Children’s Museum of Houston; and the Alief YMCA. Systematic evaluations have demonstrated the program’s successes in promoting academic achievement; in enhancing positive relationships with adults and peers; and in assisting participants’ families in meeting needs.

In this presentation….. The focus is on evaluation findings concerning the separate and combined effects of program site (or contextual) characteristics and individual participant differences in determining program outcomes – most notably the outcome of school attendance.

Some Background….. In a previous evaluation Houston’s Kids participants were found to have far fewer days absent from school than their control group counterparts at each of the program’s four sites. (In total the participants averaged about 525 fewer school days absent compared to control subjects.) But school days absent ranged considerably among participants across the four sites. What produced the attendance differences from program site to site?

Evaluation Data Both a process and outcomes evaluation were conducted. The outcomes evaluation was conducted among approximately 375 individual kindergarten through intermediate school participants and their parents. True panels of data that tracked changes in the same individual participants over a school year were employed. Data were also gathered on control or comparison subjects from the same sites and school grades.

Statistical Modeling Methods In order to get at the separate and combined effects of individuals’ characteristics and the program sites, two modeling approaches were used: 1) Contextual analysis modeling in which least-squares estimation procedures were employed to investigate separate and combined impacts. 2) Hierarchical procedures in which models were estimated separately for each site and the resulting parameters (constants and coefficients) compared.

Principal Findings: Contextual Modeling It was only possible to account for about eight percent (8%) of the total variance with models that incorporated both individual and site characteristics. (These and other data are available upon request.) The level of individuals’ participation in the program was found to have the greatest predictive power on absences. The program site was found to have almost no influence on school absences. No evidence was found of any interactions between site and individual characteristics (e.g., being female and participating at a middle school site).

Principal Findings: Hierarchical Modeling When the four sites were examined separately, participation in the out-of-school time program was found to be the most important predictor of school absences (the higher the participation, the fewer the absences) at all sites. Model coefficients varied considerably from site to site. (e.g., at one site an Hispanic background was found to result in fewer absences while at a second it had a slight negative one.) Individual characteristics (e.g., gender, ethnic status, poverty status, etc.) were found to be more explanatory predictors at some sites than others. But in no case were they found to account for more than 15% of the total variance.

The Big “Take-Aways”: Back to Logic Models None of the statistical models – contextual or hierarchical – provided much explanation of variations in school absences among out-of-school-time participants. Many logic models are always possible; perhaps the wrong one was chosen to explain variations in school absences. A logic model that incorporated close, peer group relationships might have been a better one. Such a logic model requires a different statistical strategy: the use of dyadic data

Some Useful References Gelman, A.&Hill J. (2006), Data analysis using regression and multilevel/hierarchical models. Cambridge Press, 2006. Luke, D.A. (2004), Multilevel modeling. Thousand Oaks, Ca: Sage. HLM (2013). Retrieved from http://www.ssicentral.com/hlm/examples.html