LiveWell Colorado Evaluation LiveWell Colorado Taskforce Meeting July 16, 2009

LWCO Evaluation Team: Mission To guide and support LiveWell Colorado communities in evaluation and to report the overall outcomes and impact of LiveWell initiatives.

LWCO Evaluation Team: Who We Are Sr. Research Asst. Deanna McQuillan, MA Sr. Research Specialist Erica Morse, MA Sr. Research Asst. Lisa Harner, MA Tristan Sanders, BS Sr. Evaluation Manager Bonnie Leeman-Castillo, PhD Data Analysis Data Collection Community Reports & TA Data Management & Reports Statistical Analyst Sophia Newcomer, MPH KPCO Community & Local Govt Relations Corina Lindley, MPH Monica Buhlig, MPH Principal Investigator Diane King, PhD

What We Do Collect Enter Analyze Report …………………………………………………Data

Primary Sources of Data Demographic & Process Data Community Action Plan (year end) Continuation Progress Report (year end) Annual Site Visits (summer) Key informant interviews with key community coalition members (fall) TA Reports (monthly) Outcomes Data Continuation Progress Report (year end) IVR Surveys (varies by community)

Other Sources of Data Media and Website Hits Community Highlights Local Evaluators Reports School surveys CDPHE surveys e.g., BRFSS and Child Health Survey

Todays Plan LWC IVR Survey: Analytic Update Progress Milestones Current indicators of future success Community Reports Small group discussion

LiveWell Colorado IVR Survey: Analytic Update Sophia Raff Newcomer, MPH Biostatistician Institute for Health Research, Kaiser Permanente Colorado

Agenda Whats new with the IVR survey? IVR survey updates Timing of community surveys Analytic update Discussion of raw vs. weighted data Examples

Whats new with the IVR survey? Timing of survey Survey will be conducted: M&P Year Year 1 (first year of implementation) Year 5 Year 10 Questions Consistent core questions Limiting number of questions on survey

Unweighted data In the LiveWell Colorado IVR survey, a sample (n) of community members participate in a phone survey. The intent of the survey is to use the responses from that sample of community members (n) to better understand key health and behavior indicators of the entire community (N). Community (N) Sample (n)

Unweighted data (cont.) The data collected is raw data Raw data: the actual responses from survey participants Sometimes refered to as unweighted If we use the raw data from the sample (n) to draw conclusions about the community (N), then we assume that: Non coverage and non response bias is the same throughout all demographic groups in the community The sample (n) is representative of the community These assumptions are violated in the LWCO IVR

Example: Park Hill 2006 Question: # of fruits and vegetables/day N=19,315 (2000 Census) n= 321

Weighted survey analysis (Park Hill data) Weighted survey analysis adjusts the raw, collected data from the sample (n) to try to look like the population (N) Weighting on age and sex, the individual case weights (ICW) are: Small cell sizes; collapse with next age group

Definitions…and then some data! Survey sample size: number of respondents that started the IVR survey (provided a valid age and pressed a key for the first question) Item sample size: number of respondents who provided a valid response for that question 95% confidence interval: range of values in which the true weighted percentage is expected to occur, with 95% probability If we drew 100 samples from that same community, we would expect the weighted percentage from 95 of those samples to fall within the reported 95% confidence interval

Have you heard of Park Hill Thriving Communities?

Weighted percentage for response=Yes %20%30%40% 17.0% 12.4%21.6% % 28.0%38.7% % 33.8%44.8%

Meeting guidelines of 30 minutes of moderate physical activity at least 5 days per week?

Meeting guidelines of 5 or more servings of fruits and vegetables per day?

Have you heard of LiveWell Commerce City?

Meeting guidelines of 30 minutes of moderate physical activity at least 5 days per week?

Weighted percentage of respondents meeting moderate PA guidelines COMMERCE CITY %30%40% 26.7% 18.9%34.6% % 20.4%34.2% % 27.9%43.1%

Meeting guidelines of 5 or more servings of fruits and vegetables per day?

Data weighting summary Advantages: Allows the collected IVR data to look more like the population from which it is sampled Limitations 2000 Census data Weighting on other sociodemographic factors, such as race, is impractical Small sample sizes and extreme ICWs

Any questions?

Progress Milestones

How do we answer that question????? Collaborative Partnerships & Community Capacity 4 Intermediate Outcomes that Predict Impact Volume of changes : number of new or modified community & systems changes. Intensity of changes: modifying access & removing barriers more intense than providing information about services. Permanence (durability) of changes: ongoing versus once Penetration of changes: was contact made (reach) through multiple sectors & settings (adopters) or geographic areas.

Collaborative Partnerships & Capacity Building Are Partners Broadly Representative? e.g., 4 core sectors, parents, policy makers, police, target populations, non-profit organizations Are Partners (including residents) Engaged? e.g., shared leadership & work teams Is Community Capacity Increasing? e.g., amount of resources leveraged

Volume & Intensity How many organizational systems changes adopted? How many environmental changes made? How many sectors are you working in?

Permanence (a.k.a maintenance) Are strategies durable? e.g., policy change > curriculum change > events Has HEAL mission been adopted by partner organizations? Has HEAL language been incorporated into planning & policy documents?

Broad Penetration (a.k.a. reach) What proportion of target population(s) have been reached? Are residents being reached at multiple levels of the ecological model? e.g., cooking class (individual / program); corner store/farmers markets (family / environment); school rules involving snacks (organization / policy)

Broad Penetration (a.k.a. adoption) Are changes holistic (i.e., inter-linked)? e.g., park improvements + safe routes to park > park improvements alone Are single goals addressed in multiple sectors? e.g., breastfeeding addressed through: healthcare (training providers to counsel, changing formula bags for lactation kits), social marketing (norms & attitudes), workplaces (space, time, supervisor awareness)

Community Reports

Community Reports: Activity Questions for Discussion: Who would use this information: partners, stakeholder, and/or residents? How might it be used? Is it a good way to display this information? Does it need more/less text? What would you add or subtract on this topic/subsection of the report?