Presentation on theme: "Don Compton, Ph. D. Senior Evaluator DNPAO"— Presentation transcript:
1Don Compton, Ph. D. Senior Evaluator DNPAO dcompton@cdc Don Compton, Ph.D. Senior Evaluator DNPAO (770)
2Each Has an Advisory Group Made up of State and CDC Staff Evaluating PartnershipsPatricia Rieker and Jan JerniganDeveloping and Using an Evaluation Consultation GroupMichael Baizerman and Don ComptonEvaluation of State Nutrition, Physical Activity, and Obesity PlansPaul Mattessich and Don Compton
4Executive Director, Wilder Research Center, has worked in the fields of human services, social policy,and social research since In 1982, he becameexecutive director of Wilder Research and has served as amember of the Wilder Foundation's senior leadershipteam since then. He has authored or co-authored morethan 250 publications. He has also served on a variety ofgovernment and nonprofit boards of directors and specialtask forces. He is the author of 4 books: Collaboration:What Makes It Work; Community Building: What MakesIt Work; the Manager’s Guide to Program Evaluation; andInformation Gold Mine – all with wide nonprofitaudiences.
6Overview of process for developing an evaluation of the State Plan Scope of State Plan evaluationState Plan evaluation / evaluation of initiativesState program initiatives / partner initiativesInvolvement of stakeholders and partners
7Describes a comprehensive, integrated approach for improving nutrition and physical activity, to prevent/reduce obesity in a stateTypically created collaboratively by “partners” throughout the state: state and local health departments, education agencies, health organizations, nonprofit organizations, advocates, insurers, and others
8Development of a partnership to create, publish, and disseminate the state plan Development of an implementation plan for the state planDevelopment of an evaluation plan for the state plan (and for the implementation plan)Ongoing evaluation and the reporting of evaluation results
91. Increase physical activity. 2. Increase consumption of fruits and vegetables.3. Decrease the consumption of sugar-sweetened beverages.4. Increase breastfeeding initiation and duration.5. Reduce the consumption of high-energy-dense foods.6. Decrease television viewing.
10Information to enable states to develop and implement plans as effectively and efficiently as possibleStates developing first plan: process evaluation to strengthen their work, improve stakeholder involvement, etc.States with plan: “outcome evaluation”States revising plan: evaluation results (from their state and others) as input into development of new plan
11Activities: Production/Dissemination Intermediate Outcomes OutputsState Plan (completed and distributed to all intended audiences)Implementation Plan (completed, distributed to all intended audiences, used)Partnerships formed, continuedActivities: Production/DisseminationStakeholder Engagement, Partnership FormationWriting of Plan and Implementation PlanApprovals SecuredRaise Awareness - Plan Distribution, Presentations, Forums, Education of Intended AudiencesInputsData on Nutrition, Physical Health, and Obesity ConditionsHealth Department InfrastructureHuman and Financial Inputs (for writing)Initial OutcomesIncreases in:Policies and standards…Access to and use of environments…Social and behavioral approaches… …to support healthful eating and physical activityImplementation of initiativesImplementation of evaluationIntermediate Outcomes1. Increased physical activity.2. Increased consumption of fruits and vegetables.3. Decreased consumption of sugar-sweetened beverages.4. Increased breastfeeding initiation and duration.5. Reduced consumption of high-energy-dense foods.6. Decreased television viewing.Ultimate OutcomesReduced obesityReduced chronic diseasesDecreased mortality rates
12Primary focus, for annual review: InputsActivitiesOutputsInitial OutcomesSecondary focus, for longer term review (and use in revising the state plan):Intermediate and Ultimate Outcomes
13Information regarding inputs, activities and outputs provides insight on plan development, and whether plan design/content can be improved.Information regarding initial outcomes indicates whether the plan achieves its initial, short-term purpose to make certain that state and local health departments and their partners organize and improve policies, access, and approaches for the prevention/reduction of obesity among the state’s residents.
14Information on intermediate and ultimate outcomes Is important for future revisions in the state planIs available, e.g., through BRFSSHowever, it is not necessary to evaluate every specific initiative or policy in the state plan in order to do a “state plan evaluation”. (Those initiatives can be the subjects of separate evaluations.)
15Plan identifies initiatives for implementation by state and local health departments, education agencies, health organizations, nonprofit organizations, advocates, insurers, and othersTherefore: Evaluation of Plan determines whether these were implemented Because this implementation falls into the “Initial Outcomes” of the State Plan
16Questions about Process Questions about ContentQuestions about DisseminationQuestions about AwarenessQuestions about Initial OutcomesQuestions about Intermediate OutcomesQuestions about Ultimate Outcomes
17Examples and Potential Methods/Sources QuestionMethod/SourceWere the appropriate stakeholders involved in the development of the state plan and the implementation plan?Stakeholder SurveyWere the stakeholders satisfied with their level of involvement?Do the involved stakeholders feel “ownership” of the state plan?
181. I had sufficient opportunities to give input on the plan. SAADSDNS2. The amount of time requested of me as a Task Force member was reasonable (i.e., not excessive).3. I believe that my ideas and suggestions were taken into account in developing the plan.4. The plan reflects my personal values/priorities related to obesity prevention.5. The plan supports the mission or priorities of the work I do.6. It is clear to me how the work I do fits into the plan.7. I believe that the plan will be useful in guiding or supporting my work.8. Overall, I enjoyed serving on the Task Force.9. Overall, I am satisfied with the Nutrition and Physical Activity Plan.
19Examples and Potential Methods/Sources QuestionMethod/SourceWas a state plan completed?ObservationWas an implementation plan completed?
20Examples and Potential Methods/Sources QuestionMethod/SourceDoes the state plan include sufficient data describing nutrition, physical activity, and obesity within the state’s population?Key informant ratings?“State Plan Index”?Are objectives clearly organized and logical?Does the plan reflect the current state of the art and science in public health? Is it evidence/science based?Does the plan incorporate national goals for Nutrition, Physical Activity, and Obesity?Are goals and objectives stated in SMART (or similar) format?
21Examples and Potential Methods/Sources QuestionMethod/SourceHas a strategy for dissemination of the plan been developed and accepted by stakeholders?Observation, ChecklistHave all intended audiences been reached and involved, in accordance with the implementation plan?Checklist?Survey?
22Examples and Potential Methods/Sources QuestionMethod/SourceDo those who should know about the plan actually know about it?SurveyDo those who should know about the plan (or parts of it) actually understand what it says?Do partners link to plan on their web site?Observation/Checklist
23State Plan Index Nutrition and Physical Activity Program To Prevent Obesity and Other Chronic Diseases The Centers for Disease Control (CDC) has provided a State Plan Index to assist with an overall assessment of the plans developed by states. The index was set up to score each item on a scale of 1 to 5, once the plan is written. However, reviewing the criteria as we are writing the plan should assist with making sure that we have considered the following points:
24Presentation of Data on Disease Burden and Existing Efforts to Prevent and Control Obesity Data are presented on disease burden of obesity and chronic diseases related to poor nutrition and physical inactivityEpidemiologic data are from reliable source(s) (e.g., BRFSS, NHANES)State-level data are provided, including results of state-specific epidemiologic or evaluation studies
25Examples and Potential Methods/Sources QuestionMethod/SourceHas an increase occurred in policies and standards to support healthful eating and physical activity?InventoryHas an increase occurred in access to, and use of, environments which support healthful eating and physical activity?Has an increase occurred in social and behavioral approaches used to support healthful eating and physical activity?SurveyHave agencies / organizations identified by the plan for specific roles or actions carried out their assignments?Review/checklist, based on implementation planSurvey?Do established public health programs conform to plan?Inventory – programs compared with plan
26Evaluation of the “Initial Outcomes” of the State Plan may be the same as evaluation of the Implementation Plan
27Indicator 16: Number of Women, Infants, & Children Program (WIC) clinics funded by DSHS WIC for obesity prevention activitiesIndicator 17: Percent of partners who implement specific items from the state plan that are related to their workIndicator 19: Number of policy and/or environmental changesIndicator 20: Number of legislative bills passed related to obesity preventionNumber of worksites that have adopted worksite wellness programs that meet a qualified standardIndicator 23 (Target 19): Percent of school districts that have implemented an approved school wellness policyIndicator 24: Percent of schools with a school health committee or advisory group
28Examples and Potential Methods/Sources QuestionMethod/SourceDid physical activity increase?BRFSSDid consumption of fruits and vegetables increase?BFRSSDid obesity rates decline?
29Stakeholder Involvement Strategy How Evaluation Can Assist
31Enables you to communicate with stakeholders during the development of the State Plan Use feedback to make changes in the plan itself, or in the development processRelates to Inputs, Activities, and Outputs boxes in logic model for State Plan Evaluation
33Select questionsSelect methods – and establish schedule for data collectionAssign personnel, allocate resourcesFinalize all procedures – review with stakeholders, test methods, etc.
34Examples and Potential Methods/Sources QuestionMethod/SourceWere the appropriate stakeholders involved in the development of the state plan and the implementation plan?Stakeholder SurveyWere the stakeholders satisfied with their level of involvement?Do the involved stakeholders feel “ownership” of the state plan?
351. I had sufficient opportunities to give input on the plan. SAADSDNS2. The amount of time requested of me as a Task Force member was reasonable (i.e., not excessive).3. I believe that my ideas and suggestions were taken into account in developing the plan.4. The plan reflects my personal values/priorities related to obesity prevention.5. The plan supports the mission or priorities of the work I do.6. It is clear to me how the work I do fits into the plan.7. I believe that the plan will be useful in guiding or supporting my work.8. Overall, I enjoyed serving on the Task Force.9. Overall, I am satisfied with the Nutrition and Physical Activity Plan.
36Participation Records Surveys of StakeholdersExpert Informant SurveysAnnual Inventories (policies, initiatives, etc.)State Plan Index
37Obtain information, using methods developed in design stage Compile and store information in usable format
38Process informationPrepare for reportingPreliminary review of findings
39Present findings to key stakeholders Issue report(s)
40Information Collected Information UsedQuestions about Process, ContentEarly in process of Plan development, and throughout development phaseDuring Plan development phase – to improve the process and contentQuestions about DisseminationAfter drafts or final versions are distributedImmediately – to identify gapsQuestions about AwarenessA few months after distribution; possibly at later intervalsImmediately – to identify additional action stepsQuestions about Initial OutcomesOne year after completion of PlanFor evaluation of “Implementation”Questions about Intermediate and Ultimate OutcomesBaseline, and a few years after plan developmentPrimarily at time of revision of State Plan