Presentation is loading. Please wait.

Presentation is loading. Please wait.

Don Compton, Ph.D. Senior Evaluator DNPAO (770) 488-5258.

Similar presentations


Presentation on theme: "Don Compton, Ph.D. Senior Evaluator DNPAO (770) 488-5258."— Presentation transcript:

1 Don Compton, Ph.D. Senior Evaluator DNPAO (770)

2  Each Has an Advisory Group Made up of State and CDC Staff  Evaluating Partnerships  Patricia Rieker and Jan Jernigan  Developing and Using an Evaluation Consultation Group  Michael Baizerman and Don Compton  Evaluation of State Nutrition, Physical Activity, and Obesity Plans  Paul Mattessich and Don Compton

3  Jianglan White, GA  Emma Kirkpatrick, TX  Amanda Raftery, IN  Kristian Gordon, SC  Abdoulaye Diedhiou, SC  Laura Hutton, MN  Jan Liebhart, WI  Shelly Sutherland, MT  Don Compton, CDC  Raegan Tuff, CDC  Jennifer Kohr, CDC

4 Executive Director, Wilder Research Center, has worked in the fields of human services, social policy, and social research since In 1982, he became executive director of Wilder Research and has served as a member of the Wilder Foundation's senior leadership team since then. He has authored or co-authored more than 250 publications. He has also served on a variety of government and nonprofit boards of directors and special task forces. He is the author of 4 books: Collaboration: What Makes It Work; Community Building: What Makes It Work; the Manager’s Guide to Program Evaluation; and Information Gold Mine – all with wide nonprofit audiences.

5 Paul W. Mattessich, Ph.D. March 18, 2010

6  Overview of process for developing an evaluation of the State Plan  Scope of State Plan evaluation  State Plan evaluation / evaluation of initiatives  State program initiatives / partner initiatives  Involvement of stakeholders and partners

7  Describes a comprehensive, integrated approach for improving nutrition and physical activity, to prevent/reduce obesity in a state  Typically created collaboratively by “partners” throughout the state: state and local health departments, education agencies, health organizations, nonprofit organizations, advocates, insurers, and others

8  Development of a partnership to create, publish, and disseminate the state plan  Development of an implementation plan for the state plan  Development of an evaluation plan for the state plan (and for the implementation plan)  Ongoing evaluation and the reporting of evaluation results

9  1. Increase physical activity.  2. Increase consumption of fruits and vegetables.  3. Decrease the consumption of sugar- sweetened beverages.  4. Increase breastfeeding initiation and duration.  5. Reduce the consumption of high-energy- dense foods.  6. Decrease television viewing.

10  Information to enable states to develop and implement plans as effectively and efficiently as possible  States developing first plan : process evaluation to strengthen their work, improve stakeholder involvement, etc.  States with plan : “outcome evaluation”  States revising plan : evaluation results (from their state and others) as input into development of new plan

11 Inputs Data on Nutrition, Physical Health, and Obesity Conditions Health Department Infrastructure Human and Financial Inputs (for writing) Activities: Production/Dissemination Stakeholder Engagement, Partnership Formation Writing of Plan and Implementation Plan Approvals Secured Raise Awareness - Plan Distribution, Presentations, Forums, Education of Intended Audiences Outputs State Plan (completed and distributed to all intended audiences) Implementation Plan (completed, distributed to all intended audiences, used) Partnerships formed, continued Initial Outcomes Increases in: Policies and standards… Access to and use of environments… Social and behavioral approaches… …to support healthful eating and physical activity Implementation of initiatives Implementation of evaluation Intermediate Outcomes 1. Increased physical activity. 2. Increased consumption of fruits and vegetables. 3. Decreased consumption of sugar-sweetened beverages. 4. Increased breastfeeding initiation and duration. 5. Reduced consumption of high-energy-dense foods. 6. Decreased television viewing. Ultimate Outcomes Reduced obesity Reduced chronic diseases Decreased mortality rates

12 Primary focus, for annual review:  Inputs  Activities  Outputs  Initial Outcomes Secondary focus, for longer term review (and use in revising the state plan):  Intermediate and Ultimate Outcomes

13  Information regarding inputs, activities and outputs provides insight on plan development, and whether plan design/content can be improved.  Information regarding initial outcomes indicates whether the plan achieves its initial, short-term purpose to make certain that state and local health departments and their partners organize and improve policies, access, and approaches for the prevention/reduction of obesity among the state’s residents.

14  Information on intermediate and ultimate outcomes  Is important for future revisions in the state plan  Is available, e.g., through BRFSS  However, it is not necessary to evaluate every specific initiative or policy in the state plan in order to do a “state plan evaluation”. (Those initiatives can be the subjects of separate evaluations.)

15  Plan identifies initiatives for implementation by state and local health departments, education agencies, health organizations, nonprofit organizations, advocates, insurers, and others  Therefore: Evaluation of Plan determines whether these were implemented Because this implementation falls into the “Initial Outcomes” of the State Plan

16 1. Questions about Process 2. Questions about Content 3. Questions about Dissemination 4. Questions about Awareness 5. Questions about Initial Outcomes 6. Questions about Intermediate Outcomes 7. Questions about Ultimate Outcomes

17 Examples and Potential Methods/Sources QuestionMethod/Source Were the appropriate stakeholders involved in the development of the state plan and the implementation plan? Stakeholder Survey Were the stakeholders satisfied with their level of involvement? Stakeholder Survey Do the involved stakeholders feel “ownership” of the state plan? Stakeholder Survey

18 1. I had sufficient opportunities to give input on the plan. SAADSDNS 2. The amount of time requested of me as a Task Force member was reasonable (i.e., not excessive). SAADSDNS 3. I believe that my ideas and suggestions were taken into account in developing the plan. SAADSDNS 4. The plan reflects my personal values/priorities related to obesity prevention. SAADSDNS 5. The plan supports the mission or priorities of the work I do. SAADSDNS 6. It is clear to me how the work I do fits into the plan. SAADSDNS 7. I believe that the plan will be useful in guiding or supporting my work. SAADSDNS 8. Overall, I enjoyed serving on the Task Force. SAADSDNS 9. Overall, I am satisfied with the Nutrition and Physical Activity Plan. SAADSDNS

19 Examples and Potential Methods/Sources QuestionMethod/Source Was a state plan completed? Observation Was an implementation plan completed? Observation

20 Examples and Potential Methods/Sources QuestionMethod/Source Does the state plan include sufficient data describing nutrition, physical activity, and obesity within the state’s population? Key informant ratings? “State Plan Index”? Are objectives clearly organized and logical? Key informant ratings? “State Plan Index”? Does the plan reflect the current state of the art and science in public health? Is it evidence/science based? Key informant ratings? “State Plan Index”? Does the plan incorporate national goals for Nutrition, Physical Activity, and Obesity? Key informant ratings? “State Plan Index”? Are goals and objectives stated in SMART (or similar) format? Key informant ratings? “State Plan Index”?

21 Examples and Potential Methods/Sources QuestionMethod/Source Has a strategy for dissemination of the plan been developed and accepted by stakeholders? Observation, Checklist Have all intended audiences been reached and involved, in accordance with the implementation plan? Checklist? Survey?

22 Examples and Potential Methods/Sources QuestionMethod/Source Do those who should know about the plan actually know about it? Survey Do those who should know about the plan (or parts of it) actually understand what it says? Survey Do partners link to plan on their web site? Observation/Checklist

23 State Plan Index Nutrition and Physical Activity Program To Prevent Obesity and Other Chronic Diseases The Centers for Disease Control (CDC) has provided a State Plan Index to assist with an overall assessment of the plans developed by states. The index was set up to score each item on a scale of 1 to 5, once the plan is written. However, reviewing the criteria as we are writing the plan should assist with making sure that we have considered the following points:

24  Presentation of Data on Disease Burden and Existing Efforts to Prevent and Control Obesity  Data are presented on disease burden of obesity and chronic diseases related to poor nutrition and physical inactivity  Epidemiologic data are from reliable source(s) (e.g., BRFSS, NHANES)  State-level data are provided, including results of state-specific epidemiologic or evaluation studies

25 Examples and Potential Methods/Sources QuestionMethod/Source Has an increase occurred in policies and standards to support healthful eating and physical activity? Inventory Has an increase occurred in access to, and use of, environments which support healthful eating and physical activity? Has an increase occurred in social and behavioral approaches used to support healthful eating and physical activity? Survey Have agencies / organizations identified by the plan for specific roles or actions carried out their assignments? Review/checklist, based on implementation plan Survey? Do established public health programs conform to plan?Inventory – programs compared with plan

26  Evaluation of the “Initial Outcomes” of the State Plan may be the same as evaluation of the Implementation Plan

27  Indicator 16: Number of Women, Infants, & Children Program (WIC) clinics funded by DSHS WIC for obesity prevention activities  Indicator 17: Percent of partners who implement specific items from the state plan that are related to their work  Indicator 19: Number of policy and/or environmental changes  Indicator 20: Number of legislative bills passed related to obesity prevention  Number of worksites that have adopted worksite wellness programs that meet a qualified standard  Indicator 23 (Target 19): Percent of school districts that have implemented an approved school wellness policy  Indicator 24: Percent of schools with a school health committee or advisory group

28 Examples and Potential Methods/Sources QuestionMethod/Source Did physical activity increase? BRFSS Did consumption of fruits and vegetables increase? BFRSS Did obesity rates decline? BRFSS

29  Stakeholder Involvement Strategy  How Evaluation Can Assist

30 Stakeholder Their primary interest How involve? Contact frequency Contact details

31  Enables you to communicate with stakeholders during the development of the State Plan  Use feedback to make changes in the plan itself, or in the development process  Relates to Inputs, Activities, and Outputs boxes in logic model for State Plan Evaluation

32  Design  Data Collection  Analysis  Reporting

33  Select questions  Select methods – and establish schedule for data collection  Assign personnel, allocate resources  Finalize all procedures – review with stakeholders, test methods, etc.

34 Examples and Potential Methods/Sources QuestionMethod/Source Were the appropriate stakeholders involved in the development of the state plan and the implementation plan? Stakeholder Survey Were the stakeholders satisfied with their level of involvement? Stakeholder Survey Do the involved stakeholders feel “ownership” of the state plan? Stakeholder Survey

35 1. I had sufficient opportunities to give input on the plan. SAADSDNS 2. The amount of time requested of me as a Task Force member was reasonable (i.e., not excessive). SAADSDNS 3. I believe that my ideas and suggestions were taken into account in developing the plan. SAADSDNS 4. The plan reflects my personal values/priorities related to obesity prevention. SAADSDNS 5. The plan supports the mission or priorities of the work I do. SAADSDNS 6. It is clear to me how the work I do fits into the plan. SAADSDNS 7. I believe that the plan will be useful in guiding or supporting my work. SAADSDNS 8. Overall, I enjoyed serving on the Task Force. SAADSDNS 9. Overall, I am satisfied with the Nutrition and Physical Activity Plan. SAADSDNS

36  Participation Records  Surveys of Stakeholders  Expert Informant Surveys  Annual Inventories (policies, initiatives, etc.)  State Plan Index

37  Obtain information, using methods developed in design stage  Compile and store information in usable format

38  Process information  Prepare for reporting  Preliminary review of findings

39  Present findings to key stakeholders  Issue report(s)

40 Information CollectedInformation Used Questions about Process, Content Early in process of Plan development, and throughout development phase During Plan development phase – to improve the process and content Questions about Dissemination After drafts or final versions are distributed Immediately – to identify gaps Questions about Awareness A few months after distribution; possibly at later intervals Immediately – to identify additional action steps Questions about Initial Outcomes One year after completion of Plan For evaluation of “Implementation” Questions about Intermediate and Ultimate Outcomes Baseline, and a few years after plan development Primarily at time of revision of State Plan

41


Download ppt "Don Compton, Ph.D. Senior Evaluator DNPAO (770) 488-5258."

Similar presentations


Ads by Google