IHS Special Diabetes Program Competitive Grants Part 2: Using Idea Maps to Plan for Evaluation Diabetes Prevention Group Cynthia C. Phillips, Ph.D. Lisa Wyatt Knowlton, Ed.D. Evaluation Training ̶ March 2005

Learning Objectives  Increased knowledge of how to move from idea maps to evaluation measures  Skill to further discuss and refine idea maps with evaluation in mind  Shared understanding of draft measurement model

Overview  Connecting Idea Maps to Evaluation  Developing Evaluation Questions  Core DP Program Idea Map Example  “Hands-on” Evaluation Planning Exercise  Updating Your DPP Curriculum Idea Map  BREAK  Maps to Measures  Indicators  Measurement Model  Discussion  Implications

Connecting Idea Maps to Evaluation

Evaluation Design IM--Questions Example: Family Vacation

DPP Curriculum SAMPLE - Diabetes Prevention Program Idea Map RESOURCESACTIVITIESOUTPUTSSHORTER- TERM OUTCOMES LONGER-TERM OUTCOMES IMPACT In order to accomplish our set of activities we will need the following: In order to implement our program we will need to do: Evidence of program implementation: Results we hope to see soon (6-12 months) Results that may take longer (1-3 years, still during grant period) Results that may take even longer (beyond grant period) Funding Staff Participants Training DPP Curriculum Materials Space for classes and visits Supplies Recruit and Screen Invite IGT/IFG to participate Teach DPP Curriculum Lifestyle coaching visits People recruited and entered into project Classes taught Participation in classes and activities Goals set and progress monitored Increased knowledge of diabetes and its prevention Regular exercise Improved nutritional choices Weight loss Reduced risk of diabetes Less IGT/IFG progress to DM Reduced prevalence of diabetes

In your project teams: 1. Compare your DPP curriculum idea map (from last time) with the sample and draft protocol from yesterday. 2.Update your original idea map as needed to make sure it includes all of the elements from the sample and draft protocol. 3.Jot down a few of the issues and/or insights that came up for your group during the exercise. EXERCISE: Updating Your Idea Map

BREAK

Idea Map Questions Indicators Methods

Question Examples AudienceTypical QuestionsEvaluation Use Program Management and Staff Are we reaching our target population? Are our participants satisfied with our program? Is the project delivering the program as intended? How can we improve our program? Programming decisions, day- today operations ParticipantsDid the program help me and people like me? What would improve the program next time? Decisions about continuing participation. Community Members Is the program suited to our community needs? What is the program really accomplishing? Decisions about participation and support. Public OfficialsWho is the program serving? What difference has the program made? Is the program reaching its target population? What do participants think about the program? Is the program worth the cost? Decisions about commitment and support. Knowledge about the utility and feasibility of the program approach. FundersIs the project achieving results as intended? Is the program worth the cost? Accountability and improvement of future grantmaking efforts.

RESOURCESACTIVITIESOUTPUTSSHORTER- TERM OUTCOMES LONGER- TERM OUTCOMES IMPACT In order to accomplish our set of activities we will need the following: In order to implement our program we will need to do: Evidence of program implementation: Results we hope to see soon (6-12 months) Results that may take longer (1-3 years, still during grant period) Results that may take even longer (beyond grant period) Funding Staff Participants Training DPP Curriculum Materials Space for classes and visits Supplies Recruit and Screen Invite IGT/IFG to participate Teach DPP Curriculum Lifestyle coaching visits People recruited and entered into project Classes taught Participation in classes and activities Goals set and progress monitored Increased knowledge of diabetes and its prevention Regular exercise Improved nutritional choices Weight loss Reduced risk of diabetes Less IGT/IFG progress to DM Reduced prevalence of diabetes Is the project delivering the program as intended? Is the project achieving results as intended? ? ? ? ? SAMPLE - Diabetes Prevention Program Idea Map DPP Curriculum

Indicators  The specific statements you develop as markers of “success” for activities and outcomes  SMART across each part of the idea map  Meaningful and manageable  May require technical assistance to develop data collection tools

Indicator Examples Focus AreaIndicatorsHow to Evaluate Influential FactorsMeasures of influential factors--may require general population surveys and/or comparison with national data sets. Compare the nature and extent of influences before (baseline) and after the program. ResourcesLogs or reports of financial/staffing status.Compare actual resources acquired against anticipated. ActivitiesDescriptions of planned activities. Logs or reports of actual activities. Descriptions of participants. Compare actual activities provided, types of participants reached against what was proposed. OutputsLogs or reports of actual activities. Actual products delivered. Compare the quality and quantity of actual delivery against expected. Outcomes & ImpactsParticipant attitudes, knowledge, skills, intentions, and/or behaviors thought to result from your activities. Changes in health status factors specified for the program (uniform core) Compare the measures before and after the program

Diabetes Prevention: Measurement Model for Individual Participant LESS INTENSIVE ACTIVITIES: Community-based, awareness activities VARIABLES UNLIKELY TO CHANGE VARIABLES THAT MAY CHANGE (AND INFLUENCE OUTCOMES) OUTCOMES (INTERVENTION DESIGNED TO CHANGE) INTENSIVE ACTIVITIES: Screening - Recruitment - Consent - Baseline Exam - DPP Lifestyle Intervention - Follow up ENTRY AND EXCLUSION CRITERIA Enter: Dx Prediabetes after Screening: * OGT IGT * FBG IFG Exclude: * Current pregnancy * Active alcohol or sub- stance abuse * End stage renal disease * Previous diagnosis of diabetes * ≤ 18 years of age Demographics: * Age * Gender * Height * Education * Family History of Diabetes * Socioeconomic Status Other: * Other comorbidities * Distress * Depression * Trauma and PTSD * Alcohol use * Smoking/ tobacco use * Social support * Perceived health competence * Resilient coping * Spirituality * Knowledge, attitudes & behaviors o high fat foods o weight o exercise * Readiness to change * Participation o attendance o engagement PRIMARY * Prevent Diabetes in people with IGT, IFG SECONDARY * Risk of DM o ↓ weight o ↓ BMI o ↓ waist circumference o  physical activity o improved nutrition INDIRECT * Subjective health

In your project teams: 1.Discuss the draft measurement model and list the implications for your project on the flipchart. 2.From this list of implications:  What actions will your team need to take?  Where will your team most need assistance? 2. Review your list and be prepared to briefly share your team insight with the whole group. DISCUSSION:

Wrap-up