1. IHS Special Diabetes Program Competitive Grants Part 2: Using Idea Maps to Plan for Evaluation CVD Risk Reduction Group Cynthia C. Phillips, Ph.D. Lisa Wyatt Knowlton, Ed.D. Evaluation Training ̶ March 2005

2. Learning Objectives  Increased knowledge of how to move from idea maps to evaluation measures  Skill to further discuss and refine idea maps with evaluation in mind  Shared understanding of draft measurement model

3. Overview  Connecting Idea Maps to Evaluation  Developing Evaluation Questions  Core CVD Program Idea Map Example  “Hands-on” Evaluation Planning Exercise  Updating Your Case Management Idea Map  BREAK  Maps to Measures  Indicators  Measurement Model  Discussion  Implications

4. Connecting Idea Maps to Evaluation

5. Evaluation Design IM--Questions Example: Family Vacation

6. Case Management SAMPLE – CVD Risk Reduction Program Idea Map RESOURCESACTIVITIESOUTPUTSSHORTER- TERM OUTCOMES LONGER-TERM OUTCOMES IMPACT In order to accomplish our set of activities we will need the following: In order to implement our program we will need to do: Evidence of program implementation: Results we hope to see soon (6-12 months) Results that may take longer (1-3 years, still during grant period) Results that may take even longer (beyond grant period) Funding Staff Participants Guidelines Training Space for classes and visits Supplies Medications Materials Recruit people with DM to enter Case management team Individual care plans Treatment to target goals Self-management education People recruited and entered into project Team meets regularly Document and monitor care plan Clinic visits Medications started or changed Appointments and referrals made Participation in activities Education, classes Increased knowledge of risk of CVD in diabetes and how to reduce risk Improvements in CVD risk factors, such as BP, A1C, lipids, smoking cessation, etc. Reduced risk of CVD Reduced prevalence of CVD in people with diabetes

7. EXERCISE: Updating Your Idea Map In your project teams: 1. Compare your case management idea map (from last time) with the sample and draft protocol from yesterday. 2.Update your original idea map as needed to make sure it includes all of the elements from the sample and draft protocol. 3.Jot down a few of the issues and/or insights that came up for your group during the exercise.

8. BREAK

9. Idea Map Questions Indicators Methods

10. Question Examples AudienceTypical QuestionsEvaluation Use Program Management and Staff Are we reaching our target population? Are our participants satisfied with our program? Is the project delivering the program as intended? How can we improve our program? Programming decisions, day- today operations ParticipantsDid the program help me and people like me? What would improve the program next time? Decisions about continuing participation. Community Members Is the program suited to our community needs? What is the program really accomplishing? Decisions about participation and support. Public OfficialsWho is the program serving? What difference has the program made? Is the program reaching its target population? What do participants think about the program? Is the program worth the cost? Decisions about commitment and support. Knowledge about the utility and feasibility of the program approach. FundersIs the project achieving results as intended? Is the program worth the cost? Accountability and improvement of future grantmaking efforts.

11. RESOURCESACTIVITIESOUTPUTSSHORTER- TERM OUTCOMES LONGER- TERM OUTCOMES IMPACT In order to accomplish our set of activities we will need the following: In order to implement our program we will need to do: Evidence of program implementation: Results we hope to see soon (6-12 months) Results that may take longer (1-3 years, still during grant period) Results that may take even longer (beyond grant period) Funding Staff Participants Guidelines Training Space for classes and visits Supplies Medications Materials Recruit people with DM to enter Case management team Individual care plans Treatment to target goals Self- management education People recruited and entered into project Team meets regularly Document and monitor care plan Clinic visits Medications started or changed Appointments and referrals made Participation in activities Education, classes Increased knowledge of risk of CVD in diabetes and how to reduce risk Improvements in CVD risk factors, such as BP, A1C, lipids, smoking cessation, etc. Reduced risk of CVD Reduced prevalence of CVD in people with diabetes Case Management Is the project delivering the program as intended? ? ? ? ? Is the project achieving results as intended?

12. Indicators  The specific statements you develop as markers of “success” for activities and outcomes  SMART across each part of the idea map  Meaningful and manageable  May require technical assistance to develop data collection tools

13. Focus AreaIndicatorsHow to Evaluate Influential FactorsMeasures of influential factors--may require general population surveys and/or comparison with national data sets. Compare the nature and extent of influences before (baseline) and after the program. ResourcesLogs or reports of financial/staffing status.Compare actual resources acquired against anticipated. ActivitiesDescriptions of planned activities. Logs or reports of actual activities. Descriptions of participants. Compare actual activities provided, types of participants reached against what was proposed. OutputsLogs or reports of actual activities. Actual products delivered. Compare the quality and quantity of actual delivery against expected. Outcomes & Impacts Participant attitudes, knowledge, skills, intentions, and/or behaviors thought to result from your activities. Changes in health status factors specified for the program (uniform core) Compare the measures before and after the program Indicator Examples

14. CVD Risk Reduction: Measurement Model for Individual Participant LESS INTENSIVE ACTIVITIES: Community-based, awareness activities VARIABLES UNLIKELY TO CHANGE VARIABLES THAT MAY CHANGE (AND INFLUENCE OUTCOMES) OUTCOMES (INTERVENTION DESIGNED TO CHANGE) INTENSIVE ACTIVITIES: Screening - Recruitment - Consent - Baseline Exam - Case Management - Treatment - Follow up ENTRY AND EXCLUSION CRITERIA Enter: Dx Diabetes Exclude: * Current pregnancy * Active alcohol or sub- stance abuse * End stage renal disease * ≤ 18 years of age Demographics: * Age * Gender * Height * Education * Family History of Diabetes or CVD * Socioeconomic Status Other: * Other comorbidities * Distress * Depression * Trauma and PTSD * Alcohol use * Smoking/ tobacco use * Social support * Perceived health competence * Resilient coping * Spirituality * Knowledge, attitudes & behaviors o high fat foods o weight o exercise * Readiness to change * Participation o attendance o engagement PRIMARY * Reduce CVD in DM SECONDARY * Risk of CVD ↓ BP ↓ LDL;  HDL; ↓TG ↓ A1C ↓ BMI; wt; waist circum Smoking cessation Aspirin/day Stress reduction Improved nutrition Regular phys activity  awareness of CVD INDIRECT * Subjective health

15. DISCUSSION: In your project teams: 1.Discuss the draft measurement model and list the implications for your project on the flipchart. 2.From this list of implications:  What actions will your team need to take?  Where will your team most need assistance? 2. Review your list and be prepared to briefly share your team insight with the whole group.

16. Wrap-up