1. IF TRAINING IS THE ANSWER, WHAT’S THE QUESTION? TOWARD AN EVIDENCE-BASED SYSTEM FOR INNOVATION SUPPORT Abraham Wandersman U. Of South Carolina wandersman@sc.edu

2. QUESTIONS IF TRAINING IS THE ANSWER, WHAT’S THE QUESTION? IF TRAINING EVALUATION IS THE ANSWER, WHAT’S THE QUESTION?

3. OVERVIEW AMBITIOUS GOALS THE INTERACTIVE SYSTEMS FRAMEWORK FOR DISSEMINATION AND IMPLEMENTATION DEMYSTIFYING ACCOUNTABILITY AND GETTING TO OUTCOMES ® (TAKING A VACATION) TOWARD AN EVIDENCE BASED SYSTEM FOR INNOVATION SUPPORT SUMMARY (JEOPARDY)

4. AMBITOUS GOALS

5. Dr. von Eschenbach: I believe we are at what I call a strategic inflection in biology, which means we're at a point of unprecedented growth in three key areas related to cancer research: knowledge, technology, and resources. The integration of growth in these three sectors provides an opportunity for exponential progress. To achieve this progress, we must set a clear direction and focus our efforts into a cohesive strategy.

6. The goal of eliminating suffering and death due to cancer provides this focus. It does not mean "curing" cancer but, rather, it means that we will eliminate many cancers and control the others, so that people can live with -- not die from -- cancer. We can do this by 2015, but we must reach for it. We owe it to cancer patients around the world -- and their families -- to meet this challenge. May 16, 2003 BenchMarks

7. ADDITIONAL AMBITIOUS GOALS SAFETY PERMANENCY CHILD WELL-BEING

8. DATA - EVIDENCE

11. THE GAP BETWEEN RESEARCH AND PRACTICE IN THE DOCTOR’S OFFICE

12. OVERALL 54.9% RECEIVED RECOMMENDED CARE ASCH ET AL STUDY, NEJM, 2006

13. POSSIBLE SOLUTION VA MEDICAL SYSTEM HAS 67% RECOMMENDED CARE SYSTEM HAS ELECTRONIC MEDICAL RECORDS, DECISION SUPPORT TOOLS, AUTOMATED ORDER ENTRY, ROUTINE MEASUREMENT AND REPORTING ON QUALITY, INCENTIVES FOR PERFORMANCE

14. As Yogi Berra supposedly said, "In theory there is no difference between theory and practice, but in practice there is."

15. Gates Foundation Preventive Intervention Vaccine/Drug Mechanism Syringes Physician Health System Support System Medical Schools Government Funding

16. Stage 1 Traditional Evaluation Model Agency Funder Stage 2 Evaluator Funder

17. Partnership Model RESULTS Funder Evaluator Agency

18. HOW DO WE GET FROM HERE TO THERE ?

19. THE INTERACTIVE SYSTEMS FRAMEWORK FOR DISSEMINATION AND IMPLEMENTATION

20. io Distilling the Information—Prevention Synthesis & Translation System Supporting the Work—Prevention Support System Putting It Into Practice—Prevention Delivery System Synthesis General Capacity Building Innovation-Specific Capacity Building General Capacity Use Innovation-Specific Capacity Use Macro Policy Climate Fundin g Existing Research and Theory Translation

21. Support System Model

22. TOOLS

23. TRAINING

24. TECHNICAL ASSISTANCE

25. QUALITY IMPROVEMENT/ QUALITY ASSURANCE

26. DEMYSTIFYING ACCOUNTABILITY GTO®

27. Accountability QuestionsRelevant Literatures 1.What are the underlying needs and conditions that must be addressed? (NEEDS/RESOURCES) 1.Needs/Resource Assessment 2.What are the goals, target population, and objectives? (i.e., desired outcomes)? (GOALS) 2.Goal Setting 3.What science (evidence) based models and best practice programs can be used in reaching the goals (BEST PRACTICE)? 3.Consult Literature on Science Based and Best Practice Programs 4.What actions need to be taken so the selected program “fits” the community context? (FIT) 4.Feedback on Comprehensiveness and Fit of Program 5.What organizational capacities are needed to implement the program? (CAPACITIES) 5.Assessment of Organizational Capacities 6.What is the plan for this program (PLAN)6.Planning 7.Is the program being implemented with quality (PROCESS) 7.Process evaluation 8.How well is the program working? (OUTCOME EVALUATION) 8.Outcome and Impact Evaluation 9.How will continuous quality improvement strategies be included? (IMPROVE) 9.Total Quality Management; Continuous Quality Improvement 10.If the program is successful, how will it be sustained? (SUSTAIN) 10.Sustainability and Institutionalization

28. GTO as a Painter’s Palette #1 Needs/ Resources #2 Goals #3 Best Practices #9 Improve/ CQI #8 Outcome Evaluation #7 Implementation Process Evaluation #4 Fit #5 Capacities #6 Plan #10 Sustain RESULTSRESULTS

29. VACATION EXAMPLE

30. What Is Getting To Outcomes-2004 ? By Matthew Chinman, Pamela Imm & Abraham Wandersman A system based on ten empowerment evaluation and accountability questions that contain elements of successful programming Published by the RAND Corporation (quality review) Available at no cost at: http://www.rand.org/publications/TR/TR101/ http://www.rand.org/publications/TR/TR101/ “Best Practice Process” –WINNER OF 2008 OUTSTANDING PUBLICATION AWARD DOWNLOADED OVER 75,000 TIMES

31. Levels & Accountability CountryState County AgencyProvider Accountability Question 1. NEEDS/ RESOURCES 2.GOALS 3.EVIDENCE-BASED PRACTICES 4.FIT 5.CAPACITY 6.PLAN 7.IMPLEMENTATION 8.OUTCOME EVALUATION 9.CQI 10.SUSTAINABILITY

32. TOWARD AN EVIDENCE BASED SYSTEM FOR INNOVATION SUPPORT (EBSIS) WANDERSMAN, CHIEN, KATZ (ACCEPTED WITH REVISIONS)

33. EVIDENCE BASED SYSTEM for INNOVATION SUPPORT

34. AN ILLUSTRATION OF EBSIS THE MOMS PROJECT FOR SUBSTANCE ABUSING PREGNANT WOMEN

35. Core Components of MOMS program Client-Centered Care Getting to Outcomes ® Motivational Interviewing Rewards system

36. EBSIS Components Tools Training Technical Assistance Quality Assurance/Quality Assistance

37. Tool: MOMS Manual Developing Fall 2009-Spring 2011 Three versions: Clinician, supervisor, and client Described: – GTO process for individual treatment planning and deliver – Use of assessment tools for planning, monitoring, and evaluating client progress – Use of motivational interviewing to move clients through the GTO Steps However, we know that many tools end up sitting on shelves!

38. Training for Outcome (TFO) Developing in conjunction with LRADAC leadership and clinicians Utilized thorough assessment of training needs to assure training transfer Separated into modules tailored to specific job descriptions Delivered 10.21.10-10.22.10 to over 70 staff!

39. Technical Assistance Began immediately following training Implementation planning January 2011 – Using the Quality Implementation Tool (QIT) Regular meeting with clinical/supervisory staff around imp. Issues and program refinement Clinician booster session re: use of MI and other MOMS components 2.11

40. QI/QA Monitoring process and Evaluating Outcomes Assessment of Information System Capacity Streamline collection and utilization of information Designed to: – Inform Continuous Quality Improvement (CQI) – Evaluate multi-dimensional outcomes

41. ENHANCING THE SCIENCE AND PRACTICE OF INNOVATION SUPPORT ZOOM OUT--THE 40 CELLS OF EBSIS

42. ToolsTrainingTAQI/QA Needs & Resources Goals Best Practices Fit Capacity Plan Implement/ Process Evaluation Outcomes CQI Sustainability

43. ZOOM IN--GTO AND WEBINARS

44. GTO Step 1.Need and Resources  Identification of group to be targeted by the webinar  What does this group need to know? o Conduct a task analysis to determine the focus of webinar  Conduct a value analysis to determine whether presenting issues are worth addressing o Use cost-benefit analysis to determine if perceived benefits of training outweigh anticipated training costs in terms of financial and resource investment. 1.Goals and Objectives  Determine training goals based on needs/resource assessment data o What knowledge or skills will participants gain? (Webinars are more useful for imparting conceptual or basic procedural knowledge – as opposed to hands-on/skill building-oriented knowledge)  Ensure training goals are aligned with organization’s goals and culture 1.Best PracticesWebinar Design and Delivery  Event management capabilities (e.g., GoToMeeting, Elluminate, AdobeConnectNow) can increase efficiency by automating: o invitations to the webinar, registration processes, meeting reminders, real-time assessments of satisfaction during webinar, follow-ups and post-webinar feedback surveys, annotations and highlighting procedures, and sharing of documents  Commencement of webinar one or two minutes subsequent to the scheduled starting time in order to maximize attendance  Reduction of background noise is important; participants should be instructed to keep their phones on mute until they have a question or a comment  Presentation of an outline of the webinar as an introductory PowerPoint slide  Counterbalancing of graphics with words in Webinar slides; graphics are important as 55% of learners are visual learners  Limit the participant group to 15 to 17 people, especially when training on a new skill. This allows for high levels of interaction and enables the instructor to identify people who may be struggling with the content. Webinar Content  Content of Webinar should align with best practice literature in content area

45. 1.Fit  Customization of webinar advertisements to the population of focus  Customization of webinar content to the population of focus 1.Capacities Technical capacities  Does the audience have access to computers, internet, space free from distractions, etc.?  How many simultaneous users does Webinar software support? Human capacities  Identification of individuals for webinar roles: primary presenter, marketer, moderator, and technical coordinator  1.Plan  Scheduling of date and time for webinar  Distribution of personalized invitations to potential attendees  Development of a schedule that outlines the sequence of a webinar’s components or phases  Develop webinar content and other materials  Rehearse webinar presentation  Distribution of reminder emails to individuals who are registered for webinar  Maintenance of a library with recorded webinars; individuals who registered for the webinar but no- showed should be notified about the availability of a recording 1.Process Evaluation  Monitoring of implementation metrics, including number of registrants, and attendance rate  Monitoring of attendees’ engagement o Use of real-time polls to get feedback about participants’ opinions and attitudes o Use of a real-time “satisfaction meter” as an index of audience engagement  Administration of post-webinar survey o Collection of Likert self-report data from attendees on: webinar content, webinar design, presenter effectiveness (clarity and pace), planning and delivery  Facilitator debriefings 1.Outcome Evaluation  Were Goals and Objectives from Step 2 met? 1.CQI  Develop a systematic process for learning from previous webinars to improve future webinars. 1.Sustainability  Identify program champions  Address funding and marketing issues

46. Distilling the Information— Prevention Synthesis & Translation System Supporting the Work—Prevention Support System Putting It Into Practice—Prevention Delivery System Synthesis General Capacity Building Innovation-Specific Capacity Use General Capacity Use Innovation-Specific Capacity Use Climate Funding Existing Research and Theory Translation Implementation QI/QA Macro Policy iGTO GTO Contracting Outcomes

47. WHAT CAN FUNDERS, PRACTITONERS, AND RESEARCHERS/EVALUATORS HAVE IN COMMON?

48. Accountability Partnership Model RESULTS Funder Evaluator Agency

49. As Yogi Berra supposedly said: “It’s déjà vu all over again.”

50. SUMMARY

51. JEOPARDY AN ANSWER 1) A COST-EFFICIENT COMPONENT FOR INCREASING CAPACITY FOR KNOWLDEGE, SKILLS, AND BEHAVIOR WITHIN A BROADER SUPPORT SYSTEM 2) ACCOUNTABILITY PROGRAM IMPROVEMENT

52. WHAT IS THE PURPOSE OF TRAINING? A COST-EFFICIENT COMPONENT FOR INCREASING CAPACITY FOR KNOWLDEGE, SKILLS, AND BEHAVIOR WITHIN A BROADER SUPPORT SYSTEM

53. WHAT IS THE PURPOSE OF TRAINING EVALUATION? ACCOUNTABILITY PROGRAM IMPROVEMENT

54. STIMULUS QUESTIONS What is the purpose of IV E training? (eg what do you hope will be accomplished by IV- E training) What is the purpose of training related to CFSR?

55. As Yogi Berra supposedly said, "If you see a fork in the road, take it."

56. REFERENCES Chinman M, Imm P, Wandersman A (2004). Getting to Outcomes 2004: Promoting Accountability Through Methods and Tools for Planning, Implementation, and Evaluation. Santa Monica, CA: RAND Corporation, TR-TR101. Available at http://www.rand.org/publications/TR/TR101/. Wandersman, A., Duffy, J., Flaspohler, P., Noonan, R., Lubell, K., Stillman, L., et al. (2008). Bridging the gap between prevention research and practice: The Interactive Systems Framework for Dissemination and Implementation. American Journal of Community Psychology, 41, 171-181. Wandersman, A., Chien, V., & Katz. J. (accepted with revisions) Toward an Evidence Based System for Innovation Support (Tools, Training, Technical Assistance and Quality Improvement/Quality Assurance. American Journal of Community Psychology.