1. Perspectives on Impact Evaluation Cairo, Egypt March 29 – April 2, 2009 Presented by: Wayne M. Harding. Ed.M., Ph.D., Director of Projects, Social Science Research & Evaluation, Inc., Burlington, MA USA wharding@ssre.org Cheryl Vince-Whitman, M.B.A., Ed.M., Senior Vice President, Education Development Center, Newton, MA USA cvincewhitman@edc.org A Model for Using Technical Assistance to Improve the Evaluation Capacity of Local Programs

2. 1 Agenda Brief Overview of Service to Science (STS). Evaluation Findings About Service to Science Key Characteristics of The Technical Assistance Provided By Service To Science

3. Overview of Service to Science Initiative

4. 3 The Need for Service-to-Science Federal agencies and others in the USA have promoted the use of evidence-based programs to prevent substance abuse. The supply of “proven” programs is too limited to meet many local conditions. There are many other programs that may work, but lack evidence of their effectiveness

5. 4 Service-to-Science Goals To help innovative prevention interventions that address substance abuse (or related issues) develop, improve, and document evidence of their effectiveness. To increase the pool of effective and appropriate interventions.

6. 5 Organization of Service-to-Science Funded by Center for Substance Abuse Prevention (CSAP) in 2004. Implemented through 5 regional prevention centers (Centers for the Application of Prevention Technologies – CAPTs). In 2007, CSAP funded a related initiative for programs designed for Native Americans. Implemented by the Native American Center for Excellence (NACE).

7. 6 Service-to-Science Activities CAPTs solicit prevention program nominations from State prevention officials. CAPTs assess the readiness of candidate programs. CAPTs conduct regional Service to Science Academies. CAPTs provide follow-up technical assistance.

8. 7 Service-to-Science Activities (Continued) Service to Science Academy participants – eligible to compete for “subcontract awards” to further enhance evaluation capacity. CAPTs provide limited technical assistance post subcontract award.

9. 8 Number of Programs Served Through Service to Science As of July 2009, the CAPTs had provided 294 prevention programs with technical assistance. Since 2006, 110 programs have received the subcontracts of up to $30,000.

10. Selected Evaluation Findings

11. 10 Methods Pilot study (2006) interviews and/or online survey with program Directors. Target sample of 79 programs; 58 (73%) responded. Six case studies of 2005 STS participants and extended follow-up case studies on 5 that received mini-subcontracts. Six case studies of 2006 participants. 2007 online survey targeted 142 programs that had been participants for at least 15 months. 93 responded (a 77% rate after removing 21 programs for which respondents were no longer available).

12. Participant Program Characteristics

13. 12 Primary Behaviors Targeted for Change Percent (a) Alcohol Use/Abuse80% Illicit Drug Use/Abuse60% Tobacco Use58% Violence (including bullying) 37% HIV/AIDS 8% (a) Multiple responses allowed. N=93.

14. 13 Ages of Population(s) Served Percent (a) Adolescents (ages 13-17)71% Childhood (ages 6-12) 51% Young Adults (ages 18-25) 19% Early Childhood (ages 0-5) 10% Adults (ages 26-55) 10% Older Adults (ages >55) 7% No Distinction by Age 3% (a) Multiple responses allowed. N=93.

15. 14 Race/Ethnicity of Population(s) Served Percent (a) No Distinction85% Hispanic or Latino 9% American Indian or Native Alaskan 5% Black or African American 4% White 4% Asian American 3% Native Hawaiian or Other Pacific Islander 3% (a) Multiple responses allowed. N=93.

16. 15 Strategies Used Percent (a) Percent Improve Knowledge / Awareness 90% Mobilize Community27% Improve Skills85% Improve Identification and Referral 25% Healthy Alternatives79% Improve Policies17% Improve Norms59% Improve Laws12% Improve Mental / Physical Health 54% Improve Access to and Quality of Care 11% Build Collaboration36% Improve Enforcement 8% (a) Multiple responses allowed. N=93.

17. Service To Science Outcomes

18. 17 Value of STS Services Percent (a) Satisfied with CAPT TA (Very Satisfied or Somewhat Satisfied vs. Somewhat Dissatisfied or Very Dissatisfied) 72% CAPT TA Useful (A Little, Somewhat, or Very Useful vs. Not at All Useful) 91% Would Recommend STS to Similar Programs (Definitely Yes or Probably Yes vs. Probably No or Definitely No) 91% (a) N=93.

19. 18 Factors That Facilitated Progress When asked to discuss factors that facilitated progress on their evaluation the largest percentage of clients (79%) identified TA from the CAPTs. (a) From the 2006 pilot study. N=58.

20. 19 TA Contributed to Program Capacity (a) in 5 Most Common Issues Percent (b) Improving understanding of program rating systems such as NREPP & NASADAD 95% Improving understanding of evaluation 95% Developing and/or improving a logic model 92% Developing and/or implementing a more rigorous evaluation design 95% Improving organizational commitment to evaluation 84% (a) Contributed “A Little,” “Somewhat,” or “A Great Deal” vs. “Did Not Contribute at All.” (b) N=93.

21. 20 Seeking Recognition and/or Funding Percent (a) Plan to seek or are seeking recognition from NREPP or NASADAD 57% Sought recognition from local, regional or other national organizations 13% Prepared a journal article 13% Presented evaluation findings at professional conference or meeting 13% Used evaluation findings to help secure funding 23% (a) N=93.

22. Defining Features Of Service To Science TA Defining Features Of Service To Science TA

23. 22 Defining Features of Service to Science TA Client-centered Customized Long-term Relationship-based Flexible

24. 23 Defining Features of Service to Science TA Client Centered Customized Long-term Relationship-based Flexible

25. 24 Defining Features of Service to Science TA Client Centered Customized Long-term Relationship-based Flexible