1. Andrea Moore Information Specialist MANILA Consulting Group, Inc. American Evaluation Association Annual Meeting November 11, 2010 The Community-based Organizations Behavioral Outcomes Project: Identifying Challenges and Lessons Learned National Center for HIV/AIDS, Viral Hepatitis, STD & TB Prevention Division of HIV/AIDS Prevention, Program Evaluation Branch

2. Formative Evaluation Planning Effective Interventions Planning Assess Needs, Define Problem Process Monitoring Determining What Services Were Delivered To Whom Process Evaluation Determining If Intervention Was Implemented As Intended Outcome Evaluation Determining If Intervention Caused Outcomes Impact Evaluation Determining Broader Impacts Outcome Monitoring Determining If Intervention Achieved Its Outcome Objectives Program Evaluation Pyramid 2

3. 3 HIV Prevention at CDC: Effective Behavioral Interventions (EBIs)  What are EBIs?  One component of CDC’s national HIV prevention strategy  28 science-based, community-, group-, and individual-level HIV prevention behavioral interventions  Demonstrated evidence of effectiveness in reducing risky behaviors  CDC funds community-based organizations (CBOs) and health departments to implement EBIs targeting those at high risk of HIV infection or transmission in their communities

4. 4 PEB and the Outcome Monitoring of EBIs: The Community-based Organizations Behavioral Outcomes Project (CBOP)  Since 2006, CDC has partnered with CBOs to implement a family of outcome monitoring evaluations known as CBOPs  Determine whether implementation of EBIs in the field produces behavioral outcomes similar to those observed in controlled research settings  Through a competitive process PEB selected CBOs to receive additional funding to conduct outcome monitoring on 5 EBIs:  VOICES/VOCES  Healthy Relationships  SISTA  Many Men, Many Voices  Mpowerment

5. 5 CBOP Evaluation Questions  Do clients receiving the EBI report changes in HIV risk behaviors at two follow-ups?  What client-, intervention-, or CBO-level factors are associated with reported changes in HIV risk behaviors at two follow-ups?  What were the experiences of CBOs conducting this outcome monitoring project?

6. 6 CBOP Evaluation Design  Clients eligible for CBOP if they were eligible for the respective EBI  Repeated-measures, with no control or comparison groups  CBOs were advised to enroll 100+ clients per year and retain 80% at each follow-up

7. 7 CBOP Evaluation Design, Continued  Data were collected using standardized, national variables  Data collection templates operationalizing the variables were provided by CDC  Follow-up data collection attempted with all participants regardless of intervention completion  Contextual data was routinely collected for project management and to inform process monitoring  Data quality assurance procedures at every phase of CBOP

8. 8 VOICES/VOCES  Single session, video-based intervention for small groups of heterosexual African-American and Hispanic men and women  Designed to increase condom use  Core elements include:  Viewing culturally-specific videos portraying condom negotiation  Educating program participants about different types of condoms and their features Source: www.effectiveinterventions.org

9. 9 Healthy Relationships  5 session, small group intervention for men and women living with HIV  Aims to enable participants to make informed and safe decisions about disclosure and reduce risky behaviors  Core elements include:  Using modeling, role-play, and feedback to teach and practice skills related to coping with stress  Teaching decision-making skills about disclosure of HIV status Source: www.effectiveinterventions.org

10. 10 SISTA  5 session, small group, gender- and culturally- relevant intervention for African American women  Aims to reduce HIV sexual risk behavior  Core elements include:  Discuss cultural and gender-related barriers and facilitators to using condoms (e.g., provide information on African American women’s risk of HIV infection)  Instruct women on how to effectively and consistently use condoms Source: www.effectiveinterventions.org

11. 11 Many Men, Many VOICES  7-session, small group, HIV and STD prevention intervention designed for African American men who have sex with men (MSM) who may or may not identify as gay  Aims to foster positive self image, educate participants about their STD/HIV risks; and teach risk reduction and partner communication skills  Core elements include:  Educate clients about HIV risk and sensitize to personal risk  Enhance self-esteem related to racial identity and sexual behavior Source: www.effectiveinterventions.org

12. 12 MPowerment  Community-level intervention with a group-level component for young MSM  Designed to decrease unprotected sex by providing HIV prevention, safer sex and risk reduction messages  Core elements:  Sponsoring social events to promote community building among young gay men  Conducting a publicity campaign about the project within the community Source: www.effectiveinterventions.org

13. 13 Future Outcome Monitoring Projects  RESPECT  An individual-level, client-focused, HIV prevention intervention, consisting of two brief interactive counseling sessions  WILLOW  A four-session social-skills building and educational intervention for adult women living with HIV

14. 14 Identifying Outcome Monitoring Challenges and Lessons Learned  CDC employs numerous methodologies to document outcome monitoring experiences  Throughout all CBOP evaluations  From both CDC and CBO perspective  CDC uses this information to:  Improve the quality of data collection, data entry, and quality assurance for the project in progress  Formulate solutions to challenges  Inform the design of future outcome monitoring projects

15. 15 Identifying Challenges and Lessons Learned: Information from Participating CBOs  Project manuals  Detailed notes from communication with grantees  Required reporting documents  Data entry verification logs, error reports

16. 16 Identifying Challenges and Lessons Learned: CDC Observations and Experience  Site visit observations  On-site discussions between CDC and CBO staff  CDC CBOP staff experience  CBOP team meetings and informal communications  CDC cross-team meetings and communications

17. 17 Synthesis of Information  CBOP staff synthesized information into lessons learned documents  Summary of lessons learned in agency reports (delivered to the CBOs and CDC leadership)  CDC staff from previous and current CBOPs applied lessons learned to the design and implementation of upcoming projects  Former CBOP grantees share their experiences with new grantees on new CBOPs