1. MHHSC PROGRAM Constructing a Cross-Site Evaluation of Ethnic Minority HIV Mental Health Services The Mental Health HIV Services Collaborative (MHHSC) Program Michael Costa, Abt Barbara J. Silver, CMHS Maria Madison, Abt Tandiwe Njobe, Abt Gabriela Garcia, Abt

2. MHHSC PROGRAM Presentation Goals Describe the MHHSC Program Components Context Relevance Convey: The process of this collaborative. utilization- focused evaluation Accomplishments to date (products) Uses of the outputs of products

3. MHHSC PROGRAM Substance Abuse Mental Health Services Administration (SAMHSA) Center for Mental Health Services (CMHS) Center for Substance Abuse Prevention (CSAP) Center for Substance Abuse Treatment (CSAT)

4. MHHSC PROGRAM SAMHSA HIV/AIDS HISTORY Prior to 2001: CSAT funds targeted capacity/HIV/AIDS Substance Abuse treatment programs for African American, Hispanic/Latino, and other racial/ethnic minorities. 2001: CMHS funds a similar targeted/expanded capacity program for community based organizations (CBOs) serving African American, Hispanic/Latino, and other racial/ethnic minorities.

5. MHHSC PROGRAM HIV Infection among People with Severe Mental Illness Across all published studies, the rate of HIV infection among psychiatric patients is 10%, 25 times higher than that of the general population. Cournos & McKinnon, 1997;Krakow et al., 1998;Rosenberg et al., 2001

6. MHHSC PROGRAM Number of partners Number of risky or anonymous partners Frequency of sex trading Rates of coerced sex McKinnon et.al., 1996, 1999 COMPARED TO GENERAL POPULATION, PATIENTS HAVE FEWER EPISODES OF SEX WITH A PARTNER, BUT THEY HAVE: Sexual Risk Behavior Among People With Severe Mental Illness:

7. MHHSC PROGRAM Elevated risk for HIV infection in psychiatric Patients Risk factors: Alcohol and other drug use Unsafe sex Environmental circumstances (poverty, institutionalization, etc.) Substance use is associated with both psychiatric symptoms and HIV risk Psychiatric Disorders and Risk for HIV Infection

8. MHHSC PROGRAM MHHSC Program 21 Mental Health Service Sites – CBOs at least 2 years experience in behavioral health care services MH Centers, Substance Abuse facilities, Primary Health Care &/or HIV/AIDS clinics Abt Associates, Inc. = Coordinating Center

9. MHHSC PROGRAM MHHSC Program Congressional requirement (CBC & CHC): provide these new HIV/AIDS-related mental health services in both traditional and non-traditional settings. Funding for mental health treatment services and related case management only. However, grantees are required to develop comprehensive integrated individual treatment plans and monitor primary and substance use treatment.

10. MHHSC PROGRAM WHO ARE THE SERVICE SITES? New HIV/AIDS-Related Services New services (no prior HIV/AIDS-related MH services)– 5 sites Expanded services – 16 sites Service Delivery Settings Traditional (primarily clinic-based) – 13 sites Non-traditional (e.g., mobile treatment, ) – 1 site Both settings – 8 sites Target Populations African American -19 sites Hispanic/Latino – 14 sites Haitian – 1 site Native American – 1 site

11. MHHSC PROGRAM Demographics – Gender

12. MHHSC PROGRAM Hispanic Latino/a = 30.65% Demographics – Race / Ethnicity

13. MHHSC PROGRAM Demographics - Age Age CategoryPercent Less than 20 years1.72% 20 – 24 years4.68% 25 – 29 years7.41% 30 – 34 years10.76% 35 – 40 years20.83% 40 – 44 years22.78% 45 – 50 years14.51% 50 – 54 years8.74% 55 years and older6.01%

14. MHHSC PROGRAM DSM IV Diagnoses

15. MHHSC PROGRAM DSM IV Diagnoses Categories

16. MHHSC PROGRAM MHHSC Program Goals Expand Effective Culturally Competent Mental Health Services For PLWHIV In Minority Communities

17. MHHSC PROGRAM MHHSC Capacity Building The Coordinating Center provides technical assistance to grantees: Local and regional trainings (e.g., ethics, neuropsychology, cultural competence) Expert speakers at national meetings on topics of mental health and psychiatry, gender issues, consumer involvement, etc. Assistance with data collection and management

18. MHHSC PROGRAM CULTURAL COMPETENCE Strategies to Promote Culturally-Competence Service Delivery Demographically similar staffing as target population(s) Treatment facilities reflect cultural interests of target group(s) (e.g., artwork/decor, waiting room music, etc.) Diversity/cultural competence training Consumer feedback This is being addressed in great detail by the MHHSC Cultural Competence Subcommittee

19. MHHSC PROGRAM MHHSC Utilization-Focused Cross-site Evaluation Clinically/Programmatically Relevant Evaluation meet the needs of the clinical and other program staff in their efforts to better serve their clients The MHHSC cross-site evaluation is voluntary, except for submission of required aggregate data critical that the cross-site evaluation be clinically and programmatically relevant and valuable, otherwise no site would agree to participate

20. MHHSC PROGRAM Evaluation as a Pain in the Neck Mental health/social service providers historical experience with evaluation Intrusive reporting requirement Necessary to satisfy Local, State and Federal funding requirements. Often data are never reported back to programs Concerns that data will used to make them look bad or draw inappropriate comparisons between sites and providers

21. MHHSC PROGRAM MHHSC X-Site Evaluation Collaborative Process Engaging the key stakeholders Local site-specific evaluators Local site clinicians Program Administrators Consumer Advisory Boards (CABS)

22. MHHSC PROGRAM MHHSC X-Site Evaluation Collaborative Process Evaluation Subcommittee (ESC) Logic Model Workgroup Eval/Clinician Communication Workgroup Site Program Logic Models Site-Specific Evaluation Summaries Determination of Evaluation Foci Across Sites

23. MHHSC PROGRAM MHHSC X-Site Evaluation Collaborative Process Face-to-Face Meeting w/MHHSC Stakeholders X-Site Candidate Evaluation Foci

24. MHHSC PROGRAM X-Site Candidate Evaluation Foci

25. MHHSC PROGRAM Quantitative Methods Used Across Sites

26. MHHSC PROGRAM Qualitative Methods Used Across Sites

27. MHHSC PROGRAM MHHSC X-Site Evaluation Collaborative Process Breakout Groups (BOGs) Evaluation Design Workgroup MHHSC National Meeting Consumer Network Committee X-Site Candidate Evaluation Foci Face-to-Face Meeting w/MHHSC Stakeholders

28. MHHSC PROGRAM MHHSC X-Site Evaluation Collaborative Process Next Steps Pilot Test Instruments/Protocols Final Draft Field X-Site Evaluation Pray Everything Goes As Planned Instruments/Protocols Beta Version X-Site Design Site Review

29. MHHSC PROGRAM Final Cross Site Foci Four main domains* of interest across sites: Mental Health Client Satisfaction Client Retention and Service Utilization Quality of Life & Medical Health *Cultural Competence appears in all domains.

30. MHHSC PROGRAM Common Questions of Interest Who is being served? What are the barriers to care and to services for the target population? How do programs overcome these barriers? What services are being used by the target population? Are the services being provided in a culturally competent manner?

31. MHHSC PROGRAM Mental Health Given the target population being served, what are the Prevailing mental health diagnoses? Co-occurring disorders? (e.g., substance use) Changes in mental health symptoms? Disorders that may be more prevalent with HIV positive status? Physiological HIV disease factors that contribute to mental health symptoms?

32. MHHSC PROGRAM Client Satisfaction Questions relating to care received by clients. What care is being received? What is the frequency of care? What is the level of client involvement in care? What is the setting in which care is received? Is the client satisfied with the care?

33. MHHSC PROGRAM Client Retention & Service Utilization Client Retention How many cases become active/inactive in a given time period? What is the site definition for active/ inactive clients? What are the site policies for case closing? What reasons are given for case closings? What retention strategies have been effective with the target population?

34. MHHSC PROGRAM Client Retention & Service Utilization - continued. Service Utilization What is the type, frequency and duration of services used? How are services provided? Referrals in and out of program Agency linkages - in-house and external - to other providers Service setting – traditional/ non-traditional

35. MHHSC PROGRAM Quality of Life How does the quality of life status and general medical health of a client impact upon Treatment compliance (medications, ITP)? Response to treatment? Client retention? Service utilization? How does quality of life status and general medical health change over time with treatment?

36. MHHSC PROGRAM Methods & Indicators 1.CLIENT LEVEL INSTRUMENT confidential – unique client id Demographics Social supports Acculturation Substance use and risk behavior Trauma Medical health Medication adherence Clinicians report

37. MHHSC PROGRAM Methods & Indicators 2.CLIENT SATISFACTION SURVEY anonymous Client characteristics Service utilization Care Client involvement in care Access/ barriers to care Cultural competence in care Overall satisfaction with care

38. MHHSC PROGRAM Methods & Indicators 3.FOCUS GROUPS with clients on site Focus groups will provide qualitative backdrop to analyze quantitative data from client satisfaction survey and client instrument Types of services used Satisfaction with services Barriers to care Cultural competency

39. MHHSC PROGRAM Methods & Indicators 4.SITE VISIT DATA COLLECTION MHHSC Coordinating Center Staff Continuum of services Location of site Geographic setting Site community Service setting (traditional / non traditional) Client retention strategies Site activities to overcome identified barriers to mental health care

40. MHHSC PROGRAM Utility of Cross-Site Analyses From the Sites Perspective Site buy-in has been an on-going process Clinicians and local evaluators participate in the evaluation design work group They took ownership of the evaluation design Made decisions on utility of collecting certain types of data across sites

41. MHHSC PROGRAM Client Focused Domain Client Characteristics Clients Presenting Diagnosis Barriers/Access to Care (enhance/maintain Clients Quality of Life/Health)

42. MHHSC PROGRAM Program Context Domains Program Structure CBO VS Large System of care Staffing patterns (FT, PT) Sustainability Efforts Plan in place

43. MHHSC PROGRAM Client Focused Domains Client Characteristics Site Uses Systematic documentation type of clinical services offered to what target population across the sites Tailor services to specific populations Identify (ID) staff training & resource needs Determine success or not in reaching target population ID changing client populations Ability to pool data across all sites by client characteristics Identify types of clients (positively) impacted by type of treatment Learn from sites experiences ID promising practices

44. MHHSC PROGRAM Client Focused Domains (cont.) Mental HealthSite Uses Examine patterns/changes in symptomatology. Are symptoms unique to a given population Better assess clients Determine client case-mix Document increased client needs Gain insight on co- occurring disorders Better understand client needs Tailor services offered Inform client retention strategies Examine patterns of mental health disorders across sites Understand impact of services Inform site staff on cultural competence issues

45. MHHSC PROGRAM Client Focused Domains (cont.) Barriers / Access to CareSite Uses Identify barriers to care specific to target population Improve engagement Improve retention ID methods for overcoming stigma Determine strategies for inter- agency collaboration Gain insight into cultural competence issues Gain better understanding of ease of access to mental health services

46. MHHSC PROGRAM Program Context Domains Program StructureSite Uses Understand contexts in which other sites operate Determine relevance/applicability of findings across sites Identify replicable program components Understand uniqueness of site challenges Learn from other sites' experiences about ways to overcome barriers to care Identify promising practices Create a menu of tools to address barriers

47. MHHSC PROGRAM Program Context Domains (cont.) Site Program Structure (contd) Site Uses Understand differences in the range of program services Identify gaps in services Identify staffing/resource needs Inform program expansion and sustainability Identify promising practices in client referrals Improve client treatment fidelity Improve client retention Create a menu of tools to address barriers to care

48. MHHSC PROGRAM Program Context Domains (cont.) SustainabilitySite Uses Examine data on services provided to support funding opportunities Develop sustainability plans at individual site or cross-site level if similarity in client needs Develop plan to target specific funding sources

49. MHHSC PROGRAM Purposes/Uses of Data Cross-site analysis Attention to differences across sites, client characteristics Careful & appropriate Nuanced, not reductionist Program context critical Use of qualitative and program-level data (e.g., differences in resources/capacity available)

50. MHHSC PROGRAM Purposes/Uses of Data (Recap) UTILIZATION-FOCUSED EVALUATION Data/analysis to be provided to sites User friendly feedback A timely manner For program improvement /development of better intervention strategies NOT REPORT CARDS Individual clinicians Individual sites