1 Client Level Data Collection: The California Experience Jeffrey Bernstein, MS Center for AIDS Prevention Studies, California Technical Assistance Program Shanna Livermore, BS Steven R. Truax, PhD California Department of Health Services, Office of AIDS, HIV Education and Prevention Branch, Sacramento Presentation for the 4 th Annual Center for AIDS Prevention Studies Conference San Francisco, April 16, 2004

2 Today’s Presentation Overview of ELI What challenges have been experienced at the state and local levels? How does ELI work successfully in California at the state and local levels? How has this technology been used to inform public health decisions?

3 ELI Background Overview Development

4 What is ELI? Evaluating Local Interventions (ELI) is a web- based evaluation system that enables California’s prevention providers to collect and access information critical for the planning and implementation of effective HIV prevention programs.

5 Background Evaluation is important for planning and enhancing programs and for securing resources. Office of AIDS (OA) has long recognized the need for evaluation and technical resources required to support it.

6 Background CDC Guidance on evaluation released in 1999 added additional motivation. CDC’s and OA’s shift in new directions

7 Shift happens!!

8 Benefits Systematic data collection –Gives the client a voice Consistent reporting requirements

9 Benefits There is a strong relationship between data collection and program planning Helps in grantwriting Helps structure your services/conversations

10 Benefits Immediate feedback to providers –Over 50 “cannned reports” –Raw data available

11 Prevention Activities Required to be Evaluated Individual-level interventions Group-level interventions Outreach interventions Prevention case management Health Communication

12 ELI Development Partners Coordination & Development –California State Office of AIDS –Universitywide AIDS Research Program/UCOP Program Evaluation Technical Assistance –Center for AIDS Prevention Studies, UCSF Guidance & Input –Local Health Departments –CDC

13 Strategic Planning Process for Development of ELI Sought input from stakeholders Assessed –existing evaluation efforts –state of the art measures, methods –local feasibility and utility

14 Pilot Phase Fall 2001

15 Pilot Process for ELI ELI piloted in 5 counties Feedback collected on –data collection forms & protocols –web system –resource issues –technical assistance needs Trainings and TA developed ELI revised

16 Pilot Feedback: Key Concerns Use of individual-level forms in outreach and group settings Need for additional fields, data elements, & reports Ability to download data Integration of existing data collection efforts into ELI system Need for training and technical assistance on data collection, system use, and use of data Resources

17 OA Response to Pilot Feedback Revised protocol for data collection in outreach and group settings Created additional data fields, optional fields & reports Developed downloading capability Designed & implemented statewide trainings Developed partnership with CAPS to provide evaluation TA to support use of forms & data OA is available for –system technical assistance –consultation for integration of ELI forms into existing data collection efforts

18 Explanation of forms

19 Prevention Interventions to be Evaluated Using ELI Intervention: A specific activity intended to bring about HIV risk reduction –1 Individual-Level Interventions (ILI) –2 Group-Level Interventions (GLI) –3 Outreach –1 Health Communication –1 Prevention Case Management (PCM)

20 Individual Level Interventions

21 Group Check Sheet (front)

22 Group Check Sheet (back)

23 Overview of ELI Elements Agency Information Intervention Info. –Intervention setting –Intervention type –Referrals –Risk reduction materials Demographics –Gender –Race/ethnicity –Sexual orientation –Matching criteria Behavioral Info. –Sexual Behavior –Hepatitis & STD History –Substance Use History Supplemental Info.

24 What are the Matching Criteria & Why are They Important? Date of Birth First Letter of Last Name Gender Race/Ethnicity Zip code

25 Selecting Forms Discuss form selection Forms should be decided on by the time intervention is set-up Some degree of form flexibility upon Office of AIDS approval

26 ELI Data Collection Forms Not a script! –Provide a framework for a conversation with a client –Optional word use and question order –Use terms familiar to and appropriate for the client

27 Explanation of Reports Over 50 “canned” ELI reports available –Age by Race by Gender –Missing Fields –Intervention Effectiveness –Referrals by Risk Raw data available

28 Reports

29 Reports Budget Report Referrals by Risk Gender by Race by Age Group Sexual Orientation by Race by Age Group STD by Race by Gender STD Type by Race by Gender Drugs by Race by Age Group Materials Distributed by Risk Missing Fields Intervention List

30 Reports

31 Reports Garbage in – garbage out Reports are as rich as the data you enter

32 Challenges

33 Challenges Time Staffing Changing CDC requirements Funding –Program cuts

34 Challenges Individual client data requirements in certain settings Federal vs. State vs. Local requirements Information Technology is not always a super highway Different levels of knowledge Requires new scope of work format Intervention set-up: LHJ vs. CBO

35 Challenges Data Collection –New forms/form integration –Matching Criteria BIG challenge with some populations (e.g., clients with immigration issues)

36 Successes

37 Successes Increased communication Increased collaboration Increased technical assistance Increased training opportunities

38 Successes Accurate timely information about HIV prevention interventions Efficiency increase scope of work review Intervention Working Group (IWG)

39 Increased focus on quality interventions! NOT quantity.

40 Characteristics of Successful Interventions Science based Highly targeted Clearly defined goals and objectives Realistic goals, objectives and implementation timeline Focused on reducing specific risk behaviors Adequately trained staff

41 Characteristics of Successful Interventions Sensitive to target population Sustainable resources and support Flexible Part of community plan Culturally, developmentally competent Gender specific

42 Lessons Learned Secure funding ahead of time Lengthen and expand the pilot phase Be prepared for major delays

43 Lessons Learned Need for forms in Spanish Be prepared for varied knowledge levels –Including data issues Promote data utilization

44 Lessons Learned: Agency level data use Clearly identify the intended users of the data Develop buy-in among data stakeholders Identify evaluation questions meaningful to the intended users

45 Lessons Learned: TA and Training Issues Regional trainings require out of county travel Training design must support new and experienced users of ELI SOW vs ELI vs Real Life Site visits often perceived as punitive

46 Next Steps Outcome evaluation Continue on site agency level TA on intervention set up Data Use –More training required –Report manual

47 Next Steps Suggestions for next steps –What should be the focus of future ELI trainings? –How can we make ELI more agency friendly?

48 Contact info Shanna Livermore, Office of AIDS Jeffrey Bernstein, CAPS/CTAP