Supporting Community Priorities and Emphasizing Rigor An Approach to Evaluation Capacity Building with Tribal Home Visiting Programs Kate Lyon, MA Julie Morales, PhD Erin Geary, MA James Bell Associates -Tribal Home Visiting Evaluation Institute Aleta Meyer, US Department of Health & Human Services October 2013

Overview Evaluation priorities: Evaluation requirement and approach Expanding the evidence base Address tribal & community questions Evaluation requirement and approach PICO Process What did we learn?

Evidence-Based Policy & MIECHV Requires State MIECHV grantees to implement evidence-based home visiting models HHS conducted a systematic review of the evidence of effectiveness, known as Home Visiting Evidence of Effectiveness HomVEE: results at http://homvee.acf.hhs.gov 14 models currently meet “evidence-based criteria” for the State MIECHV program

Evidence-Based Policy & MIECHV Identified studies of home visiting programs implemented in Tribal communities for review No models met the criteria for evidence of effectiveness Tribal grantees can: Adapt an evidence-based model designed for the “general population” to Tribal setting Use an evidence-based model developed for Tribal communities (but still may need adaptation to specific setting) Develop their own model http://homvee.acf.hhs.gov/Tribal_Report_2012.pdf

Rigorous Evaluation Requirement All Tribal MIECHV grantees are required to conduct a rigorous evaluation Goal is to inform practice and build the evidence base around effective home visiting interventions with Native populations Rigorous evaluation activities include: Examining effectiveness of home visiting models in serving Native populations Examining effectiveness of adaptations of evidence-based home visiting models for Tribal communities Questions regarding implementation or infrastructure necessary to support implementation of home visiting programs in Tribal communities Emphasis on rigorous evaluation comes from the legislation

Evaluation Approach All knowledge will be generated through local evaluations No cross-site evaluation Evaluation questions are developed by grantees in consultation with their community to reflect local interests & priorities Evaluation questions are informed by findings of needs assessment and connected to implementation decisions Tribal ownership of evaluation process, data and dissemination is respected IRB and Tribal approval is required Iterative, connected, and circular process for program planning, development, implementation, and evaluation

Evaluation Approach cont. Evaluations can be limited in size and scope A focused question is answered with rigorous design and methods Flexibility to focus evaluation on a component of home visiting Evaluations will inform grantees, communities, and the field about what works in implementing home visiting in Tribal communities Intensive technical assistance is provided to increase Tribal capacity and empowerment to conduct different types of evaluation Don’t have large budgets, so we’re encouraging grantees to look at a subset of outcomes or examine a component of the home visiting program, such as just the prenatal component or a set of modules. Or maybe a cultural enhancement or implementation strategy, like training and supervision of HV or family recruitment and retention strategies.

P I C O PICO P The target population you plan to serve I   I The intervention or program to be evaluated C The comparison you will make to understand how well the program works in your community O The intended outcomes you want to see achieved I C O A framework for developing a well-built evaluation question PICO developed by Mark Testa (UNC Chapel Hill) and used extensively with the Children’s Bureau’s Permanency Innovations Initiatives to reduce long-term foster care. (find material from slides) Adapted from: Permanency Innovations Initiative Evaluation Team (2011) Logic Model and Theory of Change. Presented at The Children’s Bureau’s PII Kickoff Meeting, Washington DC, November 2010

PICO Question Example Do Native families with a child under age 1 (P) that receive home visiting (I) demonstrate greater improvements in parent knowledge of child development and child development outcomes (O) than families who receive services as usual (C)? You’ll see the PICO question on the posters. 9

PICO Discussion Helps to identify model and evaluation priorities Return to PICO elements multiple times to refine evaluation question To identify potential subpopulations To consider potential comparison groups To hone in on discrete intervention component To prioritize outcomes To help articulate theory of change To refine hypotheses You’ll see the PICO question on the posters. 10

PICO Discussion cont. Brings together program leadership and staff, stakeholders, and evaluators Collaborative discussion about community needs, program development, and evaluation PICO builds rigor into the discussion Allows participatory process within a prescribed framework

Population Who is your target population for home visiting? What are their prioritized needs? Needs assessment informs target population. From broad: what is your community like? To specific: who will you serve?

Intervention Highlights the linkages between the needs of the target population, the program, and the benefits you hope to achieve. What is the theory of change for the program(s) you have selected? What implementation supports need to be in place to have a successful program?

Comparison Walk through different types of comparisons. What is the alternative course of action that your comparison group will experience? What is the strongest contrast that is feasible? The strongest possible contrast will help to attribute any observed differences between the groups to your program.

Outcomes What are the short- and long-term outcomes you hope to achieve? Is there evidence that the program will impact those outcomes? Which outcomes are most critical for the community?

Implementation Planning C P I Needs Assessment Model Selection Implementation Planning Evaluation Question O Links the evaluation question to the needs assessment and model selection Connects the evaluation question to the intervention And helps grantee to articulate their theory of change Helps narrow the focus of the evaluation to a manageable piece Builds a rigorous design into the evaluation question Facilitates a community-engaged approach to evaluation planning 16

What Did We Learn? Recognizing and respecting the need for community input Allowing time for input Will the findings be meaningful & useful? Address the community priorities? Recognize historical relationships with research/researchers Understanding Tribal context: Sample sizes vary widely Capacity and resources available Identification of needed supports Acceptable research designs vary across tribes

What Did We Learn? Finding the balance: Translating terms Awareness of “loaded” words What does “rigor” mean? Drawing on multiple ways of knowing Indigenous knowledge Evaluation Science Qualitative and quantitative data Will the findings contribute to the community and to the evidence-base?

Questions?

For more information on TEI contact: Nicole Denmark Kate Lyon The Tribal Evaluation Institute is funded by the Office of Planning, research and Evaluation within the Administration for Children and Families. TEI was awarded to James Bell Associates in partnership with the University of Colorado’s Centers for American Indian and Alaska Native Health and Michigan Public Health Institute. For more information, contact the individuals on this slide. The Tribal Home Visiting Evaluation Institute (TEI) is funded by the Office of Planning, Research and Evaluation, Administration for Children and Families, Department of Health and Human Services under contract number HHSP23320095644WC. TEI is funded to provide technical assistance to Tribal Home Visiting grantees on rigorous evaluation, performance measurement, continuous quality improvement, data systems, and ethical dissemination and translation of evaluation findings. TEI1 was awarded to MDRC; James Bell Associates, Inc.; Johns Hopkins Bloomberg School of Public Health, Center for American Indian Health, and University of Colorado School of Public Health, Centers for American Indian and Alaska Native Health. For more information on TEI contact: Nicole Denmark Kate Lyon Federal Project Officer Project Director Office of Planning Research and Evaluation James Bell Associates, Inc. nicole.denmark@acf.hhs.gov lyon@jbassoc.com