1. Supporting Community Priorities and Emphasizing Rigor An Approach to Evaluation Capacity Building with Tribal Home Visiting Programs
Kate Lyon, MA
Julie Morales, PhD
Erin Geary, MA
James Bell Associates -Tribal Home Visiting Evaluation Institute
Aleta Meyer,
US Department of Health & Human Services
October 2013

2. Overview Evaluation priorities: Evaluation requirement and approach
Expanding the evidence base
Address tribal & community questions
Evaluation requirement and approach
PICO Process
What did we learn?

3. Evidence-Based Policy & MIECHV
Requires State MIECHV grantees to implement evidence-based home visiting models
HHS conducted a systematic review of the evidence of effectiveness, known as Home Visiting Evidence of Effectiveness
HomVEE: results at
14 models currently meet “evidence-based criteria” for the State MIECHV program

4. Evidence-Based Policy & MIECHV
Identified studies of home visiting programs implemented in Tribal communities for review
No models met the criteria for evidence of effectiveness
Tribal grantees can:
Adapt an evidence-based model designed for the “general population” to Tribal setting
Use an evidence-based model developed for Tribal communities (but still may need adaptation to specific setting)
Develop their own model

5. Rigorous Evaluation Requirement
All Tribal MIECHV grantees are required to conduct a rigorous evaluation
Goal is to inform practice and build the evidence base around effective home visiting interventions with Native populations
Rigorous evaluation activities include:
Examining effectiveness of home visiting models in serving Native populations
Examining effectiveness of adaptations of evidence-based home visiting models for Tribal communities
Questions regarding implementation or infrastructure necessary to support implementation of home visiting programs in Tribal communities
Emphasis on rigorous evaluation comes from the legislation

6. Evaluation Approach
All knowledge will be generated through local evaluations
No cross-site evaluation
Evaluation questions are developed by grantees in consultation with their community to reflect local interests & priorities
Evaluation questions are informed by findings of needs assessment and connected to implementation decisions
Tribal ownership of evaluation process, data and dissemination is respected
IRB and Tribal approval is required
Iterative, connected, and circular process for program planning, development, implementation, and evaluation

7. Evaluation Approach cont.
Evaluations can be limited in size and scope
A focused question is answered with rigorous design and methods
Flexibility to focus evaluation on a component of home visiting
Evaluations will inform grantees, communities, and the field about what works in implementing home visiting in Tribal communities
Intensive technical assistance is provided to increase Tribal capacity and empowerment to conduct different types of evaluation
Don’t have large budgets, so we’re encouraging grantees to look at a subset of outcomes or examine a component of the home visiting program, such as just the prenatal component or a set of modules. Or maybe a cultural enhancement or implementation strategy, like training and supervision of HV or family recruitment and retention strategies.

8. P I C O PICO P The target population you plan to serve II
The intervention or program to be evaluated C
The comparison you will make to understand how well the program works in your community O
The intended outcomes you want to see achieved I C O
A framework for developing a well-built evaluation question
PICO developed by Mark Testa (UNC Chapel Hill) and used extensively with the Children’s Bureau’s Permanency Innovations Initiatives to reduce long-term foster care. (find material from slides)
Adapted from: Permanency Innovations Initiative Evaluation Team (2011) Logic Model and Theory of Change. Presented at The Children’s Bureau’s PII Kickoff Meeting, Washington DC, November 2010

9. PICO Question Example
Do Native families with a child under age 1 (P) that receive home visiting (I) demonstrate greater improvements in parent knowledge of child development and child development outcomes (O) than families who receive services as usual (C)?
You’ll see the PICO question on the posters. 9

10. PICO Discussion Helps to identify model and evaluation priorities
Return to PICO elements multiple times to refine evaluation question
To identify potential subpopulations
To consider potential comparison groups
To hone in on discrete intervention component
To prioritize outcomes
To help articulate theory of change
To refine hypotheses
You’ll see the PICO question on the posters. 10

11. PICO Discussion cont.
Brings together program leadership and staff, stakeholders, and evaluators
Collaborative discussion about community needs, program development, and evaluation
PICO builds rigor into the discussion
Allows participatory process within a prescribed framework

12. Population
Who is your target population for home visiting? What are their prioritized needs?
Needs assessment informs target population.
From broad: what is your community like?
To specific: who will you serve?

13. Intervention
Highlights the linkages between the needs of the target population, the program, and the benefits you hope to achieve.
What is the theory of change for the program(s) you have selected?
What implementation supports need to be in place to have a successful program?

14. Comparison Walk through different types of comparisons.
What is the alternative course of action that your comparison group will experience?
What is the strongest contrast that is feasible?
The strongest possible contrast will help to attribute any observed differences between the groups to your program.

15. Outcomes
What are the short- and long-term outcomes you hope to achieve?
Is there evidence that the program will impact those outcomes?
Which outcomes are most critical for the community?

16. Implementation PlanningC P I
Needs Assessment
Model Selection
Implementation Planning
Evaluation Question O
Links the evaluation question to the needs assessment and model selection
Connects the evaluation question to the intervention
And helps grantee to articulate their theory of change
Helps narrow the focus of the evaluation to a manageable piece
Builds a rigorous design into the evaluation question
Facilitates a community-engaged approach to evaluation planning 16

17. What Did We Learn?
Recognizing and respecting the need for community input
Allowing time for input
Will the findings be meaningful & useful?
Address the community priorities?
Recognize historical relationships with research/researchers
Understanding Tribal context:
Sample sizes vary widely
Capacity and resources available
Identification of needed supports
Acceptable research designs vary across tribes

18. What Did We Learn? Finding the balance: Translating terms
Awareness of “loaded” words
What does “rigor” mean?
Drawing on multiple ways of knowing
Indigenous knowledge
Evaluation Science
Qualitative and quantitative data
Will the findings contribute to the community and to the evidence-base?

19. Questions?

20. For more information on TEI contact: Nicole Denmark Kate Lyon
The Tribal Evaluation Institute is funded by the Office of Planning, research and Evaluation within the Administration for Children and Families. TEI was awarded to James Bell Associates in partnership with the University of Colorado’s Centers for American Indian and Alaska Native Health and Michigan Public Health Institute. For more information, contact the individuals on this slide.
The Tribal Home Visiting Evaluation Institute (TEI) is funded by the Office of Planning, Research and Evaluation, Administration for Children and Families, Department of Health and Human Services under contract number HHSP WC.
TEI is funded to provide technical assistance to Tribal Home Visiting grantees on rigorous evaluation, performance measurement, continuous quality improvement, data systems, and ethical dissemination and translation of evaluation findings.
TEI1 was awarded to MDRC; James Bell Associates, Inc.; Johns Hopkins Bloomberg School of Public Health, Center for American Indian Health, and University of Colorado School of Public Health, Centers for American Indian and Alaska Native Health.
For more information on TEI contact:
Nicole Denmark Kate Lyon
Federal Project Officer Project Director
Office of Planning Research and Evaluation James Bell Associates, Inc.