Targeted Evaluation of Five Programs Supporting Orphans and Vulnerable Children: Background and Methods Florence Nyangara, PhD MEASURE Evaluation/Futures Group Dissemination Meeting, September 3 rd, 2009 Washington, DC

The number of OVC and their corresponding programs increasing ■SSA - 12 million orphans (0-17 yrs), 2003 ■ over 18 million orphans (O) ■Other millions are made vulnerable – HIV/AIDS, dire poverty, war, etc (V) ■Response – increased attention to the plight of OVC (funds, programs) Children on the Brink, 2004 Sub-Saharan Africa’s population of children orphaned by AIDS increasing Background

Response to the OVC Crisis  OVC programs – emergency response to areas most HIV-affected  Strategies used were based on existing cultural support systems, conventional wisdom, and lessons learned from other program areas;  Support community-based responses (capacity & resources)  Household/family support (capacity & resources)  Direct support to families & OVC (access to essential services)  Gap – lack of evidence to guide OVC programs  Call for evidence based programming  USAID funded MEASURE evaluation to conduct targeted evaluations to fill this evidence gap Background

Evaluation Goals ■Find out “what works” in terms of ■intervention models and program components ■cost effectiveness, and ■outcomes (benefits) for OVC and their caregivers in resource poor settings ■Provide evidence to guide program decisions such as; ■Scaling-up of best practices (models, strategies), and ■Modify & improve interventions - to make them effective 4

Research Preparation Activities  Funds were made available – USAID/PEPFAR/PHE (4) and USAID/Tanzania mission funded (1) program evaluation.  Research team formed - MEASURE Evaluation  Extensive literature reviews (early 2006)  Consultation meetings with stakeholders  Identified OVC programs to be evaluated  Research protocol developed  Ethical approvals obtained – US, Kenya, and Tanzania  Identified local research partners (PSRI – KE; AXIOS - TZ)

Programs Evaluated Evaluated Five programs : 2 in Kenya & 3 in Tanzania  They have different intervention models with v aried combinations of child, family/household, and/or community centered approaches (multi-faceted). *** Although, the approaches vary, the goal for all of these programs is to improve the well-being of OVC and their families.

Programs Evaluated Kenya (2) 1.Kilifi OVC Project, Catholic Relief Services (CRS)  Operating in Kilifi District for two years 2.Community Based HIV/AIDS Prevention, Care and Support Program (COPHIA), Pathfinder & Integrated AIDS Program (IAP)  Operating in Thika District for 4 years Tanzania (3) 3.CARE Tumaini Project, Allamano, CARE, FHI (Allamano)  Operating in Iringa Region for five years 4.Mama Mkubwa & Kids Club, Salvation Army (TSA)  Operating in Mbeya Region for 2-years 5.Jali Watoto, Pact/SAWAKA (Jali Watoto) – Field funded  Operating in Karagwe, Kagera Region for four years

Overview of Programs & Strategies Evaluated

Key Research Questions  Impact of indirect support:  How do efforts targeted at the structural systems surrounding children– household and community– affect:  Children well-being  Caregivers well-being  Community attitudes and support of OVC & families?  Impact of direct support on child outcomes  What is the impact of educational, health, legal support, and other direct services on child & families?

Methods Case studies (2006 and 2007)  Site visits, interviews, program document review  Provide understanding of program strategies, components, goals, and expected outcomes  Document lessons learned from implementation  Case Studies available Program expenditures (2006)  Expense data collected and social costs estimated  Quantified the costs corresponding to specific intervention (e.g. food supplementation, psychosocial service, educational support)

Methods Outcome evaluation (2007 and 2008)  Post-test study design with intervention & comparison groups  Exposed Vs. Non-exposed  Surveyed children age 8-14 “or 7-15” & their caregivers  Up to 2 children per household  Four questionnaires were applied in each household:  Q1: Household Questionnaires  Q2: Parent/Guardian/Caregiver Questionnaire  Q3: Parent/Guardian/Caregiver Regarding Child Questionnaire  Q4: Child (age 8-14 “or 7-15”) Questionnaire

Ideal Impact Assessment R

Post-test Study Design Used Exposed Non-exposed XO1 XO1 O2O2 The Groups are Not Randomly Assigned ** Jali Watoto – Study compared intact groups of intervention versus comparison

Research Designs and Sampling

Principles Guiding Questionnaires Development  Capture multiple measures for each domain  Use existing standardized scales where possible (PSS, SES)  Intervention Exposure questions to be specific to each program  To facilitate comparison across countries and program models, same survey instruments were used except intervention modules  Multi-faceted programs necessitated sufficient questions across multiple domains  Multiple perspectives on child well-being (child and caregiver)  Measures of caregiver, household & community well-being

Outcomes Examined  Psychosocial well-being – multiple measures – standard scales used where appropriate (child and caregivers)  Education – enrollment & attendance (child)  Health – self-reported health status and access to health services (child and caregiver)  HIV-prevention – HIV-knowledge (child) & HIV-testing (caregiver)  Legal protection – birth registration, alternate caregiver  Community support – stigma and in-kind support (child & caregiver)

Study Strengths and Limitations  Strengths:  Yielded immediate data on program effects  Results can be used to improve current programs  Ethical - not withholding services for experiment sake  Limitations:  Post-test design – no baseline data - impossible to make conclusions concerning change in outcomes resulting from program exposure  Selection bias - self-selection to participate and those who did not -makes it difficult to conclude with certainty that the interventions are responsible for the observed differences

Analyses Plan  Who are the OVC/MVC program beneficiaries?  Effects of community level interventions i.e.  Community care and support meetings/sensitization  Effects of household or caregiver level interventions i.e.  Community volunteer or Health Worker home visits  Caregivers participation in OVC care seminars  Effects of child level interventions i.e.  Kids clubs  Basic needs support (e.g., education, health, legal)

Statistical Analyses  Descriptive analysis (Univariate)  Bivariate analysis (ANOVA and Chi-square)  Multivariate (logistics, and linear regression)  Control variable: non-program factors e.g. socio- demographic  Child Level – Age, sex, orphan status, relationship to caregiver, and number of different homes the child had lived in the past year.  Caregiver level – Age, sex, marital status, education, illness, SES, and # children

Description of OVC Program Beneficiaries  Who is enrolled in OVC programs (MVC Profile):-  Although, these programs targeted geographic areas most affected by HIV/AIDS, MVC were identified and assisted regardless of specific causes of vulnerability  Majority of children enrolled in OVC programs are vulnerable in several fronts & not just orphanhood

OVC Profile  Orphans (over 66% across programs)  Living in food insecure HH (over 80% across programs)  P oorest households (< 2assets) – over 40%  Living with chronically ill primary caregiver (over 20%)  Living with caregiver aged 50+ (about 23%)  Lived in two or more households in past year (14%)

Thank YOU! Key Findings are presented next….

MEASURE Evaluation is funded by the U.S. Agency for International Development through Cooperative Agreement GHA-A and is implemented by the Carolina Population Center at the University of North Carolina at Chapel Hill, in partnership with Futures Group International, ICF Macro, John Snow, Inc., Management Sciences for Health, and Tulane University. The views expressed in this presentation do not necessarily reflect the views of USAID or the United States government. Visit us online at