Presentation on theme: "AIDS 2012 - Turning the Tide Together PEPFAR OVC Evaluation: How good at doing good? (GH TECH/USAID 2011) Prof Lorraine Sherr UCL London UK Acknowledgements."— Presentation transcript:
AIDS 2012 - Turning the Tide Together PEPFAR OVC Evaluation: How good at doing good? (GH TECH/USAID 2011) Prof Lorraine Sherr UCL London UK Acknowledgements to Miriam Zoll (Co-author), Natasha Croome, Adele Daniels and Jaya Karira
Task: Summarize and Analyze PEPFAR OVC Evaluations 18 Evaluations 22 Programs 9 Countries – Kenya, Tanzania, Uganda, Mozambique, Rwanda, Namibia, Zambia, South Africa and Haiti 8 basic needs categories: Food/Nutrition, Health, Education, PSS, Economic Strengthening, Child Protection, Shelter and HIV Prevention Critique methodology and summarised findings. http://resources.ghtechproject.net/content/pepfar-ovc-evaluation-how-good-doing-good
Evaluation – is it ever expedient? 1. Systematic reading by reviewers. 2. Coding - extract themes agreed by two reviewers. 3. Sub-analysis of costing. 4. Methodology adequacy criteria. 5. Seven basic needs categories. 6. Recommendations for future evaluation and evidence- based policy. Variable12Agree- ment Refusal Rates22100% Random Selection 2366.7% Gender44100% Significance44100% Comparison Group 33100% Cross Sectional5466.7% Descriptive Data77100% Double coding 6 reviews (Kappa =.83)
Evaluations Surveyed 1.Brewster, Lee, et al, Catholic Relief Servies' OVC PEPFAR Programs: A Midterm Multicountry Program Evaluation 2.Nyangara, F., Obiero, W., Kalungwa, Z., Thurman, T., Community-based Psychological Intervention for HIV-Affected Children and their Caregivers: Evaluation of the Salvation Army's Mama Mkubwa Program in Tanzania, USAID, AXIOS, Measure Evaluation, March 2009 3.Thurman, T., Hutchinson, P., Ikamari, L., Gichuhi, W., Murungaru, K., Nyangara, Community Education and Sensitization as an OVC Care and Support Strategy: Evaluation of the Integrated AIDS Program-Thika in Kenya, USAID, University of Nairobi Population Studies and Research Institute, Measure Evaluation, March 2009. 4.Thurman, T., Rice, J., Ikamari, L., Jarabi, B., Mutuku, A., Nyangara, F., An Evaluation: the Difference Interventions for Guardians Can Make: Evaluation of the Kilifi Orphans and Vulnerable Children Project in Kenya, USAID, Measure Evaluation, University of Nairobi Population Studies and Research Institute, March 2009. 5.Nyangara, F., Kalungwa, Z., Obiero, W., Thurman, T., Chapman, J., Promoting a Home-based Program Model for Supporting Children Affected by HIV/AIDS: Evaluation of Tumaini Project in Iringa Region, Tanzania, USAID, AXIOS, Measure Evaluation, March 2009
Evaluations Surveyed 6. Swarts, Brian, et al, Empowering Better Care: Report on Economic Strengthening for OVC Caregivers in Uganda, Summary of Findings, The Salvation Army World Service Office, WORTH, USAID and PACT, April 2010 7. Hutchinson, P., Thurman, T., Analyzing the Cost Effectiveness of Interventions to Benefit OVC: Evidence from Kenya and Tanzania, USAID, Measure Evaluation, April 2009 8. Nyangara, F., Obiero, W., Effects of a Community-Focused Approach Supporting the Mosts Vulnerable Children: Evaluation of SAWAKA Jali Watoto Program in Kagera, Tanzania, USAID, Measure Evaluation, August 2009 9. Thurman, T., Hoffman, A., Chatterji, M., Brown, L., A Case Study: Kilifi Orphans and Vulnerable Children Project, Measure Evaluation and Catholic Relief Services with USAID, March 2007 10. Nyangara, F., Thurman, T., Hutchinson, P., Obiero, W., Effects of Programs Supporting OVC: Key Findings, Emerging Issues and Future Directions
Evaluations Surveyed 11. Murombedzi, Cornelius, Mugumya, Firminus, Opportunity International/Habitat for Humanity: Evaluation of Sustainable Income and Housing for Orphans and Vulnerable Children in Africa, International NGO Training and Research Centre, August 2009 12. Larson, B., Hamazakaza, P., Costella, J., Laurence, S., Costing of OVC Service Delivery in South Africa and Zambia, Center for Global Health and Development and Health and Development Africa, USAID/Project SEARCH, September 10, 2010 13. Jaden Consultancies, Weaving the Safety Net: Mid-Term Evaluation, Final Report, Christian Children's Fund and USAID, March 2008 14. Jaden Consultancies, Weaving the Safety Net: Mid-Term Evaluation, Z Scores Christian Children's Fund and USAID, March 2008 15. Scott, N., Pfaff, M., Semrau, K., Leyenaar, J., Suojanen, A., Mugala, T., Simon, J., End of Project Evaluation Report: Faith-based Regional Initiatives for Orphans and Vulnerable Children (FABRIC), USAID/Project Search, boston University OVC-CARE Project, Family Health International, August 2010 16. Yates, DeeDee, Finding the Inflection Point in Health Education and Economic Strengthening: Final Evaluation of Project HOPE's Track 1 OVC Program Sustianable Strengthening of Families and Orphans and Vulnerable Children Mozambique and Namibia, Cooperative Agreement #: GPO-A-00-05-00019- 00, May 31, 2010 17. Capps, J., Bukuluki, P., Davis, S., Swarts, B., Bailey, B., Wakubona, S., Sustainable Community Support for Orphans and Vulnerable Children in Tanzania and Uganda, Final Salvation Army (TSA) Uganda, June 2010. 18.RAPIDS Evaluation Final Report 2005–2009 Key Findings; Population Council, Africare & Rural Net Associates, July 2009, Revised 2010
Age Issues Focus on Adolescents 8-14 (6) 0-4; 0-18 (2) 5-11 (1) 7-12; 7-15 (2) 13-18; 13-17 (2) 12-19; 12-17 (2) N/A – (2 costing studies) 3 with unclear age
Grading methodology STRONG: No study used a RCT design in any form none graded as "strong". ADEQUATE : Nine studies graded "adequate" - some form of before/after, case control, and post intervention comparison design. 1 qualitative (with good design) Two costing studies graded as adequate - utilised consistent audit/costing data. WEAK : Six studies were graded as "weak" and conclusions/ findings should be viewed with caution.
Quantitative vs qualitative methodology The war is over 13 used/included quantitative 11 used/included qualitative Qualitative is no excuse for poor methodology. If qualitative used, quality standards on how sampled, how recorded, how analysed – coding and themes – not just the use of quotes (which is what a number did)
Complex interventions Most programmes are complex – mechanisms of impact difficult to align.
Data Handling Vast number of data points (15 Scott) Simplistic reporting (failure to do complex analysis) Percentages versus statistical tests Power calculations
Theory Role played (low) Recourse to theory (eg knowledge does not equate to behaviour – why focus and measure of knowledge eg 9 Nyangara looks at knowledge gaining, some look at attendance at knowledge imparting (not even if they got the knowledge). Generalisability Understanding mechanisms
Two forms of Evaluation – Against Pepfar Goals or against Programme goals
Methodological Trends Baseline: PEPFAR wants evidence-based approaches but most evaluations had no baseline, which is the only way to monitor and track impact of dollars and labor invested. Quantitative: Several of the evaluations were able to tell us how many OVC received services, how many caregivers had taken out a loan and repaid it. Standardized Indicators – good methodological reasons for using validated indicators (See Tulane group for good example). Outcome - How do you operationalize efficacy? (eg Yates 16 – use attendance at Health education as the indicator, but what if you attended and slept through? Gender: many mention few analyze
Sampling – good methodological reasons for using excellent sampling and reporting refusal rates. Staff training in each year were a relatively large share of program costs. If such expenses are considered investments in human capital, the training would provide services to the project over more than one year. If there is substantial staff turnover, prior investments in training would not provide services to the project in the future Dose Issue - Exposure dose seems to be an emerging issue. Home visiting is not a standardised commodity. Need to harmonise concepts in description and quantity standards
Findings by Basic Needs: Food/Nutrition - 14 / 18 9/14 evaluations focused on food/nutrition found OVC households to be moderately or severely food insecure. Improvements shown, but questions raised about dependence and sustainability.
Health Care ––11/18 8/11 studies demonstrated improved health outcomes Specifically notable when combined with economic strengthening and high dose/follow through
Education–– 15/18 Enrollment effect Supplies positive mental health effect. 0 / 14 identified gender imbalances in school enrollment, although one reported that more girls than boys miss more than two days of school. Cost – resentment Ongoing funding need until government shoulders this.
Psychosocial––13/18 10/ 13 home- visits clearly benefited OVC but not always guardians Negatives associated with infrequent visits, low meeting attendance at Kids' Clubs Targeting carers indirectly benefits OVC Volunteers overwhelmed by caseloads and ill equipped to provide HB PSS counseling, particularly to adolescents and caregivers.
Economic Strengthening–9/18 PEPFAR's investments in local savings and loan schemes are proving to be effective in raising many OVC households out of poverty Economic strengthening intersects significantly in other core areas of basic need, including food/nutrition, health, education, psychosocial support and shelter. Thus, economic strengthening remains one of PEPFAR's most potent and effective strategies for improving the lives of OVC and their families Costing - Great variation for example – Kids Clubs: $1.96 - $14.08 – Home Visiting: $3.55 - $89.68 – Gardening $24.00 - $89.00 – Food $1.39 - $32.36 – Educational Support: $17.23 - $141.32
Child Protection––16/18 Measurable role home-based visits play in reducing child abuse, stigma and social isolation Psychosocial interventions targeting guardians often benefited OVC as well Children whose homes had been visited by a volunteer > birth certificate (36% of all children had birth certificates) Home visitor training associated with improved outcome
Shelter––6/18 Shelter was the least prioritized basic need in all evaluations. Provision of shelter made significant positive differences in multiple areas, including protection, psychosocial support and health
HIV Prevention––15/18 Children's HIV knowledge appeared to increase in 7 out of 13 evaluations. Some interventions appeared to be more thorough than others. School-based HIV prevention activities were part of 5 out of 13 prevention initiatives and were estimated to be cost effective.
Summary Quality Standards. Define PEPFAR’s quality standards for evaluations? Methodology? Standardization? The very mention of "PEPFAR" should be synonymous with quality Costing - Great variation Volunteers - costly cyclical patterns are troublesome to program continuity and quality, heavy reliance, questions of quality. Food 3 / 13 focused on food - found > 80% of household samples were severely food insecure (86%; 86.3%; 88.5%). Kids’ Clubs – Mixed findings. Often volunteers poor results on issues of stigma and discrimination. One Stop Shop - Harmonizing and integrating complex PEPFAR-funded OVC programs can yield better and multiple outputs An emergency response should be saved for emergencies Evidence- based programming should be a priority - There was weak evidence of gains and impacts in the five core areas, but with a number of caveats
The way forward Updated PEPFAR OVC Guidance (2012) stipulates: “Allocate at least 10% of program budgets to M&E” PEPFAR supporting the development and implementation of: – Methodology to track impact of child welfare and protection systems strengthening – Standardized quantitative program evaluation tool to measure child and caregiver/household outcomes across countries at population level (Will be introduced in Nigeria, Mozambique)