Case management versus M&E in the context of OVC programs: What have we learned? Jenifer Chapman, PhD Futures Group/MEASURE Evaluation

Overview  Case management tools 101  Update on the Child Status Index  Key questions

What is a case mgmt tool?  Paper-based instrument  Completed by direct service providers (generally low-literate volunteers)  Assesses child’s well-being along priority dimensions (e.g. health, social relations, etc.)  Care plan documentation

Purpose of a case mgmt tool Purpose: To improve quality of care  Highest priority = Case workers’ needs o Untested hypothesis: case workers will make better decisions if they use a tool  May also support M&E / reporting needs

What is included? Information collected  Client contact information / demographics  Wellbeing information that is changeable over time  Care plan: services & referrals provided Information flow  Most important use is at local/SDP level  Some information may flow up regional level

A plethora of tools...  Child Status Index (MEASURE Evaluation)  Child Support Index (Pact)  OVC Wellbeing Tool (CRS)  Child Status Matrix (FHI)  Parenting Map (TSA)  Etc.

We need to be cautious  Some CM tools are being applied for purposes beyond case management  Targeting (identifying beneficiaries)  Program monitoring (recording services provided)  Evaluation (aggregating wellbeing scores)  Exercise caution in using a CM tool for other purposes

Child Status Index Assessment & Update

About the CSI  5 years ago, CSI was designed for low-literate home visitors to capture children’s status across the 6 domains of PEPFAR OVC programming  Early hopes that CSI could meet a range of information needs  CSI has been implemented for different purposes: case management to program evaluation  CSI is used in at least 16 countries O'Donnell K, Nyangara F, Murphy R, Nyberg B. Child Status Index. A Tool for Assessing the Well- Being of Orphans and Vulnerable Children—MANUAL. Chapel Hill, NC: MEASURE Evaluation; 2009.

CSI Assessment: Phase I  Rationale: to systematically assess how programs are implementing & using the CSI and understand OVC program field needs for additional tools to meet care, support, and M&E demands  Study questions  For what purposes are OVC programs using CSI?  What are the advantages and limitations of CSI?  What are the unmet M&E needs of OVC programs? Cannon & Snyder The CSI Usage Assessment. Chapel Hill, NC: MEASURE Evaluation.

Summary of Findings: Phase 1  25 interviews with senior program staff in 13 countries  Program staff find the CSI useful  CSI implemented mainly by volunteers  Information collected via CSI is rarely used by volunteers except for targeting (not recommended)  Care plans and referral protocols are inadequate

Summary of Findings: Phase 1  Variation in CSI implementation, data use due to:  Unclear purpose & guidance on CSI use with desire to assess impact  Variability in training approaches  Insufficient support/funding for technical assistance, follow-up, and training (data management, analysis)  CSI is important, but one tool in the toolkit

So now what?  Study Phase II  Problem: Lack of information on the utility of the CSI at the community-level as a job aid & input from CCWs  Purpose: To understand how CCWs and care teams make decisions about children (including role of job aid / data)  Methods: Interviews with CCWs and team leads in five countries, among organizations using/not using the CSI  Revision of CSI Guidance

6 Core CSI functions (we think) 1.Builds rapport between service provider and beneficiaries 2.Orients service provider to the holistic needs of the child and encourages referrals 3.Strengthens informed care decisions by systematically considering and documenting child’s needs

4.If applied regularly with the same child, may show a child’s progress over time in particular domains 5.May be helpful in community-level planning and resource allocation decision making 6.May reveal emergency situations (a score of 1 in any outcome area) 6 Core CSI functions (we think)

Probably inappropriate CSI uses  Targeting  Unnecessarily complex  May lead to expectations of action/enrolment  1 st contact with child may not be reliable  Evaluating regional or national program impact  Children’s needs/status are assessed relative to their local community, and not to national standards

Probably inappropriate CSI uses  Producing a single combined score for the child  CSI assessment should be presented as 12 independent measures  Risk varies across domains  CSI scale values are not equal-interval, but ordinal  Evaluating implementing organizations

Back to Case Management tools generally…

Some key questions  Are case management tools effective at improving care decision making?  For all types of case workers? Formal? Informal?  How does training in both case management and tool use factor?  What specifically about a case management tool improves care decision making? For whom?  Does the benefit outweigh the burden?

Some key questions II  Are CM tools useful in managing case workers?  Are some CM tools useful also for targeting beneficiaries, monitoring outputs (services delivered, and evaluating impact?  What are the risks?

The research presented here has been supported by the President’s Emergency Plan for AIDS Relief (PEPFAR) through the United States Agency for International Development (USAID) under the terms of MEASURE Evaluation cooperative agreement GHA-A Views expressed are not necessarily those of PEPFAR, USAID or the United States government. MEASURE Evaluation is implemented by the Carolina Population Center at the University of North Carolina at Chapel Hill in partnership with Futures Group, ICF International, John Snow, Inc., Management Sciences for Health, and Tulane University.