Stakeholder data validation workshop July 2, 2012 Windhoek Namibia looking forward: Effectiveness of capacity development support to local partners

Why we are here today To share Pact Namibias internal evaluation findings with stakeholders To get stakeholder validation of the evaluation recommendations

Agenda TimeActivity 9:30–10:45Presentations 10:45–11:00Tea break 11:00–11:30Q&A 11:30–noonSmall group work: Discuss findings & review recommendations Group 1. Capacity development impact Group 2. Graduation to direct USAID funding Group 3. Lessons learned for capacity development organizations Noon–12:45Report back 12:45–1:00Conclusions & next steps 1:00–2:00Lunch

Background USAID Namibia began scaling up HIV programs in In 2007 USAID awarded Pact a 5-year project which included grants, organizational and technical assistance to CSOs, predominantly in areas of: HIV/AIDS prevention Care and support to orphans, HIV-vulnerable children and home-based clients Strengthening capacity of the Ministry of Gender Equality and Child Welfare

Background In 2010 – award changed to focus on two objectives: 1. Strengthening capacity of the Ministry of Gender Equality and Child Welfare to effectively coordinate activities relating to gender equality and orphans and vulnerable children 2. Strengthening six civil-society organizations to be eligible for U.S. direct funding while maintaining high standards of existing services

Tools to strengthen civil society Organizational Development (OD) Roadmaps measures CSO partner organizational systems and structures objectively Basis for identifying organizational efficiency gaps and prioritizing interventions Progression on a scale from 1 (nascent) to 5 (mature) Comprehensive Institutional Strengthening Plans to identify, schedule and monitor all capacity- building activities

Tools to strengthen civil society Comprehensive Institutional Strengthening Plans to identify, schedule and monitor all capacity- building activities Organizational Development (OD) Roadmaps 10 capacity areas: Purpose & planning HR management Monitoring & evaluation Networking Governanc e Organizational sustainability Financial management Grants & compliance Operations management Projects & services

Organizational Performance Index (OPI) Scored on a scale of 1 to 4 on each of the following dimensions: Effectiveness. Achieving outcomes, meeting standards Efficiency. Delivering services, increasing reach Relevance. Engaging target populations, promoting learning Sustainability. Mobilizing resources, increasing legitimacy

Why do this evaluation? 1.Assess the sustainability of graduated partners capacity development 2.Review the quality and quantity of services offered to beneficiaries of former and current CSO partners (partially done) 3.Assess the effectiveness of the Pact Namibia capacity building assistance to the partners that have graduated or plan to graduate 4.Draw lessons for other Pact programs that are facing similar challenges of graduating partners

Evaluation questions To what extent was Pact Namibias capacity development support effective? To what extent were partners ready for and able to effectively manage direct funding? To what extent did service quality and coverage change over the last two years?

Process 3 phases: 1. Designed terms of reference and data collection tools (virtual group) 2. Collected data (Pact staff from 5 country offices, for neutrality) 3. Analyzed data, drafted and disseminated report (current group of Pact staff)

Data collection tools 1.Partner survey (Mobenzi) 17 current & former partner organizations interviewed Quantitative & qualitative 2.Organizational Performance Index (OPI) 12 of 17 partners 3.Pact staff survey (Mobenzi) 4 core Pact Staff from each technical area 4.USAID tool 5.Historical OD Roadmap scores

Data analysis Quantitative data: Merged datasets and did selected frequencies with statistics and cross-tabulations using SPSS and Excel. Qualitative data: thematic analysis.

Partner breakdown OD Roadmap support, graduated to direct funding OD Roadmap support, not graduated No OD Roadmap support, graduated No OD Roadmap support, not graduated Total Number of orgs 3 (18%) 2 (12%) 9 (53%) 17 Number of partner respondents 11 (23%) 12 (25%) 3 (6%) 22 (46%) 48

Study limitations Sample size Only 17 organizations Limited number of respondents in some organizations; some interviewed were previous staff Time Could not conduct survey of beneficiaries

Presentation of the findings Capacity development impact Graduation to direct USAID Funding Lessons learned for capacity development organizations Preliminary recommendations

Capacity development impact

Key areas of analysis 1.How did length of support influence partners OPI scores and perceptions of usefulness of support? 2.How do Pact and partner perceptions of support compare? 3.How do OD Roadmap supported partners differ from other partners? 4.Which areas of Pact support did partners rate as most useful and of highest quality? 5.What were the most significant changes resulting from Pacts support?

1. Partners scored longer support less useful Strategic Financial Program thematic Monitoring & evaluation

2. But Pact scored longer support more useful Partners score longer support less useful Strategic Financial Program thematic M&E Pact scores longer support more useful (except financial) Strategic Financial Program thematic M&E

3. Longer support – higher OPI scores Strategic Financial Program thematic Monitoring & evaluation

4. OD Roadmap partners scored higher on OPI

Non-graduated partners 1a. But non-graduates more keen on financial management

5. Non-OD Roadmap partners rated Pact support more useful Non-OD Roadmap OD Roadmap

6. Partners rated M&E support highest Quality rating Usefulness rating

7. Most significant changes M&E and Financial management thecornerstones of programs (partner, Pact and USAID responses) M&E Timely reporting Capturing the right and required data Use data for decision making Finance Proper management of organization finance & USAID funds Complete and accurate finance reports Proper budget management

7. Most significant changes Moved from no system to very good system in all areas (partner responses) Buy-in from organizations (Pact responses) How do we sustain the change? Experience handover and staff retention Resources

Summing up More years of support = lower usefulness rating, but higher OPI scores Pact and partners see usefulness differently over the duration of support OD Roadmap support = lower usefulness rating, but higher OPI scores M&E support rated most useful and of highest quality Most significant change in M&E and financial management

Graduation to direct USAID funding

Key areas of analysis 1.Capacity-building support that partners would like more of before graduation 2.Graduate vs. non-graduate performance on OD Roadmap and OPI 3.Prerequisites for graduation 4.Key capacity-building support for graduated partners

1. Partners want more technical support before graduation

Non-graduated partners Graduated partners 1b. And graduates far more interested in technical support

HRM MERL Financial Networking Governance Sustainability Compliance Operations Programs Purpose & planning 2a. Graduates scored higher on OD Roadmap Graduates Non- graduates

Relevance Sustainability Effectiveness Efficiency 2b. But OPI not different Graduates Non-graduates

1.Strategic plan 2.Governance 3.Strong financial systems and internal controls 4.Administrative systems, including human resource management 5.Skilled staff and technical capacity 6.Strong monitoring and evaluation 3. Graduation prerequisites from Pact and partners

4. Key post-graduation capacity development Priority Resource mobilization support Technical/Program support As needed Monitoring and evaluation Database development Evaluations Routine data quality assessments Financial management

Summing up 1.Graduated partners would have liked more technical support before graduation 2.Graduated partners averaged 3.4 on Roadmap, up from 2.5 for non-graduated partners (no difference on OPI) 3.Six graduation prerequisites: strategic plan, governance, finance, admin, technical capacity, M&E 4.Key post-graduation needs: technical and resource mobilization support

Lessons learned for capacity development organizations

Key areas of analysis 1.Retention of systems after capacity building, and impact of who decided to engage in it 2.OPI scores and trends across partner subgroups 3.Different capacity needs identified by partners and donors 4.Challenges and advantages of graduating to direct funding

1a. OD Roadmap-supported partners more likely to Non-OD Roadmap Support OD Roadmap Support retain systems

Who decides to acquire system % retaining system N Partner alone 95%60 Both partner and Pact 83%87 Pact alone 75%24 1b. Retention likelier if system is partner- driven

1c. Retention likelier if system is partner- driven Program thematic Financial Strategic Monitoring & evaluation

2. OPI scores Average OPI scores show lower resource mobilization but high social capital Graduated partners generally score higher Current partners score low on resource mobilization and service delivery

RelevanceEffectiveness 2a. Average OPI shows lower resource mobilization, high social capital EfficiencySustainability Average OPI

RelevanceEffectivenessEfficiencySustainability 2b. Graduates need to focus on results and external evaluation of standards OD Roadmap graduates Average OPI

2b. Current partners scored lower on RelevanceEffectivenessEfficiencySustainability Average OPI Non-graduated OD Roadmap partners resource mobilization and service delivery

3a. Different stakeholders, different top-ranked capacity needs OD Roadmap partners 1. Monitoring & evaluation 2. Technical support 3. Financial management 4. Program development Pact 1. Resource mobilization 2. Financial management 3. Institutional support 4. Technical support USAID 1. Resource mobilization 2. Financial management 3. Administration 4. Monitoring & evaluation Non-OD Roadmap partners 1. Technical support 2. Financial management 3. Advocacy 4. Monitoring & evaluation

3b. Graduated partners & USAID: contrasting expectations Partners expect: 1. Same relationship/treatment with USAID as partners had with Pact 2. Continued technical assistance, but often lack adequate budget 3. Increased USAID capacity- building support after graduation USAID expects: 1. Strong organizational capacity and governance systems, ability to work independently and maintain capacity even after staff turnover 2. USAID support to partners, but also partner ability to address own needs 3. Performance like long-term international USAID partners, responsive and accountable

4. Graduates found both challenges & benefits Challenges 1. Increased accountability in line with U.S. Government rules & regulations 2. Less frequent capacity- building support 3. Would like clearer guidance from USAID on budgets, funding & programs 4. Would like more regular feedback on financial reports Benefits 1. Shorter reporting channels give more time to prepare reports 2. Improved interaction with USAID & understanding of USAID programs 3. Less cumbersome financial reporting requirements 4. International exposure & recognition 5. Good working relations

Summing up 1.Current partners most likely to retain M&E, financial and technical systems 2.Sustainability of capacity building depends on whose decision it was to engage in capacity development 3.On average partners tend to score lower in resource mobilization on OPI scoring 4.Partners still need to develop capacity, but are struggling to understand how to access support under direct funding.

Preliminary recommendations

Capacity development impact Graduation to direct USAID funding Lessons learned for capacity development (CD) organizations General

Capacity development impact Capacity development support should be provided for three or more years (more research is needed). Capacity development organizations need to keep their support fresh for organizations served for longer periods. OD roadmap support is an important approach for preparing effective partners and should continue to be used. Capacity development support should increase focus on knowledge transfer and resource mobilization for increased sustainability.

Graduation to direct USAID funding The OD Roadmap should be considered as the tool to measure preparedness for an organizations readiness for graduation. The prerequisites for transition to direct funding should include: Strong strategic planning capacity Governance structures Strong financial systems and internal controls Strong administration (including HR) systems Skilled staff and technical capacity Strong M&E An ability to mobilize resources An ability to advocate with donors to meet (internal and external) constituent needs.

Lesson learned for CD organizations CD organizations should ensure ownership of interventions for maximum impact in the area of system retention. CD organizations need to work with partners to diversify their funding base. CD service providers need to help partners to remain relevant and vibrant by planning to incorporate internal capacity development support on an on-going basis. Increased communication on expectations will improve the transition to direct funding for both partners and USAID.

General Pact should consider expanding this dataset to include more Namibian respondents and more international graduates (33 worldwide) and partners in order to make the findings more robust. Pact should assess the impact of the capacity development approach (and the transition to direct funding) at the beneficiary level.

Thank you!