1. Human Resources Development Promising Practices for a Functional National HIV Monitoring and Evaluation System
Linda Fogarty, Maya Tholandi, Laura Fitzgerald, Jhpiego
Wanda Jaskiewicz, IntraHealth International
Capacity Project

2. Acknowledgements USAID
Office of Population and Reproductive Health
Office of HIV/AIDS
Office of the Global AIDS Coordinator (OGAC) Strategic Information/M&E TWG
The rest of the study team: Reena Sethi, Alishea Galvin, Udaya Thomas (Jhpiego), Steve Yanks, Stephanie Schalk-Zaitsev (Training Resources Group; TRG)

3. Outline
Context and purpose What did we do? What did we find? What did we build? How might it help?

4. Our Purpose
Create a resource to help develop and strengthen national M&E workforces
Defining terms:
Capacity building vs. HR strengthening, workforce strengthening
SI vs. M&E
SI to include M&E, surveys and surveillance, HMIS
Note: Mostly “M” and very little “E”
Definitions:
Capacity Building: We are talking about capacity building, but HR capacity building. We are not talking about developing M&E systems, but the workforce needed to run the systems.
When we say M&E HR strengthening, we mean M&E workforce strengthening
When we say M&E, this is shorthand. We also mean the larger area of SI with its three components
In this world, we are doing mostly Monitoring and very little Evaluation. The desire to monitor and stregnthen monitoring systems is very donor driven. Donors require accountability.

5. Global Context Development Partners’ Commitment to “The Three Ones”
one national HIV strategic plan
one national HIV coordinating authority
one national HIV monitoring and evaluation system
Progress, but still gaps, in national HIV M&E systems
We know HR and HR capacity is a bottleneck as in larger health sector…

6. Human Resources for Health Action Framework (HAF)
WHO: World Health Report, 2006
Consensus among WHO, other donors, country representation
Following a call to action to address global health workforce crisis
Describe 6 Action Fields - improved workforce outcomes and eventual health outcomes
Improved Health Workforce outcomes = productivity, availability, competence

7. “The Twelve Components”
Global efforts to support M&E system strengthening
MERG, GAMET
AIMNET
PEPFAR/USG SI officers
UNAIDS TA
Organizing Framework for a Functional National HIV Monitoring and Evaluation System, UNAIDS, MERG, 2008
Organizing Framework for a Functional National HIV Monitoring and Evaluation System,
UNAIDS, MERG, 2008

8. M&E Capacity Building Guidance
Companion Piece to 12 Components: “Guidance on HIV Monitoring and Evaluation Capacity-building” UNAIDS, MERG
Capacity-building guidance at various levels
System
Organizational
Individual
For each of 12 components
National, sub-national, local and facility levels
2nd key resource: Guidance for capacity building.

9. NAC - M&E Database Manager Knowledge and Skills Requirements
Establish and manage a national HIV database in line with national M&E data needs
Install database at sub-national levels and oversee its operation
Train relevant staff at sub-national levels including key stakeholders from other sectors collecting information (e.g. MoH) on database use
Trouble-shoot database problems (see Component 10)
General:
- Results Based Management principles
M&E Technical:
- Quantitative data collection and analysis
- Indicator selection / target setting
- Information / data auditing / validation
- Information systems, HIS
Data use
Managerial:
- Management of human and financial resources
- Planning, setting standards and monitoring work
-- Organizing skills
- Foresight and problem solving
- Analytical skills
- Communication skills
NAC – Data Officer (monitoring data)
Collate data from government ministries and sub- national HIV Coordinating Units, and conduct…
Example of the level of detail provided for knowledge and skills for specific posts at different levels

10. What is needed next?
Examples of promising approaches to strengthen M&E workforce
Original Goal: provide resources to assist USG teams to write M&E strengthening sections of Country Operational Plans
Ultimate Goal: broader relevance to countries, other donors, other TA providers

11. Outline
Context and purpose What did we do? What did we find? What did we build? How might it help?

12. PP Selection Process Step 1 Step 2 Step 3 > 150 PPs Chose 53
~ 500 key informants
USG SI liaisons
UNAIDS M&E advisors
CA reps, local NGOS, civil society orgs
On-line surveys
AIMNET, Rhino
Step 2
Chose 53
Relevance
HAF area representation
SI area
Availability of multiple data sources
Completeness of information
Step 3
Final 28 cases
+12 “notable”
KI in-depth interviews
Site visits (Kenya, Ethiopia, South Africa)
Phone, in-person Ints
PPs scored
Assessed groups of PPs compared to others within each HAF
Methodology:
Gather Promising Practice (PP) nominations
Categorize PPs according to HAF action fields
Rate PPs on selected criteria
Complete in-depth interviews with key informants
Gather resource materials
Write up case study

13. PP Scoring Table Criterion Level 1 Level 2 Level 3 Program objectives
Score Justification
Program objectives
achieved
Initial results are promising; activity is too young to determine whether program objectives met
Activity has demonstrated positive results based on implementer/donor feedback appears to be meeting objectives without formal assessment
Activity met or exceeded stated objectives as evidenced by documentation 1 2 3
Score =
Funding support
Less than one year of funding
1–3 years of funding
More than 3 years of guaranteed funding
Affordability
Relies on outside technical assistance, ongoing significant financial/expert human resources
Significant financial/HR start-up costs limited to onset of activity
Relies primarily on existing government infrastructure and financial and human resources
Local ownership
Donor or implementing agency pushes for activity; little buy-in from government/local institutions; intervention/activity would likely fail without external donor/implementer (e.g. no local demand for activity, no involvement of local institutions in activity implementation)
Partial government/local institution ownership of activity; without external donor/implementer aspects of intervention/activity would not continue (e.g., local demand is moderate to high; some evidence of local operational involvement in activity implementation)
Fully integrated into government/ local institutions’ programming; does not require outside assistance for continuation (e.g., local demand is sustained at a high level; activity implemented entirely by local human and financial resources)
Replicable (within country)
No replication attempted/considered at this stage
Activity successfully replicated over time and/or geography (e.g., beyond pilot/local phase; implemented more than once; replicated in additional provinces, sites or locations)
Transferable
(to other countries)
No known example of possible or actual activity replication in another country
Activity considered for expansion or replication in another country
Activity successfully expanded/ replicated in another country
TOTAL SCORE =

14. Outline
Context and purpose What did we do? What did we find? What did we build? How might it help?

15. Human Resources Management
Case Studies N=28
Human Resources Management
(n=3)
Policy
Finance
(n=2)
Education
(n=13)
Partnership
(n=4)
Leadership
(n= 3)
A flavor of types of approaches
Education:
2- 3 week short courses/ certificate courses, for various purposes and audiences (DQA, M&E fundamentals, Community health workers, in evening for working adults)
Short, on-line trainings
1 year Masters program in M&E,
- Measure evaluations regional training centers and replications
HRM
Supportive supervision, mentoring
Performance Support
Job descriptions, minimum standards
Policy
Developing full M&E plans based on 12 components with estimated HRH needs included
Focused strategic planning of cadres, based on past experience, needs
Difficult ones:
Leadership
DIRECT: VLDP-M&E
INDIRECT: Formation of a M&E association in country
Partnership
Joint supervision missions
Finance
Course, materials provided outside M&E context

16. PP Distribution across SI Areas
Note: categories not mutually exclusive, n=29

17. PP Distribution across Regions

18. Outline
Context and purpose What did we do? What did we find? What did we build? How might it help?

19. Table of Promising Practice Case Studies, with HAF, SI and Region#
Name of Practice
HAF Area
SI Area
Region
Education
Policy
Partnership
Finance
Leadership
HRM
M&E
Surveillance
HMIS/GIS
Africa
Asia
Americas
Europe
Global 1
Planning a National M&E System for HIV in India around the 12-Components Framework 2
Planning a New Cadre of Data Capturers in South Africa 3
Planning for a New Cadre of Health Informatics Technicians in Ethiopia 4
Establishment of a Nongovernmental Organization Dedicated to M&E in Zanzibar 5
Virtual Leadership Development Program for M&E of HIV Programs 6
Partnerships to Strengthen Community-based M&E Reporting System in Kenya 7
Strengthening Partnerships in M&E for HIV: The Positive Impact of Joint Missions 8
Mukuru on the Move: Geographical Information System Mapping in an Urban Kenyan Slum 9
Partnerships for the Effective Use of Geography and GIS Technology for HIV/AIDS Interventions 10
Strengthening Human Resources for Health Financing Capacity in Peru

20. Performance Support to Improve HMIS in Uganda
HAF area: HRMS
SI area: HMIS
Geographic region: Africa
Background
The Uganda Protestant Medical Bureau (UPMB), the coordinating body for the Protestant health services in Uganda, requested assistance to improve the functionality of their health management information system (HMIS) system, and to increase the reliability and timeliness of data collection and reporting, thereby improving their capacity to make informed strategic decisions….
Description
The Capacity Project responded to the UPMB request with a performance support intervention in all 258 health facilities across the country, targeting UPMB officers engaged in HMIS management, including HMIS officers at the national unit, Diocesan Health Coordinators at the diocese level, and Health Units’ “in-charges” at health facilities and at all levels of the system (hospitals, health center II, health center III and health center IV). The intervention followed these five steps: 1) establish agreements and commitments among stakeholders; 2) decide the expected performance of local teams…
Results
A follow-up performance assessment was completed to measure progress. Average compliance with HMIS performance standards improved at all facility levels, particularly for data analysis (from 45% to 81%), reporting (34% to 79%) and decision making (28% to 72%), where the largest problems were found. Performance in hospitals improved the most (from 33% to 94%). Local action plans permitted an organized mobilization of local resources….
Recommendations
Implementers emphasize the importance of following all five performance support steps. They have found that the final step—recognition—is the most difficult for organizations to implement successfully and thus requires substantial support. …
Supporting Materials in Online Resource
Uganda HMIS performance support assessment tool June 2009
For further information, please contact the Uganda Protestant Medical Bureau
Website:
Telephone number: (0)

21. Performance Support to Improve HMIS in Uganda
HAF area: HRMS
SI area: HMIS
Geographic region: Africa
Background
The Uganda Protestant Medical Bureau (UPMB), the coordinating body for the Protestant health services in Uganda, requested assistance to improve the functionality of their health management information system (HMIS) system, and to increase the reliability and timeliness of data collection and reporting, thereby improving their capacity to make informed strategic decisions….
Description
The Capacity Project responded to the UPMB request with a performance support intervention in all 258 health facilities across the country, targeting UPMB officers engaged in HMIS management, including HMIS officers at the national unit, Diocesan Health Coordinators at the diocese level, and Health Units’ “in-charges” at health facilities and at all levels of the system (hospitals, health center II, health center III and health center IV). The intervention followed these five steps: 1) establish agreements and commitments among stakeholders; 2) decide the expected performance of local teams…
Results
A follow-up performance assessment was completed to measure progress. Average compliance with HMIS performance standards improved at all facility levels, particularly for data analysis (from 45% to 81%), reporting (34% to 79%) and decision making (28% to 72%), where the largest problems were found. Performance in hospitals improved the most (from 33% to 94%). Local action plans permitted an organized mobilization of local resources….
Recommendations
Implementers emphasize the importance of following all five performance support steps. They have found that the final step—recognition—is the most difficult for organizations to implement successfully and thus requires substantial support. …
Supporting Materials in Online Resource
Uganda HMIS performance support assessment tool June 2009
For further information, please contact the Uganda Protestant Medical Bureau
Website:
Telephone number: (0)

22. Performance Support to Improve HMIS in Uganda
HAF area: HRMS
SI area: HMIS
Geographic region: Africa
Background
The Uganda Protestant Medical Bureau (UPMB), the coordinating body for the Protestant health services in Uganda, requested assistance to improve the functionality of their health management information system (HMIS) system, and to increase the reliability and timeliness of data collection and reporting, thereby improving their capacity to make informed strategic decisions….
Description
The Capacity Project responded to the UPMB request with a performance support intervention in all 258 health facilities across the country, targeting UPMB officers engaged in HMIS management, including HMIS officers at the national unit, Diocesan Health Coordinators at the diocese level, and Health Units’ “in-charges” at health facilities and at all levels of the system (hospitals, health center II, health center III and health center IV). The intervention followed these five steps: 1) establish agreements and commitments among stakeholders; 2) decide the expected performance of local teams…
Results
A follow-up performance assessment was completed to measure progress. Average compliance with HMIS performance standards improved at all facility levels, particularly for data analysis (from 45% to 81%), reporting (34% to 79%) and decision making (28% to 72%), where the largest problems were found. Performance in hospitals improved the most (from 33% to 94%). Local action plans permitted an organized mobilization of local resources….
Recommendations
Implementers emphasize the importance of following all five performance support steps. They have found that the final step—recognition—is the most difficult for organizations to implement successfully and thus requires substantial support. …
Supporting Materials in Online Resource
Uganda HMIS performance support assessment tool June 2009
For further information, please contact the Uganda Protestant Medical Bureau
Website:
Telephone number: (0)

23. Performance Support to Improve HMIS in Uganda
HAF area: HRMS
SI area: HMIS
Geographic region: Africa
Background
The Uganda Protestant Medical Bureau (UPMB), the coordinating body for the Protestant health services in Uganda, requested assistance to improve the functionality of their health management information system (HMIS) system, and to increase the reliability and timeliness of data collection and reporting, thereby improving their capacity to make informed strategic decisions….
Description
The Capacity Project responded to the UPMB request with a performance support intervention in all 258 health facilities across the country, targeting UPMB officers engaged in HMIS management, including HMIS officers at the national unit, Diocesan Health Coordinators at the diocese level, and Health Units’ “in-charges” at health facilities and at all levels of the system (hospitals, health center II, health center III and health center IV). The intervention followed these five steps: 1) establish agreements and commitments among stakeholders; 2) decide the expected performance of local teams…
Results
A follow-up performance assessment was completed to measure progress. Average compliance with HMIS performance standards improved at all facility levels, particularly for data analysis (from 45% to 81%), reporting (34% to 79%) and decision making (28% to 72%), where the largest problems were found. Performance in hospitals improved the most (from 33% to 94%). Local action plans permitted an organized mobilization of local resources….
Recommendations
Implementers emphasize the importance of following all five performance support steps. They have found that the final step—recognition—is the most difficult for organizations to implement successfully and thus requires substantial support. …
Supporting Materials in Online Resource
Uganda HMIS performance support assessment tool June 2009
For further information, please contact the Uganda Protestant Medical Bureau
Website:
Telephone number: (0)

24. Performance Support to Improve HMIS in Uganda
HAF area: HRMS
SI area: HMIS
Geographic region: Africa
Background
The Uganda Protestant Medical Bureau (UPMB), the coordinating body for the Protestant health services in Uganda, requested assistance to improve the functionality of their health management information system (HMIS) system, and to increase the reliability and timeliness of data collection and reporting, thereby improving their capacity to make informed strategic decisions….
Description
The Capacity Project responded to the UPMB request with a performance support intervention in all 258 health facilities across the country, targeting UPMB officers engaged in HMIS management, including HMIS officers at the national unit, Diocesan Health Coordinators at the diocese level, and Health Units’ “in-charges” at health facilities and at all levels of the system (hospitals, health center II, health center III and health center IV). The intervention followed these five steps: 1) establish agreements and commitments among stakeholders; 2) decide the expected performance of local teams…
Results
A follow-up performance assessment was completed to measure progress. Average compliance with HMIS performance standards improved at all facility levels, particularly for data analysis (from 45% to 81%), reporting (34% to 79%) and decision making (28% to 72%), where the largest problems were found. Performance in hospitals improved the most (from 33% to 94%). Local action plans permitted an organized mobilization of local resources….
Recommendations
Implementers emphasize the importance of following all five performance support steps. They have found that the final step—recognition—is the most difficult for organizations to implement successfully and thus requires substantial support. …
Supporting Materials in Online Resource
Uganda HMIS performance support assessment tool June 2009
For further information, please contact the Uganda Protestant Medical Bureau
Website:
Telephone number: (0)

25. Performance Support to Improve HMIS in Uganda
HAF area: HRMS
SI area: HMIS
Geographic region: Africa
Background
The Uganda Protestant Medical Bureau (UPMB), the coordinating body for the Protestant health services in Uganda, requested assistance to improve the functionality of their health management information system (HMIS) system, and to increase the reliability and timeliness of data collection and reporting, thereby improving their capacity to make informed strategic decisions….
Description
The Capacity Project responded to the UPMB request with a performance support intervention in all 258 health facilities across the country, targeting UPMB officers engaged in HMIS management, including HMIS officers at the national unit, Diocesan Health Coordinators at the diocese level, and Health Units’ “in-charges” at health facilities and at all levels of the system (hospitals, health center II, health center III and health center IV). The intervention followed these five steps: 1) establish agreements and commitments among stakeholders; 2) decide the expected performance of local teams…
Results
A follow-up performance assessment was completed to measure progress. Average compliance with HMIS performance standards improved at all facility levels, particularly for data analysis (from 45% to 81%), reporting (34% to 79%) and decision making (28% to 72%), where the largest problems were found. Performance in hospitals improved the most (from 33% to 94%). Local action plans permitted an organized mobilization of local resources….
Recommendations
Implementers emphasize the importance of following all five performance support steps. They have found that the final step—recognition—is the most difficult for organizations to implement successfully and thus requires substantial support. …
Supporting Materials in Online Resource
Uganda HMIS performance support assessment tool June 2009
For further information, please contact the Uganda Protestant Medical Bureau
Website:
Telephone number: (0)

26. Performance Support to Improve HMIS in Uganda
HAF area: HRMS
SI area: HMIS
Geographic region: Africa
Background
The Uganda Protestant Medical Bureau (UPMB), the coordinating body for the Protestant health services in Uganda, requested assistance to improve the functionality of their health management information system (HMIS) system, and to increase the reliability and timeliness of data collection and reporting, thereby improving their capacity to make informed strategic decisions….
Description
The Capacity Project responded to the UPMB request with a performance support intervention in all 258 health facilities across the country, targeting UPMB officers engaged in HMIS management, including HMIS officers at the national unit, Diocesan Health Coordinators at the diocese level, and Health Units’ “in-charges” at health facilities and at all levels of the system (hospitals, health center II, health center III and health center IV). The intervention followed these five steps: 1) establish agreements and commitments among stakeholders; 2) decide the expected performance of local teams…
Results
A follow-up performance assessment was completed to measure progress. Average compliance with HMIS performance standards improved at all facility levels, particularly for data analysis (from 45% to 81%), reporting (34% to 79%) and decision making (28% to 72%), where the largest problems were found. Performance in hospitals improved the most (from 33% to 94%). Local action plans permitted an organized mobilization of local resources….
Recommendations
Implementers emphasize the importance of following all five performance support steps. They have found that the final step—recognition—is the most difficult for organizations to implement successfully and thus requires substantial support. …
Supporting Materials in Online Resource
Uganda HMIS performance support assessment tool June 2009
For further information, please contact the Uganda Protestant Medical Bureau
Website:
Telephone number: (0)

27. Outline
Context and purpose What did we do? What did we find? What did we build? How might it help?

28. How to use the collection
Meant to compliment Strategic M&E planning
Builds on previous work (M&E CB guidance)
Provides wide range of examples for comprehensive M&E workforce development
Not step-by-step replication guidance, but ideas for consideration and follow-up
Provides some resources and contacts for further information
Intended users
National planners
Examining how to implement M&E plan
USG country teams, development partners to improve reporting quality
UNAIDS and other M&E TA providers to add to toolbox

29. Dissemination Global M&E website USAID mission staff USG TWG partners
MERG assistance

30. Limitations Snow-ball sampling; not exhaustive search
Country “gate keepers’” enthusiasm controlled how far we could go
Fully executed interventions scored higher than up-and-coming practices
Case studies tended to be HIV-specific (because of donor goals)

31. Links of possible interest: http://www.globalhivevaluation.org/
Thank You!
Links of possible interest: