1. WORLD HEALTH ORGANIZATION Draft Report WHO/HQ Geneva – Dr. Sasha Goubarev WHO/SEARO & WHO/Nepal Presented by Karen Gladbach Contributions by Arie Rotem & Amitai Rotem U.N. Senior Fellowship Officers Meeting 2-4 November 2010 – Paris Impact Evaluation of WHO Fellowships Nepal Pilot Study

2. The Journey begins... Why Nepal? Supportive team: WHO/SEARO & WHO/Nepal, home & host institutes Quantity sample: 55 long-term Fellows (3 months+) Supporting FEL documents: readily available Core institutes: less diversity

3. The pilot study aimed to: 1) Evaluate the impact of WHO fellowships in Nepal: 2004-2008 2) Derive lessons from Contribution Analysis to evaluate impact 3) Assess availability + usefulness of existing FEL information from national & international fellowship authorities 4) Develop and/or refine tools and protocols to obtain evidence

4. The Study Sample Step 1 – Analysis Reviewed all 55 long term fellows-- three cohorts (2004-2008) Analysis of existing information available in WHO files Step 2 – Interviews Face-to-Face (F2F) interviews with 26 Fellows Focus groups with fellows, peers and home institutes Kathmandu, Dharan, and districts in Chitwan and Pokhara provinces Step 3 – Final Analysis and draft report

5. Milestones: FONTS F – Future: What is Fellow expected to do with training post-Fellowship? O – Objectives: Are objectives SMART? Will they enable Fellow to meet post- Fellowship expectations? N – National Priorities: Are national priorities clearly defined and known by Fellow, training & home institute? Was the Fellow the right choice? T – Training: Is the training institute aware of and able to address the training objectives aligned with national priorities? Will they conduct their own post- training evaluation? Are they aware of post-Fellowship expectations? S – Sharing: How will the Fellow share the new knowledge and skills? How will this sharing lead to impact?

6. Levels of Impact Sustainable Impact: Created a new unit, established new services, applied new procedures, major transfer to others, sustainable capacity building, likely to be sustained beyond the fellow’s direct involvement Considerable Impact: Significant improvement of competence leading to better performance and better outcomes, sharing with others Some Impact: Gained skills, some opportunity to apply or share with others No Immediate Impact: No significant gain in capabilities and/or inability to apply in Nepal context

7. Assessment of Impact Based on this analysis it has been concluded that: One in five fellows interviewed (19%) made sustainable impact as a result of their fellowship training Two in five (42%) demonstrated a considerable impact One third (31%) had at least some impact In two cases (8%) no impact was evident due to inappropriate placement that led to incomplete program

8. Impact rating # % Clinical specialists Senior managers District level officers Clinical academics Non clinician academics Sustainable 51931-1- Considerable 11421235- Some 83112113 No immediate 282---- Total2610075473 Overall Impact Ratings Interviewed Nepal Fellows 2004-2008

9. Impact Sustainable Impact: Established new Intensive Care unit in a major hospital Contributed to development of new medical education model Considerable Impact: Introduced new techniques for testing and preparing antigens Improved the use of phototherapy in treatment and research Taught new methodologies to colleagues and/or students Some Impact: Managed investigation and surveillance of local cholera outbreak, applying new surveillance techniques

10. Impact No Impact: Senior neurologist placed at an institution that could not provide advanced training -- Fellow returned home early with no gained capacity Surgeon not able to complete study in urology due to budget restriction -- Fellow not recognised as urologist so could not to apply new knowledge and skills Equally important to learn from no impact cases

11. WHO Findings 1) Current information collected by WHO has the potential to address data needs to assess fellowship impact Fellowship Application Form: Fellow’s details, study objectives, endorsements Letter of Award: relevant information on training program & funding Termination of Studies Report: Fellows & host institutes’ assessment of training Utilization of Studies Report : Fellow & supervisor’ assessment one-year later Fellows in-depth written reports: Required by some home institutes BUT forms are forms – the quality and depth of responses vary

12. WHO Findings 2) Face-to-Face (F2F) Interviews are crucial Better understand complexity of individual cases Elicits potentially sensitive information Offers richer, in-depth ‘st ories’ of the journey 3) Debriefing important at various intervals Post-Fellowship (1-3 years) 4) Significant contributions were made in strengthening institutional capacity and provision of services

13. Recommendations 1) Provide briefing of national needs to training institutions 2) Require progress and post-training reports from training institutions 3) Require fellows to prepare detailed and insightful termination and follow up reports 4) F2F debriefings on utilization of training with a sample of fellows 1 -3 years post- fellowship 5) Review barriers to utilization and provide further support, when feasible and appropriate 6) Develop Fellowship alumni network

14. What Next? What can we do now to improve our FEL program? Gain Fellows’ commitment to follow-up evaluations (1-5 years) At program onset, include post-training evaluation commitment in application Incorporate interviews into FEL program, not as ‘add-on’ activity but integral component Select cohort of 20 each year Share information – ‘get the message out’ to stakeholders Package’ according to audience: Ministry / government, Home institutes, Host institutes KISS Keep It Simple Sweetheart

15. The Journey Continues... What’s your next step?