Presentation is loading. Please wait.

Presentation is loading. Please wait.

Quality of Non-Lending Technical Assistance A QAG Assessment October 25, 2005 Quality Assurance Group.

Similar presentations


Presentation on theme: "Quality of Non-Lending Technical Assistance A QAG Assessment October 25, 2005 Quality Assurance Group."— Presentation transcript:

1 Quality of Non-Lending Technical Assistance A QAG Assessment October 25, 2005 Quality Assurance Group

2 2 Table Of Contents I.Executive Summary3 II.Introduction13 III.Findings21 IV.Conclusions34 V.Recommendations35

3 Quality Assurance Group 3 I. EXECUTIVE SUMMARY

4 Quality Assurance Group 4 Summary of Findings F Assessment results show that NLTA has been a highly effective knowledge transfer instrument F But…improper coding of tasks under NLTA is high and calls for remedial actions

5 Quality Assurance Group 5 WHY ASSESS NLTA ? FManagement concern over rapidly growing share of NLTA in AAA program, almost doubling in dollar terms (from $21m to $42m) and in number (from 175 tasks to 303 tasks) between FY01 and FY04 FNLTA quality never systematically reviewed by QAG

6 Quality Assurance Group 6 WHAT IS NLTA ? F The definition of free-standing NLTA applicable to the FY04 review period was human and technological inputs to the development of advisory services and instructional activities. NLTA often took the following form: –Workshops and conferences: Vehicles to share international best practice, exchange knowledge, or build consensus –Technical advice: Practical assistance to support implementation of policies, programs or projects –Technical notes: Written practical guides to assist the client in implementation of policies, programs or projects F The distinction between ESW and TA was whether or not the activity aims to produce original diagnostic/analytic content for the purpose of influencing the clients policies

7 Quality Assurance Group 7 THE SAMPLE FThe FY04 NLTA universe includes 303 tasks for a total cost of $41 million 1/ FThe NLTA sample includes 75 2/ randomly selected tasks for a total cost of $19 million, representing 25% of all tasks by number and 46% by cost 3/ FTasks costing less than US$ 20,000 were excluded from the sample FSample yields robust results at the Bank-wide level only 1/ At the time of sampling. This amount has since increased to $42.4 million 2/ The original sample included 85 NLTA tasks delivered to the client during FY04, of which 10 were Reimbursable TA tasks. It was subsequently decided that Reimbursable TA should be subjected to a separate QAG review. The sample size was reduced accordingly 3/The sample includes 15 tasks (20%) funded under the Public-Private Infrastructure Advisory Facility (PPIAF) for a total cost of $4.3 million (23%)

8 Quality Assurance Group 8 MAIN FINDINGS F48% of tasks sampled (36 4/ out of 75), accounting for 53% of the sample cost ($10.1 million), were improperly coded as NLTA. Improper coding is primarily caused by a weak overall governance framework (unclear product definition, low SAP user friendliness, insufficient guidance/support to TTLs, unclear accountability) F Only 52% of tasks sampled were assessed and rated (39 tasks 5/ out of 75 with a $ amount of 8.9 million or 47% of the total sample cost) as they were appropriately coded as NLTA 6 / 4/Including 5 PPIAF funded tasks or 7% of the number of tasks sampled, accounting for 6% of the sample cost and representing 30% of the number of PPIAF tasks sampled and 26% of their cost 5/Including 10 PPIAF funded tasks or 13% of the number of tasks sampled, accounting for 17% of the sample cost and representing 70% of the number of PPIAF tasks sampled and 74% of their cost 6/ The balance of 36 tasks could not be assessed and rated, using the NLTA guidance questionnaire, as they were not NLTA

9 Quality Assurance Group 9 F At 99% overall Moderately Satisfactory or better (SAT) for both the weighted number of tasks and their weighted value, the quality of FY04 NLTA is strong. This compares well with the 94% overall Satisfactory or better rating for the FY04 NLTA tasks reviewed under the country AAA assessment F A high number (29%) of tasks were rated Highly Satisfactory (HS) overall F Strategic Relevance and Timeliness, Internal Quality and Dialogue and Dissemination were found particularly impressive by the panels as these dimensions were rated 98% SAT overall with only 4%, 5% and 8% of tasks rated Moderately Satisfactory, respectively MAIN FINDINGS … Continued

10 Quality Assurance Group 10 FClose client involvement and solid consultant and Bank inputs resulted in both high quality content generation and significant knowledge transfer as indicated by the 99% and 96% SAT overall rating for Internal Quality and Likely Impact, respectively FManagerial attention over the implementation of PPIAF funded tasks has often been limited, resulting in missed opportunities for better dialogue and dissemination and greater impact FAlthough resources were found to have been very effectively used, panels noted a number of instances where task costs had not been fully recorded thus understating the cost of about a quarter of tasks assessed MAIN FINDINGS … Continued

11 Quality Assurance Group 11 KEY RECOMMENDATIONS OPCS should: F Simplify the presentation of the ESW and TA Decision Tree (eliminate overlapping definitions, clearly define the terms client, training, internal order and clarify the trigger for task creation) F Prepare and launch an NLTA portal, similar to the ESW portal currently under development, to provide better guidance and support to TTLs F Given SAPs low user friendliness and the extent of improper coding, allow all NLTA related entries (including the original task coding) into SAP to be made through the NLTA portal F Explore, jointly with SFR and TFO, ways to ensure full accounting of tasks costs (including TFs and other external contributions)

12 Quality Assurance Group 12 KEY RECOMMENDATIONS… continued FThe Regions and Sector Anchors should monitor the use of the TA product code by their staff to reduce the incidence of improper coding

13 Quality Assurance Group 13 II. INTRODUCTION

14 Quality Assurance Group 14 NLTA DEFINITION 1/ The definition of free-standing NLTA applicable to the FY04 review period was human and technological inputs to the development of advisory services and instructional activities. NLTA often took the following form: F Workshops and conferences: Vehicles to share international best practice, exchange knowledge, or build consensus F Technical advice: Practical assistance to support implementation of policies, programs or projects F Technical notes: Written practical guides to assist the client in implementation of policies, programs or projects. 1/ The key distinction between ESW and NLTA was that ESW aims at generating original analytic content while NLTA aims at disseminating/applying existing knowledge

15 Quality Assurance Group 15 THE NEW NLTA DEFINITION F The new (July 04) NLTA definition 2/ includes activities 3/ that are: –Aimed at enabling an external client to implement reforms or strengthen its institutions; –Free standing; and –Linked to a Bank unit accountable for the services provided (exclude client-executed grant-funded activities) 2/ The key distinction between ESW and NLTA remains that ESW aims at generating original analytic content while NLTA aims at disseminating/applying existing knowledge 3/ With the following SAP output types: (a) institutional development plan; (b) how-to guidance;(c) model/survey; (d) client document review; and (e) knowledge-sharing forum

16 Quality Assurance Group 16 FPromote accountability for quality by providing indicators on quality of NLTA work on a Bank-wide basis: FEncourage systemic change through: -Improved understanding of key determinants of NLTA quality -Disseminating assessment findings to appropriate Bank units ASSESSMENT OBJECTIVES

17 Quality Assurance Group 17 FMethodology and Guidance Questionnaire were developed by reviewing the NLTA governance framework as well as from lessons learned from a recent assessment of the quality of OESW FTasks costing between US$ 20,000-$50,000 were reviewed by a triage panel and were rated using a simplified version of the Guidance Questionnaire 4/. All other tasks were rated by a customized panel FThe overall quality of each task was assessed on five quality dimensions (Strategic Relevance and Timeliness, Internal Quality, Dialogue and Dissemination, Likely Impact and Bank Inputs and Processes) F110 panelists conducted the assessment F22 Bank staff participated in the assessments as observers 4/ The simplified questionnaire provides and overall rating and a summary rating for each quality dimension but does not include ratings for individual questions METHODOLOGY

18 Quality Assurance Group 18 FPanelists used QAGs new six point scale: -Highly Satisfactory (1): Best practice in several respects and no significant deficiencies -Satisfactory (2): Satisfactory or better on all aspects -Moderately Satisfactory (3): Satisfactory on all key aspects but significant missed opportunities -Moderately Unsatisfactory (4): Significant deficiencies in a few key aspects -Unsatisfactory (5): Significant deficiencies in several key aspects -Highly Unsatisfactory (6): A broad pattern of deficiencies METHODOLOGY …Continued

19 Quality Assurance Group 19 FThe formal trigger for entry into the universe was the standard SAP milestone Delivered to the Client FThe FY04 task universe included 303 tasks for a total cost of $41 million 5/ FAll tasks with a cost of less than $20,000 were excluded from the universe FThe sample was stratified by cost ($1m+, $400-$999k, $200-$399k, k and $20,-$50k) with over sampling of large tasks 5/At the time of sampling. This amount has since increased to $42.4 million SAMPLE

20 Quality Assurance Group 20 FThe sample includes 75 6/ randomly selected tasks with a total cost of $19 million, representing 25% of the universe and 46% of its cost 7/ FStatistical Robustness: The sample is only representative at the Bank- wide level (95% confidence level with a 9% sampling margin of error) 8/ FResults in this report are presented on a weighted basis to adjust for over/under representation in the sample because of stratification and sample design SAMPLE ROBUSTNESS 6/ The original sample included 85 tasks, of which 10 were Reimbursable TA (RTA) tasks. It was subsequently decided that RTA should be subjected to a separate QAG review. The sample size was reduced accordingly 7/ The sample includes 15 tasks (20%) funded under the Public-Private Infrastructure Advisory Facility (PPIAF) for a total cost of $4.3 million (23%) 8/ Results are less robust for Regional and Network cohorts given their small size

21 Quality Assurance Group 21 III. FINDINGS

22 Quality Assurance Group 22 F 48% of tasks sampled were miscoded (36 9/ tasks out of 75 sampled) representing 53% of the $ amount ($10.2 million out of $19.1 million) F Miscoding was found across all Regions and Networks F There is no significant correlation between tasks cost and miscoding. The average cost of miscoded tasks is $283,000 while the average cost of all tasks is $255,000 9/Including 5 PPIAF funded tasks or 7% of the number of tasks sampled, accounting for 6% of the sample cost and representing 30% of the number of PPIAF tasks sampled and 26% of their cost CODING

23 Quality Assurance Group 23 CODING …Continued FSAP not sufficiently interactive and user friendly as it does not provide adequate guidance to users to prevent task miscoding FPreparation of the ESW and TA decision tree a good initiative but: –Several TTLs found it overly complex, confusing, needing simplification –Some of its definitions are overlapping and could lead to miscoding –Some basic definitions are missing (who is a client, what is a legitimate trigger for task creation, what is training, what is an internal order etc…) FA number of Panelists as well as TTLs wondered why training was not considered part of TA for coding purposes

24 Quality Assurance Group 24 FOf the 36 Miscoded tasks: -8 were Partnership activities (PT) 10/ - 7 should not have been coded as NLTA since they were conducted for the Banks benefit -5 should have been coded as ESW given their original analytical content -4 should have been coded as project preparation/appraisal and 4 were not yet delivered to the client as work was under way -3 were not separate tasks (created simply to separately account for a funding source) -2 were a combination of several activities (ESW, preparation, IO etc…) -1 was External Training, 1 was project supervision and 1 should have been cancelled as it never started implementation 10/ Clear definition and processing guidance for this product line have not yet been issued CODING …Continued

25 Quality Assurance Group 25 F52% of tasks sampled were assessed and rated (39 tasks 11/ out of 75) as they were appropriately coded as NLTA (with a $ amount of 8.9 million or 47% of the total sample $ amount) 12/ FOverall quality of FY04 NLTA is strong (99% SAT 13/ by number of task and by value) and above the 91% satisfactory or better rating for NLTA tasks assessed under the country AAA assessment 14/ FThe most frequently stated development objective of NLTA is Institutional Development/Capacity Building 11/Including 10 PPIAF funded tasks or 13% of the number of tasks sampled, accounting for 17% of the sample cost and representing 70% of the number of PPIAF tasks sampled and 74% of their cost 12/ The balance of 36 tasks could not be assessed and rated as they were not NLTA 13/ SAT = Moderately Satisfactory or Better 14/ The country AAA assessment reviewed a total of 64 NLTA tasks (of which 1 was delivered in FY01, 9 in FY01, 20 in FY02, 23 in FY03, 10 in FY04 and 1 in FY05). The FY04 NLTA cohort was rated 94% satisfactory or better RATINGS

26 Quality Assurance Group 26 F Distribution of Ratings: RatingNo. of Tasks % of $ Amount Highly Satisfactory (HS)1129 Satisfactory (S)2156 Moderately Satisfactory (MS)61314 Moderately Unsatisfactory111 Unsatisfactory000 Highly Unsatisfactory000 Total39100 RATINGS …Continued

27 Quality Assurance Group 27 CountryTask TitleSector Board AfricaPEFA WorkshopsFin. Management ECAEnhancing Poverty Anal. & MonitoringPoverty Reduction LCRIndigenous Peoples & Sustainable Dev.Social Development MoldovaSocial Protection Policy DialogueSocial Protection NepalLiving Standards SurveyPoverty Reduction PeruLocal Governance & delivery of ServicesUrban Development Russian FederationEducation Advisory ServicesEducation SloveniaPension ModelSocial Protection Sri LankaPrimary Dealers Capital Adequacy FrameworkFinancial SwazilandRailways Legal and RegulatoryPrivate Sector Dev. WorldGlobal Labor ToolkitPrivate Sector Dev. OPERATIONS RATED HS

28 Quality Assurance Group 28 F11 tasks or 29% of both the weighted number of tasks assessed and their total $ amount ($2.3 million) were rated Highly Satisfactory (HS) FThe percentage of HS NLTA tasks (29%) is above the OESW and the FY02 ESW HS levels of 18% and 20%, respectively and comparable to the 29% SAT rating for NLTA tasks assessed under the Quality of Country AAA assessment FThe 11 HS tasks were distributed among seven Networks (PREM, HDN, PSD, INF, ESSD, OPCS and FIN) and 4 Regions (ECA, SAR, LCR and AFR) OPERATIONS RATED HS

29 Quality Assurance Group 29 FOutstanding aspects of HS tasks include (by order of importance): -Strategic relevance, particularly the consistency of the task objectives with the CAS -Internal quality, resulting from either high quality direct task team contributions or the preparation of thorough consultant TORs, a well managed consultant selection process leading to the hiring of highly qualified consultants and high quality supervision of their work by the Bank -Strong client involvement and participation and high quality interaction with the client OPERATIONS RATED HS …Continued

30 Quality Assurance Group 30 RESULTS BY QUALITY DIMENSION Quality DimensionHSSMSSAT Overall Assessment Strategic Relevance & Timeliness Internal Quality Dialogue and Dissemination Likely Impact Bank Inputs and Processes Percent of Tasks in Each Rating Category

31 Quality Assurance Group 31 STRENGHTS FAssessed tasks were implemented within a well defined strategic framework FTask objectives were clearly stated F For workshops/conferences the consultative process and high degree of client involvement were usually very instrumental in defining content, structure, venue and participants. High degree of client involvement noted for other tasks as well F Extensive and relevant knowledge from both inside as well as outside the Bank was drawn upon to generate high quality content

32 Quality Assurance Group 32 STRENGHTS … Continued F Both the quality of partnership arrangements with other donors and dissemination arrangements were very appropriate F For a number of tasks, client impact was evident very shortly following delivery (shift in policy positions, setting up of a regulatory body, introduction of competitive bidding process, modeling of pension options etc…) F Solid task team contributions often with help from highly qualified consultants, well supervised by the bank F Many panels noted that task implementation had resulted in significant knowledge transfer

33 Quality Assurance Group 33 WEAKNESSES FPanels noted that for PPIAF funded tasks, beyond the selection of qualified TTLs, managerial attention had often been limited during task implementation, resulting in missed opportunities for better dialogue and dissemination and greater impact FCosts had not been properly recorded for about a quarter of tasks assessed: FTasks often cross subsidized from other budgets/task codes FTF contribution sometimes not accounted for FOther contributions (from other donors, NGOs etc…) not accounted for FWith about 54% of tasks delivered to the client in June, the Banks internal processes, more than client needs, seem to drive the timing of delivery

34 Quality Assurance Group 34 IV. Conclusions F Results show that NLTA is a valuable knowledge transfer tool that has been used very effectively by the Bank to engage clients in a timely manner and to deal with strategically important issues with appropriate expertise F But…improper recording of tasks under NLTA is high and calls for remedial actions

35 Quality Assurance Group 35 V.RECOMMENDATIONS

36 Quality Assurance Group 36 FOPCS should: -Simplify the presentation of the ESW and TA Decision Tree (avoid overlapping definitions, clearly define the terms client, training, internal order and the trigger for task creation) -In order to provide better guidance and support to TTLs, finalize and launch an NLTA portal similar to the ESW portal currently under development -Given SAPs low user friendliness, allow all NLTA entries (coding, milestones, AIS etc…) into SAP to be made through the NLTA portal -Explore, jointly with SFR and TFO, ways to ensure full accounting of all task costs (including TFs and other external contributions) FThe Regions and Sector Anchors should m onitor the use of the TA product code by their staff to reduce the incidence of mis-coding


Download ppt "Quality of Non-Lending Technical Assistance A QAG Assessment October 25, 2005 Quality Assurance Group."

Similar presentations


Ads by Google