1. NANPA Oversight Working Group 2000 NANPA Performance Review June 18, 2001 Pat Caldwell, Chair

2. June 18, 20012000 NANPA Performance Review2 New Environment Changes to NANPA’s functions and responsibilities in 2000. First and Second NRO Orders COCUS evolves into NRUF State delegated authority Increased State activity in NPA relief planning New criteria for code requests (e.g. facilities readiness, utilization thresholds) Greater regulatory demand for information

3. June 18, 20012000 NANPA Performance Review3 Performance Evaluation Process NOWG data gathering, analysis and feedback Solicit feedback via surveys and NOWG operational reviews Analyze input Develop conclusions and recommendations Review with the FCC Preview results with NANPA Present report to the NANC

4. June 18, 20012000 NANPA Performance Review4 Input Analyzed Survey responses, operational review and observations NANPA’s Annual Report NANPA operational reviews and documentation Performance Feedback Survey responses –State Commissions16 –Industry26 –Non-USAcknowledgements

5. June 18, 20012000 NANPA Performance Review5 Response Comparisons Fewer responses than in the past

6. June 18, 20012000 NANPA Performance Review6 Analysis NOWG Activities and Process Contacted originator for clarification if survey responses seemed contradictory or unclear Aggregated analysis of survey ratings - Quantitative Aggregated analysis of survey comments- Qualitative NOWG observations and concerns Recommended steps for improvements

7. June 18, 20012000 NANPA Performance Review7 Criteria for Assessment of Comments Categories used for analysis NOWG developed chart to ensure analytical consistency of survey ratings. Note: This chart was not sent to survey respondents.

8. June 18, 20012000 NANPA Performance Review8 Criteria for Assessment of Comments Categories used for analysis NOWG developed chart to maintain analytical consistency throughout the survey comments analysis.

9. June 18, 20012000 NANPA Performance Review9 Conclusion NANPA 2000 Annual Performance Review resulted in “MET” rating The NOWG has determined that the NANPA’s performance resulted in a “Met” rating, which the NOWG determined means the following: Met the performance standards for the position. Outside the areas which may have been improved if CAS had been delivered, little improvement is needed in order to be considered fully successful in all aspects of the position. Performance was competent and reliable. Decisions and recommendations were sound in routine areas, and were generally sound in the less structured, non-routine areas. The reader should be aware that the NOWG’s performance evaluation of “Met” should not be used as a comparison to the1999 performance review conclusion of “Above Average”. The NOWG changed the rating scale and any comparison of scales may give the reader an incorrect interpretation of NANPA’s performance. In addition it should be noted that the NANPA’s “Met” rating was achieved during substantial changes in 2000.

10. June 18, 20012000 NANPA Performance Review10 Findings Summary of survey verbatims and NOWG observations Annual Report –Vast improvement with all requirements met except NANP exhaust Communications / Responsiveness –Prompt, courteous and professional –Need to improve notification and provide longer transition periods when changing processes or requirements Guidelines / Requirements –Excellent job of maintaining confidentiality –Need to conduct a full review of all information on Part 1 applications to avoid multiple re-submissions –Need to improve Part 4 record management and reporting

11. June 18, 20012000 NANPA Performance Review11 Findings (con’t) Summary of survey verbatims and NOWG observations Technical / Analysis –Demonstrated improved expertise –Need to improve familiarity with local conditions –Need more proactive communications during NPA Relief meetings –More practice resolution of contentious issues –Need consistent interpretation/clear direction from NANPA regarding (new/developing) FCC rules and guidelines  Tools The Document Distribution System received accolades Lack of information on CAS implementation and deployment date Forecasting model/analysis approach unclear and unknown

12. June 18, 20012000 NANPA Performance Review12 Findings (con’t) Survey verbatims and NOWG observations Staffing –Helpful, responsive and willing to assist –Need to improve response consistency with information provided by different NANPA personnel Web –Contains useful and valuable information –Needs timely updating with new/additional information –Somewhat difficult to navigate Tactical –Continued reports of confusion regarding responsibilities of NeuStar employees who sometimes perform functions as NANPA –Changes to administration procedures lack clear written justification and documentation

13. June 18, 20012000 NANPA Performance Review13 Recommendations Annual Report NANP exhaust forecast must be included to meet the baseline criteria per Requirements Document. NANPA has the option to include any additional information. Some suggestions that may improve the report are: 1. Caribbean numbering authority contacts and web sites to the same extent as the US and Canada 2. NPA maps 3. Information about the Binder of Decisional Principles and perhaps the index listing (see last bullet item under Section 7 of this Report) 4. Add an Index

14. June 18, 20012000 NANPA Performance Review14 Recommendations Communications / Responsiveness NPA Relief Planning Letters need to be kept current on the web site and reflect changes, when changes occur. NPA Relief Planners require training to better their facilitations skills, and ability to manage the consensus process during meetings and conference calls. NANPA must provide notification when modifying or implementing a new process giving its customers advanced notice of the change and its effective date. NANPA must revisit its practices on updating individual telephone voice messages to ensure announcements are updated in a timely manner.

15. June 18, 20012000 NANPA Performance Review15 Recommendations Guidelines / Requirements / Regulatory Directives Both NPA Relief Planners and CO Code Administrators must ensure that they maintain uniformity, and consistency in their treatment and responses to their customers. NPA Relief Planners need additional knowledge and training on local conditions, e.g. dialing plans. NANPA needs to institute a records tracking and management system to ensure that the duplicative efforts caused by misplaced Part 4 forms are avoided in the future. NANPA must make the FCC and NANC aware of concerns related to NPA relief activity. NANPA must proactively facilitate interpretation and resolution when there are conflicts between rules, requirements and/or guidelines.

16. June 18, 20012000 NANPA Performance Review16 Recommendations Staffing and Technical / Analysis Staffing Continue enhancing the training and development program, meeting facilitation skills and customer service for NANPA personnel. Technical / Analysis NANPA needs to improve on the NPA relief forecasting accuracy. If guidelines are inadequate, NANPA has an obligation to communicate the issue and provide a solution to the appropriate body for review. NANPA must continue to ensure that all CO Code administrators and NPA Relief Planners apply the industry guidelines in the same manner and eliminate inconsistent interpretations. (Note: NOWG acknowledges that states, under delegated authority, are permitted to direct NANPA to operate in a manner that does not comport with national guidelines.)

17. June 18, 20012000 NANPA Performance Review17 Recommendations Tools and Web Site Tools NANPA must provide NANC with their rollout schedule for CAS by the end of June such that the system will be available by September 1, 2001 NANPA must educate the industry and regulators about forecasting tools and assumptions. They should also be using all available data, including detailed historical demand and forecasts, to make forecasts the highest possible quality. Web Site Provide for more frequent updates to information on the web site. Provide for electronic assistance to those customers who are unable to navigate the site. Post more detailed NANPA organizational chart and contact information, including contact information for NeuStar employees who provide support to NANPA.

18. June 18, 20012000 NANPA Performance Review18 Recommendations Overall NANPA needs to review entire resource applications for completeness, and advise of all errors, before suspending or rejecting an application. There were issues raised in the 1999 performance review that came out again in 2000. CAS Consistency NANPA vs. NeuStar Annual Report

19. June 18, 20012000 NANPA Performance Review19 Next Year’s Review In the interest of continuous NOWG improvement Eliminate competing NANPA surveys during response period Distribute survey forms earlier Simplify survey format and analysis Define rating scale Document a process for managing multiple submissions from individual company’s