NANC 2016 NANPA Performance Evaluation Report Prepared by the Numbering Oversight Working Group (NOWG) June 29, 2017 2016 NANPA Performance Evaluation Report June 29, 2017 Page 2 of 27 Table of Contents Executive Summary .................................................................................... 3 Section 1.0 Performance Review Methodology ........................................... 4 Section 2.0 NANPA Reports ....................................................................... 5 Section 3.0 Program Improvement Plan (PIP) ............................................ 7 Section 4.0 2016 Performance Survey Results ........................................... 8 Section 5.0 Operational Review ................................................................ 10 Section 6.0 NANP Administration System (NAS) ...................................... 19 Section 7.0 Change Orders ...................................................................... 21 Section 8.0 NANPA Website ..................................................................... 23 Section 9.0 Conclusion and Recommendation .......................................... 25 Section 10.0 Acknowledgements & NOWG Participants ........................... 26 Section 11.0 List of Appendices .............................................................. 27 2016 NANPA Performance Evaluation Report June 29, 2017 Page 3 of 27 Executive Summary The North American Numbering Plan Administrator’s (NANPA) annual performance assessment is based upon a compilation of performance feedback surveys, monthly standing agenda conference calls, the annual operational review, and observations/interactions between the NANPA and the Numbering Oversight Working Group (NOWG). The NANPA serves under a contract with the FCC. The NOWG has compiled this data into an annual performance report for the FCC and the North American Numbering Council (NANC). NANPA’s rating for the 2016 performance year was determined by consensus of the NOWG to be Met. This rating is defined below: MET • Performance was competent and reliable • Decisions and recommendations were within requirements The Met rating was given to the NANPA for consistency in meeting all requirements which included the areas of Code Administration, NPA Relief Planning, NRUF, and Other NANP Resources. Throughout 2016, the NANPA personnel continued to exhibit their professionalism and expertise while performing NANPA duties and continued to perform at a consistent level of efficiency in the administration of all NANP resources. Highlights included: • Conducted one-on-one NRUF training and posted a “Getting Started” information sheet to assist newly authorized interconnected VoIP providers in applying for and obtaining NANP resources. • Completed a significant two-year project to reclaim 555 line numbers and implement the sunset of 555 NXX assignment guidelines. This project involved extensive communications with assignees, as well as close coordination with the FCC and the INC, resulting in the return/reclamation of 7,172 numbers. • Submitted and implemented change orders that resulted in several NANP Administration System (NAS) enhancements, which included adding a new service type for interconnected VoIP to the NRUF Form 502 and NRUF FTP file format, updating the NRUF form to a more current version of Excel™, and making modifications necessary to sunset the 555 resource. • Provided industry leadership and support in NPA Relief Planning, NRUF, Other NANP Resources, and Code Administration activities. 2016 NANPA Performance Evaluation Report June 29, 2017 Page 4 of 27 Section 1.0 Performance Review Methodology The annual NANPA Performance Evaluation Report is a summary of significant events that were accomplished during the 2016 performance year. The NOWG’s evaluation is based on its interactions with the NANPA and on information obtained about the NANPA’s performance through the following means: • 2016 NANPA Performance Survey • Monthly NOWG/NANPA status meetings • Annual operational review • NANPA NANC reports • Interaction with the industry and state regulators The following chart defines the rating categories that were used by the NOWG for the NANPA’s 2016 performance rating: Satisfaction Rating Used when the NANPA... MET • Performance was competent and reliable • Decisions and recommendations were within requirements NOT MET • Performance was unreliable and commitments were not met • Decisions and recommendations were inconsistent with requirements The NOWG will present the draft report to the FCC and the NANPA. The final report will be presented to the NANC for endorsement and then forwarded to the FCC. 2016 NANPA Performance Evaluation Report June 29, 2017 Page 5 of 27 Section 2.0 NANPA Reports In 2016, the NANPA provided reports on its numbering administration activities to the NANC and the NOWG. The reports were consistently completed and delivered when required. The NANPA also provided monthly reports to the FCC and made presentations at the March, June, September, and December NANC meetings. 2.1 Monthly Reports to the FCC As required per Sections 8.3, 8.4, and 8.5 of the NANPA Technical Requirements Document (TRD), the NANPA reported monthly to the FCC on activity regarding applications, assignments, and denials for Central Office (CO) Codes. The NANPA also reported on other numbering resources, the status of NPA relief activities, and the NANP Administration System (NAS) performance and availability. 2.2 NANC Reports In 2016, the NANPA provided monthly reports to the NANC and made presentations at NANC meetings on a variety of subject matters, which included the following: • 2016 versus 2015 CO Code and NPA Inventory Status • A summary of CO Code Activity • NPA and NANP Exhaust Forecasts and Status Reports • Trends in application activity, code assignments, and returns • NPA relief planning and implementation, new NPA assignments and NPAs placed into service, including non-geographic NPAs • Issues introduced by NANPA at INC • NRUF data collection and process improvements • Assistance and training for NRUF submission, in particular interconnected VoIP service providers • Resource reclamations for other NANP resources • Communications and outreach with the industry and state regulators • Assistance and training for NAS Users, in particular interconnected VoIP service providers • Change Orders affecting NAS, NRUF, and numbering assignments • Website additions and enhancements 2016 NANPA Performance Evaluation Report June 29, 2017 Page 6 of 27 2.3 NOWG Reports The NOWG and NANPA followed a standing agenda during scheduled monthly oversight meetings. The NANPA prepared monthly performance measurements, status reports, and trouble ticket tracking for the NOWG that were reviewed in-depth during the monthly meetings. NANPA also reported on a range of activities and services performed for the industry and regulators. The NANPA reported to the NOWG on items such as: • Assistance provided to state regulators • NAS enhancements and performance • Updates to the State Reclamation Contact List and Safety Valve Quick Sheet • Procedures for processing applications for interconnected VoIP service providers • Training for NANPA staff • Training for the industry on using NAS for NRUF submissions • General communications with industry and state regulators • Updates to the NANPA website • Updates to User Guides See Appendix A for 2016 NANPA / NOWG Standing Agenda 2016 NANPA Performance Evaluation Report June 29, 2017 Page 7 of 27 Section 3.0 Program Improvement Plan (PIP) The PIP is utilized for identifying and tracking program improvements. Highlights of the 2016 PIP included: • Worked with the Pooling Administrator to develop and post to the NANPA website a new information sheet, “Getting Started for Interconnected VoIP Providers” • Improved the “Getting Started for Interconnected VoIP Providers” information sheet to include additional information concerning NRUF reporting requirements, a link to the webpage providing state-specific information on 30-day filing requirements, and website layout changes for NRUF on the NANPA website • Modified NANPA website to include a new NRUF tool bar on the home page and a consolidation of NRUF information onto just two website pages, reducing redundancy and volume on the site • Conducted an NRUF Q&A session to assist service providers with filing their NRUF submissions • Updated the “Getting Started with CIC Assignment” document on the NANPA website with verbiage addressing the difference between CICs and OCNs See Appendix B for 2016 NANPA PIP Report 2016 NANPA Performance Evaluation Report June 29, 2017 Page 8 of 27 Section 4.0 2016 Performance Survey Results The annual 2016 NANPA Performance Survey, conducted by the NOWG to obtain feedback from the industry and state regulators regarding the NANPA’s performance, contained eight evaluation sections and a single “Comment” section at the end of the survey. The rating categories used were Met, Not Met, and N/A (Not Applicable). 4.1 Survey Ratings – Quantitative Analysis The NANPA 2016 Performance Survey was completed by a total of 76 respondents. The respondents were comprised of 47 Industry and Other respondents and 29 State Regulatory Commission respondents. Respondents provided a rating for one question in each section. Following are the aggregated response ratings: Section Met Not Met N/A CO Code (NXX) Administration 65 0 11 NPA Relief Planning 63 0 13 NRUF 69 0 7 Other NANP Resources 61 0 15 NANP Administration System (NAS) 67 0 9 NANPA Website and Reports 71 3 2 NANPA Industry Activities 48 0 28 Overall Assessment of the NANPA 76 0 0 See Appendix C for 2016 NANPA Numerical Survey Results and Bar Charts and Appendix D for 2016 NANPA Survey Cover Letter and Performance Survey 2016 NANPA Performance Evaluation Report June 29, 2017 Page 9 of 27 4.2 NANPA Survey Written Comments The comment section in the survey allowed respondents the opportunity to provide details regarding their satisfaction or dissatisfaction with NANPA’s performance in 2016. The NOWG reviewed all comments to determine if there was a common theme substantiated by multiple respondents. Following is a summary of written comments that were provided by survey respondents. The comments provided praise for individual staff members and expressed an appreciation for the quality of service provided by NANPA. The following adjectives and phrases were used by multiple respondents to describe their experiences in working with the NANPA staff: • Proactive, responsive • Provides excellent customer service • Efficient, well organized, professional • Knowledgeable, accurate The majority of comments were positive. Two of the comments indicated some dissatisfaction relating to the use of the online NRUF Form 502. There were also two comments suggesting improvements, such as a suggestion for a NAS-related enhancement to the content of the system-generated Part 4 emails. After thoroughly reviewing all of the comments received, the NOWG concluded that the written comments generally indicated a high level of satisfaction experienced by those who interacted with the NANPA. Samples of the written comments received are provided below: “NANPA provides excellent service and guidance when needed.” “NANPA Admins are very easy to work with and quickly have been able to point my team to specific guidelines that we have needed over the past year, I appreciate the professionalism and the courtesy they provide each time we have reached out. Thank you” “Communication was key the year of 2016 and NANPA nailed it! When I needed help or further explanation, I got it via email with a follow up phone call to make sure I understood it. Received references for additional information, useful suggestions and information on upcoming webinars. I would say NANPA exceeded expectation in 2016.” “When I ordered an NXX there was quite a bit interaction with NANPA and the support was very instrumental in effectively getting the NXX turned up and working properly.” See Appendix E for 2016 NANPA Survey Respondents and Respondents’ Comments 2016 NANPA Performance Evaluation Report June 29, 2017 Page 10 of 27 Section 5.0 Operational Review The NOWG members met with NANPA representatives in Sterling, Virginia on March 29-30, 2017 to conduct the annual operational review of the NANPA. During the operational review, there was a formal status presentation regarding: • 2016 Summary • NANP Administration System (NAS) • NPA Resource Administration • NPA Relief Planning • State NPA Relief Planning Activities • Numbering Resource Utilization and Forecast (NRUF) & Exhaust Projections • Central Office Code Administration • NANPA Enterprise Services • NANP Resource Administration • NANPA Forum Participation and Industry Liaison Activities The following sections summarize highlights of specific areas within the NANPA operational review. 5.1 2016 Summary The NANPA reviewed its organizational structure and personnel, as well as the current NANPA contract. The high-level summary included highlights such as: • NANPA change order approval and implementation • Sunset of the 555 line number assignment guidelines • NPA relief activities • Trends in CO code allocations The high-level summary also included details about the following items in the NANPA’s Program Improvement Plan (PIP): • Interconnected VoIP provider direct access to numbers • NRUF enhancements and training • Website enhancements 5.2 NANP Administration System (NAS) The NANPA presented the following 2016 highlights on NAS: • 1,240 NAS registered users, including 1,154 service provider or service provider consultant users, 37 federal and state regulatory users, and 49 other users • 2,647 mailing list recipients for NANPA Notifications, such as Planning Letters or Jeopardy Notifications • The Help Desk received nearly 1,050 emails and responded to more than 500 phone calls • NAS users continued to be able to submit resource applications, utilize various search/query capabilities, submit and access NRUF data, and manage NAS user profiles 2016 NANPA Performance Evaluation Report June 29, 2017 Page 11 of 27 • NAS users continued to be able to access various NANP notifications such as Planning Letters, newsletters, NPA relief activity information, rate center consolidations, and INC guideline changes • NAS continued to create the daily “NRUF on File” database utilized by both the NANPA and the PA, and also the daily list of OCNs associated with Federal Registration Numbers (FRNs) with delinquent payments that were subject to the FCC’s Red Light Rule • NAS continued to upload PAS-generated reports, such as the report to identify OCNs that had opted into pooling • NAS issued 181 NNS notifications, covering topics such as NPA relief planning and Planning Letters, NRUF, INC guideline changes, code administration, newsletters and non-geographic resources • NANPA opened and closed 2 NAS trouble tickets • NANPA implemented load balancer and firewall software upgrades, conducted its annual failover exercise, and implemented 10 NAS software builds • NANPA implemented a number of NAS enhancements, including several related to NRUF, a couple to remove two forms and unwanted characters in email notices, and to introduce a new non-geographic 5XX NPA • NAS met the 99.9% system availability requirement, having two instances of scheduled unavailability for a total of 23 minutes, and one instance of unscheduled unavailability for 20 minutes • NANPA implemented three change orders in 2016, all NAS-related 5.3 NPA Resource Administration The NANPA presented the following 2016 highlights on NPA resource administration: • 5 NPAs were assigned; 4 geographic NPAs and 1 non-geographic NPA • 7 NPAs were placed into service; 4 geographic NPAs in the United States, 2 geographic NPAs in Canada, and the 522 non-geographic NPA • In the last 10 years, there have been 55 geographic NPAs placed into service; 37 in the US and 18 outside the US • At year end, there were 414 NPAs assigned, with 392 of those in service and 22 awaiting implementation • At year end, there were 267 NPAs remaining unassigned, including 42 Easily Recognizable Codes (ERCs) and 225 General Purpose Codes (GPCs): – Of the 42 ERCs, 5 were reserved and 37 available – Of the 225 GPCs, 177 were reserved and 48 were available • A variety of NPA reports are available on the NANPA website, such as a list of NPAs introduced in the last 10 years, each NPA’s dialing plan, and the NPA database that provides the status of every NPA 5.4 NPA Relief Planning The NANPA presented the following 2016 highlights on NPA relief activities: • Initiated 5 new relief planning activities, and filed 5 relief petitions/applications with state regulators 2016 NANPA Performance Evaluation Report June 29, 2017 Page 12 of 27 • Conducted 21 industry relief meetings, including 6 initial NPA relief implementation meetings and 3 jeopardy related meetings • Shadowed 68 industry committee meetings for relief implementation spanning 11 NPAs across 8 states • Published 7 Planning Letters announcing new relief projects in 6 states • Distributed 104 NNS notifications on NPA relief planning activities, up from 2015 • Met all performance objectives for 48 tracked events associated with NPA relief planning • Responded to 510 voicemail/email contacts within one business day (100%) • Coordinated relief planning meetings and activities to avoid scheduling conflicts with other industry meetings • Updated two NPA relief related reports daily and posted them to the website • Advised state regulatory personnel on the status of NPAs within their states, such as NPA triggers being met, NPAs exhausting within 36 months, and coordinating industry input on various relief options • Tracked performance measurements and key relief planning activities and reported on those monthly to the NOWG • Conducted a quality survey for the 4 initial relief planning meetings and received an average score of 4.97 on a scale of 1.00-5.00 with 5.00 being the highest (up from 2015) • Conducted a quality survey for one non-relief planning conference call and received an average score of 4.75 on a scale of 1.00-5.00 with 5.00 being the highest • Outlined the variety of NPA relief planning information and reports that are available on the public website • Responded to 13 state regulatory activities, including attending 9 public meetings in California in person, and 2 public hearings in Texas and Washington via conference bridge • Tracked the status of various NPA relief activities throughout the year, such as 5 new relief decisions, 15 suspended relief decisions or implementation dates to be determined across 11 states, and 9 relief petitions pending decisions • Met guidelines requirements with 100% on-time completion of key relief planning tasks • Demonstrated a continued high level of satisfaction with NANPA’s facilitation of relief planning meetings, and identified and implemented relief planning process improvements 5.5 State NPA Relief Planning Activities The NANPA presented a comprehensive summary of the 2016 relief planning activities in 22 states where activity took place, a relief decision was pending, or some trigger for action is set in the future. The NANPA also identified potential relief implementation activities expected in 2017. 2016 NANPA Performance Evaluation Report June 29, 2017 Page 13 of 27 5.6 Numbering Resource Utilization and Forecast (NRUF) & Exhaust Projections The NANPA presented the following 2016 highlights on NRUF reporting, measurements and education: • Met all 5 processing metrics at 100% – Form 502s processed and confirmation notifications sent, with any errors identified, within 7 calendar days – Missing utilization notices sent within 45 days – Anomalous notifications sent within 90 days – Phone calls/emails responded to within one business day – Job Aid updates completed 60 days prior to the submission deadline • Processed 13,334 Form 502 submissions – About 57% submitted via email, approximately 31% submitted online in NAS, and 11% submitted via secure FTP – 82% were filed in January/February and July/August timeframes • Outlined the various NRUF queries and reports available to users within NAS • Provided numerous NRUF notifications to NRUF reporting carriers, including multiple “tips of the day” • Responded to more than 1,400 NRUF-related inquiries • Conducted one-on-one training sessions with newly authorized interconnected VoIP providers and updated the website’s “Getting Started” document to provide additional information about NRUF reporting requirements • Distributed an NNS notice to provide clarification on intermediate numbers as outlined in FCC Order 15-70 • Distributed 8 NNS notices about the implementation of NAS NRUF enhancements (Change Order 3) and the new June 2016 NRUF Form 502 • Conducted an informal NRUF question and answer session in July for service providers to call into the conference bridge to ask questions or just listen to the discussion; participants also could submit questions via email • Conducted NRUF refresher training in December where 72 individuals representing 64 service providers attended • Updated the NRUF on-line user guide and the NRUF job aid documents, and posted those to the NANPA website • Included NRUF-related articles in the NANPA’s quarterly newsletters regarding Change Order 3 and how to forecast in just one NPA in an overlay complex The NANPA presented the following 2016 highlights on NPA exhaust projections: • Produced and published NPA and NANP exhaust projections in April and October, using essentially the same model as previous years • Produced and published 3 delta NRUFs (updated exhaust projections) for 8 NPAs between semi-annual NPA exhaust publications • Outlined the NPA exhaust projection inputs and the methodology used • Produced and published separate exhaust projections for the non-geographic 5XX resources 2016 NANPA Performance Evaluation Report June 29, 2017 Page 14 of 27 5.7 Central Office Code Administration The NANPA presented the following 2016 highlights on CO code administration: • Processed 10,272 applications in 2016, down from 2015 – 1% submitted directly to NANPA, and 99% submitted through the PA – 31% of those submitted through the PA were initial applications and 69% were growth, compared to 20% initial and 80% growth in 2015 • Made 3,405 total and 3,184 net code assignments • Denied 683 code applications due to reasons such as codes returned containing ported numbers or blocks, code holders failing to first disconnect LRNs and/or ports or blocks, applicant’s FRN appearing on the Red Light List or no NRUF on file, and invalid switching identification • Processed 221 code disconnects and 153 cancellations • Suspended 181 applications for reasons such as returned codes containing ported numbers or, at the applicant’s request, searches for existing codes to transfer in lieu of opening a new code for LRN purposes • Processed 100% of applications within the 7 calendar day requirement, and returned 100% of calls within one business day • Made one code assignment that resulted in a code reject, but made no code assignments that resulted in a code conflict • Initiated reclamation of codes within 6 months of the effective date 100% of the time; sent 651 Part 4 reminder notices and 161 Part 4 delinquent notices • Made no processing errors on 2,217 Part 4 submittals • Denied 43 applications due to the FCC’s Red Light Rule • Worked with service providers to recover 226 codes across 14 states that were previously protected or no longer in use • Identified 160 codes as abandoned, and worked with regulators in 9 states to recover or transfer those codes • Initiated reclamation on one code for a state that has not exercised its delegated authority over reclamation • Worked with 22 states on 92 codes appearing on state delinquent lists • Resolved discrepancies on over 400 codes by proactively sending 378 reminders to service providers of the requirements to build new codes in industry databases, and to complete work necessary to transfer or return codes • Distributed 3 NNS notices concerning rate center consolidations, and 9 NNS notices concerning INC guidelines changes and jeopardy declarations • Initiated 11 INC issues in 2016; 7 were resolved and implemented, 3 were resolved and will be implemented in 2017, and 1 remained open at year-end • Conducted code administrator training to review and revise internal methods and procedures on a variety of topics, such as supporting documentation requirements and implementation of INC issue resolutions • Continued to provide guidance to service providers in understanding application, code activation and reclamation processes • Outlined the various CO code information, queries and reports available on the website, and also the various CO code queries and reports available to NAS users within NAS 2016 NANPA Performance Evaluation Report June 29, 2017 Page 15 of 27 • Provided Part 1 and Part 3 reports to authorized state regulatory personnel on a daily, weekly or monthly basis, as specified by the recipients 5.8 NANPA Enterprise Services The NANPA reviewed the operating principles for the 5 enterprise services provided: • Fees must be fair, reasonable and the same for all customers for the same service • Proprietary/confidential information must be protected • Services must be performed without jeopardizing NANPA duties and responsibilities • Fees collected are independent of the NANPA contract The NANPA contract requires NANPA to provide AOCN services for a fee. In addition to the AOCN services provided, NANPA continued to stand ready to provide four other Enterprise Services, although none were requested in 2016. Those include the entry of paper submissions of resource applications, entry of paper NRUF submissions, NANPA testimony in state regulatory hearings, and customized reports for publicly available resource assignment data. Finally, the NANPA technical requirements indicate that any and all approved Enterprise Services are subject to an audit by an independent auditor. NANPA will contract with an auditor to audit all enterprise service offerings during the first, third and fifth year of the term of the contact. The NANPA’s most recent audit was completed in September 2015, covering the 2013-2014 time period. The next audit is targeted for completion in the third quarter of 2017, covering the 2015-2016 time period. 5.9 NANP Resource Administration The NANPA presented the following 2016 highlights on NANP resource administration: • Carrier Identification Codes (CICs) – 9 FGB CICs returned/reclaimed and no assignments made, for a total of 251 FGB CICs assigned at year-end – In the last 10 years, only 11 FGB CICs have been assigned, but 168 have been returned/reclaimed in that same period – 33 FGD CICs returned/reclaimed and 23 assigned, for a total of 1,963 FGD CICs assigned at year-end – In the last 10 years, more FGD CICs have been returned/reclaimed than have been assigned; more than 7,800 FGD CICs are available – Exhaust is not a concern for either FGB or FGD CICs – NANPA identified 133 “abandoned” CICs in 2016, where companies failed to return CICs no longer needed or to notify NANPA of new contact information or legal name changes – Issues concerning CICs include entities potentially ordering access with OCNs instead of CICs, and access providers providing inaccurate or invalid data to NANPA regarding access and/or CIC usage – Outlined the various CIC reports available on the website 2016 NANPA Performance Evaluation Report June 29, 2017 Page 16 of 27 • Non-Geographic 5XX-NXX codes – 777 net assignments for a total of 4,976 codes assigned at year-end – In the last 10 years, a net total of 4,433 codes have been assigned – Assignment of 522-NXX codes began in August 2016; at year-end, the 522 NPA was projected to exhaust in 2017 – The next 5XX NPA to be assigned will be 521, the first non-ERC NPA to be used for non-geographic resources – Outlined the various 5XX reports available on the website and within NAS • 9YY-NXX codes – 5 codes returned and no assignments made, for a total of 55 codes assigned at year-end (only 20 are assigned to US entities) – In the last 10 years, 22 codes have been assigned, but 93 have been returned/reclaimed – With 698 codes available for assignment and little demand, exhaust is not a concern • 555 Line Numbers – Continued a significant effort to reclaim assigned 555 line numbers not in service – Submitted to the INC a detailed proposal for the future of the 555 NXX resource, which ultimately led to the sunset the 555 NXX Assignment Guidelines and the creation a 555 NXX reference document – Change Order 5 submitted, approved, and implemented to modify NAS to remove the ability for a NAS user to access the 555 Part A and Part C forms, and to remove the 555 reports – Published a Planning Letter announcing the sunset of the 555 NXX Assignment Guidelines • Other NANP Resources – No 800-855 line numbers were assigned or returned/reclaimed – No 456-NXXs were assigned and 1 code was returned, for a total of 2 codes remaining assigned at year-end – No assignments of vertical service codes, N11 codes or ANI II digits – Outlined the various NANP resource reports available on the website 5.10 NANPA Forum Participation and Industry Liaison Activities The NANPA presented the following 2016 highlights on its forum participation and industry liaison activities: • Provided monthly reports to the NANC membership and made presentations at NANC meetings on topics such as NPA assignment and relief activities, central office codes and other NANP resources activity, change orders, interconnected VoIP provider assistance and education, and NAS performance • Continued to manage and maintain the NANC Chair website • Provided monthly reports to and participated in monthly meetings with the NOWG to cover such topics as the PIP, interconnected VoIP provider assistance and education, the Monthly Operational Report (MOR), NAS performance and enhancements, and NANPA Change Orders 2016 NANPA Performance Evaluation Report June 29, 2017 Page 17 of 27 • In the monthly reviews with the NOWG, covered other items such as the review of the annual report, and the NANPA’s quarterly newsletter publications • Attended NANC’s Future of Numbering Working Group meetings to provide input as needed on NANPA operational capabilities and processes • Interacted with the FCC on such items as monthly reports, NAS maintenance, the VoIP direct access order, 555 NXX line number reclamation, NANPA staff changes and various FCC inquiries • Met with state regulators on a one-on-one basis as needed to discuss state- specific NPA relief planning and exhaust projections, CO code assignment activities, and NRUF reporting • Met with state regulators bi-monthly to review NPA relief activities, CO code assignment trends, INC issues, and other numbering-related activities • Actively participated at INC, submitting 11 new issues and multiple contributions covering such topics as code transfers for LRN purposes, updates to several guidelines regarding supporting documentation submitted with applications, and clarifications for interconnected VoIP providers’ applications • Provided periodic updates to INC on CIC reclamations and NANP resources, and provided real-time technical guidance during issue discussion to note impacts to NAS and NANPA’s administrative roles and responsibilities • Distributed six NNS notifications regarding INC guideline changes • In quarterly neutrality audits conducted by Ernst & Young, the auditor found no issues with NANPA’s processing of central office code and CIC applications or central office code reclamation activities • Published the 2015 Annual Report in March 2016, and updated the Binder of Decisional Principles in December 2016 • Published quarterly NANPA newsletters covering topics such as NANP resource updates, NPA relief planning activities, NRUF volumes, NANPA change order approval and implementation, CIC reporting requirements, sunset of the 555 NXX Assignment Guidelines, and other informational items • Made various updates to the NANPA website, such as: – Posted two new informational documents for interconnected VoIP providers interested in obtaining CO codes, thousands-blocks or CICs – Updated the CIC “Getting Started” document – Created a new NRUF tool bar on the home page and consolidated NRUF information to just two webpages – Added approved NANPA Change Orders to the website – Posted the updated June 2016 NRUF Form 502 and NRUF FTP file format – Posted interim and final jeopardy procedures for two NPAs • Made other updates to the website to keep information current on items such as the NANP country contacts, the List of Codes Requiring Special Handling, NRUF Common Errors & Fixes, user guides and job aids, and area code maps • NANPA Feedback items received averaged 11 per month, down slightly from 2015 • Received two email complaint submissions in 2016 but neither were due to NANPA’s performance, just inquiries on how to stop unwanted calls 2016 NANPA Performance Evaluation Report June 29, 2017 Page 18 of 27 5.11 Summary of the NANPA Operational Review Based on the information shared with the NOWG during the 2016 operational review, the NANPA not only maintained a high level of performance, but continued to proactively find ways to improve processes, NAS, and customer satisfaction. The NANPA was proactive in making information, assistance and training readily available to interconnected VoIP providers. NANPA’s operational review presentation was very thorough in providing details of the NANPA activities in 2016. The NANPA staff effectively responded to NOWG inquiries during presentations, demonstrating their knowledge and expertise. See Appendix F for 2016 NANPA Operational Review Presentation and Appendix G for 2016 NANPA Highlights 2016 NANPA Performance Evaluation Report June 29, 2017 Page 19 of 27 Section 6.0 NANP Administration System (NAS) NAS is the database used by service providers, service provider consultants, regulators, and NANPA in the assignment of NANP resources, NRUF, and various administration aspects of the NANP. NAS provides an automated system for processing numbering resource applications, collecting numbering resource utilization and forecast data, and issuing notifications to the industry on numbering matters such as area code relief and rate center consolidations. NAS provided system functionality in the following areas to support the assignment and administration of NANP resources: • Central Office Codes • Non-Geographic 5XX Resources • Other Numbering Resources • NANP Notification System • NRUF • NAS Reports • NAS User Registration Capabilities of NAS include: • Ability to submit resource applications (e.g. CO code, NPA, CIC, 5XX/9YY/456 NXX, 800-855 line number) • Ability to submit CO code applications via the Pooling Administration System (NAS-PAS interface) • Queries to search, view and print CO code forms; 5YY, 5XX, 9YY and 456 NPA forms; CIC forms; 800-855 forms • Ability for users to submit NRUF FCC Form 502 data on-line • Distribution of NANP Notifications (e.g., INC guideline changes, NRUF, Planning Letters, Newsletters, NPA relief planning, Jeopardy, rate center consolidations, geographic and non-geographic notices) • Interface to allow management of NAS user profile (e.g., add/delete OCNs, change NAS resource subscriptions, modify notices received, update contact information, reset password) • Create daily NRUF on File database which is also uploaded into PAS • Obtain FRNs delinquent in payments to the FCC and compare with NRUF data to identify OCNs on Red Light list and then upload that list into PAS • Upload PAS-generated reports which include mandatory pooling rate centers and OCNs opting in pooling In 2016, the NANPA provided excellent support and maintenance of NAS, meeting the FCC requirement for NAS system availability of 99.9%, which excluded scheduled maintenance. There were two instances of NAS scheduled unavailability to perform failover testing between Sterling VA and Charlotte NC. There was one instance of unscheduled 2016 NANPA Performance Evaluation Report June 29, 2017 Page 20 of 27 unavailability totaling of 20 minutes when NAS was switched over to Charlotte to complete a firewall software update and a DNS error kept traffic pointed to the Sterling NAS. In 2016, the following system enhancements were made to NAS: • Created a minor error notification when a user includes a non-geographic NPA on the F3-b form on the geographic NRUF Form 502 (spreadsheet and FTP) • Added column headings to the NPA database file that is posted on the NANPA public website • Expanded the size of the on-line electronic signature field on the NRUF Form 502 Rural Certification form • Added NPA 522 as the next non-geographic NPA • Allow the character “/” to appear in a rate center abbreviation • Creation of a NAS rejection notification when a user submits a blank Form 502 spreadsheet • Removal of trouble reporting contact information from the 5XX and 9YY Part C form (in response to INC Issue 803) • Permit NRUF drop-down menus in NAS containing OCNs to be listed in numerical order • Removal of unwanted characters appearing in email notices There were ten NAS software builds in 2016 to implement the system enhancements and maintenance activity. Two NAS trouble tickets were opened and closed in 2016: • One ticket involved a user’s inability to add a rate center forecast using the NRUF on-line capability. A software build was deployed to resolve this issue and the ticket was open for 20 days. • The second ticket dealt with the submission of a 5XX NPA Part C and was closed the same day. The NANPA maintained the NAS Trouble Ticket Log that is reviewed at the monthly NANPA/NOWG meetings. This log provides a description of the issue, actions taken to address the trouble ticket, and the opened and closed date of the ticket. See Appendix H for 2016 NAS Trouble Ticket Log 2016 NANPA Performance Evaluation Report June 29, 2017 Page 21 of 27 Section 7.0 Change Orders During 2016, NANPA submitted three change orders to the FCC (Change Orders 4, 5, and 6) and implemented three change orders (Change Orders 3, 4, and 5). For each of the change orders submitted, the NOWG prepared a Change Order Recommendation, recommending all of them for FCC approval. The three change orders implemented in 2016 included two changes orders that had been submitted in 2016, as well as one change order (Change Order 3) that was submitted and pending at the end of 2015. NANPA Change Order 3: NANP Administration System (NAS) NRUF Updates This change order, submitted in September 2015, was approved by the FCC and implemented by NANPA in 2016. Change Order 3 proposed updates to the NAS NRUF process in response to the FCC Order 15-70 concerning direct access to numbers by interconnected VoIP providers. The updates to NAS were to enable interconnected VoIP providers to accurately identify themselves when submitting utilization and forecast data in their NRUF report filings. Additional modifications to NAS were also included in this change order to update the NRUF Form 502 to a more current version of Excel™, and to clarify terms used on the form. NANPA Change Order 4: INC Issue 497: Identify Changes to INC Guidelines Based on NANC’s Report and Recommendation, VoIP Service Providers’ Access Requirements for NANP Resource Assignments (July 19, 2005), and FCC Order 15-70 (June 22, 2015), and INC Issue 797: Updates to the INC Guidelines Forms This change order, submitted in January 2016, was approved by the FCC and implemented by NANPA in 2016. Change Order 4 proposed modifications to the NANP Administration System (NAS) to incorporate several form changes adopted by the Industry Numbering Committee (INC) and reflected in the Numbering Guidelines. Form changes had been made by the INC to enable Interconnected VoIP Providers to request numbering resources directly from the Numbering Administrators. These changes included updates to numbering resource application forms for Central Office Codes, Carrier Identification Codes (CICs), the 5XX NPA, and the 9YY NPA. Additional changes were also made to modify various forms for consistent formatting and to update outdated information. NANPA Change Order 5: INC Issue 788: 555 Line Number Assignments and Reclamation This change order, submitted in August 2016, was approved by the FCC and implemented in 2016. Change Order 5 proposed modifications to NAS to implement changes being made to the use and management of the 555 line number resource and to support the sunset of the INC’s 555 NXX Number Assignment Guidelines. The modifications included removing the ability for a NAS registered user to access the 555 application (555 Part A) and in-service 2016 NANPA Performance Evaluation Report June 29, 2017 Page 22 of 27 certification (555 Part C) forms as well as the Search 555 Forms function. Additionally, NANPA published a Planning Letter announcing the sunset of the 555 NXX Assignment Guidelines and the 555 resource, and posted the 555 NXX Reference Document to the NANPA website. NANPA Change Order 6: NANP Administration System (NAS) to the Cloud This change order was submitted to the FCC and approved in September 2016, but it is not scheduled for implementation until 2017. Change Order 6 proposed moving the NANP Administration System (NAS) to a cloud platform, Amazon Web Service (AWS). Since the current NAS hardware is nearing the end of its life, moving NAS to the cloud should preclude the need to replace the old equipment and invest in new technology. Also, use of a cloud platform is expected to significantly improve the system’s scalability, provide the reliability of geographically discrete instances, and improve recovery time in a disaster scenario. The Change Order Log, which is maintained by the NOWG, shows a history of change orders submitted by NANPA under the current contract. See Appendix I for the 2016 NANPA Change Order Log 2016 NANPA Performance Evaluation Report June 29, 2017 Page 23 of 27 Section 8.0 NANPA Website The website maintained by the NANPA provides information relating to numbering resources and NPA relief planning for use by service providers, regulatory agencies, and the general public. In 2016, updates made to the NANPA website included: • “Getting Started for Interconnected VoIP Providers” document to assist interconnected VoIP providers interested in obtaining thousands-blocks and central office codes directly from NANPA and the PA • Information sheet with specific information concerning NRUF reporting requirements • New CIC document that provides directions for interconnected VoIP providers when applying for a CIC • “Getting Started with CIC Assignment” document on the NANPA website with verbiage addressing the difference between CICs and OCNs • New NRUF tool bar on the home page and a consolidation of NRUF information onto just two website pages, reducing redundancy and volume on the site • Approved NANPA Change Orders added to the website (under Publications) • Interim and final jeopardy procedures posted for NY 518 and PA 717 NPAs Updated the following documents on the website: • NANP country contacts and area code maps for Ohio, North Carolina, New York and Indiana • List of Codes that Require Special Handling on NANPA website (under Reports, Central Office Codes) • June 2016 NRUF Form 502, and NRUF FTP file format, to reflect the “interconnected VoIP” service type • “NRUF Common Error & Fixes” document • Binder of Decisional Principles The feedback received from the 2016 performance survey included the following comment regarding the NANPA website: “Receive updates primarily regarding new area codes, splits, and overlays. The few times I have needed to look up this information proactively, I found 2016 NANPA Performance Evaluation Report June 29, 2017 Page 24 of 27 that easy to do on the website. Keeps us out of trouble with our PBX clients so THANKS!” 2016 NANPA Performance Evaluation Report June 29, 2017 Page 25 of 27 Section 9.0 Conclusion and Recommendation The NOWG based its 2016 NANPA Performance Evaluation Rating on documentation, information collected, and observations throughout the review year. For the 2016 performance evaluation rating, the NOWG considered NANPA activities that included interaction with the NOWG and NANC, active participation at INC and other industry forums, the NANPA’s ongoing consistency in addressing PIP suggestions, the resolution of issues brought to the NANPA’s attention, and suggestions made by the NOWG throughout the calendar year. The 2016 survey results revealed a high level of satisfaction with the quality of NANPA’s work, and indicated that NANPA continued to consistently and effectively demonstrate their numbering expertise. The NOWG reviewed all aspects of the NANPA activities, which included feedback from the industry and state regulators, to determine the NANPA’s performance rating in 2016. As a result of this analysis, the NOWG gave the NANPA a “Met” rating. The NOWG makes the following recommendations for NANPA in 2017: • Provide details of the type of modifications being made to NAS in the notices sent to users when events are scheduled to implement software builds or other improvements, particularly when changes may impact FTP users or users with automated processes. • Continue to offer NRUF Training Session(s) via WebEx which would include navigation and functionality of the online NRUF Form 502. • Hold a NANPA website refresher training session via WebEx, or an unstructured NANPA website Q&A session. • Continue to proactively search for ways to improve processes, educate customers, and enhance system functionality, and document those efforts where NANPA has gone above and beyond to assist its customers. 2016 NANPA Performance Evaluation Report June 29, 2017 Page 26 of 27 Section 10.0 Acknowledgements & NOWG Participants The NOWG wishes to thank the following NANPA employees for their contributions throughout 2016, for assisting the NOWG during the annual operational review, and for participating in the NOWG’s monthly meetings: Bruce Armstrong Wayne Milby Al Cipparone Kimberly Miller Joe Cocke Beth Sprague Nancy Fears Brent Struthers Tom Foley LaShanda Tomlinson Jeremiah Jenkins Heidi Wayman John Manning NANPA Help Desk and Code Administrators The following NOWG members have participated in varying degrees by attending NOWG meetings throughout 2016, attending the annual operational review, and contributing to the development of this document: AT&T Linda Richardson CenturyLink Joy McConnell-Couch CenturyLink Jan Doell CenturyLink Phil Linse Cox Communications Beth O'Donnell Pennsylvania PUC Christopher Hepburn Sprint Rosemary Emmer Sprint Shaunna Forshee Sprint Karen Riepenkroger T-Mobile US Cathie Capita T-Mobile US Jennifer Pyn Verizon Dana Crandall Verizon Laura Dalton XO Communications Ruben Galvan The NOWG would also like to thank the following FCC staff members who assisted the NOWG throughout 2016 by answering questions, providing support, and participating in monthly NOWG/NANPA calls: Darlene Biddy Carmell Weathers Myrva Freeman Sanford Williams Marilyn Jones 2016 NANPA Performance Evaluation Report June 29, 2017 Page 27 of 27 Section 11.0 List of Appendices Appendix A 2016 NANPA / NOWG Standing Agenda Appendix B 2016 NANPA PIP Report Appendix C 2016 NANPA Numerical Survey Results and Bar Charts Appendix D 2016 NANPA Survey Cover Letter and Performance Survey Appendix E 2016 NANPA Survey Respondents and Respondents’ Comments Appendix F 2016 NANPA Operational Review Presentation Appendix G 2016 NANPA Highlights Appendix H 2016 NAS Trouble Ticket Log Appendix I 2016 NANPA Change Order Log