NANC 2016 PA Performance Evaluation Report Prepared by the Numbering Oversight Working Group (NOWG) June 29, 2017 2016 PA Performance Evaluation Report June 29, 2017 Page 2 of 24 Table of Contents Executive Summary .................................................................................... 3 Section 1.0 Performance Review Methodology ........................................... 4 Section 2.0 PA Reports ............................................................................... 5 Section 3.0 Program Improvement Plan (PIP) ............................................ 7 Section 4.0 Customer Focus ....................................................................... 8 Section 5.0 2016 PA Performance Survey Results ..................................... 9 Section 6.0 Operational Review ................................................................ 11 Section 7.0 Systems ................................................................................ 16 Section 8.0 Change Orders ...................................................................... 17 Section 9.0 Websites ................................................................................ 19 Section 10.0 p-ANI (pseudo-Automatic Number Identification) / RNA (Routing Number Administrator) ............................................................... 20 Section 11.0 Conclusion and Recommendation ....................................... 22 Section 12.0 Acknowledgements & NOWG Participants .......................... 23 Section 13.0 List of Appendices ............................................................... 24 2016 PA Performance Evaluation Report June 29, 2017 Page 3 of 24 Executive Summary The Pooling Administrator’s (PA) annual performance assessment is based upon a compilation of performance feedback surveys for the PA and the Routing Number Administrator (RNA), monthly standing agenda conference calls, the annual operational review, and observations/interactions between the PA and the Numbering Oversight Working Group (NOWG). The PA serves under a contract with the FCC. The NOWG has compiled this data into an annual performance report for the FCC and the North American Numbering Council (NANC). The PA’s rating for the 2016 performance year was determined by consensus of the NOWG to be Met. This rating is defined below: MET • Performance was competent and reliable • Decisions and recommendations were within requirements The Met rating was given to the PA for consistency in meeting and often exceeding all requirements related to Pooling Administration and p-ANI Administration. The survey results revealed a consistent level of satisfaction that respondents attributed to the responsiveness and expertise exhibited by the PA and RNA personnel throughout 2016. In 2016, the PA continued to consistently perform its required responsibilities. Highlights included: • Successfully moved RNAS (February 2016) and PAS (June 2016) to Amazon Web Services (AWS). • Processed 123,629 PAS applications and 5,710 RNAS applications. • PAS and RNAS both exceeded the required metric of 99.9% availability in 2016. • Implementation of the Direct Numbering Access Interconnected VoIP Order. • Educated interconnected VoIP providers on requirements for application processing, the proper supporting documentation, and on the information needed in 30-day notification letters. 2016 PA Performance Evaluation Report June 29, 2017 Page 4 of 24 Section 1.0 Performance Review Methodology The annual PA Performance Evaluation Report is a summary of significant events that were accomplished during the 2016 performance year. The NOWG’s evaluation is based on its interactions with the PA and on information obtained about the PA’s performance through the following means: • 2016 PA and RNA Performance Surveys • Monthly NOWG/PA status meetings • Annual operational review • PA NANC reports • Interaction with the industry and state regulators The following chart defines the rating categories that were used by the NOWG for the PA’s 2016 performance rating: Satisfaction Rating Used when the PA... MET • Performance was competent and reliable • Decisions and recommendations were within requirements NOT MET • Performance was unreliable and commitments were not met • Decisions and recommendations were inconsistent with requirements The NOWG will present the draft report to the FCC and the PA. The final report will be presented to the NANC for endorsement and then forwarded to the FCC. 2016 PA Performance Evaluation Report June 29, 2017 Page 5 of 24 Section 2.0 PA Reports 2.1 PA Annual Report The annual report prepared by the PA is a requirement in the Pooling Administrator Technical Requirements document, and the status of pooling and p-ANI administration should be reported in the annual report. At a minimum, the annual report is required to contain the following information: • Brief description of the PA and the Routing Number Administrator (RNA) • Highlights/significant milestones reached during the previous year • Identification of existing and potential pooling areas • Aggregated total, by pool, of the service providers participating in the pooled areas • Forecast results, as well as a review of forecasts vs. actual past block activations • System and performance metrics • Status of required transferable property • Industry issue identification/feedback • Volume of reports produced for entities such as: FCC, state regulatory agencies, NANC, NANPA, and service providers Part of the NOWG’s annual performance review process is to review the annual report. The PA provided the NOWG with an opportunity to review the draft copy of the 2016 Annual Report. During the operational review virtual meeting on March 9-10, 2017, the PA staff reviewed with the NOWG the 2016 highlights which were also included in the annual report. Overall, the annual report provides a comprehensive snapshot of the PA and RNA activities for 2016. The PA 2016 Annual Report was filed with the FCC and is posted for general availability on the PA’s website at www.nationalpooling.com. 2.2 PA NANC Report The PA reported its monthly numbering activities to the NANC and the NOWG. Additionally, the PA made presentations at the 2016 NANC meetings. The PA reported the status of thousands-block pooling administration and p-ANI administration, and events affecting the performance of the PA and RNA. This included the following: • Volume of pooling assignments, donations/disconnects, and applications processed • Codes opened to replenish pools and establish Location Routing Numbers (LRNs) • Pools with less than six months inventory vs. forecasts • Summaries of monthly reports to the FCC • Number of blocks reclaimed • Interconnected VoIP activities • Percent availability of PAS and RNAS 2016 PA Performance Evaluation Report June 29, 2017 Page 6 of 24 • Status and implementation of change orders • Updates to PAS and RNAS • Updates to the PA and RNA websites • p-ANI summary 2.3 NOWG Monthly Reports Throughout 2016, the NOWG and PA followed a standing agenda during the scheduled monthly calls. The PA provided monthly performance reports that were reviewed during the monthly calls with the NOWG. The quality and content of these reports provided the NOWG with valuable insight into the operations of the PA and RNA. Some of the standing agenda topics included: • Program Improvement Plan (PIP) • Thousands-block and p-ANI application processing metrics • Rate center pooling status • Customer focus items • Change Orders See Appendix A for 2016 PA / NOWG Standing Agenda 2016 PA Performance Evaluation Report June 29, 2017 Page 7 of 24 Section 3.0 Program Improvement Plan (PIP) In June 2016, the NOWG recommended that PA create a PIP that will be utilized for identifying and tracking program improvements and improvements in the user experience. Highlights of the 2016 PIP included: • Ongoing internal training to ensure consistency when responding to service providers and regulators • Training videos for p-ANI applicants • Training videos for interconnected VoIP providers See Appendix B for 2016 PA PIP Report 2016 PA Performance Evaluation Report June 29, 2017 Page 8 of 24 Section 4.0 Customer Focus At the monthly NOWG/PA meetings, the PA provided information on Pooling and p-ANI customer focus items that they executed to help service providers, interconnected VoIP providers, regulators, FCC, and other industry users of PAS and RNAS. Customer focus items covered both contractual and non-contractual initiatives related to customer service. There were 164 customer focus items for Pooling and p-ANI in 2016. Customer focus items included, but were not limited to, the following: • Provided special reports for service providers, regulators, interconnected VoIP providers, FCC, and industry users of PAS and RNAS • Provided pooling education, and research for existing and new service providers, regulators, and interconnected VoIP providers • Provided p-ANI education and assistance for existing and new p-ANI users • Provided service provider, regulator, and interconnected VoIP providers training which included individual and website training The PA and the RNA often went “above and beyond” in assisting service providers, regulators, interconnected VoIP providers, FCC, and industry users as needed with these customer focus initiatives. 2016 PA Performance Evaluation Report June 29, 2017 Page 9 of 24 Section 5.0 2016 PA Performance Survey Results The annual 2016 PA Performance Survey, conducted by the NOWG to obtain feedback from the industry and state regulators regarding the PA’s performance, contained six evaluation sections and a single ‘Comment’ section at the end of the survey. The rating categories used were Met, Not Met, and N/A (Not Applicable). 5.1 PA Survey Ratings – Quantitative Analysis The PA 2016 Performance Survey was completed by a total of 75 respondents. The respondents were comprised of 43 Industry and Other respondents and 32 State Regulatory Commission respondents. Respondents provided a rating for one question in each section. Following are the aggregated response ratings: Section Met Not Met N/A Pooling Administrator (PA) 72 0 3 Pooling Administration System (PAS) 68 0 7 PA Website 70 0 5 Miscellaneous PA Functions 68 0 7 PA Industry Activities 49 0 26 Overall Assessment of the PA 75 0 0 See Appendix C for 2016 PA Survey Metrics and Bar Charts, and Appendix D for 2016 PA Survey Cover Letter and Performance Survey 5.2 PA Survey Written Comments The comment section in the survey allowed respondents the opportunity to provide details regarding their satisfaction or dissatisfaction with the PA’s performance in 2016. All of the comments were positive, with only a few containing suggestions for areas of improvement. The NOWG reviewed all comments to determine if there was a common theme substantiated by multiple respondents. Following is a summary of written comments that were provided by survey respondents. The comments reflected appreciation for the quality of service provided by the PA and praise for individual staff members. The following adjectives and phrases were used by multiple respondents to describe their experiences in working with the PA staff: 2016 PA Performance Evaluation Report June 29, 2017 Page 10 of 24 • Professional, knowledgeable, friendly • Willing to help, conscientious, positive • Pleasant to work with, responsive, supportive • Problem solvers, proactive, patient Comments suggesting improvements included: • PAS enhancements, such as providing additional information on the subject line of Part 4 Reminder/Delinquent email notices, and the ability to upload supporting documentation one time when submitting initial requests for rate centers that share supporting documentation. • More advanced notice of PAS changes being implemented that impact automated processes of service providers. • More thorough review of Part 1A supporting documentation prior to the issuance of a Part 3 Denial. After thoroughly reviewing all of the comments received, the NOWG concluded that the written comments were not indicative of any performance issues, and generally indicated a high level of satisfaction experienced by those who interacted with the PA. Samples of the written comments received are provided below: “Being new to numbering, all personnel have been extremely helpful and very patient with me. It is very much appreciated by myself and this Commission.” “Continued excellent service from all areas of the organization. Especially appreciated have been the continuous updates and input on the entry of interconnected VoIP carriers to the process, as well as assistance from the Concord reps in coordinating special requests and requirements.” “Always helpful in numbering issues and number conservation matters. Easy place to go to solve problems.” “When I called, they were very helpful with education and sent a follow up email with additional information they thought I might need. I needed that information and am grateful for the amazing service I received.” “All of the PA's I have dealt with have always been more than willing to assist me in any issue or questions I may have had for them.” See Appendix E for 2016 PA Survey Respondents and Respondents’ Comments 2016 PA Performance Evaluation Report June 29, 2017 Page 11 of 24 Section 6.0 Operational Review The NOWG members met with the PA representatives in a virtual meeting on March 9- 10, 2017 to conduct the annual operational review. During this review, the PA staff provided presentations on the following 2016 activities: • 2016 Summary of Accomplishments • Pooling Administration Service Center • VoIP • External Relations • Industry Forum Participation • Quality Assurance and Implementation Management • Reports • Regulatory and Compliance • Amazon Web Services (AWS) and Operational Improvements • Technical Operations The presentation also included the Routing Numbering Administrator (RNA) operations and this activity is covered in Section 9.0. 6.1 2016 Summary of Accomplishments The PA presented a high-level summary of its 2016 activities which included highlights of applications for blocks processed, regulatory activities, special projects, and migration of PAS and RNAS to the Cloud (Amazon Web Services). 6.2 Pooling Administration Service Center The PA presented the following highlights on the Pooling Administration Service Center: • Total Applications Processed in 2016: – Issued 123,629 Part 3s – Assigned 45,978 thousands-blocks – Opened 3,394 CO Codes – Processed 99.999% of applications within seven calendar days or less • Customer Support Desk: – Received 875 calls – 100% of calls were answered within one business day – Managed new PAS user registrations, profile updates, and password resets • Issued quarterly pooling tips, including a supplemental tip, on the following topics: – Part 4 Obligations – Completing the Months to Exhaust and Utilization Certification Worksheet – TN Level for Growth Requests – Using the Back Arrow Button on your Internet Browser While in PAS 2016 PA Performance Evaluation Report June 29, 2017 Page 12 of 24 – Block Preference – Pooling in an NPA that is in Jeopardy • Conducted internal training which included: – M&P reviews and revisions – Review of guidelines based on revisions from INC – VoIP training – Quarterly Pooling Tips • Requests for voluntary disconnects/donations included: – For excluded rate centers made optional, the PA requested disconnects/donations for 49 rate centers and received 24 disconnects/donations – At the request of SPs, the PA requested disconnects/donations for 15 rate centers and received disconnects/donations in 11 rate centers • PA reported the following on mass modifications: – Processed 81 mass modify spreadsheets from 8 different SPs modifying 22,794 blocks in PAS – Processed 16 spreadsheets from NANPA to update 2,400 codes in PAS – Processed 340 spreadsheets from 5 different SPs updating 26,393 records of forecast data in PAS • PA Block Reclamation: – Reclaimed 4 blocks in 2016 • Overdue Part 4 Project: – Contacted 5 states with a total of 138 old overdue Part 4s – 38 of the old overdue Part 4s were received and approved and 5 other blocks were disconnected – A total of 43 old overdue Part 4s were resolved • PA continued to work on resolving abandoned codes/blocks: – Disconnected 161 abandoned blocks in BIRRDS – Sent out 145 abandoned code/block emails to request a new code and/or block holder, and found new block holders for 61 blocks and new code holders for 68 codes – The resolved abandoned codes/blocks involved 6 service providers in 6 states • Trouble Tickets: – Opened 8 and closed 10 trouble tickets – Two of the closed tickets were carried over from 2015 2016 PA Performance Evaluation Report June 29, 2017 Page 13 of 24 6.3 VoIP • A “Getting started for interconnected VoIP providers” quick sheet was developed by the PA and NANPA and posted on the PA website in February. Quick sheet included: – VoIP direct access authorization order information – 30-Day Notice to State instructions – Information to assist with applying for numbering resources after the 30- Day Notice – Website links for further support documents • Processed 5,279 Part 3s for VoIP providers (about 4% of total Part 3s) • Ongoing education for VoIP providers on application processing, proper supporting documentation, and on the information needed in 30-day notification letters • Provided the following support to states: – Held informational conference call in February to review provisions of VoIP order, 30-day notification process, and the documentation required by VoIP providers to obtain numbers – Created a VoIP Provider 30-day notification state regulatory contact sheet and posted it to the PA website – Provided updates to the states when new applications or filings were made and when the first application was submitted in the respective state – Held an additional conference call in September with the states on 30-day notifications and general process questions 6.4 External Relations The PA presented the following highlights on external relations: • Participated in monthly meetings with the NOWG providing updates on PA and RNA activities which included block and p-ANI information, trouble tickets, and change orders • Submitted three new change orders in 2016 • Implemented two change orders from 2015 and one from 2016 • There were 164 significant customer focus items for Pooling and p-ANI which included providing special reports, training, education, research, and assistance 6.5 Industry Forum Participation The PA presented the following highlights on industry forum participation: • Participated in various industry forums which included: – INC ? the PA submitted 14 new issues and 15 new contributions related to Pooling Administration, and three new issues and four new contributions related to p-ANI Administration – CIGRR – ATIS Testbed Focus Group 2016 PA Performance Evaluation Report June 29, 2017 Page 14 of 24 – LNPA WG – ESIF 6.6 Quality Assurance and Implementation Management The PA presented the following highlights on quality assurance and implementation management: • Reviewed the 2015 Census data that became available in 2016 and there was one change in the top 100 MSAs with all relevant rate centers added to the mandatory pooling category with this change • Data management, which includes quality control, on-going auditing, rate center designations, and rate center consolidations in PAS • Ongoing PAS and RNAS testing for all builds • Prepared pooling information reports for the NANPA for NPA relief planning and implementation meetings 6.7 Reports The PA presented the following highlights on reports: • PA provided annual, semi-annual, quarterly and monthly reports on Pooling and p-ANI Administration to the FCC and NOWG • PA provided ad hoc reports to service providers, state regulators, and FCC 6.8 Regulatory and Compliance The PA presented the following highlights on regulatory and compliance: • Held six state regulatory update conference calls in 2016, which included updates on pooling, p-ANI administration, the VoIP direct access order, and RNAS and PAS Amazon Web Services (AWS) implementation • Provided status reports for four NANC meetings in 2016 and also monthly performance reports to NANC members • Provided 676 external reports 6.9 Amazon Web Services (AWS) The PA presented the following highlights on AWS: • Benefits of migrating PAS and RNAS to AWS include infrastructure as a service, auto scaling, and quick testing and prototyping • RNAS was migrated to AWS in February with 55 minutes of scheduled downtime • PAS was migrated to AWS in June with 4 hours 44 minutes of scheduled downtime (copying the PAS database was the majority of this downtime) 6.10 Technical Operations The PA presented the following highlights on technical operations: 2016 PA Performance Evaluation Report June 29, 2017 Page 15 of 24 • PAS was available for use 99.995% of the time and RNAS was available for use 99.997% of the time, both exceeding their respective performance metrics of 99.9% • PAS used 4 hours and 44 minutes of scheduled downtime and RNAS used 55 minutes of scheduled downtime in 2016 • There were two instances of PAS unscheduled downtime totaling 25 minutes and 16 seconds, and one instance of RNAS unscheduled downtime of 15 minutes • There were eight PAS maintenance events and seven RNAS maintenance events in 2016; none of these resulted in PAS or RNAS customers experiencing any downtime See Appendix F for 2016 PA Operational Review Presentation and Appendix G for 2016 PA Highlights 2016 PA Performance Evaluation Report June 29, 2017 Page 16 of 24 Section 7.0 Systems 7.1 Pooling Administration System (PAS) Pooling Administration System (PAS) migrated to Amazon Web Services (AWS) in June, 2016. The PA used four hours and 44 minutes out of six hours of requested downtime to migrate to AWS. PAS was available 99.995% of the time. There were eight software builds in 2016 which covered maintenance and trouble tickets. PAS was unavailable for a total of 25 minutes and 16 seconds due to two instances of unscheduled down time. Overall, the industry appeared to be satisfied with the performance of PAS in 2016. 7.2 Routing Number Administration System (RNAS) Routing Number Administration System (RNAS) migrated to Amazon Web Services (AWS) in February, 2016. A total of 55 minutes of downtime was used to migrate RNAS to AWS. RNAS was available 99.997% of the time. There were seven maintenance events for RNAS in 2016. RNAS was unavailable for a total of 15 minutes due to one instance of unscheduled downtime. Overall, the industry appeared to be satisfied with the performance of RNAS in 2016. See Appendix H for 2016 PAS and RNAS Trouble Ticket Log 2016 PA Performance Evaluation Report June 29, 2017 Page 17 of 24 Section 8.0 Change Orders In 2016, the PA submitted three new change orders to the FCC (Change Orders # 3, 3A Revised, and 3B). These change orders related to the transition of the Number Portability Administration Center (NPAC) from Neustar to iconectiv and the interaction between the PA and the NPAC. The NOWG prepared a change order recommendation for each of the new PA change orders. Additionally, the PA implemented three change orders in 2016, which included Change Order # 3A Revised, and also Change Orders # 1 and 2 which had been submitted and approved in 2015. Below is a summary of each new PA change order submitted in 2016: PA Change Order 3: Addressing the iconectiv Proposals for Connection to the PAS This change order, submitted in March 2016, proposed functional changes to PAS to provide the necessary connectivity and access for iconectiv as the new Local Number Portability Administrator (LNPA). It addressed the PA’s support for and during the NPAC transition, and included the enhancement of the existing API for more security, flexibility and resiliency. It also proposed an update to the existing PAS application code to support two NPAC vendors during the LNPA transition, as well as other functions. The NOWG analyzed the change order, provided a series of follow-up questions to the PA, and discussed the PA’s responses to those questions. Following its change order review, the NOWG did not recommend that the FCC approve this change order. The NOWG suggested that the FCC work with the PA to significantly reduce the associated costs, or separate the change order into multiple change orders. PA Change Order 3A Revised: Specifications for API This change order, submitted in June 2016, addressed only the PAS API specifications for the NPAC transition. It proposed Neustar’s development and delivery of specifications for the API to enable iconectiv to access PAS during and after the LNPA transition. Specifically, this change order proposed PAS modifications in security, flexibility, and data transmission requirements needed to accommodate multiple?vendor access to PAS, and addressed specifications for how PAS will be accessed over the Internet. The NOWG recommended that the FCC approve this change order. This change order was approved in August, and was implemented in September 2016. 2016 PA Performance Evaluation Report June 29, 2017 Page 18 of 24 PA Change Order 3B: Development and Support of the PAS NPAC API This change order, submitted in October 2016, addressed the following: • Development of a PAS API interface to allow iconectiv secure access to PAS over the Internet • Enhancement of the PAS GUI to enable both NPAC vendors to access PAS during the transition by region • Integrated performance testing, regional cutover testing, and/or regional fallback testing with iconectiv • Support through the NPAC regional migrations The NOWG analyzed this change order, provided a series of follow-up questions to the PA, and discussed the PA’s responses to those questions. Following its change order review, the NOWG did not recommend that the FCC approve this change order because the NOWG did not believe it had enough detailed information to determine if the level of effort described in the change order was commensurate with the proposed cost. However, the NOWG suggested that the FCC work with the PA to significantly reduce the costs associated with this change order. The Change Order Log, which is maintained by the NOWG, shows a history of change orders submitted by the PA under the current contract. See Appendix I for 2016 PA Change Order Log 2016 PA Performance Evaluation Report June 29, 2017 Page 19 of 24 Section 9.0 Websites Section 9.1 National Pooling Website The website maintained by the PA provides number pooling information to service providers and regulatory agencies. In 2016, the PA continued to keep the information current on the website. In 2016, the PA created and posted to the website two new documents to assist interconnected VoIP providers in applying for resources: • “Getting started for interconnected VoIP providers” • “VoIP Provider 30-day notification state regulatory contact sheet” Overall, the industry appeared satisfied with the website. Section 9.2 RNA Website The website maintained by the RNA provides p-ANI information to service providers, PSAPs, 9-1-1 systems providers, and governmental/regulatory agencies. In 2016, the RNA continued to keep the information current on the website. There were nine p-ANI training videos created and posted in September 2016: • Create/Modify p-ANI Forecast • New p-ANI Requests • P-ANI Modification Requests • FCC License Search • Helpful Tools in RNAS • Types of Reports in RNAS • Filing p-ANI Annual Report in RNAS • Filing p-ANI Annual Report in Excel • Release of Enhancement to PAS for Regulatory Users All training videos can be downloaded from the RNA website. The p-ANI videos received 46 views by year-end in 2016. Overall, the industry appeared satisfied with the website. 2016 PA Performance Evaluation Report June 29, 2017 Page 20 of 24 Section 10.0 p-ANI (pseudo-Automatic Number Identification) / RNA (Routing Number Administrator) The PA, as the Routing Number Administrator (RNA), administers p-ANI e911 resources. The PA’s RNA is responsible for processing p-ANI applications, carriers’ annual reports, and semi-annual forecasts. Its functions in 2016 included, but were not limited to, the following: • Processed 5,710 applications (Part 3s issued), with 100% processed on time • Assigned 2,708 new p-ANI ranges • Modified 30 existing p-ANI ranges • Processed 2,894 returned p-ANI ranges • Maintained RNAS and the p-ANI website • Continued to participate in the ATIS Emergency Services Interconnection Forum (ESIF) • Prepared and published its annual p-ANI Activity and Projected Exhaust Report • Issued p-ANI Quarterly Tips to registered RNAS Users The RNA provided customer support by helping carriers locate the correct documentation needed to obtain p-ANI resources. They also continued to resolve duplicate p-ANI assignment issues by working with carriers to reconcile p-ANI ranges that had been assigned but were found to already be in use by another carrier. Based on the results of the 2016 RNA Performance Survey, with the exception of one respondent, the industry appeared satisfied with the PA’s RNA performance. Following is a sample of written comments received from the 2016 RNA survey: “I've only had positive experiences with the RNA. Very dependable. Easy website to use, always dependable, responsive and helpful.” “On every occasion, the Neustar Team met and exceeded our expectations with their speedy and helpful assistance. Despite the fact that their SLA for post requested pANI assignment was 5 days, they typically turned the request around within 24hours, which greatly assisted our Team in being able to move forward with PSAP deployments.” See Appendix J for 2016 RNA Survey Respondents and Survey Respondents’ Comments 2016 PA Performance Evaluation Report June 29, 2017 Page 21 of 24 10.1 2016 RNA Performance Survey Results – Quantitative Analysis The annual 2016 RNA Performance Survey, conducted by the NOWG to obtain feedback from the industry and state regulators regarding the RNA’s performance, contained five evaluation sections and a single ‘Comment’ section at the end of the survey. The rating categories used were Met, Not Met, and N/A (Not Applicable). Since the RNA is part of the PA, the following quantitative results were included in the PA Performance Review for 2016. The RNA 2016 Performance Survey was completed by a total of 14 respondents. The respondents were comprised of 12 Industry and Other respondents and 2 State Regulatory Commission respondents. Respondents provided a rating for one question in each section. Following are the aggregated response ratings: Section Met Not Met N/A Routing Number Administrator (RNA) 9 1 4 Routing Number Administration System (RNAS) 8 1 5 RNA Website 9 1 4 RNA Industry Activities 6 1 7 Overall Assessment of the RNA 12 1 1 See Appendix K for 2016 RNA Survey Metrics and Bar Charts and Appendix L for 2016 RNA Survey Cover Letter and Performance Survey 2016 PA Performance Evaluation Report June 29, 2017 Page 22 of 24 Section 11.0 Conclusion and Recommendation The NOWG based its 2016 PA Performance Evaluation Rating on documentation, information collected, and observations throughout the review period year. The NOWG also considered the PA’s interactions with the NOWG and NANC, active participation at INC and other industry forums, the PA’s ongoing consistency in addressing and resolving issues brought to their attention, and suggestions made by the NOWG throughout the calendar year. The 2016 survey results revealed a high level of client satisfaction with the PA’s and RNA’s performance of their duties, and the continued professionalism and expertise exhibited by the PA and RNA personnel. The PA continued to demonstrate their ability to handle a large volume of block applications while going above and beyond to assist and educate interconnected VoIP providers that were granted authority in 2016 to obtain numbering resources directly from the PA. The NOWG reviewed all aspects of the PA activities, as well as the feedback from service providers and regulators, and determined that the PA consistently performed very well in all areas of their responsibilities in 2016. As a result of the analysis, the NOWG gave the PA a “Met” rating for the 2016 performance year. The NOWG makes the following recommendations for the PA in 2017: • Continue to have internal training sessions with the PA and RNA personnel to ensure consistency in understanding and communicating processes when responding to service providers and regulators. • Provide details of the type of modifications being made to PAS and RNAS in the notices sent to users when events are scheduled to implement software builds or other improvements, particularly when changes may impact FTP users or users with automated processes. • Review the PA and RNA performance survey comments for any possible future enhancements to PAS or RNAS and consider adding them to the list of possible future enhancements; provide the comprehensive list to the NOWG for review. • Continue to proactively search for ways to improve processes, educate customers and enhance system functionality. 2016 PA Performance Evaluation Report June 29, 2017 Page 23 of 24 Section 12.0 Acknowledgements & NOWG Participants The NOWG wishes to thank the following Neustar PA personnel for their contributions throughout 2016, for assisting the NOWG during the annual operational review, and for participating in the NOWG’s monthly meetings: Bruce Armstrong Wayne Louie Jesse Armstrong Cecilia McCabe Tara Farquhar Amy Putnam Dara Flowers Shannon Sevigny Evelyn Freeman Florence Weber Kevin Gatchell Gary Zahn Linda Hymans PA Help Desk and Pooling Administrators Jeremiah Jenkins p-ANI Help Desk and p-ANI Administrators The following NOWG members have participated in varying degrees by attending NOWG meetings throughout 2016, attending the annual operational review, and contributing to the development of this document: AT&T Linda Richardson CenturyLink CenturyLink Joy McConnell-Couch Jan Doell CenturyLink Phil Linse Cox Communications Beth O'Donnell Pennsylvania PUC Christopher Hepburn Sprint Shaunna Forshee Sprint Rosemary Leist Sprint Karen Riepenkroger T-Mobile US Cathie Capita T-Mobile US Jennifer Pyn Verizon Dana Crandall Verizon Laura Dalton XO Communications Ruben Galvan The NOWG would also like to thank the following FCC staff members who assisted the NOWG throughout 2016 by answering questions, providing support, and participating in monthly NOWG/PA calls: Darlene Biddy Carmell Weathers Myrva Freeman Sanford Williams Marilyn Jones 2016 PA Performance Evaluation Report June 29, 2017 Page 24 of 24 Section 13.0 List of Appendices Appendix A 2016 PA / NOWG Standing Agenda Appendix B 2016 PA Program Improvement Plan (PIP) Appendix C 2016 PA Survey Metrics and Bar Charts Appendix D 2016 PA Survey Cover Letter and Performance Survey Appendix E 2016 PA Survey Respondents and Respondents’ Comments Appendix F 2016 PA Operational Review Presentation Appendix G 2016 PA Highlights Appendix H 2016 PAS and RNAS Trouble Ticket Log Appendix I 2016 PA Change Order Log Appendix J 2016 RNA Survey Respondents and Respondents’ Comments Appendix K 2016 RNA Survey Metrics and Bar Charts Appendix L 2016 RNA Survey Cover Letter and Performance Survey