NANC 2015 PA Performance Evaluation Report Prepared by the Numbering Oversight Working Group (NOWG) June 30, 2016 2015 PA Performance Evaluation Report June 30, 2016 Page 2 of 23 Table of Contents Executive Summary...........................................................................3 Section 1.0 Performance Review Methodology ..............................4 Section 2.0 PA Reports ..................................................................5 Section 3.0 Customer Focus ..........................................................7 Section 4.0 2015 PA Performance Survey Results.........................8 Section 5.0 Operational Review ...................................................10 Section 6.0 Pooling Administration System (PAS)........................15 Section 7.0 Change Orders ..........................................................16 Section 8.0 National Pooling Website...........................................18 Section 9.0 p-ANI (pseudo-Automatic Number Identification) / RNA (Routing Number Administrator) ......................................................19 Section 10.0 Conclusion and Recommendation................................21 Section 11.0 Acknowledgements & NOWG Participants ...............22 Section 12.0 List of Appendices ....................................................23 2015 PA Performance Evaluation Report June 30, 2016 Page 3 of 23 Executive Summary The Pooling Administrator’s (PA) annual performance assessment is based upon a compilation of performance feedback surveys for the PA and the Routing Number Administrator (RNA), monthly standing agenda conference calls, the annual operational review, and observations/interactions between the PA and the Numbering Oversight Working Group (NOWG). The PA serves under a contract with the FCC. The NOWG has compiled this data into an annual performance report for the FCC and the North American Numbering Council (NANC). The PA’s rating for the 2015 performance year was determined by consensus of the NOWG to be Met. This rating is defined below: MET ? Performance was competent and reliable ? Decisions and recommendations were within requirements The Met rating was given to the PA for consistency in meeting and often exceeding all requirements related to Pooling Administration and p-ANI Administration. As in previous years, the survey results revealed a consistent level of satisfaction that respondents attributed to the responsiveness and expertise exhibited by the PA and RNA personnel throughout 2015. In 2015, the PA continued to consistently perform its required responsibilities. Highlights included: • Successfully completed a total overhaul of the PAS system on January 11, 2015, with no major impacts to customers. • Processed a record total of 145,828 Pooling and 35,518 p-ANI Part 3s. One hundred percent (100%) of Part 3 responses were sent on time. • PAS and RNAS both exceeded the required metric of 99.9% availability in 2015. • Completed numerous special projects, including 2 MSA updates, Methods and Procedures (M&Ps) updates, and completed a manual p-ANI “reservation” process in all 305 NPAs per a request from the National Emergency Number Association (NENA). 2015 PA Performance Evaluation Report June 30, 2016 Page 4 of 23 Section 1.0 Performance Review Methodology The annual PA Performance Evaluation is a summary of significant events that were accomplished during the 2015 performance year. In addition to the annual performance review survey process, the NOWG’s interactions with the PA included the following: ? Monthly NOWG/PA status meetings ? Annual operational review ? Change Order review process ? PA NANC reports ? Interaction with the industry and state regulators The annual 2015 PA and RNA Performance Surveys, conducted by the NOWG to obtain feedback from the industry and state regulators regarding the PA’s and RNA’s performance, contained six evaluation sections for the PA Survey and five evaluation sections for the RNA Survey, along with a single ‘Comment’ section at the end of each survey. The rating categories used were Met or Not Met. The following chart defines the rating categories: Satisfaction Rating Used when the PA... MET ? Performance was competent and reliable ? Decisions and recommendations were within requirements NOT MET ? Performance was unreliable and commitments were not met ? Decisions and recommendations were inconsistent with requirements The NOWG also used the Met or Not Met categories for the PA’s 2015 overall performance rating. The NOWG will present the draft report to the FCC and the PA. The final report will be presented to the NANC for endorsement and then forwarded to the FCC. 2015 PA Performance Evaluation Report June 30, 2016 Page 5 of 23 Section 2.0 PA Reports 2.1 PA Annual Report The annual report prepared by the PA is a requirement in the Pooling Administrator Technical Requirements document, and the status of pooling and p-ANI administration should be reported in the annual report. At a minimum, the annual report is required to contain the following information: ? Brief description of the PA and the Routing Number Administrator (RNA) ? Highlights/significant milestones reached during the previous year ? Identification of existing and potential pooling areas ? Aggregated total, by pool, of the service providers participating in the pooled areas ? Forecast results, as well as a review of forecasts vs. actual past block activations ? System and performance metrics ? Status of required transferable property ? Industry issue identification/feedback ? Volume of reports produced aggregated by regulatory agencies, NANC, NANPA, and service providers Part of the NOWG’s annual performance review process is to review the annual report. The PA provided the NOWG with an opportunity to review the draft copy of the 2015 Annual Report. During the on-site operational review in Concord, California on April 20- 21, 2016, the PA staff reviewed with the NOWG the 2015 highlights which were also included in the annual report. Overall, the annual report provides a comprehensive snapshot of the PA and RNA activities for 2015. The PA 2015 Annual Report was filed with the FCC and is posted for general availability on the PA’s website at www.nationalpooling.com. 2015 PA Performance Evaluation Report June 30, 2016 Page 6 of 23 2.2 PA NANC Report The PA reported its monthly numbering activities to the NANC and the NOWG. Additionally, the PA made presentations at the 2015 NANC meetings.1 The PA reported the status of thousands-block pooling administration and p-ANI administration, and events affecting the performance of the PA and RNA. This included the following: ? Volume of pooling assignments, donations, and applications processed ? Codes opened to replenish pools and establish Location Routing Numbers (LRNs) ? Pools with less than six months inventory vs. forecasts ? Summaries of monthly reports to the FCC ? Number of blocks reclaimed ? Percent availability of PAS and RNAS ? Status and implementation of change orders ? Updates to PAS ? Updates to the PA website ? p-ANI summary 2.3 NOWG Monthly Reports Throughout 2015, the NOWG and PA followed a standing agenda during the scheduled monthly calls. The PA provided monthly performance reports that were reviewed during the monthly calls with the NOWG. The quality and content of these reports provided the NOWG with valuable insight into the operations of the PA and RNA. Some of the standing agenda topics included: ? Thousands-block and p-ANI application processing metrics ? Rate center pooling status ? Customer focus items ? Change Orders See Appendix A for 2015 PA / NOWG Standing Agenda 1 The March meeting was cancelled but the report was distributed to the NANC members. 2015 PA Performance Evaluation Report June 30, 2016 Page 7 of 23 Section 3.0 Customer Focus At the monthly NOWG/PA meetings, the PA provided information on Pooling and p-ANI customer focus items that they executed to help service providers, interconnected VoIP providers, and regulators. Customer focus items covered both contractual and non- contractual initiatives related to customer service. There were 105 customer focus items for Pooling and p-ANI in 2015. Customer focus items included, but were not limited to, the following: ? Provided pooling education and research for existing and new service providers, regulators, and interconnected VoIP providers ? Provided p-ANI education and assistance for existing and new p-ANI users ? Provided special reports for service providers and regulators ? Provided service provider and regulator training which included individual and website training The PA and the RNA often went “above and beyond” in assisting service providers and regulators as needed with these customer focus initiatives. 2015 PA Performance Evaluation Report June 30, 2016 Page 8 of 23 Section 4.0 2015 PA Performance Survey Results 4.1 PA Survey Ratings – Quantitative Analysis The PA 2015 Performance Survey was completed by a total of 79 respondents. The respondents were comprised of 44 Industry and Other respondents and 35 State Regulatory Commission respondents. Respondents provided a rating for one question in each section. Following are the aggregated response ratings: Section Met Not Met N/A Pooling Administrator (PA) 78 0 1 Pooling Administration System (PAS) 71 0 8 PA Website 76 0 3 Miscellaneous PA Functions 73 1 5 PA Industry Activities 57 0 22 Overall Assessment of the PA 79 0 0 See Appendix B for 2015 PA Survey Metrics and Bar Charts, and Appendix C for 2015 PA Survey Cover Letter and Performance Survey 4.2 PA Survey Written Comments The comment section in the survey allowed respondents the opportunity to provide details regarding their satisfaction or dissatisfaction with the PA’s performance in 2015. The majority of comments were positive, with only a few containing suggestions for areas of improvement. The NOWG reviewed all comments to determine if there was a common theme substantiated by multiple respondents. Following is a summary of written comments that were provided by survey respondents. The comments provided praise for individual staff members and expressed an appreciation for the quality of service provided by the PA. The following adjectives and phrases were used by multiple respondents to describe their experiences in working with the PA staff: • Prompt, responsible, polite • Knowledgeable, patient, helpful • Courteous, eager to assist • Professional, informative, supportive 2015 PA Performance Evaluation Report June 30, 2016 Page 9 of 23 Comments suggesting improvements pertained to the following areas: • System issues that occurred when the new Enhanced PAS was released. • Suggestions for additional PAS changes, such an addition of a “Back Button” that had been used previously in the old version of PAS, and an enhancement to the Part 4 Reminder email notice. • Process improvement needed for the PA’s building of BCD records prior to issuing Part 3s. After thoroughly reviewing all of the comments received, the NOWG concluded that the written comments were not indicative of any consistent performance issues, and generally indicated a high level of satisfaction experienced by those who interacted with the PA. Samples of the written comments received are provided below: “I am able to get the numbering resources that I need in a timely fashion. Keep up the good work.” “The performance of the Pooling Administration was nothing but positive for me in the past year. I can't think of a time when I needed assistance and called and was not helped. I appreciate all of the work the Pooling Administrators have provided for the 2015 year.” “I would like to see user guides that provide step by step instruction. The videos are... ok but still leave a lot of gaps in the process. If this information is already in place, please let me know where to go to find it.” “PA always provides answers to my questions even if the information is within the instructional documents. PA always pulls the supporting content to answer my questions.” “All inquiries and requirements of the Commission Staff were met satisfactorily by the PA over the course of 2015. The service level excellence continues.” “Interactions with the PA have been very positive in all respects.” “PA personnel were always helpful and provided prompt responses to inquiries. They provided very good help in resolving number assignment issues.” “All experiences with the PAs have been nothing but stellar. They are all very helpful and if they don't readily know the answer, are more than willing to get the answer for you.” See Appendix D for 2015 PA Survey Respondents and Respondents’ Comments 2015 PA Performance Evaluation Report June 30, 2016 Page 10 of 23 Section 5.0 Operational Review The NOWG members met with the PA representatives in Concord, CA on April 20-21, 2016 to conduct the annual on-site operational review. During this review, the PA staff provided presentations on the following 2015 activities: ? 2015 Summary of Accomplishments ? Pooling Administration Service Center ? Preparing for VoIP ? External Relations ? Industry Forum Participation ? Quality Assurance and Implementation Management ? Reports ? Regulatory and Compliance ? PAS Rollout ? Amazon Web Services (AWS) ? Technical Operations The presentation also included the RNA operations and this activity is covered in Section 9.0. 5.1 2015 Summary of Accomplishments The PA presented a high-level summary of its 2015 activities which included highlights of applications for blocks processed, regulatory activities, special projects, and rollout of the new PAS. 5.2 Pooling Administration Service Center The PA presented the following highlights on the Pooling Administration Service Center: ? Total Applications Processed in 2015: – Issued 145,828 Part 3s, a new record for Part 3s processed since the start of Pooling – Assigned 42,723 thousands-blocks – Opened 3,716 CO Codes – Processed all within seven calendar days or less ? Customer Support Desk received 914 calls; 100% were answered within one business day ? Created three new training videos – Release of Enhancements to the PAS Training Session for Service Provider and Service Provider Consultant Users – Release of Enhancements to the PAS Training Session for Regulatory Users – Chrome Browser Release How it Affects PAS Drop Down Menus (this was a temporary training video) 2015 PA Performance Evaluation Report June 30, 2016 Page 11 of 23 ? Issued quarterly pooling tips on the following topics: – Effective date preference for requests for new blocks, block modifications, and block disconnects – Disconnect LRN records in BIRRDS on pooled code returns – To begin pooling in an excluded rate center – Designated point of contact when searching for a new block/code holder ? Conducted internal training which included: – M&P reviews and revisions – Review of guidelines based on revisions from INC – FCC Order 15-70, which allows interconnected VoIP providers direct access to numbering resources – Ongoing weekly meetings ? Requests for voluntary disconnects/donations included: – For excluded rate centers made optional, the PA requested disconnects/donations for 126 rate centers and received 88 disconnects/donations – At the request of SPs, the PA requested disconnects/donations for 26 rate centers and received disconnects/donations in 6 rate centers ? PA reported the following on mass modifications: – Processed 90 mass modify spreadsheets from four different SPs modifying 31,138 blocks in PAS – Processed 26 spreadsheets from NANPA Code mass modifications to update 3,277 codes in PAS – Processed 215 spreadsheets from four different SPs updating 16,142 records of forecast data in PAS ? Reclaimed 3 blocks in 2015 ? PA continued to work on resolving abandoned codes/blocks: – Disconnected 313 abandoned blocks in BIRRDS – Sent out 142 abandoned code/block emails to request a new code and/or block holder, and found new block holders for 84 blocks and new code holders for 70 codes ? Trouble Tickets: – Opened 32 and closed 30 trouble tickets – Two remaining tickets were still open at the end of 2015 2015 PA Performance Evaluation Report June 30, 2016 Page 12 of 23 5.3 Preparing for VoIP The PA presented the following highlights on preparing for interconnected VoIP providers to obtain resources directly from the PA: ? Examined the initial order (FCC 15-70) ? Reviewed and updated guidelines/forms and submitted those updates to INC ? Met with the FCC ? Prepared the M&Ps and updated processes ? Trained staff 5.4 External Relations The PA presented the following highlights on external relations: ? Participated in monthly meetings with the NOWG providing updates on PA and RNA activities which included block and p-ANI information, trouble tickets, and change orders ? Submitted two new change orders in 2015 and implemented a change order from 2014 in January 2015 ? There were 105 significant customer focus items for Pooling and p-ANI which included providing special reports, education, research and assistance 5.5 Industry Forum Participation The PA presented the following highlights on industry forum participation: ? Participated in various industry forums which included: – INC – the PA submitted two new issues and 13 new contributions related to Pooling Administration, and two new issues and five new contributions related to p-ANI Administration – CIGRR – the PA submitted one new issue and continued to offer support for pooling related issues – ATIS Testbed Focus Group – LNPA WG – ESIF 2015 PA Performance Evaluation Report June 30, 2016 Page 13 of 23 5.6 Quality Assurance and Implementation Management The PA presented the following highlights on quality assurance and implementation management: ? Review of the 2014 Census data that became available in 2015 to see if there were any changes in the top 100 MSAs; there were none ? Data management, which includes quality control, on-going auditing, rate center designations, and rate center consolidations in PAS ? PAS testing for the rollout of the new PAS in January 2015 ? Ongoing PAS testing for subsequent PAS releases, including three maintenance builds ? Prepared pooling information reports for the NANPA for NPA relief planning and implementation meetings 5.7 Reports The PA presented the following highlights on reports: ? PA provided annual, semi-annual, quarterly and monthly reports on Pooling and RNA to the FCC ? PA provided ad hoc reports to service providers, state regulators and the FCC 5.8 Regulatory and Compliance The PA presented the following highlights on regulatory and compliance: ? Conducted individual pooling educational sessions for three states ? Held five regulatory update conference calls in 2015, which included updates on pooling, p-ANI administration, and the VoIP direct access order ? Provided status reports for four NANC meetings in 2015 and also monthly performance reports to NANC members ? Provided 637 external reports which included 65 ad hoc reports and 334 reports for reclamation 5.9 PAS Rollout The PA presented the following highlights on the PAS Rollout: ? PAS went live on January 11, 2015 with minimal issues ? Training sessions were held in January prior to the new PAS going live 2015 PA Performance Evaluation Report June 30, 2016 Page 14 of 23 5.10 Amazon Web Services (AWS) The PA presented the following highlights on AWS: ? Benefits of migrating PAS and RNAS to AWS include infrastructure as a service, auto scaling, and quick testing and prototyping ? AWS meets FCC security rules and regulations ? AWS has three regions (separate geographical locations) in the US and within each region are many Availability Zones (AZ); an AZ is a separate data center ? The multiple hardware related outages in 2015 would not have occurred if AWS had been in place ? The PA plans to move RNAS and PAS to AWS in 2016 5.11 Technical Operations The PA presented the following highlights on technical operations: ? There were 496 PAS users, 70 RNAS active users and 139 inactive RNAS users; inactive RNAS users can quickly re-activate their accounts as needed2 ? PAS was available for use 99.98% of scheduled uptime and RNAS was available for use 99.99% of scheduled uptime, both exceeding their respective performance metrics of 99.9% ? Neither PAS nor RNAS used any scheduled downtime in 2015 ? There were four instances of PAS unscheduled downtime totaling 1 hour and 25 minutes, and one instance of RNAS unscheduled downtime of 30 minutes ? There were nine PAS maintenance events and seven RNAS maintenance events in 2015; neither PAS nor RNAS customers experienced any downtime during the scheduled maintenance ? On October 18, 2015, PAS and RNAS successfully completed disaster recovery testing See Appendix E for 2015 PA Operational Review Presentation and Appendix F for 2015 PA Highlights 2 Data as of 4/26/16 2015 PA Performance Evaluation Report June 30, 2016 Page 15 of 23 Section 6.0 Pooling Administration System (PAS) The new Pooling Administration System (PAS) was released on January 11, 2015. In 2015, PAS was available 99.98% of the time. There were nine maintenance events in 2015, of which three were maintenance builds for the new PAS. No FCC-approved scheduled down time was used for these maintenance builds. PAS was unavailable for a total of one hour 25 minutes due to four instances of unscheduled down time. Overall, the industry appeared to be satisfied with the performance of PAS in 2015. However, in the performance survey comments there were some comments regarding the functionality of the new PAS along with suggestions for improvement. 2015 PA Performance Evaluation Report June 30, 2016 Page 16 of 23 Section 7.0 Change Orders In 2015 under the current contract, the PA submitted two change orders to the FCC, and filed one letter in lieu of a change order. The PA also completed the implementation of a change order that was remaining from the previous contract. The NOWG prepared a change order recommendation for each of the PA change orders, recommending that they be approved by the FCC. PA Change Order 24 (Previous Contract): Enhancement of the FTP Interface with PAS This change order, which proposed enhancements to the FTP interface to allow it to more closely conform to the GUI experience, was approved by the FCC in 2012, and was partially implemented in July 2013. The remainder of this change order was implemented in January 2015 with the roll out of the new PAS. PA Letter in Lieu of Change Order: Possible Future Impact of FCC VoIP Direct Access Order In November 2015, the PA submitted a letter to notify the FCC of the possible future impact of the FCC’s VoIP Direct Access Order (FCC 15-70) upon services provided by the PA. Because of uncertainty regarding the effect of the Order on PA operations, the PA stated that they cannot accurately determine its impact at this time, but may come back to the FCC at a later time with a change order requesting additional personnel if they are unable to manage any increased workload with their existing staff. PA Change Order 1: Move RNAS and PAS into the Cloud This change order was submitted in November 2015 to propose moving the Routing Number Administration System (RNAS), followed by the Pooling Administration System (PAS), from their existing hardware configuration to a cloud platform (Amazon Web Services). Use of a cloud platform is expected to significantly improve the systems’ scalability, provide the reliability of geographically discrete instances, and improve recovery time in a true disaster scenario. The new cloud platform is also intended to provide at least three availability zones to mirror the redundancy and reliability of the two existing data centers. The FCC approved this change order in December 2015. 2015 PA Performance Evaluation Report June 30, 2016 Page 17 of 23 PA Change Order 2: INC Guideline Form Changes This change order was submitted in December 2015 in response to INC Issue 497: “VoIP Service Providers’ Access Requirements for NANP Resource Assignments” and INC Issue 797: “Updates to the INC Guideline Forms.” This change order proposed a variety of changes to the Pooling, p-ANI, and Central Office Code forms contained in PAS and RNAS, as reflected in the INC Guidelines, to conform to the FCC’s Order 15-70 allowing interconnected VoIP providers to request numbering resources directly from the numbering administrators. Additional changes were made to update other forms contained in PAS and RNAS for consistency and formatting. As of the end of 2015, this change order was pending FCC approval. The Change Order Log, which is maintained by the NOWG, shows a history of change orders submitted by the PA under the current contract. See Appendix G for 2015 PA Change Order Log 2015 PA Performance Evaluation Report June 30, 2016 Page 18 of 23 Section 8.0 National Pooling Website The website maintained by the PA provides number pooling information to service providers and regulatory agencies. In 2015, the PA continued to keep the information current on the website. There were three new training videos created in 2015: • Release of Enhancements to PAS for Service Provider and Consultant Users • Release of Enhancement to PAS for Regulatory Users • Chrome Browser Release -- How It Affects PAS Drop Down Menus (temporary video) The most popular video once again was “New to Pooling Quick Start”, with 200 views in 2015. All training videos can be downloaded from the PA website. Overall, the industry appeared satisfied with the website. 2015 PA Performance Evaluation Report June 30, 2016 Page 19 of 23 Section 9.0 p-ANI (pseudo-Automatic Number Identification) / RNA (Routing Number Administrator) The PA, in its third full year administering p-ANIs as the permanent Routing Number Administrator (RNA), continued its work effort to reconcile p-ANI data discrepancies. The RNA worked diligently to resolve these discrepancies found where no p-ANI user reported on a p-ANI range that the previous assignor reported as assigned, and where more than one carrier reported on the same p-ANI range or part of a p-ANI range. Of the 4,561 p-ANI ranges identified with data discrepancies in the initial reporting period, only 7 p-ANI ranges remained unresolved at the end of 2015. In addition to the data reconciliation work, the PA’s RNA functions in 2015 included, but were not limited to, the following: • Processed 32,518 applications (Part 3s issued), with 100% processed on time • Assigned 2,482 new p-ANI ranges • Modified 27,021 existing p-ANI ranges • Processed 2,901 returned p-ANI ranges • Maintained RNAS with 99.99% availability, as well as the p-ANI website • Continued to participate in the ATIS Emergency Services Interconnection Forum (ESIF) • Processed 80 annual p-ANI reports and 48 semi-annual forecasts • Completed and posted the annual p-ANI Activity and Projected Exhaust Report • Issued p-ANI Quarterly Tips to registered RNAS Users The RNA also completed a special project, at the request of the NENA i3 Architecture Working Group, to set aside 50 p-ANIs in each NPA for use in the Next Generation 9-1- 1 Legacy Selective Router Gateway. The RNA will also set aside the same p-ANI ranges in new NPAs when they are added to RNAS. The FCC approved this work. The industry appeared satisfied with the PA’s RNA performance, as indicated by the responses received in the 2015 RNA Performance Survey. Following is a sample of written comments received from the survey: “We are completely satisfied with the RNA's performance. The RNA team is always available to assist in resolving issues or in providing additional information on process questions. The RNA team consistently goes above and beyond to assist us and the entire industry, and we truly appreciate their efforts.” “I had no problems and had timely results when applying for new pANI's during 2015. Great service.” “Always very helpful and responsive. Never a problem.” See Appendix H for 2015 RNA Survey Respondents and Survey Respondents’ Comments 2015 PA Performance Evaluation Report June 30, 2016 Page 20 of 23 9.1 RNA Survey Ratings – Quantitative Analysis The following quantitative results were included in the PA Performance Review Methodology for 2015. The RNA 2015 Performance Survey was completed by a total of 12 respondents. The respondents were comprised of 10 Industry and Other respondents and 2 State Regulatory Commission respondents. Respondents provided a rating for one question in each section. Following are the aggregated response ratings: Section Met Not Met N/A Routing Number Administrator (RNA) 8 0 4 Routing Number Administration System (RNAS) 9 0 3 RNA Website 10 1 1 RNA Industry Activities 6 0 6 Overall Assessment of the RNA 11 0 1 See Appendix I for 2015 RNA Survey Metrics and Bar Charts and Appendix J for 2015 RNA Survey Cover Letter and Performance Survey 2015 PA Performance Evaluation Report June 30, 2016 Page 21 of 23 Section 10.0 Conclusion and Recommendation The NOWG based its 2015 PA Performance Evaluation Rating on documentation, information collected, and observations throughout the review period year. The NOWG also considered the PA’s interactions with the NOWG and NANC, active participation at INC and other industry forums, the PA’s ongoing consistency in addressing and resolving issues brought to their attention, and suggestions made by the NOWG throughout the calendar year. The 2015 survey results revealed a high level of client satisfaction with the PA’s and RNA’s performance of their duties, and the continued professionalism and expertise exhibited by the PA and RNA personnel. The PA continued to demonstrate their ability to handle the large volume of block applications, while simultaneously completing special projects. The NOWG reviewed all aspects of the PA activities, as well as the feedback from service providers and regulators, and determined that the PA consistently performed very well in all areas of their responsibilities in 2015. As a result of the analysis, the NOWG gave the PA a “Met” rating for the 2015 performance year. The NOWG makes the following recommendations for the PA’s consideration in 2016: • Continue to have internal training sessions with the PA and RNA personnel to ensure consistency in understanding and communicating processes when responding to service providers and regulators. • Create a Program Improvement Plan (PIP) document to capture and track performance improvements suggested by the NOWG. This will be a “living” document discussed with the NOWG during monthly joint meetings throughout 2016. • Create training videos for p-ANI applicants, such as a video similar to the PA’s “New to Pooling Quick Start” video, or a video on how to locate FCC license information. Consider creating additional training videos that would assist p-ANI applicants/assignees as needed. • Continue to provide high quality service to the industry. 2015 PA Performance Evaluation Report June 30, 2016 Page 22 of 23 Section 11.0 Acknowledgements & NOWG Participants The NOWG wishes to thank the following Neustar PA personnel for their contributions throughout the year, for assisting the NOWG during the annual operational review, and for participating in the NOWG’s monthly meetings: Bruce Armstrong Wayne Louie Jesse Armstrong Cecilia McCabe Jan Connally Amy Putnam Tara Farquhar Bill Reidway Dara Flowers Shannon Sevigny Evelyn Freeman Florence Weber Kevin Gatchell Gary Zahn Linda Hymans PA Help Desk and Pooling Administrators Jeremiah Jenkins p-ANI Help Desk and p-ANI Administrators The following NOWG members have participated in varying degrees by attending NOWG meetings throughout the year, attending the annual operational review, and contributing to the development of this document: AT&T Linda Richardson CenturyLink Jan Doell Cox Communications Beth O'Donnell Pennsylvania PUC Christopher Hepburn Sprint Rosemary Emmer Sprint Shaunna Forshee Sprint Karen Riepenkroger T-Mobile USA Cathie Capita Verizon Dana Crandall Verizon Laura Dalton XO Communications Ruben Galvan The NOWG would also like to thank FCC staff members who assisted the NOWG throughout the year by answering questions and providing support. 2015 PA Performance Evaluation Report June 30, 2016 Page 23 of 23 Section 12.0 List of Appendices Appendix A 2015 PA / NOWG Standing Agenda Appendix B 2015 PA Survey Metrics and Bar Charts Appendix C 2015 PA Survey Cover Letter and Performance Survey Appendix D 2015 PA Survey Respondents and Respondents’ Comments Appendix E 2015 PA Operational Review Presentation Appendix F 2015 PA Highlights Appendix G 2015 PA Change Order Log Appendix H 2015 RNA Survey Respondents and Respondents’ Comments Appendix I 2015 RNA Survey Metrics and Bar Charts Appendix J 2015 RNA Survey Cover Letter and Performance Survey