*Pages 1--9 from þÿ* 1 Attachment 2 Ex Parte Letter to Marlene H. Dortch Attachment April 3, 2003 Page 1 of 6 THE CABS UNE- P CONVERSION SBC has already discussed in general terms the relationship between the Carrier Access Billing System (“ CABS”) UNE- P conversion in the fall of 2001 and the database reconciliation project that took place in January 2003. 1 In response to further questions from the FCC Staff, SBC provides here a more detailed explanation of the CABS UNE- P conversion itself and why it resulted in certain inaccuracies in the CABS database that SBC ultimately had to correct through the database reconciliation. The CABS UNE- P Conversion Consisted of Two Phases The CABS UNE- P conversion was a complicated effort, requiring extensive planning and resources to execute. SBC planners decided that the best approach was to implement a two- phase approach: Phase 1 would consist of converting the embedded base of UNE- P circuits from the Resale Billing System (“ RBS”) to the CABS billing system. During this Phase, it was necessary to put a freeze on posting all incoming UNE- P order activity (i. e., either new UNE- P circuits or changes to existing UNE- P circuits) to either billing system. Incoming order activity during this Phase was “held” for posting to the CABS billing system until after the embedded base was completely converted. 2 Phase 1 was expected to last approximately 4 weeks. Phase 2 would involve the processing of the UNE- P order activity held during Phase 1. SBC designed and developed tools that were expected to convert most of these “held” orders mechanically. 3 Phase 1 of the CABS UNE- P Conversion Took Longer Than Expected Although SBC estimated that the initial phase of the conversion would last approximately 4 weeks, it actually took nearly three months to complete. There were two principal reasons why SBC’s actual experience was so different from its original expectations: First, SBC planned the CABS UNE- P conversion in early 2001, at a time when UNE- P volumes were still relatively low. For example, in January 2001, SBC had approximately 1 See Flynn Aff. ¶ 6 n. 4 (App. A, Tab 12); Brown/ Cottrell/ Flynn Joint Reply Aff. ¶ 17 & n. 14 (Reply App., Tab 3); Ex Parte Letter from Geoffrey M. Klineberg, Kellogg, Huber, Hansen, Todd & Evans, P. L. L. C., to Marlene H. Dortch, FCC (Mar. 14, 2003), Attach. B, at 2- 4; Ex Parte Letter from Geoffrey M. Klineberg, Kellogg, Huber, Hansen, Todd & Evans, P. L. L. C., to Marlene H. Dortch, FCC (Mar. 28, 2003) (“ March 28 Ex Parte”), Attach. D, at 1- 2. 2 See Brown/ Cottrell/ Flynn Joint Reply Aff. ¶ 19; March 28 Ex Parte, Attach. D, at 1- 2. 3 March 28 Ex Parte, Attach. D, at 1- 2. 3 Ex Parte Letter to Marlene H. Dortch Attachment April 3, 2003 Page 2 of 6 33,000 UNE- P circuits in service throughout the entire Midwest region. When the conversion finally began in August 2001, that number had grown to nearly 450,000. Although SBC had certainly anticipated some increase in UNE- P circuits, the actual growth in 2001 far outstripped its expectations. Second, SBC experienced problems with the creation of billing account numbers (“ BANs”) for some CLECs prior to the planned conversion. This delayed SBC’s ability to convert certain circuits, thereby requiring significant rescheduling. Both of these factors caused the conversion of the embedded base to take longer than SBC had anticipated. As a consequence, the need to hold the processing of new UNE- P billing service order activity by CABS while the conversion was taking place led to a far greater number of “held” service orders than SBC had originally expected. 4 There Were Problems With the Conversion of the Embedded Base In order to create the CABS UNE- P billing process and to carry out the conversion of the embedded base of UNE- P accounts to CABS, SBC Midwest enhanced an existing tool called RoboTask. 5 Unfortunately, there were certain flaws in the RoboTask routines that created the billing orders for the conversion of the embedded base; although these flaws caused information to be placed incorrectly on some service orders, these inaccuracies were typically not so severe as to prevent the service order from posting to CABS. Consequently, some of the embedded base circuits that were converted to CABS contained inaccurate information (e. g., a circuit record might contain a code for a particular feature in the wrong place). Subsequent order activity on that same circuit could fall out (i. e., fail to post mechanically) because the record was inaccurate. Resolving these types of problems proved to be both difficult and time consuming. Phase 2 of the CABS UNE- P Conversion Involved Many More “Held” Orders than SBC Had Anticipated Although SBC estimated that Phase 2 of the conversion would require the processing of approximately 100,000 to 150,000 held UNE- P service orders, it actually required the processing of five to seven times that number. There were two principal reasons why SBC’s actual experience was so different from its original expectations: 4 See March 28 Ex Parte, Attach. D, at 1; Brown/ Cottrell/ Flynn Reply Aff. ¶ 19. 5 RoboTask is a software system that SBC had previously used to mechanically create service orders for access services and unbundled loops. SBC modified RoboTask to accept data feeds of provisioned UNE- P circuits from the provisioning database and to create corresponding CABS billing service orders. The enhancements to RoboTask allowed it to use a file from the Ameritech Customer Information System (“ ACIS”), the system that contains the order used to provision the service, to create a billing service order and then to post that billing service order to CABS. If either RoboTask fails to create the billing service order or the billing service order does not post successfully to CABS, the task of posting the order properly to CABS falls to the Local Service Center (“ LSC”). 4 Ex Parte Letter to Marlene H. Dortch Attachment April 3, 2003 Page 3 of 6 First, because Phase 1 of the conversion took so much longer to complete than had been anticipated, the period of time during which incoming UNE- P service order activity was held lasted approximately three times longer than expected. Second, during Phase 1, incoming UNE- P order activity was nearly two times higher than SBC had anticipated. As a result of these two factors, the number of orders that had to be held during Phase 1 for future processing to CABS was approximately 750,000. There Were Problems With Processing Both “Held” and New Billing Service Orders RoboTask’s problems were not limited to converting the embedded base; there were similar flaws in RoboTask’s routines designed to post both orders that had been held during the conversion process and orders that came in after the conversion process was complete. Some of these orders were adversely affected by these flaws in RoboTask, resulting in information being incorrectly positioned on the billing service order. Compounding the problem, large volumes of new UNE- P service orders continued to come in during the months immediately after the conversion. Some of these service orders did not post to CABS because of simple errors, unrelated to the conversion. These unposted orders added to the already substantial number of orders that had been held pending the conversion and that had fallen out because of RoboTask’s mistakes. The LSC was responsible for dealing with this growing backlog of unposted orders. However, in the period immediately after the conversion, the error- management tools available to the LSC service representatives simply did not allow for the efficient retrieval of information. The LSC service representatives often found themselves unable to determine the appropriate sequence of orders on a particular circuit. All of These Factors Contributed to the Inaccuracies in CABS As explained above, the conversion itself resulted in a much larger number of “held” orders than SBC had anticipated. Further, based on its experience with comparable database conversions, SBC expected that RoboTask would have been able to process approximately 90% of the orders mechanically. Instead, because of the various flaws discussed above, RoboTask’s actual mechanical-posting rate was closer to 70%. In other words, the fallout rate (30% rather than 10%) was three times greater than what SBC had expected. Therefore, of the 750,000 orders that RoboTask attempted to process mechanically, approximately 250,000 fell out for handling by the LSC. To make matters worse, the volume of unposted orders increased significantly from approximately 250,000 in December 2001 to approximately 400,000 in April 2002. The consequence of all of these factors was that the 5 Ex Parte Letter to Marlene H. Dortch Attachment April 3, 2003 Page 4 of 6 backlog of unposted orders exceeded not only SBC’s projections, but also SBC’s capacity to manage it effectively. SBC Devoted Substantial Resources to Fixing this Problem SBC Midwest undertook numerous initiatives to address the issues presented by the backlog of unposted orders. Throughout 2002, SBC Midwest significantly increased the number of service representatives assigned to handle the UNE- P fallout. Initially, the additional workforce had little impact, both because these service representatives were unfamiliar with SBC Midwest’s processes and because they did not have particularly useful tools to resolve the errors. However, in March 2002, SBC modified these tools to provide service representatives with the ability to ensure that service orders on the same circuit would be posted in the proper sequence. Subsequent enhancements to the tool allowed them to input status updates and to create follow- ups. Moreover, SBC dedicated a team focused on root cause analysis to correct and improve mechanical posting of the unposted orders. This effort resulted in enhancements to RoboTask that improved significantly the rate at which orders would mechanically post to CABS. In addition, SBC Midwest re- flowed unposted orders, using the improved mechanization enhancements to drive further reductions in the backlog. Mechanized processing of billing orders improved from 71% in December 2001 to 93% in July 2002. 6 Improvements in both mechanical and manual processing reduced the number of unprocessed orders to approximately 100,000 as of September 2002. While a dedicated team continued to work the backlog, the LSC was able to focus on the fallout from new service order activity and remain current. Although SBC had made substantial progress in reducing the backlog and in ensuring that orders that fell out for manual handling were handled appropriately and expeditiously, SBC concluded that the only way to ensure the accuracy of the CABS database going forward was to implement a mechanical reconciliation of the inventory of UNE- P billing records in CABS with the inventory of provisioned UNE- P records in ACIS. 7 That is why, “as a final quality assurance measure,” SBC implemented the database reconciliation in January 2003, which eliminated the backlog. 8 6 Data sources used in the calculation of the mechanical billing service order posting rate in December 2001 originated from the same data sources used by the “Informix” database implemented in March 2002. See infra pp. 5- 6. The data were collected manually and captured in an Excel spreadsheet. SBC has estimated the mechanical service-order posting data from December 2001 by first subtracting the number of UNE- P billing orders that fell out to the LSC for manual handling from the total number of UNE- P orders. SBC then divided this number by the total number of UNE- P billing orders processed by RoboTask, yielding an estimate for the percentage of UNE- P orders that mechanically posted to CABS. The more robust “Informix” tool was not implemented until March 2002, so SBC used this internal, order- tracking spreadsheet in the December 2001 timeframe. 7 See March 28 Ex Parte, Attach. D, at 2. 8 See Brown/ Cottrell/ Flynn Reply Aff. ¶ 17. 6 Ex Parte Letter to Marlene H. Dortch Attachment April 3, 2003 Page 5 of 6 SBC’s Bill- Posting Performance Has Improved Steadily and Significantly The table below shows the total number of UNE- P billing service orders in the Midwest region that are intended to post mechanically to CABS, followed by the number (and percentage) of those orders that actually do. The data are arranged by month, beginning in March 2002 and ending in March 2003: Table 1 Mechanical Posting of UNE- P Billing Service Orders in SBC Midwest Month Total Orders Processed by RoboTask Orders that Mechanically Post to CABS % of Orders That Mechanically Post to CABS Mar- 02 9 162,623 115,462 71% Apr- 02 342,209 253,712 74% May- 02 390,136 316,942 81% Jun- 02 396,840 355,946 90% Jul- 02 389,425 362,268 93% Aug- 02 516,082 477,038 92% Sep- 02 504,755 467,531 93% Oct- 02 510,109 468,171 92% Nov- 02 462,641 428,920 93% Dec- 02 484,063 438,983 91% Jan- 03 521,657 484,906 93% Feb- 03 516,592 490,286 95% Mar- 03 543,438 519,556 96% SBC produces the data contained in Table 1 internally as part of an ordinary and regular process in the normal course of business to ensure that its systems are functioning properly. To further enhance the management capabilities of the UNE- P billing process, SBC created an “Informix” tracking database and reporting tool. The Informix tool utilizes certain database software to track the CABS billing process from the extraction of the completed provisioning order in ACIS, through RoboTask’s creation of a billing service order, to the final posting of the bill in CABS. The Informix tool generates a status report of each order as it moves through the process. This enables the reporting tool to provide the raw data used to measure and track the mechanized processing of each order. To monitor the health of the mechanized process, the percentage of orders mechanically posting to CABS is calculated by dividing the number of orders that successfully post to the CABS billing database by the total number of RoboTask billing orders processed. The very small number of UNE- P 9 The data for March 2002 reflect orders processed between March 12 and the end of the month. 7 Ex Parte Letter to Marlene H. Dortch Attachment April 3, 2003 Page 6 of 6 orders that do not impact a CABS bill (e. g., orders that change the listing address) are never sent to CABS and consequently are not included in either the numerator or the denominator of this measurement. These statistics are audited by the reporting system to ensure that the total number of orders add up appropriately. These data are reported to the LSC leadership team on a daily basis. This management report is used as a mission critical tool to monitor both the health of the mechanized billing process as well as the LSC’s handling of those orders that do require manual handling. It is reviewed in detail each business day so that appropriate corrective action (where appropriate) can be taken immediately. BearingPoint’s Testing Confirms that SBC Had Made Substantial Improvements Upon completion of Phase 1 of the CABS UNE- P conversion in October 2001, the Commission staffs in all five of SBC Midwest’s states discussed with SBC Midwest and BearingPoint the appropriate time to conduct the test for UNE- P billing timeliness. On January 8, 2002, the Commission staffs for Illinois, Indiana, Ohio and Wisconsin directed BearingPoint to begin preparations for the testing of UNE- P billing timeliness. BearingPoint began submitting UNE- P service orders for existing and new accounts in all four states in February 2002. Not surprisingly, the results of the BearingPoint test were affected by the same problems described above, and it issued an exception applicable to all four states. When BearingPoint retested this exception from August through October 2002, however, the improvements that SBC had implemented made an enormous difference. Whereas it had found in its first test in February 2002 that only 63% to 75% of its service orders were posting to CABS in a timely manner, the August through October 2002 percentages had become 97% to 100%. 10 Conclusion The complexities associated with the CABS UNE- P conversion created unexpected problems that contributed to the large number of orders that did not post in a timely manner into CABS. SBC believes that the vast majority of the errors in the 138,000 circuits that SBC ultimately corrected in Michigan as part of the database reconciliation were caused by some combination of (1) original conversion errors, (2) errors resulting from out- of- sequence posting of billing orders, (3) manual errors, which were more likely in the environment existing at the time, and (4) errors from orders that SBC set aside at the end of the conversion process, knowing that they would be corrected by the reconciliation itself. Nevertheless, through process enhancements, additional resources and the reconciliation itself, SBC Midwest’s billing performance is now excellent, having improved significantly in the year and a half since it completed the CABS UNE- P conversion. 10 See Ohio Interim OSS Status Report (Dec. 20, 2002) at 1028; BearingPoint Exception Report 127, Version 2 (issued Aug. 1, 2002; closed Nov. 12, 2002) . 8 9