PUBLIC NOTICE Federal Communications Commission 445 12th St., S.W. Washington, D.C. 20554 News Media Information 202 / 418-0500 Fax-On-Demand 202 / 418-2830 TTY 202 / 418-2555 Internet: http://www.fcc.gov ftp.fcc.gov DA 10-670 Released: April 20, 2010 COMMENT SOUGHT ON RESIDENTIAL FIXED BROADBAND SERVICES TESTING AND MEASUREMENT SOLUTION Consumer Information and Disclosure Public Notice PLEADING CYCLE ESTABLISHED CG Docket No. 09-158, CC Docket No. 98-170, WC Docket No. 04-36 Comment Date: May 4, 2010 In the 2009 Consumer Information and Disclosure Notice of Inquiry, the Commission has sought comment on whether there are opportunities to protect and empower American consumers by ensuring sufficient access to relevant information about communications services.1 Subsequent to release of the Notice of Inquiry, the Commission issued a Public Notice as part of the National Broadband Plan proceeding to gather additional information on fixed Internet broadband service (“Fixed Services”), seeking comment on service quality and transparency issues.2 Accurate and clear information is crucial to a well-functioning market. The National Broadband Plan stated that, “[p]utting more information in the hands of consumers is a proven method to promote meaningful competition and spur innovation, both of which will generate more and better consumer choices.”3 Numerous parties have supported this goal and suggested that the Commission and private industry should work together to address this issue.4 To assist in this effort, the National Broadband Plan recommended that the FCC continue efforts to measure and publish data on actual performance of fixed broadband services, and to make that data available publicly.5 The FCC released a Request for Quotation 1 See Consumer Information and Disclosure, CG Docket No. 09158, CC Docket No. 98-170, WC Docket No. 04-36, Notice of Inquiry, 24 FCC Rcd 11380 (2009). 2 See Comment Sought on Broadband Measurement and Consumer Transparency of Fixed Residential and Small Business Services in the United States – NBP Public Notice #24, GN Docket No. 09-51, Public Notice, 24 FCC Rcd 14120 (WCB 2009) (“NBP PN #24”). 3 Omnibus Broadband Initiative, Federal Communications Commission, Connecting America: The National Broadband Plan (2010) (National Broadband Plan), 44. 4 Verizon Comments in re NBP PN #24 filed Dec. 14, 2009, at 1; AT&T Comments in re NBP PN #24, filed Dec. 16, 2009, at 1; New America Foundation Comments in re NBP PN #24, filed Dec. 14, 2009, at 2; NCTA Comments in re NBP PN #24, filed Dec. 14, 2009, at 2; Time Warner Cable Comments in re NBP PN #24, filed Dec. 14, 2009, at 1. 5 Omnibus Broadband Initiative, Federal Communications Commission, Connecting America: The National Broadband Plan (2010) (National Broadband Plan), 45. 2 (“RFQ”) pursuant to this recommendation, and has recently contracted with a third-party, SamKnows Limited, to begin this effort.6 In this Public Notice, we seek comment on the methodology SamKnows proposes to adopt. The following sections outline the approach of SamKnows to accurately measure broadband performance. There are five main elements to the approach: 1. Development and recruitment of panel of respondents The recruitment strategy has been developed following a similar project in the UK, which SamKnows carried out on behalf of Ofcom, the independent regulator and competition authority for UK communications industries.7 Panelists are selected on the basis of how they relate to the broadband population rather than any demographic criteria. The only pre-requisite of the recruitment is that the eventual panel is representative of the ‘broadband population’ in terms of technology, geography and service level. Therefore, the recruitment strategy is to initially solicit volunteers through a media campaign using social and traditional media, such as consumer and technology press, alongside Twitter and independent bloggers and opinion formers. This campaign would ask consumers whether they would like to volunteer for a study to ascertain actual broadband speeds. If interested, volunteers would be asked to complete a short form, to ascertain whether, taken in consideration with other volunteers, they are potentially representative of the ‘broadband population’ and to ensure that there are no conflicts of interest in participating in the study. When a sufficient number of volunteers have been recruited, SamKnows’ statisticians will look to select a sample of volunteers. The volunteers are asked to verify their line speed, by means of a web-based speed test. (Please note: it is widely recognized that an online speed test is not an absolute indicator of broadband speeds delivered; however, it is used in this instance as simply an indicator of performance, rather than confirmation). The volunteers are also asked to complete an additional, non-intrusive questionnaire regarding their broadband connection, general Internet usage and perceived broadband performance. This ‘refined’ pool is again reviewed by the SamKnows statisticians, who select what will become the US Broadband Performance Panel. Each successful panelist will be asked to complete a terms and conditions agreement, and on acceptance will be invited to join the project. At any point during the project, should it become necessary to target a specific geography, technology, ISP or service level, ISPs that have expressed willingness to assist in this effort may be called upon to assist in the recruitment. At no point during this process do the ISPs have access to the panelist details; their role is to point prospective volunteers to the SamKnows sign-up page, whereupon the volunteers become part of the pool. Whether a volunteer is invited to join the panel is determined by the SamKnows statisticians. 6 Additional information on SamKnows limited can be found at www.samknows.com. 7 Additional information on SamKnows work with Ofcom can be found at www.samknows.com/broadband/performance.php?page=performance-ofcom-and-samknows 3 Throughout this project, SamKnows will adhere to a strict opt-in policy that ensures that only consumers who wish to participate in the panel are included. Each panelist will have access to his/her own broadband performance data via a secure login. 2. Data collection & tests The SamKnows solution operates by having each panelist install a custom piece of hardware with specially designed pre-loaded test software – the SamKnows Whitebox – behind their modem. This Whitebox is programmed not only to run a sequence of performance tests over the ISP network, but also to assess a consumer’s experience in web browsing and other typical Internet activities. The SamKnows solution’s topology means that it is able to measure, collect and report performance data not only over the panelist’s ISP network (specially designed performance tests from modem to Internet gateway), but also to test the performance that consumers actually experience (web and other typical Internet use from behind the modem to the public Internet). Each Whitebox is programmed with a random start time. Starting at this time, each Whitebox runs its tests every sixty minutes, 24 hours a day, 7 days a week. The Whitebox tests are summarized below: Application Description Web surfing Browsing of typical web sites containing HTML, images and small rich media objections, e.g., Flash-based banner advertisement. Video download (peer to peer) Downloading of TV show or movie from a legal P2P site. Video Streaming Streaming of a TV show in real time (Hulu). File download Downloading of large data files using HTTP or FTP, e.g., downloading new device drivers or software packages. File upload Uploading a large file to a server using HTTP or FTP, e.g., uploading a video to YouTube. Online gaming Online gamers value low latency and delay when sending or receiving data to a gaming server. VOIP Quality and reliability of VOIP calls are determined by a number of factors. Specific metrics includes: packet loss, jitter, and available bit rate. SamKnows will measure, collect and report the following data on an ongoing basis: • Data speed (being a combination of the Download and Upload Speed) • Data usage • Download speed • Upload speed • Latency • Jitter • Availability 4 • Packet loss • DNS resolution time (measured in milliseconds) • DNS failures (measured as a percent of total DNS requests) • Web page load time (measured in milliseconds) SamKnows is a member of the Measurement Labs (M-Labs) research consortium. We will use the M- Labs infrastructure as destinations for our remote tests during this project. In addition, SamKnows will engage with the ISPs to encourage them to also host test nodes within their networks. 3. Maintaining consumer privacy SamKnows will maintain the following safeguards to ensure the security and confidentiality of the personally identifiable information (PII) of its panelists: Administrative safeguards: Whiteboxes are assigned a unique identifier, referred to as the hostname. All communications and data sent between the Whiteboxes and the data clusters are linked via the hostname. All data which the back-end servers then store, process and report upon are only referenced via the hostname; the PII of panelists is never included in reports or data exports. All data sent between SamKnows and third parties is encrypted and sent via SCP or SFTP. The SamKnows architecture is regularly scanned and audited for security issues. Technical safeguards: The SamKnows architecture is based around the Linux operating system; the Whiteboxes and back-end servers are hardened using standard industry methods. Panelists have no remote or local access to their Whitebox over their LAN or through the Internet. Back-end servers are protected via layered hardware firewalls. SamKnows administrators access back-end servers via IPSEC VPN to hardware firewalls and SSH via SSH keys to the physical servers. All communications between Whiteboxes and data control servers are encrypted using SSL over HTTPS. Physical safeguards: The SamKnows servers are hosted in high security data premises with 24-hour security personnel, CCTV and access control systems. Data center architecture is replicated between multiple data centers for resilience against provider network outages and critical hardware failure. Data from the program is collected, aggregated, analyzed, and reported using a strict protocol, designed to ensure confidentiality, as follows: • Panelist data is initially collected into a database which is hosted within a secure facility. • Redacted and/or aggregated data is exported from this database using pre-programmed database scripts. This data has been stripped of all PII. • This redacted/aggregated data is routed to the SamKnows project statistical team for further analysis and reporting. • This dataset is routed back to the technical team for insertion into the SamKnows reporting architecture, which deploys reports according to pre-programmed scripts. 4. Data analysis and presentation All data collected from a specific ISP network are fully disclosable to that ISP, including the detailed non- graphical data gathered. Under the terms of our panel contracts and relevant data protection legislation, the granular ISP network performance data is not disclosable to another competitor network. However, performance data from all 5 networks is statistically weighted and combined into a number of benchmarks as a comparison for an ISP’s own network’s performance in each test area. SamKnows will develop an online forum for participating panelists where they will be able to access a dashboard on performances related to their ISP package for aggregated benchmark of performance. Panelists will be invited to provide the following information: • Name • Full zipcode (+4) • Email address • Telephone number (for support reasons) • Name of broadband provider • Advertised speed of panelist’s Broadband Package • Confirmation of no employment relationship with ISP considered • Number, gender and age of people in household • Whether broadband package is “unlimited” or “capped” • Exceed monthly allowance (if exceed, panelist rejected) • How often panelist surfs the Internet • How often panelist watches online video • How often panelist watches online TV • How often panelist watches online films • How often panelist makes calls via the Internet • How often panelist plays games online • How often panelist listens to online audio • How often panelist downloads music • Panelist’s overall satisfaction with his/her broadband connection The following will be derived: • Region • Density – urban, sub-urban and rural • An accurate estimate of length of line from the panelist’s premises to the relevant exchange Information to be provided in relation to each panelist each month: • DSL or DOCSIS connection metrics • Application layer, service specific performance metrics • Time varying performance Reporting System: SamKnows has built a proprietary reporting system to publish the data. This system has been designed to pull data from other tests run by 3rd parties to allow real-time comparison. The SamKnows panel monitors networks according to a series of proprietary and regulator-certified tests that provide results in three main areas: Speed: Web download and web upload speed, DNS response time, website load time and latency. Integrity: Packet loss, failed web requests and failed DNS queries within the network. 6 Multimedia performance: Audio/visual packet loss, jitter and overall streaming performance. FCC Data: SamKnows will format and send the Project Data to the FCC on a quarterly basis. SamKnows will supply the FCC with data by panelist on an aggregated basis (i.e. one value per month for each panelist.) The FCC will also have access to the dynamic reporting system described above. The following variables are supplied for all panelists • ID • ISP • Headline speed • Access technology (DSL, cable) • Region • Rural/urban • Straight-line distance from exchange Public Website: The FCC and SamKnows will make available a public version of the real-time reporting website. This public version will contain summary reports of actual service coverage by region and provider. This data will be made available separately to the FCC in aggregated form, for the purpose of creating a broadband performance report that will be published to the general public. Publicly Accessible Data: SamKnows will also make the raw data available to researchers on a periodic basis, with all PII removed. This will allow academics to study this data freely. 5. Flexible, configurable and scalable framework The SamKnows framework has been designed to enable any number of tests to be deployed to a distributed (and representative) number of homes. Furthermore, although the initial requirement is for 10,000 homes, the SamKnows infrastructure can scale to many times this number. The SamKnows test software can also be embedded onto other hardware such as routers, gateways, games consoles, etc. --------- We seek public comments generally on this approach, as well as specifically on the five elements described above. This matter shall be treated as a “permit-but-disclose” proceeding in accordance with the Commission’s ex parte rules. See 47 C.F.R. §§ 1.1200, 1.1206. Persons making oral ex parte presentations are reminded that memoranda summarizing the presentations must contain summaries of the substance of the presentations and not merely a listing of the subjects discussed. More than a one- or two-sentence description of the views and arguments presented generally is required. See 47 C.F.R. § 1.1206(b). Other rules pertaining to oral and written ex parte presentations in permit-but-disclose proceedings are set forth in section 1.1206(b) of the Commission's rules, 47 C.F.R. § 1.1206(b). All comments should refer to CG Docket No. 09-158, CC Docket No. 98-170, WC Docket No. 04-36. Please title comments responsive to this Notice as “Comments—Consumer Information and 7 Disclosure PN, DA 10-670.” Further, we strongly encourage parties to develop responses to this Notice that adhere to the organization and structure of this Notice. Comments may be filed using (1) the Commission’s Electronic Comment Filing System (ECFS), (2) the Federal Government’s eRulemaking Portal, or (3) by filing paper copies.8 Comments can be filed through the Commission’s ECFS filing interface located at the following Internet address: http://www.fcc.gov/cgb/ecfs/. Comments can also be filed via the Federal eRulemaking Portal: http://www.regulations.gov.9 In completing the transmittal screen, commenters should include their full name, U.S. Postal Service mailing address, and the applicable docket or rulemaking number. Parties who choose to file by paper must file an original and four copies of each filing. If more than one docket or rulemaking number appears in the caption of this proceeding, filers must submit two additional copies for each additional docket or rulemaking number. Filings can be sent by hand or messenger delivery, by commercial overnight courier, or by first- class or overnight U.S. Postal Service mail. All filings must be addressed to the Commission’s Secretary, Office of the Secretary, Federal Communications Commission. § Effective December 28, 2009, all hand-delivered or messenger-delivered paper filings for the Commission’s Secretary must be delivered to FCC Headquarters at 445 12th St., SW, Room TW- A325, Washington, DC 20554. The filing hours are 8:00 a.m. to 7:00 p.m. All hand deliveries must be held together with rubber bands or fasteners. Any envelopes must be disposed of before entering the building. Please Note: The Commission's former filing location at 236 Massachusetts Avenue, NE, Suite 110, Washington, DC 20002 permanently closed on December 24, 2009. § Commercial overnight mail (other than U.S. Postal Service Express Mail and Priority Mail) must be sent to 9300 East Hampton Drive, Capitol Heights, MD 20743. § U.S. Postal Service first-class, Express, and Priority mail must be addressed to 445 12th Street, SW, Washington DC 20554. People with Disabilities: To request materials in accessible formats for people with disabilities (Braille, large print, electronic files, audio format), send an e-mail to fcc504@fcc.gov or call the Consumer and Governmental Affairs Bureau at (202) 418-0530, (202) 418-0432 (TTY). For further information about this Public Notice, please contact Dave Vorhaus, Omnibus Broadband Initiative, at (202) 418-3641. 8 See Electronic Filing of Documents in Rulemaking Proceedings, 63 Fed. Reg. 24121 (1998). 9 Filers should follow the instructions provided on the Federal eRulemaking Portal website for submitting comments.