Peer Review Panel Report On OET Report FCC/OET 08-TR-1005 Evaluation of the Performance of Prototype TV- Band White Space Devices Phase II Peer Reviewers: Mary Bucher, WTB George Dillon, EB James Higgins, EB/Columbia Martin Liebman, WTB Doug Miller, EB/Atlanta October 1, 20082 Background: The following is the peer review of OET Report FCC/OET 08-TR-1005, entitled “Evaluation of the Performance of Prototype TV-Band White Spaces Devices Phase II.” The report is a result of OET’s second phase study of hardware development devices with capabilities for operating on an unlicensed basis in the TV bands as part of the “TV white spaces” proceeding, ET Docket No. 04-186. This second phase study examined four development devices and follows OET’s initial study of TV white space development devices, which was released July 31, 2007, see “Initial Evaluation of the Performance of Prototype TV-Band White Space Devices,” OET Report FCC/OET 07-TR-1006 (Phase I Report).1 The devices examined in this second phase study were submitted by industry in response to an OET public notice inviting submittal of additional prototype white space devices and are product development platforms, rather than models of products that could actually be marketed. One of these devices has both transmitting and spectrum sensing capabilities; the other three have only spectrum sensing capability. Each of the devices has unique spectrum sensing capabilities and was examined in the laboratory and the field with respect to its ability to detect the presence of digital TV, analog TV, and/or Part 74 wireless microphone signals. The one device with a transmitter was also examined for its potential to cause interference to the reception of digital television signals and cable service. Peer review of the OET report was performed as required under the OMD Information Improvement Act for influential scientific and engineering studies. The review panel was made up of five engineers, three from the Enforcement Bureau and two from the Wireless Telecommunications Bureau. The review panel analyzed and discussed various subject areas in the OET report, both independently and jointly. Specifically, as requested in the OET memo, the review panel addressed the following: 1. Whether the scope of testing in terms of spectrum sensing abilities and signal conditions examined was appropriate and sufficient; 2. Whether the measurement methodologies used in the testing of the prototype devices spectrum sensing abilities was appropriate; 3. Whether the scope of testing of the device with transmitting capability for its potential to cause interference to digital TV, analog TV, and wireless microphone signals was appropriate 4. Whether the various tests performed were properly conducted consistent with the selected methodologies. The response of the review panel is presented below for each question shown above: 1) Whether the scope of testing in terms of spectrum sensing abilities and signal conditions examined was appropriate and sufficient. In the opinion of the review panel, the overall scope of the spectrum sensing testing was appropriate. Tests were performed against a range of devices that utilized various signal sensing methods. Initial laboratory tests were performed under controlled conditions, followed up by field tests utilizing live over-the-air television signals. 1 Five devices were submitted for this phase of testing; however, one of the devices stopped working after a few tests were performed on it and the reports contain only partial data for that device. October 1, 20083 For the laboratory tests, initial measurements were taken at multiple channels across the operating range of the prototype devices, which consisted of UHF channels 21-51, to determine if the devices responded consistently. The devices were tested against a clean digital TV signal, alone and in the presence of TV signals on both the first and second adjacent channels, as well as recorded RF captures of DTV signals to simulate various multi-path conditions. These tests were followed up by field tests at nine different locations representing a variety of topography including urban, suburban, rural, residential and business locations. Multiple measurements were taken at each location under varied indoor and outdoor conditions. The field tests were performed over the entire operating range of the devices and examined the ability of the devices to detect live over-the-air ATSC and NTSC signals. For these tests, channel occupancy based on TV contours was noted, but actual measurements were taken to determine the occupancy of each channel. The wireless microphone portion of the testing was performed both in the laboratory and in the field. Laboratory tests were performed using signals generated by two different wireless microphones. The microphones were placed in an anechoic chamber to isolate the test signal and avoid undesired signals. White space device sensitivity measurements were taken for the wireless microphone signal alone and with simulated DTV signals at various first and second adjacent channels. The field tests were performed under actual real world conditions during a pre-season football game at FedEx Field in Landover, MD and before and during a performance of a Broadway play at the Majestic Theater in New York City. Measurements were taken at multiple locations and times at these locations. The review panel believes that the approach taken and procedures followed by the staff were well thought out and provided an excellent analysis of the sensing capabilities of the WSDs. The review panel believes that if more time and testing equipment had been available for the laboratory testing, it would be more realistic to have tested multiple, adjacent DTV signals at the same time, rather than testing only a single adjacent channel at a time. In addition, if there had been no time constraints, performing more scans at the various field locations to obtain a larger set of results for determining detection reliability would have been useful. 2) Whether the measurement methodologies used in the testing of the prototype devices spectrum sensing abilities was appropriate. The review panel believes that the measurement methodologies used in the testing of the prototype devices were appropriate. OET developed a Phase II test plan which largely follows the procedures used for Phase I testing with some modifications. As in the Phase I test plan, guidance and procedures established to date by IEEE 802.22 for testing the spectrum sensing capability of fixed/access WSDs were considered where applicable. In the instances where the testing deviated from those methodologies, the review panel feels that the underlying objectives were met by the tests that were performed. OET indicated that comments and suggestions offered within the public record in the Commission’s TV white spaces proceeding with respect to both previous and current testing were considered and included in the test plan where appropriate and practicable. As stated above, both laboratory and field tests were performed to assess the scanning/sensing capability of the prototype WSDs for the television portion of the testing. Three separate bench October 1, 20084 tests were performed to determine the minimum DTV signal detection threshold for each of the prototype devices submitted for evaluation. The first laboratory test utilized a single, unimpaired, laboratory-grade DTV signal as the test input. The second laboratory test utilized recorded (captured) DTV signals that incorporated “real world” distortions resulting from reflections and multi-path fading. The third laboratory test utilized two unimpaired, laboratory-grade signals as the input, one on the detection channel and the other placed on the first or second adjacent channel. Measurements were taken to determine DTV power levels resulting in 100% detection reliability. Data was also collected for a few points above and below the breakpoint where the detection reliability degrades from 100%. In the laboratory multiple measurements were taken in order to determine the percentage of successful detections with some statistical relevance, and initial tests were performed for multiple channels to investigate potential frequency related differences in performance. For the baseline detection signal test using a single DTV input, the input DTV signal was initially set to a measurable level and then further attenuated incrementally with the calibrated step attenuator bank while exercising the scanner over the occupied channel. At each attenuation step (input power level), thirty independent trials were performed. Tests were performed on three channels in the lower (channel 21), middle (channel 36 or 37) and upper (channel 51) portions of the WSD tuning range. The review panel feels that the number of trials and channels tested was sufficient. Since the initial test results revealed that the sensing performance of the devices was fairly consistent over their tuning range, the remaining tests were performed on a single channel in the middle of the tuning range. The review panel agrees that testing on a single channel was appropriate given the results of the baseline testing. The methodologies for the two DTV signal tests and the recorded DTV signal tests were similar to the methodology used in the single DTV input signal tests. For the two DTV signal test, a second DTV input signal was introduced at N±1 and N±2. Detection threshold measurements were performed with the adjacent channel signal power set to -28 dBm (high), -53dBm (moderate) and -68dBm (low) levels. The review panel feels that it would be helpful if the report explained the rational for the selection of these signal levels (i.e., an explanation as to why -28 dBm was considered a high signal level and why -68 dBm was considered a low signal level). For both tests thirty independent trials were performed at each attenuation step (power level in detection channel) in order to determine the percentage of successful detections with some statistical relevance. Additional recorded DTV signal measurements utilizing one thousand independent trials were performed on the Motorola device to smooth out variations resulting from significant instantaneous temporal variations in the RF captures. Had time allowed, the panel feels that it would had been useful to perform these additional measurements on all of the devices. Field tests were performed utilizing live OTA ATSC and NTSC television signals at nine different locations representing a variety of topography including urban, suburban, rural, residential and business locations. At each field test site, testing was conducted at two locations under varied indoor and outdoor conditions. At each of these locations, two scans were performed with each device over its entire channel range and the results were recorded. The review panel felt that the measurements taken at these locations provide a good initial assessment October 1, 20085 of the scanners/sensors performance under “real world” conditions. However, the review panel believes that it would have been useful to consider the potential effects of interfering structures on the results (i.e., by performing tests inside and outside at the same site and describing the nature of any interfering structures). The panel also believes that it may have been worthwhile to attempt to identify areas subject to fading and shadowing and to perform tests in and around these areas. The review panel feels that the measurement methodologies used for the wireless microphone portion of the testing were appropriate to ascertain the ability of the devices to scan for and detect Part 74 wireless microphones. Two different types of microphones, FM modulated and digitally modulated, were used in these tests. Regarding these tests, the review panel did note that during most of the field testing scans were performed with the microphones either turned on or turned off. The panel believes that testing in a more dynamic environment with microphones being turned on and off intermittently during the scans would have been more beneficial. The report also indicates that there were a large number of false positive indications during the wireless microphone field testing. In relation to these results, the review panel believes that it would be helpful if the report could expand on possible explanations for these results and/or provide a rational for not performing additional tests to possibly identify their cause. 3) Whether the scope of testing of the device with transmitting capability for its potential to cause interference to digital TV, analog TV, and wireless microphone signals was appropriate. It is the opinion of the review panel that the scope of the testing of the Adaptrum WSD prototype for its potential to cause interference to digital TV was appropriate, given the study’s stated limitations. However, the review panel did not observe any discussion in the report describing tests that may have been done to analyze potential interference to analog TV signals or wireless microphone signals. The project conducted limited, or “anecdotal,” tests of the prototype WSD transmitter to provide information on its potential to interfere with digital TV reception. These tests were performed within the confines of the FCC laboratory facility with an unobstructed line-of-sight (LOS) propagation path between the WSD transmitter operating at approximately 150 mW EIRP and the DTV test receiver antenna. A test DTV receive antenna was mounted on an extendable mast to simulate a consumer installed roof-top antenna. All available channels were analyzed for the weakest receivable channel (i.e., the worst-case interference scenario) and channel 30 was ultimately used for that purpose. The WSD transmitter was then placed in the “main beam” of the receive antenna, tuned to each of the immediately adjacent channels, and activated at incremental distances from the DTV receive antenna while observing for interference effects to the picture quality. Tests were also performed with the WSD tuned to channel 30. The distance at which interference was observed was measured and recorded. The tests were repeated with the WSD transmitter located to the sides and behind the DTV receive antenna. Overall, the review panel feels that these tests provided valuable data on the effects of WSDs on DTV reception. The panel feels that additional analysis of the WSD transmitter operating at power levels other than an EIRP of 150 mW, e.g., the power level initially proposed by the Commission for mobile WSD devices, would be valuable. October 1, 20086 The panel recognizes that use of OTA signals produces only anecdotal results and the ability to perform tests were limited by the area where the tests were being performed. Under different circumstances, the review panel believes that it would have been useful for a larger number of tests to have been conducted under more uniform conditions to provide a larger set of results that could be more easily correlated. The review panel also feels that normalizing the data in Table 4- 2 for a standard path length would perhaps make the results more meaningful. The review panel recognizes that the testing was limited by the fact that the Adaptrum WSD prototype was the only available working device with an active transmitter. Had more transmitting devices been available, the panel believes that simultaneous testing of multiple devices to determine the cumulative interference affects of those devices would have yielded valuable data. 4) Whether the various tests performed were properly conducted consistent with the selected methodologies. In the opinion of the review panel, the tests were properly conducted consistent with the selected methodologies. OET developed a Phase II test plan which largely follows the procedures used for Phase I testing with some modifications. OET has indicated that comments and suggestions offered within the public record in the Commission’s proceeding on the TV white spaces with respect to both previous and current testing were considered and included in the test plan where appropriate and practicable. For the most part, OET’s testing was consistent with the established Phase II test plan and deviation from the test plan did not undermine the results. The review panel believes that the testing was well done and thorough. For more description of the testing methodology, see No 2 above.