Federal Communications Commission Report: August 11, 2021 Nationwide WEA Test December 2021 Report: August 11, 2021 Nationwide WEA Test December 2021 Public Safety and Homeland Security Bureau Federal Communications Commission — 45 L Street, NE — Washington, DC 20002 TABLE OF CONTENTS Para. I. SUMMARY 1 II. BACKGROUND 3 III. THE 2021 NATIONWIDE WEA TEST 8 A. Parameters of the 2021 Nationwide WEA Test 8 B. Participation in the 2021 Nationwide WEA Test 9 a. CMS Providers 9 b. Leveraging Partnerships with Emergency Management Agencies to Gather Nationwide WEA Test Data 10 a. CMS Provider Responses to Questions from PSHSB Bureau Chief 13 C. Test Performance 14 1. Reliability 14 a. Overall 14 b. By Wireless Service Provider 15 c. By CMS Provider versus Mobile Virtual Network Operator 16 d. By Generation of Wireless Network Technology 18 e. By Mobile Device WEA Capabilities 19 f. By Mobile Device Manufacturer 20 g. By Device Operating System 21 h. By Location (Indoor/Outdoor) 22 i. By Whether Device Was In Use 23 2. Latency 24 a. CMS Provider Network Latency Analysis 24 b. Latency Overall 29 c. By Wireless Provider 32 d. By Generation of Wireless Network Technology 34 3. Accessibility 35 D. Analysis of Reported Challenge with Duplicate Messages 36 IV. NEXT STEPS 40 V. CONCLUSION 41 I. SUMMARY 1. The Integrated Public Alert and Warning System (IPAWS) Modernization Act, 2015 (Public Law 114-143) requires the Federal Emergency Management Agency’s (FEMA’s) IPAWS Program Management Office to conduct, not less than once every three years, nationwide tests of the public alert and warning system. On August 11, 2021, FEMA, in coordination with the Federal Communications Commission (Commission), conducted a nationwide test of Wireless Emergency Alerts (WEA), which is a system designed to notify the public through their mobile devices about emergencies that may affect them. The 2021 nationwide WEA test was designed to evaluate the effectiveness of WEA to distribute an emergency message nationwide. For the first time, FEMA elected to conduct the test using the “State/Local WEA Test Alert” functionality, which only displays messages on wireless handsets on which subscribers have affirmatively opted-in to receive test messages. The Commission partnered with 11 federal, state, and local emergency management agencies to assess the performance of the nationwide WEA test through collecting and analyzing volunteer responses from across the nation’s diverse geographic and technological landscape. 2. The following report presents the Public Safety and Homeland Security Bureau’s (PSHSB) analysis of the survey responses and material submitted by the nationwide wireless providers regarding their performance in the nationwide WEA test. As detailed below, performance during the 2021 nationwide test demonstrates that, on the whole, WEA generally is performing reliably but there is room for improvement. The completed survey responses indicate that the test message was received by approximately 90% of respondents that opted into the test with a compatible device. The majority of respondents that received the test alert received it within two minutes of transmission. PSHSB finds that WEA reliability during the test was largely consistent across Commercial Mobile Service (CMS) Provider, generation of wireless network technology (i.e., 4G or 5G), mobile device manufacturer, device operating system, and whether a user was indoors or outdoors, or whether the mobile device was already in use at the time of the test. PSHSB also finds, however, that many mobile devices erroneously received a duplicate nationwide WEA test message, and that there may be opportunities to improve WEA’s reliability. Based on these findings, PSHSB recommends actions that would continue to make WEA more effective and improve upon the nation’s ability to evaluate WEA’s performance. II. BACKGROUND 3. The 2006 Warning, Alert, and Response Network Act (WARN) Act directed the Commission to establish technical requirements to enable CMS Providers to voluntarily offer emergency alerts and warnings to their subscribers. See Warning, Alert, and Response Network Act, Title VI of the Security and Accountability for Every Port Act of 2006, Pub. L. No. 109-347, 120 Stat. 1884 (2006) (WARN Act). See also 47 CFR § 10.10(d) (defining a Commercial Mobile Service Provider for the purpose of WEA as “an FCC licensee providing commercial mobile service as defined in section 332(d)(1) of the Communications Act of 1934”); 47 U.S.C. § 332(d)(1) (defining the term “commercial mobile service” as “any mobile service (as defined in 47 U.S.C. § 153) that is provided for profit and makes interconnected service available (A) to the public or (B) to such classes of eligible users as to be effectively available to a substantial portion of the public, as specified by regulation by the Commission”). Accordingly, in 2008, the Commission adopted a regulatory framework for WEA. Commercial Mobile Alert System, PS Docket No. 07-287, First Report and Order, 23 FCC Rcd 6144 (2008) (WEA First Report and Order); Commercial Mobile Alert System, PS Docket No. 07-287, Second Report and Order and Further Notice of Proposed Rulemaking, 23 FCC Rcd 10765 (2008) (WEA Second Report and Order); Commercial Mobile Alert System, PS Docket No. 07-287, Third Report and Order, 23 FCC Rcd 12561 (2008) revised by Erratum (dated Sep. 5, 2008) (WEA Third Report and Order). In 2012, the Commission and FEMA jointly announced WEA’s deployment. See FCC's Public Safety and Homeland Security Bureau Sets Timetable in Motion for CMS Providers To Develop a System That Will Deliver Alerts to Mobile Devices, PS Docket No. 07-287, Public Notice, 24 FCC Rcd 14388 (PSHSB 2009). Since its deployment in April 2012, over 550 authorized alert originators have used WEA over 61,000 times. 4. The IPAWS Modernization Act, 2015 requires the FEMA IPAWS Program Management Office to conduct, not less than once every three years, nationwide tests of the public alert and warning system. Integrated Public Alert and Warning System Modernization Act of 2015, Pub. Law No. 114-143, 130 Stat 327 (2016). In 2018, FEMA first performed a nationwide test of WEA, in conjunction with a nationwide test of the Emergency Alert System (EAS). PSHSB, Report: October 3, 2018 Nationwide WEA and EAS Test (2019), https://docs.fcc.gov/public/attachments/DOC-356902A1.pdf. FEMA conducted the 2018 nationwide test using the Presidential level alert classification, which enabled it to transmit the test alert to all WEA-capable wireless devices nationwide. See Improving Wireless Emergency Alerts and Community-Initiated Alerting, PS Docket No. 15-91, Order, 33 FCC Rcd 6816 (PSHSB 2018). PSHSB received anecdotal feedback directly from the public about the 2018 test through its Public Safety Support Center, and some WEA stakeholders voluntarily shared with the Commission and published survey data describing their experiences with the test. See, e.g., Letter from Benjamin J. Krakauer, Assistant Commissioner, Strategy & Program Development, New York City Emergency Management Department, to Marlene Dortch, Secretary, Federal Communications Commission, PS Docket Nos. 15-91 and 15-94 (filed Nov. 21, 2018); Thomas Crane, The Presidential Alert: Was it a success?, EVERBRIDGE (Oct. 15, 2018), https://www.everbridge.com/blog/thepresidential-alert-was-it-a-success/; Emily Dreyfuss, Why Didn’t I Get an Emergency Presidential Alert Text?, WIRED (Oct. 3, 2018), https://www.wired.com/story/why-didnt-i-get-emergencypresidential-alert-text/. In 2018, PSHSB reported that 61% of commenters in its Public Safety Support Center “reported no problems in receiving the WEA test message.” PSHSB, Report: October 3, 2018 Nationwide WEA and EAS Test, at 7 (2019), https://docs.fcc.gov/public/attachments/DOC-356902A1.pdf. Other sources surveying performance of the nationwide WEA test found that between 52.7% and 83% of respondents reported receiving the test. Id. at 6-7. PSHSB found that the 2018 test “demonstrated that WEA is an effective alerting tool to rapidly disseminate emergency information to the public.” Id. at 3. For the 2021 nationwide test, FEMA elected to use the State/Local WEA Test classification, which requires users to affirmatively opt in to receive the test alert. State/Local WEA Test Alerts allow state, local, Tribal, and territorial alert originators to test the WEA system for readiness and proficiency training while only displaying the tests on the mobile devices of users that affirmatively opt in to receive them. See Wireless Emergency Alerts; Amendments to Part 11 of the Commission’s Rules Regarding the Emergency Alert System, PS Docket Nos. 15-91, 15-94, Report and Order and Further Notice of Proposed Rulemaking, 31 FCC Rcd 11112, 11156, paras. 67-68 (2016) (WEA R&O and WEA FNPRM; 47 CFR § 10.350(c)(4). Participating CMS Providers deliver State/Local WEA Test Alerts through their networks “in a manner that complies with the Commission’s Alert Message requirements.” The Commission’s Alert Message Requirements are provided in Subpart D of the Part 10 WEA rules. See 47 CFR §§ 10.400 – 10.480. These requirements address classification, prioritization, message elements, character limit, embedded references, geo-targeting, and roaming. 5. WEA enables authorized federal, state, local, Tribal, and territorial government entities (alert originators) to geographically target alerts and warnings to members of the public that use WEA-capable mobile devices and are subscribers of a Participating CMS Provider. See 47 CFR § 10.10(f) (defining a Participating CMS Provider as “a Commercial Mobile Service Provider that has voluntarily elected to transmit Alert Messages under” the applicable WEA election requirements). As background regarding how this messaging works from a technical perspective, alert originators initiate WEA alerts by using FEMA-approved alert origination software to send their desired alert message to the FEMA-operated Integrated Public Alert and Warning System (IPAWS). There, the alert is authenticated, validated, and delivered to IPAWS’ Alert Gateway for dissemination to Participating CMS Providers’ own Alert Gateways. Participating CMS Providers then pass the alert message from their Alert Gateways to their co-located cell broadcast centers, which determine which CMS network facilities will be used to transmit the alert to the public. The CMS mobile core network (consisting of radio network controllers in 2G and 3G networks, mobility management entities (MMEs) in 4G networks, and access and mobility functions (AMFs) in 5G networks) then passes the alert to a constellation of cell sites near the geographic area targeted for the alert, which finally transmit WEA alerts over cellular broadcast-based control channels to mobile devices connected to their networks. See Wireless Emergency Alerts; Amendments to Part 11 of the Commission’s Rules Regarding the Emergency Alert System, PS Docket No. 15-91, 15-94, Second Report and Order and Second Order on Reconsideration, 33 FCC Rcd 1320, 1322, 1347-48, at para. 3 (2018) (WEA 2018 R&O). 6. While CMS Provider participation in WEA is voluntary, those CMS Providers that elect to participate in WEA must comply with the Commission’s rules governing the transmission and delivery of WEA messages. 47 CFR § 10.210 (WEA participation election procedures); 47 CFR § 10.10(k) (defining participation “in whole” as “CMS Providers that have agreed to transmit WEA Alert Messages in a manner consistent with the technical standards, protocols, procedures, and other technical requirements implemented by the Commission in the entirety of their geographic service area, and when all mobile devices that the CMS Providers offer at the point of sale are WEA-capable”); 47 CFR § 10.10(l) (defining participation “in part” as CMS Providers that have made the same agreement with respect to “some, but not in all of their geographic service areas, or CMS Providers that offer mobile devices at the point of sale that are not WEA-capable”). Among other requirements, the Commission has established geotargeting requirements to ensure that alerts are delivered only to those for whom the alert is relevant. Under Section 10.450(a) of the Commission’s rules, Participating CMS Providers must “deliver any Alert Message that is specified by a circle or polygon to an area that matches the specified circle or polygon. A Participating CMS Provider is considered to have matched the target area when they deliver an Alert Message to 100 percent of the target area with no more than 0.1 of a mile overshoot.” 47 CFR § 10.450(a) (“If some or all of a Participating CMS Provider’s network infrastructure is technically incapable of matching the specified target area, then that Participating CMS Provider must deliver the Alert Message to an area that best approximates the specified target area on and only on those aspects of its network infrastructure that are incapable of matching the target area. A Participating CMS Provider's network infrastructure may be considered technically incapable of matching the target area in limited circumstances, including when the target area is outside of the Participating CMS Provider’s network coverage area, when mobile devices have location services disabled, and when legacy networks or devices cannot be updated to support this functionality.”); see also WEA 2018 R&O, 33 FCC Rcd at 1327-28, para. 9 (“the requirement to match the target area applies only to new mobile devices offered for sale after November 30, 2019 and to existing devices capable of being upgraded to support this matching standard”); see also id. at 1328, para. 9 n.9 (“Existing mobile devices that cannot be upgraded to support enhanced geo-targeting will still be considered “WEA-capable” as of November 30, 2019, as long as the CMS Provider delivers Alert Messages to these devices using its “best approximation” of the target area.”); Public Safety and Homeland Security Bureau Announces that FEMA’s Integrated Public Alert and Warning System (IPAWS) is Ready to Support Certain Improvements to Wireless Emergency Alerts (WEA), PS Docket Nos. 15-91, 15-94, Public Notice, 34 FCC Rcd 12332 (PSHSB 2019) (announcing FEMA’s readiness to support enhanced WEA geo-targeting on December 19, 2019). According to CTIA, as of July 22, 2021, 34% of active smartphones support enhanced WEA geo-targeting, an increase of about 18 percentage points from the prior year. See Letter from Matthew Gerst, Vice President, Regulatory Affairs, CTIA, to Lisa Fowlkes, Bureau Chief, Public Safety and Homeland Security Bureau, Federal Communications Commission, PS Docket Nos. 15-91, 15-94, at 2 (Jul. 22, 2021) (CTIA 2021 WEA Geo-targeting Deployment Letter). According to PSHSB’s analysis, and as described in greater detail below, 12.3% of mobile devices used to participate in the nationwide WEA test survey supported enhanced WEA geo-targeting, less than the 34% that CTIA’s analysis would anticipate. To the extent that some of a Participating CMS Provider’s network infrastructure is technically incapable of matching its WEA delivery to the target area, then that Participating CMS Provider must deliver the Alert Message to its best approximation of the target area. 47 CFR § 10.450(a). CTIA projects that the majority of active smartphones will support enhanced WEA geo-targeting in 2022. CTIA 2021 WEA Geo-targeting Deployment Letter at 3. 7. The Commission’s rules also promote WEA’s accessibility and availability. WEA uses a unique audio attention signal and vibration cadence to make it accessible to all users, including individuals with disabilities and the elderly. WEA First Report and Order, 23 FCC Rcd at 6168, para. 65; 47 CFR § 10.520 (audio attention signal); 47 CFR § 10.530 (common vibration cadence). WEA-capable mobile devices must extract WEA alerts in the subscriber’s preferred language, 47 CFR § 10.500(e). present the alert on the device consistent with subscriber alerting preferences, 47 CFR § 10.500(f). detect and suppress duplicate alerts, 47 CFR § 10.500(g). and preserve alerts in a consumer-accessible format and location for at least 24 hours or until deleted by the subscriber. 47 CFR § 10.500(h). III. THE 2021 NATIONWIDE WEA TEST A. Parameters of the 2021 Nationwide WEA Test 8. The 2021 nationwide test of EAS and WEA was initiated by FEMA on August 11, 2021, at 2:20 pm Eastern Daylight Time. PSHSB Announces Nationwide Test of EAS and WEA, PS Docket Nos. 15-94, 15-91, Public Notice, DA 21-680, (PSHSB Jun. 11, 2021). The test message displayed on the wireless handsets of those subscribers who opted in to receive the test read as follows: “THIS IS A TEST of the National Wireless Emergency Alert System. No action is needed.” PSHSB Reminds EAS and Participating CMS Providers to Issue Accessible Alerts, PS Docket Nos. 15-94, 15-91, Public Notice, DA 21-798, (PSHSB Jul. 7, 2021). FEMA initiated the nationwide WEA test in both English and Spanish using the State/Local WEA Test Alert category. 47 CFR § 10.350(c). Subscribers whose mobile device language preference was set to Spanish received a nationwide WEA test message with the following text: “ESTA ES UNA PRUEBA del Sistema Nacional de Alerta de Emergencia. No se necesita acción.” As a result, unlike the 2018 nationwide WEA test, which was conducted using a national alert that is required to be displayed on all WEA-capable mobile devices, the 2021 nationwide WEA test message was displayed only on the devices of those members of the public who took affirmative steps to receive State/Local WEA Test Alerts. 47 CFR § 10.350(c)(4); see also WEA R&O and WEA FNPRM , 31 FCC Rcd at 11154 -55, para. 65. FEMA initiated the nationwide WEA test using a Federal Information Processing Standard (FIPS) geocode to describe the nationwide target area. As a result, the nationwide WEA test did not trigger enhanced WEA geo-targeting. 47 CFR § 10.450(a) (applying the enhanced WEA geo-targeting requirement only to geographic areas specified as a circle or polygon). The test followed the same dissemination path through Participating CMS Providers’ networks to the public as outlined above for WEA alerts. B. Participation in the 2021 Nationwide WEA Test 1. CMS Providers 9. FEMA initiated the 2021 nationwide WEA test by sending a State/Local WEA Test Alert to all Participating CMS Providers connected to the IPAWS Gateway. According to the Master WEA Registry, 80 CMS Providers connect to IPAWS, participating in WEA either “in whole” or “in part.” See Master WEA Registry, https://www.fcc.gov/files/weamasterregistry112019xls (last visited Nov. 12, 2021) (providing an account of the CMS Providers that participate in WEA based on PSHSB’s analysis of filings in the WEA election docket, PS Docket No. 08-146). 2. Leveraging Partnerships with Emergency Management Agencies to Gather Nationwide WEA Test Data 10. The FCC partnered with 11 federal, state and local emergency management agencies to gather information about the end user’s experience with the nationwide WEA test: FEMA; the National Weather Service; the Alabama Emergency Management Agency; Harris County (Texas) Office of Homeland Security & Emergency Management; City of Los Angeles (California) Emergency Management Department; New York City (New York) Emergency Management; Mendocino County (California) Office of Emergency Services; Ohio Emergency Management Agency; Oklahoma Department of Emergency Management and Homeland Security; City of Philadelphia (Pennsylvania) Office of Emergency Management; and the Utah Department of Public Safety. See News Release, FCC Acting Chairwoman Rosenworcel Announces Federal, State, and Local Partnerships to Assess Wireless Emergency Alert Test Performance (Jul. 20, 2021), https://docs.fcc.gov/public/attachments/DOC-374223A1.pdf. PSHSB notes that this report does not necessarily reflect the analysis or conclusions of the other partners. This first-of-its-kind approach allowed the FCC to gather data from dedicated volunteer respondents across the country on performance factors that Participating CMS Providers are not required to measure under the Commission’s current rules, such as the rate of test receipt on mobile devices, how long it takes for the test to reach mobile devices, and whether the test is presented as designed. Unlike the 2018 nationwide WEA test, the findings presented in this Report are based solely on responses to FCC’s Nationwide WEA Test Survey. 2,290 Nationwide WEA Test Survey respondents were located across the U.S. in 49 states and the District of Columbia (no respondents were located in Rhode Island). The FCC and its partners asked respondents to identify themselves with a control group number assigned to each partner and to specify the following: whether they received the test and at what time; the make, model, operating system, and wireless provider for their mobile device; their location at the time that they received the test; whether and what kind of cellular service their device had at the time they that received the test; whether their device was in use at the time they received the test; whether the test they received contained the correct test message; and whether they received a duplicate test message. See WEA Live Test Survey, OMB Control No. 3060-1269 (2021) (Nationwide WEA Test Survey). 11. PSHSB received 342 responses for which the respondent fully completed the Nationwide WEA Test Survey and used a WEA-compatible device capable of receiving State/Local Test Alerts. Many participants failed to include the control group number assigned to their sponsoring agency. PSHSB noted a significant discrepancy among results from those that reported a control group number versus those that did not. PSHSB believes that responses that failed to include a control group number may be unreliable, as a failure to include a control group number suggests that those respondents either did not have access to the Nationwide WEA Test Survey’s instructions or otherwise did not follow them. These respondents also may not have realized they needed to take affirmative measures to opt in to receive State/Local WEA Test Alerts, as explained in the instructions. Further, some Nationwide WEA Test Survey respondents participated in the test using older mobile devices that may be technically incapable of receiving the newer category of State/Local WEA Test Alerts although still capable of receiving other categories of WEA alerts (such as Imminent Threat Alerts). Such respondents may have been unaware their devices were not capable of receiving State/Local WEA Test Alerts and thus could not receive the nationwide WEA test. 12. In light of the foregoing, the analysis that follows focuses on the 342 respondents who fully completed the Nationwide WEA Test Survey and used a mobile device technically capable of receiving State/Local WEA Test Alerts. We refer to these respondents as the “completed survey response” group. Completed survey responses were received from 40 states and the District of Columbia (no completed survey responses were received from Vermont, South Dakota, Rhode Island, New Jersey, Nevada, Massachusetts, Delaware, Connecticut, Colorado, or Arkansas). These respondents are most likely to be representative of WEA’s actual performance during a real emergency, where consumers need not affirmatively opt in or use a newer handset model in order to receive a WEA message, and accordingly the analysis below is confined to this data set. Three respondents who completed surveys participated in the test using a mobile device that was not WEA capable (i.e., technically incapable of receiving any WEA test or alert message). These respondents’ responses are excluded from the data set. 3. CMS Provider Responses to Questions from PSHSB Bureau Chief 13. PSHSB submitted questions to three nationwide CMS Providers (AT&T, Verizon, and T-Mobile) eliciting information about their performance during the nationwide WEA test. See Letter from Lisa Fowlkes, Bureau Chief, Public Safety and Homeland Security Bureau, Federal Communications Commission, to Jaime (Mike) Tan, Director, Federal Regulatory, AT&T Services, Inc. (Jul. 20, 2021); Letter from Lisa Fowlkes, Bureau Chief, Public Safety and Homeland Security Bureau, Federal Communications Commission, to Robert Morse, Associate General Counsel, Verizon (Jul. 20, 2021); Letter from Lisa Fowlkes, Bureau Chief, Public Safety and Homeland Security Bureau, Federal Communications Commission, to Shellie Blakeney, Director, Federal Regulatory Affairs, T-Mobile USA, Inc. (Jul. 20, 2021). Responses from the CMS Providers were filed in PS Docket Nos. 15-91 and 15-94. The Commission asked these nationwide CMS Providers whether they received the test, what latencies the nationwide WEA test encountered at each step through their networks, how they transmitted the nationwide WEA test to the public through their cell broadcast facilities, and for a description of any complications that they experienced with test processing or transmission. See Letter from Lisa Fowlkes, Bureau Chief, Public Safety and Homeland Security Bureau, Federal Communications Commission, to Jaime (Mike) Tan, Director, Federal Regulatory, AT&T Services, Inc. (Jul. 20, 2021); Letter from Lisa Fowlkes, Bureau Chief, Public Safety and Homeland Security Bureau, Federal Communications Commission, to Robert Morse, Associate General Counsel, Verizon (Jul. 20, 2021); Letter from Lisa Fowlkes, Bureau Chief, Public Safety and Homeland Security Bureau, Federal Communications Commission, to Shellie Blakeney, Director, Federal Regulatory Affairs, T-Mobile USA, Inc. (Jul. 20, 2021). Their responses are presented in the context of our analysis of test performance below. PSHSB did not ask Participating CMS Providers to deploy their own volunteer network of respondents to report on their receipt of the nationwide WEA test. PSHSB also did not ask Participating CMS Providers for information describing how long it took for the nationwide WEA test to traverse the interface between cell sites and mobile devices, nor for how long it took the mobile devices to process and display the nationwide WEA test, nor for the percentage of mobile devices connected to their networks that received the test. Reporting on these performance factors may require Participating CMS Providers to extract information from mobile devices connected to their networks, which Participating CMS Providers currently are not required to do under the Commission’s rules. C. Test Performance 1. Reliability a. Overall 14. Based on the Nationwide WEA Test Survey data analysis, PSHSB finds that the nationwide WEA test was delivered to approximately 90% of WEA-capable mobile devices. This report generates findings that accurately describe, to the extent possible, the likely performance of mobile devices when an actual WEA alert is transmitted based on WEA’s performance during the nationwide WEA test, which was issued as a State/Local WEA Test Alert. Along with other adjustments to the data described herein, we exclude from the data set mobile devices that support the receipt of WEA alerts but do not support the State/Local WEA Test Alert. While this report finds that WEA alerts can be delivered to approximately 90% of WEA-capable mobile devices nationwide, we encourage state, local, Tribal, and territorial alert originators to confirm the Commission’s findings through independent testing within their alerting jurisdiction using State/Local WEA Test Alerts. To the extent that these alert originators wish to replicate the Commission’s survey, they may request additional information by contacting PSHSB at WEA@fcc.gov. This result should be representative of WEA’s reliability during a more typical state- or local-level activation because the State/Local WEA Test Alert that FEMA used to conduct the nationwide WEA test is required to be delivered “in a manner that complies with the Commission’s Alert Message requirements.” 47 CFR § 10.350(c). That the nationwide WEA test did not trigger enhanced WEA geo-targeting should not affect the reliability of the nationwide WEA test’s receipt on WEA-capable mobile devices within the nationwide target area. AT&T, Verizon, and T-Mobile each report that they received the nationwide WEA test and transmitted it to their subscribers in all geographic areas where they offer WEA coverage. See AT&T 2021 Nationwide WEA Test Letter at 1 (offering the caveat that, “due to a reporting issue, we do not have substantiating data from our broadcast message center (BMC)” to confirm AT&T’s belief that “the alert was transmitted successfully to all geographic areas”); Verizon 2021 Nationwide WEA Test Letter at 1; T-Mobile 2021 Nationwide WEA Test Letter at 2. Table 1 shows respondents’ success rate for test receipt. Table 1 Overall Test Reception Rates Completed Survey Response Results Test Received Test not Received Total Response Count 89.8% 10.2% 342 The data indicate that the nationwide WEA test was successfully transmitted: approximately 90% (89.8%) of respondents who completed the survey received the nationwide WEA test. As discussed above, PSHSB excludes from its analysis the 30 completed survey responses that specified that they participated in the test using a mobile device that was technically incapable of opting in to receive the State/Local WEA Test Alert. Sixteen respondents in the group stated that they were “not sure” whether they received WEA alert. To be conservative, we added these respondents to the list of respondents who reported not receiving WEA alert. While PSHSB cannot authoritatively state why 10.2% of mobile devices did not receive the nationwide WEA test, PSHSB examines potential reasons for the non-receipt of WEA alerts and recommends appropriate measures to improve WEA’s reliability below in Section IV. b. By Wireless Service Provider 15. Based on the Nationwide WEA Test Survey data, PSHSB finds that AT&T and Verizon offer a WEA service that is similarly reliable. This finding is statistically significant at the 5% significance level. See William Cochran, The X2 Test of Goodness of Fit, 23 The Annals of Mathematical Statistics 3, 334 (1952), https://www.jstor.org/stable/2236678. T-Mobile’s WEA service appears to be similarly reliable, but there are not enough completed survey respondents who identify T-Mobile as their CMS Provider for PSHSB to make a statistically significant finding in this regard. The Nationwide WEA Test Survey asked respondents to identify their wireless service provider. Table 2 shows how reliably the AT&T, T-Mobile, Verizon, and other wireless providers respectively delivered the nationwide WEA test. Table 2 Test Reception Rates By Wireless Service Provider Completed Survey Response Results Test Received Test not Received Total Response Count AT&T 89.9% 10.1% 99 T-Mobile 93.2% 6.8% 44 Verizon 91.0% 9.0% 156 Other wireless providers 81.4% 18.6% 43 Total 342 The completed survey data indicate that AT&T, Verizon, and T-Mobile respectively delivered the nationwide WEA test to 89.9%, 91.0%, and 93.2% of mobile devices. In addition to the exclusions described above, this analysis also excludes responses where the respondent reported being “unsure” of their service provider. Based upon PSHSB’s analysis of the data set, other wireless providers, including CMS Providers other than AT&T, T-Mobile, and Verizon, as well as Mobile Virtual Network Operators (MVNOs), See Communications Market Report, GN Docket No. 18-231, Report, 33 FCC Rcd 12558, 12562, para. 7 (2018) (defining MVNOs as entities that “do not own any network facilities, but instead purchase mobile wireless services wholesale from facilities-based providers and resell these services to consumers”). appear to have delivered the nationwide WEA test to about 10% fewer mobile devices than the nationwide Participating CMS Providers. c. By CMS Provider versus Mobile Virtual Network Operator 16. Based on the Nationwide WEA Test Survey data, PSHSB finds that some respondents received the nationwide WEA test while subscribed to MVNOs, rather than CMS Providers’ wireless services. There are different types of MVNOs, including those that rely on CMS Providers for some of their network functionalities, and those that rely on CMS Providers for all of their network functionalities. Table 3 shows the number of respondents who reported receiving the nationwide WEA test through an MVNO and what percentage of those respondents received the alert. Table 3 also shows how reliably MVNOs delivered the nationwide WEA test relative to CMS Providers. Table 3 Test Reception Rates By CMS Provider v MVNO Completed Survey Response Results Test Received Test not Received Total Response Count MVNO 82.1% 18.0% Note that the percentages expressed by the analyses in this report may not add up to exactly 100.0% due to rounding to the nearest tenth of a percent. 39 CMS Provider 90.8% 9.2% 303 Total 342 The completed survey data indicate that 32 respondents received the nationwide WEA test on a mobile device subscribed to an MVNO. The MVNO networks to which respondent mobile devices report being connected at the time they received the nationwide WEA test include Cricket Wireless, First Net, H20 Wireless, Pure Talk, Straight Talk, Boost Mobile, Consumer Cellular, Google Fi, Lycamobile, MetroPCS, Patriot Mobile, Pioneer Cellular, Q Link Wireless, Republic Wireless, Simple Mobile, Ting, Wal-Mart Family Mobile, Credo, Nemont, Page Plus, Spectrum, Total, TracFone, and Visible. Only one MVNO, Google’s Project Fi, has formally elected to participate in WEA. See Master WEA Registry, https://www.fcc.gov/files/weamasterregistry112019xls (last visited Nov. 12, 2021); see also Letter from Patrick Fonzo, Compliance Officer, Google North America Inc., to Marlene Dortch, Secretary, Federal Communications Commission, PS Docket No. 08-147 (Apr. 28, 2015). Based on these data, PSHSB finds that many more MVNOs are sending WEAs than have formally elected to participate in WEA. 17. Based upon analysis of this data set, MVNOs appear to have delivered the nationwide WEA test to about 10% fewer mobile devices than CMS Providers. d. By Generation of Wireless Network Technology 18. Based on the Nationwide WEA Test Survey data, PSHSB finds that there is no statistically significant difference between the reliability with which 4G and 5G wireless network technologies deliver WEA alerts. This finding is statistically significant at the 5% level. See William Cochran, The X2 Test of Goodness of Fit, 23 The Annals of Mathematical Statistics 3, 334 (1952), https://www.jstor.org/stable/2236678. The Nationwide WEA Test Survey asked respondents to identify the generation of wireless network technology to which their device was connected at the time FEMA sent the nationwide WEA test. Table 4 shows how reliably WEA performed over these wireless network technologies. Table 4 Test Reception Rates By Generation of Wireless Network Technology Completed Survey Response Results Test Received Test not Received Total Response Count 3G 33.3% 66.7% 6 4G 91.6% 8.4% 178 5G 90.7% 9.3% 97 Total 281 The completed survey data indicate that Participating CMS Provider support WEA using all wireless network technologies (2G, 3G, 4G, and 5G). 2G device performance was not sufficiently reported among respondents to reach a finding. In addition to the exclusions described, this analysis also excludes responses where the respondent did not specify the generation of network technology their phone was using at the test was performed. It also indicates that 91.6% of mobile devices that were connected to a 4G network received the nationwide WEA test as compared to 90.7% of mobile devices in the data set that were connected to a 5G network. The number of completed survey respondents who report having been connected to a 3G network at the time FEMA sent the nationwide WEA test, however, is not sufficient to make a statistically significant finding in this regard. e. By Mobile Device WEA Capabilities 19. The Nationwide WEA Test Survey asked respondents what wireless provider and mobile device make and model they used to participate in the nationwide WEA test. PSHSB used this information in conjunction with publicly available information provided by CMS Providers to determine which WEA capabilities those mobile devices support. See AT&T, Find out about Wireless Emergency Alerts, https://www.att.com/support/article/wireless/KM1009041 (last visited Aug. 15, 2021); Verizon, Wireless Emergency Alert Compatible Devices, https://www.verizon.com/support/wireless-emergency-alerts-compatible-devices/ (last visited Aug. 15, 2021); T-Mobile, Devices Tested for WEA Functionality, https://www.t-mobile.com/content/dam/t-mobile/pdf/TMobileWEA-4.pdf (last visited Aug. 15, 2021). A subset of mobile devices that CMS Providers offer as “WEA-capable” support the WEA requirements that the Commission adopted in 2016, including support for up to 360 characters of alphanumeric text, embedded references, Spanish-language alerts, and, critically, the State/Local WEA Test Alert used for this nationwide WEA test. A smaller subset of those devices also support for enhanced WEA geo-targeting. Table 5 shows the rate at which respondents received the nationwide WEA test by CMS Provider-identified WEA capabilities. Table 5 Test Reception Rates By CMS Provider-identified Mobile Device Version Completed Survey Response Results Test Received Test not Received Total Response Count 2016 WEA Capabilities 88.4% 11.6% 301 Full WEA Capabilities 100% 0.0% 41 Total 342 The completed survey data indicate that 88.4% of mobile devices with 2016 WEA capabilities received the nationwide WEA test and that 100% of mobile devices with full WEA capabilities received the nationwide WEA test. In addition to the exclusions described above, this analysis also excludes responses where the respondent reported being “unsure” of the model of their mobile device, which therefore prevented PSHSB from determining the WEA capabilities of the phone. Based on the analysis of this data set, mobile devices with full WEA capabilities appear to have delivered the nationwide WEA test more reliably than mobile devices with 2016 WEA capabilities. PSHSB will continue to evaluate the reliability of various WEA-capable mobile devices and assess whether less capable devices, including the oldest WEA-capable devices, have a significantly lower rate of WEA alert reception. f. By Mobile Device Manufacturer 20. Based on the Nationwide WEA Test Survey data, PSHSB finds no statistical difference between the performance of WEA among different brands of mobile device. This is statistically significant at the 5% significance level. See William Cochran, The X2 Test of Goodness of Fit, 23 The Annals of Mathematical Statistics 3, 334 (1952), https://www.jstor.org/stable/2236678. The Nationwide WEA Test Survey asked respondents to identify their mobile device manufacturer. Table 6 shows how reliably different brands of mobile device delivered the nationwide WEA test. Table 6 Test Reception Rates By Device Manufacturer Completed Survey Response Results Test Received Test not Received Total Response Count Apple 90.4% 9.6% 228 Samsung 86.8% 13.2% 76 Other 93.8% 6.3% 32 Total 336 The completed survey data indicate that 90.4% of Apple devices, 86.8% of Samsung devices, and 93.8% of other devices received the nationwide WEA test. In addition to the exclusions described above, this analysis also excludes responses where the respondent reported being “unsure” of the brand of their mobile device. The public should not expect to improve the likelihood that they will receive a WEA alert based on the brand of mobile device that they use. g. By Device Operating System 21. Based on the Nationwide WEA Test Survey data, PSHSB finds no statistical difference between the performance of different mobile device operating systems. The nationwide WEA test survey asked respondents to identify their mobile device operating system. Table 7 shows how reliably different mobile device operating systems delivered the nationwide WEA test. Table 7 Test Reception Rates By Device Operating System Completed Survey Response Results Test Received Test not Received Total Response Count iOS 90.4% 9.6% 230 Android 88.1% 11.9% 109 Total 339 The completed survey data indicate that 90.4% of mobile devices with an iOS operating system and that 88.1% of mobile devices with an Android operating system received the nationwide WEA test. In addition to the exclusions described above, this analysis also excludes responses where the respondent reported being “unsure” of the operating system of their mobile device. The public should not expect to improve the likelihood that they will receive a WEA alert based on their mobile device’s operating system. h. By Location (Indoor/Outdoor) 22. Based on the Nationwide WEA Test Survey data, PSHSB finds that whether a mobile device is indoors or outdoors does not have a statistically significant effect on its ability to receive a WEA alert. The Commission’s Office of Economic Analysis performed statistical analysis to generate this result and finds it to be insignificant. There is no statistical test for similarity, however. The Nationwide WEA Test Survey asked respondents to identify whether they were located indoors or outdoors at the time FEMA sent the nationwide WEA test. Table 8 shows how reliably the nationwide WEA test was delivered to respondents located indoors and outdoors. Table 8 Test Reception Rates By Location (Indoor/Outdoor) Completed Survey Response Results Test Received Test not Received Total Response Count Indoors 90.1% 9.9% 294 Outdoors 88.1% 11.9% 42 Total 336 The completed survey data indicate that 90.1% of indoor mobile devices received the nationwide WEA test as compared to 88.1% of outdoor mobile devices. In addition to the exclusions described above, this analysis also excludes responses where the respondent did not report whether they were indoors. i. By Whether Device Was In Use 23. Based on the Nationwide WEA Test Survey data, PSHSB finds no statistical difference between the performance of mobile devices based on whether they are engaged in an active voice or data session at the time they receive a WEA alert. PSHSB performed statistical analysis to generate this result and finds it to be significant. There is no statistical test for similarity, however. See Sebastian Georg, Nonparametric Testing of Distributions—the Epps–Singleton Two-Sample Test Using the Empirical Characteristic Function, 9 The Strata Journal 3, 454-465 (2009). The Commission’s WEA rules prohibit WEA-capable mobile devices from allowing a WEA alert to preempt an active voice or data session. 47 CFR § 10.510. Instead, WEA-capable mobile devices should present WEA alert simultaneous with ongoing voice or data sessions. Provided that the mobile device is technically capable of being tuned to the control channel and the voice or data network simultaneously. See ATIS, Wireless Emergency Alert (WEA) 3.0 Mobile Device Behavior (MDB) Specification, ATIS-0700036.v002, at 14 (2019); see also WEA R&O and WEA FNPRM, 31 FCC Rcd at 11152, para 59 (observing that, unlike legacy mobile devices, 4G-LTE-capable mobile devices could receive WEA alerts as soon as they are available, even while the device is engaged in an active voice or data session, because such devices can be simultaneously tuned to the voice or data network and the control channel that carries WEA alerts); Letter from Thomas Goode, General Counsel, ATIS, to James Wiley, Attorney Adviser, Cybersecurity and Communications Reliability Division, Public Safety and Homeland Security Bureau, Federal Communications Commission, PS Docket Nos. 15-91, 15-94 (Jun. 14, 2021). The nationwide WEA test survey asked respondents to identify whether their mobile device was in use at the time FEMA sent the nationwide WEA test. Table 9 shows how reliably mobile devices presented the nationwide WEA test while engaged in an active voice or data session. Mobile devices that presented the nationwide WEA test during an active voice or data session displayed the nationwide WEA test to the user while they were using their mobile device for another purpose. Table 9 Test Reception Rates By Device Usage Completed Survey Response Results Test Received Test not Received Total Response Count In-use 95.6% 4.4% 69 Not in-use 89.5% 10.5% 267 Total 336 The completed survey data indicate that 95.6% of mobile devices received the nationwide WEA test while in use, as compared to 89.5% of mobile devices that received the nationwide WEA test while not in use. 2. Latency a. CMS Provider Network Latency Analysis 24. The Commission does not require CMS Providers to deliver WEA alerts to the public within a particular timeframe from when they receive it from the IPAWS gateway. The nationwide CMS Providers’ responses, however, illuminate how WEA alerts transit the nationwide CMS Providers’ networks today from the alert gateway to the cell site, and how long each step in that process takes. AT&T reports that the nationwide WEA test traversed its network infrastructure from alert gateway to cell site in approximately 41.1 seconds. See generally AT&T 2021 Nationwide WEA Test Letter. It took 40 seconds for AT&T’s broadcast message center to process the nationwide WEA test and send it to all radio access network entities nationwide. It took under 100ms for AT&T’s Cell Radio Network Controllers/Mobility Management Entities to process and send the alert to its cell sites. It took AT&T’s cell sites approximately 1 second to process and send the nationwide WEA test to AT&T’s subscribers. Taken together, these latencies (40 seconds + 100 milliseconds + 1 second) equal 41.1 seconds. Verizon reports that the nationwide WEA test traversed its network infrastructure from alert gateway to cell site in 55.306-55.769 seconds. See generally Verizon 2021 Nationwide WEA Test Letter. Verizon provided its network latency data differently than AT&T and T-Mobile, so PSHSB calculated end-to-end latency on Verizon’s network differently from AT&T and T-Mobile. According to Verizon, on the day of the nationwide WEA test, its network received the nationwide WEA test from FEMA at 14:20:05,969 and first transmitted it to Verizon’s subscribers at 14:21:01,274. The difference is 55.305 seconds, the minimum delay. Verizon reports that its network last sent the nationwide WEA test at 14:21:01,738, yielding a maximum delay of 55:769 seconds. Therefore, the nationwide WEA test traversed Verizon’s network infrastructure from alert gateway to cell site in 55.305 to 55.769 seconds. T-Mobile reports that the nationwide WEA test traversed its network infrastructure from alert gateway to cell site in 36.679 seconds. See generally T-Mobile 2021 Nationwide WEA Test Letter. It took 35.895 seconds for T-Mobile’s WEA platform to process and transmit the nationwide WEA test to T-Mobile’s Radio Network Controllers/Mobility Management Entities/Access and Mobility Function. It took under 00:144 seconds (144 milliseconds) for T-Mobile’s Cell Radio Network Controllers/Mobility Management Entities to process and send the nationwide WEA test to its cell sites. It took T-Mobile’s cell sites less than 1 second to process and send the nationwide WEA test to T-Mobile’s subscribers. Taken together, these latencies (35.895 seconds + 144 milliseconds + 1 second) equal 36.679 seconds. Figure 1, below, illustrates the alert transmission steps about which CMS Providers were able to provide latency data (Wireless Service Provider-Identified Latency), the portion of WEA’s end-to-end transmission path about which this report seeks to present latency data by supplementing CMS Providers’ responses with data from Nationwide WEA Test Survey respondents (Overall Latency Assessed by This Report), and the full, end-to-end transmission path of a WEA alert, which further testing would be necessary to assess. Figure 1: WEA End-to-End Service Architecture 25. As illustrated by Figure 1, above, each of the nationwide CMS Providers received the nationwide WEA test from IPAWS at a logical alert gateway co-located with the CMS Providers’ cell broadcast center (known collectively as a “broadcast message center” or “WEA platform”). See AT&T 2021 Nationwide WEA Test Letter at 1; Verizon 2021 Nationwide WEA Test Letter at 1; T-Mobile 2021 Nationwide WEA Test Letter at 2. Nearly all of the latency in the nationwide CMS Providers’ processing of the nationwide WEA test occurred at this step, where their cell broadcast centers determined which CMS elements in their nationwide network to use to transmit the nationwide WEA test. Then, each of the nationwide CMS Providers’ co-located alert gateways and cell broadcast centers sent the nationwide WEA test to their respective Mobile Core Networks (radio network controllers (3G) mobility management entities (4G) and/or Access and Mobility Functions (5G), depending on their network configuration). The radio network controller and the mobility management entity are mainly responsible for interpretation of commands from the cell broadcast center, storage of messages from the cell broadcast center, providing to the cell broadcast center acknowledgement of successful execution of commands received from the cell broadcast center, reporting to the cell broadcast center failure when a command received from the cell broadcast center is not understood or cannot be executed, and routing of alert messages to the appropriate distribution area. Radio network controllers interface to only one cell broadcast center, whereas the mobility management entity may interface to one cell broadcast center or multiple cell broadcast centers. AT&T’s broadcast message center took 40 seconds to process and retransmit the alert. See AT&T 2021 Nationwide WEA Test Letter at 2. Verizon’s two geographically distinct, collocated alert gateway and cell broadcast center facilities took 55.305 seconds and 55.769 seconds, respectively, to process and transmit the alert. See Verizon 2021 Nationwide WEA Test Letter at 2. The time it took for the nationwide WEA test to transit Verizon’s network is expressed as a range between two values because, unlike the other nationwide CMS Providers, Verizon’s 2021 Nationwide WEA Test Letter provides times for both the first and last transmission of the nationwide WEA test from its Mobility Management Entities. T-Mobile’s WEA platform center took 35.895 seconds to process and transmit the nationwide WEA test. See T-Mobile 2021 Nationwide WEA Test Letter at 2. 26. In 2016, a vendor of commercial mobile service network equipment stated that it takes their cell broadcast center product 0.1 seconds (100 milliseconds) to map distribution of a WEA alert onto 10,000 cell sites. See Letter from Kim Robert Scovill, Vice President, Legal, Regulatory, and External Affairs, Comtech TCS, to Marlene Dortch, Secretary, Federal Communications Commission, PS Docket No. 16-32 at 1 (Aug. 11, 2016). The nationwide CMS Providers confirm that the performance of their WEA platforms was within normal parameters during the nationwide WEA test, and attribute the extent of the WEA platform’s delay in mapping the nationwide WEA test to network cell sites to the exponentially larger number of cell sites needed to transmit a WEA test nationwide than for a state- or local-level activation, and to the fact that the nationwide WEA test had both English and Spanish versions. See Letter from Robert Morse, Associate General Counsel, Verizon, to Lisa Fowlkes, Chief, Public Safety and Homeland Security Bureau, Federal Communications Commission, PS Docket Nos. 15-91, 15-94, at 1 (Oct. 27, 2021); Letter from Jaime (Mike) Tan, Director, Federal Regulatory, AT&T, to Lisa Fowlkes, Chief, Public Safety and Homeland Security Bureau, Federal Communications Commission, PS Docket Nos. 15-91, 15-94, at 1 (Oct. 29, 2021); Letter from Shellie Blakeney, Director, Federal Regulatory Affairs, T-Mobile, to Lisa Fowlkes, Chief, Public Safety and Homeland Security Bureau, Federal Communications Commission, PS Docket Nos. 15-91, 15-94, at 1 (Nov. 8, 2021); see also Phone call from Nicole McGinnis, Deputy Chief, Public Safety and Homeland Security Bureau, Federal Communications Commission, to Steve Sharkey, Vice President, Government Affairs, T-Mobile (Nov. 9, 2021, 2:00 pm EST). The 0.1 second (100 millisecond) delay attendant to mapping a WEA alert to 10,000 cell sites may be more representative of how nationwide CMS Providers’ WEA platforms would perform during a state- or local-level activation because state- and local-level WEA activations require CMS Providers’ WEA platforms to map WEA alerts to fewer cell sites. 27. Next, in a matter of milliseconds, the Participating CMS Providers’ Mobile Core Networks processed the test and transmitted it to their respective networks’ cell sites (NodeBs (3G), eNodeBs (4G), or gNBs (5G)). AT&T’s network did not capture data for the time it took their Mobile Core Network to process the test and send it to AT&T’s cell sites, but AT&T states that the process generally takes less than 0.1 seconds (100 milliseconds). See AT&T 2021 Nationwide WEA Test Letter at 2. Verizon’s Mobile Core Network processed and sent the nationwide WEA test to Verizon’s cell sites in between 0.001-0.003 seconds (1-3 milliseconds). See Verizon 2021 Nationwide WEA Test Letter at 2. T-Mobile’s Mobile Core Network processed the test and sent it to T-Mobile’s cell sites within 00:144 seconds (144 milliseconds). See T-Mobile 2021 Nationwide WEA Test Letter at 2. 28. Next, the nationwide CMS Providers sent the nationwide WEA test from their cell sites to customers on their respective networks. AT&T states that it takes its cell sites “approximately 1 second” to transmit WEA alerts to customers on their network. See AT&T 2021 Nationwide WEA Test Letter at 2. Verizon states that its cell sites would have processed and sent the test “within a matter of milliseconds.” See Verizon 2021 Nationwide WEA Test Letter at 2. T-Mobile states that its cell sites transmitted the test message in “less than 1 second.” See T-Mobile 2021 Nationwide WEA Test Letter at 2. Each nationwide CMS Provider takes a different approach to retransmitting WEA alerts from their cell sites. The nationwide CMS Providers’ approaches to retransmission are network policies that apply to all WEA alert and test messages, irrespective of their classification. AT&T retransmits the nationwide WEA test 30 times, once a minute for the 30-minute duration of the test. See AT&T 2021 Nationwide WEA Test Letter at 3. Verizon transmits the test message only twice on all networks, once upon receipt from IPAWS and then again 1 minute later. See Verizon 2021 Nationwide WEA Test Letter at 3. T-Mobile rebroadcast the test message 6 times over its UMTS/4G/5G networks at 5 minute intervals, and 8 times over its GSM network at 4 minute intervals. See T-Mobile 2021 Nationwide WEA Test Letter at 3. AT&T and Verizon state that they do not capture data about the latencies attendant to the processing and transmission of the nationwide WEA test beyond their cell broadcast facilities. See AT&T 2021 Nationwide WEA Test Letter at 2; Verizon 2021 Nationwide WEA Test Letter at 2. b. Latency Overall 29. Based on the Nationwide WEA Test Survey data, PSHSB finds that the majority of respondents that received the nationwide WEA test received it within two minutes of transmission from FEMA’s alert gateway. Based on the Nationwide WEA Test Survey data, PSHSB is 95% confident that the median latency value for another nationwide WEA test conducted under identical conditions would be 1 minute. This nationwide WEA test represents the first time that the Commission has formally attempted to measure how long it takes nationwide WEA tests to reach the public. It becomes more difficult for the nationwide CMS Providers to measure the timing of the nationwide WEA test once they transmit it from their cell sites because they do not capture data about message transmission in the air interface between their cell sites and mobile devices or on mobile devices connected to their networks. PSHSB treats the latency between cell sites and mobile devices as negligible because of the speed of radio frequency communication. Latencies at the level of mobile device processing may vary by device, but it is expected to be on the order of tens of milliseconds. The timing of the presentation of the WEA alert at the mobile device may vary by mobile device. 30. PSHSB’s ability to capture accurate latency data from the air interface and end-user devices is also limited. PSHSB relied on volunteer respondents to note the time indicated on their mobile device upon receipt of the State/Local WEA Test Alert, and to record that time in PSHSB’s Nationwide WEA Test Survey. Reliance on a human being to record the time that they receive a test implicitly introduces the possibility of inaccuracy, and the time presented on mobile devices is accurate to the minute, not the second. Further, anecdotal reports reflect that time presented on mobile devices can vary slightly by wireless provider. Nonetheless, for respondents that received the nationwide WEA test, the survey asked respondents to identify the time at which they received the test to the nearest minute. Table 10 shows how quickly respondents report receiving the nationwide WEA test on their mobile devices from the minute that FEMA transmitted it from its alert gateway (2:20 pm EDT). Table 10 Completed Survey Response Results Overall Latency (Minutes) Respondents’ Observations 25th percentile Median (50th percentile) 90th percentile 99th percentile 295 0 1 5 208 Min Mode Mean Max 0 1 4.48 241 The completed survey data indicate that at least half of the WEA-capable mobile devices that received the nationwide WEA test received it within two minutes of FEMA’s transmission. Where values shown in this report’s latency tables represent alert receipt within the stated minute (e.g., “0” in the table represents that the nationwide WEA test was received between 0 and 59 seconds from FEMA’s transmission; “1” in the table represents that the nationwide WEA test was received between 1 minute and 1 minute and 59 seconds from FEMA’s transmission). At least 25% of these devices received the nationwide WEA test in less than one minute (represented as 0). At least 90% of these devices received the nationwide WEA test in less than 6 minutes. The other 10% of these devices took longer, with a maximum of 241 minutes (4 hours and 1 minute) after FEMA’s transmission. 31. The time that it took for the nationwide WEA test to reach the public from FEMA’s alert gateway is not necessarily representative of how long it would take a typical WEA alert to reach the public. First, because FEMA, rather than a state or local emergency management agency, initiated the nationwide WEA test, the nationwide WEA test could not be used to measure the latency of a typical WEA alert from alert origination software to IPAWS or within IPAWS itself. Second, the nationwide WEA test did not trigger enhanced WEA geo-targeting because its target area was specified with a nationwide FIPS code rather than with circles or polygons. Upon receipt of a WEA alert with a target area specified by circles or polygons, mobile devices capable of enhanced WEA geo-targeting can potentially take from 0 to 255 seconds (4 minutes and 15 seconds) to determine their location, compare their location to the targeted area, and display the alert only if their location is within 0.1 miles of the targeted area. See ATIS, Wireless Emergency Alerts (WEA) 3.0 Device-Based Geo-Fencing, ATIS 0700041, at 10 (2019). Finally, as discussed above, CMS networks needed to map the nationwide WEA test to significantly more cell sites than a typical WEA alert, which would have a more limited geographic target area. PSHSB will continue to examine how quickly mobile devices can receive typical WEA alerts. c. By Wireless Provider 32. Based on the Nationwide WEA Test Survey data, PSHSB finds that AT&T’s network delivered the nationwide WEA test at a speed that was statistically different and quicker than Verizon’s network. This finding is statistically significant at the 10% significance level but not the 5% significance level. See William Cochran, The X2 Test of Goodness of Fit, 23 The Annals of Mathematical Statistics 3, 334 (1952), https://www.jstor.org/stable/2236678. T-Mobile’s delivery of the nationwide WEA test appears to be faster than both AT&T’s and Verizon’s but there are not enough completed survey respondents who identify T-Mobile as their CMS Provider to make a statistically significant finding in this regard. Table 11 shows how quickly mobile devices received the nationwide WEA test from the minute that FEMA transmitted it from its alert gateway. Table 11 Completed Survey Response Results Latency by Wireless Provider (Minutes) Respondent Observations Mean 25th % Median (50th %) 90th % 99th % Min Max Mode AT&T 85 1.34 0 1 2 29 0 29 0 T-Mobile 39 2.28 0 0 3 40 0 40 0 Verizon 137 7.85 1 1 4 231 0 241 1 Other 34 1.29 0 1 2 18 0 18 1 Total 295 The completed survey data indicate that at least half of the WEA-capable mobile devices that were connected to AT&T’s and Verizon’s networks and received the nationwide WEA test received it within two minutes of transmission. Verizon’s network took statistically longer than other wireless providers to deliver the nationwide WEA test. The Commission’s Office of Economics and Analytics performed a log rank test to establish this finding. Theodore G. Karrison, Versatile Tests for Comparing Survival Curves Based on Weighted Log-Rank Statistics, The Stata Journal, https://www.stata-journal.com/article.html?article=st0449. The log rank test is the standard test to determine whether one group took longer to perform an action than another. At least 90% of mobile devices connected to Verizon’s network received the nationwide WEA test within five minutes. At least 90% of mobile devices connected to AT&T’s network received the nationwide WEA test within three minutes. Although it appears that at least 90% of mobile devices connected to T-Mobile’s network also received the nationwide WEA test within three minutes, there are not enough completed survey responses to reach a statistically significant finding in this regard. 33. AT&T’s and Verizon’s own network latency analyses explain an approximately 15 second difference between the speed at which their networks delivered the nationwide WEA test. However, PSHSB cannot explain the remainder of this difference with confidence except to attribute it to a step in the test transmission process on which AT&T and Verizon did not report: the speed at which the CMS Providers were able to deliver the test across their wireless networks, given their respective retransmission policies. An additional step in the test transmission process on which AT&T and Verizon did not report is the time that it takes mobile devices in receipt of a WEA test to present the test message to the public, but PSHSB does not attribute this difference in overall transmission time to differences in mobile device performance because mobile devices were evenly distributed across AT&T’s and Verizon’s networks in the data set. PSHSB finds therefore that it is likely that AT&T’s network was able to deliver the nationwide WEA test across its wireless network, on average, faster than Verizon, given that AT&T retransmitted the nationwide WEA alert once a minute for the duration of the nationwide WEA test’s 30-minute active period and Verizon retransmitted the nationwide WEA test only once, one minute after its initial transmittal. CMS Providers’ retransmission periodicity can only explain difference in test message delivery during the test’s 30-minute active period. Delivery of the nationwide WEA test reported outside of this 30-minute window (e.g., 231 or 241 minutes after FEMA’s transmission) are not explained by this policy difference alone. d. By Generation of Wireless Network Technology 34. Based on the Nationwide WEA Test Survey data, PSHSB finds no statistically significant difference between the time it takes 4G and 5G networks to deliver WEA alerts. The Commission’s Office of Economics and Analytics performed a log rank test to establish this finding. Theodore G. Karrison, Versatile Tests for Comparing Survival Curves Based on Weighted Log-Rank Statistics, The Stata Journal, https://www.stata-journal.com/article.html?article=st0449. Table 12 shows how quickly mobile devices received the nationwide WEA test from the minute that FEMA transmitted it from its alert gateway. Table 12 Completed Survey Response Results Latency by Generation of Wireless Network Technology (Minutes) Respondent Observations Mean 25th % Median (50th %) 90th % 99th % Min Max Mode 4G 158 5.39 0 1 3 208 0 231 1 5G 86 4.12 0 1 2 241 0 241 0 The completed survey data indicate that at least half of the respondents on 4G and 5G networks that received the nationwide WEA test received it within two minutes of transmission. At least 90% of mobile devices on 4G networks that received the nationwide WEA test received it within four minutes and at least 90% of mobile devices on 5G networks that received the nationwide WEA test received it within three minutes. 3. Accessibility 35. No respondent reported any issue with the accessible presentation of the nationwide WEA test, including the audio attention signal, the vibration cadence, and the visual presentation of the test message. Prior to the nationwide WEA test, PSHSB reminded CMS Providers to take the necessary steps to ensure the accessibility of WEA messages to individuals with disabilities. See Public Safety and Homeland Security Bureau Reminds EAS Participants and Participating CMS Providers of Requirements to Issue Accessible EAS and WEA Alerts, Public Notice, PS Docket Nos. 15-91, 15-94, DA 21-798 (PSHSB rel. Jul. 7, 2021). PSHSB’s nationwide WEA test survey asked respondents whether the text of the message that they received matched the text of the test that FEMA initiated. The survey did not ask respondents to identify any other issues with the presentation of the nationwide WEA test. The survey did, however, include a field for “other comments” that permitted respondents to identify whether they experienced any issues with the test, including presentation issues. D. Analysis of Reported Challenge with Duplicate Messages 36. Wireless providers participating in the nationwide WEA test erroneously delivered a second, duplicative test message to many mobile devices. The nationwide WEA test survey asked respondents whether they received the State/Local WEA Test Alert more than once. Table 13 shows, by CMS Provider, the rate at which respondents that report receiving the nationwide WEA test also report receiving a duplicate. Table 13 Duplicate Test Reception Rates Overall and by Wireless Provider Completed Survey Response Results Duplicate Test Received Duplicate Test not Received Total Response Count AT&T 88.0% 12.0% 83 T-Mobile 7.5% 92.5% 40 Verizon 2.9% 97.1% 137 Other 41.2% 58.8% 34 Overall total 32.0% 68.0% 294 The completed survey data indicate that 32.0% of respondents that received the nationwide WEA test also received a duplicate test message. In addition to the exclusions described above, this analysis also excludes responses that did not receive the nationwide WEA test once. 88.0% of respondents that received the nationwide WEA test on AT&T’s network also received a duplicate test message; 7.5% of respondents that received the nationwide WEA test on T-Mobile’s network also received a duplicate test message; 2.9% of respondents that received the nationwide WEA test on Verizon’s network also received a duplicate test message; and 41.2% of respondents that received the nationwide WEA test on other wireless providers’ networks also received a duplicate test message. 37. PSHSB examined this duplicate test message issue. PSHSB confirmed with AT&T that a software issue caused much of the duplicate message delivery on their network. AT&T states that, to its knowledge, “this software issue has not affected any live WEA alerts.” Letter from Jaime (Mike) Tan, Director, Federal Regulatory, AT&T, to Lisa Fowlkes, Chief, Public Safety and Homeland Security Bureau, FCC, PS Docket Nos. 15-91, 15-94, at 1 (filed Aug. 25, 2021). AT&T has brought this issue to the attention of the software vendor and reports that, once it is resolved, “the nationwide duplication issue experienced by some users is not expected to recur.” Id. 38. AT&T’s software failure, however, does not explain duplicate message delivery on Verizon, T-Mobile, and other wireless providers’ networks. With respect to those networks’ duplicate messages, T-Mobile explains that mobile devices perform duplicate alert suppression based on a serial number assigned to the alert at the broadcast message center. See Letter from Shellie Blakeney, Director, Federal Regulatory Affairs, T-Mobile, to Lisa Fowlkes, Chief, Public Safety and Homeland Security Bureau, Federal Communications Commission, PS Docket Nos. 15-91, 15-94, at 1 (Nov. 8, 2021); see also Phone call from Nicole McGinnis, Deputy Chief, Public Safety and Homeland Security Bureau, Federal Communications Commission, to Steve Sharkey, Vice President, Government Affairs, T-Mobile (Nov. 9, 2021, 2:00 pm EST). When a CMS Provider retransmits an alert during the alert’s active period to promote WEA’s reliability, it assigns the same serial number to subsequent iterations of the same alert. See Letter from Shellie Blakeney, Director, Federal Regulatory Affairs, T-Mobile, to Lisa Fowlkes, Chief, Public Safety and Homeland Security Bureau, Federal Communications Commission, PS Docket Nos. 15-91, 15-94, at 1 (Nov. 8, 2021); see also Phone call from Nicole McGinnis, Deputy Chief, Public Safety and Homeland Security Bureau, Federal Communications Commission, to Steve Sharkey, Vice President, Government Affairs, T-Mobile (Nov. 9, 2021, 2:00 pm EST). When a mobile device receives a second iteration of an alert that contains the same serial number, it suppresses the second iteration as a duplicate and does not present it to the device’s user. See Letter from Shellie Blakeney, Director, Federal Regulatory Affairs, T-Mobile, to Lisa Fowlkes, Chief, Public Safety and Homeland Security Bureau, Federal Communications Commission, PS Docket Nos. 15-91, 15-94, at 1 (Nov. 8, 2021); see also Phone call from Nicole McGinnis, Deputy Chief, Public Safety and Homeland Security Bureau, Federal Communications Commission, to Steve Sharkey, Vice President, Government Affairs, T-Mobile (Nov. 9, 2021, 2:00 pm EST). Each wireless provider, however, may assign a different serial number to the same alert. See Letter from Shellie Blakeney, Director, Federal Regulatory Affairs, T-Mobile, to Lisa Fowlkes, Chief, Public Safety and Homeland Security Bureau, Federal Communications Commission, PS Docket Nos. 15-91, 15-94, at 1 (Nov. 8, 2021); see also Phone call from Nicole McGinnis, Deputy Chief, Public Safety and Homeland Security Bureau, Federal Communications Commission, to Steve Sharkey, Vice President, Government Affairs, T-Mobile (Nov. 9, 2021, 2:00 pm EST). As a result, if a user is roaming across networks during the active period of an alert, they may receive one iteration of an alert with one serial number on their home wireless network, and then receive a second, duplicate iteration of that same alert with a different serial number on a roaming network’s subsequent retransmission. See Letter from Shellie Blakeney, Director, Federal Regulatory Affairs, T-Mobile, to Lisa Fowlkes, Chief, Public Safety and Homeland Security Bureau, Federal Communications Commission, PS Docket Nos. 15-91, 15-94, at 1 (Nov. 8, 2021); see also Phone call from Nicole McGinnis, Deputy Chief, Public Safety and Homeland Security Bureau, Federal Communications Commission, to Steve Sharkey, Vice President, Government Affairs, T-Mobile (Nov. 9, 2021, 2:00 pm EST). The capacity for users to receive duplicate WEA alerts while roaming appears to be a systemic issue that predictably leads end users to receive duplicative alerts. 39. An additional potential explanation for some users’ experience of a duplicate alert is that some mobile devices may trigger their attention signal and vibration cadence a second time if the device user does not tap to view the WEA message notification when the notification is first displayed. See Letter from Shellie Blakeney, Director, Federal Regulatory Affairs, T-Mobile, to Lisa Fowlkes, Chief, Public Safety and Homeland Security Bureau, Federal Communications Commission, PS Docket Nos. 15-91, 15-94, at 1 (Nov. 8, 2021); see also Phone call from Nicole McGinnis, Deputy Chief, Public Safety and Homeland Security Bureau, Federal Communications Commission, to Steve Sharkey, Vice President, Government Affairs, T-Mobile (Nov. 9, 2021, 2:00 pm EST). This would not be an example of a duplicate alert, per se, but it might be interpreted as a duplicate alert. This additional presentation of WEA’s audio attention signal and vibration cadence is optional in WEA’s mobile device standards, and its implementation may vary by device. See Letter from Shellie Blakeney, Director, Federal Regulatory Affairs, T-Mobile, to Lisa Fowlkes, Chief, Public Safety and Homeland Security Bureau, Federal Communications Commission, PS Docket Nos. 15-91, 15-94, at 1 (Nov. 8, 2021); see also Phone call from Nicole McGinnis, Deputy Chief, Public Safety and Homeland Security Bureau, Federal Communications Commission, to Steve Sharkey, Vice President, Government Affairs, T-Mobile (Nov. 9, 2021, 2:00 pm EST). As discussed in further detail below, PSHSB continues to evaluate the duplicate alert issue and make recommendations to address it, where appropriate. IV. NEXT STEPS 40. The Commission places the highest priority on ensuring that WEA continues to be an effective alerting tool and that alerts are received in a timely manner, enabling prompt and potentially life-saving action on the part of the public. To further improve WEA based on its performance during the 2021 nationwide test, the Bureau plans to take the following actions (or, where applicable, recommend action by the Commission or external parties): · Benchmark WEA’s performance at demonstrated levels. PSHSB will submit this report into the WEA record, PS Docket No. 15-91, so that it can inform the Commission’s future consideration of WEA’s performance. Since 2016, the Commission has sought comment on whether it would be feasible for Participating CMS Providers to report to the Commission information about the reliability and latency of the WEA service they offer, and whether the Commission should adopt performance metrics, such as benchmarks regarding the extent to which WEA alerts have been received. See WEA R&O and FNPRM, 31 FCC Rcd at 11210, para. 161; Amendment of Part 11 of the Commission’s Rules Regarding the Emergency Alert System; Wireless Emergency Alerts, PS Docket Nos 15-94, 15-91, Report and Order and Further Notice of Proposed Rulemaking, 33 FCC Rcd 7086, 7105-06, paras. 46-48 (2018); see also Letter from Ajit Pai, Chairman, Federal Communications Commission, to Maria Cantwell, Ranking Member, Committee on Commerce, Science, and Transportation, United States Senate (Jul. 21, 2021); Government Accountability Office, Emergency Alerting; Agencies Need to Address Pending Applications and Monitor Industry Progress on System Improvements, GAO 20-294 (2020). The 2021 nationwide WEA test marks the first time that the Commission has generated statistical findings about WEA’s reliability and latency. Of particular note: o Approximately 90% of WEA-capable mobile devices received the nationwide WEA test. o CMS networks can transmit nationwide WEA tests end-to-end, from their alert gateway to their cell sites in approximately 36-56 seconds, with the majority of mobile devices that receive the test receiving it within 2 minutes of transmittal. PSHSB will continue to examine the reliability of older WEA-capable mobile devices that are not technically capable of receiving a State/Local WEA Test Alert and WEA’s end-to-end latency for non-nationwide activations. · Generate recommended WEA alert origination practices and encourage localized testing to confirm WEA’s demonstrated nationwide performance in local alerting jurisdictions. PSHSB will work with FEMA to publicize the following takeaways for WEA alert origination based on WEA’s performance during the nationwide WEA test: o Alert originators should plan to use other mass notification tools in conjunction with WEA to promote receipt of potentially life-saving information by the approximately 10% of wireless subscribers in the target area that may not receive WEA alerts intended for them. o Alert originators should expect WEA alerts to be delivered with comparable reliability irrespective of which wireless providers offer service within their alerting jurisdiction or the predominant wireless network technology in their alerting jurisdiction o Alert originators should expect WEA alerts to be received on a comparable basis to people located indoors or outdoors, and irrespective of whether people are using their mobile devices for other purposes at the time their device receives a WEA alert. WEA’s performance nationwide is not necessarily indicative of its performance in any particular alerting jurisdiction. WEA’s performance at the local level may vary based on the extent of CMS Providers’ local participation in WEA. PSHSB encourages state, local, Tribal, and territorial alert originators to use State/Local WEA Test Alerts to confirm WEA’s demonstrated performance at the national level within their local jurisdictions. To this end, PSHSB will consider making its survey questions available to alert originators upon request and anticipates working with alert originators going forward to measure WEA’s performance during future tests. · Continue to work with FEMA to improve WEA testing. PSHSB will work with FEMA to ensure that future WEA tests 1) evaluate the accuracy and latency of WEA geo-targeting; 2) address the challenges of opting in to receive State/Local WEA Test Alerts, and 3) promote an evenly distributed sample of survey respondents as follows: 1. PSHSB will seek out opportunities to test the accuracy and latency of WEA’s geo-targeting by working with state, local, Tribal, and territorial emergency management agencies. PSHSB could not use this test to evaluate WEA geo-targeting because of its nationwide nature. 2. The use of the State/Local WEA Test Alert category for the 2021 nationwide WEA test had the benefit of not transmitting a WEA test to the general public, many of whom may only be interested in receiving WEA alerts for actual emergencies. People annoyed by their receipt of an unwanted WEA test could have responded to the test by opting out of WEA alert message classifications used for actual emergencies. On the other hand, the incomplete test survey results suggest that opting into the State/Local WEA Test Alert may have proven difficult or confusing for some test respondents. As a result, the Commission drew from a smaller data set than it could have to generate the findings in this report. The size of PSHSB’s reliable data set at times limited its ability to reach statistically significant conclusions on some questions. 3. PSHSB will plan to take additional steps to promote diversity in its survey responses among CMS networks and states in preparation for future WEA tests. PSHSB observes two potential areas for improvement with its approach to the design and execution of its survey. First, completed survey respondents were concentrated on Verizon’s network (155 responses) compared to AT&T’s (99 responses) and T-Mobile’s (44 responses). Second, while efforts were made to ensure broad geographic distribution of Nationwide WEA Test Survey respondents, further steps could be taken in this respect. FEMA and PSHSB improved upon their 2018 approach to the nationwide WEA test in 2021 and PSHSB will plan to work with FEMA to continue to do so going forward. · Examine non-receipt of the nationwide WEA test and generate recommendations for improving WEA’s reliability. PSHSB will attempt to identify why 10.2% of mobile devices in the completed survey response group did not receive the nationwide WEA test, develop recommendations for closing this gap, and continue to assess WEA’s reliability through further testing. The fact that every mobile device in the group that failed to receive the nationwide WEA test was identified by Participating CMS Providers as incapable of enhanced WEA geo-targeting may suggest that causes of WEA non-receipt have already been addressed by the newest smartphone models. · Examine duplicate receipt of the nationwide WEA test and generate recommendations for improving WEA’s performance. PSHSB will also continue to attempt to identify why 32.0% of mobile devices in the completed survey response group received a duplicate test message, develop recommendations for preventing duplicate WEA messages from being delivered to the public, and continue to assess WEA’s performance in this regard through future testing. Recommended measures may include allowing a unique alert message ID to be associated with each WEA alert, either at the point of alert origination or at IPAWS, which mobile devices could use to suppress duplicate alerts more effectively than the current approach based on CMS Provider-assigned serial numbers. · Remind CMS Providers about critical Communications Security, Reliability, and Interoperability Council (CSRIC) network reliability best practices likely to affect their provision of reliable WEA service. PSHSB encourages Participating CMS Providers to follow CSRIC network reliability practices regarding the administration and operation of their WEA service. See CSRIC Best Practices, https://opendata.fcc.gov/Public-Safety/CSRIC-Best-Practices/qb45-rw2t/data (last visited Oct. 15, 2021). For example, CSRIC network reliability best practice #12-10-0559 states that Network Operators, Service Providers, and Public Safety should consider validating upgrades, new procedures and commands in a lab or other test environment that simulates the target network and load prior to the first application in the field. PSHSB plans to remind Participating CMS Providers about the CSRIC network reliability best practices that bear on their performance of WEA to help to prevent or decrease recurrence of system reliability and integrity issues. · Examine the extent to which MVNOs and other communications service providers support WEA and encourage them to formally participate. PSHSB recommends assessing how MVNOs participate in WEA, and to what extent. The 2021 nationwide WEA test demonstrates that the public is able to receive WEA alerts while on MVNO networks. PSHSB, Report: October 3, 2018 Nationwide WEA and EAS Test, at 6 n.24 (2019), https://docs.fcc.gov/public/attachments/DOC-356902A1.pdf (identifying survey respondents as receiving the nationwide WEA test on MVNO networks in 2018). Survey respondents report receiving WEA alerts while on the networks of MVNOs that have not formally submitted to the Commission letters attesting to their intent to participate in WEA pursuant to the Commission’s rules. Understanding the extent which MVNOs and other service providers participate in WEA is an important part of enabling emergency management agencies to understand how WEA will work in practice within their jurisdictions and an important part of the Commission’s situational awareness. Insofar as MVNOs participate in WEA, they may formally elect to do so by filing a letter in the Commission’s WEA election docket. PS Docket No. 08-146. V. CONCLUSION 41. The 2021 nationwide WEA test successfully demonstrated that the nationwide WEA system would largely perform as designed if needed for a national emergency. The 2021 nationwide WEA test was also successful in allowing PSHSB to measure, for the first time, WEA’s reliability and latency. At the same time, the test exposed deficiencies within the system that require improvement. Over 10% of mobile devices failed to receive the test message intended for them. Thirty-two percent of respondents completing the survey report receiving a duplicate test message. The Bureau will continue to work with FEMA, wireless providers, and other WEA stakeholders to improve the system and ensure that WEA remains effective and can transmit timely and accurate alerts to the public when they are needed the most. 2