Federal Communications Commission DA 21-1487 DA 21-1487 Released: November 30, 2021 REQUIREMENTS FOR CONVERSATIONAL MEAN OPINION SCORE TESTING OF INTERACTIVE VOICE SERVICE PROVIDED OVER CONNECT AMERICA FUND-SUPPORTED HIGH-LATENCY NETWORKS WC Docket No. 10-90 Consistent with the Performance Measures Satellite Reconsideration Order, Connect America Fund, WC Docket No. 10-90, Order on Reconsideration, 34 FCC 8081, 8088-89, para. 19 (WCB/WTB/OET 2019) (Performance Measures Satellite Reconsideration Order). this Public Notice provides guidance on the environmental and procedural requirements to be used for conversational Mean Opinion Score (MOS) testing of voice service provided over Connect America Fund (CAF)-supported high-latency networks. MOS testing applies to those technologies unable to meet the 100 ms latency standard (such as traditional geostationary satellite services) and that instead must demonstrate a MOS score of four or higher and that 95 percent or more of all testing hour measurement of network round trip latency are at or below 750 ms. See, e.g., Connect America Fund; ETC Annual Reports and Certifications; Rural Broadband Experiments, Report and Order and Further Notice of Proposed Rulemaking, 31 FCC Rcd 5949, 5960-61, para. 30 (2016). As a general matter, carriers subject to the MOS testing requirement should consult and follow guidance in the ITU-T Recommendation P.800 ITU-T Recommendation P.800, (08/1996), Methods for subjective determination of transmission quality. and the more recent ITU-T Recommendation P.805 ITU-T Recommendation P.805 (04/2007), Subjective evaluation of conversational quality. wherever the Commission, the Wireline Competition Bureau, the Wireline Telecommunications Bureau, or the Office of Engineering and Technology Together, the Wireline Competition Bureau, the Wireless Telecommunications Bureau, and the Office of Engineering and Technology are referred to in this document as the Bureaus and OET. has not specified more stringent requirements. The ITU recommendations provide generic methods and procedures for conversational MOS testing, from which the requirements outlined below are adapted for the specific purpose of testing CAF-supported high-latency networks. Background ITU-T Rec. P.805 defines a “conversation test” as a “subjective test in which two participants have a real-time conversation, as described in Annex A to [ITU-T P.800] and in [b-Telephonometry].” ITU-T Rec. P.805, p. 1. Test subjects participate in the MOS conversation tests as paired sets of communicators. Id. at p. 2. The Bureaus and OET allow for three variations of CAF MOS conversation testing depending on the locations of the test subjects (i.e., the endpoints of the communications connection over which a conversation is conducted). The Performance Measures Order permits testing to be conducted over a connection with at least one endpoint at an active subscriber location and where the second endpoint “may be [another active subscriber location or] a centralized location from which the Reviewer conducts live interviews with the subscriber to determine the subscriber’s MOS evaluation.” Connect America Fund, WC Docket No. 10-90, Order, 33 FCC Rcd 6509, 6525-26, para. 45 (WCB/WTB/OET 2018) (Performance Measures Order). The order does not preclude testing where both endpoints of the conversation are at active subscriber locations. The subsequent Performance Measures Satellite Reconsideration Order permits testing to be done with both test subjects at a centralized location that can be served by the carrier’s CAF-funded network. Carriers subject to the MOS testing requirement may choose from among the three variations (i.e., active subscriber testing, centralized testing, and subscriber-to-centralized location testing) to implement CAF MOS testing, subject to meeting certain requirements, specified below. These requirements help ensure that MOS testing verifies adequate voice quality of CAF-funded networks under conditions that subscribers experience in the real world. Because there is significant commonality in the requirements for the three variations of MOS testing, we present the common requirements first, followed by additional requirements specific to each testing variation. General MOS Testing Requirements Requirements Common to All Testing Variations. (1) MOS testing must be implemented using the operational network infrastructure, such as satellite links, and not laboratory-based simulations intended to reproduce service conditions. (2) MOS testing must use the same satellite beams, and network equipment, systems and processes that are used to provide service to end user locations funded by high-cost universal service. The actual network infrastructure used to provide supported services must be used in the testing. (3) MOS test Reviewers must be from an independent agency or organization. Neither test subjects nor Reviewers may be affiliated with the high latency provider whose network is being tested. Additionally, test subjects may not be members of the testing organizations. (4) MOS test Reviewers must conduct live interviews of test subjects to determine their opinion scores immediately following completion of the conversational part of the test. Survey forms, mail-in documentation, automated phone calls, or other non-interactive and non-person-to-person interviews are not permitted. However, in-person or person-to-person telephone interview calls are allowed. (5) MOS testing must be conducted over a “single hop” satellite connection from the subscribers’ or test subjects’ locations. (6) The use of simulation tools and sound-proof rooms are not acceptable, since our purpose is to verify adequate performance of the CAF-funded networks themselves under real-world conditions experienced by subscribers. Carriers should maintain records affirming that testing was not conducted in a soundproof room, but measurements of specific noise levels in testing locations are not required. (7) Procedural requirements regarding MOS conversation tasks, post-task questions and opinion scoring, as described later in this document must be followed. Additional Requirements for Active Subscriber Testing. (1) Two test subjects are required for each instance of testing; test subjects must be selected randomly from among the carrier’s active subscribers as described in the Performance Measures Order. Carriers should maintain records regarding how the selection was conducted for auditing purposes. (2) MOS Testing must be implemented with the end-user equipment that the subscribers typically use with the carrier’s voice service. Additional Requirements for Centralized Testing. (1) MOS testing must be conducted from a location other than a carrier location with test subjects selected at random from the available pool of test subjects, which cannot include the carrier’s or testing service employees. (2) MOS testing must be implemented using end-user equipment that is typical of that provided to subscribers to the carrier’s service. (3) Two test subjects are required for each instance of testing; they must be in separate rooms or otherwise positioned within the centralized location so that they cannot see or hear each other, except over the satellite network connection, during their conversation and subsequent scoring of the service. The purpose of this requirement is to ensure that communications between test subjects at the centralized location is over the connection being tested, and not shouted across the test room, or indicated through hand gestures, facial expressions and other forms of non-verbal communications. Similarly, the requirement to keep test subjects apart during the interview and scoring process is to ensure that one subject’s rating of the service does not influence the second subject’s rating. This requirement should not be misinterpreted to mean that test subjects may be placed in sound-proof rooms. (4) Each semi-annual testing cycle must be conducted from a separate centralized location served by a different network beam – assuming the provider has two or more operational beams serving CAF locations – so that the same part of the network is not tested each time. (5) The centralized location must rotate through the states where the carrier has active subscribers. Each test cycle must be conducted in a different state without repeats until all states having active subscribers have been covered. (6) The carrier must document how subjects are selected for testing, the location(s) of the centralized test site(s), and describe the facilities used for testing; this documentation should be retained for at least 10 years See 47 CFR § 54.320(b). and produced in the event of an audit. Specific MOS Testing Procedures, Tasks, and Scoring ITU-T Rec. P.805 lists what it calls main characteristics of conversation tests. The characteristics of key importance to CAF MOS testing are these: · To be very close to a real conversation where people are required to interact and may adapt their behavior to accommodate the system under test. · The use of a task to stimulate a conversation with equal participation of both parties. · Devices under test and simulation tools must be available at the test locations. ITU-T Rec. P.805, p. 2. We note generally that for CAF testing, the use of simulation tools is not acceptable, since our purpose is to verify adequate voice quality of CAF-funded networks themselves under conditions that subscribers experience in the real world. Instead, voice service provided over the carrier’s CAF-funded network and typical subscriber equipment must be used at all test locations. As noted earlier, test subjects participate in the conversation test as paired sets of communicators. Id. Test subjects may participate in only one conversation test. New test subjects are required for each test. The procedure is as follows: 1. Two test subjects are assigned a task to accomplish through their conversation. Examples of tasks suitable for use in CAF MOS testing are given in ITU-T Rec. P.805, Appendices VII to IX. Carriers must use one or more of these tasks to implement CAF MOS testing. 2. Each pair of test subjects conducts a conversation over the CAF network to accomplish its assigned task. 3. At the completion of the task, Reviewers ask the test subjects questions and record their answers. Reviewers will first ask a series of multiple-choice questions aimed at getting the test subjects to think seriously about the quality of the connection they just used to complete their conversation. The following questions from ITU Recommendation P.805 Id. at pp. 8-10. must be used to focus subjects’ thinking on the quality of the voice connection. Subjects’ responses to these questions should be recorded and maintained for audit purposes. However, they are not to be reported to the FCC or included in the calculation of the MOS score: a. “Did you or your partner have any difficulty in talking or hearing over the connection?” · Yes · No The Reviewer allocates the following values to the responses: Yes = 1; No = 0. b. “How would you assess the sound quality of the other person's voice?” The five-point scale descriptors are: · No distortion at all, natural · Minimal distortion · Moderate distortion · Considerable distortion · Severe distortion The following values are allocated to the responses: No distortion at all, natural = 5; Minimal distortion = 4; Moderate distortion = 3; Considerable distortion = 2; Severe distortion = 1. c. “How would you assess your level of effort to converse back and forth during the conversation?” The five-point scale descriptors are: · No special effort required · Minimal effort required · Moderate effort required · Considerable effort required · Severe effort required The following values are allocated to the responses: No special effort required = 5; Minimal effort required = 4; Moderate effort required = 3; Considerable effort required = 2; Severe effort required = 1. d. “Did you detect echo during the conversation?” · Yes · No The Reviewer allocates the following values to the responses: Yes = 1; No = 0. “If yes, how annoying was it?” The five-point scale descriptors are: · No annoyance · Minimal annoyance · Moderate annoyance · Considerable annoyance · Severe annoyance The following values are allocated to the responses: No annoyance = 5; Minimal annoyance = 4; Moderate annoyance = 3; Considerable annoyance = 2; Severe annoyance = 1. e. “Did you detect delay that interfered with the conversation?” · Yes · No The Reviewer allocates the following values to the responses: Yes = 1; No = 0. “If yes, how annoying was it?” The five-point scale descriptors are: · No annoyance · Minimal annoyance · Moderate annoyance · Considerable annoyance · Severe annoyance The following values are allocated to the responses: No annoyance = 5; Minimal annoyance = 4; Moderate annoyance = 3; Considerable annoyance = 2; Severe annoyance = 1. Test subjects’ responses to the preceding questions are not to be used or averaged to determine the subjects’ opinion scores. 4. Reviewers must then ask the following question from ITU-T Rec. P.805. Id. at p. 10. Responses to this question and this question alone are to be included in the calculation of the carrier’s MOS: “What is your opinion of the connection you have just been using?” The five-point scale descriptors are: · Excellent quality · Good quality · Fair quality · Poor quality · Bad quality The Reviewer allocates the following values to the responses: Excellent quality = 5; Good quality = 4; Fair quality = 3; Poor quality = 2; Bad quality = 1. Each subject’s quality rating corresponds to a single number that summarizes his or her overall opinion of the connection. This single number, which must be an integer between 1 and 5 inclusive, is the opinion score of the test subject. Each test conversation produces two opinion scores, one from each test subject. All opinion scores from all test subjects over all test conversations must be reported to the Universal Service Administrative Company (USAC) in the USAC-approved format. Using these data, USAC will determine the carrier’s MOS as described below. Prior to the calculation of the MOS, Commission staff or USAC will perform a statistical analysis to identify outliers, if any, in the carrier’s submitted opinion data. Performance Measures Satellite Reconsideration Order, 34 FCC Rcd at 8089, para. 20. These outliers will be excluded from calculation of the MOS. Commission staff will then calculate the arithmetic mean (i.e., the average) of all remaining opinion scores to obtain the carrier’s Mean Opinion Score. Since the MOS is the arithmetic mean of opinion scores, the MOS will not necessarily be an integer. Reporting the Data Carriers subject to the MOS testing requirement must report the opinion scores obtained from each conversation test. A conversation test record must include the date of the test, the test variation (i.e., active subscriber testing, centralized testing, or subscriber-to-centralized location testing), start time of the conversation, the approximate length of the conversation in minutes, and the opinion scores and location identifiers of both test subjects who participated in the test. The reported opinion scores must be those obtained from the test subjects in response to the question: “What is your opinion of the connection you have just been using?” The location identifier for an active subscriber must be the unique subscriber identifier submitted to USAC for use in the random selection of test subjects. The location identifier for a centralized location should identify both the state and city in which testing is taking place (e.g., NY-Albany). Conversation test records must be submitted for all conversation tests that have been conducted. All conversations and associated opinion scores must be reported. Trimming of response data is not permitted. See id. at 8089-90, paras. 20, 22. For additional information on this proceeding, contact Suzanne Yelen (Suzanne.Yelen@fcc.gov) or Stephen Wang (Stephen.Wang@fcc.gov) of the Wireline Competition Bureau, Telecommunications Access Policy Division, (202) 418-7400. - FCC - 2