2.1.1. Aptis test takers and speech samples

We obtained archived speaking performances, courtesy of the British Council, from 50 Aptis general test takers (hereafter speakers) spread across 19 operational forms of the test, which included four samples per speaker (one per Aptis speaking task: short response, picture description, picture comparison, long response). The sample of speakers were stratified by overall Common European Framework of Reference (CEFR) Speaking level (based on Aptis speaking score), with 10 at each of the A1–C levels and featured seven countries of residence/test location (Vietnam = 15, Spain = 10, Albania = 5, Azerbaijan = 5, Colombia = 5, Mexico = 5, Saudi Arabia = 5) that resulted in the inclusion of L1 Albanian, Arabic, Azer, Spanish and Vietnamese speakers. Their Aptis speaking section scores ranged from 5 to 48 (out of 50; mean = 31.18, SD = 13.81). In total, 198 speaking samples were included in analyses, as two separate A1-level speakers did not produce audible speech in one of their responses (one for Task 1, one for Task 4).

2.1.2. Listeners

We recruited 570 listeners via the online research participant pool Prolific (https://www.prolific.co), which included 340 US-based (female = 156, male = 184) and 230 UK-based (female = 102, male = 128) listeners. The eligibility criteria for listeners required them to be aged 18–64 (mean = 36.50, SD = 10.60, min = 18, max = 64), use English as their primary language, have no reported hearing or language impairments, and be capable of touch typing. Listeners were recruited from both the United States and United Kingdom, as these two locations are common destinations for Aptis test takers (i.e., the speaking sample in this study). Listeners indicated they had lived in a predominantly English-speaking country for 33.80 years (SD = 12.60) and used English 97% of the time (SD = 12%) in their current profession. They assessed their familiarity with non-native English speech as 5.92 (SD = 2.35) on a 9-point scale (1 = “not at all familiar,” 9 = “very familiar”).

2.2.1. Speech sample preparation

All speech samples were edited using Audacity 3.3.3 (https://www.audacityteam.org/) to enhance audio clarity and minimize background noise. Extended silences at the outset and conclusion were edited out of each sample, as was any personally identifying information. Each audio file was initially transcribed using Amazon Transcribe (Amazon Web Services, n.d.). Manual corrections were initially performed by the first author, with subsequent corrections made by the other authors, as needed. Speech tokens deemed as unintelligible after review by all authors and after reference to task prompts were recorded as /XXXX/.

For comprehensibility ratings, the 198 full speech samples had a mean duration of 106.67 s (SD = 30.25 s, range = 21.58–135.24 s). For intelligibility ratings, our research team segmented each of the 198 samples into AS units. An AS unit is defined as “a single speaker’s utterance consisting of an independent clause or sub-clausal unit, together with any subordinate clause(s) associated with either” (Foster et al., Reference Foster, Tonkyn and Wigglesworth2000, p. 365). In cases where AS-unit-based segments exceeded 24 words, we identified a logical break in the segment (e.g., clause boundary) to allow for additional segmentation. Initial segmentation was conducted by the first author and then reviewed by the second author. In total, segment length ranged from 2 to 24 words, with a mean length of 9.31 words. The resulting procedure yielded 2846 files for intelligibility transcriptions (mean # of segments = 14.23 per speaker per task, range = 2–33). The number and duration of segments by task are detailed in Table 1.

Table 1.

Number and duration of segments in intelligibility speech samples

2.2.2. Data collection

All data collection procedures were carried out remotely using listeners’ personal computers via the online experiment platform Gorilla (Anwyl-Irvine et al., Reference Anwyl-Irvine, Massonnié, Flitton, Kirkham and Evershed2020). After providing consent and filling in a background questionnaire, listeners completed the experiment. As a means to mitigate potential listener biases (e.g., Kang & Rubin, Reference Kang, Rubin, Levis and Moyer2014), listeners knew only that the speech they would hear was from a set of L2 speakers. Given the large number of speech samples to be rated and transcribed, a sparse rating design was employed (i.e., not all listeners heard all files; see Isbell, Reference Isbell, Kang and Ginther2018). First, the 198 speech samples were divided into 4 blocks of 50 files each (hereafter labeled A, B, C, D). As adequate overlap between listeners is necessary within a sparse rating design, each of the four blocks were then combined to form four supersets of speech samples (AB, BC, CD, DA; i.e., each individual block existed in 2 supersets, with each superset thus comprising 100 files). Each superset was finally divided so that the 100 files were grouped into 25 subsets of 4 (i.e., across the 4 supersets there existed 100 subsets). Each subset was balanced for task, test taker’s CEFR level and test taker’s L1. Each listener was then assigned to two supersets (e.g., AB and CD) and completed one subset within each (e.g., one subset within AB for comprehensibility ratings, one subset within CD for intelligibility transcriptions). More simply, each listener assessed four speech samples for comprehensibility and transcribed four speech sample sets for intelligibility. This design minimized, but did not eliminate, the chance of a listener being exposed to responses to the same prompt across their comprehensibility and intelligibility assessments. Aggregated at the speaker level (across all four tasks), there was an average of 45.3 judgments (SD = 2.02, min = 41, max = 49) per speaker.

Comprehensibility was assessed on a 9-point scale (1 = “hard to understand,” 9 = “easy to understand”). Listeners could select (and change) their rating at any time while hearing each sample, but were required to listen to the entirety of each file. Prior to beginning the comprehensibility section, listeners were asked to rate three practice speech samples, which were chosen to represent speakers with low (1–3), moderate (4–6) and high (7–9) comprehensibility.

Intelligibility was measured via transcription accuracy. Listeners transcribed complete Aptis speaking task responses presented as a series of consecutive audio segments. For each segment, listeners could begin typing the moment the sound file began to play and manually advance to the next segment once the audio finished and they were satisfied with their transcription of the segment. They were only permitted to hear each segment once. During training, listeners were informed that they could ignore punctuation, capitalization and fillers (e.g., uh, eh). In total, listeners transcribed the speech of four speakers (one task each). Prior to beginning the intelligibility section, listeners completed a typing test as a warm up and were then asked to transcribe the speech of two practice speakers.

2.3.1. Data quality checks

For comprehensibility ratings, we examined data quality by looking for uniform response patterns in the training task and the primary task. Ultimately, we decided to exclude two listeners’ data from the comprehensibility analyses. These two listeners demonstrated uniform response patterns (all comprehensibility ratings in the primary rating task were the same, i.e., all 4s, all 6s). Three other listeners had no variability in their practice judgments, but exhibited variation in their ratings for the primary task and were retained in analyses. In total, 567 listeners were included in comprehensibility analyses. For intelligibility, seven listeners were excluded due to one or both of the following reasons: (1) a large proportion of blank transcriptions (no transcription for over 50% of assigned segments) or (2) an average transcription length that was very short (<20 characters). One of the two listeners who were identified as contributing poor-quality data to the comprehensibility ratings was also flagged and removed from intelligibility analyses; the other listener was found to have demonstrated adequate effort/completion of transcriptions and was maintained. In total, 562 listeners were included in intelligibility analyses.

2.3.2. Scoring intelligibility responses

We scored transcriptions by calculating the proportion (expressed as a percentage) of words in a listener’s transcription which were in a criterion transcription. From original verbatim transcriptions (see Speech sample preparation), a criterion transcript which excluded extraneous elements, such as filled pauses (e.g., um), repetitions (e.g., the the) and self-corrections (with the correction being preserved), was created to have the evaluation of listener transcriptions best reflect the message a speaker intended to convey. For each segment, a listener’s transcription was searched for a match with each word of the criterion transcription and an aggregate accuracy rate of a listener’s transcription was calculated based on all segments in a sample.

2.3.3. Aggregating comprehensibility and intelligibility judgments

To arrive at aggregate measures of intelligibility and comprehensibility that accounted for differences in listeners’ ability to decode and transcribe speech (intelligibility) and severity of judgment (comprehensibility), we used many-facet Rasch measurement (MFRM) to analyze the listener response data in Facets (Linacre, Reference Linacre2021). Sparse rating designs, as used in this study, are also easily accommodated in MFRM provided that elements are adequately linked (Isbell, Reference Isbell, Kang and Ginther2018; Myford & Wolfe, Reference Myford and Wolfe2000). We constructed three-facet models which included facets for speakers, listeners and Aptis speaking tasks to produce measures of ability, severity and difficulty, respectively. For the comprehensibility model, a 9-point rating scale was specified, but for intelligibility we modeled the percentages using a scale based on 100 binomial trials. MFRM ability measures for speakers can be represented on the original scale as fair averages, which are adjusted to account for differences in other facets (e.g., listener severity). Thus, for intelligibility we transformed Rasch-based estimates of ability to percentages (ranging from 0% to 100%) and comprehensibility ability estimates were transformed to range between 1 and 9 points.

To generate fair averages for each task performance, we estimated models using data only from a given task. Because there were not adequate links in the data when each task was analyzed separately, we anchored the listener measures from the aggregated models, allowing us to generate task-level speaker estimates based on known levels of listener ability/severity.