Hostname: page-component-89b8bd64d-x2lbr Total loading time: 0 Render date: 2026-05-08T01:50:43.200Z Has data issue: false hasContentIssue false
  • English
  • Français

METHODOLOGICAL CHOICES IN RATING SPEECH SAMPLES

Published online by Cambridge University Press:  26 November 2015

Mary Grantham O’Brien
Mary Grantham O’Brien*
Affiliation:
University of Calgary
Get access Rights & Permissions [Opens in a new window]

Abstract

Much pronunciation research critically relies upon listeners’ judgments of speech samples, but researchers have rarely examined the impact of methodological choices. In the current study, 30 German native listeners and 42 German L2 learners (L1 English) rated speech samples produced by English-German L2 learners along three continua: accentedness, fluency, and comprehensibility. The goal was to determine whether rating condition, that is, (a) whether each speech sample is rated along all three continua after it is heard once or (b) whether all speech samples are rated along one continuum before being rated along the next continuum, and continuum order (e.g., whether participants rate speech samples for accentedness before comprehensibility or fluency) have an effect on listeners’ ratings. Results indicate no significant overall effects of rating condition or continuum order, but there is evidence of rating condition effects by listener group. The results have implications for laboratory and classroom assessments of L2 speech.

Information

Type
Research Report
Information
Studies in Second Language Acquisition , Volume 38 , Issue 3 , September 2016 , pp. 587 - 605
Check for updates
Copyright
Copyright © Cambridge University Press 2015 

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

Article purchase

Temporarily unavailable

References

REFERENCES

Abrahamsson, N., & Hyltenstam, K. (2008). The Robustness of aptitude effects in near-native second language acquisition. Studies in Second Language Acquisition, 30, 481–509. http://dx.doi.org/10.1017/S027226310808073X.Google Scholar
American Council on the Teaching of Foreign Languages. (2012). ACTFL Proficiency Guidelines. http://actflproficiencyguidelines2012.org/speaking.Google Scholar
Anderson-Hsieh, J., Johnson, J., & Koehler, K. (1992). The relationship between native speaker judgments of nonnative pronunciation and deviance in segmentals, prosody and syllable structure. Language Learning, 42, 529–555. http://dx.doi.org/10.1111/j.1467-1770.1992.tb01043.x.Google Scholar
Bongaerts, T., Mennen, S., & van der Slik, F. (2000). Authenticity of pronunciation in naturalistic second language acquisition: The case of very advanced learners of Dutch as a second language. Studia Linguistica, 54, 298–308. http://dx.doi.org/10.1111/1467-9582.00069.Google Scholar
Bortfeld, H., Leon, S. D., Bloom, J. E., Schober, M. F., & Brennan, S. E. (2001). Disfluency rates in conversation: Effects of age, relationship, topic, role, and gender. Language and Speech, 44, 123–147.Google Scholar
Brand, C., & Götz, S. (2011). Fluency versus accuracy in advanced spoken learner language: A multi-method approach. International Journal of Corpus Linguistics, 16, 255–275. http://dx.doi.org/10.1075/ijcl.16.2.05bra.Google Scholar
Braun, B., Lemhöfer, K., & Mani, N. (2011). Perceiving unstressed vowels in foreign-accented English. The Journal of the Acoustical Society of America, 129, 376–387. http://dx.doi.org/10.1121/1.3500688.Google Scholar
Caspers, J. (2010). The influence of erroneous stress position and segmental errors on intelligibility, comprehensibility and foreign accent in Dutch as a second language. Linguistics in the Netherlands, 27, 17–29. http://dx.doi.org/10.1075/avt.27.03cas.Google Scholar
Chakraborty, R., & Goffman, L. (2011). Production of lexical stress in non-native speakers of American English: Kinematic correlates of stress and transfer. Journal of Speech, Language, and Hearing Research, 54, 821–835. http://dx.doi.org/10.1044/1092-4388(2010/09-0018.Google Scholar
Chen, H. (2010). Second language timing patterns and their effects on native listeners’ perceptions. Concentric: Studies in Linguistics, 36, 183–212.Google Scholar
Clark, H. H., & Fox Tree, J. E. (2002). Using uh and um in spontaneous speaking. Cognition, 84, 73–111.Google Scholar
Council of Europe. (2001). Common European framework of reference for languages: Learning, teaching, assessment. Cambridge: Cambridge University Press. http://www.coe.int/t/dg4/linguistic/source/framework_en.pdf.Google Scholar
Cucchiarini, C., Strik, H., & Boves, L. (2002). Quantitative assessment of second language learners’ fluency: Comparisons between read and spontaneous speech. Journal of the Acoustical Society of America, 111, 2862–2873. http://dx.doi.org/10.1121/1.1471894.Google Scholar
de Jong, N. H., Steinel, M. P., Florijn, A., Schoonen, R., & Hulstijn, J. H. (2013). Linguistic skills and speaking fluency in a second language. Applied Psycholinguistics, 34, 893–916. http://dx.doi.org/10.1017/S0142716412000069.CrossRefGoogle Scholar
Derwing, T. M., & Munro, M. J. (1997). Accent, intelligibility, and comprehensibility: Evidence from four L1s. Studies in Second Language Acquisition, 19, 1–16.Google Scholar
Derwing, T. M., & Munro, M. J. (2009a). Putting accent in its place: Rethinking obstacles to communication. Language Teaching, 42, 476–490. http://dx.doi.org/10.1017/S026144480800551X.Google Scholar
Derwing, T. M., & Munro, M. J. (2009b). Comprehensibility as a factor in listener interaction preferences: Implications for the workplace. The Canadian Modern Language Review/La Revue Canadienne Des Langues Vivantes, 66, 181–202. http://dx.doi.org/10.3138/cmlr.66.2.181.Google Scholar
Derwing, T. M., & Munro, M. J. (2013). The development of L2 oral language skills in two L1 groups: A 7-year study. Language Learning, 63, 163–185. http://dx.doi.org/10.1111/lang.12000.Google Scholar
Derwing, T. M., Munro, M. J., & Thomson, R. I. (2008). A longitudinal study of ESL learners’ fluency and comprehensibility development. Applied Linguistics, 29, 359–380. http://dx.doi.org/10.1093/applin/amm041.CrossRefGoogle Scholar
Derwing, T. M., Rossiter, M. J., & Ehrensberger-Dow, M. (2002). “They speaked and wrote real good”: Judgements of non-native and native grammar. Language Awareness, 11, 84–99. http://dx.doi.org/10.1080/09658410208667048.Google Scholar
Derwing, T. M., Rossiter, M. J., Munro, M. J., & Thomson, R. I. (2004). Second language fluency: Judgments on different tasks. Language Learning, 54, 655–679. http://dx.doi.org/10.1111/j.1467-9922.2004.00282.x.Google Scholar
Flege, J. E. (1984). The detection of French accent by American listeners. Journal of the Acoustical Society of America, 76, 692–707. http://dx.doi.org/10.1121/1.391256.Google Scholar
Flege, J. E. (1988). Factors affecting degree of perceived foreign accent in English sentences. Journal of the Acoustical Society of America, 84, 70–79. http://dx.doi.org/10.1121/1.396876.Google Scholar
Freed, B. F. (1995). What makes us think that students who study abroad become fluent? In Freed, B. F. (Ed.), Second language acquisition in a study-abroad context (pp. 123–148). Amsterdam: John Benjamins.Google Scholar
Goethe Institut. (2004). Einstufungstest [Placement test]. http://www.goethe.de/cgi-bin/einstufungstest/einstufungstest.pl.Google Scholar
González-Bueno, M. (1997). Voice-onset-time in the perception of foreign accent by native listeners of Spanish. International Review of Applied Linguistics, 35, 251–267. http://dx.doi.org/10.1515/iral.1997.35.4.251.Google Scholar
Gui, M. (2012). Exploring differences between Chinese and American EFL teachers’ evaluations of speech performance. Language Assessment Quarterly, 9, 186–203. http://dx.doi.org/10.1080/15434303.2011.614030.Google Scholar
Gut, U. (2009). Non-native speech: A corpus-based analysis of phonological and phonetic properties of L2 English and German. Frankfurt: Peter Lang.Google Scholar
Hahn, L. D. (2004). Primary stress and intelligibility: Research to motivate the teaching of suprasegmentals. TESOL Quarterly, 38, 201–223. http://dx.doi.org/10.2307/3588378.Google Scholar
Hanna, P. L., & Allen, A. (2012). Educator assessment: Accent as a measure of fluency in Arizona. Educational Policy, 27, 711–738. http://dx.doi.org/10.1177/0895904811429293.Google Scholar
Hayes-Harb, R., & Hacking, J. F. (2015). Beyond rating data: What do listeners believe underlies their accentedness judgments? Journal of Second Language Pronunciation, 1, 43–64. http://dx.doi.org/10.1075/jslp.1.1.02hay.CrossRefGoogle Scholar
Hughes, A., & Lascaratou, C. (1982). Competing criteria for error gravity. ELT Journal, 36, 175–182. http://dx.doi.org/10.1093/elt/36.3.175.Google Scholar
Isaacs, T., & Thomson, R. I. (2013). Rater experience, rating scale length, and judgments of L2 pronunciation: Revisiting research conventions. Language Assessment Quarterly, 10, 135–159. http://dx.doi.org/10.1080/15434303.2013.769545.Google Scholar
Isaacs, T., & Trofimovich, P. (2011). Phonological memory, attention control, and musical ability: Effects of individual differences on rater judgments of second language speech. Applied Psycholinguistics, 32, 113–140. http://dx.doi.org/10.1017/S0142716410000317.CrossRefGoogle Scholar
Isaacs, T., Trofimovich, P., Yu, G., & Chereau, B. M. (2015). Examining the linguistic aspects of speech that most efficiently discriminate between upper levels of the revised IELTS Pronunciation scale. IELTS Research Report Series, 4. https://www.ielts.org/pdf/Isaacs%20et%20al%20final.pdf.Google Scholar
Kang, O. (2010). Relative salience of suprasegmental features on judgments of L2 comprehensibility and accentedness. System, 38, 301–315. http://dx.doi.org/10.1016/j.system.2010.01.005.Google Scholar
Kang, O. (2012). Impact of rater characteristics and prosodic features of speaker accentedness on ratings of international teaching assistants’ oral performance. Language Assessment Quarterly, 9, 249–269. http://dx.doi.org/10.1080/15434303.2011.642631.Google Scholar
Kang, O., Rubin, D., & Pickering, L. (2010). Suprasegmental measures of accentedness and judgments of language learner proficiency in oral English. The Modern Language Journal, 94, 554–566. http://dx.doi.org/10.1111/j.1540-4781.2010.01091.x.CrossRefGoogle Scholar
Kennedy, S., & Trofimovich, P. (2008). Intelligibility, comprehensibility and accentedness of L2 speech: The role of listener experience and semantic context. The Canadian Modern Language Review, 64, 459–489. http://dx.doi.org/10.3138/cmlr.64.3.459.Google Scholar
Kim, Y.-H. (2009). An investigation into native and non-native teachers’ judgments of oral English performance: A mixed methods approach. Language Testing, 26, 187–217. http://dx.doi.org/10.1177/0265532208101010.CrossRefGoogle Scholar
Koponen, M., & Riggenbach, H. (2000). Overview: Varying perspectives on fluency. In Riggenbach, H. (Ed.), Perspectives on fluency (pp. 5–24). Ann Arbor: University of Michigan Press.Google Scholar
Kormos, J., & Dénes, M. (2004). Exploring measures and perceptions of fluency in the speech of second language learners. System, 32, 145–164. http://dx.doi.org/10.1016/j.system.2004.01.001.Google Scholar
Leikin, M., Ibrahim, R., Eviatar, Z., & Sapir, S. (2009). Listening with an accent: Speech perception in a second language by late bilinguals. Journal of Psycholinguistic Research, 38, 447–457. http://dx.doi.org/10.1007/s10936-009-9099-1.CrossRefGoogle Scholar
Lennon, P. (1990). Investigating fluency in EFL: A quantitative approach. Language Learning, 40, 387–417. http://dx.doi.org/10.1111/j.1467-1770.1990.tb00669.x.CrossRefGoogle Scholar
Long, M. H. (1990). Maturational constraints on language development. Studies in Second Language Acquisition, 12, 251–285. http://dx.doi.org/10.1017/S0272263100009165.Google Scholar
Mackay, I. R. A., Flege, J. E., & Imai, S. (2006). Evaluating the effects of chronological age and sentence duration on degree of perceived foreign accent. Applied Psycholinguistics, 27, 157–183. http://dx.doi.org/10.1017/S0142716406060231.Google Scholar
Major, R. C. (2007). Identifying a foreign accent in an unfamiliar language. Studies in Second Language Acquisition, 29, 539–556. http://dx.doi.org/10.1017/S0272263107070428.Google Scholar
Molnar, H. (2010). The influence of the age factor on pronunciation competence in a second language: Results of a pilot study with learners of German as a second language [Der Einfluss des Faktors Alter auf die Aussprachekompetenz in der L2: ErgebnisseeinerPilotstudiemitDaZ-Lernern]. Zeitschriftfür Interkulturellen Fremdsprachenunterricht, 15, 42–60. http://zif.spz.tu-darmstadt.de/jg-15-1/beitrag/Molnar1.htm.Google Scholar
Moyer, A. (1999). Ultimate attainment in L2 phonology. Studies in Second Language Acquisition, 21, 81–108.Google Scholar
Moyer, A. (2011). An investigation of experience in L2 phonology: Does quality matter more than quantity? The Canadian Modern Language Review/La Revue Canadienne Des Langues Vivantes, 67, 191–216. http://dx.doi.org/10.3138/cmlr.67.2.191.Google Scholar
Munro, M. (1995). Nonsegmental factors in foreign accent: Rating of foreign speech. Studies in Second Language Acquisition, 17, 17–33. http://dx.doi.org/10.1017/S0272263100013735.CrossRefGoogle Scholar
Munro, M. J., & Derwing, T. M. (1994). Evaluations of foreign accent in extemporaneous and read material. Language Testing, 11, 253–266. http://dx.doi.org/10.1177/026553229401100302.Google Scholar
Munro, M., & Derwing, T. M. (1995). Foreign accent, comprehensibility and intelligibility in the speech of second language learners. Language Learning, 45, 73–97.Google Scholar
Munro, M. J., & Derwing, T. M. (2001). Modeling perceptions and the accentedness and comprehensibility of L2 speech. Studies in Second Language Acquisition, 23, 451–468.Google Scholar
Munro, M. J., Derwing, T. M., & Burgess, C. S. (2010). Detection of nonnative speaker status from content-masked speech. Speech Communication, 52, 626–637. http://dx.doi.org/10.1016/j.specom.2010.02.013.Google Scholar
Munro, M. J., Derwing, T. M., & Morton, S. L. (2006). The mutual intelligibility of L2 speech. Studies in Second Language Acquisition, 28, 111–131. http://dx.doi.org/10.1017/S0272263106060049.CrossRefGoogle Scholar
Neuendorf, J. A. (2010). The intelligibility and comprehensibility of learner speech in Russian: A study in the salience of pronunciation, lexicon, grammar and syntax (Unpublished doctoral dissertation). Bryn Mawr College. Google Scholar
O’Brien, M. G. (2004). Pronunciation matters. Die Unterrichtspraxis: Teaching German, 37, 1–9. http://dx.doi.org/10.1111/j.1756-1221.2004.tb00068.x.Google Scholar
O’Brien, M. G. (2014). L2 learners’ assessments of accentedness, fluency and comprehensibility of native and nonnative German speech. Language Learning, 64, 715–748. http://dx.doi.org/10.1111/lang.12082.Google Scholar
Pinget, A-F., Bosker, H. R., Quené, H., & de Jong, N. H. (2014). Native speakers’ perceptions of fluency and accent in L2 speech. Language Testing, 31, 349–365. http://dx.doi.org/10.1177/0265532214526177.Google Scholar
Préfontaine, Y. (2013). Perceptions of French fluency in second language speech production. Canadian Modern Language Review, 69, 324–348. http://dx.doi.org/10.3138/cmlr.1748.CrossRefGoogle Scholar
Reinisch, E., & Weber, A. (2012). Adapting to suprasegmental lexical stress errors in foreign-accented speech. Journal of the Acoustical Society of America, 132, 1165–1176. http://dx.doi.org/10.1121/1.4730884.Google Scholar
Riazantseva, A. (2001). Second language proficiency and pausing: A study of Russian speakers of English. Studies in Second Language Acquisition, 23, 497–526.Google Scholar
Riggenbach, H. (1991). Towards an understanding of fluency: A microanalysis of nonnative speaker conversations. Discourse Processes, 14, 423–441.Google Scholar
Riney, T. J., Takada, M., & Ota, M. (2000). Segmentals and global foreign accent: The Japanese flap in EFL. TESOL Quarterly, 34, 711–737. http://dx.doi.org/10.2307/3587782.Google Scholar
Riney, T. J., Takagi, N., & Inutsuka, K. (2005). Phonetic parameters and perceptual judgments of accent in English by American and Japanese listeners. TESOL Quarterly, 39, 441–466. http://dx.doi.org/10.2307/3588489.Google Scholar
Rossiter, M. J. (2009). Perceptions of L2 fluency by native and non-native speakers of English. The Canadian Modern Language Review, 65, 395–412. http://dx.doi.org/10.3138/cmlr.65.3.395.Google Scholar
Saito, K., Trofimovich, P., & Isaacs, T. (forthcoming). Second language speech production: Investigating linguistic correlates of comprehensibility and accentedness for learners at different ability levels. Applied Psycholinguistics. http://dx.doi.org/10.1017/S0142716414000502.Google Scholar
Saito, K., Webb, S., Trofimovich, P., & Isaacs, T. (forthcoming). Lexical correlates of comprehensibility versus accentedness in second language speech. Bilingualism: Language and Cognition. http://dx.doi.org/10.1017/S1366728915000255.Google Scholar
Segalowitz, N. (2007). Access fluidity, attention control, and the acquisition of fluency in a second language. TESOL Quarterly, 41, 181–186. http://dx.doi.org/10.1002/j.1545-7249.2007.tb00047.x.Google Scholar
Towell, R., Hawkins, R., & Bazergui, N. (1996). The development of fluency in advanced learners of French. Applied Linguistics, 17, 84–119. http://dx.doi.org/10.1093/applin/17.1.84.Google Scholar
Trofimovich, P., & Baker, W. (2006). Learning second language suprasegmentals: Effect of L2 experience on prosody and fluency characteristics of L2 speech. Studies in Second Language Acquisition, 28, 1–30. http://dx.doi.org/10.1017/S0272263106060013.Google Scholar
Trofimovich, P., & Isaacs, T. (2012). Disentangling accent from comprehensibility. Bilingualism: Language and Cognition, 15, 905–916. http://dx.doi.org/10.1017/S1366728912000168.Google Scholar
Trofimovich, R., Gatbonton, E., & Segalowitz, N. (2007). A dynamic look at L2 phonological learning: Seeking processing explanations for implicational phenomena. Studies in Second Language Acquisition, 29, 407–448. http://dx.doi.org/10.1017/S027226310707026X.CrossRefGoogle Scholar
Wilkerson, M. (2010). Identifying accent in German: A comparison of native and non-native listeners. Die Unterrichtspraxis, 43, 144–153. http://dx.doi.org/10.1111/j.1756-1221.2010.00076.x.Google Scholar
Winke, P., & Gass, S. (2013). The influence of second language experience and accent familiarity on oral proficiency rating: A qualitative investigation. TESOL Quarterly, 47, 762–789. http://dx.doi.org/10.1002/tesq.73.Google Scholar