Background

Affect generally refers to the subjective experience of internal feelings, emotions, moods, dispositions, and temperaments, often visible through behaviour (Frijda, Reference Frijda and Davidson1994). Individuals show affect through displays—that is, intentional or unintentional behaviours that convey an individual’s orientations or reactions to stimuli and other people. Although communicated verbally through word choice or prosody, affect is primarily conveyed through a speaker’s facial expressions (Kappas et al., Reference Kappas, Krumhuber, Küster, Hall and Knapp2013). A speaker may display a feeling, orientation, or stance at one moment as a state, or they may be disposed to react to particular experiences in a similar way over time as a trait element of their personality. Affect was historically thought to originate in individual cognition, whereby appraisals of environmental stimuli activate emotional responses (Arnold, 1960). Others have argued that due to the lack of one-to-one physiological correlates, affect may arise within socially distributed processes amongst individuals in interpersonal interactions (Parkinson, Reference Parkinson1996). Furthermore, interpretations of affect may differ depending on the cultural background of speakers and listeners (Uchida et al., Reference Uchida, Townsend, Rose Markus and Bergsieker2009) and may even be felt differently across cultures (Mesquita, Reference Mesquita2022). These varying issues highlight methodological challenges in the measurement of affect; the social context should be carefully documented, and cultural variables should be controlled if possible.

Affect displays present distinct challenges for L2 speakers in cross-cultural settings because, to communicate effectively, they must “puzzle out unfamiliar behaviors, to identify what triggers which ‘emotions’ and when, to learn how particular ‘emotions’ may be managed and to discover what cues to pay attention to and how to interpret verbal and non-verbal ‘emotion displays’” (Pavlenko, Reference Pavlenko2014, p. 247). Furthermore, the presence of internal affective responses may have a facilitating or limiting effect on the language they produce, determining “not only whether they even attempt to use language in a given situation, but also how flexible they are in adapting their language use to variations in the setting” (Bachman & Palmer, Reference Bachman and Palmer1996, p. 65). Affect may then drive differential performance outcomes in learners based on their ever-changing reactions to social stimuli.

Indeed, trait-like affective factors have received much attention in the second language acquisition (SLA) literature because they have been found to relate to various pedagogical outcomes. Higher measures of anxiety (e.g., foreign language anxiety, test anxiety), for example, have been found to correspond with lower levels of language proficiency and course achievement (Botes et al., Reference Botes, Dewaele and Greiff2020; MacIntyre et al., Reference MacIntyre, Noels and Clément1997; Teimouri et al., Reference Teimouri, Goetze and Plonsky2019). In contrast, confidence (especially L2 self-confidence) has a positive relationship with language knowledge measures and performance outcomes (Ahammer et al., Reference Ahammer, Lackner and Voigt2019; Clément, 1986; Noels & Clément, Reference Noels and Clément1996; Stankov et al., Reference Stankov, Lee, Luo and Hogan2012), perhaps given its close relationship with other individual differences such as motivation and willingness to communicate (MacIntyre et al., Reference MacIntyre, Clément, Dörnyei and Noels1998). Positive emotions (e.g., enjoyment, happiness) may also drive or correspond with L2 acquisitional processes by “broadening a person’s perspective and opening the individual to absorb the language” (MacIntyre & Gregersen, Reference MacIntyre and Gregersen2012, p. 193), with some evidence that these correspond with achievement gains as well (Botes et al., Reference Botes, Dewaele and Greiff2020; Dewaele & Li, Reference Dewaele and Li2022; Li et al., Reference Li, Dewaele and Jiang2020). Nonetheless, the relationship between at least some of these measures and achievement or proficiency outcomes may be reciprocal or bidirectional, with growth in language skills leading to, for example, greater self-confidence in one’s ability to communicate (Edwards & Roger, Reference Edwards and Roger2015; Li et al., Reference Li, Dewaele and Jiang2020). In many of these studies, these affective traits were measured using student self-report surveys rather than external observations, which offer limited evidence about how listeners conceptualize spoken language ability considering dynamic differences in affect.

Although these affective traits lend important insight into longer-term outcomes in SLA such as course achievement, perceived affective states may impact outcomes as performances unfold in brief encounters, such as conversations or interviews. Engagement, for example, broadly defined as an individual’s level of interest and evidence of participation in an event, is often regarded as an orientation composed of interacting cognitive, social, behavioural, and affective dimensions (Philp & Duchesne, Reference Philp and Duchesne2016). Engagement, in particular social engagement, may also be critical to how successful individuals are in interactive tasks, leading to enhanced performance outcomes (Storch, Reference Storch2008). Being perceived as engaged, as well as displaying confidence, attentiveness, interactiveness, and low anxiety, may factor into positive impressions of communicative effectiveness in spoken assessment settings (Ducasse & Brown, Reference Ducasse and Brown2009; May, Reference May2011; Sato & McNamara, Reference Sato and McNamara2019). Other affective phenomena, such as displaying warmth (e.g., friendliness, empathy) or competence, have been found to correspond with performance outcomes outside of SLA research in organizational settings (Cuddy et al., Reference Cuddy, Glick and Beninger2011).

Few empirical studies to date have measured the relationships between perceived state affect and language-related outcomes. Nagle et al. (Reference Nagle, Trofimovich, O’Brien and Kennedy2022) considered subjectively perceived measures of anxiety and collaborativeness (a measure of social engagement and interaction) on scores of L2 comprehensibility. In a dataset of short dyadic interactions, speakers and their interlocutors each repeatedly measured their partner’s perceived affective states and comprehensibility. The authors found that high collaborativeness (a measure of social engagement) and low anxiety explained roughly 60% of the variance in comprehensibility scores, with small differences depending on the type of speaking task used. The authors hypothesized that high anxiety related to lower comprehensibility due to the visual cues anxious speakers display (e.g., lack of expressiveness, gaze aversion), which make speech processing more effortful for the listeners. In another study, Chong and Aryadoust (Reference Chong and Aryadoust2023) investigated the relationship between automated measurements of seven basic emotions (happiness, sadness, anger, surprise, fear, disgust, and a neutral state) and language proficiency outcomes provided by four human raters using TOEFL integrated speaking rubrics. In this study, the variance in test scores attributable to emotions ranged from 8–34%, with the authors concluding that “only some part of the observed variance in test scores can or should be associated with the emotions of participants when they are using academic language” (p. 7431). However, the researchers did not interpret which emotions were associated more consistently with language-related outcomes.

The current study

Although the literature has shown a possible link between perceived affect and language-related outcomes, affect is often treated as a trait variable in individual differences studies, often measured through self-report data (Botes et al., Reference Botes, Dewaele and Greiff2020; Clément, 1986; Dewaele & Li, Reference Dewaele and Li2022; Li et al., Reference Li, Dewaele and Jiang2020; MacIntyre et al., Reference MacIntyre, Noels and Clément1997; Noels & Clément, Reference Noels and Clément1996; Teimouri et al., Reference Teimouri, Goetze and Plonsky2019). Studies that have operationalized affect as a state variable have often relied on verbal reports focusing on communication as a whole (Sato & McNamara, Reference Sato and McNamara2019) or interactional competence more narrowly (Ducasse & Brown, Reference Ducasse and Brown2009; May, Reference May2011). While these studies noted the affective component of L2 speech, little is known about the size of its relationship. What empirical research exists has investigated outcomes of comprehensibility (Nagle et al., Reference Nagle, Trofimovich, O’Brien and Kennedy2022) and integrated skill ratings of spoken language proficiency (Chong & Aryadoust, Reference Chong and Aryadoust2023). A range of other components of language proficiency, such as fluency, grammar, and lexical range and accuracy, have yet to be considered. It is important to consider a broader range of variables given that affect may be related to some language skills more than others (Dewaele & Li, Reference Dewaele and Li2022).

Given the relatively understudied role affect may play in ratings of L2 speech, the research question (RQ) that guided this study was the following:

Based on the literature reviewed, we hypothesized that affective perceptions such as confidence, engagement, and low anxiety would correspond with higher ratings in language outcomes broadly. Overall, though, our investigation was exploratory as we had few expectations regarding the direction and size of the effects.

Speech samples

We borrowed 30 speech samples recorded in a high-stakes, oral proficiency interview context from a test provider (International English Language Testing System [IELTS]; IELTS, n.d.) to be used as the basis for the affect and spoken language ability ratings. We signed non-disclosure agreements with IELTS to protect the dataset and the privacy of the test takers, and all raters signed non-disclosure agreements indicating that they would not remove or report on the samples they watched. The test takers in this dataset had indicated consent for their data to be used in research, but because of the intellectual property of the testing data, we were unable to share video or audio from the test sessions with readers. High-stakes test recordings were optimal as many contextual elements were controlled: the recordings were conducted on a standardized laptop, the noise was controlled in the environment, the IELTS-employed examiners who conducted the interviews were trained, and the language of the test content was validated for the ability levels of the test takers and test purposes.

The recordings were taken from the same section of the speaking test (Part 3) for all samples. The section was a semi-scripted conversation between a trained IELTS examiner and the test taker on abstract issues and ideas (IELTS, n.d.). Although the interview topics and examiners varied, the samples contained a similar number of opportunities for the test takers to speak and clarify answers across the segments. We selected segments of approximately 2 mins from each test taker (M = 2 min, 11 s; SD = 14 s) from the beginning of this part of the test. The length of these segments varied slightly because we sought to trim the samples as close to the 2-min mark as possible when the test taker had reached a natural conclusion to their turn. We chose relatively short (2-min) samples from the longer test as the basis of rating so that participants would make quick, intuitive impressions rather than impressions based on a wider range of evidence. Using short samples also allowed us to keep the total experiment participation time for the volunteer raters at around 2 hours.

Table 1 displays information about the speech samples. The individuals in the 30 samples (labelled S01–S30) were all Chinese L2 English speakers. Their sample label indicated their test score ranking in comparison with other test takers. For example, sample S01 had the lowest proficiency test score on IELTS (3.5, approximately A2 on the Common European Framework of Reference [CEFR]; Council of Europe, 2020) and was thus labelled as 01. On the other hand, sample S30 tied for the highest score (6.5, approximately a high B2/low C1 on the CEFR), and was listed last (IELTS scores range from 0 to 9). The test takers appeared to be of a similar age (approximately college age), and the distribution of gender appeared to skew female (23 females, 7 males), but these demographic variables were not provided with the dataset. Proficiency scores that accompanied the dataset showed that test takers were evenly distributed across multiple ability levels. The last column in Table 1 shows the length of each of the speech samples. More information about how the dataset was compiled and trimmed is available in Supplement 2 (Burton & Winke, Reference Burton and Winke2025).

Table 1.

Speech samples

Rating scales

The rater participants in this study assigned subjective ratings on affect and spoken language ability using a set of 14 semantic differential scales. Semantic differential scales are simple, often one-word adjectives or nouns that are paired with a contrasting term set on two ends of the same scale, similar to a Likert scale (e.g., good/bad, interesting/uninteresting) (Ploder & Eder, Reference Ploder, Eder and Wright2015). These scales may have multiple points between terms for raters to indicate both the directionality and strength of an association with a term. Semantic differentials allow participants to make quick, intuitive decisions on their perceptions of stimuli that do not require rater training, unlike fully developed rating rubrics/scales (Snider & Osgood, Reference Snider and Osgood1969). Semantic differentials also allow participants to bring their understanding and interpretation of phenomena to a rating event rather than restricting these interpretations through training or scale wording. This subjective, relatively open approach was desirable for this study to capture more generalizable impressions of speech and affect.

We constructed a set of scales with categories describing spoken language performance and interpersonal perceptions of affective states. The language features were fluency, vocabulary, grammar, and comprehensibility, roughly corresponding to categories of language proficiency frequently of interest in the SLA literature. Comprehensibility, rather than pronunciation, was chosen as a category to align with ongoing research in this area (e.g., Isaacs & Trofimovich, Reference Isaacs and Trofimovich2012; Nagle et al., Reference Nagle, Trofimovich, O’Brien and Kennedy2022). Participants may have had more of an intuitive understanding of what is comprehensible rather than constituent elements of pronunciation (e.g., phonemic control, prosody), which may have led to an overt focus on accent. Even though we determined through piloting that these four categories were relatively straightforward for participants to understand, we provided brief definitions of these four categories to reduce ambiguity (e.g., broad/narrow definitions of fluency; Lennon, Reference Lennon1990). These brief definitions are listed in Table 2. More information about the piloting process and how the scales were developed is available in Burton (Reference Burton2023).

Table 2.

Definitions of scale categories

We chose 10 categories of affect for participants to rate based on frequently discussed affective state perceptions discussed in the literature on SLA, language testing, and psychology. These categories were: engagement, anxiety, confidence, warmth, attentiveness, expressiveness, happiness, competence, interactiveness, and attitude. Of particular interest were those states that have been discussed in relation to language proficiency or achievement, such as engagement (Ducasse & Brown, Reference Ducasse and Brown2009; Jenkins & Parra, Reference Jenkins and Parra2003; May, Reference May2011; Nakatsuhara et al., Reference Nakatsuhara, Inoue and Taylor2021; Philp & Duchesne, Reference Philp and Duchesne2016; Sato & McNamara, Reference Sato and McNamara2019), anxiety (Botes et al., Reference Botes, Dewaele and Greiff2020; McIntyre et al., Reference MacIntyre, Noels and Clément1997; Sato & McNamara, Reference Sato and McNamara2019; Teimouri et al., Reference Teimouri, Goetze and Plonsky2019), and confidence (Clément, 1986; Ducasse & Brown, Reference Ducasse and Brown2009; Noels & Clément, Reference Noels and Clément1996; May, Reference May2011). We were also interested in how socio-affective perceptions of speakers might relate to language judgements, and for this reason, we included attentiveness (Ducasse & Brown, Reference Ducasse and Brown2009; May, Reference May2011) and interactiveness (relating to interactional competence; Galaczi & Taylor, Reference Galaczi and Taylor2018; and also, willingness to communicate; MacIntyre et al., Reference MacIntyre, Babin and Clément1999). In addition, we included a measure of expressiveness, as this has been frequently mentioned as a positive element in relation to proficiency judgements when referring to overall nonverbal behaviour (e.g., Jenkins & Parra, Reference Jenkins and Parra2003; Neu, Reference Neu, Scarcella, Andersen and Krashen1990), as well as happiness and attitude, relating to measures of enjoyment and positive psychology in the SLA literature (Botes et al., Reference Botes, Dewaele and Greiff2020; Dewaele & Li, Reference Dewaele and Li2022; Li et al., Reference Li, Dewaele and Jiang2020; MacIntyre & Gregersen, Reference MacIntyre and Gregersen2012; MacIntyre et al., Reference MacIntyre, Gregersen and Mercer2019). Finally, warmth and competence were chosen based on findings in psychology relating these perceptions to success in organizational settings (Cuddy et al., Reference Cuddy, Glick and Beninger2011). As opposed to the language scales, definitions were not provided for the affect-related adjectives, as raters were expected to bring their own interpretations of these variables to the study.

Each scale category was presented with its adjective and the related antonym on a 7-point scale, as shown in Figure 1. We used a 7-point scale to enhance measurement precision over scales with fewer categories (Simms et al., Reference Simms, Zelazny, Williams and Bernstein2019). This also allowed a midpoint for cases where judgements were ambiguous. The polarity of the adjectives was reversed for half of the scales so that positive or negative associations were mixed on each side of the 7-point scale. This was to reduce survey acquiescence bias (e.g., straightlining), and to encourage participants to read each line carefully. The language scales were presented earliest and in the same order to establish the primacy of rating language. The affect scales were presented below the language scales in a random order for each speech sample rating. The random order was to prevent primacy of any of the affect categories, while also encouraging raters to pay close attention to the category they were rating each time. To determine the feasibility of the instruments and the design of the rating study, the scales were piloted with 25 participants (not included in this rater group) with a separate set of 10 speech samples collected in a prior study. A many-facet Rasch analysis of the pilot data indicated that each scale functioned well, with scale units ordered as intended with no misfit. The pilot participants indicated that the scales were straightforward to use when rating, and the number of scales was not an issue given the quick nature of the rating process. Specific details regarding the pilot data analysis are available in Burton (Reference Burton2023).

Figure 1.

Rating scales.

Procedure

The videos and scales were incorporated into an online rating platform built in Qualtrics. An example of the system is presented in Supplement 3 (Burton & Winke, Reference Burton and Winke2025). Raters were first introduced to the study, signed a consent form and a non-disclosure agreement, and were then allowed to review the scale categories and their meanings. Although language terms were defined briefly, affect terms were not, as semantic differentials generally allow users to bring their internal definitions of terms (Ploder & Eder, Reference Ploder, Eder and Wright2015). Participants were required to rate two practice videos using the scales to calibrate their orientations to spoken language proficiency. One video was of a highly proficient speaker, and the other was of a much less proficient speaker. After rating, participants were not provided with feedback on their scores, but rather with a general description of the language performances (not including affect), as it was desirable for participants to focus first and foremost on spoken language ability. The descriptions were framed in terms of the strength of language and communicative effectiveness. After completing the practice section, raters began the main study.

The study took place on two different days with a 24-hr gap between them. Each day featured 15 randomized speech samples for each participant. The rating was spread out to minimize fatigue, as each rating day took roughly 1 hr to complete (Day 1: M = 61 min, SD = 18 min; Day 2 = 62 min, SD = 20 min). Participants conducted the ratings remotely, though they were instructed to choose a location that was quiet and free from distractions. The videorecordings of the speech samples were presented in a large format on one single Qualtrics page without the rating scales. Participants could not pause, stop, replay, or download the videos. Immediately after the video ended, raters were taken to a second page with the rating scales and instructed to rate the video. The stimuli and scales were presented on separate pages to reduce distractions (e.g., rating while listening but not watching), as we wanted the raters to pay close attention to the videos during the entirety of the performance. After the second day, raters completed a short follow-up survey that served to monitor any technical issues with the system.

Data analysis

When preparing the dataset for analysis, the polarity of the scale scores was realigned so that negative judgements (e.g., low comprehensibility, weak grammar, anxious) had an endpoint of 1, while positive judgements (e.g., comprehensible, strong grammar, at ease) aligned with an endpoint of 7. We first calculated polychoric correlations between the scales to determine associations across variables using the polychoric function in the psych package (version 2.0.8) in R. Polychoric correlations were the basis of analysis because Pearson correlations may attenuate relationships amongst ordinal or ordered categorical variables (Winke et al., Reference Winke, Zhang and Pierce2023), and polychoric correlations provide a more accurate representation of the data (Holgado-Tello et al., Reference Holgado–Tello, Chacón–Moscoso, Barbero–García and Vila–Abad2010). We checked the stability of the correlation matrix against a multilevel Pearson correlation matrix of the same data (reported in Supplement 6), and we found that the single-level correlation matrix was robust for this analysis.

Due to the large number of variables, we then ran exploratory factor analysis (EFA) to determine whether the dataset could be reduced for regression analysis. We ran EFA rather than PCA because component scores are less interpretable than factor scores (Tabachnik & Fidell, Reference Tabachnik and Fidell2013), and we hypothesized that the variables would likely show a factor structure due to their semantically related nature (e.g., happy and warm share similar connotations). We first verified that assumptions were met for factor analysis. The relationships between variables were linear, and variance inflation factors were below 4, which satisfied the assumption of a lack of multicollinearity. The Bartlett test of sphericity (p < .001) and Kaiser–Meyer–Olkin measure of sampling adequacy (all values > .90) indicated factorability. We used parallel analysis to determine the number of factors rather than eigenvalues greater than 1, as parallel analysis tends to be less biased (Franklin et al., Reference Franklin, Gibson, Robertson, Pohlmann and Fralish1995). Parallel analysis indicated the presence of four factors. We then used exploratory factor analysis with the polychoric correlation matrix using maximum likelihood estimation and a promax rotation to produce a factor solution. We used the polychoric correlation matrix because all variables were ordinal (Holgado-Tello et al., Reference Holgado–Tello, Chacón–Moscoso, Barbero–García and Vila–Abad2010). We used an oblique rotation rather than an orthogonal rotation to allow factors to correlate.

Because the dataset included nested data (each participant rated the same 30 samples, thus each participant’s scores would exhibit correlations), there was a risk that the factor structure may vary for each rater participant. We were unable to find suitable factor analytic solutions that take into account multilevel data that allow random effects, and pursuing confirmatory factor analysis or structural equation modelling (in which this is possible) was beyond the scope of this study. However, to verify the invariance of the computed factor structure, we bootstrapped the factor analysis with 1,000 samplings of 50 participants from the total pool. The full procedure we followed is detailed in Supplement 6. We found that the factor solution was stable in 97.2% of the bootstrapped calculations, and thus we concluded that the factor structure was stable for this sample despite the multilevel nature of the dataset. Using factor analysis on the full dataset, we extracted factor scores using the Ten Berge method in the factor.scores function in R. The Ten Berge method, which is the default method in R, minimizes residuals, produces unbiased estimates of the factor loadings, and is one of the more interpretable methods of producing factor scores (Ten Berge & Kiers, Reference Ten Berge and Kiers1991). This method represented the dataset better than orthogonal extraction methods such as the Anderson method, as this method would leave factors uncorrelated, which would have misrepresented the dataset. We then used the three affect-related factor scores in the following regression analysis as independent variables.

To determine how the factor scores related to the language judgements, we built four cumulative logit mixed effects models with the language judgements as separate independent variables. We used cumulative logit mixed effects models rather than multilevel generalized ordered logit models or multilevel ordinal probit regression because we were interested in the overall effects of the predictors rather than modelling the effects at individual thresholds. Cumulative logit models are more parsimonious and can aid in interpretability. The clmm function from the ordinal package (v.2019.12–10) in R was used for modelling. Random effects of both the rater and sample were entered into the models to account for these sources of variance. We entered all main effects at once, and we used a logit link flexible threshold for all models. We tested the model with main effects against the null model and the same model with random effects removed to ensure that accounting for these sources of variance was meaningful. We applied Bonferroni corrections to the significance threshold to account for the four sets of analyses, with α = .0125. All assumptions for these models were met apart from the assumption of proportional odds, which could only be tested using Brant’s tests (Brant, Reference Brant1990) on models with random effects removed using polyr. The proportional odds assumption held for some but not all of the main effects in each model. Harrell (Reference Harrell2020) stated that in this case, a violation of the proportional odds assumption is not necessarily problematic as long as the focus of the study is to observe average odds ratios for main effects. This was indeed the case in this study, and we thus used the ordinal models instead of less parsimonious multinomial regression models.

The code sheet, written in R, and the dataset are available for analysis in Supplement 4 and Supplement 5 in the Open Science Framework (Burton & Winke, Reference Burton and Winke2025).

Relationships between affect and spoken language ability

The polychoric correlations between the factor scores and the four original language scores were all positive and moderate to strong, ranging from .48 to .72, as shown in Table 5. Each language feature was more strongly associated with assuredness and involvement than positivity. These associations were stronger for fluency than the other language scores. Regarding positivity, it showed the strongest correlation with comprehensibility (.60). The weakest associations were with grammar.

Table 5.

Polychoric correlations between factor scores and language scores

We built four sets of mixed effects ordinal regression models to determine which of the factor score reductions of the affect variables had the greatest impact on each score category after taking into account the variance attributable to each participant and each speech sample. In these models, the model with main effects fit better than the null model with main effects and no predictors (null model 1) or the model with main effects with random effects removed (null model 2), as shown in Table 6. The models of fluency (Table 7), vocabulary (Table 8), and grammar (Table 9) showed similar patterns in their main effects. In each of these models, only assuredness and involvement were significant predictors of each language score, with the strongest associations between assuredness and fluency, β = .81, odds ratio = 2.25, and involvement and fluency, β = .79, odds ratio = 2.22. These indicate that the likelihood of a score category increase on the 7-point scale in fluency, vocabulary, and grammar was roughly two times greater with 1-point increases in perceptions of assuredness and involvement. These models explained roughly a fifth to a quarter of the variance in each model using Nagelkerke’s Pseudo R ², fluency = .27, vocabulary .23, grammar = .18.

Table 6.

Tests of model fit

Table 7.

Fluency model

Note: p < .0125.

Table 8.

Vocabulary model

Note: p < .0125.

Table 9.

Grammar model

Note: p < .0125.

The model for comprehensibility, shown in Table 10, differed slightly. In this model, all three main effects were significant, with somewhat smaller standardized coefficients and odds ratios. This model deviated from the previous models in that the relationship between involvement (rather than assuredness) and the outcome was the strongest, though only slightly (odds ratio for involvement = 1.78, assuredness = 1.66). As opposed to the previous three models, positivity was a significant, positive predictor of comprehensibility, β = .36, odds ratio 1.44, showing that the likelihood that speakers were classified as one point easier to understand on the 7-point scale was about 1.44 times higher when there were 1-point increases in perceived displays of positive affect. The model for comprehensibility explained about one fifth of the variance in the comprehensibility scores, Nagelkerke’s Pseudo R ² = .22.

Table 10.

Comprehensibility model

Note: p < .0125.

Implications

This study has theoretical implications for how spoken language ability is conceptualized. Models of language ability or communicative competence (e.g., Bachman & Palmer, Reference Bachman and Palmer1996; Canale & Swain, Reference Canale and Swain1980) describe second-language communication almost entirely in terms of how linguistic components of speech and writing convey meaning in sociocultural contexts. Affect and behaviour are generally not considered to be major components in these models apart from minor compensatory roles in strategic competence (Canale & Swain, Reference Canale and Swain1980). Models of interactional competence (e.g., Galaczi & Taylor, Reference Galaczi and Taylor2018) are more inclusive of behaviour and affect, especially their role in turn-taking and conversation management, but these models do not explain how affect may closely interact within a broader communicative system. Hymes’s (Reference Hymes and Duranti1972) original concept of communicative competence, however, allowed for affective orientations to interact with language ability through the ability for use, or the functional ability of an individual to deploy their linguistic competence. The findings in this study that assuredness and involvement closely overlap with judgements across all rated categories of speech appear to support a model which includes ability for use, suggesting that these models may need closer inspection in the future. Nonetheless, the exact mechanisms of how affect relates to the various competencies in such a model need more attention before further conclusions are drawn.

These results also have implications for research and practice. For one, although constructs such as comprehensibility have received ever more attention in recent literature, the complex relationship between affect and ease of understanding complicates its measurement if visual content is included. Researchers may need to consider or control for elements of engagement (e.g., backchanneling, mutual gaze) and positive affect (e.g., smiling, laughing), as these elements appear to be able to enhance second language speech comprehension. In terms of pedagogy, there may be a question of whether behavioural elements of positive affect should be taught or encouraged in the language classroom with the belief that this could result in beneficial outcomes for the speaker. There is the possibility that this may be true in the cultural context of this study (the United States), as these types of social and psychological behaviours are common in service encounters such as when dining out, or within sensitive or high-stakes communicative contexts (health care settings, negotiations with law enforcement, diplomatic negotiations or even when one is asking for homework to not be counted as late). However, it may also be the case that the detection of stilted affect (unnecessarily formal affect) could backfire. Studies considering natural or forced affect could be revealing as to whether there is any pedagogical value in encouraging certain behaviours or affect displays.

For language assessment practice, this study raises key considerations about the fairness of including or ignoring affect in proficiency evaluations where test takers are visible to examiners. One would have serious doubts about the fairness of a test if, for example, a candidate who smiled less received a lower test score. Language tests are stressful and marked by unequal distributions of power between the examiner and test taker, which could very well result in more serious-appearing behaviour. Positive affect in this study, however, did not predict changes in fluency, grammar, or vocabulary scores when other measures of affect were considered, which appears to support fairness regarding this particular measurement. Even though this study did not find that positive affect related to these linguistic measures, certain populations may exhibit different patterns of behaviours or affect depending on group or individual differences. For example, research has shown that culture mediates how nonverbal behaviour may be encoded by speakers and decoded by listeners (Matsumoto & Hwang, Reference Matsumoto, Hwang, Matsumoto, Hwang and Frank2016), and even emotional affect is a culturally constructed phenomenon (Mesquita, Reference Mesquita2022). If there are behavioural norms in one culture (e.g., avoiding eye contact with superiors) that are seen as less appropriate in another (e.g., eye contact is seen as polite), there may be effects that arise when individuals from these cultural backgrounds perceive each other’s speech. This could result in unfair or biased test scores if certain groups receive higher scores simply because their cultural norms match those expected by the raters. Likewise, other individuals exhibit neurological differences that may conflict with what is seen as “standard” in the broader population. Test takers with autism, for example, may avert their gaze more and produce repetitive motions (American Psychiatric Association, 2013) while producing speech that may be identical or equivalent to neurotypical test takers. In test-taker populations that include neurodiverse individuals or groups from different cultures, investigations of behavioural bias on test scores are critical. If bias is found to exist, steps should be taken to ameliorate this. For neurodiverse test takers, this may mean providing accommodations or modifications to how the test is rated that ensure fairness.

The fairness of considering assuredness and involvement in test scores, as this study showed, is more complex. If indeed assuredness is a bidirectional result of language proficiency, and if assuredness is partially a cognitive mechanism (e.g., Stankov et al., Reference Stankov, Lee, Luo and Hogan2012), being perceived as more confident or less anxious may in fact reveal something about an individual’s L2 ability in some cases. Testing situations are anxiety-producing, though, and the mere fact of feeling anxious or having lower confidence may have more to do with that person’s personality than underlying ability. Showing nonverbal evidence of engagement, attention, and interactiveness, however, are all important skills in effective L2 communication, especially within subdomains such as interactional competence (Galaczi & Taylor, Reference Galaczi and Taylor2018; Plough et al., Reference Plough, Banerjee and Iwashita2018) or goal-directed communicative effectiveness (Morreale et al., Reference Morreale, Spitzbert and Barge2013). Displaying this type of affect alone is not enough to overcome very low proficiency, but it may help facilitate intercultural encounters. Nevertheless, while these affective responses may provide implicit information for raters about underlying spoken language ability, scale development and rater training programs should be investigated to ensure fairness in these measurements as well, especially when working with diverse test-taking populations (Randez & Cornell, Reference Randez and Cornell2023).

Limitations

This study naturally comes with several limitations. Notably, using untrained, naïve raters results in a large amount of variance in how outcomes are perceived, and part of this variance may be due to differing internal definitions of each of these language categories. For example, even though a narrow definition of fluency was provided to the participants to differentiate this from the broader definition generally used in society (Lennon, Reference Lennon1990), follow-up qualitative data showed that some participants oriented to fluency as a general measure of language proficiency (see Burton, Reference Burton2023). The same held for the term comprehensibility, which was occasionally discussed in relation to comprehension, though this was rare. These varying internal representations of facets of language could have skewed or attenuated some of the relationships found with affect variables. However, modelling raters as a random effect in the mixed effects model accounts for these varying patterns and thus strengthens the inferences from the regression models reported in this study. Future work should consider whether these effects exist even in pools of trained language professionals using more descriptive rating scales.

Similarly, even though pilot testing showed sufficient separability in the constructs being measured, there is always a risk of raters marking diverse rating criteria too similarly, known as the halo effect. The halo effect can muddy boundaries between otherwise distinct constructs. This type of effect may be responsible for at least part of the variance in the correlations reported in this study, though we note that the distinctive clustering patterns in the factor models indicate that the participants viewed these criteria as sufficiently different. Future work could separate counterbalanced rating sessions by affect and language ability to determine whether these relationships still stand when these varying constructs are observed on different occasions. Likewise, like the previous point, training raters on more descriptive language ability criteria would also serve to reduce potential halo effects and result in stronger inferences about the rated constructs.

Finally, a limitation and simultaneously a strength of the study was the controlled nature of the backgrounds of both the L1 raters (US-born English speakers) and the L2 test takers (Chinese-born speakers of one or more dialects of the Chinese language). Controlling for the participants’ backgrounds helped to isolate the effects of nonverbal behaviour and prevent some covariance due to culture (although regional variation in affect displays and their interpretation may have been present in the dataset). These controlled backgrounds did not allow us to measure how the impact of assuredness, involvement, and positivity might have remained invariant depending on different L1 and L2 groups. There is some indication from the literature that it may not (Uchida et al., Reference Uchida, Townsend, Rose Markus and Bergsieker2009; although cf. Tsunemoto et al., Reference Tsunemoto, Lindberg, Trofimovich and McDonough2022). We do not know if, for example, the L1 American listeners would demonstrate the same relationships between affect and the L2 speech of European, South American, or African test takers. Similarly, we do not know if different L1 groups from, for example, the United Kingdom, India, Australia, or any of the growing spheres of English as a lingua franca would demonstrate the same relationships when watching and listening to the same L2 groups. This is a complex area of study, so isolating and identifying these effects in various L1–L2 groupings is a promising area of future research.