5.1. Descriptive statistics, reliability and validity

Five participants were excluded in the first questionnaire, because their ratings did not correlate with the rest of the participants (r < 0.1). The five questionnaires had high internal reliability with all Cronbach’s alphas being larger than 0.97. Table 1 shows the descriptive statistics of the EmoPro in English as L2. The English words in the present study had relatively high EmoPro, with mean and median being larger than 3.2/5. The initial piece of validity evidence is derived from the correlation results pertaining to English as L1 (as indicated by previous studies) and English as L2 (as observed in the present study). We retrieved 92 English words from one salient database on English (L1) EmoPro (Shaver et al., Reference Shaver, Schwartz, Kirson and O’Connor1987) and found that EmoPro ratings in L1 and L2 English were closely related, N = 92, r = 0.50, p < 0.001. Specifically, Shaver et al. (Reference Shaver, Schwartz, Kirson and O’Connor1987) instructed a group of English native speakers to rate a list of 213 emotion words on EmoPro, with the evaluation being carried out on a four-point scale ranging from “I definitely would not call this an emotion (1)” to “I definitely would call this an emotion (4).” Despite the differences in wording between the instructions of Shaver et al. (Reference Shaver, Schwartz, Kirson and O’Connor1987) and the present study, the two measurements converged in EmoPro. However, the number of identical words between the two studies (92) was relatively small. To further verify the correlation between L1 and L2 EmoPro ratings, a group of 34 English native (mean age: 30.97 ± 6.23, 10 males) speakers were recruited from Prolific, an online participant recruitment platform that has been widely used (Stanton et al., Reference Stanton, Carpenter, Nance, Sturgeon and Villalongo Andino2022). To be eligible, participants must be between the age of 18 and 45 years and have current residence in the United States or the United Kingdom. Only monolingual English speakers were recruited, and participants will be excluded from the study if they self-reported speaking more than one language.

Table 1.

Descriptive statistics for English EmoPro rating in L2 and L1

The English native speakers were tasked with rating 232 English emotion words from the 1122 words included. We meticulously selected words with the highest EmoPro while adhering to the protocol outlined by Shaver et al. (Reference Shaver, Schwartz, Kirson and O’Connor1987) to eliminate overlapping words sharing the same root (e.g., “sadness” and “sad”). The internal reliability of our selection was found to be adequate, with Cronbach’ s alpha exceeding 0.98. Notably, we observed a significant correlation (N = 232, r = 0.41, p < 0.001) between the EmoPro ratings of English as both L1 and L2. However, it is noteworthy that this correlation (r = 0.41) closely resembled the correlation (r = 0.50) observed between our EmoPro ratings and those reported by Shaver et al. (Reference Shaver, Schwartz, Kirson and O’Connor1987). This moderate level of correlation suggests a potential divergence in EmoPro between L1 and L2. In an effort to aid future researchers in selecting the most representative emotion-label words in both languages, we prioritized the most prototypical emotion-label words in the L2 for EmoPro rating by L1 speakers. Despite some convergence observed between the two languages, certain emotion-label words deemed most prototypical in the L2 were rated lower in EmoPro by L1 speakers. For instance, words like “honor” received ratings of 4.13 in the L2 and 1.53 in the L1 and “favorable” received ratings of 4.23 in the L2 and 1.65 in the L1. This inconsistency may be attributed to cultural and linguistic backgrounds, highlighting the potential linguistic specificity in understanding emotions. To further delve into this issue and bolster the validity of our findings, we conducted a correlation analysis between EmoPro ratings in Chinese and English, both obtained from the same population of Chinese–English bilinguals.

Specifically, the additional validity evidence emerges from the correlation results of English translations of Chinese/Spanish as L1 (according to previous studies) and English as L2 (as demonstrated in the present study). This section would additionally serve to further illustrate the cross-cultural and cross-language disparities as well as similarities. By using Baidu Translator to translate a recent Chinese (L1) EmoPro database (Wu, Reference Wu2023) into English (L2), we found there were 230 Chinese–English translation equivalents and the EmoPro ratings in the Chinese (L1) and English (L2) were also correlated among the Chinese–English bilinguals, N = 230, r = 0.62, p < 0.001. Among the 230 Chinese–English translation equivalents, 68 words were contained in the 232 English words rated by English speakers for EmoPro. Moreover, these 68 words had converging EmoPro ratings, r = 0.35. We also identified 304 English words that are translation equivalents of Spanish words from one recent EmoPro Spanish database (Pérez et al., Reference Pérez, Stadthagen-Gonzalez, Guasch, Hinojosa, Fraga, Marin and Ferré2021). The EmoPro of the two languages was related, N = 304, r = 0.38, p < 0.001, confirming the validity of EmoPro in L2 English words. One possible explanation for this medium-level correlation is the nonequivalence of emotions between languages. Previous research has indicated that certain emotion words are challenging to translate accurately from one language to another (Altarriba, Reference Altarriba2003; Kayyal & Russell, Reference Kayyal and Russell2013). The correlation amplitude between L1 and L2 within the same language (i.e., English, N = 92, r = 0.5) is greater than that across different languages (N = 304, r = 0.38). This suggests that variations in language may impact EmoPro and its development.

5.2. Correlations between EmoPro rating and other psycholinguistic variables

The correlations between EmoPro and other psycholinguistic and affective variables are presented in Table 2 (the correlation matrix for all variables is provided in the Supplementary Material). The close relationship between EmoPro and these variables provided further validity evidence for EmoPro, such that there was a negative association between EmoPro and concreteness. This finding was attributed that emotion words are usually more abstract than neutral words (Altarriba, Reference Altarriba2003; Altarriba & Bauer, Reference Altarriba and Bauer2004; Altarriba et al., Reference Altarriba, Bauer and Benvenuto1999), and high EmoPro emotion-label words, such as happiness, are by nature abstract words. However, Wu (Reference Wu2023) did not observe a correlation between EmoPro and concreteness. This disparity could potentially be ascribed to the comparatively low variance in the concreteness of the words within the database utilized by Wu (Reference Wu2023). The limited range of concreteness values in that particular database might have hindered the detection of a relationship between EmoPro and concreteness.

Table 2.

Correlation between EmoPro and other psycholinguistic variables (word count)

Note: *p < 0.05, ^**p < 0.01, ^***p < 0.001.

Word frequency, a widely explored psycholinguistic variable, was found to be positively related to EmoPro, indicating that the typical emotion-label words were frequently used, in line with previous studies (Niedenthal et al., Reference Niedenthal, Auxiette, Nugier, Dalle, Bonin and Fayol2004; Pérez et al., Reference Pérez, Stadthagen-Gonzalez, Guasch, Hinojosa, Fraga, Marin and Ferré2021). Unexpectedly, there was no significant correlation between EmoPro and valence, a finding in contrast with a negative correlation between EmoPro and valence (Pérez et al., Reference Pérez, Stadthagen-Gonzalez, Guasch, Hinojosa, Fraga, Marin and Ferré2021; Wu, Reference Wu2023). Precisely, Wu (Reference Wu2023) found that the more unpleasant a word was, the more prototypical of an emotion word. The inconsistency might be attributed to the equal distribution of negative and positive words in the present study. We adopted a comparable number of negative and positive words, making a possible nonsignificant correlation between EmoPro and valence. We separated the words into positive words (531 words, valence >4) and negative words (591 words, valence <4) and computed the correlation between valence and EmoPro. The results showed a positive association between valence and EmoPro for positive words (N = 531, r = 0.376, p < 0.001), while there was no correlation between valence and EmoPro for negative words (N = 591, r = −0.025, p > 0.05). By inspecting the negative words, we found that there were many negative emotion-laden words, such as death and devil. The emotion-laden words had low EmoPro and valence, but negative emotion-label word, such as terror, had high EmoPro and low valence. Mixing emotion-laden words and emotion-label words might contribute to the nonsignificant correlation between EmoPro and valence for negative words. However, mixing the two kinds of word in this database is necessary, because this enabled us to select and compare emotion-label words and emotion-laden words in L2.

In addition to the factors that have been explored in the prior examinations, the present study also explored the relationship between EmoPro and other factors (i.e., socialness and SER). The positive correlation between EmoPro and socialness was in line with the finding of Diveica et al. (Reference Diveica, Pexman and Binney2022) who observed that socialness was positively related to valence and arousal. Diveica et al. (Reference Diveica, Pexman and Binney2022) adopted a comprehensive concept of socialness that measures the social content of words in any aspect, including social roles (e.g., teacher), social institutions (e.g., school), social relationship (e.g., friend), or personal features in socialness (e.g., outgoing). This comprehensive concept of socialness allowed a close relationship between socialness and emotional factors, because emotion plays a vital role in social development (Eggum et al., Reference Eggum, Eisenberg, Kao, Spinrad, Bolnick, Hofer and Fabricius2011). Besides socialness, EmoPro was also associated with SER that measures the extent to which a word activates sensory experience when an individual reads that word (Juhasz et al., Reference Juhasz, Yap, Dicke, Taylor and Gullick2011). However, the negative relationship between SER and EmoPro was in contrast to the finding from one recent study of SER and emotional experience in Chinese (Song & Li, Reference Song and Li2021; Wu, Reference Wu2023). Specifically, Song and Li (Reference Song and Li2021) recently found that emotional experience was positively related to SER, suggesting that sensory experience was embedded in emotional experience construction. The positive correlation between SER and valence was also observed in the study of Diveica et al. (Reference Diveica, Pexman and Binney2022). The inconsistency was probably due to that in L2, sensory information was not as strongly activated as in L1, reflecting a disembodied account of L2 processing (Pavlenko, Reference Pavlenko2012). Additionally, certain emotion-laden words characterized by low EmoPro values also exhibit high SER, potentially contributing to a negative correlation between EmoPro and SER.

5.3. Predictors for the EmoPro

We conducted linear regression analyses to explore how EmoPro was predicted by affective and psycholinguistic variables and found that for the 740 words containing the following variables, valence, β = −0.136, t = −4.435, p < 0.05; arousal, β = 0.134, t = 4.357, p < 0.05; AoA, β = −0.293, t = −7.469, p < 0.05; concreteness, β = −0.543, t = −15.285, p < 0.05; and socialness, β = 0.161, t = 4.958, p < 0.05, could predict EmoPro. However, word frequency, β = −0.01, t = −0.249, p > 0.05, and semantic diversity, β = 0.115, t = 0.609, p > 0.05, failed to predict EmoPro. For the 242 words that contained the SER, we found that SER could predict EmoPro, β = 0.164, t = 2.609, p < 0.05, along with other significant predictors such as valence, β = −0.116, t = −2.069, p < 0.05; arousal, β = 0.116, t = 2.019, p < 0.05; AoA, β = −0.209, t = −2.613, p < 0.05; concreteness, β = −0.513, t = −7.636, p < 0.05; and socialness, β = 0.190, t = 3.127, p < 0.05, for EmoPro.

The regression analyses showed emotional and semantic factors were contributing to EmoPro, indicating that constructing L2 EmoPro requires valence, arousal, and also semantic factors including concreteness, socialness, and AoA. These results were to a large extent comparable with those obtained from the study by Wu (Reference Wu2023). The recent study (Wu, Reference Wu2023) demonstrated that arousal, valence, and AoA significantly predicted EmoPro. Valence and arousal are two essential dimensions in constructing emotion (Barrett, Reference Barrett1998; Russell & Barrett, Reference Russell and Barrett1999). Therefore, forming typical emotion categories that are represented by emotion-label words relies on valence and arousal. In addition, emotion-label words with high EmoPro are more abstract than those with low EmoPro, reflecting a categorization of emotional episodes that are represented by emotion-laden words with low EmoPro (Russell, Reference Russell2015). Furthermore, AoA negatively predicted EmoPro, implying that emotion categories were established in early development. EmoPro was positively predicted by socialness, also in line with the previous correlation between EmoPro and socialness. Many typical emotion-label words, such as pride, shame and embarrassment, are closely related to social and personal relationship and resides in social contexts, such as rewarding reception, examination or dancing stages. It was also worth noting that the ratings of affective and semantic factors were obtained from L1 English speakers, while the EmoPro was gathered from L2 English speakers. The converging pattern between EmoPro and these factors clearly suggested that emotion construction in L2 was also very similar to that in L1. That is, emotion category formation in L2 depends on both affective and semantic factors. However, some evidence showed that the concreteness effect (concrete words were recognized faster than abstract words) was restricted within L1 rather than L2 (Ferré et al., Reference Ferré, Anglada-Tort and Guasch2018), supporting that the link between L2 and sensorimotor information was weaker than that in L1 (Foroni, Reference Foroni2015; Pavlenko, Reference Pavlenko2012). The present study revealed a possible link between emotion concepts and semantics as well as sensory experiences (as reflected in SER). L2 speakers, at least when deciding the EmoPro in L2, need to retrieve affective and semantic information, but how the relevant information is at play during L2 emotion word processing should be explored in future experiments.

5.4. How EmoPro predicted lexical decision and naming performance

To explore how EmoPro and other affective and semantic factors predicted the lexical decision and naming performance, we conducted a series of regression analyses and found that EmoPro predicted naming reaction time of naming in L1 but did not predict the lexical decision performance in L1 (see Tables 3 and 4). The former finding was in line with the facilitation of EmoPro in word processing but the latter result was not in agreement with one recent experiment showing that EmoPro facilitated Spanish word recognition in lexical decision task (Haro et al., Reference Haro, Calvillo, Poch, Hinojosa and Ferré2022). It was plausible that EmoPro in L2 could predict L2 English rather than L1 English lexical decision performance, since the lexical representation between L1 and L2 was still distinctive, especially for semantic activation being weaker in L2 than in L1 (Nakayama et al., Reference Nakayama, Sears, Hino and Lupker2013; Zhang et al., Reference Zhang, Wu, Zhou and Meng2019b). The disadvantage of L2 semantics constrained the prediction of L2 EmoPro on lexical decision performance in L1. In line with this notion, analysis of L2 English lexical decision performance data (Brysbaert et al., Reference Brysbaert, Keuleers and Mandera2021) revealed that EmoPro demonstrated predictive capability for L2 English lexical decision accuracy rates rather than reaction times (refer to Table 5). This observation substantiates the validity of English EmoPro. Moreover, there was a noticeable inclination for L2 EmoPro to display a more pronounced correlation with L2 lexical processing than its connection with L1 lexical processing. Along with this notion, L2 EmoPro, β = 0.101, t = 2.055, p < 0.05, rather than L1 EmoPro (obtained from native English speakers), β = −0.023, t = −0.490, p > 0.05, could positively predict reaction time in L2 lexical decision performance. Interestingly, EmoPro exhibited a negative predictive relationship with L2 lexical decision accuracy rates and a positive predicative relationship with L2 lexical decision reaction time, indicating that an increase in L2 EmoPro was associated with a decrease in accuracy and an increase in reaction time – a contrast to the facilitation effect observed in L1 (Haro et al., Reference Haro, Calvillo, Poch, Hinojosa and Ferré2022). A plausible explanation for this discrepancy may be derived from the disembodied account (Pavlenko, Reference Pavlenko2012), which suggests that in L2, emotion words activate emotions through semantics. Consequently, higher EmoPro in L2 words activates heightened emotion and, simultaneously, triggers more substantial semantic information, potentially hindering efficient word processing. Nevertheless, a comprehensive exploration of how EmoPro modulates L2 word processing warrants further investigation in future studies. Furthermore, Zhang et al. (Reference Zhang, Wu, Yuan and Meng2019a) discovered that certain emotion-laden words with low EmoPro values exhibited faster and more accurate recognition than emotion-label words. The inclusion of these emotion-laden words with low EmoPro in the L2 lexical decision task could potentially contribute to a negative correlation between EmoPro and accuracy rates.

Table 3.

Multiple regression analyses on lexicon decision performance (Balota et al., Reference Balota, Yap, Hutchison, Cortese, Kessler, Loftis and Treiman2007)

The significant predictors are written in bold.

Table 4.

Multiple regression analyses on naming performance (Balota et al., Reference Balota, Yap, Hutchison, Cortese, Kessler, Loftis and Treiman2007)

The significant predictors are written in bold.

Table 5.

Multiple regression analyses on L2 lexicon decision performance (Brysbaert et al., Reference Brysbaert, Keuleers and Mandera2021)

The significant predictors are written in bold.

AoA and word frequency were two robust predictors for performance in the two tasks (see Tables 3 and 4). The results demonstrated that AoA and word frequency jointly facilitated word recognition across tasks, substantiating many previous studies (Brysbaert & New, Reference Brysbaert and New2009; Keuleers et al., Reference Keuleers, Lacey, Rastle and Brysbaert2012; Kuperman et al., Reference Kuperman, Stadthagen-Gonzalez and Brysbaert2012). In addition, in the lexical decision task, valence negatively predicted the reaction times, indicating that with the increase of valence (changing from negative words to positive words), the processing speed rose. This has been validated by many prior examinations showing that positive words were recognized faster than negative words (Kazanas & Altarriba, Reference Kazanas and Altarriba2015a; Wu et al., Reference Wu, Zhang and Yuan2021; Zhang et al., Reference Zhang, Wu, Zhou and Meng2019b). However, the valence failed to predict naming performance; it was possible that lexical decision task required semantic retrieval more than naming task did (Morrison & Ellis, Reference Morrison and Ellis2000). Morrison and Ellis (Reference Morrison and Ellis2000) also observed that imageability predicted naming performance rather than lexical decision performance, which was in line with the present study showing L2 EmoPro predicting L1 naming performance. Altarriba et al. (Reference Altarriba, Bauer and Benvenuto1999) found that emotion words were also high in imageability, enabling people to form images, and high imageability facilitated naming of the words.

However, it is worth noting that the predictive capacity of L2 EmoPro ratings on L1 performance is not fully guaranteed. Additionally, examining the predictive capacity of L2 EmoPro (obtained from Chinese native speakers) on L2 performance from bilinguals whose L1 is not Chinese is likely also not tenable. Consequently, this section exploring the prediction of EmoPro on L1 and L2 lexical decision and naming performance should be explained tentatively with caution.

5.5. Limitations and future directions

Several limitations warranted further elaboration. First, the sample in the present study was not gender-balanced, such that there were only 18% males, which was common for extant EmoPro rating studies (Wu, Reference Wu2023; Zheng et al., Reference Zheng, Zhang, Guo, Guasch and Ferré2023). Therefore, a gender-balanced sample for EmoPro rating was expected in future studies. Second, subjective language proficiency was collected from the participants. Future studies could consider collecting objective L2 language proficiency by brief tests, such as LexTALE (Lemhöfer & Broersma, Reference Lemhöfer and Broersma2012) to holistically describe participants’ L2 proficiency. Third, as the majority of the words were high-frequency words, we did not add the option of “I do not know the word.” However, it is still advisable that future studies could consider adding this option to identify any possibility that the participant might not recognize the word. This would provide a more comprehensive understanding of participants’ responses and help to address potential limitations in the study. By including this option, researchers can better account for individual differences in word recognition and ensure that the results are more accurate and reliable. Fourth, using translation tools to identify translation equivalents is effective and efficient. However, we were unable to determine multiple translations through this method. Therefore, future studies should also take into account the ambiguity in translation, especially for emotion-label words that might exhibit significant variance in cross-language comparisons (Basnight-Brown et al., Reference Basnight-Brown, Kazanas and Altarriba2020). Finally, a number of emotion-laden words were chosen in the present study. Regrettably, this choice might give rise to a rather low correlation between the current database and other databases. In future investigations, it would be advisable to add a greater quantity of emotion-label words and even construct a database that contains solely emotion-label words in order to explore the relationship across different databases.