2.1. Participants

Participants were recruited from a Canadian university via institutionally approved campus advertisements and invitations posted to course websites. They received either monetary compensation ($10) or course credit for their participation. Only participants who were at least 18 years of age, spoke English as a primary language, and had no uncorrected vision deficits were eligible. Eighty-nine participants (17 males; 72 females), ranging in age from 18 to 54 years (M = 23.49; SD = 6.41) participated. The ratio of males to females in this sample is not balanced, but approximates that typically observed in introductory psychology courses of the type we recruited from (Dickinson et al., Reference Dickinson, Adelson and Owen2012). Given the inbalance we did not conduct gender-based analyses. On average, participants had completed 16.07 (SD = 1.76) years of education. The majority of participants (79.8%) self-identified as Caucasian.

2.2. Stimuli

Emotion words and/or phrases (maximum six words) were generated via an unpublished study (Zupan & Babbage, Reference Zupan and Babbage2008) in which 82 participants (34 males; 48 females) freely labelled facial (dynamic and static) and vocal emotion expressions. These emotion expressions had been elicited in 10 male and 10 female speakers using stimuli (film clips; novel excerpts; Zupan & Babbage, Reference Zupan and Babbage2017) equally representing happy, sad, angry, fearful and neutral emotions. During this free labelling tasks, participants provided a total of 494 unique words/phrases (hereafter referred to as emotion words). Three words were erroneously included twice, resulting in a total of 497 words. As in Strauss and Allen (Reference Strauss and Allen2008), the words were randomly divided into three lists, each consisting of 166–167 words. Each participant was presented with one list of words. The three duplicate words each appeared in different lists so no participant received any word twice. Ratings for the repeated words were combined in analyses.

2.3. Procedure

The cross-sectional design required that participants individually attend a single 60-minute face-to-face session. Each participant first completed a brief demographic questionnaire. Participants were then sequentially assigned to one of the three-word lists in the order in which they entered the study, with a total of 30 participants assigned to lists one and two, and 29 to list three. Using Cedrus SuperLab software, words were presented in size 36 black serif font on a white background via a 21.5-inch computer screen. To control for potential order effects, word order was randomised.

Using 9-point Likert scales, participants were prompted to rate each word first on intensity (1 = extremely low; 9 = extremely high), then valence (1 = extremely negative; 9 = extremely positive), then the degree to which the word represented each of five emotion categories (i.e. happy, sad, angry, fearful, neutral); the order of these categories remained consistent for each stimulus. The instructions for these ratings is provided in the Appendix. These five categories are considered distinctive without requiring complex appraisals, thoughts or judgments (Ekman & Cordaro, Reference Ekman and Cordaro2011) and are commonly used as the minimal response options in emotion studies (Airdrie et al., Reference Airdrie, Langley, Thapar and van Goozen2018; Ben-David et al., Reference Ben-David, van Lieshout and Leszcz2011; Zupan & Babbage, Reference Zupan and Babbage2017). Evidence suggests that some emotion words present with neutral semantics (Larsen et al., Reference Larsen, Mercer and Balota2006), hence the use of a neutral category. These categories also align to those used in previous studies aimed to build a standardised corpora of emotion words in English (Stevenson et al., Reference Stevenson, Mikels and James2007; Strauss & Allen, Reference Strauss and Allen2008). Following Strauss and Allen (Reference Strauss and Allen2008), we opted to have participants rate words on valence and intensity. We opted to use intensity rather than arousal, another important dimension of emotion perception, because the former is easier to present to participants (i.e. it does not require visual depictions). Though intensity and arousal are not the same, Laukka et al. (Reference Laukka, Juslin and Bresin2005) report a strong correlation between the two.

Participants used a response keypad with only 10 keys; numbers one through nine appeared on two lines (equally spaced); a tenth key labelled ‘next’ appeared on a third line. Limiting and spacing the keys reduced the likelihood of participants mistyping. Participants were not provided feedback at any time.

2.4. Data analysis

Descriptive statistics were calculated for demographics, valence, intensity, and degree-of-fit ratings, but only degree-of-fit ratings are reported here. Mean degree-of-fit ratings ranged from 1 to 9 in each of five emotion categories; higher scores indicated a better fit for the stimulus word. Since nine was the highest possible score, we considered ratings of 8.00 or greater as indicating the emotion category provided an excellent fit for the stimulus word.

Next, we calculated a Simpson Diversity Index Score (SDIS) using degree-of-fit ratings, an approach that has been applied in other studies with a psychosocial focus (Koffer et al., Reference Koffer, Ram, Conroy, Pincus and Almedia2016; Quoidbach et al., Reference Quoidbach, Gruber, Mikolajczak, Kogan, Kotsou and Norton2014; Ram et al., Reference Ram, Conroy, Pincus, Hyde, Molloy, Harring and Hancock2012; Zupan & Eskritt, Reference Zupan and Eskritt2022). The SDIS takes into account the emotion category that had the highest degree-of-fit rating and a frequency count for the number of times each emotion category was rated highest for each word. The resulting index is a nonparametric statistic representing the dispersion of scores across the emotion categories for each word. Dispersion values fall between 0 and 1, with values closer to 0 representing a more definite categorisation of the emotion. As the value of the SDIS increases, so does diversity across participant responses (Gregorius & Gillet, Reference Gregorius and Gillet2008; Zupan & Eskritt, Reference Zupan and Eskritt2022). Using existing guidelines, words with values between 0.1 and 0.4 were considered to have a low degree of diversity, words with values between 0.41 and 0.6 were classified as having moderate diversity and those with values of 0.61 or higher were considered to have widely dispersed ratings (Guajardo, Reference Guajardo2015). Thus, a value of 0.61 or higher indicates that when asked to rate the degree to which a word fit each emotion category, participants rated numerous emotion categories similarly.

3.1. Participants

Participants were recruited online using snowball sampling on the authors’ social media platforms. We required that participants be at least 18 years of age and speak English as a primary language. All participants provided informed consent. A total of 113 participants began the study; however, six participants exited the survey before completing the demographic questions. Of the remaining 107 participants, only 56 (7 males; 49 females) proceeded beyond the demographic questions to the ratings. These 56 participants ranged in age from 19 to 72 years (M = 47.5; SD = 12.63) and had completed 16.8 (SD = 3.12) years of education on average. Thirty-four participants were from Canada, 12 from Australia, six from the United States, and four from the United Kingdom.

3.2. Stimuli

In selecting stimuli to be used in Study 2, we did not rely solely on degree of fit or SDISs from Study 1. We made this decision because it appeared that limited response options may have influenced Study 1 ratings. This was reflected in the data in two ways. First, review of individual participant data showed that for some words (e.g. astonished, apprehensive, eager, miserable, sympathetic) no categories were identified as a good fit. In these cases, participants either selected ‘1’ for all five categories or selected ‘1’ for all but one category and gave the fifth category a low rating. Second, the SDISs for words in the three negatively-valenced categories (Sad, Angry, Fearful) were moderate to high, with very few words receiving low scores. It is unclear whether the higher overall diversity was because words that fell into these categories could be interpreted as any one of these three negatively-valenced emotions, or whether participants identified the word as negative, but did not feel that any of these three basic categories accurately fit the word. For example, guilty was identified as Sad with a degree-of-fit rating of 6.45. However, Angry and Fearful had similarly high ratings of 5.34 and 6.03, respectively, resulting in an SDIS of 0.68.

To reduce the length of task and minimise the likelihood of cognitive fatigue, a particular risk in online studies (Saleh & Bista, Reference Saleh and Bista2017), we opted to decrease the size of stimulus set in Study 2. We carefully reviewed the Study 1 words and excluded the following types of words: those that were better characterised as relating to appearance or character than emotion (e.g. ugly, goofy, silly, dumb), those considered slang (e.g. snarky, miffed), phrases (e.g. creeped out, put upon), and words that matched the category names of the response options (e.g. happy, angry). Next, all three authors independently reviewed the word list for each emotion category, and identified words that they would use in the sentence ‘I feel [WORD]’. Only words identified by all three authors as fitting within that sentence frame were considered in the next step of the selection process. As was the case in Study 1, the total number of words remaining in each category varied widely, but Happy still included the highest total number (n = 23). Table 2 lists the words identified from Study 1 for possible use in Study 2, including mean degree-of-fit ratings for the five emotion categories and SDISs.

Table 2.

Mean rating and Simpson diversity index for words from study 1 considered for inclusion in study 2

Next, the word list for each emotion category was analysed for word frequency to ensure, inasmuch as possible, that participants would be familiar with the words. Since Study 2 would be conducted online and include participants with English as a first language across different countries, we calculated word frequency using the average rating between the Corpus of Contemporary American English (Davies, Reference Davies2008) and the British National Corpus (Davies, Reference Davies2004) (see Table 2). We then selected 30% of the most frequently used words in each emotion category. This yielded a stimulus set of 24 words (identified in Table 2 using an *) distributed as follows: Happy (7); Sad (7); Angry (3); Fearful (6); Neutral (1).

3.3. Procedure

The study was conducted online using the Survey Monkey platform (http://www.surveymonkey.com). After viewing study details and providing informed consent, participants confirmed they met inclusion criteria and then answered demographic questions regarding gender, age, country of residence, and years of education. Next, they were presented with a word and asked to rate how well the word fit a number of different categories (see the Appendix), using a scale of 1 to 9 (1 = not at all; 9 = extremely well). Response options were ordered alphabetically and remained consistent for each stimulus.

Thirteen categories were provided as response options. Included were an equal number of basic and complex emotions, and an equal number of positive and negative valenced emotions. Neutral was also retained as a response option. To extend the basic emotion categories to six, we opted to include Disgust and Joy as response options. Disgust is well known as a basic emotion category (Ekman, Reference Ekman1992). Joy has also been identified as basic and although this category is understood to contain Happy, people appear to differentiate these words by intensity (Zupan & Eskritt, Reference Zupan and Eskritt2020). Surprise was not included because it cannot be clearly identified as positive or negative in valence (Schlegel et al., Reference Schlegel, Grandjean and Scherer2012; Zupan & Eskritt, Reference Zupan and Eskritt2020). The final six emotion categories represented complex emotions – four that were positively-valenced (Amusement, Contentment, Pride, Relief) and two that were negatively-valenced (Anxiety, Irritated).

To maximise participant engagement, Revilla and Ochoa (Reference Revilla and Ochoa2017) report that the ideal length of an online study should be less than 20 minutes. In an effort to meet this target, participants were informed that they could rate as many of the categories as they wished for each word, but if they felt a word did not fit into a category, they could leave the rating blank and a rating of ‘1’ would be assumed. However, a minimum of three ratings was required for each word to increase the likelihood that participants were fully reflecting on the degree of fit across the range of response options. The study was expected to take approximately 15 minutes to complete.

3.4. Data analysis

The same analyses used in Study 1 were applied to Study 2.

4.1. Limitations and future directions

Findings must be considered in light of two types of limitations. The first involves choice of emotion words, both in regard to response options and the word lists. In regard to response options, there are a few notable absences. Though Surprise is one of the six basic emotion categories, we opted not to include it, both because it is not clearly positive or negative and because we sought an equal number words of each valence. However, future studies might consider including Pleasant Surprise, which has been successfully used in studies of vocal emotion recognition (Dupuis & Pichora-Fuller, Reference Dupuis and Pichora-Fuller2015), to make this distinction. We also did not include Compassion (considered a social emotion) or Contempt (a negative emotion, related to Anger and Disgust) (Ekman, Reference Ekman1992). Also, some of the words we had participants rate could be considered their own category, particularly if we look to more social emotions (e.g. guilt, worry). We acknowledge that inclusion of these words may have influenced the pattern of responding seen in Study 2. Future studies might vary the types of emotion words included as response options; specifically, we suggest studies include more social emotions (e.g. compassion) so that researchers can explore how such words are conceptualised. Results also suggest that people may conceptualise words as belonging to word families. Future studies might explore this (e.g. by asking participants to group words that belong together). Such studies have the potential to provide insight into hierarchies and the degree to which intensity influences how people conceptualise emotion words.

Participants were not provided with a ‘none’ or ‘I do not know’ option, something typical of forced-choice paradigms, which require participants to select only one option, regardless of how many they may attribute to the stimulus (Nelson & Russell, Reference Nelson and Russell2013). Instead, we allowed participants the flexibility to rate each stimulus according to its degree of fit with multiple response options. There may still have been times where participants felt uncertain of the meaning of the word; in such instances an escape option such as ‘none’, ‘I do not know’, or ‘unfamiliar’ could have reduced the likelihood of artificial agreements. However, other work (Frank & Stennett, Reference Frank and Stennett2001) suggests that the absence of an escape option is unlikely to have had much effect on the pattern of results. Finally, we acknowledge that order effects are possible since the order of response options remained consistent for each stimulus.

The word list, which was quite extensive in Study 1, was significantly reduced in Study 2. As a result conclusions are based on a relatively small number of words in each category. This makes further investigation of our findings important. It was necessary to reduce the list in Study 2 for pragmatic reasons. To do this, we had to make choices (detailed previously) about which emotion words we would include. Though we used word frequency as a guiding principal it is still possible that some words were unknown to participants. In future studies, researchers might consider including a response option that allows participants to indicate this. Also, for Study 2 we sought to keep word counts in each category in proportion to the category counts in Study 1. In retrospect, we should have selected an equal number of words from each category. The fact that we had relatively few Neutral words made it difficult to assess whether it was being used as a ‘catch all’ category or whether participants actually perceived some words as emotionally neutral. Finally, for Study 2 we chose words which people identified as fitting strongly into one of the five categories from Study 1. However, there were some emotion words where ratings in two categories were quite similar (e.g. sympathetic had similar Happy and Sad ratings, categories that were opposite in valence). Perhaps it is these not easily categorised words that require further investigation. Future studies involving such words might yield more valuable insights into how people conceptualise emotion words.

A second type of limitation concerns context. Participants were not provided any context to guide interpretation of word meaning which may have contributed to the high diversity ratings. Fehr and Russell (Reference Fehr and Russell1984) suggest that the degree to which people rate an emotion as belonging to an emotion category is dependent upon their appraisal of that emotion, which itself depends on their interpretation of the situation in which the emotion occurs (Siemer et al., Reference Siemer, Mauss and Gross2007). It is possible that the absence of context in Studies 1 and 2 limited participants’ ability to categorise words. With no prescribed context participants may have created their own, which would differ from person to person. For example, in Study 1 the primary rating for console was Sad, while the secondary rating was Happy. One would not expect the primary and secondary categories to be opposite in valence. However, perhaps respondents imagined a situation in which they were being consoled but differently appraised the valence of the word depending on whether they focused on the need to be consoled (negative), or its impact (positive). Individual differences in meaning appraisal could similarly have contributed to diversity scores in Study 2. To determine the extent to which the lack of context (or individual variation in inferred context) influences people’s interpretation of a word, future studies may consider using a standard carrier phrase (e.g. If someone said ‘I am feeling X’, I would categorise that word as meaning: happy, sad, etc.). The use of such a carrier phrase does not provide context indicative of a specific word meaning but could support more consistency in approach.