Introduction
Several studies have documented impaired affect recognition in adults with moderate or severe TBI (Babbage et al., Reference Babbage, Yim, Zupan, Neumann, Tomita and Willer2011; Bornhofen & McDonald, Reference Bornhofen and McDonald2008a; Rosenberg, McDonald, Dethier, Kessels, & Westbrook, Reference Rosenberg, McDonald, Dethier, Kessels and Westbrook2014). Affect recognition impairments have been found to have a negative effect on the rehabilitation of other cognitive functions, such as executive functions (Spikman et al., Reference Spikman, Boelen, Pijnenborg, Timmerman, van der Naalt and Fasotti2013); may negatively affect post-rehabilitation social life (Bornhofen & McDonald, Reference Bornhofen and McDonald2008a; Radice-Neumann, Zupan, Babbage, & Willer, Reference Radice-Neumann, Zupan, Babbage and Willer2007); and may be treatable (Bornhofen & McDonald, Reference Bornhofen and McDonald2008b; Neumann, Babbage, Zupan, & Willer, Reference Neumann, Babbage, Zupan and Willer2015; Statucka & Walder, Reference Statucka and Walder2013). Thus, evaluation of affect recognition is an important element of neurorehabilitation.
Most studies of affect recognition ask respondents to label expressions of emotions on faces in isolation. While experiments using isolated faces have provided rich information on emotion processing in typical children and adults and revealed deficits in patients with TBI (Croker & McDonald, Reference Croker and McDonald2005; Neumann, Zupan, Malec, & Hammond, Reference Neumann, Zupan, Malec and Hammond2014; Rigon, Turkstra, Mutlu, & Duff, Reference Rigon, Turkstra, Mutlu and Duff2016; Rosenberg et al., Reference Rosenberg, McDonald, Dethier, Kessels and Westbrook2014; Schmidt, Hanten, Li, Orsten, & Levin, Reference Schmidt, Hanten, Li, Orsten and Levin2010; Tonks, Williams, Frampton, Yates, & Slater, Reference Tonks, Williams, Frampton, Yates and Slater2007; Watts & Douglas, Reference Watts and Douglas2006; Williams & Wood, Reference Williams and Wood2010b; Zupan & Neumann, Reference Zupan and Neumann2014), faces alone do not capture the rich visual, cultural, and social contexts in which we interpret affective displays in everyday life (Barrett, Mesquita, & Gendron, Reference Barrett, Mesquita and Gendron2011). Context is important to consider because tests using faces alone may overestimate everyday performance (e.g., if in real life, an individual tends to be distracted by context cues) or underestimate it (e.g., when affect is ambiguous, and real-world context cues add useful information). There is evidence that context exerts a strong influence on recognition and interpretation of affect when that context is visual, as when a face is superimposed on a complex scene (de Gelder et al., Reference de Gelder, Meeren, Righart, van den Stock, van de Riet and Tamietto2006); verbal, as when faces are presented after written comments (Schwarz, Wieser, Gerdes, Muhlberger, & Pauli, Reference Schwarz, Wieser, Gerdes, Muhlberger and Pauli2013); or social, such as faces presented with other faces in the periphery that vary in affect (Mumenthaler & Sander, Reference Mumenthaler and Sander2012). To date, studies of emotion in context have used artificially constructed stimuli (e.g., a face superimposed on a photograph of a scene). The ecological validity of artificially constructed stimuli, and hence their clinical utility, is questionable because the images generally are unlike faces and scenes one might encounter in everyday life. An alternative may be to use photographs of natural scenes that include people with different emotional expressions, which may more closely approximate what is actually encountered in everyday life.
In addition to using everyday scenes, allowing respondents to state what they see rather than choosing from amongst a limited set of options may allow us to capture subtler variations in emotion and capture richer descriptions of recognised emotion. Most affect recognition studies to date have constrained response options to the six ‘basic’ emotions characterised by Ekman (Reference Ekman1992): happy, sad, angry, disgusted, afraid, and surprised. These emotions are thought to be universal across ages and cultures, and identifiable by facial configuration cues (Ekman, Reference Ekman1992; Russell, Reference Russell1994). Whilst studies of basic emotion recognition have been informative, in real-life contexts, we may be more likely to interpret affect cues as reflecting more than just basic emotions. For example, some affective displays can be interpreted as reflecting social emotions that are interpretable only in the context of social information, including information in the social context and interactions with others as well as inferences about others’ mental states (i.e., Theory of Mind; Adolphs, Baron-Cohen, & Tranel, Reference Adolphs, Baron-Cohen and Tranel2002; Buck, Reference Buck1988). Interpretations that reference context and mental states are evidence of the inherent human desire to attribute agency to others (and even to inanimate objects) (Brune, Reference Brune2001), and thus may be a core feature of human social interaction.
Social emotions can be negative, such as guilt and despair; or positive, such as admiration and pride (Tamietto, Adenzato, Geminiani, & de Gelder, Reference Tamietto, Adenzato, Geminiani and de Gelder2007). Social and basic emotions have different developmental trajectories (Klapwijk et al., Reference Klapwijk, Goddings, Burnett Heyes, Bird, Viner and Blakemore2013) and may decline disproportionately with age (Rakoczy, Harder-Kasten, & Sturm, Reference Rakoczy, Harder-Kasten and Sturm2012). There also is some evidence that they are differentially impaired in some clinical populations, including schizophrenia (Bora, Eryavuz, Kayahan, Sungu, & Veznedaroglu, Reference Bora, Eryavuz, Kayahan, Sungu and Veznedaroglu2006), epilepsy (Broicher et al., Reference Broicher, Kuchukhidze, Grunwald, Kramer, Kurthen and Jokeit2012), multiple sclerosis (Charvet et al., Reference Charvet, Cleary, Vazquez, Belman and Krupp2014), and Huntington's Disease (Eddy, Sira Mahalingappa, & Rickards, Reference Eddy, Sira Mahalingappa and Rickards2012). Whilst the term ‘social emotions’ has not been used often in the TBI literature, there is strong evidence of impairments recognising ‘complex emotions’ that involve Theory of Mind and can be characterised as social (e.g., sarcasm) (Channon, Pellijeff, & Rule, Reference Channon, Pellijeff and Rule2005; Martin & McDonald, Reference Martin and McDonald2005; McDonald, Reference McDonald1992, Reference McDonald2000; McDonald et al., Reference McDonald, English, Randall, Longman, Togher and Tate2013; McDonald & Flanagan, Reference McDonald and Flanagan2004; Turkstra, McDonald, & Kaufmann, Reference Turkstra, McDonald and Kaufmann1996). Thus, we may be better able to characterise affect recognition in adults with TBI if we consider not only basic emotions but also social emotions.
Another way to gain a more nuanced understanding of affect recognition is to consider cognitive-state terms, defined as terms referring to what a person is thinking rather than feeling (e.g., cautious, insincere, knowledgeable) (Adrian, Clemente, Villanueva, & Rieffe, Reference Adrian, Clemente, Villanueva and Rieffe2005). Cognitive-state terms have a different developmental trajectory from either basic or social emotion terms (Bretherton & Beeghly, Reference Bretherton and Beeghly1982), and also may be differentially impaired in adults with TBI. For example, Byom & Turkstra (Reference Byom and Turkstra2012) found that typical adults used more emotional terms and fewer cognitive-state terms as topic intimacy increased, whereas adults with TBI did not. Examining each of these three characterisations of emotion – basic emotions, social emotions, and cognitive-state terms – can inform both our understanding of affect recognition in individuals with TBI and also our approach to assessment and intervention.
In summary, tasks using isolated faces and basic emotion choices may give an incomplete account of what respondents are actually thinking when they see a facial expression in everyday life. This limitation has major implications both for our understanding of affect recognition in the real world and for clinical management. As a first step in developing methods to evaluate everyday affect recognition in individuals with TBI, we conducted two studies. First, we asked a group of undergraduate students to label photographs of faces in context or in isolation, using an open-response format (Study 1). We hypothesised that students would be more likely to use basic emotion labels when they saw faces alone, and social emotion labels than when they saw faces embedded in visual scenes. Second, to explore the clinical utility of using context-based, open-ended tasks, we administered the same stimuli to a pilot sample of young adults with TBI (Study 2) and scored their responses using a coding scheme developed from Study I data. As the pilot study was exploratory, we had no specific hypothesis, although the previously cited evidence of impaired affect recognition and Theory of Mind suggested that responses of adults with TBI would differ qualitatively from those of healthy comparison adults (e.g., might not be classifiable using the categories from Study 1).
Study 1
Methods
Participants
Participants were 148 undergraduate students (74 females) from a large university in the Midwest region of the United States. Students completed the study for extra course credit, if they were eligible, or were paid $12.50 USD for their participation. The inclusion criterion was that participants were native English speakers born and raised in the United States, by self-report, to avoid potential confounds due to cultural differences in interpretation of social contexts or differences in language use amongst native and second-language speakers.
Emotion-in-context (EIC) task
The EIC task was comprised of 50 photographs from the LIFE Magazine online archives, chosen because they appeared to be emotionally evocative, captured people in real-life scenarios, showed at least one individual with a clearly visible facial affect display, had visual contexts that could influence interpretation of affective displays, and appeared to depict a range of basic and social emotions (emotion types were not determined a priori, as that was the goal of the study). Each photograph was presented in one of two formats: As a full photograph in its original form, with a 1-inch square box drawn around the face to be labelled (face-in-context items, FC), and as a cropped image of only the face (face only, FO). Sample FC and FO stimuli are shown in Figure 1. Photographs were presented in one of two surveys administered online. Each photograph was randomly assigned to appear in either the FC or FO condition in survey 1 and was then shown in the alternate format in survey 2; thus all 50 photos appeared in both surveys. Stimulus order was randomised within each survey. Students were recruited in two waves: 116 students completed Survey 1 (58 females), and 32 completed Survey 2 (16 females).
Sample FC and FO stimuli. Reproduced with permission from Getty Images.
Procedures
Students who met inclusion criteria and consented to participate completed the EIC online via a research survey service. For each image, the participant was asked, ‘What emotion is this person showing?’ Participants were required to type a response for each photograph in order to advance to the next stimulus and completed the task at their own pace. The relevant institutional review board approved all procedures.
Scoring
The second and third authors and one undergraduate research assistant, blinded to stimulus format (FO or FC), assigned each survey response to one of the following categories: (1) basic emotions, which included happy, sad, angry, afraid, disgusted, and surprised, and alternate forms of those words (e.g., happiness, sadness, anger); (2) social emotions, defined as words that could be interpreted only with reference to a social context (e.g., jealous, content); (3) cognitive-state terms, which referred to thinking not feeling (e.g., bored, skeptical, confused); (4) appraisals that did not refer to the pictured individual's emotion but instead were the rater's opinions about the pictured person (e.g., infamous) or the person's appearance (e.g., glamorous); and (5) no interpretable response, which included answers with no data or random letters, non-words (e.g., laughter-full), and responses that were observations rather than emotions (e.g., laughter, winning). After individually coding each response, the three raters held a consensus conference in which each answer was discussed. This first analysis revealed that many responses were synonyms for basic emotions (e.g., gleeful vs. happy), so the second author identified synonyms for the six basic emotions using the online dictionary Merriam-Webster.com and re-categorised responses using these synonyms. An undergraduate research assistant, blinded to stimulus condition, independently categorised all responses using the same categories as the second author, and inter-rater agreement was 99.81% for Survey 1 and 95.92% for Survey 2. All disagreements were resolved by discussion.
Statistical analysis
The hypothesis was that context would influence responses; that is, respondents would produce more basic emotion labels for a face shown in the FO condition and would be more likely to use a social emotion label when the same face was shown in the FC condition. This hypothesis was tested with a chi-squared test comparing the proportion of basic vs. social emotion labels for the same items in the two conditions. For this comparison, basic and basic synonym categories were combined. The criterion alpha was set at .025 to correct for the two comparisons.
Results
Table 1 shows the per cent of label types in each category across all items in both EIC task versions. There were 7,400 total responses. When we eliminated grammatical variations of words (e.g., bliss, blissful) and pseudo-variations (e.g., blissfulness), there were 715 unique words. Synonyms varied primarily in magnitude (e.g., joy, glee, cheerful). Sample responses for Figure 1 are listed in Appendix A. The full list of responses is available from the authors on request.
Per cent of All Study I Responses in Each Category (N = 7,400)
There were no sex-based differences for any variable, t’s < .01, p’s < .57, so data from men and women were combined. There was no significant difference in proportion of basic vs. social emotion labels for the same items in the two conditions, χ(1) = .04, p > .05.
When basic emotions and basic emotion synonyms were combined across both tasks (see Table 2), 42% of responses were in this category. Social emotion labels were the next largest category, also varied in magnitude (e.g., concerned vs. tormented), and were positive (e.g., proud, fulfilled, triumphant, grateful) or negative (e.g., contempt, dejected, jealous, frustration, disdain). Appraisal terms comprised the next largest category of responses and included opinions related to the face or scene, such as awkward and mundane. Cognitive state terms were the smallest category and included words such as distracted and serious.
Per cent of All Labels that Represented Basic Emotions or Basic Emotion Synonyms, and Other Categories (Social Emotions, Cognitive-state Terms, Appraisals) for Face-only (FO) Items vs. Face-in-context (FC) Items. ‘Other’ Responses are Excluded. N = 7,240
Study I Discussion
The aim of this study was to discover how adults would label facial affect when faces were presented alone vs. as part of a visual scene. Context did not appear to affect labels, as the proportion of basic-emotion-type labels was not significantly different when faces were presented alone or in context. Overall, however, most labels were not the basic emotions described by Ekman (Reference Ekman1992). Instead, respondents produced a rich variety of words to label facial affect, including synonyms of basic emotions that differed in intensity, social emotion words, terms that described cognitive states rather than feelings, and words referring to participants’ opinions of pictured individuals.
Study 2
Methods
Participants
Participants were five adult males with moderate-to-severe TBI, recruited from the Midwestern United States as part of a larger study of social cognition in adults with TBI. Participant characteristics are shown in Table 3, including scores from cognitive tests recommended for TBI research (Wilde et al., Reference Wilde, Whiteneck, Bogner, Bushnik, Cifu, Dikmen and von Steinbuechel2010). Although there were no sex-based differences in Study 1, there have been some reports of sex-based differences amongst adults with TBI (Kessels, Montagne, Hendriks, Perrett, & de Haan, Reference Kessels, Montagne, Hendriks, Perrett and de Haan2014; Rigon et al., Reference Rigon, Turkstra, Mutlu and Duff2016), so we chose an all-male sample to avoid confounding results with sex-based differences in emotion recognition. We also chose the youngest participants from the larger study so they were close in age to the undergraduate students from Study 1, to minimise any age effects. TBI severity was defined according to standard injury criteria (Malec et al., Reference Malec, Brown, Leibson, Flaada, Mandrekar, Diehl and Perkins2007), i.e., a loss of consciousness of 30 minutes or more or post-traumatic amnesia of 24 hours or more, or a lowest Glasgow Coma Scale (Teasdale & Jennett, Reference Teasdale and Jennett1974) score of less than 13 in the first 24 hours; and evidence of cortical or brainstem damage. Other inclusion criteria were self-identification as a Native English speaker and no reported history of a diagnosis of language or learning disability or neurological disorder affecting the brain, other than the TBI. Exclusion criteria were failing a pure-tone hearing screening test at 20 dB HL at 500, 1,000, 2,000, and 4,000 Hz; failing standard screenings for far and near vision; or testing in the aphasic range on the Western Aphasia Battery Bedside Screening Test (Kertesz, Reference Kertesz2006).
TBI Participant Characteristics. CVLT = California Verbal Learning Test (Delis, Kramer, Kaplan, & Ober, Reference Delis, Kramer, Kaplan and Ober2000), Trails B = Trailmaking Test Part B (Tombaugh, Reference Tombaugh2004), WAIS PSI = Wechsler Adult Intelligence Scale (Wechsler, Reference Wechsler2008) Processing Speed Index. CVLT and Trails B scores are z-scores, and WAIS PSI scores are scaled scores, M = 100, SD = 15
a Medical records not available. Patient reported that he was unconscious for 6 weeks.
EIC Task
The EIC task was administered individually via a laptop computer, in a quiet room. Stimuli were presented in the order and format of Survey 2 above. Faces in the FO condition and boxed faces in the FC condition were centred on the screen (i.e., scenes were displayed so that the boxed face was at the centre of the screen). Forty-seven of the 50 faces (boxed in the FC condition or alone in the FO condition) were sized at 1 to 2 inches per side. Three images of faces alone had one or two sides that were 3 inches. Participants were seated approximately 16 inches from the laptop display. For each image, the participant was asked, ‘What emotion is this person feeling?’ Spoken responses were written down verbatim by a research assistant seated out of sight of the participant. Participants completed the task at their own pace.
Procedure
Participants completed the EIC task on their second day of testing for the larger study. Within that session, the EIC task was scheduled after a non-emotion task to avoid priming of affective responses. Participants had completed two forced-choice emotion recognition tasks in a previous testing session that was completed one day to two-and-a-half months prior to the EIC, depending on participant availability and scheduling constraints. Potential effects of that earlier testing will be considered in the discussion. The relevant institutional review board approved all procedures, and the authors assert that all procedures contributing to this work comply with the ethical standards of the relevant national and institutional committees on human experimentation and with the Helsinki Declaration of 1975, as revised in 2008.
Scoring
Responses were scored using a two-step process. If the response was one of the 719 unique words from Study I, the scorer assigned the category that was assigned in Study I. If the response was not on the Study I list, a trained undergraduate student categorised that response using the rules developed for Study I. A second independent rater, blinded to photo condition, also categorised those responses and inter-rater reliability was 89%. All differences were resolved by discussion. When a participant gave more than one response, we recorded all responses but scored only the first one, as emotion labelling in real life is an automatic process and we wanted to capture participants’ most immediate responses.
Statistical analysis
We calculated the number of labels in each of the five categories from Study 1: (1) basic emotions or basic emotion synonyms; (2) social emotions; (3) cognitive state terms; (4) appraisals; and (5) no interpretable response. These data are reported descriptively. As an exploratory comparison of response patterns in participants with vs. without TBI, we also used a chi-squared test to compare the proportion of basic/basic synonym vs. other labels that were given for items in the FO vs. FC conditions.
Results
Rates of item labelling in each of the five categories based on 245 responses (five responses were missing data because of technical errors) are shown in Table 4, and responses by condition (FO vs. FC) are shown in Table 5. Responses were distributed across the five categories, and 71% of labels were words other than basic emotions or basic emotion synonyms. There was no significant difference between the TBI group and the undergraduates from Study 1 in per cent of responses that were basic emotions/basic emotion synonyms vs. other categories, χ 2 (1) = .86, p > .05. Like participants in Study 1, participants with TBI gave fewer basic emotion or basic emotion synonym labels than other types of labels overall.
Per cent of all TBI Group Responses in Each Category
Per cent of all TBI Group Labels that Represented Basic Emotions, Synonyms, or other Categories (Social Emotions, Cognitive-state Terms, Appraisals) for Face-only (FO) Items vs. Face-in-context (FC) Items. ‘Other’ Responses are Excluded. N = 216
Several participants with TBI gave multi-word responses, which prompted us to compare the number of multi-word items in Study 1 vs. Study 2 as exploratory analysis. In Study 1, .01% of responses contained more than one descriptor (e.g., ‘nostalgic/tired’, ‘sad/worried’), and one-third of these (.003%) were from one participant who gave multiple labels for 44 of the survey items. In Study 2, 21% of responses contained more than one descriptor (e.g., ‘either tired and disgusted or in pain’), and an additional 28% were multi-word descriptors of one construct (e.g., ‘looks disgusted, very mild disgusted’).
General Discussion
When adults with and without TBI were asked to label emotions in photographs of everyday people, they used a rich variety of words. A minority of these words were the six canonical basic emotion words described by Ekman (Reference Ekman1992) and commonly used in TBI research. Many words were synonyms of the six basic emotion words, varying primarily in magnitude, but others represented inferences about the pictured individuals’ thoughts and intents beyond basic emotions. As Barrett, Henzi, and Rendall (Reference Barrett, Henzi and Rendall2007) noted, ‘There is abundant evidence that we are heavily prone to perceiving and interpreting other components of the world, besides ourselves, in anthropocentric terms’ (p. 563). In other words, we tend to view others’ behaviours through the lens of our own thinking, and that means considering not only others’ displayed emotions but also the thoughts and intents behind them (i.e., using Theory of Mind). Identifying affect for the sake of affect may not always be helpful in our social world, where relations amongst affect displays, spoken words, and thoughts can be complex and changeable.
There was a trend for adults with and without TBI to use more social emotion labels when faces were shown in a visual scene than when they were shown alone, and participants with TBI gave more multi-word answers than undergraduates, sometimes expressing more than one emotion and other times qualifying the emotion they identified. In the remainder of this paper, we consider these findings in the context of prior studies of affect recognition and discuss implications for clinical assessment.
Context Effects
Although there was no significant difference between labels for faces alone vs. in context, the high per cent of non-basic responses overall suggested that participants were imposing a social or personal context on the images, even when none was present. For example, the first item in Survey 1 was a face alone, and whilst most responses were basic emotions related to ‘sad’, there also were responses such as ‘condescension’, ‘intrigue’, and ‘jealous’, which imply a social context. The automatic imposition of context is consistent with evidence that it is processed automatically when we look at facial expressions (see review by Barrett et al., Reference Barrett, Mesquita and Gendron2011). Automatic processing makes sense from a cognitive resource perspective, as it is much more efficient than having to consciously think through ways in which the context could influence what we are seeing on a communication partner’s face (and might lead to labelling errors; Fernández-Dols, Carrera, Barchard, & Gacitua, Reference Fernández-Dols, Carrera, Barchard and Gacitua2008). It also explains why we often are wrong in our inferences about others, particularly if we interpret affect cues based on our own mental states and experience.
We studied visual context, but context can be spoken, gestured, or even sensed (e.g., via physical touch). It can be literal or implied (e.g., a person saying ‘I'm sad’ vs. ‘I had a bad day’), semantic (e.g., knowing facts about a person or situation) or pragmatic (e.g., in the case of cultural differences), and internal to the viewer (e.g., a person who is in a bad mood might judge others more negatively) or a product of the viewer's inferences about others’ mental states (e.g., a person who is in a bad mood might read a ‘neutral’ expression as happy). Context and affective displays also change over time, and assessment tools must capture these dynamic and fleeting cues (McDonald & Saunders, Reference McDonald and Saunders2005). Given the complex and dynamic nature of affective and context cues, it is no surprise that all of us make emotion labelling errors and that emotion recognition is vulnerable to damage in adults with TBI.
The high per cent of basic emotion responses overall suggests that basic facial configuration cues have a powerful influence on our perceptions of emotions, or at least on our labels for these emotions, but a surprising number of labels in both FO and FC conditions were social emotions, cognitive-state terms, or evaluations of the person's attributes. This was particularly true for participants in the TBI group, who labelled more than 70% of items with a word other than a basic emotion or basic emotion synonym. This finding was consistent with other evidence that non-canonical emotion labels emerge if response choices are unconstrained. For example, Limbrecht-Ecklundt and colleagues (Reference Limbrecht-Ecklundt, Scheck, Jerg-Bretzke, Walter, Hoffmann and Traue2013) showed 33 typical adults a set of 96 photographs that had been validated using forced-choice methods as showing the six basic Ekman emotions. When participants labelled these emotions using an open-label format, they produced 245 unique words, of which judges were unable to categorise 106 (44%) to any of the six basic emotion categories. The per cent of basic emotions may have been even smaller than Limbrecht-Ecklundt et al. (Reference Limbrecht-Ecklundt, Scheck, Jerg-Bretzke, Walter, Hoffmann and Traue2013) reported, as several responses categorised by Merriam Webster as basic synonyms could be viewed as social, such as helpless, approvingly, irony, suspicion, unfavourable, worried, loathing, and patronisingly. Thus, open-label formats may be more likely to yield responses other than basic emotions.
The open-label format revealed interesting patterns in adults with TBI. One finding was that adults with TBI produced more than twice as many words as there were items in the task, which is consistent with the tendency for some individuals with TBI to be verbose (Moran, Kirk, & Powell, Reference Moran, Kirk and Powell2012); although the reverse also has been reported (e.g., by Coelho et al., Reference Coelho, Le, Mozeiko, Hamilton, Tyler, Krueger and Grafman2013). In most cases, these labels were variations on a single label (e.g., ‘pensively thinking about something, concentrating’, ‘Blah, neutral, unenthused’), again consistent with a tendency to say more when less will suffice. In some cases, however, the labels had different meanings (e.g., ‘bored, agitated’, ‘questioning, a bit disgusted’), suggesting that open-ended tasks can reveal error patterns in adults with TBI beyond simple accuracy measures. The total word data must be viewed with the caveat that undergraduates answered in writing and participants with TBI gave spoken responses, but neither was constrained by instructions or time (or typing space, in the case of Study 1), so both had the opportunity to write more than single words. In the future, it may be interesting to use a ‘think-aloud’ protocol to learn more about real-time affect processing in people with TBI, as has been done in studies of emotion labelling in typical adults (e.g., Lee, Liu, Chan, Fang, & Gao, Reference Lee, Liu, Chan, Fang and Gao2005). There are epistemological concerns about think-aloud protocols, however (see review by Hansen, Reference Hansen2005), and they also pose dual-task demands that could confound interpretation of results in individuals with cognitive impairments.
Cognitive-State Terms
Cognitive-state labels accounted for about 6% of undergraduate students’ responses. By contrast, they accounted for 31% of TBI group responses. In a previous study of conversations between men and familiar male partners (Byom & Turkstra, Reference Byom and Turkstra2012), men with TBI used more cognitive-state terms and fewer emotion words when the topic became more intimate, whereas men without TBI showed the expected pattern of talking more about emotions when topics were more intimate. If these findings are replicated, they would fit with evidence that emotional functioning can be impaired at many levels in adults with TBI, including not only recognition of emotions shown by others, but perceiving one's own emotions and having empathy towards others (Neumann et al., Reference Neumann, Zupan, Malec and Hammond2014; Williams & Wood, Reference Williams and Wood2010a). These findings have clear implications for intervention, as they imply that training affect recognition alone will not fully address the multi-faceted social problems of adults with TBI.
Limitations and Future Directions
A limitation of the study was that the use of news photographs was at the expense of experimental control. Photographs varied in image quality and in number and nature of elements and people in each scene, so they are not directly comparable to each other or to other stimulus sets. Future studies might use experimenter-generated images of scenes, to control factors such as exposure, focal length, and spatial frequency.
As there was no existing scoring system for stimuli like these, we had to develop study-specific rules and others might debate these rules. For example, Merriam Webster lists mousy and fainthearted as synonyms for fear, and thankful and satisfied as synonyms for happy, but it could be argued that some of these words refer to a social context (e.g., mousy is a social judgement). Likewise, we classified words like worry and amusement as social emotion words because their interpretation was related to social norms and expectations, but in some systems they would be considered variants of fear and happiness. Our categorisation rules were reliable across raters, so the results were internally consistent, but the validity of these rules still may be debated.
A recent paper by Rosenberg, McDonald, Rosenberg, and Westbrook (Reference Rosenberg, McDonald, Rosenberg and Frederick Westbrook2016) described a novel standardised emotion recognition task, the Complex Audio Visual Emotion Assessment Task (CAVEAT), which considers both the categorisation scheme used in our study (i.e., basic vs. social emotions) and also valence and complexity dimensions. CAVEAT scores differentiated adults with TBI from controls, and were significantly correlated with measures of mood, apathy, and disinhibition, as well as number of friends. CAVEAT is based on video vignettes rather than still photographs, and thus may better capture cognitive processes engaged in everyday emotion recognition.
We chose a categorical analysis method, and other approaches might be informative. Alternatives to categorical models are dimensional models and their close relatives, circumplex models (after work by Russell, Reference Russell1980). Dimensional and circumplex models address how we construct mental representations of emotion, and how we use these mental representations to make inferences about others’ feelings. These models grew out of early work by Schlosberg (Reference Schlosberg1952), who proposed that emotions could be conceptualised along two dimensions: pleasant–unpleasant and attention–rejection (labelled arousal and valence in later models; Russell, Reference Russell1980). Schlosberg (Reference Schlosberg1952) asked participants to sort cards showing photographs or words so that emotions they thought of as more related were closer to each other in space. The aim was to characterise participants’ mental representations of different emotions in a similarity space; in other words, to view emotions as part of an organised system of ‘fuzzy sets’ rather than a finite number of discrete categories (Shaver, Schwartz, Kirson, & O'Connor, Reference Shaver, Schwartz, Kirson and O'Connor1987). To construct this space, Shaver and colleagues (Reference Shaver, Schwartz, Kirson and O'Connor1987) asked undergraduates to sort 135 emotion word cards into groups of similar words. Words included many of the responses seen in our study, such as synonyms of basic emotions (e.g., grief, sorrow, despondency) and what we defined as social emotion words (e.g., pity, disappointment, rejection, encouragement). The authors then generated a 135 × 135 co-occurrence matrix showing the number of times each pair of cards was put in the same category, totalled these matrices across participants, and used cluster analysis to group item totals in a multi-dimensional similarity space. Item groupings supported both the notion of basic emotions (i.e., clusters were related to anger, sadness, fear, joy, surprise, or love) and a three-dimensional organisation of emotions (valence, arousal, and dominance). Relevant to the present study, this analysis method allowed categories to emerge based on participants’ mental representations of emotions.
A multi-dimensional, bottom-up analysis approach may be fruitful for comparing typical and clinical groups, as asking participants to group photographs rather than labelling not only could identify atypical semantic networks but also obviate the need for verbal responses when language problems are a concern. There is early evidence that dimensional models are important parameters for measuring social cognition in clinical populations, as either or both can be affected. For example, Rosenberg and colleagues (Reference Rosenberg, McDonald, Dethier, Kessels and Westbrook2014) reported that adults with TBI had disproportionate deficits in recognising emotions from partial affect displays (e.g., 40% vs. 100% of a frown), but showed no interaction of valence type and error rate. These factors could be important to consider in assessment and intervention.
Because our goal was to examine context effects and response patterns rather than response accuracy, the results do not tell us if participants in either group gave ‘correct’ answers. Indeed, we would argue that there are several acceptable responses for many of these images, because at the point at which participants view them the pictured affect is underdetermined. The faces have a visual context, but as Barrett and colleagues (Reference Barrett, Mesquita and Gendron2011) noted, affect also is influenced by both the viewer's perceptual processes (the ‘perceiver-base context’), and the cultural context in which the viewer judges the image, as well as the types of context factors discussed above (e.g., spoken or sensed contexts).
Perhaps, the most important limitation of our stimuli, at least in terms of ecological or predictive validity, is that participants had unlimited time to respond. Full-face emotional expressions may last between 0.5 to 4 seconds (Ekman, Reference Ekman2003), and micro-expressions, which are most often discussed in the context of lie detection (Ekman & Friesen, Reference Ekman and Friesen1969), are even more fleeting – as brief as 150 milliseconds (Yan, Wu, Liang, Chen, & Fu, Reference Yan, Wu, Liang, Chen and Fu2013). Video assessments such as The Awareness of Social Inference Test (Flanagan, McDonald, & Rollins, Reference Flanagan, McDonald and Rollins1998) and morph tasks such as the Emotion Recognition Test (Kessels et al., Reference Kessels, Montagne, Hendriks, Perrett and de Haan2014; Montagne, Kessels, De Haan, & Perrett, Reference Montagne, Kessels, De Haan and Perrett2007) are a step towards analysis of in-the-moment affect recognition, but finer-grained timing measures are needed.
It is possible that participants with TBI were influenced by having completed forced-choice affect labelling tasks in a prior testing session, as part of the larger study in which they were enrolled. This prior exposure would have primed participants to give basic emotion labels, as those were the labels on the prior tasks. One participant completed session 1 on the previous day, and inter-session times for the other four ranged from 14 to 75 days. These are long intervals for priming effects, and participants completed several non-emotion tests and tasks on each day, but effects of prior exposure to basic emotion choices cannot be excluded. Thus, it is possible that participants with TBI would have used even fewer basic emotion labels than observed here.
A potential limitation in regard to participants with TBI was that some might have received therapy for affect recognition. Our clinical impression is that affect recognition treatment is relatively rare outside of experimental settings, but the potential contribution of direct treatment should be considered in future studies.
Clinical Implications
Impairments in affect recognition have been reported in individuals with a wide variety of brain disorders in addition to TBI, including multiple sclerosis (Charvet et al., Reference Charvet, Cleary, Vazquez, Belman and Krupp2014; Henry et al., Reference Henry, Phillips, Beatty, McDonald, Longley, Joscelyne and Rendell2009), brain tumors (Mu et al., Reference Mu, Huang, Li, Ke, Chen, Jin and Chen2012), stroke (Yuvaraj, Murugappan, Norlinah, Sundaraj, & Khairiyah, Reference Yuvaraj, Murugappan, Norlinah, Sundaraj and Khairiyah2013), Parkinson disease (Heller, Dogan, Schulz, & Reetz, Reference Heller, Dogan, Schulz and Reetz2014), Huntington's Disease (Kipps, Duggins, McCusker, & Calder, Reference Kipps, Duggins, McCusker and Calder2007), frontotemporal dementia (Kumfor & Piguet, Reference Kumfor and Piguet2012), and Mild Cognitive Impairment (McCade, Savage, & Naismith, Reference McCade, Savage and Naismith2011). The prevalence of emotion recognition impairments supports the need for standardised assessment tools. There are a few existing clinical tools for assessment of emotion recognition (e.g., Flanagan et al., Reference Flanagan, McDonald and Rollins1998; Schlegel, Grandjean, & Scherer, Reference Schlegel, Grandjean and Scherer2014; Schlegel & Scherer, Reference Schlegel and Scherer2015), but these focus on basic emotions. This is in contrast to the research literature, in which there has been debate about whether the six basic emotions have truly unique physiognomy (Jack, Garrod, & Schyns, Reference Jack, Garrod and Schyns2014), whether they are the building blocks for more complex emotions (e.g., ‘happy surprise’, Du, Tao, & Martinez, Reference Du, Tao and Martinez2014), if they are truly universal (Russell, Reference Russell1994), and how we recognise gradations of those basic emotions (e.g., slightly angry vs. very angry; Kessels et al., Reference Kessels, Montagne, Hendriks, Perrett and de Haan2014). Future clinical assessment tools could consider this more nuanced view of emotions and how they are manifest in facial affect.
Future assessments also might use open-label methods rather than forced choice. Forced-choice methods yield relative judgements (i.e., participants choose whatever is closest to what they already have in mind), and if respondents’ spontaneous labels are missing from the choices, the examiner will not know what respondents actually think. As Russell (Reference Russell1993) pointed out more than two decades ago, answers can be ‘wrong’ not only if they are slightly different from the prototype, but also if they represent fundamentally different emotions. Russell gave the example of anger vs. contempt, and showed that when contempt was included in the choice options, 76% of participants chose this label for a stimulus labelled as anger in the iconic photographs used by Matsumoto and Ekman (Reference Matsumoto and Ekman1994). This was about the same percentage that labelled that stimulus as anger when only the basic six emotions were included. Whilst open-ended methods and natural scenes have the limitations discussed earlier, the benefits of capturing everyday emotion recognition may outweigh the costs.
Conclusions
There is growing evidence of impairments in affect recognition amongst adults with TBI, but data are primarily from studies using forced-choice recognition of affect in isolated faces. In this study, young adults with and without TBI labelled facial affect with a rich variety of words that did not fit easily into basic emotion categories and were influenced by the visual context in which faces were shown. Results support the importance of studying affect recognition in contexts that more clearly approximate the way we perceive others’ emotional displays in everyday life and using methods other than forced-choice. Studies using these methods will not only inform our understanding of basic mechanisms underlying emotion processing, but also allow us to translate laboratory – and clinic-based findings into predictions about everyday emotion recognition in individuals with TBI.
Acknowledgements
The authors thank Emily Hosokawa and Kim Vidmar for assistance with data scoring and paper preparation, Dr Jee-Seon Kim for statistical advice, and Dr Julia Evans for comments on an earlier draft of the paper.
Financial Support
This work was supported by the National Center for Medical Rehabilitation Research/National Institutes of Health grant number R01 HD071089, and the Walker Foundation (L.T.).
Conflict of Interest
None.
Ethical Standards
The authors assert that all procedures contributing to this work comply with the ethical standards of the relevant national and institutional committees on human experimentation and with the Helsinki Declaration of 1975, as revised in 2008.
