A smile is the chosen vehicle for all ambiguities.
— Herman Melville (Reference Melville1852/1996, Pierre, or, The Ambiguities, p. 84)Smiles can be simple things. Individuals show very high agreement in their assignment of the label “smile” to photographs of facial gestures with certain structural features (Izard Reference Izard1971). This is true across cultures (Ekman Reference Ekman1994; Haidt & Keltner Reference Haidt and Keltner1999). Yet, smiles are also complicated things. Although smiles often communicate that the expresser feels “happiness” or “joy” (Frank et al. Reference Frank, Ekman and Friesen1997; Frank & Stennett Reference Frank and Stennett2001; Messinger et al. Reference Messinger, Fogel and Dickson2001), some smiles signal affiliative intent or a responsiveness to group norms; others express more complex interpersonal or status motivations (Abel Reference Abel and Abel2002; Fogel et al. Reference Fogel, Nelson-Goens, Hsu and Shapiro2000; LaBarre Reference LaBarre1947; Keltner Reference Keltner1995; Tipples et al. Reference Tipples, Atkinson and Young2002).
How do individuals interpret the meaning of a smile? One possibility is that each smile has its own specific facial morphology, which constitutes slightly different visual features. Meanings associated with these configurations could then be learned. Ekman (Reference Ekman2001) identified 18 types of smiles and proposed that there might be as many as 50 in all. If visual facial features did all of the work in grounding meaning, we would expect very few errors in interpreting smiles across individuals and cultures. Yet, even though errors in classification of a smile as such are not frequent (Haidt & Keltner Reference Haidt and Keltner1999), errors of interpretation of specific smile meanings are much more so (Bernstein et al. Reference Bernstein, Young, Brown, Sacco and Claypool2008).
In the present article we argue that observers of smiles sometimes construct an embodied simulation of the nuanced affective state conveyed by the smile that is supported by the brain's reward, motor, somatosensory, and affective systems. They then use this simulation to represent the smile's intended meaning. Our approach also outlines the conditions under which simulations, in contrast to other bases of processing, are actually used, and why and when these alternative processes may lead to errors.
To accomplish our goals, we devote the first section of the article to a review of research on the meaning of smiles from the view of the person doing the smiling. Based on existing functional accounts, we characterize smiles as produced by positive emotion (enjoyment smiles), by positive social motives (affiliative smiles), and as a way of communicating and maintaining social status (dominance smiles).
In the second section of the article, we review research on the neural bases of smile processing in the perceiver of the smile. Possible roles of the brain's reward centers, orbital prefrontal cortex, amygdala, motor regions, and somatosensory cortices are outlined, with accounts of motor processing linked to research on facial mimicry from social psychology and the neurosciences.
In a third section, eye contact is discussed. Our novel proposal is that eye contact automatically triggers an embodied simulation of what a smile means. A large literature in social and developmental psychology supports this claim, and we rely on it to draw conclusions from recent neuroscience findings. Finally, we bring together these summaries to motivate a comprehensive model of smile interpretation for three smile types: enjoyment, affiliative, and dominance smiles.
1. Recognition and access to meaning of expressions
Different processes can support people's ability to recognize smiles and what they mean (Adolphs Reference Adolphs2002; Atkinson Reference Atkinson2007; Kirouac & Hess Reference Kirouac and Hess1999). We begin with visual facial cues and then consider a variety of other cues.
The classification of expressions into basic categories usually relies on a perceptual analysis of the stimuli, sometimes called pattern matching (Buck Reference Buck1984). Smith et al. (Reference Smith, Cottrell, Gosselin and Schyns2005) have shown that distinct facial features can be used to classify facial expressions and that these features correspond to a de-correlation model. In their view, configurations of muscles have emerged during phylogenetic development of the human species that maximize differences between the six basic emotional expressions and produce efficient recognition of these expressions. At the neural level, this process seems to be supported by the occipito-temporal cortices (Adolphs Reference Adolphs2002).
Whereas the analysis of visual facial features may be sufficient to classify prototypical expressions in simple tasks, this process is unlikely to be sufficient to recognize less prototypic, perhaps more realistic, emotional expressions, or to represent their subtle meanings. In such cases, we propose that perceivers must call on various sources of non-visual information, such as conceptual emotion knowledge about the expresser and the social situation (Kirouac & Hess Reference Kirouac and Hess1999; Niedenthal Reference Niedenthal, Lewis, Haviland-Jones and Barrett2008). For example, faces provide information about the sex, age, and race of the other person, and individuals possess expectations and stereotypes about how members of these social groups react emotionally (Hess et al. Reference Hess, Adams and Kleck2005). Such conceptual knowledge about emotion has been shown to exert effects early in the processing of ambiguous facial expressions (e.g., Halberstadt & Niedenthal Reference Halberstadt and Niedenthal2001; Halberstadt et al. Reference Halberstadt, Winkielman, Niedenthal and Dalle2009; Hess et al. Reference Hess, Adams and Kleck2009a).
Embodied simulation also supports the recognition and access to meaning of facial expressions (e.g., Atkinson Reference Atkinson2007; Decety & Chaminade Reference Decety and Chaminade2003; Reference Decety, Chaminade, Hurley and Chater2004; Gallese Reference Gallese2003; Reference Gallese2005; Goldman & Sripada Reference Goldman and Sripada2005; Keysers & Gazzola Reference Keysers and Gazzola2007; Niedenthal Reference Niedenthal2007; Niedenthal et al. Reference Niedenthal, Barsalou, Winkielman, Krauth-Gruber and Ric2005b; Winkielman et al. Reference Winkielman, McIntosh and Oberman2009). When we use the term “embodied simulation,” we mean that a facial expression has triggered a simulation of a state in the motor, somatosensory, affective, and reward systems that represents the meaning of the expression to the perceiver. In an embodied simulation account, the perception of a facial expression is accompanied by the bodily and neural states associated with the expression and its correspondent emotion. This simulation is then used as the representation of meaning on which an interpretation or judgment is based.
In social psychology, such a view partly involves the marriage of facial feedback theory and affect as information theory. The first holds that facial musculature produces afferent feedback that alters subjective state (McIntosh Reference McIntosh1996; Zajonc et al. Reference Zajonc, Murphy and Inglehart1989). The latter holds that when individuals believe that their affective state was caused by the current object of perception, they use that state to evaluate affective features of the object (Clore & Storbeck Reference Clore, Storbeck and Forgas2006). In neuroscience, embodied simulation has been closely linked to the construct of mirror neurons and mirror systems, and the notion that brains resonate with the motor and affective states of perceptual objects with appropriate biological similarity (e.g., Gallese Reference Gallese2007; Keysers & Gazzola Reference Keysers and Gazzola2007).
The rest of this article is about the role of embodied simulation in representing the meaning of the smile. After providing theoretical justification for our account of smiles, we describe the simulation components for different types of smiles and describe the conditions under which the meaning of a smile is represented by this process. The integration of these ideas is called the Simulation of Smiles Model (SIMS).
2. What is a smile?
The smile is characterized by the upward turn of the corners of the lips, which is produced by the contraction of the zygomaticus major muscle (Ekman & Friesen Reference Ekman and Friesen1978). The zygomaticus major, like other muscles involved in the production of facial expression, is activated by the seventh cranial nerve, or the facial nerve (Rinn Reference Rinn, Feldman and Rimé1991). The facial nerve can be innervated by one of two motor systems. The subcortical motor system, also known as the extrapyramidal circuit, supports non-voluntary, facial expression. The cortical motor system, also known as the pyramidal circuit, supports learned, voluntarily facial expression, which may vary across cultures and be produced and inhibited intentionally.
Early research on the smile revealed that the frequency, intensity, and duration of the zygomaticus major muscle activity positively predicted self-reported happiness of the smiler (Ekman et al. Reference Ekman, Friesen and Ancoli1980; Cacioppo et al. Reference Cacioppo, Petty, Losch and Kim1986). Zygomaticus major contraction, however, is observed not only when positive emotions are experienced, but may also be observed when individuals report feeling negative emotions such as disgust (Ekman et al. Reference Ekman, Friesen and Ancoli1980), disappointment (Kraut & Johnston Reference Kraut and Johnston1979), sadness and uncertainty (Klineberg Reference Klineberg1940), and general discomfort (see Ekman et al. Reference Ekman, Davidson and Friesen1990, for a review).
2.1. True and false smiles
Due to the failure to observe a clear correspondence between activation of the zygomaticus major and positive feelings, several theorists have suggested that the smile should not be treated as a single category of facial expression (e.g., Ekman & Friesen Reference Ekman and Friesen1982). In the proposed distinctions, “true” or “sincere” smiles were defined as involuntary displays of positive affect, whereas ”false” or “insincere” smiles were defined as smiles voluntarily used to communicate that a positive emotion was felt when it was not (and when, in fact, it served to mask negative feelings).
Several morphological and dynamic markers have been proposed to distinguish these two types of smiles. The most frequently cited morphological indicator of a true smile is the Duchenne marker, thanks to Duchenne's empirical work on smiles and other facial expressions (Duchenne Reference Duchenne de Boulogne1862). The Duchenne marker involves the contraction of the muscle around the eye, the orbicularis oculi, pars lateralis. The orbicularis oculi causes a lifting of the cheeks, a narrowing of the eye opening, and wrinkles around the eyes. The combination of zygomaticus major contraction, along with orbicularis oculi contraction, is sometimes indicative of positive emotion (Frank et al. Reference Frank, Ekman and Friesen1993; Soussignan Reference Soussignan2002). The perceiver of the “Duchenne” smile also interprets it as expressive of positive emotion (Miles & Johnston Reference Miles and Johnston2007) and may respond to it with positive affect (Surakka & Hietanen Reference Surakka and Hietanen1998). Smiles lacking the marker have been referred to as “false,” “masking,” and “non-Duchenne” smiles (see Figure 1; École Nationale Supérieure des Beaux-Arts 1999). Other research shows that dynamic features of smiles, such as their symmetry, smoothness, duration, and synchrony, may distinguish true and false smiles as well (Cacioppo et al. Reference Cacioppo, Petty, Losch and Kim1986; Ekman et al. Reference Ekman, Friesen and Ancoli1980, reviewed in Frank Reference Frank and Abel2002; Hess & Kleck Reference Hess and Kleck1990).
The top two photographs show the Duchenne (left) and non-Duchenne (right) smiles as elicited by Guillaume-Benjamin Duchenne de Boulogne himself, using electrical impulses to manipulate relevant facial muscles. The bottom two photographs show more recent posed versions of the same.
Recent research, however, has shown that the utility of the Duchenne versus non-Duchenne distinction is limited (e.g., see Abe et al. Reference Abe, Beetham, Izard and Abel2002, for a review). For instance, some studies have demonstrated that dynamic characteristics of the smile can override the Duchenne marker's importance in determining judgments of how true (or sincere) the smile is (Hess & Kleck Reference Hess and Kleck1990; Krumhuber et al. Reference Krumhuber, Manstead and Kappas2007). Others have shown that non-Duchenne smiles may be associated with self-reported happiness in adults (e.g., Hecht & LaFrance Reference Hecht and LaFrance1998; Hess et al. Reference Hess, Banse and Kappas1995; Jakobs et al. Reference Jakobs, Manstead and Fischer1999), and, conversely, that Duchenne smiles may be displayed in situations in which false smiles would be predicted (Bourgeois & Hess Reference Bourgeois and Hess2008). Finally, there is some evidence that the importance of the Duchenne marker varies with culture (Thibault et al. Reference Thibault, Levesque, Gosselin and Hess2008).
2.2. Functional smiles
The distinction between true and false smiles itself may be largely superceded by another more useful distinction, namely, distinctions based on smile function (e.g., Barrett Reference Barrett and Abel2002). We describe three types of smiles that we believe have important and discrete functions, and which may map onto identifiable brain systems that represent different meanings.
As we have already implied, and consistent with experience, many smiles are simply readouts of positive internal states such as happiness (Buck Reference Buck1984). The “play-face” in primates, such as chimpanzees, held to be a homologue of laughter in humans, corresponds in musculature to the human smile (e.g., Parr & Waller Reference Parr and Waller2006; see our Figure 2, left panel). Humans and some primates smile spontaneously during experiences of pleasure, including visual, auditory, gustatory, and tactile stimulation (Ekman & Friesen Reference Ekman and Friesen1982). Smiles that are readouts of happy feelings reinforce the behaviors that elicited them in the first place. Thus, the communication of positive emotion through the smile is essential, among other things, for learning in infants, when mothers smile at babies to encourage desired behaviors (Klinnert et al. Reference Klinnert, Campos, Sorce, Emde, Svejda, Plutchik and Kellerman1983). We will refer to smiles that express happiness as enjoyment smiles.
Two chimpanzee facial expressions related to the human smile. The left panel shows a play face believed to be a homologue of laughter and sharing morphological features with the human enjoyment smile (Parr & Waller Reference Parr and Waller2006). The right panel shows a silent bared-teeth display, used in affiliative and appeasement contexts, believed to be homologous with the human affiliative smile and sharing similar musculature (Parr & Waller Reference Parr and Waller2006). Photos courtesy of Dr. Lisa Parr, National Primate Research Center, Emory University, Atlanta, GA. Used with permission.
Second, smiles can be readouts of positive social intentions that are essential for the creation and maintenance of social bonds, without necessarily being about personal enjoyment (Cashdan Reference Cashdan2004; Fridlund Reference Fridlund1991; Reference Fridlund and Abel2002). Such smiles may include the “greeting” smile (Eibl-Eibesfeld Reference Eibl-Eibesfeldt1972), as well as those of appeasement, and perhaps the smile component of embarrassment (Keltner Reference Keltner1995; Hess et al. Reference Hess, Beaupre, Cheung and Abel2002). In most primates, the silent bared-teeth display (right panel of Figure 2) serves to communicate that the smiler intends no harm and that there is no threat of aggression (van Hooff Reference van Hooff and von Cranach1976; Waller & Dunbar Reference Waller and Dunbar2005). In primates with relatively egalitarian social systems, including some macaques, mandrills, Gelada baboons, and chimpanzees, the silent bare-teeth display is also seen in grooming, sexual solicitation, and reconciliations (Preuschoft & van Hooff Reference Preuschoft, van Hooff, Segerstrale and Molnàr1997; Waller & Dunbar Reference Waller and Dunbar2005). Smiles that express positive social motives will be called affiliative smiles.
Finally, what we will call dominance smiles are expressions that reflect social status or control, and may include displays that have been called “scheming smiles” (e.g., Öhman et al. Reference Öhman, Lundqvist and Esteves2001; Tipples et al. Reference Tipples, Atkinson and Young2002), “critical smiles,” and perhaps the smile component of the expression of pride (Tracy & Robins Reference Tracy and Robins2004; Reference Tracy and Robins2008). Darwin also referred to derisive or sardonic smiles in his discussion of sneering and defiance (Darwin Reference Darwin1872, p. 251). Recent analyses of the signals of leadership and dominance in human societies refer to this type of smile. Senior et al. (Reference Senior, Phillips, Barnes and David1999) note that (former Prime Minister of Britain) Tony Blair and (former American President) Bill Clinton have been called “skilled proponents of the dominant smile” (p. 344).
Tony Blair, among other world leaders, has been said to be a “skilled proponent of the dominant smile” (Senior et al. Reference Senior, Phillips, Barnes and David1999), a fact that has not been ignored by caricature artists. Left: Photo © Crown Copyright. Right: Caricature by Paul Baker, courtesy of Paul Baker.
Whether a smile is voluntary on the part of the smiler is probably not a factor that best distinguishes enjoyment, affiliative, and dominance smiles. That is, the involvement of the Duchenne marker will probably not turn out to be diagnostic of one type of smile. Specific instances of the proposed functional types may however be associated with specific postural or other facial features (e.g., the greeting smiles contains an “eyebrow flash,” and the embarrassed smile is part of a larger gesture). Thus, it might in principle be possible to construct a visual description of the full bodily and facial characteristics of enjoyment, affiliative, and dominance smiles. But the present issue is: How do perceivers arrive at those meanings? We propose that perceivers use a set of neural and behavioral processes to extract a smile's meaning that allows them to distinguish between the three functional smile categories in terms of the feelings they generate in the perceiver.
3. What is simulated in the simulation of a smile?
In this section, we explore the neural structures that could play a central role in the embodied simulations that represent different smile meanings. Specifically, we address how the basal ganglia, prefrontal cortex, amygdala, motor system, and somatosensory system all contribute to the experienced meaning of a perceived smile on another person's face. Later sections will discuss when and how these simulations are triggered, and the implications for emotional experience. Although the focus is on specific neural structures, this does not mean that we endorse a localization approach. It will remain for neuroscientists to fully establish the larger distributed circuits that support the functions we suggest that these structures play.
3.1. Subcortical and cortical affective systems
3.1.1. Basal ganglia
As people perceive smiles, the reward system in their brain may simulate the experience of reward. Research shows that the smile can function as a social reward for both adult and infant humans (Trevarthen Reference Trevarthen1974; Tronick et al. Reference Tronick, Als, Adamson, Wise and Brazelton1978).
This view is consistent with evolutionary and ecological treatments of the smile (e.g., Buck Reference Buck1991; Burgoon et al. Reference Burgoon, Buller and Woodall1996; Haith Reference Haith1972; McArthur & Baron Reference McArthur and Baron1983). Research on both primates and rats implicates the striatum and the ventral tegmental area (VTA) in reward processing (Kawagoe et al. Reference Kawagoe, Takikawa and Hikosada1998; Parkinson et al. Reference Parkinson, Cardinal and Everitt2000; Schultz et al. Reference Schultz, Gauthier, Klin, Fulbright, Anderson and Volkmar2000). Further studies indicate that these regions are similarly related to reward processing in humans (Damasio et al. Reference Damasio, Grabowski, Bechara, Damasio, Ponto, Parvizi and Hichwa2000; Davidson & Irwin Reference Davidson and Irwin1999; Lane et al. Reference Lane, Reiman, Ahern, Schwartz and Davidson1997; Reference Lane, Chua and Dolan1999; Mobbs et al. Reference Mobbs, Greicius, Abdel-Azim, Menon and Reiss2003; O'Doherty et al. Reference O'Doherty, Critchley, Deichmann and Dolan2003; Rauch et al. Reference Rauch, Shin, Dougherty, Alpert, Orr, Lasko, Macklin, Fischman and Pitman1999; Redoute et al. Reference Redoute, Stoleru, Gregoire, Costes, Cinotti, Lavenne, Le Bars, Forest and Pujol2000). Recent research has also linked the reward areas of the basal ganglia with the perception of smiling faces (Chakrabarti et al. Reference Chakrabarti, Kent, Suckling, Bullmore and Baron-Cohen2006; Lawrence et al. Reference Lawrence, Chakrabarti and Calder2004; Lee et al. Reference Lee, Josephs, Dolan and Critchley2006; Morris et al. Reference Morris, Frith, Perrett, Rowland, Young, Calder and Dolan1996; Reference Morris, Friston, Buchel, Frith, Young, Calder and Dolan1998; Okun et al. Reference Okun, Bowers, Springer, Shapira, Malone, Rezai, Nuttin, Heilman, Morecraft, Rasmussen, Greenberg, Foote and Goodman2004; Phillips et al. Reference Phillips, Bullmore, Howard, Woodruff, Wright, Williams, Simmons, Andrew, Brammer and David1998; Whalen et al. Reference Whalen, Rauch, Etcoff, McInerney, Lee and Jenike1998). When people perceive smiles, they experience them as rewarding.
3.1.2. Prefrontal cortex
Simulations of reward as people perceive smiles may also reflect contributions from prefrontal cortex. The reward circuitry just described in the basal ganglia is associated with what Davidson (Reference Davidson, Lewis and Haviland1993) has termed pre-goal attainment positive affect. In his view, activation in subcortical reward centers supports an organism's approach toward an appetitive goal. Davidson contrasts the approach state with post-goal attainment positive affect, which represents a functionally different type of positive emotion, characterized by feelings of affiliation and attachment (biologists refer to the former as incentive value and the latter as reward outcome).
Post-goal attainment positive affect has been linked in a number of studies with activation in the orbitofrontal cortex (OFC) (e.g., Anderson et al. Reference Anderson, Qin, Sohn, Stenger and Carter2003; Elliott et al. Reference Elliott, Friston and Dolan2000; O'Doherty et al. Reference O'Doherty, Kringelbach, Rolls, Hornak and Andrews2001; Rolls Reference Rolls2000). Findings by Davidson and colleagues, for example, demonstrate that smiles displayed by offspring are related to OFC activation in mothers who perceive these smiles (Nitschke et al. Reference Nitschke, Nelson, Rusch, Fox, Oakes and Davidson2004). Specifically, the OFC differentiates the sight of one's own smiling baby from the sight of an unknown smiling baby (who is nevertheless cute and positive in impact). This finding has recently been replicated and extended (e.g., Minagawa-Kawai et al. Reference Minagawa-Kawai, Matsuoka, Dan, Naoi, Nakamura and Kojima2009). The reward outcome system in the brain probably extends to medial orbital and medial prefrontal (dorsal to orbital) cortices, as well as to anterior cingulate cortex (O'Doherty et al. Reference O'Doherty, Critchley, Deichmann and Dolan2003). This system may also be involved in reward learning. In addition, and very importantly, the system is implicated in processing status relations (Zink et al. Reference Zink, Tong, Chen, Bassett, Stein and Meyer-Lindenberg2008), and thus may contribute to distinguishing dominant versus submissive smile meanings.
Although the specific terms, pre-goal and post-goal affect, are not necessary for our theorizing, the distinction is relevant for differentiating the perception of enjoyment and affiliative smiles. Because the OFC and contiguous areas process learned emotional responses (Rolls Reference Rolls2004), OFC activation may distinguish the basic rewarding properties of smiles from the reward of experiencing smiles made by people with whom an individual has a significant, previously established emotional, relationship (e.g., family members, in-group members). Indeed, other evidence and reasoning suggests that reward systems in prefrontal cortex play a distinct role in responding to affiliative smiles, reflecting their association with attachment information (e.g., Schore Reference Schore2001).
3.1.3. Amygdala
As people perceive smiles, the amygdala may produce states that further contribute to simulations that underlie how these smiles are interpreted. The amygdala's involvement in perceiving fear expressions has been considered at length (e.g., Adolphs Reference Adolphs2008; Atkinson Reference Atkinson2007; Heberlein & Atkinson Reference Heberlein and Atkinson2009; LeDoux Reference LeDoux2007). Lesion and neuroimaging studies initially indicated that the amygdala was not only vital for recognizing the emotional expression of fear, but that it further supported the full experience of fear and its behavioral implications (e.g., Calder et al. Reference Calder, Lawrence and Young2001). These studies did not demonstrate any role for the amygdala in smile processing (e.g., Whalen et al. Reference Whalen, Rauch, Etcoff, McInerney, Lee and Jenike1998), and patients with amygdala lesions had not been reported as showing deficits in recognizing happy faces (Adolphs et al. Reference Adolphs, Tranel, Damasio and Damasio1994; Hamann et al. Reference Hamann, Stefanacci, Squire, Adolphs, Tranel, Damasio and Damasio1996; Calder et al. Reference Calder, Young, Rowland, Perrett, Hodges and Etcoff1996).
Yet, when measuring amygdala activity in human volunteers during rapid visual presentations of fearful, smiling, and neutral faces, Breiter et al. (Reference Breiter, Etcoff, Whalen, Kennedy, Rauch, Buckner, Strauss, Hyman and Rosen1996) found that the amygdala also responded preferentially to smiles versus neutral faces (see also a meta-analysis by Fusar-Poli et al. Reference Fusar-Poli, Placentin, Carletti, Landi, Alle, Surguladze, Benedetti, Abbamont, Gasparott, Baral, Perez, McGuire and Politi2009). Other studies supported the conclusion that all emotional expressions can activate the amygdala. For example, a functional magnetic resonance imaging (fMRI) study by Winston et al. (Reference Winston, O'Doherty and Dolan2003) used emotion perception tasks that were either implicit (ratings of maleness) or explicit (deciding whether a face was more or less emotional). Both tasks activated broad cortical and subcortical regions for disgusted, fearful, happy, and sad expressions of either low or high emotion intensity. Most importantly, all expressions activated the amygdala, a finding further supported by Fitzgerald et al. (Reference Fitzgerald, Angstadt, Jelsone, Nathan and Phan2006). Additional results demonstrate higher activation of the amygdala for high versus low intensity emotions for explicit and implicit tasks (cf. Surguladze et al. Reference Surguladze, Brammer, Young, Andrew, Travis, Williams and Phillips2003).
Taken together, these results suggest that the amygdala responds to stimuli of motivational significance independently of emotion and processing goals. Indeed, more recent evidence favors an even broader account of amygdala function, namely, that it responds to everything of uncertain meaning to the organism (e.g., Murphy et al. Reference Murphy, Michael, Robbins and Sahakian2003; Sander et al. Reference Sander, Grafman and Zalla2003; Whalen et al. Reference Whalen, Shin, McInerney, Fischer, Wright and Rauch2001). Applying this hypothesis to the smile, Yang and colleagues proposed that individuals may be uncertain about the meaning of a smile, not merely because of its perceptual features, but also because of its meaning as reflected in the current social or experimental context (Yang et al. Reference Yang, Menon, Eliez, Blasey, White, Reid, Gotlib and Reiss2002). These authors further note that when the meaning of a smile is unclear (such as a smile displayed by an enemy), additional information is needed, similar to our proposal here (see also Lee et al. Reference Lee, Josephs, Dolan and Critchley2006).
3.2. Facial mimicry in the motor system and shared neural substrates
As people perceive smiles, the motor system may simulate the experience of performing the perceived action, further contributing to how the perceived smile is understood.
3.2.1. Facial mimicry
Facial mimicry is defined as the visible or non-visible use of facial musculature by an observer to match the facial gestures in another person's facial expression. Perceivers of smiles often automatically mimic these smiles. For instance, electromyographic (EMG) recordings reveal that when individuals view a smile, their zygomaticus major muscle contracts, usually within 500 milliseconds after the onset of the stimulus (Dimberg & Thunberg Reference Dimberg and Thunberg1998). Mojzisch et al. (Reference Mojzisch, Schilbach, Helmert, Pannasch, Velichkovsky and Vogeley2006) similarly demonstrated that observers automatically mimic smiles expressed by virtual characters in dynamic animations, as did Hess and Bourgeois (in press) in an interactive live setting (see also a review in Hess et al. Reference Hess, Blairy, Philippot, Philippot, Feldman and Coats1999). Automatically mimicking a smile interferes significantly with simultaneously production of an incongruent facial expression, such as anger (Lee et al. Reference Lee, Dolan and Critchley2007).
3.2.2. Link to corresponding emotions
As anticipated by facial feedback theory, facial mimicry may be accompanied by self-reports of a corresponding emotion, sometimes called emotional contagion (Hatfield et al. Reference Hatfield, Cacioppo, Rapson and Clark1992; Reference Hatfield, Cacioppo and Rapson1993; Laird et al. Reference Laird, Alibozak, Davainis, Deignan, Fontanella, Hong, Levy and Pacheco1994; Strayer Reference Strayer1993; Wild et al. Reference Wild, Erb and Bartels2001; Soussignan Reference Soussignan2002). For afferent feedback to contribute to an embodied simulation of a perceived smile, however, the perceiver does not necessarily have to experience a conscious change in emotional state–such simulations often appear to have unconscious impact. Findings that mimicry produces emotional effects implicitly are therefore also important.
In an innovative study that assessed the causal relationship between facial mimicry and implicit emotion, Botulinum Toxin (BOTOX) was used to block facial mimicry for expressions of anger (Hennenlotter et al. Reference Hennenlotter, Dresel, Castrop, Ceballos-Baumann, Wohlschläger and Haslinger2009). Participants were directed to mimic angry and sad facial expressions in still photographs. During the anger (though not sadness) mimicry task, participants whose brows had received BOTOX injections exhibited significantly less activation in the limbic system compared to control individuals who received no injection. The result for anger causally links facial mimicry to emotion, given that disabling the facial musculature decreased emotion activation. When the muscle pattern associated with anger is blocked, part of the embodied meaning associated with anger is lost, such that the emotion is experienced less intensely. Consistent with this experimental finding, and specifically relevant to smiles, is a correlational finding by Lee et al. (Reference Lee, Josephs, Dolan and Critchley2006). In that study, participants mimicked faces expressing smiles, as well as other nonemotional facial movements and expressions. The more participants mimicked the observed smiles, the greater the activations in their striatum and amygdala (see also Schilbach et al. Reference Schilbach, Wohlschlaeger, Kraemer, Newen, Shah and Fink2006).
A recent paper by Schilbach et al. (Reference Schilbach, Eickhoff, Mojzisch and Vogeley2008), who collected fMRI and EMG data simultaneously, is also noteworthy here. Their results showed that spontaneous mimicry occurred during the perception of smiles, which was accompanied by neural activity in the motor system, specifically in the inferior left precentral gyrus. Consistent with previous findings, this study also implicated the medial temporal lobe in the spontaneous mimicry of observed emotional expressions. Finally, the dorsal midbrain was also active, which can be interpreted as signaling arousal induced by direct eye gaze.
In sum, there is evidence that mimicry has a role in causing emotion. Our interest here, however, is not with changes in consciously reported emotional state. Instead, we focus on the fundamental role of mimicry in creating the embodied feeling of a smile, which becomes part of its meaning. In a study motivated by a similar idea, Zajonc et al. (Reference Zajonc, Adelmann, Murphy and Niedenthal1987) showed that the appearance of spouses who had lived together for at least 25 years and had strong marriages grew more similar in facial appearance over time. Their interpretation was that mimicry supports empathy through afferent feedback, with the incidental effect of producing more similar facial musculature (not through more similar emotion). This preliminary interpretation requires further empirical tests (see Bastiaansen et al. Reference Bastiaansen, Thiox and Keyers2009 for further discussion and insight).
3.2.3. Shared neural substrates
Facial mimicry receives considerable attention in the literature on so-called mirror neurons and the notion of emotional resonance. Mirror neurons were first observed in the brains of monkeys in response to limb actions (Gallese et al. Reference Gallese, Fadiga, Fogassi and Rizzolatti1996; Rizzolatti & Craighero Reference Rizzolatti and Craighero2004; Rizzolatti et al. Reference Rizzolatti, Fadiga, Gallese and Fogassi1996). Recordings of mirror neurons in the monkey motor cortex, particularly area F5, associated with the production of hand and mouth movements, were interpreted as constituting a mechanism that could support the implicit comprehension of an action, specifically, the goal of an action (Gallese et al. Reference Gallese, Fadiga, Fogassi and Rizzolatti1996; Rizzolatti et al. Reference Rizzolatti, Fadiga, Gallese and Fogassi1996). More recently, a similar mirror system for perceiving and performing action has been described in humans (Fadiga et al. Reference Fadiga, Fogassi, Pavesi and Rizzolatti1995; Gallese et al. Reference Gallese, Fadiga, Fogassi and Rizzolatti1996; Iacoboni et al. Reference Iacoboni, Woods, Brass, Bekkering, Mazziotta and Rizzolatti1999; Rizzolatti et al. Reference Rizzolatti, Fadiga, Gallese and Fogassi1996). The work on humans points to the premotor and parietal areas as the most likely human homologue of monkey area F5 (e.g., Decety & Grezes Reference Decety and Grèzes1999).
Inspired by this approach, Carr et al. (Reference Carr, Iacoboni, Dubeau, Mazziotta and Lenzi2003) found that the passive perception and the intentional mimicry of emotional facial expression activate overlapping sets of brain structures, including the ventral premotor cortex, the insula, and the amygdala (see also Wicker et al. Reference Wicker, Keysers, Plailly, Royet, Gallese and Rizzolatti2003). Mimicry relative to perception, however, was accompanied by greater activation in regions related to emotion processing, in particular, the amygdala and anterior insula, and also the primary motor cortex. Similarly, Hennenlotter et al. (Reference Hennenlotter, Schroeder, Erhard, Catrop, Haslinger, Stoecker, Lange and Ceballos-Baumann2005) found that voluntary production and perception of smiles activated both the right premotor cortex and the inferior frontal cortex, as well as the right parietal operculum and the left anterior insula. These findings were replicated in a careful study by van der Gaag et al. (Reference van der Gaag, Minderaa and Keysers2007). Although the reviewed findings are compelling, and do suggest that emotion processing is affected by mimicry, we note that the mirror neuron construct and its roles in human emotion recognition have received considerable criticism (e.g., Turella et al. Reference Turella, Pierno, Tubaldi and Castiello2009; Dinstein et al. Reference Dinstein, Gardner, Jazayeri and Heeger2008; Jacob Reference Jacob2008; Reference Jacob2009). A clearer understanding of this construct awaits further research.
The robust activation of premotor areas during the observation of facial expressions (unaccompanied by mimicry) also requires discussion. This finding has been interpreted as demonstrating the presence of an as-if loop, meaning that perception activates the programs for producing facial expressions (e.g., Leslie et al. Reference Leslie, Johnson-Frey and Grafton2004). The processing of as-if loops has also been called off-line simulation (e.g., Atkinson Reference Atkinson2007). One interpretation of these findings is that as-if simulations, when they occur, contribute to smile meaning. In support of this view, research has found that perception-plus-mimicry tends to produce stronger affective responses to smiles than perception alone (e.g., Carr et al. Reference Carr, Iacoboni, Dubeau, Mazziotta and Lenzi2003). Motor mimicry seems to play an important role in how smiles and their meanings are simulated.
3.2.4. Roles of mimicry in processing perceived facial expression
If facial mimicry is important in constructing embodied simulations for smiles, then mimicry should affect performance on tasks that measure recognition and access to meaning (Adolphs Reference Adolphs2002; Reference Adolphs2003; Heberlein & Atkinson Reference Heberlein and Atkinson2009; McIntosh Reference McIntosh2006). In their review of facial mimicry, however, Hess and colleagues (Hess et al. Reference Hess, Blairy, Philippot, Philippot, Feldman and Coats1999) did not find evidence that mimicry was causally related to the simple recognition of emotional facial expressions, either directly or as mediated by changes in self-reported emotional state (e.g., Blairy et al. Reference Blairy, Herrera and Hess1999). Hess and Blairy (Reference Hess and Blairy2001) considered the possibility that failure to support a causal path from facial mimicry to emotion recognition may have resulted from the use of very prototypical facial expressions. When they used naturalistic dynamic stimuli to remedy this problem, they observed facial mimicry, along with a relationship between the perceived expression and self-reported emotional responses (e.g., happiness when viewing smiles). Nevertheless, they found no evidence of a link from motor mimicry to recognition accuracy, either directly or through changes in emotional responding.
Many additional findings further demonstrate that mimicry does not always play a central role in emotion recognition tasks. As noted previously, recognition tasks on prototypic expressions can be accomplished by perceptual analysis alone, without motor mimicry (e.g., Adolphs Reference Adolphs2002). Indeed, high-functioning autistic individuals, who do not spontaneously mimic others' facial expressions (e.g., McIntosh et al. Reference McIntosh, Reichmann-Decker, Winkielman and Wilbarger2006), perform as well as controls when categorizing facial expressions of emotion (e.g., Spezio et al. Reference Spezio, Adolphs, Hurley and Piven2007a). Similarly, individuals with facial paralysis perform quite normally on various recognition tasks (e.g., Calder et al. Reference Calder, Keane, Cole, Campbell and Young2000a; Reference Calder, Keane, Manes, Antoun and Young2000b; Keillor et al. Reference Keillor, Barrett, Crucian, Kortenkamp and Heilman2002). Finally, actively keeping individuals from mimicking, for example, by asking them to turn the corners of their mouth down while seeing a smile, does not hinder emotion recognition (Blairy et al. Reference Blairy, Herrera and Hess1999).
As all of these findings indicate, facial mimicry is not always required to recognize emotional expressions in simple recognition tasks. In some cases, however, mimicry does facilitate recognition. Niedenthal et al. (Reference Niedenthal, Brauer, Halberstadt and Innes-Ker2001), for example, observed effects of mimicry when participants had to detect the boundary of facial expression between happiness and sadness. In a more recent study, Stel and van Knippenberg (Reference Stel and van Knippenberg2008) found that blocking mimicry affected the speed, but not the accuracy, of categorizing facial expressions as positive or negative. Additionally, individuals showing strong automatic facial mimicry tend to have high levels of empathy (Sonnby-Borgström Reference Sonnby-Borgström2002; Zajonc et al. Reference Zajonc, Adelmann, Murphy and Niedenthal1987). These findings point to the possibility that simulation does become important in recognition tasks when they require fine distinctions in smile meaning, such as the processing of different smile types.
3.2.5. Beyond facial mimicry
As we noted earlier, postural and other non-facial gestures are also important components of the meaning of a smile. We have discussed facial mimicry so far, but representation of the full meaning of a smile will not be independent of the entire bodily representation of meaning. The sign of appeasement involves a smile, usually a non-Duchenne smile in fact, but also a number of other head and hand gestures (Keltner Reference Keltner1995). Tracy and Robins (Reference Tracy and Robins2008) have described the expression of pride, which involves a small smile but also a backward tilt of the head, resulting in a lifted chin, as well as a typical posture. We expect the embodiment of the entire expression to be useful in the interpretation of meaning.
3.3. Simulating embodied experience: Somatosensory cortices
As people perceive smiles, the somatosensory system may simulate the embodied experience of how the perceived smiles feel, further contributing to representing their meaning. One account of how facial expression could be simulated in an embodied manner is as the output of a simulator (e.g., Barsalou Reference Barsalou1999). From this perspective, somatosensory cortices may be involved in simulating the feeling of a perceived smile while processing its meaning (Adolphs Reference Adolphs2002; Gallese & Goldman Reference Gallese and Goldman1999; Keyers et al. 2004).
Right-hemisphere cortices are likely to be involved in simulating emotional expressions (Adolphs et al. Reference Adolphs, Damasio, Tranel and Damasio1996; Bowers et al. Reference Bowers, Bauer, Coslett and Heilman1985). For example, patients with lesions in the right somatosensory cortex are poorer at recognizing facial expressions than individuals without such lesions. Specifically, Adolphs et al. (Reference Adolphs, Damasio, Tranel, Cooper and Damasio2000) assessed 108 subjects with focal brain lesions and found that the right somatosensory cortex was central for recognizing the facial expressions associated with the six basic emotions. Consequently, these researchers concluded that the right somatosensory cortex generates a representation or an “image” of the felt state, which feeds into the recognition system as a diagnostic cue (e.g., Adolphs Reference Adolphs2002; Atkinson Reference Atkinson2007).
Following Adolphs et al. (Reference Adolphs, Damasio, Tranel, Cooper and Damasio2000), Pourtois and colleagues used transcranial magnetic stimulation (TMS) to selectively interfere with right somatosensory cortex function while participants performed a same/different facial expression-matching task (see Pourtois et al. Reference Pourtois, Sander, Andres, Grandjean, Reveret, Olivier and Vuilleumier2004). This selective interference disrupted task performance. Thus, both lesion and TMS studies implicate somatosensory simulations in the recognition of perceived facial expressions.
Although these previous studies focused on the simple recognition of facial expressions, it is likely that somatosensory simulations also support more subtle interpretation of facial expressions. Not only do somatosensory simulations facilitate recognition, they probably contribute to how perceivers experience the meaning of these facial expressions, specifically, as a felt emotion. An important issue for future research is to assess whether somatosensory simulations indeed play this additional role.
3.4. What do smiles feel like for the perceiver?
In the previous sections we suggest that individuals know whether a smile means that the smiler is expressing enjoyment, affiliation, or dominance, because those smiles feel differently in terms of reward, action, and somatosensory experience. Next, we explore how these different experiences help distinguish enjoyment, affiliative, and dominance smiles. From our perspective, we propose that an enjoyment smile involves a basic rewarding feeling of positive affect and that an affiliative smile involves a positive feeling of attachment and intimacy (where the positive feelings of enjoyment vs. attachment/intimacy are distinct). Our analysis of dominance smiles relies on several further assumptions. In hierarchical primate societies such as ours, highly dominant alpha individuals pose a certain threat insofar as they can claim territory or possessions (e.g., food) from lower status group members (Menzel Reference Menzel and Menzel1973; Reference Menzel, Schrier and Stollnitz1974). Hence, the perceived presence of a dominant other should lead to increased vigilance and preparedness for withdrawal (Coussi-Korbel Reference Coussi-Korbel1994). Although speculative, we suggest that dominance smiles, like smiles hiding negative intentions or feelings, are associated with the experience of negative rather than positive affect, as indicated by right-lateralized activation (Davidson et al. Reference Davidson, Ekman, Saron, Senulis and Friesen1990; Boksem et al. Reference Boksem, Smolders and de Cremer2009). Thus, the meaning of dominance smiles should not involve the forms of positive emotion associated with the other two smiles.
In summary, we have documented possible neural systems that could contribute to the embodied simulations that occur while perceiving emotional facial expressions, thereby contributing to their interpreted meaning. A full account of embodied simulation, however, requires the construct of a trigger that initiates embodied simulation as facial expressions are perceived. Social and developmental considerations suggest that eye contact with the expresser of an emotion launches embodied simulations automatically.
4. Triggering embodied simulation
For eyes can speak and eyes can understand.
— ChapmanIn the previous sections, we saw that embodied simulation is not always implicated in the recognition of emotional facial expressions, indicating that simulation is not always required (for analogous findings and accounts, see Barsalou et al. Reference Barsalou, Santos, Simmons, Wilson, De Vega, Glenberg and Graesser2008; Hess & Blairy Reference Hess and Blairy2001; Kosslyn Reference Kosslyn1976; Niedenthal et al. Reference Niedenthal, Winkielman, Mondillon and Vermeulen2009; Solomon & Barsalou Reference Solomon and Barsalou2004; for related situated-simulation assumptions, see Semin & Cacioppo Reference Semin, Cacioppo, Semin and Smith2008; Smith & Semin Reference Smith and Semin2007; Strack & Deutsch Reference Strack and Deutsch2004). In this section, we review recent research suggesting that eye contact modulates the presence versus absence of embodied simulation as people perceive smiles.
4.1. Definitions related to eye gaze
Eye gaze is the direction of one's gaze at another's eyes, presumably during the search for information useful to attributing the cause of the other's behavior (von Cranach Reference von Cranach and Esser1971). Mutual gaze refers to two people gazing at each other's faces. Eye contact involves two people gazing at each other's eyes. All such behavior has become of great interest in scientific research lately, usually under the theoretical rubric of “social relevance” (e.g., Adams et al. Reference Adams, Gordon, Baird, Ambady and Kleck2003; Hess et al. Reference Hess, Adams and Kleck2007; Klucharev & Sams Reference Klucharev and Sams2004; Mojzisch et al. Reference Mojzisch, Schilbach, Helmert, Pannasch, Velichkovsky and Vogeley2006; Richeson et al. Reference Richeson, Todd, Trawalter and Baird2008). Our primary interest here is in eye contact. Specifically, we argue that eye contact counts as a sufficient, but not a necessary, trigger of embodied simulation as observers perceive smiles.
4.2. Gaze and simulation of meaning
4.2.1. Eye contact and intimacy
Several sources of evidence suggest that eye contact triggers embodied simulation. In part, the roots of this prediction lie in research on intimacy. Argyle (Reference Argyle1972) suggested that increased eye contact during social interaction indicates an increase in intimacy, which is also consistent with Patterson's (Reference Patterson1982; Reference Patterson1983) functional analysis. Additional findings support these proposals: Increased eye contact is associated with increased maternal sensitivity (Lohaus et al. Reference Lohaus, Keller and Voelker2001). Individuals make eye contact with people with whom they have close relationships more frequently and for longer durations than with strangers (Russo Reference Russo1975). Men show more approach behavior after repeated eye contact (Walsh & Hewitt Reference Walsh and Hewitt1985). Dating couples of both sexes tend to look in the eyes of their partners more often than unacquainted couples (Iizuka Reference Iizuka1992).
Importantly, the pupils themselves impart no actual descriptive information (beyond dilation and constriction, which inform arousal). Thus, these findings suggest that eye contact triggers something beyond itself that increases intimacy. We suggest that eye contact triggers an embodied simulation of the perceived facial expression and its correspondent feeling for use in interpretation. A classic study by Bavelas et al. (Reference Bavelas, Black, Lemery and Mullett1986) supports this proposal. In their study, a confederate faked experiencing an injury and expressed apparent pain facially. When participants viewed the pained facial expression, they inadvertently mimicked it. The critical manipulation in the experiment was whether the victim of the painful stimulus made eye contact with participants. Analyses revealed that eye contact significantly affected the pattern and timing of participants' mimicry. Specifically, participants mimicked the perceived expressions of pain most clearly when eye contact was made.
Consistent with this initial finding, Schrammel and colleagues (Reference Schrammel, Pannasch, Graupner, Mojzisch and Velichkovsky2009) demonstrated that, when viewing emotional expressions, participants' zygomaticus major (i.e., smile) activity was higher for happy than for angry or neutral faces, and, most importantly, that this effect was stronger under conditions of eye contact. Furthermore, angry faces elicited more negative affect following eye contact, and happy faces elicited more positive affect, relative to the no-eye contact condition. For both emotions, eye contact modulated the intensity of the experienced emotion. Although both studies just described found that participants in the no-eye contact conditions recognized smiles accurately, the eye contact effects suggest that these participants may not have produced embodied simulations.
It is important to note, however, that drawing a strong conclusion about eye contact's role as a trigger for the mimicry component of embodied simulation, is difficult. First, as we have seen, some smiles are rewarding stimuli (e.g., the enjoyment smiles used in most research). Therefore, a perceiver of a smile could smile, even without making eye contact with the smiler, simply as a resulted of the positive emotion associated with the perceived smile (Mojzisch et al. Reference Mojzisch, Schilbach, Helmert, Pannasch, Velichkovsky and Vogeley2006). In this case, the elicited smile would not count as mimicry, but rather as the readout of the perceiver's positive emotional response to the smile. This interpretation implies that much past research on facial mimicry is not definitive because it does not distinguish mimicking a perceived smile from responding with positive emotion to the sight of a smile. Nevertheless, because a smile can have functionally different meanings, it is an ideal expression for distinguishing mimicry versus emotional responding. Specifically, researchers could determine when perceivers make correct versus incorrect interpretations of different smile types as a function of simply responding with enjoyment to a smile or whether they mimic it. As described next for the SIMS model, we present conditions under which smiles simply produce reward responses versus embodied simulations of the perceived smile and its associated emotion. If the model is correct, then embodied simulation should underlie the correct interpretations of smiles that have specific meanings.
4.2.2. Developmental considerations
The developmental literature offers further support for a relationship between eye contact and embodied simulation of emotional states. In general, eye contact is very important for infants. For instance, Farroni et al. (Reference Farroni, Csibra, Simion and Johnson2002) demonstrated that three-day-old infants looked longer at faces with direct gaze, as opposed to simultaneously presented faces with averted gaze. Thus, infants prefer faces that establish eye contact. Furthermore, eye contact is associated with stronger neural processing, as demonstrated by analyses of the infant N170 (Farroni et al. Reference Farroni, Csibra, Simion and Johnson2002; Farroni et al. Reference Farroni, Massaccesi, Pividori and Johnson2004). Infants also show enhanced neural processing of angry expressions when these expressions are accompanied by direct eye gaze (Striano et al. Reference Striano, Kopp, Grossmann and Reid2006). Eye contact has thus been called the “main mode of establishing a communicative context between humans” (Farroni et al. Reference Farroni, Csibra, Simion and Johnson2002, p. 9602).
Infants' interest in eyes has been interpreted as an interest in the perceptual features of the eyes per se, and as an interest in direct gaze. Because infants benefit when adults understand their internal states and needs, however, and because they have limited means to express those needs, their interest in eye contact could instead reflect an evolutionary-based mechanism for triggering embodied simulation in caretakers. Such an interpretation receives additional support from research showing that infants use behaviors such as smiling, gurgling, and flirting to achieve eye contact with adults who are not looking at them (Blass & Camp Reference Blass and Camp2001). Infants also engage longer and smile more often at individuals who make eye contact (Hains & Muir Reference Hains and Muir1996; Symons et al. Reference Symons, Hains and Muir1998).
4.2.3. Role of the amygdala
An increasingly clear connection between eye contact and amygdala activation further supports this account (Dalton et al. Reference Dalton, Nacewicz, Johnstone, Schaefer, Gernsbacher, Goldsmith, Alexander and Davidson2005; Pourtois & Vuilleumier Reference Pourtois and Vuilleumier2006). Complete amygdala lesions result in a severe reduction in eye contact during actual conversations (Spezio et al. Reference Spezio, Huang, Castelli and Adolphs2007b). Conversely, as neural activity in the amygdala increases, monitoring for emotional gaze events in others increases (Hooker et al. Reference Hooker, Paller, Gitelman, Parrish, Mesulam and Reber2003).
A recent study that recorded electroencephalogram (EEG) and EMG while participants viewed smiles and sad expressions showed a similar effect (Achaibou et al. Reference Achaibou, Pourtois, Schwartz and Vuilleumier2008). Peak EEG activation around 100 msec (P1) was associated with greater correspondent EMG activity slightly later, at around 300 msec. Achaibou et al. interpreted the increased P1 amplitude as a signature of the participant's attentional state, suggesting that deeper visual processing of the facial expression was associated with enhanced mimicry, as indexed by EMG. Although amygdala activation was not measured, all the findings reviewed here on functions of the amygdala suggest that, when the amygdala becomes active, it directs attention to the eyes, and that resulting eye contact elicits greater or more correspondent facial mimicry. The link between amygdala activation and mimicry further follows from involvement of the dorsal midbrain, which has been linked to increases in arousal produced by direct gaze (Donovan & Leavitt Reference Donovan and Leavitt1980). Although we have assumed conservatively here that amydala activation increases motor mimicry in smiling, it is also possible that it increases embodied simulation as well, thereby producing stronger emotional responses.
4.3. Perceptual information from the eye region versus eye contact
A potential confound exists between looking at the eyes as a way to extract perceptual information and making eye contact as a possible trigger to simulation. Adophs and colleagues, for example, reported extensive study of patient S.M., who had a bilateral brain lesion that encompassed all nuclei of the amygdala (Adolphs et al. Reference Adolphs, Gosselin, Buchanan, Tranel, Schyns and Damasio2005). In initial assessments, S.M. showed a compromised ability to recognize fearful expressions. In subsequent assessments, this failure was attributed to not using information from the eyes. Failure to look at the eye region during smiles may increase compensatory looking at the mouth region. Because S.M. used information from the mouth normally, she was able to correctly distinguish smiles from fear expressions. More recent findings from the same laboratory found that high-functioning autistic children also avoid gazing at the eye region when performing a facial expressions recognition task (Spezio et al. Reference Spezio, Adolphs, Hurley and Piven2007a). Here, too, the autistic group performed as well as the control group in recognition accuracy (based on impoverished information provided in the Bubbles technique), perhaps reflecting the use of other diagnostic perceptual cues, such as the mouth.
The specific ability to distinguish Duchenne from non-Duchenne smiles has been shown to be compromised in autistic adults, who failed to look at the eyes while making judgments (Boraston et al. Reference Boraston, Corden, Miles, Skuse and Blakemore2008). More specifically, this result was interpreted as reflecting failure to use information in the perceived expression provided by contraction of the orbicularis oculi muscle in the vicinity of the eyes. Although these findings have typically been interpreted as showing that information from the eye region carries specific perceptual information, they can be also interpreted as demonstrating the importance of the eye region for triggering embodied simulations. As described in the SIMS, we propose that these findings actually demonstrate that eye contact has evolved as a trigger for embodied simulations, which endow smiles with their different functional meanings.
5. Conceptual knowledge about facial expression
We have already mentioned that individuals possess and use stereotyped knowledge and expectations about the meanings of smiles (Halberstadt et al. Reference Halberstadt, Winkielman, Niedenthal and Dalle2009; Kirouac & Hess Reference Kirouac and Hess1999). A challenge for the present approach is to suggest how this knowledge is represented and also how it relates to embodied simulations of perceived smiles. As will be seen in later sections, conceptual knowledge of smiles plays a central role in the SIMS model of how people interpret their meanings.
The classic view of emotion concepts relies on general models of representation in the cognitive sciences that view concepts as redescriptions of the input from modality-specific systems into an abstract representation language (e.g., Bower Reference Bower1981; Johnson-Laird & Oatley Reference Johnson-Laird and Oatley1989; Ortony et al. Reference Ortony, Clore and Foss1987). According to these accounts, higher-order mental content is represented in an abstract, language-like code that has an amodal character (e.g., Fodor Reference Fodor1975). Notably, an amodal representation does not preserve anything analogical about the perceptual experience of the object, event, or state. Instead it is abstracted away from this experience and transduced into some sort of representation language that is abstract and amodal in format. Whether the resulting representation takes the form of something like a logical structure, a feature list, or vectors on which different values can be positioned, the assumption is that the multi-modal experience of something and its conceptual representation do not take place in the same system (Barsalou et al. Reference Barsalou, Niedenthal, Barbey, Ruppert and Ross2003; Niedenthal et al. Reference Niedenthal, Barsalou, Winkielman, Krauth-Gruber and Ric2005b).
The social psychology and emotion literatures are filled with evidence, now supported by findings from neuroimaging studies, that is consistent with an embodied or grounded account of conceptual processing (Barsalou Reference Barsalou1999; Reference Barsalou, Cohen and Lefebvre2005; Reference Barsalou2008; Gallese & Lakoff Reference Gallese and Lakoff2005; Niedenthal Reference Niedenthal, Lewis, Haviland-Jones and Barrett2008). From this perspective, the modality-specific states that represent perception, action, and introspection when one interacts with a particular entity or has a particular subjective experience also represent these same stimuli and events when processed in their absence. For example, retrieving the memory of a landscape involves reactivating (simulating) parts of the visual states that were active while perceiving it. From this perspective, then, having a concept is having the ability to reenact experiencing an instance of the respective category.
As can be seen from these two accounts of concepts, the embodied account provides a natural way to link conceptual knowledge about smiles and the related social situation to the actual perception of smiles. Rather than assuming that a smile activates an amodal knowledge structure to represent its meaning, the embodied simulation account proposes instead that a smile triggers a simulation of a smile experience that includes emotion, motor activity, and somatosensory experience (Niedenthal et al. Reference Niedenthal, Barsalou, Ric, Krauth-Gruber, Feldman-Barrett, Niedenthal and Winkielman2005a). Once this simulation becomes active, it provides a conceptual interpretation of the perceived smile, going beyond the information given to place it in a context. Clearly, much about this account remains to be developed, especially the relevant neural systems and their role in social processing.
The grounded cognition position is important in presenting the SIMS model for a specific reason, namely, because an embodied simulation is not always a response to a currently perceived smile. Additionally, an embodied simulation can also be triggered by the activation of related knowledge or expectations (e.g., stereotypes). For instance, when someone activates conceptual knowledge about infants, this could produce an embodied simulation of an infant. Significantly for the SIMS account to follow, these conceptually triggered simulations may not always be relevant or correct in the current situation, when a different embodied simulation triggered by the actual situation and facial expression offer a more correct interpretation. Rather than mimicking and feeling the reward value of an enjoyment smile present in the current situation, for example, an individual who expects to see a dominance smile may experience negative feelings and interpret the smile as expressing superiority, as the result of unjustified conceptual processing (Halberstadt et al. Reference Halberstadt, Winkielman, Niedenthal and Dalle2009).
6. The simulation of smiles (SIMS) model
We now present a model that seeks to integrate brain and behavior in accounting for the representation of smile meaning. We focus on the possible meanings of a smile as expressing enjoyment, affiliation, or dominance. These judgments can be seen as having correct answers only in so far as smiles with certain identity, postural, morphological, and dynamic markers have been selected or developed as stimuli. In addition, stimulus development would require validation by reference to the intentions and feelings of the person who is smiling, as well as the demonstration of reasonable consensus in interpretation by perceivers. For example, a set of empirically derived enjoyment smiles would be static or dynamic facial displays of people who were experiencing joy, as indicated by self-report, at the time of the smile.
First, we present the core SIMS model, which establishes how grounding smiles differentially in neural systems and behavior causes them to be interpreted differently as meaning enjoyment, affiliation, or dominance (see Figure 4 in sect. 6.1.1). Then, we propose conditions under which perceptual cues, experiential cues, and conceptual knowledge are used to interpret smiles in these different ways (Figures 5–7; Adolphs Reference Adolphs2002; Kirouac & Hess Reference Kirouac and Hess1999). The SIMS model does not attempt to account for the entire neural circuitry that underlies the processing of emotional facial expressions (as discussed, e.g., in Adolphs Reference Adolphs2002; Reference Adolphs2006; Heberlein & Atkinson Reference Heberlein and Atkinson2009), nor does the SIMS model attempt to account for the details of real-time neural time courses. Instead, the focus is on the conditions under which embodied simulation and other cues are used to arrive at judgments of the three smile meanings. Thus, our account aims to represent cognitive-behavioral function, but not the timing of neural activity.
The SIMS model has been largely developed using data collected in Western countries. Nevertheless, it is essential to note that cultural differences may modulate our account. Some clear predictions across cultures can be imagined and have been articulated elsewhere (Niedenthal et al., in preparation).
6.1. Details of the SIMS model
6.1.1. Core SIMS model
We begin by considering how smiles are interpreted in the most ecologically valid situation, that is, in which the smile has an uncertain meaning (in the sense of being unexpected in the context). First, consider the empirically derived enjoyment smile illustrated in the top panel of Figure 4. As the result of eye contact, a reward experience is generated in the basal ganglia, and motor mimicry is generated in the cortical motor regions (in particular, those described by Schilbach et al. Reference Schilbach, Eickhoff, Mojzisch and Vogeley2008). In turn, these two activations produce corresponding bodily experience in the somatosensory cortex. The top panel of Figure 4 represents this overall pattern. Thus, the most straightforward case of processing a smile's meaning begins with the detection of uncertainty (producing amygdalar activation), which, in turn, directs gaze toward the eyes (eye contact), followed by the generation of reward, motor mimicry, and corresponding somatosensory experience. As this initial example illustrates, an embodied simulation of enjoyment grounds visual perception of the perceived smile, leading to the interpretation that the person is “happy.”
The top panel illustrates the case of an enjoyment smile (A) presented such that the meaning is initially uncertain. The perception of the smile is accompanied by activation in the amygdala (B). Research suggests that amygdala activation enhances the probability that eye contact with the smiling person will be made (C). In the SIMS model, eye contact has evolved as a trigger for embodied simulation. Eye contact thus produces increased activation in the reward centers of the basal ganglia (D1) and in motor regions, described by Schilbach et al. (Reference Schilbach, Eickhoff, Mojzisch and Vogeley2008) (D2), that support motor mimicry (E). These motor and limbic processes then produce bodily sensations in somatosensory cortex (F). On the basis of these neural activations and behaviors, the smile is judged as indicating that the smiling individual feels happy (G). The middle panel illustrates the process that results in the judgment of a smile as affiliative. The only difference between the content of the two panels (A′–G′) is the additional OFC (D3′) activation, which in theory supports distinctive positive affect related to attachment. The bottom panel shows the processing of a dominance smile (A″). Amygdala activation would again be expected (B″) and eye contact would be predicted to occur (C″). Dominance smiles may be associated with a pattern of asymmetrical neural activation related to withdrawal-related negative affect (e.g., Davidson et al. Reference Davidson, Ekman, Saron, Senulis and Friesen1990; D1″). Activation in relevant motor regions (D2″) would be expected and output resulting in mimicry (E″). Because of the role of prefrontal cortices in processing status, OFC or contiguous regions may also be involved (D3″). Implications of these supported by somatosensory cortices (F1″) will ground a judgment of a smile as a smile of superiority of some type (G″).
The middle panel of Figure 4 illustrates the same process for the judgment that a smile is an affiliative smile (e.g., that the smile is understood as “friendly” or “conciliatory”). Many of the relevant brain regions are very similar to those just described for the judgment of an enjoyment smile. For an affiliative smile, however, processing would also involve OFC activation and perhaps the involvement of closely related prefrontal regions. As discussed previously, these areas may selectively support the distinctive positive feeling of seeing an individual smile with whom one has a close relationship, as in the finding mentioned earlier when OFC differentiates the sight of one's own smiling baby from the sight of an unknown smiling baby (e.g., Minagawa-Kawai et al. Reference Minagawa-Kawai, Matsuoka, Dan, Naoi, Nakamura and Kojima2009; Nitschke et al. Reference Nitschke, Nelson, Rusch, Fox, Oakes and Davidson2004).
The bottom panel of Figure 4 illustrates the processing of a dominance smile. Again, amygdala activation would be expected to signal the expression's uncertainty and its potential significance. Here, however, negative rather than rewarding affect may be experienced. Although speculative, we have suggested that dominance smiles, like smiles hiding negative intentions or feelings, are associated with the experience of negative rather than positive affect, as indicated by right-lateralized activation (Boksem et al. Reference Boksem, Smolders and de Cremer2009; Davidson et al. Reference Davidson, Ekman, Saron, Senulis and Friesen1990). Because OFC and contiguous regions are central to processing social status (Zink et al. Reference Zink, Tong, Chen, Bassett, Stein and Meyer-Lindenberg2008), ventral frontal cortex may also be involved. As before, activation of cortical motor regions and correspondent mimicry also occur and produce corresponding somatosensory experience associated with the feeling of being dominated (e.g., that the smile is experienced as “superior” or “condescending”).
6.1.2. Behavioral constraints in SIMS
Altering the core SIMS model in principled ways facilitates further understanding of smile processing.
6.1.2.1. Experimental inhibition of behavior
In contrast to the three cases in Figure 4, the two cases in Figure 5 assume that embodied simulation is not likely to occur. An enjoyment smile is used as the example stimulus event. As the top panel of Figure 5 illustrates, no eye contact is achieved with the smiler as was the case for Figure 4. This may occur experimentally when the smile has low uncertainty, as in a blocked design when many smiles are presented one after the other. As a result, the rich behavior and brain activity associated with eye contact is absent. Instead, recognition of the expression as a smile only reflects processing visual features of the mouth. The matching of this visual input to perceptual representations of stored smiles is indicated in the top panel of Figure 5 by activation in occipito-temporal cortices (Adolphs Reference Adolphs2002; Calder & Young Reference Calder and Young2005). Further interpretation of the smile's meaning beyond simple recognition could occur if conceptual knowledge produces expectations about smiling and possibly a simulated smile (as described in further detail later). Because eye contact is not available to trigger embodied simulation, some other source of information must be consulted to provide meaning, should additional meaning be relevant.
The top panel shows processing of an enjoyment smile (A) when eye contact is not achieved for experimental reasons (B), as in a blocked design (in which uncertainty is low). As a result, processing primarily focuses on visual features of the smile. The matching of visual input for the perceived smile to stored perceptual representations of previously experienced smiles is indicated by the activation of occipito-temporal cortices (C) (Adolphs Reference Adolphs2002; Calder & Young Reference Calder and Young2005). Semantic associations to the perceptual representation requiring involvement of prefrontal cortex, such as OFC (D), could be relied on for producing a judgment (E). The bottom panel depicts a situation in which motor mimicry to an enjoyment smile (A′) is inhibited for experimental reasons, as through the blocking of facial musculature (e.g., Oberman et al. Reference Oberman, Winkielman and Ramachandran2007). Perception of the smile will be associated with amygdala activation (B′), and eye contact will be made (C′). Because motor mimicry is inhibited, activation in motor systems and emotion systems will be absent or reduced. Matching of visual input to stored perceptual representations still occurs in occipito-temporal cortices (D1′), and premotor regions may be weakly active (D2′), reflecting the processing of an as-if motor loop. Again, semantic associations, requiring involvement of prefrontal cortex (E′), would be necessary for a specific judgment of meaning (F′).
The bottom panel depicts a second case in which facial mimicry to a smile is inhibited. Inhibition can occur experimentally when spontaneous expression is blocked by mechanical means (e.g., Oberman et al. Reference Oberman, Winkielman and Ramachandran2007). In these situations (when uncertainty is again high), perception of the smile will be associated with amygdala activation. Because motor mimicry in inhibited, however, activation in motor systems and emotion systems will be absent or reduced, although premotor regions may be partially active, reflecting the as-if motor loop that simulates motor activity at a relatively abstract level. Because actual motor activity does not occur however, the smile is not felt in somatosensory areas, and recognition occurs on the basis of matching visual input about the perceived face to stored perceptual representations. Again, interpretation of the smile's specific meaning based on embodied simulation could occur, but would require the use of conceptual knowledge or expectations, given the absence of embodied simulation that results from eye contact.
6.1.2.2. Social inhibition of behavior
The experimental situations represented in Figure 5 also occur in natural social situations, but for far more interesting reasons. Eye contact and mimicry will sometimes be suppressed when a smile is displayed by an individual with whom exchanging eye contact and mimicry is too intimate, risky, or aversive (Lee et al. Reference Lee, Josephs, Dolan and Critchley2006; Likowski et al. Reference Likowski, Mühlberger, Seibt, Pauli and Weyers2008; Mondillon et al. Reference Mondillon, Niedenthal, Gil and Droit-Volet2007). Unlike in the artificial experimental context, however, it is likely that embodied simulations will become active and contribute to judging a smile's meaning. Such simulations, however, will again not be responses to the perceived smile, but will result from activating conceptual knowledge.
Emotions caused by conceptual knowledge can indeed affect the interpretation of a perceived facial expression via embodied emotion simulations. Niedenthal et al. (Reference Niedenthal, Halberstadt, Margolin and Innes-Ker2000), for example, induced states of happiness, sadness, and neutral emotion in participants, triggered by conceptual knowledge. As participants then viewed movies of faces whose expressions changed from happiness or sadness to neutral emotion, their task was to ascertain when the initial emotional expression on the face ended. The induced emotional state of the participants affected their perception of that offset point. In other words, an emotion-congruence effect occurred between a conceptually activated embodied simulation and a perceptually processed face. Participants in a happy state saw smiles persist longer into the ambiguous neutral zone of the expressions, whereas sad participants saw sadness persist somewhat longer. Emotional traits and styles similarly affect performance on this task, further indicating that conceptual knowledge contributes to the interpreted meaning of a perceived facial expression via embodied simulations of emotion (Feldman-Barrett & Niedenthal Reference Feldman-Barrett and Niedenthal2004; Fraley et al. Reference Fraley, Niedenthal, Marks, Brumbaugh and Vicary2006). Again, conceptual knowledge about the smiler (or about the context in which the smile was encountered) triggered embodied emotion simulations that affected judgments of perceived smiles (e.g., Halberstadt et al. Reference Halberstadt, Winkielman, Niedenthal and Dalle2009).
Figures 6 and 7 illustrate socially inhibited eye contact and mimicry, respectively. For the former, the components are similar to the analogous ones in Figure 4. Here, however, the suppressed social behavior detaches processing from the stimulus and renders it knowledge-based, with embodied simulation and emotional content being provided by conceptual knowledge in memory for the respective facial expressions (indicated in portion A of each panel; Niedenthal Reference Niedenthal, Lewis, Haviland-Jones and Barrett2008). Facial expression may occur, but is not necessarily related to the perceived smile, therefore not counting as mimicry. Instead, such facial expressions reflect simulation of emotions associated with conceptual knowledge. As a result, the observer's smile may not constitute mimicry (this may have occurred in Mojzisch et al. Reference Mojzisch, Schilbach, Helmert, Pannasch, Velichkovsky and Vogeley2006), and may obscure the smile's functional meaning. An interesting question for future research is to examine the neural basis of using one's own knowledge to generate an embodied emotion simulation versus using facial cues in perceived expressions, such as eye contact (e.g., Decety & Grèzes Reference Decety and Grèzes2006). In Figure 7, facial mimicry is the behavior that is inhibited. As discussed earlier (sect. 3.2.2.), inhibited mimicry reduces emotional processing. Even when a prior expectation for a smile meaning exists (such as the expectation that the person is expressing an enjoyment smile), emotional processes may still be inhibited because the motor system cannot become engaged as usual. Under these conditions, conceptual knowledge about emotion in memory may be used instead to establish the meaning of the smile, rather than embodied simulation triggered by eye gaze. Unlike the process illustrated in Figure 6, however, perceivers here may rely more on “disembodied” linguistically based associations and social concepts. Brain regions involved in the use of semantic associations depend on the task and the way in which the semantic knowledge is called for (e.g., Barsalou et al. Reference Barsalou, Santos, Simmons, Wilson, De Vega, Glenberg and Graesser2008).
All three panels illustrate situations in which eye contact does not occur, analogous to the three cases in Figure 4 when eye contact does occur. Although the neural activations are very similar, the suppressed social behavior in Figure 6 detaches any emotional processing that occurs from the perceived smile and renders it knowledge-based. Thus, the emotional content is determined by simulation of conceptual knowledge in memory, rather than being driven by the experience of eye contact (Niedenthal Reference Niedenthal, Lewis, Haviland-Jones and Barrett2008). The top panel illustrates the case in which a smile is believed to be a smile of enjoyment (A). Perception of the smile is accompanied by activation in the amygdala (B). We have defined this as a situation in which eye contact is avoided (C). Nevertheless, because the smile is believed for other reasons to be an enjoyment smile, activation of the reward centers of the basal ganglia (D1) occurs, and also the motor brain regions described by Schilbach et al. (Reference Schilbach, Eickhoff, Mojzisch and Vogeley2008) (D2). Correspondent smiling (E) occurs, but is determined by simulation of conceptual knowledge and does not count as mimicry (as in Halberstadt et al. Reference Halberstadt, Winkielman, Niedenthal and Dalle2009). Conceptual implications are represented by somatosensory cortex (F) and confirmation of the smile as an enjoyment smile (G) is made on this basis. Note, of course, that the judgment could be wrong with regard to smiler intention. The middle panel illustrates the same process that results in the judgment that a smile is an affiliative smile (A′–G′), again without eye contact. The only difference with the top panes is the again additional robust OFC activation (D3′), which in theory supports positive affect related to attachment. The bottom panel shows the processing of a smile believed to be a smile of dominance (A″), where again amygdala activation simply reflects visual cues from the mouth (B″) and eye contact does not occur (C″). Withdrawal-related negative affect (e.g., Davidson et al. Reference Davidson, Ekman, Saron, Senulis and Friesen1990; D1″) and activation in relevant motor regions (D2″) that support smile production is again expected (E″), as described earlier for Figure 4. Because of the role of prefrontal cortices in processing social status (Zink et al. Reference Zink, Tong, Chen, Bassett, Stein and Meyer-Lindenberg2008), OFC or contiguous regions may also be involved (D3″). The representation of these inputs in somatosensory cortices (F″) will serve to confirm the interpretation of the smile as an expression of dominance (G″).
The figure illustrates the judgment that a smile communicates enjoyment when mimicry is inhibited. Perception of the smile is associated with amygdala activation (B) and eye contact is made (C). Because motor mimicry is inhibited, activation in motor systems and emotion systems is absent or reduced. Premotor regions may be active (D1), reflecting the processing of an as-if motor loop, driven by semantics in the temporal and prefrontal cortex (D2) to grounding interpretation (E).
6.2. Judgments of smiles as true versus false
One particular distinction between types of smiles has received considerable attention: whether a smile is true or false (e.g., Miles & Johnstone 2007). Asking this question already implies a culture-laden value that smiles can be false per se (Fridlund Reference Fridlund and Abel2002). Some cultures and some individuals seem to mean by false that there is a desire to deceive or manipulate (Frank Reference Frank and Abel2002). This would be akin to saying that a smile put on to hide a negative intent is a false one. Alternatively, individuals sometimes mean that the smile was produced voluntarily and that this act is itself inauthentic or false (Fridlund Reference Fridlund and Abel2002). This would be akin to saying that any intentional smile is, by definition, false. Although the definitions of true and false smiles will ultimately have to be bootstrapped, at least by culture, we can propose a working definition here: Judging a smile as a true smile is the normative or default judgment, and it means that the smile signifies positive feelings of some kind (e.g., pleasure or joy) or positive intentions. Conversely, judging a smile as false means that the perceiver believes that the smile was motivated by a desire to hide, moderate, or justify something negative (e.g., a lie, a criticism, a feeling of superiority or contempt, a manipulation, an inappropriate affect).
Rather than a clear dichotomy, the true-false distinction is a continuum that is most often represented in the English language by a value on a scale of “genuineness,” “authenticity,” or “sincerity” (Hess & Kleck Reference Hess and Kleck1990). Different measurable features of smiles have been claimed to be associated with these judgments. Indeed, Duchenne smiles are often judged as more genuine than non-Duchenne smiles. Individuals also judge smiles with a slow onset, a slow offset, and a shorter apex as more genuine than smiles that have a sudden onset, a sudden offset, and a long apex (Krumhuber & Kappas Reference Krumhuber and Kappas2005). The former smiles are further associated with more positive personality judgments (e.g., Krumhuber et al. Reference Krumhuber, Manstead and Kappas2007). Notably, however, these features do not in any way map isomorphically to the true/false judgment. For instance, as already mentioned, the importance of Duchenne markers for judging smiles in still photos varies with culture (Thibault et al., submitted). In addition, the Duchenne marker is far less important in dynamic as compared to still smile stimuli (e.g., Krumhuber et al. Reference Krumhuber, Manstead and Kappas2007).
The SIMS model predicts when a smile will be judged as more or less genuine and on what basis. For example, enjoyment and affiliative smiles with initially uncertain meaning should always be judged as true smiles. This is because the default judgment of a smile being genuine is based on the presence of positive affect and motor mimicry in an embodied simulation associated with it. In contrast, dominance smiles should be more likely to be judged as false smiles, due to the greater presence of negative affect and withdrawal in their associated simulations (e.g., Davidson et al. Reference Davidson, Ekman, Saron, Senulis and Friesen1990).
In contrast, when eye contact is not made, or when mimicry is inhibited by social factors, true/false judgments should be guided by cultural beliefs and stereotypes stored in conceptual knowledge. As a result, true/false judgments may be much less systematic than when eye contact and mimicry both support the embodied simulation.
A recent study by Maringer et al. (in press) supports this proposal. Participants were given empirically derived “true” dynamic smiles or “false” dynamic smiles to see and asked to rate them on scales of genuineness (Krumhuber et al. Reference Krumhuber, Manstead and Kappas2007). Orthogonally crossed with the true-false manipulation, half of the participants were able to freely mimic the smiles, and the remaining half held a pencil in their mouths so as to block facial mimicry. The results were clear: In the mimicry condition, participants who saw true smiles rated them as more genuine than participants who saw false smiles. In the mimicry-blocked condition, participants did not distinguish between the two types of smiles. Thus, the ability to mimic facial expressions was essential for distinguishing true from false smiles, implicating embodied simulation in performing these judgments accurately.
In a second study, participants viewed true smiles and were told either that these smiles occurred in a context in which a true smile would be expected by cultural stereotypes or that these smiles occurred in a context in which a false smile would be expected. Mimicry was blocked in half of the participants, and all participants rated the smiles on genuineness. Results showed that when mimicry could occur, cultural beliefs and stereotypes did not affect the perceived genuineness of the smiles, but when mimicry was blocked, smiles that occurred in contexts associated with true smiles were judged as more genuine than those that occurred in contexts in which false smiles could also occur. Again, embodied simulation appears to be a critical cue for establishing that a smile is genuine.
7. Beyond smiles
The present article and model focus on the smile because it is probably the most complex of all emotional expressions. Not only are smiles highly diverse in the conditions under which they occur, they are also highly diverse in their possible meanings. Significantly, however, similar embodied processes should support interpreting the expression of other basic emotions (e.g., anger, sadness), as well as the expression of secondary emotions (e.g., shame, embarrassment). In general, the perception of any facial expression should trigger affective, motor, and somatosensory experience that gives the expression meaning. Thus, the structure, but not the content, of the SIMS model can be used analogically to investigate the interpretation of facial expressions besides the smile.
As an illustrative example, consider anger. Expressions of anger are used to establish and maintain social hierarchies, among other social functions (Keltner & Haidt Reference Keltner and Haidt1999). Anger expressions are slightly different than those of contempt (Ekman & Heider Reference Ekman and Heider1988), but may occur in similar situations. Indeed a family of anger-related emotions exists, ranging from frustration to outrage. Given their different social functions, it would be essential for socially competent individuals to correctly experience their subtle meanings in the respective situations (e.g., Fischer & Roseman Reference Fischer and Roseman2007).
A functional analysis of anger expressions is likely to suggest different social determinants of eye contact and mimicry than those for smiling. For instance, there is evidence that anger expressions are mimicked less to the degree that a situation is social, unless anger is expressed with averted gaze indicating that the source of the anger is not the perceiver (Bourgeois & Hess Reference Bourgeois and Hess2008; Hess et al. Reference Hess, Adams and Kleck2009a). Furthermore, because anger is statistically less frequent in most environments than the smile, and has different implications for the observer, some neural circuits, such as those involving the amygdala may be activated more often and be less sensitive to environmental determinants of uncertainty. Perceiving and mimicking an anger expression may also have different somatosensory experiences than perceiving and mimicking related emotions, such as frustration, contempt, and outrage. Finally, anger has been linked to neural structures that do not appear relevant for interpreting smiles, including the ventral striatum (Calder et al. Reference Calder, Keane, Lawrence and Manes2004) and dopamine receptors (Lawrence et al. Reference Lawrence, Calder, McGowan and Grasby2002).
This example is not intended to be comprehensive. Rather, it simply suggests that the present model can be extended to other emotions, but cannot be extended effectively without due consideration of the specific circumstances, together with the accompanying behavioral and neural responses. We believe, however, that the basic structure of the model, with its emphasis on integrating specific social behaviors with related forms of embodied simulation, is viable in modeling the meaning of all emotional facial expressions.
8. Conclusions
The impact of Darwin's (1872) writing on facial expression and its centrality in more recent theories of emotion (e.g., Ekman Reference Ekman, Wagner and Manstead1989) has pushed the study of emotion expression toward a dictionary of facial muscles and their combinations as signs to internal states. The notion that a physical description has access to disembodied amodal knowledge, which can, in turn, be used to interpret the perceptual input, has been a natural way to think about the processing of emotional facial expressions in the context of the Cognitive Revolution. Given more recent developments in grounded cognition and neuroscience, however, it seems increasingly likely that a construct along the lines of embodied simulation is necessary for a full account of how people establish the meaning of facial expressions. Increasingly, new empirical techniques provide additional ways of measuring embodied simulation and establishing causal evidence for its roles in social processing (e.g., the use of transcranial magnetic stimulation [TMS] in Pitcher et al. Reference Pitcher, Garrido, Walsh and Duchaine2008). The SIMS model of the smile is an attempt to show how behavioral and neuroimaging findings can be integrated to generate new and productive process models of facial expression more generally. Solving the riddle of the smile will, we believe, provide important groundwork for understanding the full array of emotional facial expressions.
ACKNOWLEDGMENTS
The authors thank Ralph Adolphs, Anthony Atkinson, Markus Brauer, Julie Grèzes, and Piotr Winkielman for comments on an earlier draft of the paper. We extend a particularly grateful thanks to Lawrence Barsalou for providing very specific and detailed feedback on multiple drafts. The work of Paula M. Niedenthal and Martial Mermillod was supported by a grant (FaceExpress–Blanc CSD9 2006) from the Agence National de Recherche (ANR), France; the work of Ursula Hess by a grant from the National Science Foundation (NSF) and the Social Sciences and Humanities Research Council of Canada.
Target article
Emotion simulation and expression understanding: A case for time
Related commentaries (1)
The Simulation of Smiles (SIMS) model: Embodied simulation and the meaning of facial expression