Action Spaces

One major approach to understanding the representation of action knowledge was influenced by previous work investigating the mental representation of objects (e.g. Reference Beymer and PoggioBeymer & Poggio, 1996; Reference EdelmanEdelman, 1998; Reference GärdenforsGärdenfors, 2004; Reference Kriegeskorte, Mur and BandettiniKriegeskorte et al., 2008a, b). These studies develop the idea that known actions are described by multidimensional ‘spaces’ (see Figure 3), in which each type of action occupies a point in that space (Reference Dima, Tomita, Honey and IsikDima et al., 2022; Reference Kabulska and LingnauKabulska & Lingnau, 2022; Reference Lingnau and DowningLingnau & Downing, 2015; Reference Thornton and TamirThornton & Tamir, 2022; Reference Tucciarelli, Wurm, Baccolo and LingnauTucciarelli et al., 2019; Reference Watson and BuxbaumWatson & Buxbaum, 2014; Reference Zhuang and LingnauZhuang & Lingnau, 2022). Traversing along one hypothetical dimension, actions should vary systematically on one action property; furthermore, by hypothesis, the distance between a pair of actions in such a space should be directly related to the perceived subjective similarity of those actions (Reference Dima, Tomita, Honey and IsikDima et al., 2022; Reference Tucciarelli, Wurm, Baccolo and LingnauTucciarelli et al., 2019).

Figure 3 Illustration of the action ‘spaces’ idea. Action kinds may be construed as atom-like points in representational spaces, the dimensions of which may correspond to psychologically meaningful distinctions. Positions of actions reflect their values on hypothetical mental dimensions. Distances between actions are proportional to subjective judgments of the similarity between them. Here we present only a reduced example for the sake of clarity; realistic action spaces would be far more complex.

Several possible action spaces have been identified in studies that combine a data-driven element such as subjective similarity ratings or analyses of text corpora, with data-reduction methods such as hierarchical cluster analysis, principal component analysis, or multidimensional scaling. For example, in a series of studies based on analyses of large text corpora and other measures, Reference Thornton and TamirThornton & Tamir (2022) identified Abstraction, Creation, Tradition, Food, Animacy, and Spiritualism as key action dimensions (referred to as the ACT-FASTaxonomy). These dimensions successfully captured variance in the judged similarity of action words, and also described the socially relevant features of actions (e.g. how, why and by whom an action is performed).

Focusing instead on action images, Reference Tucciarelli, Wurm, Baccolo and LingnauTucciarelli et al. (2019) required participants to group pictures of actions by their perceived similarity in a multi-item arrangement task (Reference Kriegeskorte and MurKriegeskorte & Mur, 2012). Analysis by k-means clustering and principal component analysis revealed several meaningful action categories, including food-related actions, communicative actions, and locomotion. These categories fell along dimensions that were organized according to the type of change induced by the action, and the type of need to be fulfilled by the action (e.g. basic/physiological versus higher social needs). Reference Kabulska and LingnauKabulska & Lingnau (2022) used a very similar approach with pictures of 100 different actions that revealed additional action categories such as Interaction, Gestures, and Aggressive actions. Analyses of action features provided evidence for a strong dimension capturing the positive or negative valence of the action. Still further evidence shows how typically intended outcomes may also shape action spaces. For example, Reference Tarhan, de Freitas and KonkleTarhan et al. (2021) analysed data from a multi-arrangement task to find that a goal similarity model predicted judgments of action similarity better than models based on movement similarity or on visual similarity.

Together, this family of approaches has revealed a diverse set of candidate principles and dimensions that organize action knowledge. The ‘spaces’ that emerge from a given study depend on the kinds of actions being considered, and the specific task set for the observers (a topic that we will return to in Section 3). It may be that multiple distinct spaces, or hierarchically nested spaces, best capture the huge and diverse range of actions that observers can understand, rather than a single space. Indeed, in each of the studies described, a large proportion of variability remained unexplained, suggesting that there are additional organizing principles not yet identified.

Our knowledge about actions changes as we learn; and in different contexts, different aspects of actions may be more or less immediately relevant. Accordingly, action spaces are likely to be dynamic over both short and longtime scales. Attention or context effects may dynamically ‘warp’ the shape of action spaces as they are used in making action judgments. For example, Reference Shahdloo, Çelik, Urgen, Gallant and ÇukurShahdloo et al. (2022) used neuroimaging to reveal how the distributed patterns of activity evoked by actions were modulated by changing the observers’ immediate task to attend either to their communicative or their locomotion characteristics. At longer time scales, effects of experience and expertise are relevant. For one example, over the course of years of practice, a gymnast must build a dense and detailed ‘space’ representing her specialist events, which would likely have more, and more meaningful dimensions relative to a novice observer. Moreover, attention and experience might modify the weights of specific dimensions (Reference GärdenforsGärdenfors, 2004). To our knowledge, these ideas have not been explored empirically from the action spaces perspective.

By definition, the organization of observed actions into action spaces shows similarities with semantic networks (e.g. Reference Collins and QuillianCollins & Quillian, 1969) and semantic categories generally (see e.g. Reference LevinLevin, 1993; Reference PinkerPinker, 1989; Reference TalmyTalmy, 1985). However, the organization of actions depicted by visual stimuli and by verbal material is bound to differ in important ways since visually presented actions are concrete instantiations of a specific action, whereas language has the flexibility to refer to actions at varying levels of abstraction. For related discussions, see Reference Tucciarelli, Wurm, Baccolo and LingnauTucciarelli et al. (2019); Reference Vinson and ViglioccoVinson & Vigliocco (2008); and Reference Watson and BuxbaumWatson & Buxbaum (2014).

Action Frames

The ‘space’ metaphor is very powerful for capturing the key dimensions of action knowledge as well as subjective judgments of the similarity of different kinds of action. One limitation of the approach, however, is that it obscures some of the rich internal structure that constitutes our knowledge of familiar actions. This is not easily captured in a dimensional representation that treats action concepts as single points in a mental space. Accordingly, building on previous conceptions of knowledge frames (Reference MinskyMinsky, 1975) and scripts (Reference Bower, Black and TurnerBower et al., 1979; Reference Schank and AbelsonSchank & Abelson, 1977) here we consider the idea of action frames. Related ideas have also been more recently explored in the context of action understanding (Reference Aksoy, Orhan and WörgötterAksoy et al., 2017; Reference Chersi, Ferrari and FogassiChersi et al., 2011; Reference Zacks, Speer, Swallow, Braver and ReynoldsZacks et al., 2007), although these have tended to focus more narrowly on specific issues such as the sequential nature of actions. An action frame may be seen as a schematic representation that describes, abstractly, important features of an action, such as its intended outcomes or goals; means by which the goals typically are achieved; and the kinds of movements, postures, objects, and locations associated with that kind of action (Figure 4). These associations are assumed to be picked up from statistical co-occurrences in our natural environment. Action frames may help to identify action kinds by interacting with the output of perceptual systems that recognize objects and scenes (Reference Epstein and BakerEpstein & Baker, 2019), detect and classify people (Reference Pitcher and UngerleiderPitcher & Ungerleider, 2021), and estimate their poses and movements (Reference Giese and PoggioGiese & Poggio, 2003). These perceptual systems analyse an observed action, abstracting over some details (e.g. the colour of a knife) while emphasizing others (e.g. its position relative to the ingredients, and its motion related to the movements of the chef). Consistent evidence gathered in the perceptual systems and schematic action frame representations mutually reinforce each other, whereas inconsistent evidence leads to suppression. Recent perspectives have also highlighted the perceptual significance of typical relationships amongst scene elements (Reference Bach, Knoblich, Gunter, Friederici and PrinzBach et al., 2005; Reference Green and HummelGreen & Hummel, 2006; Reference Hafri and FirestoneHafri & Firestone, 2021; Reference Kaiser, Quek, Cichy and PeelenKaiser et al., 2019), which will also have diagnostic value for distinguishing among different kinds of actions. We can understand action classification as emerging from competitive interactions amongst perceptual systems, and between perceptual systems and action frames, such that (normally) this system rapidly converges on an interpretation that best coheres with the available evidence (cf. Reference ErnstErnst, 2006; Reference Netanyahu, Shu, Katz, Barbu and TenenbaumNetanyahu et al., 2021).

Figure 4 Illustration of the ‘action frames’ perspective. A, B: Perceptual subsystems process objects, body postures, movements, and scenes to extract relevant aspects of the action, and the relationships among them. C: Mental ‘action frames’ capture the roles, relationships, and reasons that comprise our action knowledge. Slots of a given frame gather perceptual evidence about scene elements. Matches increase the evidence for one action (<cooking>) relative to others (<cleaning>). Normally, interactions between perceptual subsystems and action frames cohere rapidly to select one action frame; action understanding is this convergence of activity. Links omitted for clarity.

Action frames must be abstract enough to encompass the wide perceptual variety of different action instances that we described earlier. They must also be flexible or probabilistic, rather than rigid, to account for our ability to tolerate variations: for example, cooking normally happens in the kitchen but may also take place outdoors in a campsite. Further, a key aspect of our knowledge about action kinds is an understanding of the desired outcomes that normally motivate a given action. Accordingly, frames need to describe not only knowledge about the directly observable elements that constitute an action; they also need to include descriptions of the expected mental states of the actors. Finally, they also require access to more general semantic knowledge of the physical and social world. This includes, for example, knowledge about typical cause-and-effect relationships (cooking pasta makes it soft and edible; stealing from someone makes them angry). Likewise, we deploy knowledge about the ways in which the properties of objects like tools make them suited to specific kinds of manipulations for specific kinds of outcomes – the shape, hardness, and weight distribution of a hammer makes it useful for driving in nails (e.g. Reference Buxbaum, Shapiro and CoslettBuxbaum et al., 2014; Reference Osiurak and BadetsOsiurak & Badets, 2016; see also Reference Binkofski and BuxbaumBinkofski & Buxbaum, 2013).

Action frames as described here might offer several useful properties. First, they may describe the highly predictable way in which actions generally unfold over time that is not readily captured by a semantic space of actions. For example, purchasing food ingredients is not just semantically related to cooking; one typically precedes the other in a predictable way. Likewise, at a finer grain, preparing a soup may include obtaining, washing, peeling, and slicing vegetables, sub-actions that only make sense in a specific order. These regularities enable an observer to anticipate what is likely to follow next (Reference Aksoy, Orhan and WörgötterAksoy et al., 2017; Reference Chersi, Ferrari and FogassiChersi et al., 2011; Reference Schank and AbelsonSchank & Abelson, 1977; Reference Zacks, Speer, Swallow, Braver and ReynoldsZacks et al., 2007). It may be difficult to capture these kinds of relationships in a scheme in which action kinds are considered as ‘points’ in a multidimensional Euclidean space. A more abstract and compositional representation may be better suited to capture the temporal and causal relationships that describe typical chains of actions.

Prediction

We previously highlighted the important theme of prediction in action understanding. Here we briefly explore how expectations and predictions might play out from the perspectives of action spaces and action frames. For one example, Tamir, Thornton, and colleagues have proposed that the proximity of actions in a space reflects not only semantic similarity, but also transitional probability. In general, one cooking event is more likely to immediately follow another cooking action than (say) a vehicle-repair action. In a series of studies, Reference Thornton and TamirThornton & Tamir (2021a) found that participants’ ratings of transition probabilities between actions corresponded well to actual rates of action transitions (determined on the basis of several large naturalistic datasets). More important, Reference Thornton and TamirThornton & Tamir (2021b) demonstrated that actions that were close to each other in the ACT-FAST action space described above were also more likely to follow each other.

From the action frames perspective, expectations – for example, evidence that an action will unfold in a kitchen – allows relevant action frames (e.g. for cooking, eating, and washing up) to compete with and suppress less relevant ones. In turn, this enables pre-activation of cooking-relevant objects (e.g. a knife), again at the expense of other unrelated objects (e.g. pliers). The net effect of these competitive interactions should be a relative advantage in understanding actions that are consistent with expectations, by suppression of unlikely alternatives. Indeed, when actions are embedded in an incongruent context, they take longer to be processed in comparison to actions embedded in a neutral or congruent context (Reference Wurm and SchubotzWurm & Schubotz, 2012, Reference Wurm and Schubotz2017). Likewise, ambiguous actions are recognized with higher accuracy when taking place in a congruent context in comparison to incongruent or neutral contexts (Reference Wurm and SchubotzWurm & Schubotz, 2017; Reference Wurm, Artemenko, Giuliani and SchubotzWurm et al., 2017a). Here, the surrounding context (e.g. the emotional facial expression of an agent) shapes the interpretation of the action (e.g. an approaching fist with the intention to punch or to greet the observer with a fist bump; see e.g. Reference Kroczek, Lingnau, Schwind, Wolff and MühlbergerKroczek et al., 2021), just as ambiguous objects (e.g. Reference Brandman and PeelenBrandman & Peelen, 2017) and emotional facial expressions (Reference Aviezer, Trope and TodorovAviezer et al., 2012) are interpreted in reference to their immediate context in the domains of scene and body perception.

To summarize, here we have considered two complementary perspectives on how the mind organizes long-term knowledge about familiar actions. These are not mutually exclusive ideas: as action understanding is so complex, each perspective may better describe different aspects of what we know about actions, how that knowledge is applied to understanding ‘what’ a given action is, and how that supports predictions about the actors and events that we interact with.

Observational Learning

Observational learning (sometimes ‘social learning’) refers, in the broadest sense, to acquiring knowledge about the contingencies between behaviour and outcomes by observing others (Reference Bandura and WaltersBandura & Walters, 1977). By observation, without the need for first-hand experience that may be ineffective, slow, or even dangerous, we can learn that touching an electrified fence is painful; that others find playing a musical instrument rewarding; or that posting controversial opinions to Twitter attracts both praise and condemnation.

One focus area within this broad theme concerns the transfer of learning from one domain (normally vision) to motor performance. What can we learn about serving a tennis ball, shaping a clay pot, or performing brain surgery, by watching an expert do those things? A fundamental issue in this literature concerns the role of symbolic or cognitive representations in mediating the benefits of observational learning. As an example, a tennis novice observing a coach in order to learn to serve might try to segment the observed movement according to summary cues such as ‘down together’, ‘up together’, ‘back’, ‘hit’, and ‘follow-through’. In a classic study of motor sequence learning, Reference Bandura and JeffreyBandura & Jeffrey (1973) compared participants’ learning and retention of simple manual action sequences as a function of rehearsal type. Those participants who were instructed to encode observed sequences in verbal terms (e.g. with letter codes) recalled sequences better than those who were not, especially over longer intervals. The insight is that at least in some cases, symbolic representations are more informationally compact and durable (Reference Uithol, van Rooij, Bekkering and HaselagerUithol et al., 2012), and therefore more readily rehearsed and retrieved at a later time, compared to ‘raw’ motor representations.

Related research asks whether the action representations acquired from observation are explicit, in the sense of being overtly retrievable and usable as part of a strategy, or instead implicit, in the sense of being acquired without awareness. For example, serial response tasks require participants to rapidly press one of several keys corresponding to the location of a single target. Typically, if the sequence of locations is repeated in a second-order cycle, response times improve with practice. However, explicit knowledge of the sequence may be absent, for example as tested in a subsequent task requiring participants to guess the next item in a series (Reference SegerSeger, 1997). In contrast, studies of observational learning – in which participants learned keypress sequences simply by watching the target events appear – found that sequence learning was mediated mainly by explicit, verbalizable knowledge (Reference Kelly, Burton, Riedel and LynchKelly et al., 2003). Interestingly, Reference Bird, Osman, Saggerson and HeyesBird & colleagues (2005) found that when sequences were observed not simply as visual events, but as the outcomes of a live actor’s behaviour, implicit learning was also revealed, suggesting that the actor’s presence encouraged a more first-person like encoding of the action sequences.

The preceding examples all relate to categorical actions and action sequences – pressing one of four keys, for example – for which the specific action dynamics were not relevant. Other studies have examined observational learning with actions that involve more continuous variables. For example, Reference Mattar and GribbleMattar & Gribble (2005) required participants to make simple reaches under the influence of an unseen ‘force field’ that deflected those movements. Participants who first watched another actor perform this task before attempting it showed stable benefits (e.g. smaller disruptions to their own reach trajectories) compared to controls. Notably, this observational learning remained essentially intact even when it took place under a demanding concurrent cognitive load, suggesting a relatively automatic and implicit form of learning (see also Section 3).

Conversely, other work has examined the transfer of motor learning to visual action judgments. Reference Casile and GieseCasile & Giese (2006) demonstrated how learning to perform an unusual pattern of walking movements selectively improved visual detection of those movements when they were rendered as point-light animations. In a more naturalistic context, Reference Aglioti, Cesari, Romani and UrgesiAglioti et al. (2008) demonstrated that experienced basketball players made better predictions about the outcome of observed free throws in comparison to individuals with similar visual experience (experienced coaches, sports journalists) and to novices. Improved performance of players in this example, compared to experienced coaches, invites the interpretation that motor experience specifically contributes to improved action understanding. In a similar vein, Reference Knoblich and FlachKnoblich & Flach (2001) found that participants were better able to judge from a video where a thrown dart would land, when that video depicted a previous throw that they had performed themselves, compared to another thrower. An important feature of each of these motor-to-vision studies is that the observed actions were seen from a side view, that is, one that is normally unavailable for one’s own actions. Therefore, the learning exhibited in those situations must extend over modalities (from motor to visual) and must also generalize across visual perspective.

The preceding findings imply a close overlap between an observer’s own motor repertoire and her ability to understand actions. Yet other findings show that these two variables can be dissociated. For example, a series of studies of individuals with congenital dysplasia who lack upper arms (and therefore have no upper-limb motor representations), revealed essentially normal performance in a variety of tasks. These included different aspects of action understanding, including the ability to name pantomimes and point-light animations, to learn new actions, and to predict the outcome of basketball free-throws (Reference Vannuscorps and CaramazzaVannuscorps & Caramazza, 2016; but see Reference Vannuscorps and CaramazzaVannuscorps & Caramazza, 2023). Developmental studies reveal similar dissociations; for example, three-month-old infants have been shown to interpret observed actions as goal-directed before they are able to perform reach and grasp actions themselves (Reference Liu, Brooks and SpelkeLiu et al., 2019; see also Reference SouthgateSouthgate, 2013). In sum, whereas several studies suggest that the ability to detect subtle differences in the kinematics of observed movements is modified by the observer’s experience, relevant motor experience is not always a necessary requirement for the ability to understand actions.

Imitation

Observational learning generally relates to the effects of experience on later performance (or perception) of an action. In contrast, imitation concerns the attempt to immediately replicate another person’s action. Here, key research questions have concerned the development of imitation (to what extent is imitation present from birth?) and automaticity (e.g. to what extent does imitation occur in spite of the observers’ current goals?).

The claim that even newborn infants possess not only the ability to imitate facial expressions but a tendency to do so spontaneously (Reference Meltzoff and MooreMeltzoff & Moore, 1977) has been highly influential, although the core findings have been questioned by more recent large-scale replication efforts (e.g. Reference Oostenbroek, Suddendorf and NielsenOostenbroek et al., 2016). Similarly, evidence for ‘automatic’ imitation in adults has proven fruitful. A simple procedure, typically called the automatic imitation task, was developed by Brass and colleagues (Reference Brass, Bekkering, Wohlschläger and PrinzBrass et al., 2000; Reference Cracco, Bardi and DesmetCracco et al., 2018). Here, participants lift either their index or middle finger in response to a visually presented numeric cue. At the same time as the cue, an on-screen hand is shown to lift either the index or middle finger. While the finger movement is task-irrelevant, participants are nonetheless normally faster when that movement also matches the action they are required to execute, compared to when it does not match. Variants of this procedure have been developed to understand this compatibility effect, to identify its neural correlates (Reference Darda and RamseyDarda & Ramsey, 2019), to assess its malleability following training (Reference Catmur, Walsh and HeyesCatmur et al., 2007), and to test the claim that it is ‘automatic’ (Reference Cracco, Bardi and DesmetCracco et al., 2018).

In contrast to these relatively simple and controlled tasks, researchers in social psychology have asked whether, in more naturalistic settings, participants tend to unwittingly mimic the movements or body postures of confederates. For example, Reference Chartrand and BarghChartrand & Bargh (1999) reported a ‘chameleon effect’ whereby individuals may unintentionally match others’ overt behaviours, and moreover that the experience of being imitated in this fashion increases liking. In general, then, there is some evidence of the tendency for irrelevant or incidental actions of others to influence the observer’s own concurrent behaviours, even in the absence of an explicit goal to imitate.

A final important distinction is that between imitation and emulation, where the latter refers to an achievement of the same end state via different specific motoric means (see also Reference Bekkering, Wohlschlager and GattisBekkering et al., 2000; Reference CsibraCsibra, 2008; Reference HeyesHeyes, 2001; Reference Tomasello, Kruger and RatnerTomasello et al., 1993). For example, given no specific instructions, preschool children will tend to emulate the target of an action (e.g. reaching for the right ear) instead of producing a faithful copy of the observed action (e.g. reaching for the right ear with the contralateral hand; Reference Bekkering, Wohlschlager and GattisBekkering et al., 2000). This finding illustrates that actions may normally be understood by default from the ‘intentional stance’ – as deliberate and rational behaviours, performed by an agent for a reason – a topic we return to in Section 2.3.

‘How’ beyond Observational Learning

The specific manner in which an action is performed (e.g. grasping a bottle at the top or the bottom) can provide cues about the immediate goal of an actor (e.g. to move the bottle or to use it to pour a drink). Observers may use a variety of sources, such as the kinematics and the preshaping of the hand, as well as perceived gaze direction (e.g. Reference Aglioti, Cesari, Romani and UrgesiAglioti et al., 2008; Reference Ambrosini, Costantini and SinigagliaAmbrosini et al., 2011; Reference Cavallo, Koul, Ansuini, Capozzi and BecchioCavallo et al., 2016) to anticipate how an action will unfold, and to coordinate actions of two or more actors (see also Reference Azaad, Knoblich and SebanzAzaad et al., 2021). Access to the precise way in which an action is performed also plays a role in the predictive coding framework of action understanding (Reference KilnerKilner, 2011; Reference Kilner, Friston and FrithKilner et al., 2007). We will return to this point in Section 4.

Studies from the direct perception tradition (Reference GibsonGibson, 1979/2014) and more recently from the social vision framework (Reference Adams, Adams, Ambady, Nakayama and ShimojoAdams et al., 2011) examine how the observed patterns of others’ movements provide rich clues about the states and traits of other individuals (with the caveat that such cues may not be fully valid). For example, studies of point-light recordings of actors performing simple actions revealed that they support above-chance identification of the actor (Reference Loula, Prasad, Harber and ShiffrarLoula et al., 2005) and discrimination of emotion (Reference Atkinson, Dittrich, Gemmell and YoungAtkinson et al., 2004), gender (Reference Kozlowski and CuttingKozlowski & Cutting, 1977), or sexuality (Reference Johnson, Gill, Reichman and TassinaryJohnson et al., 2007). Those studies guided by the direct perception framework have tried to identify simple physical properties of movement patterns that reliably cue social variables without the need for complex cognitive analysis. For example, Reference Kozlowski and CuttingKozlowski & Cutting (1977) identified that a lower centre of movement reliably signals female as opposed to male actors from walking patterns. Studies in the related social vision framework have tended to focus on outcomes, as seen, for example, in the finding that observers’ judgments of actors’ health from movement patterns was a reliable predictor of which actor would be selected in a hypothetical political election (Reference Kramer, Arend and WardKramer et al., 2010). In sum, the details of action dynamics, even from minimal stimuli like ‘point-light’ animations, can provide information about the states and traits of the actors that perform those actions. In the following section, we examine how observed actions also provide evidence about more complex mental states such as goals and beliefs.

Physiological Measures of Motor System Activity

One way to examine the level of activation of the motor system is to induce a brief electrical current in the primary motor cortex via transcranial magnetic stimulation (TMS). This can trigger measurable motor evoked potentials (MEPs) in contralateral peripheral muscles such as in the hand. The strength of these MEPs indexes the excitability of the corresponding stimulated motor region (for a review, see Reference Bestmann and KrakauerBestmann & Krakauer, 2015). Moreover, the comparison of MEPs between different muscles of the hand that are involved in specific types of grasping movements enables an examination of muscle-specific activation of the motor cortex during action observation. In general, findings that the excitability of the motor cortex can be modulated by passively observed compatible actions have been taken as evidence of a ‘mirror-like’ mechanism in humans (Reference Rizzolatti, Fogassi and GalleseRizzolatti et al., 2001).

Using this approach, several studies found that observing manual grasping actions leads to a muscle-specific activation of some of the same motor pathways that would be used if the observer were to perform that action (Reference Baldissera, Cavallari, Craighero and FadigaBaldissera et al., 2001; Reference Fadiga, Fogassi, Pavesi and RizzolattiFadiga et al., 1995; Reference Maeda, Kleiner-Fisman and Pascual-LeoneMaeda et al., 2002; Reference Strafella and PausStrafella & Paus, 2000). Such findings have been interpreted as a sign of an automatic ‘resonance’ in the observer’s motor system caused by observing the action. Other studies demonstrated that MEPs are sensitive to predicted action outcomes. For example, Reference Aglioti, Cesari, Romani and UrgesiAglioti et al. (2008) found that in expert basketball players, specific MEPs were evoked during the observation of missed free-throws, upon release of the ball but before the outcome was known (see also Reference Gangitano, Mottaghy and Pascual-LeoneGangitano et al., 2001, Reference Gangitano, Mottaghy and Pascual-Leone2004; Reference Kilner, Vargas, Duval, Blakemore and SiriguKilner et al., 2004). More recent studies demonstrated that MEPs are not only modulated by low level kinematic features of an action, but are also affected by higher level processes, such as the difference between honest and deceptive actions (Reference Finisguerra, Amoruso, Makris and UrgesiFinisguerra et al., 2018) or the congruence between the context and the action (Reference Amoruso and UrgesiAmoruso & Urgesi, 2016; Reference Betti, Finisguerra, Amoruso and UrgesiBetti et al., 2022). Findings like these have contributed to a debate about whether MEPs reflect an automatic motor resonance, or whether instead they are also modulated by top-down influences (for a review, see Reference Amoruso and FinisguerraAmoruso & Finisguerra, 2019).

Another series of experiments exploited the mu rhythm, a frequency in the cortical EEG signal in the range of 8–13 Hz over sensorimotor cortex. In general terms, the mu rhythm is suppressed during selective attention and motor preparation, and the mu rhythm can be sensitive to the type of movement and handedness (for review, see Hobson & Bishop, 2007). Mu suppression, like MEPs, has been used as an index of motor system activity during the passive observation of others’ actions. In common with the properties ascribed to some mirror neurons, for example, suppression of the mu rhythm is stronger during the observation of a precision grip of an object compared to a mimicked precision grip in the absence of an object (e.g. Reference Muthukumaraswamy and JohnsonMuthukumaraswamy et al., 2004a, Reference Muthukumaraswamy, Johnson and McNairb). However, the view of the mu rhythm as an index of human mirror neuron activity also remains debated (e.g. Reference Hobson and BishopHobson & Bishop, 2017). In particular, it is not straightforward to determine whether a suppression in the 8–13 Hz window originates from sensorimotor areas, or whether it instead stems from a modulation of the alpha rhythm originating from occipital cortex. This alternative indicates that the modulation of the mu rhythm during action observation might instead, or additionally, reflect visual attention or perceptual processes.

Studies from the brain stimulation and mu rhythm lines of work have been useful to explore how the states of the observers’ motor system are influenced by what the observer sees and understands about an action. However, the functional implications of some of these findings remain debated, in that several interpretations remain about what processes these neural measures reveal.

Human Neuroimaging

Early human neuroimaging studies using PET (Reference Grafton, Arbib, Fadiga and RizzolattiGrafton et al., 1996; Reference Rizzolatti, Fadiga and MatelliRizzolatti et al., 1996) and fMRI (Reference Iacoboni, Woods and BrassIacoboni et al., 1999) adopted the logic that anatomical overlap between brain areas that are recruited during the observation of actions, and the execution, imagination, or imitation of actions, would provide evidence of ‘mirror-like’ human brain representations. Some common findings in these initial studies laid the foundation for later human neuroimaging investigations. For example, fMRI studies demonstrated that during passive observation of goal-directed actions, participants recruit a consistent set of brain region including the ventral premotor cortex (PMv) extending into the posterior IFG, the preSMA, somatosensory cortex, anterior and superior sections of the parietal cortex, and portions of the lateral occipitotemporal cortex (see Figure 7B). As a shorthand, these regions are often collectively referred to as the ‘action observation network’. Later studies showed how parts of this network (premotor, parietal, and somatosensory areas) also overlap with the areas involved during motor imagery and/or movement execution (for meta analyses, see e.g. Reference Arioli and CanessaArioli & Canessa, 2019; Reference Caspers, Zilles, Laird and EickhoffCaspers et al., 2010; Reference Hardwick, Caspers, Eickhoff and SwinnenHardwick et al., 2018; but see Reference Turella, Pierno, Tubaldi and CastielloTurella et al., 2009). Together, findings like these have been taken to show a common neural representation of the corresponding visual and motor aspects of actions, as a possible system-level homologue of the mirror neuron.

However, an influential commentary by Reference Dinstein, Thomas, Behrmann and HeegerDinstein et al. (2008) noted limitations in this logic, namely that spatially overlapping activations (e.g. of regions responding to observed and to executed actions) may reflect overlapping but distinct neural populations rather than a shared representation (see also Reference Peelen and DowningPeelen & Downing, 2007). Better evidence for a ‘mirror like’ representation would be found in a demonstration that neuronal populations within overlapping regions are selective for specific motor acts. Accordingly, several studies have investigated cross-modal action selectivity using fMRI adaptation or repetition suppression (e.g. Reference Grill-Spector and MalachGrill-Spector & Malach, 2001). This method is based on the observation that the repetition of a specific stimulus property, such as object category, leads to an attenuation of the fMRI signal in neuronal populations that represent the repeated stimulus property.

Several studies followed this approach to seek evidence for cross-modal action selectivity as suggested by Reference Dinstein, Thomas, Behrmann and HeegerDinstein et al. (2008). Neuronal populations with such properties should adapt when the same action is repeated, across performance or observation of that action, compared to different actions. Reference Dinstein, Hasson, Rubin and HeegerDinstein et al. (2007) and Reference Press, Weiskopf and KilnerPress et al. (2012) obtained action-selective adaptation during observation and also during execution in overlapping parietal regions. However, they did not observe cross-modal adaptation – that is, for observation of an action followed by its execution, or vice versa. Reference Chong, Cunnington, Williams, Kanwisher and MattingleyChong et al. (2008) found cross-modal adaptation in the right IPL, but only tested for execution followed by observation (see also Reference de la Rosa, Schillinger, Bülthoff, Schultz and Umildagde la Rosa et al., 2016). In contrast, Reference Lingnau, Gesierich and CaramazzaLingnau et al. (2009) tested for cross-modal adaptation in both directions; this effect was found in the left IPL, but only when observation was followed by execution. Finally, using a similar approach, Reference Kilner, Neal, Weiskopf, Friston and FrithKilner et al. (2009) found cross-modal adaptation in the IFG in both directions.

Following these initial contradictory results, doubts arose about one of the key assumptions underlying these studies: namely, that mirror neurons adapt to repetition in the same way as other types of neurons. Reference Caggiano, Pomper and FleischerCaggiano et al. (2013) reported that mirror neurons in F5 do not reduce their response amplitude following two repetitions. By contrast, Reference Kilner, Kraskov and LemonKilner, Kraskow, & Lemon (2014) found a modulation of the firing rate, the latency, and beta band power of the local field potential in this region, but only after repetitions of 7–10 trials.

Together, human neuroimaging studies using repetition suppression to examine cross-modal action selectivity remain inconclusive. It is likely that at least one contribution to this is the variety of tasks, stimuli, and action types that have been tested. For example, the combined effects of action type (e.g. object-directed vs intransitive), viewpoint (e.g. first- or third-person), and meaningfulness (e.g. simple movements vs grasps vs pantomimes) have not been factorially explored within a single repetition suppression study of action understanding.

Multivoxel pattern analysis (MVPA; Reference Norman, Polyn, Detre and HaxbyNorman et al., 2006) approaches offer another way to identify shared visual and motor representations of actions that may avoid the issues with interpreting ‘overlap’ identified by Reference Dinstein, Thomas, Behrmann and HeegerDinstein (2008). For example, in a series of studies, Reference Oosterhof, Wiggett, Diedrichsen, Tipper and DowningOosterhof et al. (2010, Reference Oosterhof, Tipper and Downing2012a, Reference Oosterhof, Tipper and Downingb; reviewed in Reference Oosterhof, Tipper and DowningOosterhof et al., 2013) used whole-brain surface-based ‘searchlight’ MVPA (Reference Kriegeskorte, Goebel and BandettiniKriegeskorte et al., 2006; Reference Oosterhof, Wiggett, Diedrichsen, Tipper and DowningOosterhof et al., 2010) to identify brain regions in which the local patterns of activity are a) similar for a given action, whether passively observed or performed by the participant; and also b) dissimilar for different actions. This logic captures the core concept of the mirror neuron in carrying representations that generalize over modality and also distinguish between different actions. These studies consistently revealed regions of the anterior parietal and lateral occipitotemporal cortex that met those defining criteria. Further, patterns of activity in the ventral premotor cortex were also cross-modal and action specific, but only for actions viewed from the first-person perspective – in contrast to initial evidence on mirror neurons that exhibited at least some evidence for selectivity to third-person views of action (see also Reference Caggiano, Fogassi and RizzolattiCaggiano et al., 2011). By contrast, viewpoint independence of cross-modal action-selective representations was obtained in parietal and occipitotemporal cortex only (Reference Oosterhof, Tipper and DowningOosterhof et al., 2012a).

Finally, the most direct way to examine whether the human brain contains cells with mirror properties is to perform direct recordings in humans undergoing preparation for neurosurgery. Reference Mukamel, Ekstrom, Kaplan, Iacoboni and FriedMukamel et al. (2010) recorded extracellular single and multiunit activity from a group of neurons in patients being treated for epilepsy. The authors found neurons that responded both during observation and execution of actions in the supplementary motor area, the hippocampus, and other nearby regions. A subset of these neurons showed excitation during execution, but inhibition during observation (see also Reference Kraskov, Dancause, Quallo, Shepert and LemonKraskov et al., 2009). The presence of both excitation and inhibition is in line with computational models of action planning (see e.g. Reference CisekCisek, 2007) that assume that several potential actions are specified in parallel and compete with each other until there is enough sensory evidence in favour of one of these actions.

The preceding section has briefly laid out some of the main neuroscientific approaches that have been used to apply the mirror neuron logic to the human brain. Overall, the results of these studies converge to implicate several key regions in one or more aspects of action understanding (see Figure 6). Where they diverge is in the extent to which they confirm or fail to confirm the key concepts of cross-modal, view-invariant, and action-specific representations that were inherited from initial descriptions of mirror neurons.

Expertise

If one’s own motor representations play a causal role in action understanding, it stands to reason that the richness of those representations should influence the nature of understanding. Accordingly, several studies have examined how different kinds and levels of action expertise (and specifically motor expertise) change the way these actions are processed in brain regions of the action observation network. The general logic is that relative to the novice, an expert’s richer motor representations of an action repertoire enable an improved, or even qualitatively different, understanding of observed actions from that domain.

Observers’ expertise modulates fMRI activity within the action observation network (see Reference Turella, Wurm, Tucciarelli and LingnauTurella et al., 2013, for a review). For example, one series of studies examined brain responses of expert dancers from two disciplines (ballet and capoeira). In their domain of expertise, dancers exhibited more activity in prefrontal and parietal regions relative to dance movements of the other domain (Reference Calvo-Merino, Glaser, Grèzes, Passingham and HaggardCalvo-Merino et al., 2005) and to dance movements of the expert domain that were motorically but not visually familiar (Reference Calvo-Merino, Grèzes, Glaser, Passingham and HaggardCalvo-Merino et al., 2006; see also Reference Cross, Hamilton and GraftonCross et al., 2006, and Reference Jola, Abedian-Amiri, Kuppuswamy, Pollick and GrosbrasJola et al., 2012). The interpretation of these findings was that motoric aspects of dance expertise influenced the way that experts visually perceived and understood actions, by way of a cross-modal visuo-motor representation.

An apparent paradox in this literature is that in some cases the effect of experience appears to decrease rather than increase the activity in action observation regions (see e.g. Reference Gardner, Aglinskas and CrossGardner et al., 2017). For example, Reference Petrini, Pollick and DahlPetrini et al. (2011) found such a pattern of results when comparing the neural activity elicited by observing ‘point light’ animations of drumming actions, in experienced versus novice drummers. These divergent effects may reflect two different facets of expertise: on the one hand, expertise (e.g. with performing a class of actions) provides a rich framework by which observed actions may be assigned meanings that are not accessible to novices; hence a relative increase in activity in relevant regions for experts. In contrast, expertise also entails familiarity with actions from the relevant domain, supporting an improved ability to predict what will be seen next. Indeed, the literature on perceptual expectations emphasizes the suppressing effect of expectations on neural activity in line with predictive coding models (Reference Summerfield, Trittschuh, Monti, Mesulam and EgnerSummerfield et al., 2008).

Modulation by Task Requirements

In Section 3, we discussed the automaticity of action understanding. Neuroscientific studies have also approached this question by asking to what extent brain activity is modulated by manipulations of the observers’ task, such as by instruction to attend to an action or instead to an object in a scene (Reference Wurm, Ariani, Greenlee and LingnauWurm et al., 2015); to attend to the goal or to the effector involved in an action (Reference Lingnau and PetrisLingnau & Petris, 2013); to attend to the type of action performed by an animal or rather its taxonomic category (Reference Nastase, Connolly and OosterhofNastase et al., 2017; see also Reference KemmererKemmerer, 2021); or to attend to the type of action, the actor, or the colour of the object (Reference Orban, Ferri and PlatonovOrban et al., 2019).

Here, typically the task modulates the engagement of specific brain regions implicated in action understanding. For example, one study showed a higher response in the lateral occipitotemporal cortex when focusing on the ‘what’ of an action, and a higher response when focusing on the ‘why’ of an action in several areas, including the dorsomedial prefrontal cortex and the temporal pole (Reference Spunt, Satpute and LiebermanSpunt et al., 2011; but see Reference Spunt, Kemmerer and AdolphsSpunt et al., 2016). Part of the logic of such studies is to apply reverse inference from previous findings. For example, activity in the ‘action observation network’ may be interpreted as evidence for processing the ‘how’ of an action (Reference Rizzolatti and CraigheroRizzolatti & Craighero, 2004; Reference Rizzolatti and SinigagliaRizzolatti & Sinigaglia, 2010; Reference Caspers, Zilles, Laird and EickhoffCaspers et al., 2010), whereas activity in regions linked to mentalizing tasks is taken to reveal an effort to understand the intentions behind an action (‘why’; e.g. Reference Van Overwallevan Overvalle, 2009; Reference Van Overwalle and Baetensvan Overvalle & Baetens, 2009). More generally, these studies reinforce the view discussed in Section 3, namely that action understanding is not reflex-like, but rather recruits neural processes that adapt to serve the observer’s current goals.

Generalization and Abstraction

Which brain regions show selectivity for specific observed actions, and how abstract or generalized are those representations? Initially, motivated by findings from the mirror-neuron literature, many studies used region-of-interest (ROI) approaches to focus on regions such as the PMv and the IPL. To establish whether these regions demonstrate action selectivity – a response that can distinguish between different observed actions – several studies relied on fMRI adaptation. For example, Reference Hamilton and GraftonHamilton & Grafton (2006) reported that the anterior IPS encodes the object that is the target of the reach, in a way that generalizes over the specific trajectory that is required to reach that object. Similarly, Reference Hamilton and GraftonHamilton & Grafton (2008) reported a representation of the outcome of an action (e.g. an opened or closed box) that generalizes over the specific kinematics required to achieve that outcome, in the right IPL, the left aIPS and the right IFG (see also Reference Majdandžić, Bekkering, van Schie and ToniMajdandžić et al., 2009). Finally, using a related approach called TMS adaptation, Reference Cattaneo, Sandrini and SchwarzbachCattaneo et al. (2010) adapted participants to the observation of hand or foot actions manipulating an object. TMS applied to the IPL and the PMv led to shorter response times for repeated actions relative to non-repeated ones, irrespective of the effector. By contrast, TMS applied to the STS revealed effector-specific adaptation, suggesting action representations at different hierarchical levels in the STS and the IPL/PMv. Together, these studies are a good early example of how neuroimaging and brain stimulation methods could answer qualitative questions about levels of neural action representation, and how they vary across different brain regions.

Tests of the abstractness of an action representation have also been addressed with a cross-decoding MVPA approach. Here, a classifier might initially be trained to distinguish between two observed actions (A and B), based on the activity patterns within a given brain region. Next, that classifier is tested to see whether it can still distinguish between the two actions following a variation in the way the actor performed the action. Using this logic with a whole-brain searchlight approach, several studies reported that it is possible to decode observed actions from patterns of activity in the lateral occipitotemporal cortex (LOTC) and in the IPL across different target objects (Reference Wurm, Ariani, Greenlee and LingnauWurm et al., 2015) and across objects and the kinematics required to manipulate these objects (Reference Wurm and LingnauWurm & Lingnau, 2015). Similarly, Reference Hafri, Trueswell and EpsteinHafri, Trueswell, & Epstein (2017) were able to distinguish between different interaction categories (e.g. biting, kicking, slapping) across different visual formats (static images versus dynamic videos) based on activity in several regions, including occipitotemporal, parietal and left premotor cortex. And Reference Wurm and CaramazzaWurm & Caramazza (2019) were able to decode actions from videos to written descriptions and vice versa from activation patterns in human LOTC.

Together, what these MVPA decoding findings show is that distributed activity patterns can reveal rich information about viewed actions that go beyond a literal description of a single instance of an action, to extend to more general properties. One important point of focus in this body of work has been around the anatomical regions implicated. As noted, the initial human neuroimaging work focused on the role of ventrolateral frontal and parietal regions. However, a typical pattern in a growing number of more recent human studies (e.g. Oosterhof et al., 2012; Reference Wurm and LingnauWurm & Lingnau, 2015; Reference Wurm, Ariani, Greenlee and LingnauWurm et al., 2015, Reference Wurm, Caramazza and Lingnau2017b) is that these abstract action representations are instead found more consistently in posterior occipitotemporal regions. In part, this discrepancy may reflect different neural distributions in different regions, which may be more or less visible to MVPA. Indeed, by using single-cell recordings from two tetraplegic patients with electrode arrays in the posterior parietal cortex, Reference Aflalo, Zhang and RosarioAflalo et al. (2020) were able to decode manipulative actions across different stimulus formats in human parietal cortex.

The studies reviewed in this section so far clearly indicate how rich information about observed actions is implicit in the activity patterns seen in human brain regions beyond the core motor system. Indeed, a common pattern over multiple studies is that the highest degree of generalization, in common with object vision, is found in higher-level visual areas and the parietal cortex (see e.g. Reference Ayzenberg and BehrmannAyzenberg & Behrmann, 2022).

Organization of Observed Actions in Space and Time

In Section 2.1, we described the logic of multidimensional ‘spaces’ that could describe some aspects of knowledge about action categories. More recent studies have adapted this logic – which emerged from work on the representations of concepts, objects, and faces (Reference GärdenforsGärdenfors, 2004; Reference ShepardShepard, 1958; Reference Valentine, Lewis and HillsValentine et al., 2016) – to examine how patterns of brain activity might describe similar neural ‘spaces’ for action representation. Many of these studies adopt the representational similarity analysis (RSA) approach, which uses measures of similarity to describe the notional geometry of a representation of a class of events or stimuli (Reference Kriegeskorte, Mur and BandettiniKriegeskorte et al., 2008a). In this way, comparisons between behavioural and neural measures, or between two different neural measures, are possible at a level of abstraction above the specific items. For example, Reference Tucciarelli, Wurm, Baccolo and LingnauTucciarelli et al. (2019; see also Reference Tarhan, de Freitas and KonkleTarhan et al., 2021; Reference Zhuang, Kabulska and LingnauZhuang et al., 2023) compared representational geometries based on the perceived semantic similarity of observed actions from behavioural measurements, with geometries derived from fMRI multi-voxel activity patterns. They found that neural activity patterns in a set of regions along the ventral and dorsal stream resembled the behaviourally determined action space, in the sense that there was a significant positive relationship between the ‘space’ inferred from behavioural judgments, and that determined from the patterns of brain activity. Thus, these approaches can link, at an abstract level, subjective and neural representations of action knowledge.

If patterns of neural activity capture action ‘spaces’, what is the organization of these spaces? Reference Tarhan and KonkleTarhan & Konkle (2020) identified five distinct distributed clusters of brain regions covering the lateral and ventral occipitotemporal cortex and the intraparietal sulcus. These carried information about body parts and the target of an action during the passive observation of short naturalistic video clips. Responses in four of the identified clusters were organized by the spatial scale of the action (e.g. from small, precise movements involving the hands to large movements involving the entire body). Using a similar approach, Reference Thornton and TamirThornton & Tamir (2022) were able to decode amongst observed actions on the basis of their six-dimensional ACT-FAST taxonomy, based on fMRI activity measured from a widespread set of occipitotemporal, parietal and frontal regions. Finally, using EEG during passive observation of short video clips depicting everyday actions in combination with behavioural ratings, Reference Dima, Tomita, Honey and IsikDima et al. (2022) observed a temporal gradient in action representations. Over a period from 60 to 800 ms, the shape of action ‘spaces’ changed from an emphasis on visual features, to action-related features, and then to social-affective features. Together, studies like these show how specific action-space models can be developed and tested on the basis of neuroimaging data.

Multiple studies have found particularly strong evidence that LOTC plays a role in representing action spaces. For example, Reference Tucciarelli, Wurm, Baccolo and LingnauTucciarelli et al. (2019) found that patterns of activity across the LOTC best capture the semantic similarity structure of observed actions, when variability due to specific action features such as body parts, scenes, and objects is removed. In that study, actions related to locomotion, communication, and food formed clusters both in the behaviourally determined and in the neural action space. Given evidence for abstract action ‘spaces’ in LOTC, is there evidence of any anatomical organization to the patterns of activity within this region? We have previously made the case for representational gradients across the LOTC, such that the way it encodes an action property (e.g. the extent to which it is person or object-directed) varies continuously across the region. Specific proposed gradients include a posterior-anterior gradient for the dimensions concrete-abstract and visual-multimodal, and a dorsal-ventral gradient for the dimensions intentional-perceptual and animate versus inanimate (e.g. Reference Papeo, Agostini and LingnauPapeo et al., 2019; Reference Tarhan, de Freitas and KonkleTarhan et al., 2021; Reference Wurm, Caramazza and LingnauWurm et al., 2017b; for reviews, see Reference Lingnau and DowningLingnau & Downing, 2015; Reference Wurm and CaramazzaWurm & Caramazza, 2022).

Together, this family of findings shows that the representational similarity approach can test hypotheses about how action knowledge is captured in distributed patterns of brain activity. Moreover, these studies have highlighted the role of the LOTC, and point to several action-relevant features that are captured in this region. At the same time, this review highlights that there is not yet consensus on a single set of organizing dimensions. Indeed, given the flexibility with which observers can process an action depending on their attentional state or task set, such a consensus may not be expected.