Introduction: The problem of reconstructing cognition

Most psychologists and cognitive scientists and almost all linguists investigate the nature of the mind, intelligence, and language from a uniquely human perspective. But the origins of these attributes lie in the deep history of the primate brain, and all organic explanations for them must derive from non-human and human primate evolutionary history. There are a variety of approaches to reconstructing the origins of cognition and language. The fossil record is the only direct evidence of human evolution, but it is fragmentary. It can tell us little about the roots of cognition and language because the soft tissues involved do not fossilize.

Our understanding of human evolution is therefore greatly enhanced by evidence from living primates, especially those most closely related to us, the great apes – the chimpanzee (Pan troglodytes), bonobo (P. paniscus), gorilla (Gorilla gorilla), and orangutan (Pongo pygmaeus). While we must be careful not to consider apes as living fossils, they are exemplars of the way in which natural selection molds large-brained, social primates to the natural environment. They exhibit learned, interpopulational culturally varied behavior beyond that of any animal other than ourselves. They provide us with a sense of the likeliest range of options, behavioral and cognitive, that natural selection would have taken with ancestral forms of humans.

As an anthropologist, I study non-human primates because they inform us about the likely course that the evolution of human form and function has taken.

Introduction

Homologies and language evolution

We are interested here in homologous brain structures in the human and the macaque, those which may be characterized as descended from the same structure of the brain of the common ancestor. However, quite different paths of evolution, responsive to the need of different organisms for similar functions, may yield organs with similar functions yet divergent evolutionary histories – these are called homoplasic, rather than homologous. Arbib and Bota (2003) forward the view that homology is not usefully treated as an all-or-none concept except at the grossest level, such as identifying visual cortex across mammalian species. Even if genetic analysis were to establish that two brain regions were homologous in that they were related to a common ancestral form it would still be important to have access to a measure of similarity to constrain too facile an assumption that homology guarantees similarity across all criteria.

Indeed, from the perspective of computational and comparative neuroscience, declared homologies may be the start, rather than the end, of our search for similarities that will guide our understanding of brain mechanisms across diverse species. When two species have diverged significantly from their common ancestors, a single organ x in ancestor A might differentiate into two organs y and y′ in modern species B and three organs z, z′, and z″ in modern species C. For example, a region involved in hand movements in monkey might be homologous to regions involved in both hand movements and speech in humans.

Introduction

This chapter reviews recent evidence that the linguistic representation of action is grounded in the mirror neuron system. Section 10.2 summarizes the major semantic properties of action verbs and argument structure constructions, focusing on English but also considering cross-linguistic diversity. The theoretical framework is Construction Grammar, which maintains that the argument structure constructions in which action verbs occur constitute basic clausal patterns that express basic patterns of human experience. For example, the sentence She sneezed the napkin off the table exemplifies the Caused Motion Construction, which has the schematic meaning “X causes Y to move along path Z,” and the sentence She kissed him unconscious exemplifies the Resultative Construction, which has the schematic meaning “X causes Y to become Z” (Goldberg, 1995).

Section 10.3 addresses the neuroanatomical substrates of action verbs and argument structure constructions. A number of neuroimaging and neuropsychological studies are described which suggest that different semantic properties of action verbs are implemented in different cortical components of the mirror neuron system, especially in the left hemisphere: (1) motoric aspects of verb meanings (e.g., the type of action program specified by kick) appear to depend on somatotopically mapped primary motor and premotor regions; (2) agent–patient spatial–interactive aspects of verb meanings (e.g., the type of object-directed path specified by kick) appear to depend on somatotopically mapped parietal regions; and (3) visual manner-of-motion aspects of verb meanings (e.g., the visual movement pattern specified by kick) appear to depend on posterior middle temporal regions.

Framework

In this chapter I show in outline how human language as we know it could have evolved incrementally from mental capacities it is reasonable to attribute to lower primates and other mammals. I do so within the framework of a formal computational theory of language understanding (Hobbs et al., 1993). In the first section I describe some of the key elements in the theory, especially as it relates to the evolution of linguistic capabilities. In the next two sections I describe plausible incremental paths to two key aspects of language − meaning and syntax. In the final section I discuss various considerations of the time course of these processes.

Strong AI

It is desirable for psychology to provide a reduction in principle of intelligent, or intentional, behavior to neurophysiology. Because of the extreme complexity of the human brain, more than the sketchiest account is not likely to be possible in the near future. Nevertheless, the central metaphor of cognitive science, “The brain is a computer,” gives us hope. Prior to the computer metaphor, we had no idea of what could possibly be the bridge between beliefs and ion transport. Now we have an idea. In the long history of inquiry into the nature of mind, the computer metaphor gives us, for the first time, the promise of linking the entities and processes of intentional psychology to the underlying biological processes of neurons, and hence to physical processes.

Introduction

The Mirror System Hypothesis (MSH) (see Arbib, Chapter 1, this volume) postulates that higher communication skills and language may be grounded in the basic skill of action recognition, enhanced by the ability of movement pantomime and imitation. This chapter will examine from a computational point of view how such recognition and imitation skills can be realized. Our goal is to take our formal knowledge of how to produce simple actions, also called movement primitives, and explore how a library of such movement primitives can be built and used to compose an increasingly large repertoire of complex actions. An important constraint in the development of this material comes from the need to account for movement imitation and movement recognition in one coherent framework, which naturally leads to an interplay between perception and action.

The existence of motor primitives (a.k.a. synergies, units of actions, basis behaviors, motor schemas, etc.) (Bernstein, 1967; Arbib, 1981; Viviani, 1986; Mataric, 1998; Miyamoto and Kawato, 1998; Schaal, 1999; Sternad and Schaal, 1999; Dautenhahn and Nehaniv, 2002) seems, so far, the only possibility for how one could conceive that biological and artificial motor systems are able to cope with the complexity of motor control and motor learning (Arbib, 1981; Schaal, 1999, 2002a, 2002b; Byrne 2003). This is because learning based on low-level motor commands, e.g., individual muscle activations, becomes computationally intractable for even moderately complex movement systems. Our computational approach to motor control with movement primitives is sketched as an abstract flowchart in Fig. 6.1 (Schaal, 1999).

Introduction

“Mirror” neurons are found in area F5 of the monkey brain, and they fire both when a monkey grasps an object and when the monkey observes another individual grasping the object (e.g., Rizzolatti et al., 1996; see Arbib, Chapter 1, this volume, for further discussion). Mirror neurons have also been found in the rostral part of the monkey inferior parietal lobule (Gallese et al., 2002). Like mirror neurons, signers must associate the visually perceived manual actions of another signer with self-generated actions of the same form. Sign language comprehension and production requires a direct coupling between action observation and action execution. However, unlike mirror neurons for hand movements recorded in monkey, signing is not tied to object manipulation. Mirror neurons for grasping in monkey fire only when an object is present or understood to be present and do not fire when just the grasping movement is presented (Umiltà et al., 2001). Furthermore, unlike grasping and reaching movements, sign articulations are structured within a phonological system of contrasts. The hand configuration for a sign is determined by a phonological specification stored in the lexicon, not by the properties of an object to be grasped. These facts have interesting implications for the evolution of language and for the neural systems that underlie sign language and action.

The fact that sign language exhibits form-based patterning of meaningless elements (i.e., phonology) distinguishes signs from actions, even when the two appear quite similar on the surface; for example, the American Sign Language (ASL) sign TO-HAMMER resembles the act of hammering.

Introduction: duality of patterning

Language can be viewed as a structuring of cognitive units that can be transmitted among individuals for the purpose of communicating information. Cognitive units stand in specific and systematic relationships with one another, and linguists are interested in the characterization of these units and the nature of these relationships. Both can be examined at various levels of granularity. It has long been observed that languages exhibit distinct patterning of units in syntax and in phonology. This distinction, a universal characteristic of language, is termed duality of patterning (Hockett, 1960). Syntax refers to the structuring of words in sequence via hierarchical organization, where words are meaningful units belonging to an infinitely expandable set. But words also are composed of structured cognitive units. Phonology structures a small, closed set of recombinable, non-meaningful units that compose words (or signs, in the case of signed languages). It is precisely the use of a set of non-meaningful arbitrary discrete units that allows word creation to be productive.

In this chapter we outline a proposal that views the evolution of syntax and of phonology as arising from different sources and ultimately converging in a symbiotic relationship. Duality of patterning forms the intellectual basis for this proposal. Grasp and other manual gestures in early hominids are, as Arbib (Chapter 1, this volume) notes, well suited to provide a link from the iconic to the symbolic.

Introduction

Our progress towards an understanding of how the human brain evolved to be ready for language starts with the mirror neurons for grasping in the brain of the macaque monkey. Area F5 of the macaque brain is part of premotor cortex, i.e., F5 is part of the area of cerebral cortex just in front of the primary motor cortex shown as F1 in Fig. 1.1 (left). Different parts of F5 contain neurons active during manual and orofacial actions. Crucially for us, an anatomically segregated subset of these neurons are mirror neurons. Each such mirror neuron is active not only when the monkey performs actions of a certain kind (e.g., a precision pinch or a power grasp) but also when the monkey observes a human or another monkey perform a more or less similar action. In humans, we cannot measure the activity of single neurons (save when needed for testing during neurosurgery) but we can gather comparatively crude data on the relative blood flow through (and thus, presumably, the neural activity of) a brain region when the human performs one task or another. We may then ask whether the human brain also contains a “mirror system for grasping” in the sense of a region active for both execution and observation of manual actions as compared to some baseline task like simply observing an object. Remarkably, such sites were found in frontal, parietal, and temporal cortex of the human brain.

There are many ways to approach human language – as a rich human social activity, as a formal system structured by rules of grammar, and as a pattern of perception and production of utterances, to name just a few. The present volume uses this last concern – with the perception and production of utterances – as its core. The aim is not to ignore the other dimensions of language but rather to enrich them by seeking to understand how the use of language may be situated with respect to other systems for action and perception.

The work is centered on, but in no way restricted to, the Mirror System Hypothesis (introduced by Arbib and Rizzolatti in 1997). This is the hypothesis that the mirror neuron system for the recognition of movements of the hands in praxic action – which is present both in monkey and in human in a number of areas including Broca's area (generally considered to be the frontal speech area) – provides the evolutionary basis for the brain mechanisms which support language. The Mirror System Hypothesis sees the ancestral action recognition system being elaborated through the evolution of ever more capable neural mechanisms supporting imitation of hand movements, then pantomime emerging on the basis of displacement of hand movements to imitate other degrees of freedom. A system of “protosign” emerges as conventionalized codes extend the range of manual communication, and serves as scaffolding for “protospeech.”

Introduction

The aim of the present volume is to enrich human language dimensions by seeking to understand how the use of language may be situated with respect to other systems for action and perception. There is strong evidence that higher human cognitive functions, such as imitation and language, emerged from or co-evolved with the ability for compositionality of actions, already present in our ancestors (Rizzolatti and Arbib, 1998; Lieberman, 2000; Arbib, 2003; Arbib, Chapter 1, this volume). Corroborating evidence from psychology (Greenfield et al., 1972; Iverson and Thelen, 1999; Glenberg and Kaschak, 2002), neurobiology (Pulvermüller, 2003) and cognitive sciences (Siskind, 2001; Reilly, 2002) strongly support a close relationship between language, perception, and action. Social abilities, such as imitation, turn-taking, joint attention and intended body communication, are fundamental for the development of language and human cognition. Together with the capacity for symbolization, they form the basis of language readiness (Rizzolatti and Arbib, 1998; Arbib, 2003).

The work presented in this chapter takes inspiration from this body of experimental evidence in building a composite model of the human's cognitive correlates to action, imitation and language. The model will contribute to develop a better understanding of the common mechanisms underlying the development of these skills in human infants, and will set the stage for reproducing these in robots and simulated agents.

A recent trend of robotics research follows such views, by equipping artifacts with social capabilities.

Tools to investigate the anatomy of the central autonomic systems

The functional specificity of the autonomic regulation of target organs and the neurophysiological recordings from peripheral autonomic neurons (see Chapter 4) argue that central autonomic systems must be differentiated. It seems likely then that the central organization is reflected in the micro- and macroanatomy of the central autonomic systems. Originally, central stimulation and lesion studies, involving recordings from autonomic nerves or autonomic effector responses (e.g., blood pressure, heart rate, gastrointestinal motility), gave clues about the location of the autonomic centers in the neuraxis. However, the results from these studies were imprecise so that it was not possible to identify anatomically distinct populations of central neurons as being associated with distinct autonomic output systems. Furthermore, focal electrolytical lesioning and focal electrical stimulation did not discriminate between destruction or excitation of cell bodies and passing axons. With the introduction of chemical stimulation of cell bodies (by microionophoretic application of excitatory amino acids, such as glutamate, aspartate or dl-homocystic acid, or inhibitory amino acids, such as γ-aminobutyric acid [γ-aminobutyric acid] or glycine) it was possible to topically excite or inhibit small populations of central neurons selectively. This technique turned out to be a valuable tool in physiological experimentation.