Recent years have seen the emergence of a wave of influential theories that highlight the predictive nature of cognition – the so-called predictive brain framework (see Bar Reference Bar2011; Clark Reference Clark2013). A common denominator of these theories is that they paint a picture of the mind wherein our mental representations of the world become active before we engage with reality (i.e., so-called “top-down” processing); this view contrasts with traditional perspectives that assumed that our representation of the current state of the world emerges only after we have acquired evidence from our sense organs (i.e., “bottom-up” processing).
A prominent theory within this framework is the Predictive Processing (PP) approach (e.g., Bar et al. Reference Bar, Kassam, Ghuman, Boshyan, Schmidt, Dale, Hämäläinen, Marinkovic, Schacter, Rosen and Halgren2006; Friston Reference Friston2005). Proponents of PP argue that every encounter we have with reality is akin to scientific hypothesis testing. For example, a person who is about to open the fridge already has prior representations of what they are about to see (e.g., a milk carton); to the extent that this representation successfully predicted the event to come, there is no need for much additional cognitive processing; however, when a discrepancy between the prior representation and bottom-up inputs is detected (i.e., if there is no milk left, a so-called prediction error), then there is a need to update the mental representation in light of the new evidence. Often, theories within the PP camp argue that such updates mimic rules of normative reasoning and, specifically, Bayesian inference.
The ideas of the PP approach have long been influential in the domain of perception science (e.g., Gregory Reference Gregory1980; Helmholtz Reference Helmholtz and Shipley1860/1961) and have become paradigmatic in the neuroscientific literature on this topic (e.g., Bar et al. Reference Bar, Kassam, Ghuman, Boshyan, Schmidt, Dale, Hämäläinen, Marinkovic, Schacter, Rosen and Halgren2006; Friston Reference Friston2005; Rao & Ballard Reference Rao and Ballard1999). More recently, generalizations of PP theory to the domain of action (most notably, active inference theory further described in section 3, e.g., Friston et al. Reference Friston, Daunizeau and Kiebel2009) have been able to account for diverse phenomena in areas such as decision-making (e.g., Schwartenbeck et al. Reference Friston2013) and psychiatry (e.g., Barrett et al. Reference Barrett, Quigley and Hamilton2016; Friston et al. Reference Friston, Stephan, Montague and Dolan2014; Powers et al. Reference Powers, Mathys and Corlett2017). Some have argued that the PP approach, and specifically Active Inference Theory, may provide a unified theory of brain function (Hohwy Reference Hohwy2013) and a grand paradigm for cognitive science (Clark Reference Clark2013).
Another recent group of influential theories in the “predictive brain” camp may not subscribe to an all-encompassing role for predictive mechanisms – but nonetheless, ascribe a critical role to prediction. Specifically, these approaches stress that cognition greatly relies on prospection (or future-oriented mental time travel; Suddendorf & Corballis Reference Suddendorf and Corballis2007; Tulving Reference Tulving1984) – the ability to deliberately create explicit representations of future events, and use these representations to guide behavior. Like PP theory, research on prospection has been highly influential across numerous domains of investigation. Within memory research, research on prospection argues that the function of declarative memory is in enabling simulation of future events (e.g., Schacter et al. Reference Schacter, Addis and Buckner2007). Within the Reinforcement Learning (RL) literature, research on prospection investigates how organisms often make decisions by relying on so-called model-based algorithms (e.g., Daw et al. Reference Daw, Gershman, Seymour, Dayan and Dolan2011) that explicitly simulate future outcomes (e.g., Redish Reference Redish2016). Within comparative psychology, research on prospection suggests that the evolutionary success of humansFootnote 1 stems from our capacity for future-thought (e.g., Suddendorf Reference Suddendorf2013).
It is commonly held that humans’ advanced ability for prospection must rely on similarly advanced representational capacities. Broadly speaking, these representational abilities are referred to as the ability for abstraction or as abstract thought – and are contrasted with more concrete thought. This distinction between relatively abstract and concrete mental representation has been integral to theories of the predictive brain. For example, the literature on mental time travel (e.g., Schacter et al. Reference Schacter, Addis and Buckner2007) argues that we are able to imagine and predict future events by relying on the abstract representational capacities afforded by declarative memory (specifically, episodic memoryFootnote 2) as opposed to procedural/non-declarative memory. In the RL literature, the concrete-abstract dimension is reflected in the distinction between model-free learning that relies on relatively simple associations (e.g., Schultz et al. Reference Schultz, Dayan and Montague1997), and model-based processing that relies on hierarchical, structured cognitive models (Tolman Reference Tolman1948) of potentially complex state-spaces (e.g., Daw & Dayan Reference Daw and Dayan2014). Within PP theory, higher-level, more abstract units in a representational hierarchy form predictions that inform and interact with lower-level units.
In light of the centrality of abstract mental representations in theories of the predictive brain, an in-depth account of representational abstractness seems essential for developing a comprehensive account of predictive cognition. The present article aims to provide such an account. Luckily, this work does not have to start from scratch. Decades of research on higher-order cognition have generated rich and intricate theoretical conceptualizations of the many different representational entities that give rise to abstract thought. We believe that, to date, this richness and intricacy has not been sufficiently tied to the newly evolving paradigm of the predictive brain and its neural substrates. Possibly, the lack of integration between cognitive science's rich past and its present hinders future development.
This dissociation can be traced to an earlier problem of insufficient integration across different theories of abstract cognition, and a lack of a joint vocabulary across different influential frameworks. In the current manuscript, we provide a unified conceptualization of abstraction, which we further integrate into the newly evolving framework of the predictive mind. In doing so, we aim to generate a comprehensive account of the representational bases of people's ability to traverse the here and now. Our account, which evolved from previous research on Construal Level Theory (e.g., Liberman & Trope Reference Liberman and Trope2008; Reference Liberman and Trope2014; Trope & Liberman Reference Trope and Liberman2010) aims to achieve two goals. First, we attempt to shed light on the diversity of abstract mental representation; second, we wish to integrate this diversity under a unified framework that could be tested, refined and revised through further experimentation.
We begin by suggesting a definition of the act of abstraction and describing how it relates to the different representational entities that have been designated as “abstract” in previous literature (sections 1–2). We then provide our account of the process of “mental travel” (sect. 3) and explicate how the different abstract representational entities expand humans’ mental travel ability (sect. 4). Finally, we discuss how the current framework can provide a theoretical basis for understanding the neural substrates of the predictive brain (sect. 5).
1. How do abstract mental entities emerge?
Despite the wide use of the term “abstraction,” there have been few attempts to provide a definition of this fundamental construct. We begin by providing a definition that will serve as the basis for our subsequent analysis of abstract mental representations and their role in prediction.
1. Our definition focuses on abstraction as a phenomenon that pertains to mental statesFootnote 3 (i.e., beliefs, desires, intentions). These mental states are “directed at” a (physical or mental) object, and have conditions of satisfactionFootnote 4, Footnote 5 – for example, when a person believesFootnote 6 that the earth is round, then this belief can be satisfied when novel observations suggest that this is indeed the case; when a person desires ice cream, then this desire can be satisfied if she eats ice cream; when a person intends to file her taxes, then this intention can be satisfied once the taxes are filed. In other words, satisfaction is akin to minimizing a discrepancy between one's internal state and the state of the world.Footnote 7
2. Whenever there exists a mental state that is satisfiable by object A, but not by object B, these objects can be said to exist as subjectively distinguishable objects in the mind of the perceiver.
3. We define the act of abstraction as the formation of a beliefFootnote 8 that two or more subjectively distinguishable objects satisfy a belief, a desire or an intention.Footnote 9
For example, forming a belief that both jogging and dieting would satisfy my intention to lose weight is an act of abstraction. Here are a few additional examples:
1. An infant desires milk. An instinct causes it to put various objects in its mouth. A perceptual pattern, which – from an external perspective – we call “mother,” repeatedly satisfies the infant's desire, regardless of whether mother is wearing a (tickling) sweater or a (smooth) T-shirt. According to our definition, once this substitutability is represented as a new entity in the infant's mental system (once it forms the belief that the distinguishable objects satisfy the desire) it has performed an act of abstraction. Such acts of abstraction are often discussed under the term generalization (e.g., Pearce Reference Pearce1987; Shepard Reference Shepard1987) and recognition, in research into basic learning processes.
2. A rat learns that pressing a lever in a red cage is substitutable with pressing a lever in other red cages (where it produces a reward), but not with pressing a lever in a blue cage. Such scenarios are discussed under the term situation/state recognition (e.g., Gershman et al. Reference Gershman, Blei and Niv2010; Redish et al. Reference Redish, Jensen, Johnson and Kurth-Nelson2007) in the RL literature.
3. A child is told by her parents “look at the kitty” whenever they encounter a cat, and forms the belief that various hairy, four-legged creatures are substitutable in satisfying the belief “(this) is a kitty.” Such processes are treated under the heading concept/category/word formation/learning/acquisition (e.g., Bloom & Markson Reference Bloom and Markson1998; Carey Reference Carey2009; Kruschke Reference Kruschke1992; Tenenbaum et al. Reference Tenenbaum, Kemp, Griffiths and Goodman2011).
4. An interviewer forms the implicit belief that a job applicant is predominantly similar to other candidates from the same ethnic background, rather than to other candidates with similar work experience. Such acts are discussed under the heading social stereotyping in the social-psychological literature (e.g., Banaji et al. Reference Banaji, Hardin and Rothman1993; Zarate & Smith Reference Zarate and Smith1990), and can be seen as a specific case of categorization or similarity judgment/analogical comparison (e.g., Gentner Reference Gentner1983; Medin et al. Reference Medin, Goldstone and Gentner1993).
All these examples highlight the commonality of different acts of abstraction. Just like the act of prediction, abstraction appears as an omnipresent regularity of the mind – and a defining property of cognition. However, it must be stressed that this description of abstraction pertains to the computational level of analysis (Marr Reference Marr1982), namely, it is an (abstract) description that does not correspond to one specific mechanism or neural hardware.Footnote 10
It is also important to note that forming the belief that two distinguishable objects satisfy a mental state does not mean that in order to perform an act of abstraction one must consider two particular exemplars. Although it is possible to perform an act of abstraction in a bottom-up manner, by having a thought like: “both object A and object B give rise to a sensation of tastiness,” it is clearly also possible to perform the act of abstraction by relying on the outputs of previous acts of abstraction, for example, by having a thought such as: “things that are made of chocolate give rise to a sensation of tastiness” and “things that are tasty are often made of chocolate.” Importantly, in all cases, abstraction deems (at least) two subjectively distinct objects as equivalent – as well as any additional objects that would satisfy the same mental state.
1.1. The outputs of the act of abstraction
As follows from the definition, the output of abstraction is a belief that two or more subjectively distinguishable objects satisfy a belief, a desire or an intention. The emergent output of this belief is that a person possesses a mental representation that allows them to associate between the objects and the mental state that they satisfy.
1. We refer to the set of distinguishable objects as the concreta.Footnote 11
2. We refer to the rule (or algorithm, function) that determines/picks-out the set of equivalent objects (the concreta) for a given mental state as the criterion of substitutabilityFootnote 12 (a notion related to a sense in the Fregian theory, i.e., Frege Reference Frege, Geach and Black1892/1952b).
As noted above, criteria of substitutability can take a form such as “things that are tasty are often made of chocolate”; this means that they can implement a theory (Murphy & Medin Reference Murphy and Medin1985). Theories allow us to generate predictions of (or imagine) future members of a set, rather than just assign a probability of class membership given a list of features (i.e., they implement a generative model, Ng & Jordan Reference Ng, Jordan, Dietterich, Becker and Ghahramani2002).
3. We refer to the newly generated mental object that instantiates (1) and stands for (2) as the abstractum.
4. Because cognition allows the outputs of abstraction to serve as concreta for additional acts of abstraction (Berwick et al. Reference Berwick, Friederici, Chomsky and Bolhuis2013), we can speak of mental representations as forming a continuum of abstractness. We define abstractness as a relative term that refers to the relation between two abstracta. Whenever we can say that abstractum X is part of the concreta of abstractum Y, we will say that abstractum Y is more abstract than abstractum X.
In our definition, if two objects are not distinguishable in any sense by the observer, then there is only a single object in mind, hence there is no abstraction. The requirement of distinguishability means that abstraction involves having at least two dimensionsFootnote 13 in mind: one dimension on which the stimuli differ, and another dimension on which they will be considered identical. Thus, when performing an act of abstraction, one makes a (conscious or non-conscious) decisionFootnote 14 on which dimension is central, and by doing so, one designates other dimensions as secondary or irrelevant in the current context (see Shapira, Liberman et al. Reference Shapira, Liberman, Trope and Rim2012, for a related definition of abstraction).
Because abstraction entails selecting/attending to one dimension and disregarding other dimensions that might be salient, many acts of abstraction likely rely on cognitive operations often referred to as “cognitive control” and “selective attention” (e.g., Botvinick et al. Reference Botvinick, Braver, Barch, Carter and Cohen2001; Macleod Reference Macleod1991). The degree to which cognitive control is needed for an act of abstraction depends on the relative salience of the (attended) dimension of substitutability versus the (ignored) dimension on which the concreta subjectively differ (see Deacon Reference Deacon1997, for a related idea).
Consider a psychology experiment wherein a child sees a picture of a person who has a mustache and wears a dress. The child can form the belief that the person is substitutable with other males (because of the mustache) or with other females (because of the dress). What made “clothing type” a relevant and salient dimension for the child? (see Fodor Reference Fodor1998; Goodman Reference Goodman and Goodman1972).Footnote 15 Clearly, the dimensions used for generating new criteria of substitutability come from our prior network of beliefs (Murphy & Medin Reference Murphy and Medin1985). However, explaining the formation of new beliefs merely based on prior beliefs leads to an infinite regress. The dimensions that form our criteria of substitutability must have been originally introduced into our mind at some point.
The question of where these dimensions come from is a major topic of investigation in the field of developmental psychology and concept formation. Broadly speaking, these dimensions can be acquired in three ways: they can be innate (e.g., infants may have an innate biological mechanisms that determine that the velocity and direction of an object is an important dimension to attend to in the newly-discovered world), shaped by personal experience and the gradual discovery of statistical regularities (e.g., a rat may discover that a specific auditory cue is a useful dimension along which to group outcomes), or transferred from other people (e.g., a child may learn from society that gender is a meaningful dimension along which to categorize people).
2. The diverse representational ontology that cognitive science should not ignore
As noted earlier, the ability to use abstracta as inputs for further acts of abstraction generates a continuum of abstractness. However, we argue (and in sect. 5, review evidence) that there are qualitative distinctions along this continuum. Performing the act of abstraction by relying on different types of inputs and different types of dimensions gives rise to qualitatively different types of abstracta.
2.1.1. Modality-specific features, objects, and relations.Footnote 16
The first steps in moving beyond a concrete representation can be traced to the discovery of identity between different perceptual features (e.g., color, loudness, a nose), and the formation of representational permanence of objects on the basis of perceptual pattern similarity (or “object permanence”; Piaget Reference Piaget1954). Object permanence relies on the formation of a belief that different sensory impressions that appear across different spatial and temporal contexts are equivalent, in the sense that they all pertain to the same object (e.g., “the image of my dog”). Likewise, perceptual patterns which are best described as relations (e.g., sound A is louder than sound B; objects A and B are similar in color; e.g., Martinho & Kacelnik Reference Martinho and Kacelnik2016) can perhaps be represented in a modality-specific manner, and be deemed equivalent across different instantiations. Such modality-specific abstracta probably often rely on different innate, hard-wired dimensions such as pitch and color (Baillargeon et al. Reference Baillargeon, Spelke and Wasserman1985; Carey Reference Carey2009).
2.1.2. Multimodal features, objects, and relationsFootnote 17
Once individuals generate some set of modality-specific abstracta, they may use them in further acts of abstraction. This is because percepts that cannot be grouped together based on perceptual pattern similarity may also be deemed as substitutable – whenever they are bound together by spatiotemporal contiguity. For example, when a toddler experiences the (modality-specific) sound of a dog and the (modality-specific) sight of the dog at the same time, it generates the multimodal abstractum of the dog. As another example, an animal that repeatedly hears a bell before it gets food can bind these two patterns together. Whereas the discovery of modality-specific abstracta often relies on innate substitutability criteria, multimodal abstracta are often acquired via personal experience.
Two types of multimodal representations warrant special consideration. First, a person might group together various different objects that share a temporal context (e.g., “the dog chasing the Frisbee in the park.”) The resulting abstractum can be called a mental episodeFootnote 18 (Tulving Reference Tulving1984). A second important class of multimodal representations are lemmas, which are the entities of our mental lexicon (Roelofs Reference Roelofs1992). These are the abstracta whose concreta include modality-specific representations of a feature/object/relation as well as the linguistic sign (written, spoken) that co-occurs with it. For example, the lemma of a banana will group images of bananas, taste experiences associated with bananas, alongside with the visual symbol “banana,” and the auditory pattern “ba-na-na.”
2.1.3. Categories
Multimodal abstracta can be used as building blocks for further acts of abstraction – even in situations where they cannot be grouped together based on spatiotemporal co-occurrence (Murphy & Medin Reference Murphy and Medin1985). This results in a category.Footnote 19 Categories arise, for example, when we form the belief that different multimodal abstracta such as bats and whales are all substitutable as mammals. Even the brightest child will probably be unable to acquire the category “mammal” (that includes both bats and whales) without being taught by someone. In other words, many categories probably rely on socially acquired dimensions and substitutability criteria that are subserved by lemmas.
Categories are often discussed within the context of perceptions and objects but clearly extend to actions and relations (see Bargh Reference Bargh2006; Hommel et al. Reference Hommel, Musseler, Aschersleben and Prinz2001; Weingarten et al. Reference Weingarten, Chen, McAdams, Yi, Hepler and Albarracin2016, for evidence of category-based organization of actions). For example, when we form the belief that different relations between people (e.g., “giving change to a beggar” and “giving a gift to a friend”) are substitutable acts of “giving” we have generated a categorical representation of action (Semin & Fiedler Reference Semin and Fiedler1991; Vallacher & Wegner Reference Vallacher and Wegner1987; see Mahler Reference Mahler1933 for initial research on substitutability in action categories).
An especially important type of category is one whose concreta are intangible entities. As noted, basic acts of abstraction involve transforming a percept of a particular spatiotemporal event into a modality-specific object (e.g., the image of my bicycle); forming a multimodal representation is another step (my bicycle), which can be followed by the formation of a category which includes different multimodal objects (the category “methods of transportation”). However, even non-particular (i.e., “universal”) objects (“methods of transportation”) and relations (e.g., “behind”) can refer to entities whose particular instantiations are detected by the senses. Yet, categorical abstracta can pertain to objects that are undetected by our senses, for example, “energy,” and “future.” These objects are often referred to as “abstract objects”; this terminology, however, is confusing, because, as we explained above, seemingly “concrete objects” such as “bicycle” are also the product of abstraction.
Whenever two or more intangible entities are believed to satisfy a mental state, we will refer to the resulting abstractum as an intangible abstractum. The big question surrounding intangible abstracta is how intangible objects, features, and relations take their place in the mind in the first place; this ability has been suggested as the crucial point of divergence between humans and non-human animals (Penn et al. Reference Penn, Holyoak and Povinelli2008). The question of how intangible features are initially “perceived” is beyond the scope of our current discussion. However, broadly speaking, it seems safe to assume that although some intangible dimensions may have an innate basis, or may be emergent properties discovered via personal experience (using socially-unmediated statistical learning mechanisms), many intangible dimensions (e.g., using zodiac signs as a criterion according to which to predict romantic success) are likely transmitted from mind to mind via the use of lemmas and words and as such rely on the pre-existence of multimodal abstracta (i.e., they are socially acquired).
Moreover, it should be noted that the discovery and use of many intangible abstracta are likely especially difficult, in that it requires us to disregard dimensions of equivalence that are typically salient and important. For example, the idea that hateful rhetoric, as well as peaceful rhetoric, may be deemed as identical – in that they are protected under the principle of “free speech” – requires that we ignore dimensions that are typically deep-seated in our mind (the malevolence/benevolence of people).
2.2. Forming complex mental structures
We have broadly outlined how abstraction generates modality-specific abstracta, how these modality-specific abstracta can serve as the concreta of multimodal abstracta – that, in turn typically serve as the concreta for categories. By performing recursive acts of abstraction, humans organize these representations within larger-scale structures (e.g., Smith Reference Smith, Gilbert, Fiske and Lindzey1998), mostly discussed in the literature on semantic and episodic memory (e.g., Tulving Reference Tulving1986).
2.2.1. Network structure
The assumption that when we create an abstractum we retain the association between its constituent elements entails that our representations will gradually form a network/graph structure (e.g., Hintzman Reference Hintzman1986). Regularities in the world entail that the same abstracta will be formed repeatedly, which in turn implies that some abstracta will have a greater degree of association with each other than others. For example, if the abstractum “cake” is repeatedly categorized as “dessert” (rather than, “baked goods”) – and if the intention to eat dessert is repeatedly satisfied by cakes (rather than “chocolate”) – cake will become a prototypical concretum of dessert. Namely, it should be processed faster, will be more likely and more confidently retrieved when prompted with “dessert,” and will be judged as a better instantiation of this category than chocolate (Rosch & Mervis Reference Rosch and Mervis1975).
2.2.2. Hierarchical structure
Once we decide (or are told) that for some purpose, dogs and chimpanzees are substitutable, we can create (for example) the abstractum mammal. We can then continue and decide that mammals and fish are also substitutable (being vertebrates), and so forth. In doing so, we gradually create a hierarchical system of mental representations that are organized in a tree-like structure of many-to-one relations (e.g., Collins & Quillian Reference Collins and Quillian1969).
Likewise, in the domain of actions, when deciding, for example, that swimming and running are substitutable as a means to achieve the goal of exercising, and that exercising and eating healthy are substitutable as means to achieve the goal of maintaining health, one gradually builds action hierarchies which serve as the basis of directing and representing goal-directed behavior (Carver & Schier Reference Carver and Scheier2001; Cohen Reference Cohen2000; Vallacher & Wegner Reference Vallacher and Wegner1987).
2.2.3. Temporal structure
Earlier we defined episodes as a type of multimodal abstracta that bind abstracta based on close temporal contiguity. This may give the impression that episodic memory is a fragmented tapestry of discrete events. However, discrete episodes may also be bound together by spatiotemporal contiguity (e.g., Tulving Reference Tulving1985), thereby generating a hierarchical structure of multimodal abstracta that can extend across hours or even days. For example, episode A (going to the gym) and episode B (taking a shower) may be bound together into episode C (gym and shower), which may be bound together with later episode D, and so forth (see Corballis Reference Corballis2014 for a discussion of the recursive structure of episodic memory).
Episodes that occur at temporally distant contexts (i.e., do not share temporal contiguity) may also be grouped to generate a type of category often called a script (Schank & Abelson Reference Schank and Abelson1975). For example, various episodes of airport visits can be deemed as equivalent, generating an airport script. Scripts can likewise be organized within increasingly abstract action categories (“going to the airport,” “traveling to a different country,” “traveling”), as described earlier.
2.3. How can mental structures interact with other mental structures?
The final entity we posit in our ontology (and which has been widely discussed in linguistics, but rarely in psychology and neuroscience) is the predicator. Like the other types of abstracta we have discussed, predicators instantiate a rule that determines the set of entities that are equivalent in satisfying a particular mental state (the criterion of substitutability). For example, the predicator “red” defines a certain visual-processing property as the dimension along which stimuli are deemed as equivalent, ignoring dissimilarity on other dimensions (such as object identity; its concreta contain different objects like “red dog” and “red Corvette”). Crucially, in order for the abstractum “red” to be a predicator (rather than being the “ordinary” category “red”) it must call for the specification of a subset of its concreta – by taking another entity (a different abstractum; e.g., “dog,” “apple”) as an input argument. As such, predicators are representations that behave like functions.
The theoretical distinction between “regular” abstracta and predicators can be traced to Frege (Reference Frege, Geach and Black1892/1952a) who famously noted the difference between “saturated” abstract entities (or, in his terminology, “objects”) – that do not need other entities in order to be functional, and “unsaturated” abstract entities (or “concepts”) – that rely on input objects in order to function. The existence of predicators in our minds is believed to be reflected in (overt) language useFootnote 20 (e.g., Pinker Reference Pinker2007). For example, in order for verbs to be functional they require the specification of a noun phrase as an argument (e.g., “lost” is not a meaningful utterance – until you specify who lost what, such as “Cleveland lost the championship”).
The interdependence of a predicator and its argument means that the operation of predicators (or predication) is best described as an interaction; moreover, according to the view presented herein, this interaction is characterized by an asymmetrical relationship of the predicator and its argumentFootnote 21 – the “saturated” abstractum, which (prior to its encounter with the predicator) was perfectly fine with denoting a wide variety of concreta (e.g., “dog”), is forced by the predicator into denoting a more limited set (i.e., it no longer denotes any dog, but specifically, a “red dog”). Thus, predicators can be thought of as “concretization machines,” in that they modify their input argument in a specific manner.
This specification that predicators entail plays a crucial role in human cognition: it allows us to modify representations in a systematic, rule-based/algorithmic manner (Bogdan Reference Bogdan2009; Fodor & Pylyshyn Reference Fodor and Pylyshyn2014). In doing so, predicators enable a purported “language of thought” (Fodor Reference Fodor1975) – a platform upon which we use mental representations to systematically orchestrate the modification of mental representations.
The computational process whereby predicators modify their arguments has long posed a challenge to theories of cognition (Fodor & Pylyshyn Reference Fodor and Pylyshyn2014). Most notably, theories of concepts that assume that the act of abstraction entails grouping together sets of particular exemplars (i.e., “exemplar theories”; e.g., Medin & Schaffer Reference Medin and Schaffer1978) or generating some statistical summary representation of these exemplars (i.e., “prototype theories”; Rosch & Mervis Reference Rosch and Mervis1975) may struggleFootnote 22 to explain how predication can generate previously never-encountered and non-typical objects (e.g., “a smoking caterpillar”; see Fodor & Pylyshyn Reference Fodor and Pylyshyn2014 for a discussion). In contrast to exemplar and prototype theories, we argued that abstraction requires applying a criterion of substitutability that can serve as a generative model for finding instantiations that will satisfy a particular mental state, whether previously encountered or not. According to such a view, predication might be best seen as a type of mental algebra that entails applying the criterion of substitutability of the predicator on top of the criteria of substitutability of its argument (e.g., “tasty cockroaches” are “insects that run on the ground” and also “make you feel good when you eat them”).Footnote 23
Some predicators have more than one argument (Marcus Reference Marcus2001). For example, the predicator lifted requires that you provide it with an argument for the agent (e.g., Dana) and for the patient (e.g., the dog). If given proper inputs, Lifted(Dana, the dog) will instantiate a specific relation between them (i.e., the dog is in the air, Dana is not). In this way, a predicator that operates upon more than one argument (a relational predicator, see Doumas et al. Reference Doumas, Hummel and Sandhofer2008; Gentner & Markman Reference Gentner and Markman1997; Halford et al. Reference Halford, Wilson and Phillips2010; Marcus et al. Reference Marcus, Vijayan, Rao and Vishton1999; Markman & Stilwell Reference Markman and Stilwell2001) modifies its arguments by specifying their relation to each other.
Of special importance are intangible relational predicators. They are distinct from representations of predicators such as Lifted, because, as their name suggests, they denote a relation that does not have specific perceivable instantiations (for example, “A thinks B” and “A is like B”). Relational predicators likely play an especially important role in higher-order cognition. For example, mastering the use of a system of logical relations allows for the emergence of the formal systems of reasoning (Evans Reference Evans2003; Goodwin & Johnson-Laird Reference Goodwin and Johnson-Laird2013; Kuczaj & Daly Reference Kuczaj and Daly1979). This ability is possibly subserved by a toolkit of intangible relational predicators such as if(A,B), cause(A,B), or(A,B), that designate the specific intangible deontology between the arguments.
Just like in the case of logical relations, the intangible relational predictor that determines that A symbolizes B may be of special importance. This Symbolize(A,B) predicator allows us to designate the specific intangible relation between a symbol (e.g., a written word) and its referent, B (i.e., “A can replace B in the mental world or in communication, but not in the real world”). This predicator might play a crucial role in humans’ ability to safely manipulate mental objects (now symbols), without worrying about manipulating real objects (DeLoache Reference DeLoache2004; Leslie Reference Leslie1987), a point to which we return later (see sect. 4). Relatedly, the ability to designate that a representation is explicitly about other mental representations (a “meta-representation,” Pylyshyn Reference Pylyshyn1978) may have crucial importance (Leslie Reference Leslie1987). For example, the predicator Believe(A,B) is impervious to who is believing, and what is believed – but designates the complex relation between the thing that is believed, the mind of the believer, and the world. As we discuss later, such predicators may be crucial for social cognition.
2.5. Interim summary: Sections 1–2
In sections 1–2 we proposed an ontology of abstract mental representations, based on extant theorizing and research. If cognitive scientists were right in assuming the existence of (at least some) of these entities, then any theory of the mind – theories of the predictive brain included – may need to integrate this representational diversity into their conceptualization. As we will argue in section 4, this diversity seems crucial in order to account for humans’ capacity to “traverse the here and now.” However, before going into this explication, we must first unpack what this capacity entails.
3. Interlude – What is Mental Travel?
Millions of years of evolution have ingrained within us mechanisms for acquiring and using information. For example, without resourcefulness and effort on our behalf, we are born with nerve endings that detect extreme heat, and the motor reflexes that tell us to distance ourselves from the source of heat. However, although our innate senses supply us with access to quite a lot of information, there is clearly much more knowledge in the world than what meets the eye (or other sense organs).
A proto-human would have had better chances to reproduce, if one day she would have woken up with the knowledge that: 10 kilometers up-north there is a waterfall; tomorrow, boars will visit it; she has a 40% chance of catching one; her friends will want to steal her prey. Such knowledge, however, oftentimes remains obscure, as it cannot be drawn from one's direct experience. Within Construal Level Theory (e.g., Liberman & Trope Reference Liberman and Trope2008; Reference Liberman and Trope2014; Trope & Liberman Reference Trope and Liberman2010), the aforementioned epistemic barriers are termed “dimensions of psychological distance.” Trope and Liberman posit that much of the ignorance we face in life is a result of spatial, temporal, and social distance, and because of uncertainty concerning the ontological status of things (hypotheticality) – and that the attempt to mitigate ignorance and uncertainty is an important force in human cognition.
The crucial importance of mitigating uncertainty is also a hallmark of PP theory. Most notably, Active Inference Theory (Friston Reference Friston2010) subsumes all cognitive activity under a single epistemic imperative – the attempt to reduce the surprise Footnote 24 that we expect to experience in our next interaction with reality (a process termed “free-energy”Footnote 25 minimization). In information-theoretic terms, expected surprise is the same as uncertainty; thus, Active Inference Theory suggests that every action an organism makes is an attempt to reduce uncertainty.
Several key insights emerge from this account. A first insight (which echoes discussions of the intentionality of the mind; Brentano Reference Brentano1874; for example, Searle Reference Searle1983; Velleman Reference Velleman1992) is that prior representations/expectations can become consistent with reality via two substitutable routes: organisms can either update these representations so that they cohere with sense data, or act in a manner that alters the world (and makes these predictions accurate). A second key insight is that the epistemic imperative can explain supposedly non-epistemic (i.e., utilitarian) phenomena such as mate-seeking and defensive behaviors. This is because organisms have prior representations/expectations that they will continue to live (e.g., will eat, will not be predated)Footnote 26; given these predictions, and given the substitutability of belief and action, the imperative “to be right” brings about a state wherein predictions are often fulfilled, and organisms survive.
Other theories in the predictive brain landscape (e.g., Suddendorf & Corballis Reference Suddendorf and Corballis2007) may not endorse the idea according to which the mitigation of ignorance and uncertainty is the goal of the mind, but nonetheless, stress that the ability to predict the future is what led to humans' evolutionary success. Simply stated, these theories argue that the person who was just about to go into a bear's cave, but suddenly gained a glimpse into the unfortunate outcome of such an action, would have outlived his future-myopic friend. Although our account is consistent with theories of prospection, we wish to highlight that futurity is just one of a number of epistemic barriers that humans may face. As described earlier, other dimensions of psychological distance (i.e., spatial distance, social distance, hypotheticality) also engender uncertainty.
Moreover, many of the unknowns of reality do not stem directly from divergence from one's own experience of the here and now. For example, one can observe lightning hit the ground without knowing what caused it, even though this occurrence is not distant in time, space, and social perspective. In other words, understanding the cause of a phenomenon (e.g., by developing a theory of electricity), which affords additional advantages (in the form of artifacts such as electrical lighting), is not reducible to traversing one of the psychological distance dimensions.
Finally, it should be stressed that a central aspect of the uncertainty faced by humans also extends to social reality. Traversing social distance may concern the mental states of individuals (i.e., perspective-taking/mentalizing/theory-of-mind; e.g., Baron-Cohen et al. Reference Baron-Cohen, Leslie and Frith1985; Heider Reference Heider1958; Jones & Davis Reference Jones and Davis1965; Kelley Reference Kelley1973; Malle & Holbrook Reference Malle and Holbrook2012; Trope Reference Trope1986;) or may concern social constructions (for example, in attempting to understand the distinction between homicide in the first vs. second degree). Moreover, even when the object of a person's inquiry does not appear to be social in nature (e.g., what is the shape of the earth), people might perceive other minds to be of much relevance, and wish to align themselves with the beliefs of others (Echterhoff et al. Reference Echterhoff, Higgins and Levine2009; Janis Reference Janis1972).
Indeed, research shows that (sometimes) the function of beliefs might not be to accurately represent reality – but rather to facilitate traversing social distance by creating unity of minds. Studies have shown that individuals’ desire to arrive at “shared reality” (Echterhoff et al. Reference Echterhoff, Higgins and Levine2009) can overshadow the motivation to attain veridical beliefs about the world (e.g., Asch Reference Asch and Guetzkow1951; Turner Reference Turner1991; for a meta-analysis, see Bond & Smith Reference Bond and Smith1996). For example, when a leader announces that the sun is an almighty God, she could refer to a factual state of the world (in which case it would be relevant to consider evidence for and against this proposition); she may, however, implicitly mean something like “let us create a social group that would be united by the belief that the sun is an almighty god, leaving outside of our group anybody who does not believe so.” The generation of unifying myths (Campbell Reference Campbell1991/1959), and more broadly, the generation of institutional facts (Searle Reference Searle2010), have immense societal ramifications – they facilitate traversing social distance within groups of individuals and can thereby form the basis for creating large social structures such as tribes, religions, nations, and ideologies (which, in and of themselves likely contributed to the evolutionary fitness of humans; Henrich Reference Henrich2015).
Thus, despite the critical importance of prospection, the future is just one of many epistemic and social barriers that humans try to traverse. Importantly, similarly to Active Inference Theory, we argue that the process of traversing temporal distance shares many commonalities with the attempt to traverse other epistemic barriers. Therefore, in our subsequent discussion of prospection, we refer to this process under the more general heading of mental travel.
3.1. How does mental travel occur?
In the preceding section, we echoed theories of the predictive brain and suggested that the need to traverse the unknown is a central functionality of our brain and cognition. How is the feat of mental travel performed, and what is the role of our reservoir of abstract mental entities in this process? In the remainder of the manuscript, we present our answer to this question, which is mostly focused on deliberative acts of mental travel carried out by human beings.
As noted earlier, the act of abstraction creates a rich storehouse of various mental representations. Theories of prospection often stress that our memories of specific episodes are the critical reservoir of information upon which we build simulations of future worlds (e.g., Suddendorf & Corballis Reference Suddendorf and Corballis2007). However, we wish to stress that all other types of abstract mental representations (e.g., categories, predicators, hierarchies, scripts) are just as important in mental travel. We shall refer to the reservoir of mental representations, which serve as the basis for the act of mental travel as the reservoir of source representations.
Importantly, when facing a specific problem of mental travel, people access their reservoir of source representations and generate a representation that models the specific problem at hand. For example, in order to decide whether to split the check in a restaurant on a first romantic date, we may access our knowledge regarding gender roles, the social background and the likely dispositions of the person in front of us, and recollect episodes of going on a first date. Based on this plethora of information we generate a target representation. In generating this representation, we make decisions regarding the relevant dimensions of the situation. For example, when deciding whether to split the check, I might consider the color of the clothes of my date to be an irrelevant dimension, but the gender and age of this person may seem to provide relevant information. Thus, the generation of the target representation constitutes an act of abstraction in and of itself.
However, generating a target representation is not the end of the story. Some mental processes must rely on this model of the situation and generate tenable predictions. A central tenet in many theories of prospection is the idea that people prospect by simulating future events (e.g., Barsalou Reference Barsalou2009; Bechara et al. Reference Bechara, Damasio, Damasio, Shinnick-Gallagher, Pitkanen, Shekhar and Cahill2003; Gilbert & Wilson Reference Gilbert and Wilson2007; Redish Reference Redish2013; Schacter et al. Reference Schacter, Addis and Buckner2007). For example, when John is considering whether he should call or text his date from the day before, he could imagine calling her: “The phone is ringing, no one is picking up. I am ending the call. Now I don't know whether she's busy or whether she's not interested. This is very stressful.” Having simulated this, John might decide that his best course of action is to send a text message on Facebook so that he could see whether the message was read.
The functionality of a simulation stems from the fact that the person running the simulation self-projects into it, that is, becomes an agent in the simulated situation. When the simulation is vivid and detailed (and thus similar to direct perception), reality-oriented processes (e.g., sensorimotor and affective/motivational processing, spatiotemporal associations, scripts) respond similarly to how they would react in real life (Gallese & Goldman Reference Gallese and Goldman1998; Gordon Reference Gordon1986; Moulton & Kosslyn Reference Moulton and Kosslyn2009). By “reading” the responses of the simulated self, one can generate new knowledge about the situation and decide how to act (e.g., Gallese & Goldman Reference Gallese and Goldman1998).
Although the dating scenario described above represents a case wherein simulation is used to reason about consequences in a relatively novel scenario, simulation is also important in situations wherein organisms learn action-outcome contingencies from repeated experience (i.e., Reinforcement Learning; [RL]; e.g., Redish Reference Redish2016; Skinner Reference Skinner1938; Tolman Reference Tolman1948). Research in this area distinguishes between “model-free” and “model-based” learning. In model-free learning, organisms choose their actions based on a representation that is simply an aggregation of previous hedonic outcomes associated with an action; as such, model-free behavior is impervious to sudden changes in the future (expected) utility of an action (for example, because of sudden devaluation of the rewards; Dickinson Reference Dickinson1985). In contrast to model-free learning, many organisms can also learn to represent action-outcome contingencies in a “cognitive” (Tolman Reference Tolman1948), “model-based” (Daw & Dayan Reference Daw and Dayan2014) manner – namely, as a structured representation of the different possible states of the world, and the possible transitions between these states. Such target representations enable deliberative prediction processes (i.e., prospection), in which actions are selected based on consideration of future utilities rather than force of habit (Niv et al. Reference Niv, Joel and Dayan2006).
Much research suggests that model-based decision-making relies on simulation processes, or, as it is often referred to in the RL context – a process of “vicarious trial and error” (Redish Reference Redish2016; Tolman Reference Tolman1948). During vicarious trial and error, organisms mentally test-out the different alternatives outlined in their model, and “read” the rewards and costs from the simulated scenario in order to choose the most favorable action. Indeed, single-cell recording studies have conclusively shown that when trying to choose between two arms in a T-maze, rats activate place cells that correspond to different routes they may take, thereby “mentally traveling” through the maze while sitting still (Amemiya & Redish Reference Amemiya and Redish2016; Johnson & Redish Reference Johnson and Redish2007). Moreover, once the rat “arrives” at its goal, the simulation activates reward circuitry, allowing the rat to evaluate the action (van der Meer et al. Reference van der Meer, Johnson, Schmitzer-Torbert and Redish2010). Such findings provide compelling evidence for the importance of mental simulation in decision-making.
Despite the emphasis in the literature on the process of prospection via simulation (e.g., Barsalou Reference Barsalou2009; Schacter et al. Reference Schacter, Addis and Buckner2007), we contend that simulation is not the only route by which people can traverse the unknown; rather, people can also use theory-based inference; namely, “mentally travel” by relying on analogical reasoning (e.g., Gentner & Medina Reference Gentner and Medina1998; Hummel & Holyoak Reference Hummel and Holyoak1997; Reeves & Weisberg Reference Reeves and Weisberg1994) and on deduction (e.g., Evans Reference Evans2003; Goodwin & Johnson-Laird Reference Goodwin and Johnson-Laird2013; Kuczaj & Daly Reference Kuczaj and Daly1979). For example, I can employ Sherlock-Holmes-like skills and reason that because my date studied at Vassar College, there is a good a chance that she has a negative view of traditional gender roles, which means that I should not offer to pay the check. Similarly, if I am to decide on whether to choose radio station X or Y, I do not necessarily need to simulate the expected outcomes. Instead, I can simply engage in proposition-based deduction (e.g., “station X plays classical music and station Y plays jazz; I am a person who prefers classical music; I should choose that which I prefer”). Unlike simulation, this form of inference does not require that one construct a representation that resembles sensory reality or experienced outcomes (e.g., hearing the music in my mind, feeling pleasure); therefore, theory-based inference is more likely to rely on mental representations of higher abstractness such as highly abstract categories, intangible abstracta, and predicators.
The distinction between theory-based inference and simulation has been a topic of much discussion within the literature on perspective-taking/mentalizing (i.e., the “theory-theory” vs. “simulation theory” debate; e.g., Apperly Reference Apperly2008; Gallese & Goldman Reference Gallese and Goldman1998; Gilead et al. Reference Gilead, Boccagno, Silverman, Hassin, Weber and Ochsner2016; Gordon Reference Gordon1986), which, in our terminology, addressed the process of traversing social distance. Yet, it has been almost entirely absent from the literature on prospection.
When would people rely on theory-based inference and when will they simulate? As defined herein, simulation is a process that entails the projection of the self into a specific spatiotemporal context. Thus, whenever such a projection is difficult or unhelpful, we should predict that individuals will instead rely on theory-based inference:
When attempting to predict an event occurring at a specific spatiotemporal context (e.g., “where should we go for dinner?”) simulation is a good idea (“I am eating this dish that I love. This is fun. Uh-oh, here comes the check.”) However, when predicting outcomes that are not reducible to a representative scenario (e.g., “what will happen to divorce rates in the UK in the next five years?”) simulation is largely irrelevant. One can simulate episodes contributing to a specific couple getting a divorce (e.g., an episode of infidelity) – yet such simulations do not really tell us what will happen on the national scale. Instead, it is better to use theory-based inference process such as proposition-based deduction (e.g., “the economy is in decline… economic hardship can increase marital distress”).
When outcomes are not determined by human agents, the usefulness of being able to “read” the response of the simulated agent goes away, and theory-based inference should be more likely. For example, when trying to predict whether a specific greyhound will win at the races, individuals will likely perform some calculation based on past statistics and betting odds (rather than simulating the greyhound running, the wind blowing in his ears). Moreover, when outcomes are determined by members of a conflict group, these individuals are sometimes dehumanized to such an extent that they are deemed as no different from objects or animals (e.g., Haslam & Loughnan Reference Haslam, Loughnan and Fiske2014); in such cases people will likewise often rely on abstract theory-based inferences (e.g., theories that explain all human behavior in terms of rewards and punishments) rather than on simulation.
When simulated others are dissimilar from the self, the utility of “reading” one's own responses to the simulated scenario diminishes, rendering the simulation less useful. For example, if I wish to figure out whether a close friend will be amused by a joke I can probably simulate my own response to the situation. In contrast, if I am interacting with a person who is socially distant (e.g., someone much younger) I might lack many of the specific parameters needed for the simulation and will be more likely to rely upon abstract knowledge and theory-based inference.
When events have some precedence in one's reservoir of specific episodic memories (e.g., “how successful will I be as a comedian?”) simulating these events can be useful (e.g., remembering times where your jokes fell flat). However, when no relevant episodes exist (e.g., “how successful will I be as a professional wrestler?”) one can only rely on more abstract knowledge concerning the self (e.g., “I am an out of shape academic, this is not the typical demographic you see in professional wrestling; there might be a reason for that”).
Despite these considerations, people often rely on simulation even when projection to a specific spatiotemporal context is less appropriate. For example, theory-based inference will probably be less likely when it requires the application of long, complex computations, and whenever cognitive resources are depleted (e.g., Stanovich & West Reference Stanovich and West2000). Furthermore, because theory-based inference often depends on socially-acquired knowledge (e.g., heuristics and stereotypes, rules of normative probabilistic or logical inference, math, and so on), individuals who lack such knowledge (e.g., because of their young age, illiteracy) are more likely to use simulation during mental travel.
3.3. Interim summary: Section 3
In section 3 we proposed a bird's-eye account of mental travel. This account is somewhat at odds with several aspects of current theorizing: (i) contrary to some theories of prediction, we highlight that futurity is just one of the many epistemic barriers humans overcome (ii); contrary to theories that suggest that prediction is purely an epistemic process, we highlight that in humans, mental travel is often guided by the motivation to arrive at a state of shared belief with other humans – regardless of the truthfulness of those beliefs; (iii) contrary to theories that put a focus on episodic memory and simulation as the primary conduits of prospection, we highlight that an act of mental travel draws on a plethora of “source representations,” some are relatively concrete and others are more abstract – and on both simulation and theory-based inference.
This account will now serve as the basis for our discussion of the role of different abstract mental representations in mental travel. In section 4, we explain why the diverse representational ontology we described in section 2 is indispensable in allowing mental travel. Finally, in section 5, we will review the empirical neuroscientific evidence that supports this account of representational diversity.
4. The members of our diverse representational ontology all help in meeting the challenges of mental travel
The challenge of mental travel stems from the fact that no target-representation will ever be identical to reality. However, what seems different on a concrete level could be seen as similar on a more abstract level. At the most rudimentary level, abstraction makes mental travel possible by introducing invariance among distinct experiences. Thus, although you cannot step twice into the same river, as the famous aphorism from Heraclitus goes, having a multimodal abstractum of water introduces stability into this endless variety, allowing the prediction that any time you put your feet in the river, it would feel wet. The organization of abstracta within networks allows us to predict which abstracta are likely to co-occur with a given abstractum (i.e., “this is a river; there must be fish around here.”) The organization of categories within hierarchical structures allows us to draw inferences concerning properties of novel objects, based on their place in a hierarchy (i.e., “this is a fish; therefore, it must be edible like other fish.”). The ability to organize episodes in temporal structures can allow us to re-play extended sequences and predict the conclusion of a possible course of action (“last time I ate fish, I ended up nauseous.”) Finally, the cross-temporal organization of abstracta in scripts let us know what to expect and how to behave in situations that happen repeatedly (“whenever I feel nauseous, eating rice helps me feel better.”)
In other words, the representational structures described earlier form the bridges that allow us to traverse uncertainty. In light of this, we believe that the link between abstraction and mental travel is fundamental to any consideration of these constructs; there is no mental travel without abstraction, and there is no need for abstraction but to support mental travel.
Beyond this fundamental claim, we argue that the different representational entities described earlier all play crucial roles in several (often-unrecognized) challenges associated with mental travel. We now turn to explicate this point.
4.1. The challenge of optimizing the accuracy/detail tradeoff of the target representation
In order for a target-representation to be functional, it must be accurate and detailed. When either condition is not met, the target representation is useless. For example, when trying to decide whether to go on a blind date, you may ponder, “what will my date look like?” If you tell yourself - “she will look like a human” you will be accurate, but this prediction will not provide you with any detail to guide your love life. In contrast, if you predict that - “she is 5’11 and has blonde hair,” you have generated a detailed prediction, which could be useful except that it is less likely to correspond to reality.
As noted, the act of abstraction generates hierarchies of representations at varying levels of abstractness (e.g., “mammal,” “human,” “young human male with blonde hair”). This hierarchical organization of mental representations facilitates the construction of target representations in a manner that optimizes the accuracy-detail tradeoff.Footnote 27 By assessing the amount of knowledge at hand, one can tune the degree to which her prediction will be detailed and specific. Based on this logic, Construal Level Theory predicts that whenever people contemplate events that are more psychologically distant – and therefore involve more uncertainty – the optimal point of the accuracy/detail tradeoff moves towards higher-level abstractness and less detail (Shapira et al. Reference Shapira, Liberman, Trope and Rim2012). Such logic is also consistent with the normative principle of Occam's razor,Footnote 28 according to which the best model is the one that introduces the least amount of (unsubstantiated) assumptions.Footnote 29
Much research based on this theory shows that cognition abides by these normative principles and that the degree of psychological distance from an occurrence is a critical factor in determining the degree of representational abstractness. To give just a few examples, research has shown that people use more inclusive categories (Kruger et al. Reference Kruger, Fiedler, Koch and Alves2014; Liberman et al. Reference Liberman, Sagristano and Trope2002) and more abstract language (Bhatia & Walasek Reference Bhatia and Walasek2016; Fujita et al. Reference Fujita, Trope, Liberman and Levin-Sagi2006; Liberman & Trope Reference Liberman and Trope1998; Snefjella & Kuperman Reference Snefjella and Kuperman2015) when imagining or making predictions of more distant situations (for reviews, see Liberman & Trope Reference Liberman and Trope2014; Trope & Liberman Reference Trope and Liberman2010).
Upon this view, the capacity to mentally travel to distant locations, planning farther into the future, imagining counterfactual worlds that are less similar to one's experiences, and taking the perspectives of more socially distal others – should co-occur with each other, as well as with an improved ability to form relatively more abstract mental representations. This co-occurrence should be evident across human evolution (e.g., the co-occurrence of greater interaction with distant others and the emergence of belief in imaginary moralizing deities; Norenzayan Reference Norenzayan2013), throughout ontogeny (e.g., the co-occurrence of delay of gratification and advanced reasoning ability; e.g., Rodriguez et al. Reference Rodriguez, Mischel and Shoda1989), as well as in covariance across individuals (e.g., co-morbidity between deficiencies in social perspective-taking and delay of gratification; e.g., Faja et al. Reference Faja, Murias, Beauchaine and Dawson2013).
4.2. The challenge of making the construction of target-representations computationally efficient
Hoarding memories in a massive library of source representation will not do any good if transforming these into a target representation takes too long. What are the representational capacities that allow for efficient construction of target representations?
Consider the example of two international travelers. John goes to an airfare search website and seeks out a flight to Bangkok by looking at the entire list of flights, and considering whether each of them suits him in terms of price, time, and number of layovers. Jane is likewise searching for a flight to Bangkok but filters the flights and looks only at prices and times within the acceptable range. Alhtough both travelers will eventually find a flight, Jane used the taxonomic organization of the flight database and should find a flight more efficiently. Likewise, our database of mental representations is organized as a hierarchical taxonomy, allowing us to efficiently and quickly retrieve task-relevant source-representations from memory, and use them in order to construct target-representations. This provides the computational infrastructure that helps us to handle our massive storehouse of mental representations in an efficient manner (Bower et al. Reference Bower, Clark, Lesgold and Winzenz1969; Cohen Reference Cohen2000).
If it is indeed the case that humans’ capacity to organize abstracta within complex hierarchical structures increases retrieval efficiency, two straight-forward predictions follow: First, as demands for efficiency increase (for example, as individuals accrue more and more knowledge in a specific domain) people should be more likely to represent information in a structured, hierarchical manner (rather than, for example, strictly based on temporal order). Second, it should be the case that hierarchical organization in memory will indeed facilitate fast retrieval from long-term memory. The degree to which we organize our knowledge based on temporal contiguity or category-based hierarchies can be readily gauged by examining the clustering of items during free recall (Kahana Reference Kahana1996). However, memory research has not yet examined how the efficiency of subsequent retrieval is affected by the organization of material in long-term memory, nor has it examined the relation between the amount of information encoded (e.g., because of expertise) and the nature of its organization. These fundamental issues concerning memory volume, efficiency, and organization await further research.
Finally, like hierarchies of abstracta, predicators also allow efficient use of mental representations in that they contain free variables that make them applicable across various domains. For example, encoding “hang” as a predicator allows one to generate both “hang a picture” and “hang a towel” without requiring the learning of each of the specific instantiations. Indeed, research on primate language acquisition has shown that chimpanzees that were taught how to use signs as predicators (e.g., when they learned the meaning of a sign for “give me” as an entity that exists separately from “give me a banana” and “give me an apple”) could process a multitude of different assertions (e.g., “give me juice,” “give me carrots”) – without the need be explicitly instructed on the meaning of each of these compositions, which would have been an intractable task (Savage-Rumbaugh et al. Reference Savage-Rumbaugh, Rumbaugh and Boysen1978).
4.3. The challenge of creating a richer repertoire of possible target-representations
If our repertoire of target representations would have been limited to previously-experienced events, mental travel would have been less useful. Theories of prospection stress that reshuffling of episodic memories allows us to generate many novel constructions, and thereby imagine previously unencountered events. According to our account, the reservoir of mental content we draw upon when constructing target-representations is not limited to episodic memories. Rather, humans use their full arsenal of source representations in order to enhance their ability to construe a multitude of alternative worlds. Below we outline three routes by which this is achieved.
4.3.1. Analogical transfer
One route by which a new target-representation can be generated is by importing some (but not all) aspects of existing source-representations into a new domain via analogical transfer (Gentner & Markman Reference Gentner and Markman1997). Consider the case of a newly-elected president, facing a crisis of increasing tension with a foreign country. Lacking experience in foreign policy, the president might seek an analogy to this political situation. She might invoke a playground script - “if you share your toys with other kids they will like you – I should offer concessions”. Thinking that “foreign politics are like a playground” suggests that it involves a relation of reciprocity wherein you give X now, and you shall receive Y later – but does not imply that it involves a sandbox.
It is widely argued that without relational predicators that instantiate relatively abstract relations (e.g., “containment,” “transitivity”; in our example - “reciprocity”) the capacity for systematic analogical thought and inference would have been limited (see Gentner Reference Gentner1983; Penn et al. Reference Penn, Holyoak and Povinelli2008) – and accordingly, our ability to construct novel target representations and traverse the unknown would have been less impressive.
4.3.2. Permutation
A second route by which abstraction broadens the scope of possible target-representations is by facilitating permutation (or “conceptual combination”; e.g., Fauconnier & Turner Reference Fauconnier and Turner2008). Consider the example of a cook contemplating a new dish. He can rely on an existing recipe and simulate changing the preparation method, ingredients, and so forth. We hope, one of the different combinations will yield a novel, tasty dish.
Again, it is his ability for systematic permutation of arguments within a predicator (e.g., fry(X); boil(Y)) that can greatly enhance the number of compositions he can create, and accordingly, the number of possible target-representations. Furthermore, the mere diversity in our representational capacities (i.e., the emergence of intangible abstracta, multimodal abstracta, categories) and the ensuing multitude of permutable objects very likely contributes to our capacity for representational generativity.
4.3.3. Cultural transfer
A third, critical route by which our diverse representational capacities increase the scope of possible target-representations is by facilitating language and cultural transfer (e.g., Cavalli-Sforza & Feldman Reference Cavalli-Sforza and Feldman1981; Henrich Reference Henrich2015), namely, by subserving the processes of symbolic interaction (Mead Reference Mead1934). Using language and other symbols (e.g., mathematical formula), we can efficiently adopt target-representations created by other minds. For example, my own experiences are unlikely to be helpful when generating a hypothesis regarding the outcome of a physics experiment; in such a case, I must rely on more abstract scripts that were gradually generated throughout human history, and on the cultural transfer of this knowledge.
It goes without saying that linguistic entities enable cultural transfer and that they are entities of a relatively high level of abstractness. In fact, language and abstract representations are so closely intertwined that they are sometimes thought of as synonymous. Language inevitably makes use of categories and lemmas that symbolize their referents. Furthermore, the generativity of language may rely upon the use of predicators (Bogdan Reference Bogdan2008).
4.4. The challenge of decoupling the target-representation from the real world
In order for a target-representation to be functional, it must not be confused with reality. In the terminology of Nichols and Stich (Reference Nichols and Stich2000), the segregation of the target-representation from the real world requires that the mental traveler creates a metaphorical “possible-worlds-box,” the contents of which are distinct from reality. For example, when I imagine being chased by a tiger, or when I read about a person running away from a tiger, I should not confuse these thoughts with the presence of an actual tiger – as the appropriate reaction is quite different.
Theories of embodied cognition (e.g., Barsalou Reference Barsalou2008) suggest that these different processes rely on the same representational bases, such that when we read about a tiger or imagine a tiger we activate the same perceptual and motor representations that become activated when we encounter a real tiger. Supporting this idea, much research shows that imagining visual stimuli activates the same neural regions involved in direct perception (e.g., Pearson et al. Reference Pearson, Naselaris, Holmes and Kosslyn2015; Redish Reference Redish2013); furthermore, merely reading verbs that pertain to sensorimotor states (e.g., “kick,” “toss”) activates brain regions that control hand or leg movement (e.g., Hauk et al. Reference Hauk, Johnsrude and Pulvermüller2004). Thus, it could have been easy to confuse simulation with reality, leading to maladaptive responses.
Non-psychotic adults do not typically confuse real experiences with imagined ones, although sometimes confusions do occur. For example, under some extreme conditions (e.g., sleep deprivation) normative individuals could be prone to hallucinations, which they deem to be reality (Babkoff et al. Reference Babkoff, Sing, Thorne, Genser and Hegge1989). Furthermore, memory research (e.g., Johnson & Raye Reference Johnson and Raye1981; Goff & Roediger Reference Goff and Roediger1998) has shown that concrete, vivid simulations can easily give rise to false memories, wherein an imagined event is mistakenly thought of as one that really occurred.
There are several possibilities regarding the manner by which our brains distinguish between simulation and reality. One possibility is that the “possible worlds box” is implemented via different modes of processing in the same neuronal populations.Footnote 30 An additional possibility proposed by our model is that in order not to confuse imagination with reality individuals rely on representations of higher abstractness, supposedly subserved by different neuronal populations (Gilead et al. Reference Gilead, Liberman and Maril2012; Reference Gilead, Liberman and Maril2013). One advantage of abstract representations is their lower correlation with their referent. For example, the abstractum animal refers to many instances of animals; when one activates this abstractum, this activation may diffuse across many representations (e.g., cats, birds, and dogs), weakly activating each one of them. Thinking of Danny's dog playing in the garden yesterday activates a smaller set of representations, resulting in a more vivid mental image that could be more readily misinterpreted with truly seeing a dog.
If it is indeed the case that more concrete representations are more readily perceived as being real, it should be expected that people will make strategic use of this situation whenever they want to modulate the perceived factuality of displaced reference statements (i.e., statements that pertain to events that do not occur in the here and now). For example, it could be predicted that a comparison between the language used by prosecutors, who try to convince the judge and the jury that a transgression occurred, and defense attorneys, who have the opposite aim, will reveal that the latter use more abstract language (e.g., “Mr. Johnson lied about the value of the car he sold Mr. Smith” vs. “the defendant did not commit any wrongdoing in his business transaction with the plaintiff”).
Finally, even when we accurately designate an event as being a simulation (e.g., something we heard about from others rather than experienced ourselves), we still need to designate whether this event is fictional or non-fictional. We regularly distinguish between biographies and a fairy-tales, documentaries, and non-documentaries, and (we hope) fake news and real news. It is likely that in order to make these distinctions we do not rely on different modes of neural processing or different neuronal populations; rather, we rely on the symbol-modifying capacities of predicators. For example, intangible relational predicators such as “not,” “if,” “imagine,” and “believe” may play a critical role in keeping representations of fact, fiction, and hypotheticals logically and functionally distinct.
4.5. The specific challenges of traversing social distance
As noted, one of the most important types of mental travel is the traversing of social distance – the attempt to understand the beliefs, desires, and intentions of others, and to arrive at states of joint beliefs, desires, and intentions. In fact, several theories of cognitive evolution contend that the need to traverse social distance was the selection pressure that drove the development of the abstract representational capacities of humans (Deacon Reference Deacon1997; Dunbar & Dunbar Reference Dunbar and Dunbar1998). We will now turn to explain how these representational capacities play a particularly important role in overcoming challenges associated with the attempt to traverse social distance.
4.5.1. Traversing the distance between two minds
Unlike any other dimensions of psychological distance, social distance can be traversed not only by using theory-based inference and/or simulation, but also by two straight-forward routes: (i) by simply asking the other person what is in her/his mind; (ii) by telling another person what is on your mind (thereby, making them to align themselves with the content of your mind). Clearly, representations that serve as the building blocks of symbolic interaction (i.e., lemmas, categories, predicators) are crucial for this type of communication-based alignment of minds (e.g., Austin Reference Austin1975).
Furthermore, a unique complexity arises in the meeting of two minds, in that these minds engage in simultaneous, interactive predictions. In order to predict what my competitor in a chess match will do, I must try to represent what she believes I believe she will do, which depends on what she believes I believe she will do, and so forth (Camerer et al. Reference Camerer, Ho and Chong2002). The ability to process such complex recursion may rely on the iterative use of predicators that designate mental states, such as believe and think.
Indeed, the idea that mental state predicators are constitutive for performance on tasks that require mentalizing is supported by much research in developmental psychology (see Milligan et al. Reference Milligan, Astington and Dack2007, for a meta-analysis). For example, research has provided evidence for a causal relation between mothers’ use of mental state verbs and children's subsequent performance on false-belief tasks (e.g., Ruffman et al. Reference Ruffman, Slade and Crowe2002).
4.5.2. Traversing the distance across an entire society
The ability to share mental states at larger scales (e.g., across nations, religions) provided the basis for large-scale cooperation in child rearing, agriculture, knowledge transfer, all of which contributed to humans’ evolutionary success (Henrich Reference Henrich2015). Consider, for example, the emergence of a system of laws and norms – that undoubtedly facilitated social coordination. The use of abstract categories of behavior (i.e., murder) that are organized within a hierarchy of higher-order laws, constitutions, virtues and values (e.g., “sanctity of life”), allows potential offenders (and judges) to infer which behavior is prohibited –despite the infinite concrete ways by which a person can be murdered (see Hahn & Chater Reference Hahn and Chater1998). Moreover, the emergence of effective norms and laws may require impressive recursive capacities. For example, it has been argued (e.g., Searle Reference Searle2010) that the ability to form a modern economy relies on intricate mentalizing: When a ruler declares that a note is valuable (i.e., a legal tender), citizens must believe that this ruler has the capacity to assign value to objects, that other people believe that it is indeed the case and that others believe that other people believe that is the case. As noted, this capacity may depend on the iterative use of mental state predicators.
Finally, in order for people to be willing to cooperate with non-kin individuals, it is useful to have credible displays of one's belonging and devotion to the group, a sort of “secret handshake” that allows entrance to an arbitrarily-established clique (Henrich Reference Henrich2015). The knowledge and practice of intricate myths and rituals (e.g., “thunder represents God's fury”) are one especially potent means to achieve these goals (Campbell Reference Campbell1968). Importantly, such myths typically pertain to intangible abstracta (e.g., god) that cannot be deduced empirically from one's sense experience and logic – and therefore cannot be known without admission into the congregation and its specific teachings.
If it is indeed the case that beliefs pertaining to intangible abstracta are especially potent in generating strong bonds between strangers, it could be predicted that social groups that define their beliefs in terms of intangible ideas should become more cohesive. Future work could examine this prediction by investigating whether individuals who discuss their group membership in terms of more intangible ideas (e.g., “I am a Republican because I believe in liberty”), rather than in terms of concrete policy/action preferences (e.g., “I am a Republican because I don't want the government to take away my guns”) exhibit greater solidarity with the group, and are more willing to engage in self-sacrificing behaviors in the name of the group.
4.6. Interim summary: Section 4
After describing an ontology of abstract representations in section 2, and describing the process of mental travel in section 3, in section 4 we have presented an account of how the diverse abstract representational entities that inhabit our mind all play a crucial role in mental travel. At this point we hope to have conveyed the message that mental travel likely relies on a plethora of diverse abstract representational entities – and that it may be insufficient to characterize the representational bases of mental travel by using relatively broad constructs such as “episodic memory,” “cognitive model,” or by relying on an undifferentiated, continuous hierarchy of mental representations of different levels of abstractness.
It is possible that alternative models of mental travel could account for the diverse competencies described herein, by assuming a more parsimonious representational toolkit (e.g., explaining the separation between reality and fiction without recourse to entities such as predicators). Such attempts could help refine or revise the model presented herein.
In the final part of this manuscript, we argue that if it is indeed the case that the process of mental travel builds upon these representational entities, understanding the neural bases of mental travel requires understanding the neural bases of these different types of mental representations.
5. Understanding the neural bases of the diverse representational architecture of the mind is essential to understanding the neural mechanisms of the predictive brain
Currently, functional neuroimaging is the central method by which cognitive neuroscientists study the neural bases of humans’ predictive cognition. As implied by its name, functional imaging is most often employed to examine cognitive functions (e.g., predicting the future, memory retrieval, multisensory integration). Such functions are progressive mental acts that operate upon some object, a mental representation.Footnote 31 Thus, when we observe the predictive brain at work, we must remember to ask ourselves – does our observation reflect the working of a type of cognitive function, or the generation or use of a representational type? (see Wood & Grafman Reference Wood and Grafman2003 for a similar perspective).
In the final section, we apply our representation-focused, pluralistic perspective to the neuroscientific literature on predictive cognition. We begin by examining whether/how the neuroscientific literature coheres with our ontology of representational types, and then go on to demonstrate why a better understanding of the neural substrates of abstract mental representation may be crucial for research on the predictive brain.
5.1. Does neural evidence support the diverse representational ontology we have described?
In section 2, we provided an account of the different types of abstracta that correspond to different meanings ascribed to the term “abstract representation”: modality-specific abstracta, multimodal abstracta, categories, several complex structures, and predicators. In this section, we will examine whether there is neuroscientific evidence that these entities are indeed distinct from each other.
5.1.1. Modality-specific abstracta
The existence of neural mechanisms specifically tuned to process modality-specific perceptual patterns has been demonstrated conclusively. Most notably, Hubel and Wiesel (Reference Hubel and Wiesel1962; Reference Hubel and Wiesel1968) used single-cell recordings and showed that early visual processing operates in a hierarchical manner: neurons on the retina and the lateral geniculate nucleus respond to light at specific points of the physical world (Kuffler Reference Kuffler1953); their projections converge to early visual cortex “simple-cells” that show selectivity to lines in a specific location and orientation; these “simple cells” converge to “complex-cells” which are orientation- but not location-specific (Hubel & Wiesel Reference Hubel and Wiesel1968).
This convergent architecture wherein neurons serve increasingly abstract modality-specific features is believed to continue until the generation of complex perceptual gestalts. Indeed, research has shown that along the inferior temporal cortex, cell assemblies converge to respond to gradually more invariant visual properties such as faces (e.g., Desimone et al. Reference Desimone, Albright, Gross and Bruce1984) and entire scenes (e.g., Epstein & Kanwisher Reference Epstein and Kanwisher1998). Despite these advances, we still do not have a complete computational account of how complex modality-specific objects, features, and relations (e.g., the image of a nose, the sound of a dog bark) are abstracted and stored. However, such an account may be inching closer, as scientists gain an increased theoretical understanding of the workings of deep (i.e., multi-layered) artificial neural networks (see LeCun et al. Reference LeCun, Bengio and Hinton2015).
A prevailing paradigm in neural network research is that experience-based representation learning occurs when a specific pattern of neuronal firing in a “lower-level” cell-assembly (i.e., pattern A) alters the strength of connections between the lower-level neurons and higher-order neurons (or a “deeper” layer) – in a way that increases the probability that a specific pattern in the deeper layer (i.e., pattern B) will recur in the futureFootnote 32; specifically, pattern B becomes more likely as activity at the lower level becomes more similar to pattern A. The higher-level layer contains fewer neuronsFootnote 33 and therefore generates a more compact representation of the information in the lower-level layer. This means that different patterns in the lower layer are substitutable with each other in giving rise to the same pattern in the higher layer. For example, different percepts of triangles (e.g., equilateral, isosceles) which correspond to different patterns in the lower-level layer, may generate the same pattern of activity in the higher-level layer.
Some types of neural network architectures (especially those used in the PP theory; e.g., the Helmholtz Machine; Hinton et al. Reference Hinton, Dayan, Frey and Neal1995) contain both bottom-up and top-down connections between the layers, such that activation of the higher-level layer (e.g., triangle) can re-generate the pattern of activity in the lower-level layer (e.g., an equilateral triangle). Endowing neural networks with such a “generative” capacity gives rise to many of the competencies observed in biological perception: Generative architectures allow the higher-level representation to predict future inputs, allow the network to “imagine” percepts, and retrieve modality-specific representations based on partial inputs (i.e., perform “pattern completion”; Hopfield Reference Hopfield1982; O'Reilly & McClelland Reference O'Reilly and McClelland1994). Such evidence suggests that the artificial neural network literature may indeed provide a good model of how modality-specific abstracta are generated and used during mental travel.
5.1.2. Multimodal abstracta
How are multimodal abstracta represented? According to the modality-specific, widely distributed processing hypothesis (e.g., Farah & McClelland Reference Farah and McClelland1991; Barsalou Reference Barsalou1999; Kiefer & Pulvermüller Reference Kiefer and Pulvermüller2012) which is inspired by the classic research into artificial neural networks (McClelland et al. Reference McClelland, Rumelhart and Research Group1986), multimodal representation are not subserved by specialized neural assemblies. Rather, the multimodal representation of, for example, “dog” is instantiated via reinstatement of patterns of activity across modality-specific cell assemblies in the visual, auditory, and somatosensory cortices. This hypothesis contradicts our model, in that it suggests that multimodal representations do not constitute a distinct representational type.
Despite the parsimony of this hypothesis, it is inconsistent with recent research. Whereas relatively posterior temporal regions subserve permanent (visual) representations, evidence from single-cell recordings performed on humans (e.g., Mormann et al. Reference Mormann, Kornblith, Quiroga, Kraskov, Cerf, Fried and Koch2008; Quiroga et al. Reference Quiroga, Reddy, Kreiman, Koch and Fried2005; Quiroga et al. Reference Quiroga, Kraskov, Koch and Fried2009) has shown that neurons further downstream in the Medial Temporal Lobe (MTL; i.e., in the hippocampus and in adjacent areas) selectively respond to both visual and verbal presentation of specific places and people, and thus may represent abstracted knowledge.
As noted earlier, whereas modality-specific abstracta can be grouped together based on perceptual pattern similarity, multimodal abstracta can be grouped together based on temporal contiguity. Much research has shown that the MTL, and especially the hippocampus, is involved in binding together perceptually distinct patterns based on the experience of their co-occurrence (e.g., Danker et al. Reference Danker, Tompary and Davachi2016; Davachi Reference Davachi2006; Eichenbaum et al. Reference Eichenbaum, Yonelinas and Ranganath2007; Gottlieb et al. Reference Gottlieb, Wong, de Chastelaine and Rugg2012; Sargolini et al. Reference Sargolini, Fyhn, Hafting, McNaughton, Witter, Moser and Moser2006). This research suggests that the hippocampal system may be critical in the encoding and retrieval of multimodal abstracta.Footnote 34
Compelling evidence against the “modality-specific, widely distributed processing hypothesis” comes from extant fMRI research. In a comprehensive meta-analysis of functional neuroimaging studies of semantic processing, Binder et al. (Reference Binder, Desai, Graves and Conant2009) concluded that “semantic” knowledge lies within a wide, distributed network of regions, which includes the posterior inferior parietal lobe and the angular gyrus, middle temporal gyrus, the fusiform and parahippocampal gyri in the MTL, ventral, and dorsomedial prefrontal cortex, posterior cingulate gyrus, and the Left Inferior Frontal Gyrus (LIFG). These findings were replicated in research that examined patterns of neural activity that were common to the presentation of specific objects across different modalities (e.g., Fairhall & Caramazza Reference Fairhall and Caramazza2013). The set of regions identified in these studies overlap with the set of brain regions referred to as the “Default-Mode Network” (DMN; Raichle et al. Reference Raichle, MacLeod, Snyder, Powers, Gusnard and Shulman2001). Importantly, with the exception of the fusiform and parahippocampal gyri (which are involved in the explicit mental imagery of concrete words; Gilead et al. Reference Gilead, Liberman and Maril2013; Wang et al. Reference Wang, Conder, Blitzer and Shinkareva2010), this large swath of the cortex has no overlap with brain areas involved in sensory and motor processing – which suggests that multimodal abstracta are distinct from modality-specific abstracta (Binder & Desai Reference Binder and Desai2011).
As noted above, the idea of layers of cell-assemblies that are abstracted away from their modality-specific instantiations is inconsistent with classic connectionist models (e.g., Farah & McClelland Reference Farah and McClelland1991). However, it is consistent with our view, as well as with more recent reincarnations of connectionist modeling (deep neural networks) that have shown that increasingly higher-level layers of cell assemblies can come to represent highly-abstract entities, that are coded in a (relatively) localized (rather than distributed) format (Bowers Reference Bowers2009) – and thus correspond to a-modal knowledge.
5.1.3. Categories
Multimodal objects that cannot be grouped based on spatial-temporal contiguity may nonetheless be deemed as substitutable, and generate categories. As noted, one does not need to encounter an image of a poodle with the associated word “mammal” to be able to categorize poodles as mammals (because poodles suckle milk and have hair). Categorical abstracta generate complex hierarchies of increasing abstractness, that often rely on the explicit linguistic transfer of socially-constructed criteria of substitutability (rather than on discovery via associative learning). However, is it the case that categories are neurally distinct from multimodal abstracta, as suggested by our model?
According to Rosch et al. (Reference Rosch, Mervis, Gray, Johnson and Boyesbraem1976), people categorize objects into so-called “subordinate level” concepts (e.g., poodle), “basic-level” (e.g., dog) and “superordinate level” concepts (e.g., mammal). Unlike superordinate concepts such as mammal, basic-level concepts such as dog are more likely to be discovered by associative learning (e.g., observing various different dogs while hearing the word “dog”) – and thus are more likely to evoke non-categorical representations (specifically, multimodal or modality-specific abstracta). Studies that have contrasted the processing of superordinate-level concepts such as mammal with basic-level concepts such as dog have found that processing the former involves greater activation within the Left Inferior Frontal Gyrus (LIFG) and the middle temporal gyrus (e.g., Raposo et al. Reference Raposo, Mendes and Marques2012).
As noted earlier, categories also exist in the domain of goal-directed action. For example, the goal to “put on running shoes” is subordinate to the goal “going jogging,” which is in turn subordinate to the goal of “maintaining health.” Research into the functional architecture of the lateral prefrontal cortex (e.g., Badre & D'Esposito Reference Badre and D'Esposito2007; Reference Badre and D'Esposito2009; Badre et al. Reference Badre, Kayser and D'Esposito2010) suggests that it is organized according to a hierarchy of abstractness wherein more anterior and inferior lateral frontal regions (e.g., LIFG) code more abstract, superordinate actions.
Furthermore, as noted, in contrast to modality-specific and multimodal abstracta, categories can refer to intangible entities. Therefore, another way to investigate whether categorical abstracta rely on a distinct neural population is by examining the neural substrates of processing intangible (vs. tangible) concepts. A comprehensive meta-analysis on intangible language processing (Wang et al. Reference Wang, Conder, Blitzer and Shinkareva2010) has shown that processing intangible words (e.g., justice, energy) as compared with processing of concrete words (dog, door) activates the LIFG and the anterior middle temporal gyrus.
Another way to investigate the neural substrates of categories is to observe brain activity as participants engage in tasks in which they attempt to find relations between stimuli that do not rely on spatiotemporal contiguity or perceptual similarity. Such acts are required during abstract analogical reasoning. Again, research into analogical reasoning points toward left-lateralized anterior frontal cortex as critical for this type of cognitive processing (e.g., Bassok et al. Reference Bassok, Dunbar and Holyoak2012; Bunge et al. Reference Bunge, Wendelken, Badre and Wagner2004; Whitaker et al. Reference Whitaker, Vendetti, Wendelken and Bunge2018; see Hobieka et al. Reference Hobeika, Diard-Detoeuf, Garcin, Levy and Volle2016 for meta-analysis).
Thus, research suggests that processing categories typically recruits anterior left-lateralized frontal and potentially frontotemporal cortical regions, implicated in linguistic processing/interaction (e.g., Kanwisher Reference Kanwisher2010). This provides tentative evidence that, indeed, it may be warranted to consider categories as functionally and anatomically distinct from multimodal abstracta.
As noted, most categorical abstracta (e.g., superordinate level concepts like “mammal,” intangible concepts like “inflation rate”) cannot be discovered without symbolic interaction with other people. The fact that the processing of categorical abstracta seems to rely on anterolateral frontal and temporal areas that are also involved in linguistic processing may reflect an affinity between the primary route by which criteria of substitutability are acquired and the neural systems upon which their associated abstracta eventually rely. Future research could investigate this hypothesis further by delineating the neural bases of acts of abstraction that rely on innate, personally experienced spatiotemporal contingencies, and socially-mediated criteria of substitutability – and examine how these three routes relate to subsequent retrieval of abstracta.
The acquisition of categorical knowledge via symbolic interaction has not been widely addressed in computational models of neural processing. The prevailing paradigm in neural network technology relies on inductive learning that associates between numerous instances (e.g., images of dogs and cats) and “labels” (e.g., the word dog or cat). Although this approach has led to impressive technological successes, the acquisition of categories in humans (and the intelligent behavior this affords) often relies on “one-shot learning” of criteria of substitutability, transferred from one mind to another. We do not present children with pictures of mammals alongside with the label “mammal” in order to teach them about mammals – we supply them a definition that allows them to recognize mammals and even imagine new instances of mammals. Thus, despite the impressive successes of technologies that followed the connectionist tradition, future advances in artificial intelligence research may require a rapprochement between “symbolic” and deep neural network architectures (see Lake et al. Reference Lake, Ullman, Tenenbaum and Gershman2017 for a similar position).
5.1.4. Complex structures
As noted in section 2, theories of cognition have long suggested that representational entities bind together in an organized manner to form complex structures (i.e., temporal structures such as scripts and long-winding episodes, hierarchical taxonomies) that subserve mental travel. Given that our model views these structures as amalgams of abstracta, we probably should not expect to observe them as a distinct representational type – rather, as a product of functional connections between regions that subserve different types of abstracta described earlier, that is, as widely-distributed patterns of processing. Even so, according to our representational approach, investigators can still ask, for example, whether some of these complex structures predominantly rely on neural systems that subserve categories, multimodal abstracta, and modality-specific abstracta.
For example, one important question is how the neural representation of specific episodes differs from that of scripts. Whereas memory for particular episodes can contain information concerning specific perceptual details (e.g., the taste of the fish I ate), scripts represent information that is invariant across different particular situations (e.g., visits to different restaurants), and should rely less on modality-specific and multimodal abstracta. Indeed, research examining the neural representation of specific episodes versus general events (e.g., see Martinelli et al. Reference Martinelli, Sperduti and Piolino2013 for a meta-analysis) shows that the retrieval of specific episodes relies on visual areas (which subserve modality-specific abstracta) and on the MTL (which may subserve multimodal abstracta); in contrast, the retrieval of general event knowledge activates frontotemporal regions (which may subserve categorical abstracta). Importantly, contrary to some approaches in memory research, our perspective suggests that such findings should not be interpreted as representing a distinction between “semantic” and “episodic” autobiographical memory systems (e.g., Conway & Pleydell-Pearce Reference Conway and Pleydell-Pearce2000), but rather between systems that subserve categorical abstracta and multimodal/modality-specific abstracta.
Our pluralistic representational perspective can also inform attempts to reconcile findings concerning the neural substrates of semantic and episodic memory. Although facts (“semantic” memory) are often represented more abstractly than episodes, our model does not posit a one-to-one mapping between fact-knowledge and categories (or between event-knowledge and multimodal/modality-specific abstracta). For example, you can learn the whereabouts of the Empire State Building via symbolic interaction, when you are told that “the Empire State Building is in New York City”; in such a case, this fact will be encoded and represented by frontotemporal regions associated with categorical abstracta. However, the same fact can be discovered based on spatiotemporal contiguity experienced first-hand (e.g., repeatedly passing by the building when visiting NYC) or by repeatedly noting the unstructured association between the lemmas “New York” and “Empire State Building” in books and movies. Thus, our model suggests a partial dependence of semantic memory on neural substrates that are critical for episodic memory. In light of this, our model can explain why focal bilateral lesions to the MTL (critically associated with episodic memory) do not spare (nor do they obliterate) fact-knowledge, but rather cause partial anterograde and retrograde semantic amnesia (e.g., Lah & Miller Reference Lah and Miller2008; Stark et al. Reference Stark, Stark and Gordon2005; Tulving et al. Reference Tulving, Hayman and Macdonald1991).
Furthermore, our account predicts that research that attempts to localize the “semantic system” by asking participants to process words versus non-words (e.g., cloth vs. sworf) will activate the multimodal representations associated with lemmas, and therefore, should often evoke MTL activations (see, e.g., Montefinese Reference Montefinese2019, for evidence supporting this prediction). As such, our perspective helps make sense of supposedly contradictory findings in the literature showing that the processing of “semantic” word meanings and “episodic” memory largely overlap (Binder et al. Reference Binder, Desai, Graves and Conant2009).
5.1.5. Predicators
As noted, since Frege (Reference Frege, Geach and Black1892/1952a), the existence of so-called “unsaturated entities,” sometimes simply referred to as “concepts,” has been posited by philosophers of mind and language – as these were thought to underlie the ability of mental representations to serve as functions that modify other mental representations. In contrast to the standard psychological approach (and consistent with approaches in linguistics; e.g., Kratzer & Heim Reference Kratzer and Heim1998) our representational ontology posits that “unsaturated” entities (i.e., predicators) are distinct from “regular,” “saturated” categories. Notably, this conjecture concerning the distinction between predicators and other categories has received very little attention in cognitive science (see Pylkkänen et al. Reference Pylkkänen, Brennan and Bemis2011, for a discussion of this topic as an example for the disconnect between linguistic theory and neuroscience).
One way to examine the neural basis of predication is to look at the processing of verbs that differ in the number of arguments they require in order to be saturated (e.g., one-argument verbs such as “cringe,” vs. two- and three-argument verbs such as “teach” that requires a specification of who taught who and whom). A recurring finding from such studies (e.g., Thompson et al. Reference Thompson, Bonakdarpour, Fix, Blumenfeld, Parrish, Gitelman and Mesulam2007; see Williams et al. Reference Williams, Reddigari and Pylkkänen2017 for a discussion) is that the left angular gyrus increases in activity with increased demands for argument saturation – suggesting that this region may be crucial for predication. To the extent that the left angular gyrus indeed subserves predicators, our model predicts that it should be especially important in argument-manipulation processes such as those evident in logical deduction (e.g., application of predicators such as “if,” and “or”), mathematical reasoning (e.g., the application of operations such as “minus” and “plus”), and ToM reasoning tasks (which may involve the application of mental state predicators such as “believe” and “think”). Indeed, in all these domains, there is evidence that lesions to the left angular gyrus result in significant behavioral decrements (e.g., Dehaene et al. Reference Dehaene, Piazza, Pinel and Cohen2003; Eimontaite et al. Reference Eimontaite, Goel, Raymont, Krueger, Schindler and Grafman2018; Zimmerer et al. Reference Zimmerer, Varley, Deamer and Hinzen2019).
Thus, to summarize, the research reviewed herein provides compelling evidence for the distinction between modality-specific abstracta and multi-modal abstracta, and suggests a tentative model wherein the DMN, implicated in semantic cognition, may be parsed into (i) an MTL hub, that subserves multimodal abstracta; (ii) a left anterolateral frontotemporal hub, that subserves categorical abstracta; and (iii) a temporal-parietal hub, that subserves the unique class of “unsaturated” categories, namely, predicators. Much further work is needed in order to test, refine, or revise this neural model of mental representation and conceptual cognition; however, such an endeavor is essential in order to provide cognitive scientists with an accurate ontology of the representational entities that exist in our mind – and that subserve predictive cognition.
5.2. The neural bases of prediction
In this final section of the manuscript, we will apply our model to the neuroscientific research on prospection/self-projection, RL, and PP. In doing so, we highlight important issues that need to be addressed in future research.
5.2.1. Self-projection
Much research has investigated the neural bases of various types of goal-directed mental travel (often referred to as “self-projection”; Buckner & Carroll Reference Buckner and Carroll2007) – such as contemplating future events (e.g., Addis et al. Reference Addis, Wong and Schacter2007), imagining hypothetical scenarios (e.g., Hassabis et al. Reference Hassabis, Kumaran and Maguire2007), and taking the perspective of others (e.g., Frith & Frith Reference Frith and Frith1999). This research shows that it is appropriate to consider different types of mental travel as sharing an important common denominator. These various tasks all reveal neural activity localized to the medial prefrontal cortex, posterior cingulate cortex, and the angular gyrus – which are sub-components of the DMN (for meta-analyses see Gilead et al. Reference Gilead, Liberman and Maril2013; Spreng et al. Reference Spreng, Mar and Kim2009; for within-participants comparisons of mentalizing and prospection tasks see Spreng & Grady Reference Spreng and Grady2010; DuPre et al. Reference DuPre, Luh and Spreng2016).
As noted, activity in these regions of the DMN is also associated with episodic memory retrieval (Kim Reference Kim2016). In light of this, one of the most prominent theories of the DMN is that it is responsible for mental travel that occurs via simulations that rely on episodic memory (i.e., the “self-projection via episodic simulation” hypothesis; Buckner & Carroll Reference Buckner and Carroll2007). Based on this influential hypothesis, some studies have interpreted activation of the DMN as a neural marker for the occurrence of episodic simulation processes (e.g., Peters & Büchel Reference Peters and Büchel2010; Tamir & Mitchell Reference Tamir and Mitchell2011; Tamir et al. Reference Tamir, Bricker, Dodell-Feder and Mitchell2015).
However, as noted, recent research suggests that virtually all of the different components of DMN also subserve the representation of semantic (rather than episodic) knowledge (e.g., Binder et al. Reference Binder, Desai, Graves and Conant2009). In light of this, we suggest that the involvement of DMN regions in mental travel may be attributed to its role in subserving multimodal abstracta, categories and predicators. Whenever we see that the MTL regions of the DMN are involved in mental travel, this indeed may reflect the retrieval of particular episodes and a more vivid simulation process (Madore et al. Reference Madore, Szpunar, Addis and Schacter2016); however, when MTL activity is absent (and activity in the angular gyrus and anterolateral frontotemporal cortex is evident), we contend that that mental travel likely occurred via a “theory-based” inferential process, that relied on more abstract representations. Our account predicts that although episodic simulation can contribute to prospection, it is not essential. Indeed, this prediction is supported by research showing that individuals with extensive MTL damage have a preserved ability to reason about future events in a rational, normative manner (despite their deficiency in generating vivid simulations; e.g., De Luca et al. Reference De Luca, Benuzzi, Bertossi, Braghittoni, di Pellegrino and Ciaramelli2018; Kwan et al. Reference Kwan, Craver, Green, Myerson, Boyer and Rosenbaum2012).
Construal Level Theory argues that because people are more ignorant about occurrences that are more psychologically distant, increased distance entails reliance on representations of higher abstractness, as these pertain to a greater number of possible alternatives and reduce error. Based on this, it could be predicted that when people contemplate more distant situations, they should rely on higher-level, superordinate categories which might be subserved by the LIFG and anterior temporal lobe. Partly supporting this prediction, Packer and Cunningham (Reference Packer and Cunningham2009) have shown that thinking of the more distant future resulted in activation in the LIFG and anterior temporal lobe. Similarly, Tamir and Mitchell (Reference Tamir and Mitchell2010) and Majdandzic et al. (Reference Majdandzic, Amashaufer, Hummer, Windischberger and Lamm2016) have shown that activity in the LIFG increases as participants predict the beliefs of increasingly dissimilar others.
5.2.2. Reinforcement learning
As noted, it is widely held that there are two routes by which organisms can make decisions in the context of RL tasks: in habitual, “model-free” learning, the organism makes decisions using pre-computed values that were calculated based on the history of rewards associated with specific actions; in “model-based” learning, the organism predicts potential rewards by using a hierarchical mental representation that models the latent causal structure of events, and that allows deliberate prospection.
Much research provides compelling evidence that model-based RL indeed relies on simulation processes of the type discussed in the “self-projection” literature (e.g., Doll et al. Reference Doll, Duncan, Simon, Shohamy and Daw2015; Johnson & Redish Reference Johnson and Redish2007; Redish Reference Redish2016; van der Meer et al. Reference van der Meer, Johnson, Schmitzer-Torbert and Redish2010). However, as we argued in section 3, episodic simulation may not be the only route by which humans prospect. With the help of representational conduits such as categories, predicators, and scripts, we can form innumerable different models (in our terminology, target representations) upon which different types of theory-based inference processes (as well as simulations) can operate.
Such theory-based inferences may also be important in repetitive value-based decisions (of the type discussed in the RL literature). Consider the example of a person who has to decide each morning whether to drive to work through the city (which is often busy with traffic), or the turnpike (which is less crowded, but entails a fee). As she enters her car, she can generate a vivid simulation of driving through the city (e.g., seeing the traffic slowly inching forward; feeling stressed by the prospect of being late). Such a simulation will likely rely on activity in regions associated with episodic memory retrieval (e.g., hippocampus) navigation and mental imagery (e.g., parahippocampal gyrus), as well as in regions involved in affective valuation (e.g., amygdala, anterior ventral striatum, orbitofrontal cortex). However, this person can also make her decision based on an abstract, theory-based inference – that does not require her to generate a facsimile of reality (e.g., “this is summer so more people are on vacation, which means that the city might be less crowded; however, rent prices keep soaring, so people must have less disposable income to go on vacation; the city will be swamped.”) Such inference likely relies on regions that subserve the retrieval and systematic manipulation of categorical abstracta (i.e., left-lateralized frontotemporal regions). Thus, although research has conclusively shown that humans do rely on concrete simulation during model-based RL – this does not rule out the possibility that theory-based inferences may also support model-based decisions. The extent to which humans rely on such abstract inference processes in the context of value-based decisions remains an open question.
A fuller understanding of the abstract representational bases of cognition (and their diverse, hierarchical nature) is also crucial for future research on model-free learning (which do not rely on deliberation and prospection). RL studies have identified that what animals learn is dependent on the learning situation (e.g., responses that were acquired in a specific situation do not necessarily transfer to other situations). In light of this, as noted by Redish et al. (Reference Redish, Jensen, Johnson and Kurth-Nelson2007, p. 790), a major question faced by the decision-maker is: “not a decision-process question – Should I act or not? – but rather a cognitive question – Which situation am I in? … the recognition that the agent's current situation shares properties with previous (similar) situations.”
Our perspective suggests that the same exact environmental situation can be categorized/construed at different levels of abstractness. For example, a couple attending a basketball game can construe the situation as “a night out in town” (which may entail a craving for a nice glass of red wine) or “sitting on a stadium seat at Madison Square Garden” (which would evoke craving for a hot dog). Research within Construal Level Theory has demonstrated that increasing psychological distance from a situation makes people construe it more abstractly. Much behavioral work within this framework has highlighted how this regularity in the process of construal/situation-recognition can explain various behavioral outcomes in the domain of decision-making (e.g., variation in intertemporal discounting – Liberman & Trope Reference Liberman and Trope1998; Trope & Liberman Reference Trope and Liberman2000; self-control failures – Fujita et al. Reference Fujita, Trope, Liberman and Levin-Sagi2006; melioration – Pick-Alony et al. Reference Pick-Alony, Liberman and Trope2014; exploration–exploitation decisions – Halamish & Liberman Reference Halamish and Liberman2017; Yudkin et al. Reference Yudkin, Pick, Hur, Liberman and Trope2019).
The fact that the same situation can be categorized at different levels of abstractness means that it is important to distinguish between situation-recognition processes that are primarily based on modality-specific or multimodal properties (e.g., a basketball game is deemed similar to other crowded, loud gatherings) and categorical characteristics (e.g., a basketball game is deemed similar to other “competitive situations”). Our model predicts that the former will rely on retrieval of representations in the hippocampal network and sensory cortices, and the latter will rely on the left lateral frontotemporal cortex. Future research is warranted in order to examine these predictions.
Such considerations also give rise to novel, potentially important research questions. For example, we know that “model-free,” habitual behaviors can be triggered by situation-recognition processes that pertain to modality-specific abstracta (e.g., a red light in a Skinner box). However, can habitual responding be likewise activated by recognition processes that rely on categorical abstracta? For example, can model-free RL mechanisms be used to condition a person to light a cigarette whenever she (specifically) reads the works of French existential philosophers (but not of Russian Existentialists)?
5.2.3. Neurobiological models of predictive processing
We argued that abstract representations have a causal role in cognitive processing. Moreover, echoing the “symbol-processing view” (Newell & Simon Reference Newell and Simon1972), we suggested that some abstract mental representations (predicators) have the capacity to interact with other representations in a systematic manner, thereby giving rise to a sort of “language of thought.” Like language, such a system may be best modeled by its own unique principles (i.e., rather than being modeled by the same principles that explain visual perception and motor action).
The historic alternative to this view is the connectionist (or “subsymbolic”) perspective (McClelland et al. Reference McClelland, Rumelhart and Research Group1986). Connectionist models adopt a parsimonious architecture, wherein a simple set of computations suffice to explain all aspects of cognitive functioning – from basic perception to language. On this view, the unique characteristics of different types of abstract mental representations, and the unique functions they may afford, are epiphenomenal to understanding the algorithms of the mind. In recent years, developments of connectionist models (i.e., deep neural networks) have demonstrated the utility of this approach, by achieving remarkable success in solving real-world computational challenges.
Neurobiological process-models of PP have been inspired by neural network models (e.g., Hinton et al. Reference Hinton, Dayan, Frey and Neal1995; Hinton et al. Reference Hinton, Osindero and Teh2006) and likewise often adopted rather parsimonious mechanisms. Most notably, similar to connectionist models, a major strength of PP theory (and specifically, Active Inference Theory) is that it shows how the complexity of cognition can naturally arise from a canonical computation repeated across different layers of a single continuum of representational abstractness.
Contrary to accounts that seek a “neat” organization of cognition, we advanced the case of the so-called “scruffies” (see Schank & Abelson, in Clark Reference Clark2013; Marcus Reference Marcus2009), namely, those who believe that cognition has a “varied bag of tricks” (Clark Reference Clark2013). We do not deny the importance of subsymbolic processes and representations but rather endorse a pluralistic perspective according to which both subsymbolic and symbolic representations play crucial roles in cognition (see Griffiths et al. Reference Griffiths, Chater, Kemp, Perfors and Tenenbaum2010 for a similar pluralistic approach). Specifically, we argued that when we zoom-in into different layers in the hierarchy of mental representations, we reveal meaningful neurobiological distinctions (e.g., neural substrates) and functional distinctions (e.g., different roles in prediction) – whose explication and integration with theories of the predictive brain could further develop theory.
One area where such potential integration could be especially meaningful is in the domain of mental health. PP models of psychopathology have provided compelling accounts of mental illnesses as stemming from aberrant belief-updating dynamics between different layers of the representational hierarchy. Specifically, the theory points at dysfunctionalities in the weight given to update signals between higher- and lower-level layers, as giving rise to “false beliefs,” and phenomena such as psychosis and depression (e.g., Adams et al. Reference Adams, Stephan, Brown, Frith and Friston2013; Clark et al. Reference Clark, Watson and Friston2018). However, it is possible that some pathologies of belief formation can also be attributed to aberrations of the unique computations that occur within specific layers.
For example, if categories and predicators give rise to a “language of thought,” one could ask how the dynamics within such a highly-abstract system are related to illness. As illustrated by Asimov's (Reference Asimov1950) depiction of the robot Herbie, who went mad once he realized that he could not abide by some of his imperatives without breaking others, empirical research suggests that irreconcilable inconsistencies within one's system of abstract beliefs about the self (i.e., “cognitive dissonance”; Festinger Reference Festinger1962) can lead to emotional distress. When cast in terms of Bayesian belief dynamics, confidently believing two contradictory abstract propositions about the self (or “splitting”; Kernberg Reference Kernberg1975; e.g., “I am a bad person,” “I am a saint”) may generate instability in the top-down predictions of the self-evidencing architecture of the brain, giving rise to psychopathology (e.g., unstable self-worth; borderline personality disorder). As a response, more resilient individuals may construct a novel model that accounts for conflicting beliefs (e.g., “I am human, and humans are multi-dimensional”); once such a model is selected, it may “suppress” prediction errors and increase the stability of the system.
Importantly, one unique principle of the layer of language-like mental representations is that it is directly accessible via symbolic interaction (e.g., through self-talk, or via talking with others). Indeed, since Freud it has been argued that “speech therapy,” in which patients are given novel interpretations of their experiences can alleviate distress. Such symbolically-mediated belief-updating processes – that are posited to be fundamental to many aspects of human life – are currently not addressed within the PP literature. As such, we suggest that attempts to model the dynamics that occur within humans’ system of symbolic representations present a future challenge for PP models, and a critical test of their ability to provide a comprehensive account of psychological distress.
6. Concluding remarks
In recent years, cognitive scientists have increasingly taken to investigate the role of prediction as a fundamental process of cognition. It is widely held that humans’ advanced capacity for prospective thought is subserved by similarly advanced capacities for abstract mental representation. In light of this, in the current manuscript, we attempted to provide an explicit and integrative account of the abstract representational bases which allow humans to “transcend the here and now.”
Based on theorizing and research in philosophy and in the neural and cognitive sciences, in sections 1–2 we provided a bird's-eye view of the ontogeny and ontology of abstract mental representations. In line with influential theories in the predictive brain framework, we suggested that abstractions are built as a hierarchy, ranging from the highly concrete to the highly abstract. However, importantly, we argued that the different computational challenges associated with prediction give rise to important qualitative differences between different types of abstractions that exist along this hierarchy, generating meaningful diversity in the representational substrates of the mind.
Specifically, echoing views from philosophy, we suggested that the representational hierarchy can be parsed into three qualitatively distinct levels: modality-specific representations that are primarily instantiated on perceptual pattern similarity; multimodal representations that are primarily instantiated on spatiotemporal association; and categorical representations, that are primarily instantiated on social interaction. These representational elements serve as the building blocks for more complex structures discussed in cognitive psychology (i.e., episodes and scripts, networks and hierarchies), and are the inputs for mental representations that behave like functions, and have been discussed mainly in linguistics, namely, predicators. We offer this ontology as our best attempt at an “elemental table” of the mind – to be revised, extended, or replaced.
We provided two types of arguments to support our model: first, we explained how the different elements in this ontology are all needed in order to account for humans’ impressive predictive cognition; this argument may be contrasted by alternative theoretical models that explain how the capacity for mental travel can be explained by a different, perhaps simpler, ontology. Second, we examined how the neuroscientific evidence coheres with our account and highlighted future research that could further support or contrast our model.
If our conceptualization is somewhat right, then this means that theories of the predictive cognition have endorsed an overly simplistic picture of the representational architecture of the mind and that this simplicity may hinder the ability of these models to account for behavioral and neural phenomena. We highlighted several directions by which our diverse representational ontology could guide future research into the predictive brain (e.g., regarding the functionality of the DMN, the role of theory-based inference in RL, and the role of symbolic representations in PP models of psychiatric illness).
In conclusion, the evolving framework of the predictive brain offers an opportunity for greater integration across the cognitive sciences. Psychologists, neuroscientists, and philosophers have long been working on piecing together ideas concerning the representational bases of cognition. The current manuscript attempts to build a bridge between this rich history and the newly evolving framework of predictive cognition. It is our hope that this bridge will assist scientists in their future mental travels.
Acknowledgments
We wish to thank Greg Murphy, Gary Marcus, Dedre Gentner, Tal Eyal, Ran Hassin, Yoav Bar-Anan, Ruvi Dar, Yakir Levine, Jochen Weber, Rani Moran, Alexa Hubbard, Raphael Gerraty, Dave Kalkstein, Miri Perkas, Zvee Gilead, Talya Sadeh, Britt Hadar, Hadar Ram, David Kashtan, Chelsea Helion and the anonymous reviewers for immensely helpful feedback.
Target article
Above and beyond the concrete: The diverse representational substrates of the predictive brain
Related commentaries (31)
A challenge for predictive coding: Representational or experiential diversity?
A modern materialist approach to abstraction, concreteness, and explanation in cognition
Above and beyond the content: Feelings influence mental simulations
Abstracting abstraction in development and cognitive ability
Abstracting reward
Abstraction still holds its feet on the ground
Abstraction: An alternative neurocognitive account of recognition, prediction, and decision making
Abstractions, predictions, and speech sound representations
Are all distances created equal? Insights from developmental psychology
Catching the intangible: a role for emotion?
Cognitive representations and the predictive brain depend heavily on the environment
Dynamic hierarchical cognition: Music and language demand further types of abstracta
Experiences of liking versus ideas about liking
Is it always so? Unexpected visions
Language as a mental travel guide
Mind wandering as data augmentation: How mental travel supports abstraction
Neuronal codes for predictive processing in cortical layers
On the implications of object permanence: Microhistorical insights from Piaget's new theory
Other and other waters in the river: Autism and the futility of prediction
Prospection does not imply predictive processing
Representation and agency
Representation, abstraction, and simple-minded sophisticates
Scale-free architectures support representational diversity
Shared reality and abstraction: The social nature of predictive models
Simulation across representation: The interplay of schemas and simulation-based inference on different levels of abstraction
Simulation and the predictive brain
Structured event complexes are the primary representation in the human prefrontal cortex
Successful simulation requires bridging levels of abstraction
The productive mind: Creativity as a source of abstract mental representations
The role of sleep in the formation and updating of abstract mental representations
Touch me if you can: The intangible but grounded nature of abstract concepts
Author response
Above and beyond “Above and beyond the concrete”