Postdiction and the speed of consciousness

The strongest kind of evidence on the speed of consciousness comes from psychological effects called “postdictive effects” – in which the way one consciously perceives a stimulus presented at time t changes because of events occurring at t+1. Let us use visual masking as a simple example to illustrate this. In metacontrast masking, presenting a mask 70ms after a target – a 70ms stimulus-onset asynchrony (SOA) – can make the target invisible (Breitmeyer & Ogmen, Reference Breitmeyer and Ogmen2006). This is a postdictive effect. Had the target been processed consciously within 70ms, the mask would have arrived too late to make it invisible. So, this effect indicates that conscious perception takes at least 70ms (assuming that the target is processed equally fast irrespective of whether or not the mask is presented).

We can use psychological effects of this kind to identify a lower bound for the speed of consciousness. Emphasis on “lower bound.” The fact that the mask is ineffective when presented 130ms after stimulus onset, but effective at 70ms, does not show that conscious perception occurs within 130ms. Why? Because it does not show that no method could make the target invisible (or retrospectively change its subjective appearance) more than 130ms after its onset. It just shows that we cannot achieve this using a metacontrast mask aloneFootnote ⁴ . With this caveat in mind, the important point is that we can investigate a lower bound for the speed of consciousness using postdictive effects of this kind.

How far have psychologists pushed the lower bound of the speed of consciousness? Postdictive effects in the 50-200ms time range have been studied for decades (Eagleman & Sejnowski (Reference Eagleman and Sejnowski2000); Kolers & Von Grünau (Reference Kolers and Von Grünau1976); see Hogendoorn (Reference Hogendoorn2022); Shimojo (Reference Shimojo2014) for reviews). For example, in the Poggendorff illusion, a straight line interrupted by a shape (such as a rectangle) appears misaligned, even though the line segments are actually collinear. Sugita et al. (Reference Sugita, Hidaka and Teramoto2018) showed that presenting the rectangle up to 200ms after the line segments causes subjects to perceive the lines as misaligned (Figure 2A). Since presenting the rectangle 200ms after the lines determines the perceptual fate of the lines, conscious perception of the lines must have waited at least 200ms.

Figure 2.

Postdictive effects. A) Asynchronous presentation of the elements creating the Poggendorff illusion. Presenting a rectangle up to 200 milliseconds after collinear line segments changes the perceived collinearity of the lines. Source: adapted from Sugita, Hidaka, & Teramoto (Reference Sugita, Hidaka and Teramoto2018). B) Sequential Metacontrast paradigm (SQM). A Vernier (with an offset to the right) and an anti-Vernier (offset to the left) fuse into a neutral Vernier (no offset) when presented in rapid succession. This is true even when the Vernier and anti-Vernier are presented within a stream composed of neutral Verniers. In those conditions, presenting an anti-Vernier up to 450ms after the start of the stream postdictively changes perception of the entire stream, such that no Vernier or anti-Vernier is consciously perceived at all. Source: Adapted from Drissi-Daoudi et al. (Reference Drissi-Daoudi, Doerig and Herzog2019). C) A Vernier and anti-Vernier presented in succession normally fuse into a neutral Vernier. However, applying transcranial magnetic stimulation to early visual cortex up to 320ms after the presentation of the Vernier and anti-Vernier can postdictively change what participants consciously experience.

We can go further with “long-lasting” postdictive effects (see Herzog et al. (Reference Herzog, Drissi-Daoudi and Doerig2020) for a review). Many of the relevant experiments involve visual stimuli called “Vernier” – two vertically superimposed bars with horizontal offset (Figure 2B). These stimuli fuse remarkably well (Scharnowski et al., Reference Scharnowski, Hermens, Kammer, Öğmen and Herzog2007). When a Vernier is followed by an anti-Vernier (a Vernier with the opposite offset), the two stimuli fuse into a neutral Vernier – the offsets cancel one another. This effect is postdictive in nature. Integration also happens for Verniers presented in a moving stream of Verniers (Figure 2B). Notably, if a Vernier is presented at the start of the stream, subjects perceive its offset as transported along the stream even if all the following stimuli do not have an offset.

A striking discovery from Drissi-Daoudi, Doerig, and Herzog (Reference Drissi-Daoudi, Doerig and Herzog2019) is that presenting an anti-Vernier in the stream up to 450ms after the first Vernier can postdictively change the appearance of the entire stream (Figure 2B). Doing so brings discrimination performance for the Vernier offsets from 75% to chance level: if the first Vernier in the stream has an offset to the right, subjects who would otherwise be able to discriminate it at 75% accuracy become unable to do so when it is succeeded by a Vernier with the opposite offset 450ms later. This integration seems mandatory: it is not disrupted by saccades during the stream (Drissi-Daoudi et al., Reference Drissi-Daoudi, Ögmen, Herzog and Cicchini2020) or by the presentation of distractors (Vogelsang, Drissi-Daoudi, & Herzog, Reference Vogelsang, Drissi-Daoudi and Herzog2024). The other remarkable aspect of the sequential metacontrast paradigm (SQM) is that there is a discrete time window during which Verniers integrate. A Vernier presented at 100ms in the stream does not integrate with an anti-Vernier presented 550ms after the start of the stream (that is, 450ms after the Vernier). The window of integration closes about 450ms after the start of the stream, and the anti-Vernier is now outside of this discrete window.

Motion does not have to be involved to obtain long-lasting postdictive effects with Verniers. Scharnowski et al. (Reference Scharnowski, Hermens, Kammer, Öğmen and Herzog2007) discovered that Vernier dominance in Vernier fusion can be manipulated up to 350ms after the presentation of a Vernier by Transcranial Magnetic Stimulation (TMS) to the early visual cortex. That is, the way in which subjects perceive the fusion of two Verniers depends on whether or not a TMS pulse will be delivered 350ms later.

Several long-lasting postdictive effects involve attention. In a series of experiments Sergent et al. showed that attracting exogenous attention to a stimulus location up to 400ms after its offset can postdictively improve its discriminability and reported subjective visibility – a phenomenon called “retro-perception” (Sergent et al., Reference Sergent, Wyart, Babo-Rebelo, Cohen, Naccache and Tallon-Baudry2013; Thibault et al., Reference Thibault, Van Den Berg, Cavanagh and Sergent2016; Xia, Morimoto, & Noguchi, Reference Xia, Morimoto and Noguchi2016). Importantly, in a Gabor orientation discrimination task, Thibault et al. (Reference Thibault, Van Den Berg, Cavanagh and Sergent2016) found that the retro-cue does not affect the precision of the representations, but mainly changes the “guess” rate. This suggests that the retro-cue does not simply consolidate the representation of an already visible target, but instead postdictively determines the target’s conscious or unconscious fate. Other postdictive attentional effects include distortions of visual space by retrospective attentional cues (Ono et al., Reference Ono, Yamada, Takahashi, Sasaki and Ariga2023; Ono & Watanabe, Reference Ono and Watanabe2011). In the attentional repulsion effect, attending to a location makes a stimulus next to that location appear further away than it really is (Lawrence, Kulzhabayeva, & Pratt, Reference Lawrence, Kulzhabayeva and Pratt2020). Ono and Watanabe (Reference Ono and Watanabe2011) showed that the attentional repulsion effect can occur postdictively and found that it is preserved even when the attentional cue is presented up to 400ms after stimulus offset.

Postdictive effects also occur in other modalities. One of the most famous effects is the cutaneous rabbit illusion – a tactile illusion in which the perceived location of a tap on the skin is affected by the location of a subsequent tap. This postdictive effect is robust in the 300ms range (Flach & Haggard, Reference Flach and Haggard2006; Geldard & Sherrick, Reference Geldard and Sherrick1972; Kilgard & Merzenich, Reference Kilgard and Merzenich1995). One can also find postdiction across modalities (Stiles, Tanguay, & Shimojo, Reference Stiles, Tanguay and Shimojo2022), although effects have so far only been demonstrated under 200ms. For instance, the McGurk effect can be induced with an asynchrony between sound and movement, suggesting a window of 200ms for the integration of sound and visual information (Van Wassenhove, Grant, & Poeppel, Reference Van Wassenhove, Grant and Poeppel2007). As another example, Stiles et al. (Reference Stiles, Li, Levitan, Kamitani and Shimojo2018) have shown that sounds postdictively affect vision. In the audio-visual rabbit illusion, observers presented with a sequence of spatially separated flashes paired with sounds perceive an illusory flash located between the real flashes if a sound occurs between the flash-sound pairs. That is, the location of an illusory flash depends on whether a flash-sound pair will be presented 130ms after the onset of the first flash-sound pair.

We have reviewed long-lasting postdiction effects in the 350-400ms range. These effects suggest that conscious perception does not generally occur before 350ms after stimulus onset. If conscious perception occurred before that, manipulating post-stimulus processing would be too late to change it. The fact that the fate of a percept can still be changed in that time frame indicates that it was not already conscious. This means that consciousness is slow. It is preceded by a window of unconscious sensory integration that can last up to 450ms in the most extreme cases (Herzog et al. Reference Herzog, Drissi-Daoudi and Doerig2020). We now answer some objections to this interpretation.

What long-lasting postdiction shows

Let us address two objections. A first objection is that subjects might be phenomenally conscious of the real sequence of stimuli, but can only cognitively access the integrated sequence. That is, phenomenal consciousness overflows cognitive access. In the SQM, for instance, subjects are phenomenally conscious of the offset of the first Vernier, but by the time they start thinking about what they saw, this information is no longer available for report (this is similar to what Dennett (Reference Dennett1991) called an “Orwellian” account). Call this the “overflow gambit,” because the response says that phenomenal consciousness overflows cognitive access (Block, Reference Block1995, Reference Block2007).

There are two main problems with the overflow gambit. The first problem is that it is ad hoc. To see why, consider the fact that the worry overgeneralizes: one could say the same to deny that short-term postdictive effects like visual masking have an effect on phenomenal consciousness. Just like visual masking, long-lasting postdiction experiments include not only introspective reports but also forced-choice discrimination tasks. For example, in the SQM, participants cannot discriminate the orientation of the first Vernier when a Vernier with an opposite offset appears 300ms later in the stream (Drissi-Daoudi et al. Reference Drissi-Daoudi, Doerig and Herzog2019). This effect persists even when researchers explicitly inform participants about the paradigm and instruct them to focus on the first Vernier. The evidential standard used to assess perception in long-lasting postdictive effects matches the standard used to demonstrate the effectiveness of visual masking. This means that maintaining the overflow gambit requires non-question-begging reasons to claim that long-lasting postdictive effects involve phenomenal overflow while shorter postdictive effects like visual masking do not.

Admittedly, cases of phenomenal consciousness without cognitive access are plausible in the (near) absence of attention. And it is indeed, in those cases that phenomenal overflow is usually posited (Block, Reference Block2007; Lamme, Reference Lamme2003, Reference Lamme2010). But when no attentional manipulations are involved, and working memory is not otherwise overloaded, phenomenal overflow seems much less plausible. This is why proponents of the overflow thesis like Lamme (Reference Lamme, Metzinger and Windt2016) believe that there is no overflow in cases like visual masking:

The same is true in many of the effects reviewed above (Michel & Doerig, Reference Michel and Doerig2022). Participants do attend to the relevant stimuli in long-lasting postdictive effects. They have nothing else to do but to identify the relevant stimuli. Yet, they still fail to identify them and deny seeing them. Given this, we see no theory-independent reason to maintain that they nevertheless consciously experience the relevant stimuli. If one wants to maintain that there is inaccessible phenomenal consciousness in long-lasting postdiction when all the available indicators suggest unconscious processing instead, the burden of proof is on them.

The second problem with the overflow gambit is that one of its predictions is incompatible with the evidence, at least in the case of relatively fast postdictive effects. Cowan and Greenspahn (Reference Cowan and Greenspahn1995) conducted an experiment where subjects viewed an apparent motion display. In half of the trials, a target appeared to move continuously from left to right, though it was actually presented first on the left and then on the right with a 100ms delay. On the other half of the trials, the target was shown on the left twice, making it impossible for participants to predict whether apparent motion would occur. The experiment had two conditions. In one condition, Cowan and Greenspahn instructed subjects to respond as fast as possible when they saw the target moving through the midpoint of the display. In the other condition, subjects had to respond when they saw the target reach the endpoint. The overflow gambit predicts that subjects should respond faster in the endpoint condition. Indeed, according to this view, subjects should first perceive the target at the end point, and then perceive the apparent motion only after this. This is not what Cowan and Greenspahn (Reference Cowan and Greenspahn1995) found. Reaction times were the same in both conditions, suggesting that the whole perceptual episode was released at the same time (See White (Reference White2024) for an in-depth discussion). Further experiments of this kind could be carried out with long-lasting postdictive effects, but for now, there is at least one postdictive effect in which the overflow gambit is wrong.

A second possible objection to the view that consciousness is slow is that the lag between consciousness and reality should be noticeable. You might wonder how it is possible for consciousness to be that slow, and even think that you have good introspective data indicating otherwise. Take a sport like baseball, for instance. A ball thrown at 100mph (161 km/h) reaches the batter in about 375 milliseconds. If it took 350ms for the batter to construct a conscious percept of the ball, the batter would consciously perceive it far behind its actual location (Nijhawan, Reference Nijhawan2008). The batter would have to hit the ball when their conscious perception is as of the ball leaving the pitcher’s hand. But this obviously contradicts introspective evidence: the batter presumably feels like they are hitting the ball as the ball is approaching toward them. So, one might think that a lag of 350ms between action and conscious perception should be noticeable.

This objection confuses the properties of mental states and the properties represented by mental states. A mental state might represent that something is green without itself being green. This distinction applies to temporal properties as well. We need to distinguish between a mental state occurring at a time t, and a mental state representing events occurring at a time t (Dennett & Kinsbourne, Reference Dennett and Kinsbourne1992). Just because the batter’s experience represents the ball approaching at time t does not mean that the experience itself occurred at t. All that is needed to account for the feeling that one hits the ball at the time one sees the ball approaching is an experience that represents those two events occurring at the same time. The experience itself might occur at a later time. A temporal gap between an event and an experience of that event need not imply an experience that represents the temporal gap – an experience of the gap. This view – according to which the temporal character of experience is explained by temporal properties represented by mental states instead of being entirely inherited from the temporal relations between mental states – is increasingly gaining in popularity (Dennett & Kinsbourne, Reference Dennett and Kinsbourne1992; Droege, Reference Droege2022; Herzog, Kammer, & Scharnowski, Reference Herzog, Kammer and Scharnowski2016, Reference Herzog, Drissi-Daoudi and Doerig2020; Hogendoorn, Reference Hogendoorn2022; Lee, Reference Lee2014; Prosser, Reference Prosser2018; White, Reference White2023). For our purposes, its main advantage is that it can explain why we do not notice the lag between consciousness and reality despite the fact that consciousness is slow: our introspective access is limited to the contents represented in experience – which feature no temporal gap.

Before we continue, let us add two clarifications. First, we do not claim that perception in general is slow. Perception is fast, especially for superordinate categories (e.g., “animal” vs. “non-animal”). For instance, when presented with two pictures, only one of which is an animal, subjects can initiate a saccade to the correct image in less than 150ms (Kirchner & Thorpe, Reference Kirchner and Thorpe2006). In the same way, subjects can initiate saccades towards faces in just a 100ms (Crouzet, Kirchner, & Thorpe, Reference Crouzet, Kirchner and Thorpe2010). The fact that sports such as ping-pong and badminton are even possible is an existence proof that skilled actions can be triggered remarkably fast based on visual cues. But this does not imply that conscious perception is fast. Our claim is that those fast responses are likely triggered by unconscious sensory information. We survey empirical evidence for this claim in the next section.

Second clarification: above, we discussed the fact that experimental evidence from postdictive effects can only provide us with a lower bound for conscious perception. This does not imply that the lower bound is always the same across contexts. We do not claim that conscious perception always takes 450ms, as in the SQM. This might be an extreme case. Consciousness could be faster in other contexts and may even vary within a given visual scene depending on factors such as visual eccentricity or attention (Jovanovic & Mamassian, Reference Jovanovic and Mamassian2019, Reference Jovanovic and Mamassian2020a, Reference Jovanovic and Mamassian2020b). We discuss how the speed of consciousness might be somewhat flexible in the section “The emergence of reality monitoring.” Our claim is: the fact that visual consciousness is typically slow – and sometimes very slow – reveals something about the kinds of functions it might be involved in. The relative sluggishness of visual consciousness gives us a clue as to what visual consciousness might be for. In particular, it is incompatible with a role in immediate reactions to stimuli.

Qualitative differences in conscious control

What might those slower mental processes be? A fruitful perspective, grounded in both animal learning theory and reinforcement learning, is that different “controllers” can drive behavior (Daw & O’Doherty, Reference Daw and O’Doherty2014; Dolan & Dayan, Reference Dolan and Dayan2013; Hunt et al., Reference Hunt, Daw, Kaanders, MacIver, Mugan, Procyk, Redish, Russo, Scholl, Stachenfeld, Wilson and Kolling2021). A Pavlovian controller enables reactive responses to biologically significant events by associative learning (classical conditioning). These responses are rapid, stereotyped, and reflexive. A habitual controller enables more flexible learning than afforded by Pavlovian control, by reinforcing stimulus-response linkages that previously led to reward (obeying Thorndike’s “law of effect”; Thorndike, Reference Thorndike1911). However, the habitual (or “model-free”) controller is still relatively inflexible, in that if new information comes to light that makes it unwise to pursue a previously rewarded action, it is unable to make use of this information. As noted by LeDoux (Reference LeDoux2023), model-free processes are rigid because “they are controlled by the past” (p.163). This is famously shown in studies of devaluation – if a rat is trained that a lever gives it a food reward, but this food is then devalued (by feeding to satiety, or making it subjectively aversive), then the habitual controller will not “know” the food is now not wanted, and the lever may continue to be pressed (Adams & Dickinson, Reference Adams and Dickinson1981; Dickinson, Reference Dickinson1985; this over-expression of habitual responding is a canonical model of addiction, Everitt et al., Reference Everitt, Belin, Economidou, Pelloux, Dalley and Robbins2008).

Many of these simpler forms of learning and behavioral control may proceed unconsciously. A well-studied example is provided by fear conditioning. Studies in animal models have described subcortical pathways that elicit rapid (reflexive or Pavlovian) defensive responses to threatening stimuli (LeDoux & Pine, Reference LeDoux and Pine2016). In humans, similar pathways can be activated by subliminal stimuli (Morris, Öhman, & Dolan, Reference Morris, Öhman and Dolan1998; Whalen et al., Reference Whalen, Rauch, Etcoff, McInerney, Lee and Jenike1998) and may support affective blindsight (De Gelder et al., Reference De Gelder, Vroomen, Pourtois and Weiskrantz1999; Tamietto & De Gelder, Reference Tamietto and De Gelder2010), leading to the potential for nonconscious forms of threat learning (Hamm, Reference Hamm2003; Raio et al., Reference Raio, Carmel, Carrasco and Phelps2012; Taschereau-Dumouchel et al., Reference Taschereau-Dumouchel, Cortese, Chiba, Knotts, Kawato and Lau2018). Subliminal stimuli may also elicit positive motivational responses accompanied by heightened activation in the ventral striatum, a key target of midbrain dopaminergic projections (Pessiglione et al., Reference Pessiglione, Petrovic, Daunizeau, Palminteri, Dolan and Frith2008)Footnote ⁶ . Some of these simple forms of learning also do not even require a brain: associative learning is found in animals such as jellyfish (Bielecki et al., Reference Bielecki, Dam Nielsen, Nachman and Garm2023), learning can occur independently of inputs from the brain in spinalized rats (Grau, Barstow, & Joynes, Reference Grau, Barstow and Joynes1998; Liu et al., Reference Liu, Ferguson, Crown, Bopp, Miranda and Grau2005), and even trace conditioning can be achieved in decerebrate animals, such as decerebrate guinea pigs (Kotani, Kawahara, & Kirino, Reference Kotani, Kawahara and Kirino2003). Finally, a core psychological feature of habits is that they may proceed independently of conscious control processes (Wood & Rünger, Reference Wood and Rünger2016). In summary, “model-free” learning and behavioral control can occur unconsciously.

In contrast to Pavlovian and habitual control, model-based or goal-directed control chimes with our usual conception of what it means to consciously control an action. We know things about our environment, and can use this information to reach our goal. Briefly, the current state of the art in computational neuroscience proposes that the brain maintains and updates an internal model of the world, and uses this to rapidly compute how hypothetical actions in the world would or would not lead to reward (Doll, Simon, & Daw, Reference Doll, Simon and Daw2012; Hunt et al., Reference Hunt, Daw, Kaanders, MacIver, Mugan, Procyk, Redish, Russo, Scholl, Stachenfeld, Wilson and Kolling2021; Mattar & Lengyel, Reference Mattar and Lengyel2022). Model-based control is thought to depend on inferring transitions through (sensory) states. For instance, a plan for how to obtain coffee may include transitioning between my office, the street, and the coffee shop, prior to obtaining the coffee. Computing transitions between states is flexible – because if the route to my favorite coffee shop is blocked, I can plan around it (in the form of a different set of state transitions, such as walking around the block). There are many candidate algorithms for planning, which are outside the scope of this article. However, core to all planning algorithms is an internal model of the environment: an agent’s representation of how the environment is affected by the agent’s actions, allowing counterfactual computations of the form “what would happen if I do X?” (Battaglia, Hamrick, & Tenenbaum, Reference Battaglia, Hamrick and Tenenbaum2013; Craik, Reference Craik1943; Mattar & Lengyel, Reference Mattar and Lengyel2022; McNamee & Wolpert, Reference McNamee and Wolpert2019).

Notably, this “planning” process is distinct from the effortful thought characteristic of conscious deliberative planning (although this would also be an example of higher-level model-based control). Instead, the online “planning” that is achieved through rapid simulation of future sensory states is intuitive and (relatively) fast. A focus of current neuro-scientific research on online planning is on replay (or “preplay”; Foster, Reference Foster2017; Hunt et al., Reference Hunt, Daw, Kaanders, MacIver, Mugan, Procyk, Redish, Russo, Scholl, Stachenfeld, Wilson and Kolling2021; Mattar & Daw, Reference Mattar and Daw2018; Singer et al., Reference Singer, Carr, Karlsson and Frank2013). Here, neural representations of different points in an internal model of the environment (such as positions in a maze or elements of a connected graph) are sequentially activated prior to acting. Neural replay might support internal “searches” through possible future paths that the agent could take and has been documented in both rodents and humans (Dragoi & Tonegawa, Reference Dragoi and Tonegawa2011; Gupta et al., Reference Gupta, Van Der Meer, Touretzky and Redish2010; Liu et al., Reference Liu, Dolan, Kurth-Nelson and Behrens2019; Ólafsdóttir et al., Reference Ólafsdóttir, Barry, Saleem, Hassabis and Spiers2015) with a time lag of around 20-50ms between sequentially activated states (Kay et al., Reference Kay, Chung, Sosa, Schor, Karlsson, Larkin, Liu and Frank2020; Liu et al., Reference Liu, Mattar, Behrens, Daw and Dolan2021). The entire replayed sequence is compressed into a time window of <200ms, consistent with online planning taking up time – but not that much time (Hunt et al., Reference Hunt, Daw, Kaanders, MacIver, Mugan, Procyk, Redish, Russo, Scholl, Stachenfeld, Wilson and Kolling2021; Liu et al., Reference Liu, Dolan, Kurth-Nelson and Behrens2019). For instance, Kay et al. (Reference Kay, Chung, Sosa, Schor, Karlsson, Larkin, Liu and Frank2020) found neural activity in the rodent hippocampus encoding two possible future scenarios (two upcoming maze paths) in constant alternation at 8 Hz: one scenario per 125ms cycle. Importantly, this fast replay activity might also involve sensory representations when relevant (Doll et al., Reference Doll, Duncan, Simon, Shohamy and Daw2015; Kurth-Nelson et al., Reference Kurth-Nelson, Economides, Dolan and Dayan2016; Y. Liu et al., Reference Liu, Dolan, Kurth-Nelson and Behrens2019; Schwartenbeck et al., Reference Schwartenbeck, Baram, Liu, Mark, Muller, Dolan, Botvinick, Kurth-Nelson and Behrens2023; Wittkuhn et al., Reference Wittkuhn, Krippner, Koch and Schuck2024).

It is likely that the use of internal models to navigate external environments (and associated neural replay) occurs routinely in the background, in the absence of conscious awareness (see the “Reality monitoring” section below; Wittkuhn et al. Reference Wittkuhn, Krippner, Koch and Schuck2024). However, there is nevertheless a general assumption that switching from habitual to model-based control involves conscious experience of the relevant sensory information – an experience of what the internal model represents (Butlin, Reference Butlin2020; Frith, Reference Frith2021; Pennartz, Reference Pennartz2018). For instance, the global workspace model of consciousness proposes that consciousness is associated with the flexible usage of perceptual information in a global workspace, where perceptual states can be flexibly mapped onto a variety of possible actions (Baars, Reference Baars1988; Dehaene & Naccache, Reference Dehaene and Naccache2001). Some of the clearest evidence for this link comes from the process-dissociation procedure, in which model-based and habitual responses are placed in opposition to each other and their influences on subjects’ behavior measured (Jacoby, Reference Jacoby1991). In the classic version of this paradigm, subjects are provided with a prime stimulus (e.g., referee) and asked to complete a word stem (e.g., ref……) with any word that comes to mind, except the prime word. When the prime word is visible/conscious, then this is straightforward, and subjects are able to flexibly refrain from completing the stem with the prime, exhibiting goal-directed control (in which the goal is “don’t use the prime”). But when the prime is subliminal, the frequency of word stems completed by the prime is higher than expected by chance – showing inflexible, habitual responding (See also the “Perruchet effect” for a similar effect; Perruchet, Reference Perruchet1985).

Similar dissociations have been identified in blindsight – a phenomenon caused by lesions in primary visual cortex, in which patients can perform visual tasks in the absence of reported conscious visual experiences (Weiskrantz, Reference Weiskrantz2009). Persaud & Cowey (Reference Persaud and Cowey2008) presented blindsight patient GY with a grating stimulus in either the upper or lower quadrant of his visual field. Crucially, the required response was to say the opposite of its location (e.g., reporting “Down” if the stimulus was in the upper quadrant). GY had no trouble following exclusion instructions in his normal, sighted field, but tended to erroneously respond with the real location when the grating appeared in his blind field. Strikingly, this pattern of erroneous responding actually increased as the grating contrast was increased – suggesting that the impact of blindfield stimuli on behavior is mediated through inflexible, habitual responses rather than flexible model-based control. This same logic has been used to argue for a qualitative dissociation between unconscious and conscious usage of visual information in macaque monkeys (Ben-Haim et al., Reference Ben-Haim, Dal Monte, Fagan, Dunham, Hassin, Chang and Santos2021). In summary, there is growing evidence that switching to model-based control requires being conscious of the relevant sensory information (see also Tsushima, Sasaki, & Watanabe, Reference Tsushima, Sasaki and Watanabe2006).

Conversely, the online control of action can be guided by largely unconscious sensory information. A famous example of this is the distinction between dorsal and ventral visual streams (Goodale & Milner, Reference Goodale and Milner2018; Milner & Goodale, Reference Milner and Goodale2006, Reference Milner and Goodale1993). Damage to the ventral visual stream in patient DF led to impairments in conscious shape perception, but left the online guidance of action intact (Goodale et al., Reference Goodale, Milner, Jakobson and Carey1991). Similar dissociations between conscious perception and action guidance have been observed in healthy observers. For instance, in the hollow mask illusion, prior knowledge of faces creates a powerful visual illusion that a concave mask is a normal convex 3D face. And yet when observers are asked to “flick” targets on the surface of the mask, the flicking movements are directed at the real, not illusory locations of the targets (Króliczak et al., Reference Króliczak, Heard, Goodale and Gregory2006; for dissociations using other illusions, see Aglioti, DeSouza, & Goodale, Reference Aglioti, DeSouza and Goodale1995; Chen, Jayawardena, & Goodale, Reference Chen, Jayawardena and Goodale2015; Ganel, Tanzer, & Goodale, Reference Ganel, Tanzer and Goodale2008; Ozana & Ganel, Reference Ozana and Ganel2020). Dissociations have also been identified between online eye movements and explicit perceptual reports (e.g., Wong & Mack, Reference Wong and Mack1981; see Spering & Carrasco, Reference Spering and Carrasco2015 for a review). In general, eye movements provide a fast orienting system to stimuli that are yet to reach conscious perception (Crouzet et al., Reference Crouzet, Kirchner and Thorpe2010; Kirchner & Thorpe, Reference Kirchner and Thorpe2006). Finally, the existence of masked priming effects on behavior, discussed above, is a prima facie example of short-timescale unconscious influences on action (Vorberg et al., Reference Vorberg, Mattler, Heinecke, Schmidt and Schwarzbach2003).

As Goodale and Milner (Reference Goodale and Milner2013) write, “…the dorsal (action) stream works in real time and stores the required visuomotor coordinates only for a very brief period… The ventral (perception) stream, on the other hand, is designed to operate over a much longer time scale.” (p. 81–82). This longer time scale enables the ventral stream to create the mental furniture that allows us to think about the world and plan our actions.

Computational perspectives on the timescale for consciousness and control

As noted above, model-based planning requires an internal model of the world in which to plan. This can be a model retrieved from memory – for instance, I can “install” a model of my home office, and consider whether the desk I am thinking of buying would fit into that space. In humans, planning is often oriented towards goals that are relatively distant in the future: for instance, we take a shopping bag when leaving the house because we plan to pick up groceries (Suddendorf, Redshaw, & Bulley, Reference Suddendorf, Redshaw and Bulley2022). But it is arguably much more common (particularly in the evolutionary history of land animals) to need to plan in the here-and-now, for the immediate futureFootnote ⁷ . I see a predator moving behind some trees, and I need to rapidly – in a few tens of milliseconds – figure out an escape route (Figure 3B). It is easy to see that an agent with a large sensory horizon has an advantage here – because a larger (albeit still partially observable) state space is immediately available to it.

The problem is that the here-and-now is always changing – the predator is here now, but it wasn’t a few seconds ago. There is thus a tension between engaging in model-based simulation within a stable model of the world, and losing our dynamic coupling with a constantly changing environment. An attractive proposal is that the temporal window of integration of conscious experience provides this model of the world, allowing us to imagine and plan in the immediately remembered present – what the world was like in the past 300-500ms. This window of integration provides a stable platform for imagination and simulation, while also remaining sensitive to important changes in the environment. This architecture could be further finessed by making the window of integration itself sensitive to important environmental changes and/or to natural “boundaries” between events – a point we come back to in the next section (J. M. Zacks, Reference Zacks2020).

This view is compatible with (but does not assume) specific predictive processing views of brain function (Clark, Reference Clark2013; Hohwy, Reference Hohwy2013). Predictive processing holds that perception is realized by a process of inference on the causes of sensation, mediated by a continual interplay between background expectations and beliefs and current sensory inputs. When the internal model and the sensory input fail to match, a “prediction error” ensues, and our hypotheses are subtly altered to better approximate reality. Predictive processing accounts themselves do not distinguish between unconscious and conscious perception. However, extensions of this framework suggest that a key property of the sub-personal inferences leading to conscious perception is identifying a sufficiently reliable world model for use in the service of mental simulation and planning (Fleming, Reference Fleming2020; Hohwy, Reference Hohwy2013). Hohwy explicitly ties conscious perception to “active” aspects of inference (Parr, Pezzulo, & Friston, Reference Parr, Pezzulo and Friston2022), suggesting that the brain is engaged in a dynamic interplay in which a model of the world is selected and “fixed” for subsequent planning, before a new model is built, and so on. It is intriguing to note that the timescale of this proposed alternation between perceptual and active inference is similar to that identified for the window of conscious perception, around 300-500ms. Recent discoveries in animals also indicate that the neural correlates of current versus hypothetical experience can alternate within a theta cycle (Kay et al., Reference Kay, Chung, Sosa, Schor, Karlsson, Larkin, Liu and Frank2020).

In summary, in this section, we have argued that conscious vision is closely associated with the capacity for model-based planning, and that the timescale of conscious perception allows for the processes of (covert) active inference and mental simulation that are thought to subserve model-based behavior. In contrast, a wealth of evidence supports the view that online action control can be achieved unconsciously.

One important clarification before we close this section. In what follows, we will argue that the expanded sensory horizons characteristic of terrestrial animals underpin the utility of conscious perception and model-based control. This does not mean, however, that we are only conscious of objects that are at a sensory distance. Engaging in perceptual inferences over longer timescales becomes advantageous for larger sensory horizons. But once slower conscious perception is acquired, it can incorporate any sensory input into its world model – including inputs from the agent’s own body, and peripersonal space. On this view, while online action control may be achieved unconsciously, we can still be conscious of our unconsciously-guided actions.

Sensory horizons and consciousness

Returning to our evolutionary analysis, in recent computational studies of how organisms escape from predators, Mugan and MacIver (Reference Mugan and MacIver2020) have shown that in reduced sensory volumes typical of an aquatic environment, there is simply no time for planning, and instead, fish escape from predators by relying on habitual, model-free responses. Such rapid control of online action, pervasive prior to the water-to-land transition, does not require consciousness. In contrast, in terrestrial conditions, with much greater visual range, there is (some) time for planning – and such planning significantly boosts predator avoidance especially in cluttered environments where the predator might disappear behind occluders, and then appear again. As discussed in the previous section, the timescale of human consciousness is consistent with a central role in fixing a perceptual model of the world that is then used for internal simulation and planning (we will argue in a subsequent section on reality monitoring that the simulations themselves are largely unconscious).

This argument leads naturally to the view that organisms that evolved prior to the water-to-land transition, and which predominantly rely on small aquatic motor volumes, do not need either model-based control or conscious visual experience. If it were possible to do the kind of visual psychophysics we describe in Section 1 on fish, then we would predict that the temporal window of integration of postdictive effects on discrimination would scale with the sensory horizon. In other words, aquatic organisms would not be susceptible to long-lasting postdictive illusions because they have no need to integrate sensory information over ∼3-500ms to enable a stable world model in which to control their actions. They are rooted in the present, constrained by small sensorimotor volumes.

Most aquatic animals are not part of the “Buena Vista Sensing Club” (MacIver, Reference MacIver, Robbins and Aydede2009), and therefore, might not have developed model-based planning. There are exceptions to this general view. Most notably, land mammals that have returned to the water and use long-range echolocation, such as dolphins and whales, may retain the benefits of larger sensory horizons for planning (MacIver & Finlay, Reference MacIver and Finlay2022). Another key variable influencing the benefits of model-based planning in simulation is the presence of occlusions, making clues to adversaries cryptic. Terrestrial habitats such as savannahs provide selective benefits to planning over large sensory horizons (Mugan & MacIver, Reference Mugan and MacIver2020). But there may be aquatic species and environments for whom similar benefits emerge. The cluttered terrestrial-like environments of coral reefs and tide pools may be unusually rich environments for the evolution of model-based visual cognition. For instance, collaborative hunting where fish indicate the location of prey hiding in the reef has been documented in bluefin trevally-krait teams (Somaweera et al., Reference Somaweera, Udyawer, Amarasinghe, De Fresnes, Catherall and Molchanova2023) and grouper-moray eel teams (Vail, Manica, & Bshary, Reference Vail, Manica and Bshary2013), and experiments indicate that tide-pool dwelling gobies can rapidly form cognitive maps of shallow waters to enable one-shot learning of tide-pool location when jumping from one to another (Aronson, Reference Aronson1971). The octopus is also unusual in developing sophisticated deception strategies to deceive predators and conspecific competitors over an intermediate visual range (Godfrey-Smith, Reference Godfrey-Smith2024). Finally, as noted in footnote 5, species that are aquatic dwelling but have the ability to take advantage of terrestrial visual range may have developed model-based control and conscious visual experience. These species may include extinct transitional tetrapods at the water-to-land transition (such as Tiktaalik) and/or the archerfish, which hunts insects above the water’s surface using water jets (Volotsky, Donchin, & Segev, Reference Volotsky, Donchin and Segev2024).

Because consciousness (as measured in humans, which is the natural starting point; Bayne et al., Reference Bayne, Seth, Massimini, Shepherd, Cleeremans, Fleming, Malach, Mattingley, Menon, Owen, Peters, Razi and Mudrik2024) is associated with slow temporal windows of integration (Section 1), the evolutionary pressures to create such states would in general only manifest for larger sensory horizons. In contrast, organisms with restricted sensory horizons may only require model-free control, acting on the sensory present, rather than the remembered present. We have argued that these aspects of control are unconscious in humans. It is parsimonious to think that they would also be unconscious in other animalsFootnote ⁸ . Aquatic animals with restricted sensorimotor volumes, therefore, have no evolutionary need for conscious visual states. This might especially be the case for animals in the “pelagic zone” – away from the shores or coral reefs – who do not live in cluttered environments. Instead, we propose that a major discontinuity in the evolution of visual consciousness occurred at the water-to-land transition.

Reality monitoring

Larger sensory horizons afford time to use the perceptual system both online (for building a model of the world) and offline (for simulation, planning, and imagination). This presents a deep computational problem. For conscious perception to be a useful basis for planning and simulation, the percept should not itself be changed by the simulationFootnote ⁹ . If I imagine (simulate) an aggressive dog jumping over the fence next to me, I should not now perceive the dog jumping over the fence. The simulation allows me to realize it’s not a good idea to stand near the fence. It should not lead me to start running away from a (hallucinated) dog. A rational agent needs to keep apart its representation of the world, and its simulations or (active) inference through this representation of the world. This is known as “reality monitoring” (Dijkstra et al., Reference Dijkstra, Kok and Fleming2022).

One solution to perceptual reality monitoring (PRM) is to develop a system of higher-order representations indicating which first-order representations are reliable reflections of the world as it is now (Cortese & Kawato, Reference Cortese and Kawato2024; Lau, Reference Lau2019, Reference Lau2022; Michel, Reference Michel, Herzog, Doerig and SchurgerForthcoming). Under the PRM theory of consciousness, conscious representations are the result of the process of selecting reliable perceptual signals for use in the rational (model-based) control of action (see also Gershman, Reference Gershman2019). The machinery for reality monitoring tags perceptual signals as reflecting the outside world as it is now – flagging the present (Droege, Reference Droege2022; Gregory, Reference Gregory1996). In other words, conscious experience is the platform on which further cognition and planning are built – it cannot itself be changed by cognitive operations.

An empirical prediction of this view is that if the current world model is conscious via the operation of PRM, then the accompanying internal simulations should be largely unconscious. This is plausible – it seems unlikely that the rapid neural signatures of replay discussed above hypothesized to underpin model-based planning are consciously experienced. After all, these bursts of internal simulation are compressed in time, with individual trajectories through future possible states (e.g., possible routes through a maze, or courses of action to avoid a predator; Figure 3B) typically being completed in less than 200ms (Kay et al., Reference Kay, Chung, Sosa, Schor, Karlsson, Larkin, Liu and Frank2020; Y. Liu et al., Reference Liu, Dolan, Kurth-Nelson and Behrens2019). If we experienced such replay bursts, then they would constantly be in competition with perception itself. Consistent with this view, a recent study using fMRI in humans suggests that replay of previous visual experiences serves to update task representations, but is independent of conscious experience (Wittkuhn et al., Reference Wittkuhn, Krippner, Koch and Schuck2024). There is also evidence that predictions about future visual stimuli are encoded in the visual cortex (Ekman, Kusch, & De Lange, Reference Ekman, Kusch and De Lange2023; Kok, Mostert, & De Lange, Reference Kok, Mostert and De Lange2017; Kurth-Nelson et al., Reference Kurth-Nelson, Barnes, Sejdinovic, Dolan and Dayan2015; Turner, Blom, & Hogendoorn, Reference Turner, Blom and Hogendoorn2023). Yet, those predictions are not consciously experienced, or at least not always. Otherwise, one wouldn’t be able to experience the omission of a stimulus from a predictable sequence (Li, Theeuwes, & Van Moorselaar, Reference Li, Theeuwes and Van Moorselaar2024). This hypothesis is ripe for future empirical investigation.

One obvious counterexample is conscious visual imagery, which is often used to motivate the need for reality monitoring (Dijkstra, Kok, and Fleming, Reference Dijkstra, Kok and Fleming2022). According to our account, visual imagery occurs consciously as a result of being tagged by the reality monitor as reliably representing the world. This predicts that conscious mental imagery does not wear its unreality on its sleeve, so to speak. Instead, it has some level of “assertoric force” (Siegel & Sillins, Reference Siegel, Sillins and Matthen2015), perhaps proportional to its degree of vividness. It is first taken at face value, and this impression is then overridden post-facto based on non-perceptual signals (reviewed in Dijkstra et al. Reference Dijkstra, Kok and Fleming2022). This hypothesis is supported by studies showing that external and internal (imagined) signals are intermixed during near-threshold perception, with their combined strength determining judgments of reality and confidence (Dijkstra, Mazor, & Fleming, Reference Dijkstra, Mazor, Kok and Fleming2021, Reference Dijkstra, Mazor and Fleming2024; Dijkstra & Fleming, Reference Dijkstra and Fleming2023). Hyperphantasia (which is characterized by vivid conscious mental imagery) also seems associated with the experience of hallucinations (Oertel et al., Reference Oertel, Rotarska-Jagiela, Van De Ven, Haenschel, Grube, Stangier, Maurer and Linden2009), thus indicating that those who experience more vivid imagery tend to confuse it with genuine perceptual representations. This view would also suggest that far from being dysfunctional, aphantasics (individuals who do not experience mental imagery) may in fact be optimal, when considered from a reality monitoring standpoint – consistent with their ability to perform well in tasks requiring offline visual processing (Keogh et al., Reference Keogh, Wicken and Pearson2021; Pounder et al., Reference Pounder, Jacob, Evans, Loveday, Eardley and Silvanto2022). Of course, this is not to say that conscious visual imagery does not play a functional role. It undoubtedly does (just think of using visual imagery to determine whether a couch will fit through a door). But the extent to which this functional role depends on visual imagery occurring consciously (as opposed to unconsciously) is an open question, especially given emerging evidence suggesting functional unconscious visual imagery in aphantasia (Michel et al., Reference Michel, Morales, Block and Lau2025; Nanay, Reference Nanay2021; Pounder et al., Reference Pounder, Jacob, Evans, Loveday, Eardley and Silvanto2022; Weber et al., Reference Weber, Christophel, Görgen, Soch and Haynes2024).

In sum, the capacity for model-based planning promotes the development of reality monitoring capacities allowing organisms to distinguish, among all mental representations, which representations should impact one’s beliefs about what is happening in the world right now. Unconscious mental models do not compel us to believe that the world is as they represent it to be right now. Conscious experiences do. This epistemic contrast is explained if conscious representations are those selected by a reality monitoring system – an account that is also consistent with a long tradition in philosophy of mind and epistemology positing an essential tie between conscious perceptual experiences and perceptual beliefs (Armstrong, Reference Armstrong1968; Byrne, Reference Byrne2016; Glüer, Reference Glüer2009; Lyons, Reference Lyons2009; Smithies, Reference Smithies2019).

Hamlet’s problem in perception

Fodor (Reference Fodor and Garfield1987) noted that cognitive systems face Hamlet’s problem – a version of the frame problem in artificial intelligence:

Thinking more and exploring relevant options is generally beneficial. But thinking has to end at some point. How to know when to stop thinking (e.g., planning) is Hamlet’s problem. A similar problem arises in the case of perception. Indeed, when engaged in model-based control, the timescales for planning and perception need to be considered jointly. In order to act in a timely manner, one needs to curtail unnecessary planning; in order to plan in a timely manner, one needs to curtail unnecessary evidence gathering when forming an internal model of the environmentFootnote ¹⁰ .

Suppose that an organism has a limited sensory horizon, like that of an electric fish (Snyder et al., Reference Snyder, Nelson, Burdick and MacIver2007). With a short sensory horizon, the environment itself imposes a strict limit to how long the organism can afford to wait before reaching a decision and initiating a response. Organisms with short sensory horizons are not free to flexibly determine how much sensory evidence they should get. But animals with large sensory horizons have that luxury: with more time comes the freedom to wait for more sensory evidence to build more reliable internal models. Animals with large sensory horizons have also developed mechanisms to flexibly control evidence search – the most obvious of which are the evolution of the neck in early tetrapods (Carstens, Reference Carstens and Carstens2023), and a drastically increased capacity for targeted eye movements (Land, Reference Land2015). Not only that, but long sensory horizons also afford time to integrate sensory information with background beliefs: animals can now take time to think about what they see before acting, potentially leading them to decide whether or not to gather more evidence.

Since the environment itself does not dictate when the search for evidence should end, and since organisms following the water-to-land transition have more time and more flexible means for scanning large sensory horizons, they face Hamlet’s problem in perception: how to know when to stop searching and accumulating sensory evidence. We argued above that humans sometimes integrate evidence during a 400ms window before reaching a conscious perceptual decision. Hamlet’s problem in perception is the problem of determining how long this window should be.

We suggest that a metacognitive mechanism monitoring the reliability of one’s own perceptual states – such as the reality monitoring system hypothesized above – could be repurposed for flexibly opening and closing windows of sensory integration. As a first approximation, this mechanism could simply operate by tracking the signal-to-noise ratio of sensory representations, essentially applying the principles of signal detection theory (Green & Swets, Reference Green and Swets1966) on the sensory representations themselves (Lau, Reference Lau2022; Morales, Reference Morales2024). Whenever a signal is judged reliable enough, the monitoring system closes the window of sensory integration, thereby settling on a given interpretation of the way the world is over a given time window (Figure 4A).

Figure 4.

Reality monitoring and windows of integration. A) Higher-order states in generative models of perception can provide information about the reliability or precision of a world model (Fleming, Reference Fleming2020). B). The timescale of conscious perception allows a reliable world model to be built over a given time window of integration, and then be used for offline simulation and planning. Source: adapted from Herzog et al. (Reference Herzog, Drissi-Daoudi and Doerig2020).

The view that a higher-order mechanism is partly responsible for flexibly closing windows of sensory integration makes a central prediction: evidence accumulation in animals with large sensory horizons should be (at least in part) under metacognitive control.

There is preliminary evidence to confirm this prediction. For starters, information-seeking behavior seems guided by metacognitive states. Desender, Boldt, and Yeung (Reference Desender, Boldt and Yeung2018) designed a visual task where accuracy was matched across conditions that differed in confidence levels. By giving subjects the opportunity to take a second look at the evidence, they showed that information seeking is guided by confidence, and not just accuracy (see also Schulz, Fleming, & Dayan, Reference Schulz, Fleming and Dayan2023). Another dissociation between performance accuracy and metacognition in information-seeking behavior is found in blindsight monkeys. Kato et al. (Reference Kato, Zeghbib, Redgrave and Isa2021) trained blindsight monkeys to search for a hidden target in their blind visual field. Although monkeys were able to perform a saccade right on target in the blind field, they then kept looking for it. A speculative interpretation could be that since monkeys were not metacognitively sensitive to the success of their visual search in the blind field, they did not properly terminate the visual search.

Evidence accumulation itself might also be partly under metacognitive control. Balsdon, Wyart, and Mamassian (Reference Balsdon, Wyart and Mamassian2020) designed a task in which participants could stop evidence accumulation whenever they wanted to reach a certain level of performance. They showed that the capacity to appropriately set a bound on evidence accumulation was strongly related to metacognitive efficiency. This prompted Balsdon et al. (Reference Balsdon, Wyart and Mamassian2020) to suggest that “confidence decisions are not the result of some inert post-decisional process, but reflect an online control process that moderates sensory evidence accumulation” (p.7). According to this model, confidence is computed before the decision is fully reached, as seems plausible given recent results indicating that confidence signals are computed relatively early in the prefrontal cortex (Brosnan et al., Reference Brosnan, Sabaroedin, Silk, Genc, Newman, Loughnane, Fornito, O’Connell and Bellgrove2020; Gherman & Philiastides, Reference Gherman and Philiastides2018; Xue et al., Reference Xue, Zheng, Rafiei and Rahnev2023). Similar models have been proposed in the case of value-based decision making (Bénon et al., Reference Bénon, Lee, Hopper, Verdeil, Pessiglione, Vinckier, Bouret, Rouault, Lebouc, Pezzulo, Schreiweis, Burguière and Daunizeau2024; Lee, Daunizeau, & Pezzulo, Reference Lee, Daunizeau and Pezzulo2023), and this work is also in line with other well-established online modulations of visual processing by prefrontal cortex (e.g., Duan et al., Reference Duan, Zhan, Gross, Ince and Schyns2024; Kar & DiCarlo, Reference Kar and DiCarlo2021; Yan et al., Reference Yan, Zhan, Ince and Schyns2023).

The fact that evidence gathering is partlyFootnote ¹¹ under metacognitive control in humans does not mean that information-seeking behavior in conditions of uncertainty is evidence of metacognition, or that metacognition is the only way to set a decision boundary for evidence accumulation. What we tried to show instead is that evidence accumulation is under partial metacognitive control, in line with the view that a metacognitive mechanism helps solve Hamlet’s problem in perception.

Temporal windows of integration

The computational solutions to reality monitoring and Hamlet’s problem sketched above are two sides of the same coin. Terrestrial animals with long-range sensory horizons need to build a reliable model of the world in which to plan, but not dawdle over doing so – otherwise they risk becoming dynamically uncoupled from their environment, and rooted in the past. This is a tradeoff with reliability at its core. Too little sensory evidence accumulation, and the internal model will be too impoverished to enable useful model-based control. But too much evidence accumulation and the model will never get built in the first place. We suggest that metacognitive signals tracking the reliability of a first-order world model provide a natural computational basis for managing this tradeoff. Once one’s confidence is increased to a particular threshold, an internal model can be “fixed” and used for rapid offline simulation. The temporal window of conscious experience is the result of this delicate balance achieved from simultaneously solving perceptual reality monitoring and Hamlet’s problem.

Our view makes novel empirical predictions. First, it predicts a link between metacognitive confidence and the timing of bursts of neural replay (assuming that the latter provides a neural substrate for online planning) – with replay occurring only once a subject is confident enoughFootnote ¹² . This view also predicts a link between reality monitoring and confidence, with recent studies of the interplay between perception and imagination suggesting that judgments of reality are tightly coupled to fluctuations in confidence (Dijkstra et al., Reference Dijkstra, Mazor and Fleming2024). Another prediction is that reality monitoring capacities should only be found in animals with larger sensory horizons and capable of internal simulation. Finally, it predicts that the window of integration of conscious experience discussed above should be in part governed by the reliability of sensory signals – with longer windows of integration being adopted in lower-reliability environments.

Consciousness and reality monitoring

The view that consciousness emerges with a reality monitoring system, as described above, has significant explanatory benefits. Support for this “perceptual reality monitoring theory of consciousness” (PRM) comes from an inference to the best explanation (Lau, Reference Lau2022; Michel, Reference Michel, Herzog, Doerig and SchurgerForthcoming). Let us finish by listing some of the explanatory benefits that are most salient given the hypothesis developed in this article.

First, we often take the remarkable coherence of our conscious experiences for granted. But given the discussion above, one would not necessarily expect this from considering what sensory areas represent at any given time. Again, it is striking that our conscious experiences remain coherent despite the fact that several sensory representations coexist at any given moment, fulfilling multiple roles for perception and cognition, from the maintenance of sensory information over time to sensory activity involved in learning, planning, and predicting (Ekman et al., Reference Ekman, Kusch and De Lange2023; Kay et al., Reference Kay, Chung, Sosa, Schor, Karlsson, Larkin, Liu and Frank2020; Kok et al., Reference Kok, Mostert and De Lange2017; Liu et al., Reference Liu, Dolan, Kurth-Nelson and Behrens2019; Wittkuhn et al., Reference Wittkuhn, Krippner, Koch and Schuck2024). For instance, King and Wyart (Reference King and Wyart2021) showed that images presented in a rapid visual stream can be robustly decoded from brain activity up to one second after stimulus presentation. Yet, subjects presumably feel like they are experiencing a single image at a time – perhaps together with a sense of the relations between this image and the previous image(s). There is an apparent tension between long-lasting parallel encoding and the apparently serial nature of the contents of experience. A recent focus in neuroscience on identifying neural activity related to a state other than the agent’s current state is prima facie evidence that the brain contains a multitude of internal processes that do not correspond to our current experience (Liu et al., Reference Liu, Mattar, Behrens, Daw and Dolan2021; Mattar & Lengyel, Reference Mattar and Lengyel2022). If conscious experiences are constituted by representations selected and tagged as reliably reflecting the world as it is now by a reality monitoring mechanism, this explains why our conscious visual experiences are coherent, instead of being meaningless jumbles of represented features.

A second advantage of the account is that it explains the unique epistemic profile of conscious representations. For our purposes, there are three relevant epistemic features that seem to separate conscious perceptual states from unconscious perceptual states. The first is the apparent connection between conscious experiences and belief justification. Conscious perceptual experiences justify perceptual beliefs – for instance, a conscious experience of an apple justifies the belief that there’s an apple. A common view in epistemology is that unconscious representations do not likewise justify beliefs (Byrne, Reference Byrne2016; Huemer, Reference Huemer2006; Smithies, Reference Smithies2019). This epistemic asymmetry is explained if conscious representations are precisely those that are selected by a reality monitoring mechanism to have a direct impact on belief formation. The same hypothesis accounts for the second epistemic feature: conscious experiences lead to stubborn tendencies to believe (Lau, Reference Lau2022). As noted by Armstrong (Reference Armstrong1968) when discussing this idea:

By contrast, unconscious perceptual states do not seem to result in a stubborn propensity to believe – for instance, blindsight subjects never insist that the world is as their unconscious states represent it to be (Weiskrantz, Reference Weiskrantz2009). This contrast is explained if a function of a reality monitoring system is to select which perceptual states should impact our beliefs (partly for the purpose of model-based planning). A third relevant epistemic feature of consciousness is that conscious perceptual decisions are available for explicit metacognitive evaluation, whereas unconscious perceptual decisions are not (Persaud, McLeod, & Cowey, Reference Persaud, McLeod and Cowey2007, Reference Persaud, Davidson, Maniscalco, Mobbs, Passingham, Cowey and Lau2011; Vlassova, Donkin, & Pearson, Reference Vlassova, Donkin and Pearson2014). That is, explicit metacognition seems consciousness-selective (See Michel (Reference Michel2023) for a review). This feature is explained if the representations selected by the reality monitor also become available for metacognition, which can then reuse reality monitoring capacities (i.e., assessing signal-to-noise ratio) for explicit confidence ratings.

The perceptual reality monitoring theory, coupled with the account developed here, also provides a principled explanation for why vision used in the online guidance of action is not conscious – as supported by the influential two visual stream hypothesis (Kozuch, Reference Kozuch2023; Milner & Goodale, Reference Milner and Goodale2006; Wu, Reference Wu2013). Vision for action is unconscious because it needs to be fast. The visual system responsible for vision for action – the dorsal visual stream – does not face the reality monitoring problem and Hamlet’s problem, which the reality monitoring system evolved to solve. The reality monitoring system is not meant to assess the reliability of visual representations in the dorsal visual stream – which is why those representations are unconscious.

This is just a relevant subset of the explanatory benefits of the hypothesis that consciousness is linked to the activity of a perceptual reality monitoring system (for more, see Lau, Reference Lau2022; Michel, Reference Michel, Herzog, Doerig and SchurgerForthcoming)Footnote ¹³ . If this view is on the right track, visual consciousness comes online with the emergence of perceptual reality monitoring capacities. In turn, reality monitoring capacities emerge in response to the reality monitoring problem and Hamlet’s problem in perception – computational problems that arise for animals with large sensory horizons.

Let us nevertheless emphasize that our account is compatible with a variety of alternative views. Instead of going through the list, let us paint with broad strokes. One of the main dividing lines in consciousness research is between those who think that consciousness fulfills a perceptual role (as in the local recurrence theory; Lamme, Reference Lamme, Metzinger and Windt2016), and those who think that consciousness is for cognition (as in the global workspace theory; Mashour et al., Reference Mashour, Roelfsema, Changeux and Dehaene2020, or the attention schema theory; Graziano, Reference Graziano2015). Our account is compatible with ‘cognitivist’ theories of all kinds, as long as they hypothesize that the kind of cognition that consciousness supports is model-based.

Beyond vision

The current paper has focused on vision, which affords a particularly precise connection between sensory horizons, evolutionary biology, and the speed of consciousness. In future work, it would be of interest to develop this theoretical project for other sensory modalities – for instance, drawing on empirical work on postdictive phenomena in conscious audition (e.g., Alais & Burr, Reference Alais and Burr2003). More broadly, we envisage a generalization of our argument that encompasses world models in general, regardless of their sensory origins. We would expect that the capacity for flexible model-based planning would scale with the organism’s (multimodal) sensory horizon. Such a prediction could potentially be tested by imposing artificial restrictions on the sensory horizon in one modality (such as introducing visual fog) and asking how another modality may be prioritized to compensate. Also relevant here are nascent investigations of multimodal perceptual metacognition and domain-general forms of confidence (Deroy, Spence, & Noppeney, Reference Deroy, Spence and Noppeney2016; Masset et al., Reference Masset, Ott, Lak, Hirokawa and Kepecs2020; Morales, Lau, & Fleming, Reference Morales, Lau and Fleming2018).

What about sensory modalities for which the idea of sensory horizon does not straightforwardly apply, such as touch, and affective states, such as fear or pain? Our view is that those states do not occur consciously if they are only recruited for model-free processes. From here, there are two possible paths to extend our account. One possibility is that model-based planning (and reality monitoring) is tied to large sensory horizons (including but not restricted to visual horizons). If this is correct, our account predicts that organisms with short sensory horizons do not have conscious states. But there is another possibility consistent with our hypothesis: organisms that are otherwise conscious (e.g., consciously experience pain) might operate with unconscious vision as long as vision is not recruited for model-based planning. After all, it is possible for one type of state (i.e., vision) to occur unconsciously while other types of states occur consciously – as in blindsight subjects. Which of those possibilities turns out to be correct depends on future research on the origins and functions of model-based planning for other sensory modalities and affective states (e.g., in the case of fear, see LeDoux, Reference LeDoux2019, Reference LeDoux2023).