Skip to main content
    • Aa
    • Aa

Whatever next? Predictive brains, situated agents, and the future of cognitive science

  • Andy Clark (a1)

Brains, it has recently been argued, are essentially prediction machines. They are bundles of cells that support perception and action by constantly attempting to match incoming sensory inputs with top-down expectations or predictions. This is achieved using a hierarchical generative model that aims to minimize prediction error within a bidirectional cascade of cortical processing. Such accounts offer a unifying model of perception and action, illuminate the functional role of attention, and may neatly capture the special contribution of cortical processing to adaptive success. This target article critically examines this “hierarchical prediction machine” approach, concluding that it offers the best clue yet to the shape of a unified science of mind and action. Sections 1 and 2 lay out the key elements and implications of the approach. Section 3 explores a variety of pitfalls and challenges, spanning the evidential, the methodological, and the more properly conceptual. The paper ends (sections 4 and 5) by asking how such approaches might impact our more general vision of mind, experience, and agency.

  • View HTML
    • Send article to Kindle

      To send this article to your Kindle, first ensure is added to your Approved Personal Document E-mail List under your Personal Document Settings on the Manage Your Content and Devices page of your Amazon account. Then enter the ‘name’ part of your Kindle email address below. Find out more about sending to your Kindle.

      Note you can select to send to either the or variations. ‘’ emails are free but can only be sent to your device when it is connected to wi-fi. ‘’ emails can be delivered even when you are not connected to wi-fi, but note that service fees apply.

      Find out more about the Kindle Personal Document Service.

      Whatever next? Predictive brains, situated agents, and the future of cognitive science
      Available formats
      Send article to Dropbox

      To send this article to your Dropbox account, please select one or more formats and confirm that you agree to abide by our usage policies. If this is the first time you use this feature, you will be asked to authorise Cambridge Core to connect with your Dropbox account. Find out more about sending content to Dropbox.

      Whatever next? Predictive brains, situated agents, and the future of cognitive science
      Available formats
      Send article to Google Drive

      To send this article to your Google Drive account, please select one or more formats and confirm that you agree to abide by our usage policies. If this is the first time you use this feature, you will be asked to authorise Cambridge Core to connect with your Google Drive account. Find out more about sending content to Google Drive.

      Whatever next? Predictive brains, situated agents, and the future of cognitive science
      Available formats
Hide All

A exceptionally large number of excellent commentary proposals inspired a special research topic for further discussion of this target article's subject matter, edited by Axel Cleeremans and Shimon Edelman in Frontiers in Theoretical and Philosophical Psychology. This discussion has a preface by Cleeremans and Edelman and 25 commentaries and includes a separate rejoinder from Andy Clark. See:

Linked references
Hide All

This list contains references from the content that can be linked to their source. For a full set of references and notes please see the PDF or HTML where available.

F. Adams & K. Aizawa (2001) The bounds of cognition. Philosophical Psychology 14(1):4364.

D. Alais & D. Burr (2004) The ventriloquist effect results from near-optimal bimodal integration. Current Biology 14:257–62.

A. Alink , C. M. Schwiedrzik , A. Kohler , W. Singer & L. Muckli (2010) Stimulus predictability reduces responses in primary visual cortex. Journal of Neuroscience 30:2960–66.

M. L. Anderson (2007) The massive redeployment hypothesis and the functional topography of the brain. Philosophical Psychology 20(2):143–74.

B. Arthur (1994) Increasing returns and path dependence in the economy. University of Michigan Press.

M. Bar (2007) The proactive brain: Using analogies and associations to generate predictions. Trends in Cognitive Sciences 11(7):280–89.

L. F. Barrett (2009) The future of psychology: Connecting mind to brain. Perspectives in Psychological Science 4:326–39.

L. F. Barrett & M. Bar (2009) See it with feeling: Affective predictions during object perception. Philosophical Transactions of the Royal Society of London B: Biological Sciences 364(1521):1325–34.

M. Berniker & K. P. Körding (2008) Estimating the sources of motor errors for adaptation and generalization. Nature Neuroscience 11:1454–61.

D. Bindra (1959) Stimulus change, reactions to novelty, and response decrement. Psychological Review 66:96103.

R. T. Born , J. M. Tsui & C. C. Pack (2009) Temporal dynamics of motion integration, In: Dynamics of visual motion processing, ed. U. Ilg & G. Masson . pp. 3754. Springer.

H. Brown , K. Friston & S. Bestamnn (2011) Active inference, attention and motor preparation. Frontiers in Psychology 2:218. doi: 10.3389/fpsyg.2011.00218.

J. Burge , C. Fowlkes & M. Banks (2010) Natural-scene statistics predict how the figure–ground cue of convexity affects human depth perception. Journal of Neuroscience 30(21):7269–80.

N. Chater & C. Manning (2006) Probabilistic models of language processing and acquisition. Trends in Cognitive Sciences 10(7):335–44.

A. Clark (1987) The kludge in the machine. Mind and Language 2(4):277300.

A. Clark (2006a) Language, embodiment and the cognitive niche. Trends in Cognitive Sciences 10(8):370–74.

A. Clark (2008) Supersizing the mind: Action embodiment, and cognitive extension. Oxford University Press.

A. Clark & D. Chalmers (1998) The extended mind. Analysis 58(1):719.

C. W. G. Clifford , M. A. Webster , G. B. Stanley , A. A. Stocker , A. Kohn , T. O. Sharpee & O. Schwartz (2007) Visual adaptation: Neural, psychological and computational aspects. Vision Research 47:3125–31.

M. Coltheart (2007) Cognitive neuropsychiatry and delusional belief (The 33rd Sir Frederick Bartlett Lecture). The Quarterly Journal of Experimental Psychology 60(8):1041–62.

P. R. Corlett , C. D. Frith & P. C. Fletcher (2009a) From drugs to deprivation: A Bayesian framework for understanding models of psychosis. Psychopharmacology (Berlin) 206(4):515–30.

P. R. Corlett , J. K. Krystal , J. R. Taylor & P. C. Fletcher (2009b) Why do delusions persist? Frontiers in Human Neuroscience 3:12. doi: 10.3389/neuro.09.012.2009.

P. R. Corlett , J. R. Taylor , X. J. Wang , P. C. Fletcher & J. H. Krystal (2010) Toward a neurobiology of delusions. Progress in Neurobiology 92(3):345–69.

P. Dayan (1997) Recognition in hierarchical models. In: Foundations of computational mathematics, ed. F. Cucker & M. Shub , pp. 4357. Springer.

P. Dayan & G. Hinton (1996) Varieties of Helmholtz machine. Neural Networks 9:1385–403.

P. Dayan , G. E. Hinton & R. M. Neal (1995) The Helmholtz machine. Neural Computation 7:889904.

S. Deneve (2008) Bayesian spiking neurons I: Inference. Neural Computation 20:91117.

H. E. M. den Ouden , J. Daunizeau , J. Roiser , K. J. Friston & K. E. Stephan (2010) Striatal prediction error modulates cortical coupling. Journal of Neuroscience 30:3210–19.

R. Desimone & J. Duncan (1995) Neural mechanisms of selective visual attention. Annual Review of Neuroscience 18:193222.

L. de-Wit , B. Machilsen & T. Putzeys (2010) Predictive coding and the neural response to predictable stimuli. Journal of Neuroscience 30:8702–703.

E. A. Di Paolo (2009) Extended life. Topoi 28(1):921.

S. O. Dumoulin & R. F. Hess (2006) Modulation of V1 activity by shape: image-statistics or shape-based perception? Journal of Neurophysiology 95:3654–64.

T. Egner , J. M. Monti & C. Summerfield (2010) Expectation and surprise determine neural population responses in the ventral visual stream. Journal of Neuroscience 30(49):16601–608.

C. Eliasmith (2007) How to build a brain: From function to implementation. Synthese 159(3):373–88.

A. K. Engel , P. Fries & W. Singer (2001) Dynamic predictions: Oscillations and synchrony in top–down processing. Nature Reviews: Neuroscience 2:704–16.

M. O. Ernst (2010) Eye movements: Illusions in slow motion. Current Biology 20(8):R357–59.

M. O. Ernst & M. S. Banks (2002) Humans integrate visual and haptic information in a statistically optimal fashion. Nature 415:429–33.

M. Fabre-Thorpe (2011) The characteristics and limits of rapid visual categorization. Frontiers in Psychology 2:243. doi: 10.3389/fpsyg.2011.00243.

H. Feldman & K. J. Friston (2010) Attention, uncertainty, and free-energy. Frontiers in Human Neuroscience 4:215. doi:10.3389/fnmuh.2010.00215.

J. Feldman (2010) Cognitive science should be unified: Comment on Griffiths et al. and McClelland et al. Trends in Cognitive Sciences 14(8):341.

P. Fletcher & C. Frith (2009) Perceiving is believing: A Bayesian approach to explaining the positive symptoms of schizophrenia. Nature Reviews: Neuroscience 10:4858.

T. C. A. Freeman , R. A. Champion & P. A. Warren (2010) A Bayesian model of perceived head-centred velocity during smooth pursuit eye movement. Current Biology 20:757–62.

K. Friston (2002) Beyond phrenology: What can neuroimaging tell us about distributed circuitry? Annual Review of Neuroscience 25:221–50.

K. Friston (2003) Learning and inference in the brain. Neural Networks 16(9):1325–52.

K. Friston (2005) A theory of cortical responses. Philosophical Transactions of the Royal Society of London B: Biological Sciences 360(1456):815–36.

K. Friston (2009) The free-energy principle: A rough guide to the brain? Trends in Cognitive Sciences 13(7):293301.

K. J. Friston (2010) The free-energy principle: A unified brain theory? Nature Reviews Neuroscience 11(2):127–38.

K. Friston (2011b) What is optimal about motor control? Neuron 72:488–98.

K. J. Friston , J. Daunizeau , J. Kilner & S. J. Kiebel (2010) Action and behavior: A free-energy formulation. Biological Cybernetics 102(3):227–60.

K. Friston & S. Kiebel (2009) Cortical circuits for perceptual inference. Neural Networks 22:1093–104.

K. Friston , J. Mattout & J. Kilner (2011) Action understanding and active inference. Biological Cybernetics 104:137–60.

K. Friston & K. Stephan (2007) Free energy and the brain. Synthese 159(3):417–58.

C. Frith , R. Perry & E. Lumer (1999) The neural correlates of conscious experience: An experimental framework. Trends in Cognitive Sciences 3(3):105.

P. Gerrans (2007) Mechanisms of madness. Evolutionary psychiatry without evolutionary psychology. Biology and Philosophy 22:3556.

J. N. Gold & M. N. Shadlen (2001) Neural computations that underlie decisions about sensory stimuli. Trends in Cognitive Sciences 5(10):16 238–55.

R. L. Gregory (1980) Perceptions as hypotheses. Philosophical Transactions of the Royal Society of London B 290(1038):181–97.

T. Griffiths , N. Chater , C. Kemp , A. Perfors & J. B. Tenenbaum (2010) Probabilistic models of cognition: Exploring representations and inductive biases. Trends in Cognitive Sciences 14(8):357–64.

K. Grill-Spector , R. Henson & A. Martin (2006) Repetition and the brain: Neural models of stimulus-specific effects. Trends in Cognitive Sciences 10(1):1423.

R. Grush (2004) The emulation theory of representation: Motor control, imagery, and perception. Behavioral and Brain Sciences 27:377442.

S. Harnad (1990) The symbol grounding problem. Physica D 42:335–46.

H. Helbig & M. Ernst (2007) Optimal integration of shape information from vision and touch. Experimental Brain Research 179:595605.

G. E. Hinton (2002) Training products of experts by minimizing contrastive divergence. Neural Computation 14(8):1711–800.

G. E. Hinton (2007a) Learning multiple layers of representation. Trends in Cognitive Sciences 11:428–34.

G. E. Hinton (2010) Learning to represent visual input. Philosophical Transactions of the Royal Society, B. 365:177–84.

G. E. Hinton , P. Dayan , B. J. Frey & R. M. Neal (1995) The wake-sleep algorithm for unsupervised neural networks. Science 268:1158–60.

G. E. Hinton & Z. Ghahramani (1997) Generative models for discovering sparse distributed representations. Philosophical Transactions of the Royal Society B 352:1177–90.

G. E. Hinton , S. Osindero & Y. Teh (2006) A fast learning algorithm for deep belief nets. Neural Computation 18:1527–54.

G. E. Hinton & R. R. Salakhutdinov (2006) Reducing the dimensionality of data with neural networks. Science 313(5786):504507.

S. Hochstein & M. Ahissar (2002) View from the top: Hierarchies and reverse hierarchies in the visual system. Neuron 36(5):791804.

J. Hohwy (2007) Functional Integration and the mind. Synthese 159(3):315–28.

J. Hohwy , A. Roepstorff & K. Friston (2008) Predictive coding explains binocular rivalry: An epistemological review. Cognition 108(3):687701.

T. Hosoya , S. A. Baccus & M. Meister (2005) Dynamic predictive coding by the retina. Nature 436(7):7177.

C. Q. Howe , R. B. Lotto & D. Purves (2006) Comparison of bayesian and empirical ranking approaches to visual perception. Journal of Theoretical Biology 241:866–75.

A. Iriki & M. Taoka (2012) Triadic (ecological, neural, cognitive) niche construction: A scenario of human brain evolution extrapolating tool use and language from the control of reaching actions. Philosophical Transactions of the Royal Society B 367:1023.

M. Kawato , H. Hayakama & T. Inui (1993) A forward-inverse optics model of reciprocal connections between visual cortical areas. Network 4:415–22.

D. Knill & A. Pouget (2004) The Bayesian brain: The role of uncertainty in neural coding and computation. Trends in Neuroscience 27(12):712–19.

T. Kohonen (1989) Self-organization and associative memory. Springer-Verlag.

P. König & N. Krüger (2006) Symbols as self-emergent entities in an optimization process of feature extraction and predictions. Biological Cybernetics 94(4):325–34.

K. P. Körding , J. B. Tenenbaum & R. Shadmehr (2007) The dynamics of memory as a consequence of optimal adaptation to a changing body. Nature Neuroscience 10:779–86.

S. M. Kosslyn , W. L. Thompson , I. J. Kim & N. M. Alpert (1995) Topographical representations of mental images in primary visual cortex. Nature 378:496–98.

K. Kveraga , A. Ghuman & M. Bar (2007) Top-down predictions in the cognitive brain. Brain and Cognition 65:145–68.

T. K. Landauer & S. T. Dumais (1997) A solution to Plato's problem: The Latent Semantic Analysis theory of the acquisition, induction, and representation of knowledge. Psychological Review 104:211–40.

T. K. Landauer , P. W. Foltz & D. Laham (1998) Introduction to Latent Semantic Analysis. Discourse Processes 25: 259–84.

R. Langner , T. Kellermann , F. Boers , W. Sturm , K. Willmes & S. B. Eickhoff (2011) Modality-specific perceptual expectations selectively modulate baseline activity in auditory, somatosensory, and visual cortices. Cerebral Cortex 21(12):2850–62.

M. Lee (2010) Emergent and structured cognition in Bayesian models: Comment on Griffiths et al. and McClelland et al. Trends in Cognitive Sciences 14(8):345–46.

S. H. Lee , R. Blake & D. J. Heeger (2005) Traveling waves of activity in primary visual cortex during binocular rivalry. Nature Neuroscience 8(1):2223.

T. S. Lee & D. Mumford (2003) Hierarchical Bayesian inference in the visual cortex. Journal of Optical Society of America, A 20(7):1434–48.

D. Leopold & N. Logothetis (1999) Multistable phenomena: Changing views in perception. Trends in Cognitive Sciences 3:254–64.

D. J. C. MacKay (1995) Free-energy minimization algorithm for decoding and cryptoanalysis. Electron Letters 31:445–47.

L. T. Maloney & P. Mamassian (2009) Bayesian decision theory as a model of visual perception: Testing Bayesian transfer. Visual Neuroscience 26:147–55.

L. T. Maloney & H. Zhang (2010) Decision-theoretic models of visual perception and action. Vision Research 50:2362–74.

D. Marr (1982). Vision: A computational approach. Freeman.

J. McClelland , M. Botvinick , D. Noelle , D. Plaut , T. Rogers , M. Seidenberg & L. Smith (2010) Letting structure emerge: Connectionist and dynamical systems approaches to cognition. Trends in Cognitive Sciences 14(8):348–56.

J. McClelland & D. Rumelhart (1981) An interactive activation model of context effects in letter perception: Part 1. An account of basic findings. Psychological Review 88:375407.

L. Melloni , C. M. Schwiedrzik , N. Muller , E. Rodriguez & W. Singer (2011) Expectations change the signatures and timing of electrophysiological correlates of perceptual awareness. Journal of Neuroscience 31(4):1386–96.

R. Menary (2007) Cognitive integration: Attacking the bounds of cognition. Palgrave Macmillan.

M. Meng & F. Tong (2004) Can attention selectively bias bistable perception? differences between binocular rivalry and ambiguous figures. Journal of Vision 4:539–51.

B. Merker (2004) Cortex, countercurrent context, and dimensional integration of lifetime memory. Cortex 40:559–76.

D. Milner & M. Goodale (2006) The visual brain in action, 2nd edition. Oxford University Press.

L. Muckli (2010) What are we missing here? Brain imaging evidence for higher cognitive functions in primary visual cortex V1. International Journal of Imaging Systems Technology (IJIST) 20:131–39.

D. Mumford (1992) On the computational architecture of the neocortex. II. The role of cortico-cortical loops. Biological Cybernetics 66(3):241–51.

S. O. Murray , D. Kersten , B. A. Olshausen , P. Schrater & D. L. Woods (2002) Shape perception reduces activity in human primary visual cortex. Proceedings of the National Academy of Sciences USA 99(23):15164–69.

S. O. Murray , P. Schrater & D. Kersten (2004) Perceptual grouping and the interactions between visual cortical areas. Neural Networks 17(5–6):695705.

R. M. Neal & G. Hinton (1998) A view of the EM algorithm that justifies incremental, sparse, and other variants. In: Learning in graphical models, ed. M. I. Jordan , pp. 355–68. Kluwer.

B. A. Olshausen & D. J. Field (1996) Emergence of simple-cell receptive field properties by learning a sparse code for natural images. Nature 381(6583):607609.

K. Overy & I. Molnar-Szakacs (2009) Being together in time: Musical experience and the mirror neuron system. Music Perception 26(5):489504.

C. C. Pack & R. T. Born (2001) Temporal dynamics of a neural solution to the aperture problem in visual area MT of macaque brain. Nature 409:1040–42.

A. Pascual-Leone & R. Hamilton (2001) The metamodal organization of the brain. Progress in Brain Research 134:427–45.

R. Pfeifer , M. Lungarella , O. Sporns & Y. Kuniyoshi (2007) On the information theoretic implications of embodiment – principles and methods. Lecture Notes in Computer Science (LNCS), vol. 4850. Springer.

M. J. Pickering & S. Garrod (2007) Do people use language production to make predictions during comprehension? Trends in Cognitive Sciences (11):105110.

A. Pouget , P. Dayan & R. Zemel (2003) Inference and computation with population codes. Annual Review of Neuroscience 26:381410.

J. J. Prinz (2005) A neurofunctional theory of consciousness. In: Cognition and the brain: Philosophy and neuroscience movement, ed. A. Brook & K. Akins , pp. 381–96. Cambridge University Press.

R. P. N. Rao & D. H. Ballard (1999) Predictive coding in the visual cortex: A functional interpretation of some extra-classical receptive-field effects. Nature Neuroscience 2(1):7987.

K. Rauss , S. Schwartz & G. Pourtois (2011) Top-down effects on early visual processing in humans: A predictive coding framework. Neuroscience and Biobehavioral Reviews 35(5):1237–53.

L. Reddy , N. Tsuchiya & T. Serre (2010) Reading the mind's eye: decoding category information during mental imagery. NeuroImage 50(2):818–25.

L. Reich , M. Szwed , L. Cohen & A. Amedi (2011) A ventral stream reading center independent of visual experience. Current Biology 21:363–68.

A. Roepstorff , J. Niewohner & S. Beck (2010) Enculturing brains through patterned practices. Neural Networks 23(8–9):1051–59.

M. Rowlands (1999) The body in mind: Understanding cognitive processes. Cambridge University Press.

O. Schwartz , A. Hsu & P. Dayan (2007) Space and time in visual context Nature Reviews Neuroscience 8:522–35.

L. Shams , W. J. Ma & U. Beierholm (2005) Sound-induced flash illusion as an optimal percept. NeuroReport 16(10):1107–10.

Yun Q. Shi & H. Sun (1999) Image and video compression for multimedia engineering: Fundamentals, algorithms, and standards. CRC Press.

L. Smith & M. Gasser (2005) The development of embodied cognition: Six lessons from babies. Artificial Life 11(1):1330.

P. L. Smith & R. Ratcliff (2004) Psychology and neurobiology of simple decisions. Trends in Neuroscience 27:161–68.

M. W. Spratling (2008a) Predictive coding as a model of biased competition in visual attention. Vision Research 48(12):1391–408.

M. V. Srinivasan , S. B. Laughlin & A. Dubs . (1982) Predictive coding: A fresh view of inhibition in the retina. Proceedings of the Royal Society of London, B 216:427–59.

K. Sterelny (2007) Social intelligence, human intelligence and niche construction. Philosophical Transactions of the Royal Society of London, Series B: Biological Sciences 362(1480):719–30.

K. Stotz (2010) Human nature and cognitive–developmental niche construction. Phenomenology and the Cognitive Sciences 9(4):483501.

C. Summerfield & T Egner (2009) Expectation (and attention) in visual cognition. Trends in Cognition Science 13:403409.

C. Summerfield , E. H. Trittschuh , J. M. Monti , M. M. Mesulam & T. Egner (2008) Neural repetition suppression reflects fulfilled perceptual expectations. Nature Neuroscience 11(9):10041006.

E. Todorov & M. I. Jordan (2002) Optimal feedback control as a theory of motor coordination. Nature Neuroscience 5(11):1226–35.

P. Verschure , T. Voegtlin & R. Douglas (2003) Environmentally mediated synergy between perception and behaviour in mobile robots. Nature 425:620–24.

I. Vilares & K. Körding (2011) Bayesian models: The structure of the world, uncertainty, behavior, and the brain. Annals of the New York Academy of Science 1224:2239.

P. Waelti , A. Dickinson & W. Schultz (2001) Dopamine responses comply with basic assumptions of formal learning theory. Nature 412:4348.

Y. Weiss , E. P. Simoncelli & E. H. Adelson (2002) Motion illusions as optimal percepts. Nature Neuroscience 5(6):598604. doi:10.1038/nn858.

M. Wheeler & A. Clark (2009) Culture, embodiment and genes: Unravelling the triple helix. Philosophical Transactions of the Royal Society of London, B 363(1509):3563–75.

R. A. Wilson (1994) Wide computationalism. Mind 103:351–72.

R. A. Wilson (2004) Boundaries of the mind: The individual in the fragile sciences – cognition. Cambridge University Press.

A. J. Yu (2007) Adaptive behavior: Humans act as Bayesian learners. Current Biology 17:R977–80.

A. Yuille & D. Kersten (2006) Vision as Bayesian inference: Analysis by synthesis? Trends in Cognitive Science 10(7):301308.

K. Zahedi , N. Ay & R. Der (2010) Higher coordination with less control – a result of information maximization in the sensorimotor loop. Adaptive Behavior 18(3–4):338–55.

Recommend this journal

Email your librarian or administrator to recommend adding this journal to your organisation's collection.

Behavioral and Brain Sciences
  • ISSN: 0140-525X
  • EISSN: 1469-1825
  • URL: /core/journals/behavioral-and-brain-sciences
Please enter your name
Please enter a valid email address
Who would you like to send this to? *



Altmetric attention score

Full text views

Total number of HTML views: 508
Total number of PDF views: 3304 *
Loading metrics...

Abstract views

Total abstract views: 5175 *
Loading metrics...

* Views captured on Cambridge Core between September 2016 - 28th July 2017. This data will be updated every 24 hours.