Book contents
- Frontmatter
- Contents
- Foreword
- Preface
- Contributors
- 1 General introduction
- Part I Rational and optimal decision making
- Part II Computational neuroscience models
- 9 Introduction to Part II: computational neuroscience models
- 10 Action selection and refinement in subcortical loops through basal ganglia and cerebellum
- 11 Cortical mechanisms of action selection: the affordance competition hypothesis
- 12 Toward an executive without a homunculus: computational models of the prefrontal cortex/basal ganglia system
- 13 Hierarchically organised behaviour and its neural foundations: a reinforcement-learning perspective
- 14 The medial reticular formation: a brainstem substrate for simple action selection?
- 15 Understanding decision-making deficits in neurological conditions: insights from models of natural action selection
- 16 Biologically constrained action selection improves cognitive control in a model of the Stroop task
- 17 Mechanisms of choice in the primate brain: a quick look at positive feedback
- Part III Action selection in social contexts
- Index
- Plate section
- References
12 - Toward an executive without a homunculus: computational models of the prefrontal cortex/basal ganglia system
from Part II - Computational neuroscience models
Published online by Cambridge University Press: 05 November 2011
Book contents
- Frontmatter
- Contents
- Foreword
- Preface
- Contributors
- 1 General introduction
- Part I Rational and optimal decision making
- Part II Computational neuroscience models
- 9 Introduction to Part II: computational neuroscience models
- 10 Action selection and refinement in subcortical loops through basal ganglia and cerebellum
- 11 Cortical mechanisms of action selection: the affordance competition hypothesis
- 12 Toward an executive without a homunculus: computational models of the prefrontal cortex/basal ganglia system
- 13 Hierarchically organised behaviour and its neural foundations: a reinforcement-learning perspective
- 14 The medial reticular formation: a brainstem substrate for simple action selection?
- 15 Understanding decision-making deficits in neurological conditions: insights from models of natural action selection
- 16 Biologically constrained action selection improves cognitive control in a model of the Stroop task
- 17 Mechanisms of choice in the primate brain: a quick look at positive feedback
- Part III Action selection in social contexts
- Index
- Plate section
- References
Summary
Summary
The prefrontal cortex (PFC) has long been thought to serve as an ‘executive’ that controls the selection of actions, and cognitive functions more generally. However, the mechanistic basis of this executive function has not been clearly specified, often amounting to a homunculus. This chapter reviews recent attempts to deconstruct this homunculus by elucidating the precise computational and neural mechanisms underlying the executive functions of the PFC. The overall approach builds upon existing mechanistic models of the basal ganglia (BG) and frontal systems known to play a critical role in motor control and action selection, where the BG provide a ‘Go’ versus ‘NoGo’ modulation of frontal action representations. In our model, the BG modulate working memory representations in prefrontal areas to support more abstract executive functions. We have developed a computational model of this system that is capable of developing human-like performance on working memory and executive control tasks through trial-and-error learning. This learning is based on reinforcement learning mechanisms associated with the midbrain dopaminergic system and its activation via the BG and amygdala. Finally, we briefly describe various empirical tests of this framework.
Introduction
There is widespread agreement that some regions of the brain play a larger role in controlling our overall behaviour than others, with a strong consensus that the prefrontal cortex (PFC) is a ‘central executive’ (e.g., Baddeley, 1986; Christoff et al., 2009; Conway et al., 2005; Duncan, 2001; Koechlin and Summerfield, 2007; Miller and Cohen, 2001; Shallice, 1988). However, this central executive label raises many more questions than it answers. How does the PFC know what actions or plans to select? How does experience influence the PFC? How do the specific neural properties of the PFC enable this kind of function, and how do these differ from those in other, non-executive areas? Without answers to these kinds of questions, the notion of a central executive is tantamount to positing a homunculus (small man) living inside the PFC and controlling our actions.
- Type
- Chapter
- Information
- Modelling Natural Action Selection , pp. 239 - 263Publisher: Cambridge University PressPrint publication year: 2011
References
1986 Parallel organization of functionally segregated circuits linking basal ganglia and cortexAnnu. Rev. Neurosci. 9 357CrossRefGoogle ScholarPubMed
1998 A neuropsychological theory of multiple systems in category learningPsychol. Rev. 105 442CrossRefGoogle ScholarPubMed
2005 An integrative theory of locus coeruleus-norepinephrine function: adaptive gain and optimal performanceAnnu. Rev. Neurosci. 28 403CrossRefGoogle ScholarPubMed
2009 Is the rostro-caudal axis of the frontal lobe hierarchicalNature reviews 10 659CrossRefGoogle ScholarPubMed
2009 Hierarchical cognitive control deficits following damage to the human frontal lobeNature Neurosci. 12 515CrossRefGoogle ScholarPubMed
1998 Goal-directed instrumental action: contingency and incentive learning and their cortical substratesNeuropharmacology 37 407CrossRefGoogle ScholarPubMed
1998 A computational model of how the basal ganglia produces sequencesJ. Cogn. Neurosci. 10 108CrossRefGoogle Scholar
2004 Parameterization of connectionist modelsBehav. Res. Meth. Inst. Comp. 36 732CrossRefGoogle ScholarPubMed
2004 Conflict monitoring and anterior cingulate cortex: an updateTrends Cogn. Sci. 8 539CrossRefGoogle ScholarPubMed
2009 Hierarchically organized behavior and its neural foundations: A reinforcement learning perspectiveCognition 113 262CrossRefGoogle ScholarPubMed
1999 Cognition and control in schizophrenia: a computational model of dopamine and prefrontal functionBiol. Psych. 46 312CrossRefGoogle ScholarPubMed
2000 On the control of control: the role of dopamine in regulating prefrontal function and working memoryControl of Cognitive Processes: Attention and Performance, XVIIICambridge, MAMIT Press713Google Scholar
1997 A parametric study of frontal cortex involvement in human working memoryNeuroImage 5 49CrossRefGoogle ScholarPubMed
1995 A computational model of prefrontal cortex functionAdvances in Neural Information Processing SystemsCambridge, MAMIT PressGoogle Scholar
2005 Learned predictions of error likelihood in the anterior cingulate cortexScience 307 1118CrossRefGoogle ScholarPubMed
1999 How the basal ganglia use parallel excitatory and inhibitory learning pathways to selectively respond to unexpected rewarding cuesJ. Neurosci 19 10502CrossRefGoogle ScholarPubMed
2004 How laminar frontal cortex and basal ganglia circuits interact to control planned and reactive saccadesNeural Networks 17 471CrossRefGoogle ScholarPubMed
2009 Left, but not right, rostrolateral prefrontal cortex meets a stringent test of the relational integration hypothesisNeuroImage 46 338CrossRefGoogle Scholar
1999 Memory for serial order: a network model of the phonological loop and its timingPsychol. Rev. 106 551CrossRefGoogle Scholar
2009 Pupillometric and behavioral markers of a developmental shift in the temporal dynamics of cognitive controlProc. Nat. Acad. Sci. USA 106 5529CrossRefGoogle ScholarPubMed
2000 The frontopolar cortex and human cognition: evidence for a rostrocaudal hierarchical organization within the human prefrontal cortexPsychobiology 28 168Google Scholar
2009 Prefrontal organization of cognitive control according to levels of abstractionBrain Res. 1286 94CrossRefGoogle ScholarPubMed
2007 Cortical mechanisms of action selection: the affordance competition hypothesisPhil. Trans. Roy. Soc. B Biol. Sci. 362 1585CrossRefGoogle ScholarPubMed
2002 Computational perspectives on dopamine function in prefrontal cortexCurr. Opin. Neurobiol. 12 223CrossRefGoogle ScholarPubMed
1997 A computational approach to prefrontal cortex, cognitive control and schizophrenia: recent developments and current challengesPhil. Trans. Roy. Soc. B. Biol. Sci. 351 1515CrossRefGoogle Scholar
1990 On the control of automatic processes: a parallel distributed processing model of the Stroop effectPsychol. Rev. 97 332CrossRefGoogle ScholarPubMed
1997 Temporal dynamics of brain activity during a working memory taskNature 386 604CrossRefGoogle Scholar
1992 Context, cortex, and dopamine: a connectionist approach to behavior and biology in schizophreniaPsychol. Rev. 99 45CrossRefGoogle Scholar
1992 A parallel distributed processing approach to auomaticityAm. J. Psychol. 105 239CrossRefGoogle Scholar
1999 A predictive reinforcement model of dopamine neurons for learning approach behaviorJ. Comput. Neurosci 6 191CrossRefGoogle ScholarPubMed
2005 Working memory capacity and its relation to general intelligenceTrends Cogn. Sci. 7 547CrossRefGoogle Scholar
2007 Bilinearity, rules, and prefrontal cortexFront. Comput. Neurosci. 1CrossRefGoogle ScholarPubMed
1995 A model of corticostriatal plasticity for learning oculomotor associations and sequencesJ. Cogn. Neurosci. 7 311CrossRefGoogle ScholarPubMed
2001 An adaptive coding model of neural function in prefrontal cortexNature Rev. Neurosci. 2 820CrossRefGoogle ScholarPubMed
1999 A neurocomputational theory of the dopaminergic modulation of working memory functionsJ. Neurosci. 19 2807CrossRefGoogle ScholarPubMed
2002 The computational role of dopamine D1 receptors in working memoryNeural networks 15 561CrossRefGoogle ScholarPubMed
2000 Dopamine-mediated stabilization of delay-period activity in a network model of prefrontal cortexJ. Neurophysiol. 83 1733CrossRefGoogle Scholar
2005 Dynamic dopamine modulation in the basal ganglia: a neurocomputational account of cognitive deficits in medicated and nonmedicated ParkinsonismJ. Cogn. Neurosci. 17 51CrossRefGoogle ScholarPubMed
2006 Hold your horses: a dynamic computational role for the subthalamic nucleus in decision makingNeural Networks 19 1120CrossRefGoogle ScholarPubMed
2006 Anatomy of a decision: striatoorbitofrontal interactions in reinforcement learning, decision making, and reversalPsychol. Rev. 113 300CrossRefGoogle ScholarPubMed
2001 Interactions between the frontal cortex and basal ganglia in working memory: a computational modelCogn., Affect. Behav. Neurosci. 1 137CrossRefGoogle ScholarPubMed
2006 A mechanistic account of striatal dopamine function in human cognition: psychopharmacological studies with cabergoline and haloperidolBehav. Neurosci. 120 497CrossRefGoogle ScholarPubMed
2007 Hold your horses: impulsivity, deep brain stimulation, and medication in parkinsonismScience 318 1309CrossRefGoogle ScholarPubMed
2007 Testing computational models of dopamine and noradrenaline dysfunction in attention deficit/hyperactivity disorderNeuropsychopharmacol. 32 1583CrossRefGoogle ScholarPubMed
2007 Understanding decision-making deficits in neurological conditions: insights from models of natural action selectionPhil. Trans. Roy. Soc. B. Biol. Sci. 362 1641CrossRefGoogle ScholarPubMed
2004 By carrot or by stick: cognitive reinforcement learning in ParkinsonismScience 306 1940CrossRefGoogle ScholarPubMed
1989 Mnemonic coding of visual space in the monkey's dorsolateral prefrontal cortexJ. Neurophysiol. 61 331CrossRefGoogle ScholarPubMed
1989 The Prefrontal Cortex: Anatomy, Physiology and Neuropsychology of the Frontal LobeNew YorkRaven PressGoogle Scholar
1995 Adaptive neural networks in the basal gangliaModels of Information Processing in the Basal GangliaCambridge, MAMIT Press103Google Scholar
2001 A computational model of action selection in the basal ganglia. I. A new functional anatomyBiol. Cybern 84 401CrossRefGoogle ScholarPubMed
2007 Towards an executive without a homunculus: computational models of the prefrontal cortex/basal ganglia systemPhil. Trans. Roy. Soc. B. Biol. Sci. 362 105CrossRefGoogle ScholarPubMed
2006 Banishing the homunculus: Making working memory workNeuroscience 139 105CrossRefGoogle ScholarPubMed
2010 Neural mechanisms of acquired phasic dopamine responses in learningNeurosci. Biobehav. Rev. 34 701CrossRefGoogle Scholar
2006 Neural mechanisms of cognitive control: an integrative model of Stroop task performance and FMRI dataJ. Cogn. Neurosci 18 22CrossRefGoogle ScholarPubMed
1997 Neurochemical architecture of the human striatumJ. Comp. Neurol. 384 13.0.CO;2-5>CrossRefGoogle ScholarPubMed
1995 A model of how the basal ganglia generate and use neural signals that predict reinforcementModels of Information Processing in the Basal, GangliaCambridge, MAMIT Press233Google Scholar
2007 Action selection and refinement in subcortical loops through basal ganglia and cerebellumPhil. Trans. Roy. Soc. B. Biol. Sci. 362 1573CrossRefGoogle ScholarPubMed
1995 Distributed modular architectures linking basal ganglia, cerebellum, and cerebral cortex: their role in planning and controlling actionCereb. Cortex 5 95CrossRefGoogle ScholarPubMed
1996 The representation of temporal information in perception and motor controlCurr. Opin. Neurobiol. 6 851CrossRefGoogle ScholarPubMed
2008 SAL: an explicitly pluralistic cognitive architectureJ. Exp. Theor. Artif. Intell 9 197CrossRefGoogle Scholar
2002 Actor-critic models of the basal ganglia: new anatomical and computational perspectivesNeural Networks 15 535CrossRefGoogle ScholarPubMed
2007 Genetically determined differences in learning from errorsScience 318 1642CrossRefGoogle ScholarPubMed
2007 An information theoretical approach to prefrontal executive functionTrends Cogn. Sci. 11 229CrossRefGoogle ScholarPubMed
1971 Prefrontal cortical unit activity and delayed alternation performance in monkeysJ. Neurophysiol. 34 337CrossRefGoogle ScholarPubMed
1993 Topography of pyramidal neuron intrinsic connections in macaque monkey prefrontal cortex (Areas 9 46)J. Comp. Neurol. 338 360CrossRefGoogle Scholar
1999 A role for NMDA-receptor channels in working memoryNature Neurosci. 1 273CrossRefGoogle Scholar
2000 Basal ganglia output and cognition: evidence from anatomical, behavioral, and clinical studiesBrain Cogn. 42 183CrossRefGoogle ScholarPubMed
2001 An integrative theory of prefrontal cortex functionAnnu. Rev. Neurosci. 24 167CrossRefGoogle ScholarPubMed
1996 Neural mechanisms of visual working memory in prefontal cortex of the macaqueJ. Neurosci. 16CrossRefGoogle Scholar
1996 The basal ganglia: focused selection and inhibition of competing motor programsProg. Neurobiol. 50 381CrossRefGoogle ScholarPubMed
1988 Neuronal correlate of pictorial short-term memory in the primate temporal cortexNature 331 68CrossRefGoogle ScholarPubMed
1997 A framework for mesencephalic dopamine systems based on predictive Hebbian learningJ. Neurosci. 16 1936CrossRefGoogle Scholar
2008 A role for dopamine in temporal decision making and reward maximization in ParkinsonismJ. Neurosci. 28 12294CrossRefGoogle ScholarPubMed
2003 Developmental and computational neuroscience approaches to cognition: the case of generalizationCogn. Stud. 10 76Google Scholar
1996 Biologically plausible error-driven learning using local activation differences: the generalized recirculation algorithmNeural Comput. 8 895CrossRefGoogle Scholar
2006 Biologically based computational models of high level cognitionScience 314 91CrossRefGoogle ScholarPubMed
1999 A biologically based computational model of working memoryModels of Working Memory: Mechanisms of Active Maintenance and Executive ControlNew YorkCambridge University Press375CrossRefGoogle Scholar
2006 Making working memory work: a computational model of learning in the prefrontal cortex and basal gangliaNeural Comput. 18 283CrossRefGoogle ScholarPubMed
2007 PVLV: The primary value and learned value Pavlovian learning algorithmBehav. Neurosci. 121 31CrossRefGoogle ScholarPubMed
2000 Computational Explorations in Cognitive Neuroscience: Understanding the Mind by Simulating the BrainCambridge, MAThe MIT PressGoogle Scholar
2002 Prefrontal cortex and dynamic categorization tasks: representational organization and neuromodulatory controlCereb. Cortex 12 246CrossRefGoogle ScholarPubMed
2010 Integrating what and how/where with instrumental and Pavlovian learning: a biologically-based computational modelCognition and Neuropsychology: International Perspectives on Psychological ScienceOxfordPsychology PressGoogle Scholar
2009
2006 A robot model of the basal ganglia: behavior and intrinsic processingNeural Networks 19 31CrossRefGoogle ScholarPubMed
1997 Patterns of intrinsic and associational circuitry in monkey prefrontal cortexJ. Compar. Neurol. 376 6143.0.CO;2-4>CrossRefGoogle Scholar
1999 The basal ganglia: a vertebrate solution to the selection problemNeuroscience 89CrossRefGoogle ScholarPubMed
1972 A theory of Pavlovian conditioning: variation in the effectiveness of reinforcement and non-reinforcementClassical Conditioning II: Theory and ResearchNew YorkAppleton-Century-Crofts64Google Scholar
2009 Developing PFC representations using reinforcement learningCognition 113 201CrossRefGoogle ScholarPubMed
2005 Prefrontal cortex and the flexibility of cognitive control: rules without symbolsProc. Nat. Acad. Sci. 102 7338CrossRefGoogle Scholar
2002 Learning representations in a gated prefrontal cortex model of dynamic task switchingCogn. Sci. 26 503CrossRefGoogle Scholar
2006 Behavioral inhibition and prefrontal cortex in decision-makingNeural Networks 19 1255CrossRefGoogle ScholarPubMed
1993 Responses of monkey dopamine neurons to reward and conditioned stimuli during successive steps of learning a delayed response taskJ. Neurosci. 13 900CrossRefGoogle ScholarPubMed
1995 Reward-related signals carried by dopamine neuronsModels of Information Processing in the Basal, GangliaCambridge, MAMIT Press233Google Scholar
2004 The principal features and mechanisms of dopamine modulation in the prefrontal cortexProg. Neurobiol. 74 1CrossRefGoogle ScholarPubMed
1988 From Neuropsychology to Mental StructureNew YorkCambridge University PressCrossRefGoogle Scholar
2007 Biologically constrained action selection improves cognitive control in a model of the Stroop taskPhil. Trans. Roy. Soc. B. Biol. Sci. 362 1671CrossRefGoogle Scholar
2007 Basolateral amygdala lesions abolish orbitofrontal dependent reversal impairmentsNeuron 54 51CrossRefGoogle ScholarPubMed
2001 Modeling functions of striatal dopamine modulation in learning and planningNeuroscience 103 65CrossRefGoogle ScholarPubMed
1988 Learning to predict by the method of temporal differencesMach. Learn. 3 9CrossRefGoogle Scholar
2002 Dopamine controls fundamental cognitive operations of multi-target spatial working memoryNeural Networks 15 573CrossRefGoogle ScholarPubMed
2008 Role of the lateral prefrontal cortex in executive behavioral controlPhysiol. Rev 88 37CrossRefGoogle ScholarPubMed
2008 Learning to use working memory in partially observable environments through dopaminergic reinforcementAdvances in Neural Information Processing Systems (NIPS) 21Norwich, UKCurran AssociatesGoogle Scholar
2007 Selective reinforcement learning deficits in schizophrenia support predictions from computational models of striatal-cortical dysfunctionBiol. Psych. 62 756CrossRefGoogle ScholarPubMed
1999 Synaptic basis of cortical persistent activity: the importance of NMDA receptors to working memoryJ. Neurosci. 19CrossRefGoogle ScholarPubMed
1995 Effects of local connectivity on striatal function: simulation and analysis of a modelSynapse 20 281CrossRefGoogle ScholarPubMed
1960
2008 Forward frontal fields: phylogeny and fundamental functionTrends Neurosci. 31 599CrossRefGoogle ScholarPubMed
2003 Human prefrontal cortex: processing and representational perspectivesNature Rev. Neurosci. 4 139CrossRefGoogle ScholarPubMed
2004 The neural basis of error detection: conflict monitoring and the error-related negativityPsychol. Rev. 111 931CrossRefGoogle ScholarPubMed
1991 Recurrent network model of the neural mechanism of shortterm active memoryNeural Comput. 3 179CrossRefGoogle Scholar