Skip to main content
    • Aa
    • Aa

The case for the development and use of “ecologically valid” measures of executive function in experimental and clinical neuropsychology


This article considers the scientific process whereby new and better clinical tests of executive function might be developed, and what form they might take. We argue that many of the traditional tests of executive function most commonly in use (e.g., the Wisconsin Card Sorting Test; Stroop) are adaptations of procedures that emerged almost coincidentally from conceptual and experimental frameworks far removed from those currently in favour, and that the prolongation of their use has been encouraged by a sustained period of concentration on “construct-driven” experimentation in neuropsychology. This resulted from the special theoretical demands made by the field of executive function, but was not a necessary consequence, and may not even have been a useful one. Whilst useful, these tests may not therefore be optimal for their purpose. We consider as an alternative approach a function-led development programme which in principle could yield tasks better suited to the concerns of the clinician because of the transparency afforded by increased “representativeness” and “generalisability.” We further argue that the requirement of such a programme to represent the interaction between the individual and situational context might also provide useful constraints for purely experimental investigations. We provide an example of such a programme with reference to the Multiple Errands and Six Element tests. (JINS, 2006, 12, 194–209.)

Corresponding author
Reprint request to: Dr. Paul W. Burgess, Institute of Cognitive Neuroscience, University College London, 17 Queen Square, London, WC1E 6BT, UK. E-mail:
Linked references
Hide All

This list contains references from the content that can be linked to their source. For a full set of references and notes please see the PDF or HTML where available.

Alderman, N., Burgess, P.W., Knight, C., & Henman, C. (2003). Ecological validity of a simplified version of the multiple errands shopping test. Journal of the International Neuropsychological Society, 9, 3144.

Anderson, C.V., Bigler, E.D., & Blatter, D.D. (1995). Frontal lobe lesions, diffuse damage, and neuropsychological functioning in traumatic brain-injured patients. Journal of Clinical and Experimental Neuropsychology, 17, 900908.

Bergner, M., Bobbitt, R.A., Carter, W.B., & Gilson, B.S. (1981). The Sickness Impact Profile: Development and final revision of a health status measure. Medical Care, 19, 787805.

Bird, C.M., Castelli, F., Malik, O., Frith, U., & Husain, M. (2004). The impact of extensive medial frontal lobe damage on “Theory of Mind” and cognition. Brain, 127, 914928.

Bruce, V. & Young, A.W. (1986). Understanding face recognition. British Journal of Psychology, 77, 305327.

Burgess, P.W. (2000b). Strategy application disorder: The role of the frontal lobes in human multitasking. Psychological Research, 63, 279288.

Burgess, P.W., Alderman, N., Evans, J., Emslie, H., & Wilson, B.A. (1998). The ecological validity of tests of executive function. Journal of the International Neuropsychological Society, 4, 547558.

Burgess, P.W., Quayle, A., & Frith, C.D. (2001). Brain regions involved in propsective memory as determined by positron emission tomography. Neuropsychologia, 39, 545555.

Burgess, P.W., Scott, S.K., & Frith, C.D. (2003). The role of the rostral frontal cortex (area 10) in prospective memory: A lateral versus medial dissociation. Neuropsychologia, 41, 906918.

Burgess, P.W. & Shallice, T. (1996). Response suppression, initiation and strategy use following frontal lobe lesions. Neuropsychologia, 34, 263273.

Burgess, P.W., Veitch, E., Costello, A., & Shallice, T. (2000). The cognitive and neuroanatomical correlates of multitasking. Neuropsychologia, 38, 848863.

Channon, S. & Green, P.S.S. (1999). Executive function in depression: The role of performance strategies in aiding depressed and non-depressed participants. Journal of Neurology, Neurosurgery and Psychiatry, 66, 162171.

Channon, S., Pellijeff, A., & Rule, A. (2005). Social cognition after head injury: Sarcasm and theory of mind. Brain and Language, 93, 123134.

Chaytor, N. & Schmitter-Edgecombe, M. (2003). The ecological validity of neuropsychological tests: A review of the literature on everyday cognitive skills. Neuropsychology Review, 13, 181197.

Chorover, S.L. & Cole, M. (1966). Delayed alternation performance in patients with cerebral lesions. Neuropsychologia, 4, 17.

Clark, C., Prior, M., & Kinsella, G.J. (2000). Do executive function deficits differentiate between adolescents with ADHD and Oppositional Defiant/Conduct Disorder? A neuropsychological study using the Six Elements Test and Hayling Sentence Completion Test. Journal of Abnormal Child Psychology, 28, 403414.

Cohen, J.D., Dunbar, K., & McClelland, J.L. (1990). On the control of automatic processes: A parallel distributed processing account of the Stroop effect. Psychological Review, 97, 33261.

Cohen, J.D. & Servan-Schreiber, D. (1992). Context, cortex and doapmine: A connectionist approach to behaviour and biology in schizophrenia. Psychological Review, 99, 4577.

Coltheart, M. & Davies, M. (2003). Inference and explanation in cognitive neuropsychology. Cortex, 39, 18891.

Corcoran, R. & Upton, D. (1993). A role for the hippocampus in card sorting?Cortex, 29, 293304.

Dehaene, S., Kerszberg, M., & Changeux, J.-P. (1998). A neuronal model of a global workspace in effortful cognitive tasks. Proceedings of the National Academy of Science, 95, 14,52914,534.

Duchaine, B.C., Dingle, K., Butterworth, E., & Nakayama, K. (2004). Normal greeble learning in a severe case of developmental prosopagnosia. Neuron, 43, 469473.

Duchaine, B.C., Wendt, T.N., New, J., & Kulomaki, T. (2003). Dissociations of visual recognition in a developmental agnostic: Evidence for separate developmental processes. Neurocase, 9, 380389.

Duncan, J., Burgess, P.W., & Emslie, H. (1995). Fluid intelligence after frontal lobe lesions. Neuropsychologia, 33, 261268.

Eslinger, P.J. & Damasio, A.R. (1985). Severe disturbance of higher cognition after bilateral frontal lobe ablation: Patient E.V.R. Neurology, 35, 17311741.

Evans, J.J., Chua, S., McKenna, P.J., & Wilson, B.A. (1997). Assessment of the dysexecutive syndrome in schizophrenia. Psychological Medicine, 27, 635646.

Farias, S.T., Harrell, E., Neumann, C., & Houtz, A. (2003). The relationship between neuropsychological performance and daily functioning in individuals with Alzheimer's disease: Ecological validity of neuropsychological tests. Archives of Clinical Neuropsychology, 18, 655672.

Fortin, S., Godbout, L., & Braun, C.M.J. (2003). Cognitive structure of executive deficits in frontally lesioned head trauma patients performing activities of daily living. Cortex, 39, 273291.

Frith, C.D. & Frith, U. (2003). Development and neurophysiology of mentalizing. Philosophical Transactions of the Royal Society of London B, 358(1431), 459473.

Garden, S.E., Phillips, L.H., & MacPherson, S.E. (2001). Midlife aging, open-ended planning, and laboratory measures of executive function. Neuropsychology, 15, 472482.

Gilbert, S.J., Frith, C.D., & Burgess, P.W. (2005). Involvement of rostral prefrontal cortex in selection between stimulus-oriented and stimulus-independent thought. European Journal of Neuroscience, 21, 14231431.

Gioia, G.A. & Isquith, P.K. (2004). Ecological assessment of executive function in traumatic brain injury. Developmental Neuropsychology, 25, 135158.

Goel, V. & Grafman, J. (1995). Are the frontal lobes implicated in “planning” functions? Interpreting data from the Tower of Hanoi. Neuropsychologia, 33, 62342.

Goel, V. & Grafman, J. (2000). The role of the right prefrontal cortex in ill-structured problem solving. Cognitive Neuropsychology, 17, 415436.

Goel, V., Grafman, J., Tajik, J., Gana, S., & Danto, D. (1997). A study of the performance of patients with frontal lesions in a financial planning task. Brain, 120, 18051822.

Goldman-Rakic, P.S. (1995). Architecture of the prefrontal cortex and the central executive. Annals of the New York Academy of Science, 769, 212220.

Goldstein, L.H., Bernard, S., Fenwick, P.B.C., Burgess, P.W., & McNeil, J. (1993). Unilateral frontal lobectomy can produce strategy application disorder. Journal of Neurology, Neurosurgery and Psychiatry, 56, 274276.

Goldstein, K.H. & Scheerer, M. (1941). Abstract and concrete behaviour: An experimental study with special tests. Psychological Monographs, 53 (No. 2) (Whole No. 239).

Grant, D.A. & Berg, E.A. (1948). A behavioural analysis of degree or reinforcement and ease of shifting to new responses in a Weigl-type card sorting problem. Journal of Experimental Psychology, 38, 404411.

Greve, K.W., Brooks, J., Crouch, J.A., Williams, M.C., & Rice, W.J. (1997). Factorial structure of the Wisconsin Card Sorting Test. British Journal of Clinical Psychology, 36, 283285.

Harlow, J.M. (1868). Recovery from the passage of an iron bar through the head. Reprinted in Miller, E. (1993). History of Psychiatry, 4, 271278.

Higginson, C.I., Arnett, P.A., & Voss, W.D. (2000). The ecological validity of clinical tests of memory and attention in multiple sclerosis. Archives of Clinical Neuropsychology, 15, 185204.

Hodgson, T.L., Bajwa, A., Owen, A.M., & Kennard, C. (2000). The strategic control of gaze direction in the Tower-of-London task. Journal of Cognitive Neuroscience, 12, 894907.

Jacobsen, C.F. (1931). A study of cerebral function in learning: The frontal lobes. Journal of Comparative Neurology, 52, 271340.

Jelicic, M., Henquet, C.E., Derix, M.M., & Jolles, J. (2001). Test-retest stability of the behavioural assessment of the dysexecutive syndrome in a sample of psychiatric patients. International Journal of Neuroscience, 110, 738.

Kanwisher, N. (2000). Domain-specificity in face perception. Nature Neuroscience, 3, 759761.

Kibby, M.Y., Schmitter-Edgecombe, M., & Long, C.J. (1998). Ecological validity of neuropsychological tests: focus on the California Verbal Learning Test and the Wisconsin Card Sorting Test. Archives of Clinical Neuropsychology, 13, 523534.

Kingstone, A., Smilek, D., & Ristic, J. (2003). Attention, researchers! It is time to take a look at the real world. Current Directions in Psychological Science, 12, 176180.

Kirsh, D. (1995). The intelligent use of space. Artificial Intelligence, 73, 3168.

Kliegel, M., McDaniel, M.A., & Einstein, G.O. (2000). Plan formation, retention, and execution in prospective memory: A new approach and age-related effects. Memory and Cognition, 28, 10411049.

Knight, C., Alderman, N., & Burgess, P.W. (2002). Development of a simplified version of the multiple errands test for use in hospital settings. Neuropsychological Rehabilitation, 12, 231255.

Leichnetz, G.R. (1986). Afferent and efferent connections of the dorsolateral precentral areas (area 4, hand/arm region) in the macaque monkey, with comparisons to area 8. Journal of Comparative Neurology, 254, 460492.

Levine, B., Freedman, M., Dawson, D., Black, S.E., & Stuss, D.T. (1999). Ventral frontal contribution to self-regulation: Convergence of episodic memory and inhibition. Neurocase, 5, 263275. (Special issue on orbitofrontal cortex).

Levine, B., Robertson, I., Clare, L., Carter, G., Hong, J., Wilson, B.A., Duncan, J., & Stuss, D.T. (2000). Rehabilitation of executive functioning: An experimental–clinical validation of goal management training. Journal of the International Neuropsychological Society, 6, 299312.

MacSweeney, M., Campbell, R., Woll, B., Giampicto, V., David, A.S., McGuire, P.K., Calvert, G., & Brammer, M.J. (2004). Dissociating linguistic and non-linguistic gestural communication in the brain. NeuroImage, 22, 16051618.

Manchester, D., Priestly, N., & Jackson, H. (2004). The assessment of executive functions: Coming out of the office. Brain Injury, 18, 10671081.

Manly, T., Hawkins, K., Evans, J., Woldt, K., & Robertson, I.H. (2002). Rehabilitation of executive function: Facilitation of effective goal management on complex tasks using periodic auditory alerts. Neuropsychologia, 40, 271281.

Marenco, S., Coppola, R., Daniel, D.G., Zigun, J.R., & Weinberger, D.R. (1993). Regional cerebral blood flow during the Wisconsin Card Sorting Test in normal subjects studied by xenon-133 dynamic SPECT: Comparison of absolute values, percent distribution values, and covariance analysis. Psychiatry Research, 50, 177192.

McDonald, S., Flanagan, S., Martin, I., & Saunders, C. (2004). The ecological validity of TASIT: A test of social perception. Neuropsychological Rehabilitation, 14, 285302.

Mentzel, H.J., Gaser, C., Volz, H.P., Rzanny, R., Hager, F., Sauer, H., & Kaiser, W.A. (1998). Cognitive stimulation with the Wisconsin Card Sorting Test: Functional MR imaging at 1.5 T. Radiology, 207, 399404.

Miller, E. (1984). Verbal fluency as a measure of verbal intelligence and in relation to different types of cerebral pathology. British Journal of Clinical Psychology, 23, 5357.

Milner, B. (1963). Effects of different brain regions on card sorting. Archives of Neurology (Chicago), 9, 90100.

Miyake, A., Friedman, N.P., Emerson, M.J., Witzki, A.H., Howerter, A., & Wager, T.D. (2000). The unity and diversity of executive functions and their contributions to complex “Frontal Lobe” tasks: A latent variable analysis. Cognitive Psychology, 41, 49100.

Morris, R.G., Miotto, E.C., Feigenbaum, J.D., Bullock, P., & Polkey, C.E. (1997). The effect of goal-subgoal conflict on planning ability after frontal- and temporal-lobe lesions in humans. Neuropsychologia, 35, 114757.

Nelson, H.E. (1976). A modified card sorting test sensitive to frontal lobe defects. Cortex, 12, 313324.

Nilsson, J.P., Soderstrom, M., Karlsson, A.U., Lekander, M., Akerstedt, T., Lindroth, N.E., & Axelsson, J. (2005). Less effective executive functioning after one night's sleep deprivation. Journal of Sleep Research, 14, 16.

Odhuba, R.A., van den Broek, M.D., & Johns, L.C. (2005). Ecological validity of measures of executive functioning. British Journal of Clinical Psychology, 44, 269278.

Owen, A.M., Herrod, N.J., Menon, D.K., Clark, J.C., Downey, S.P., Carpenter, T.A., Minhas, P.S., Turkheimer, F.E., Willimas, E.J., Robbins, T.W., Sahakian, B.J., Petrides, M., & Pickard, J.D. (1999). Redefining the functional organization of working memory processes within human lateral prefrontal cortex. European Journal of Neuroscience, 11, 567574.

Penfield, W. & Evans, J. (1935). The frontal lobe in man: A clinical study of maximum removals. Brain, 58, 115133.

Perret, E. (1974). The left frontal lobe of man and the suppression of habitual responses in verbal categorical behaviour. Neuropsychologia, 12, 323330.

Ramnani, N. & Owen, A.M. (2004). Anterior prefrontal cortex: Insights into function from anatomy and neuroimaging. Nature Reviews Neuroscience, 5, 184194.

Ramnani, N., Toni, I., Passingham, R.E., & Haggard, P. (2001). The cerebellum and parietal cortex play a specific role in coordination: A PET study. NeuroImage, 14, 899911.

Rowe, J.B., Owen, A.M., Johnsrude, I.S., & Passingham, R.E. (2001). Imaging the mental components of a planning task. Neuropsychologia, 39, 315327.

Semkovska, M., Bedard, M.A., Godbout, L., Limoge, F., & Stip, E. (2004). Assessment of executive function during activities of daily living in schizophrenia. Schizophrenia Research, 69, 289300.

Shallice, T. (1982). Specific impairments of planning. Philosophical Transactions of the Royal Society of London B, 298, 199209.

Shallice, T. & Burgess, P.W. (1991a). Deficits in strategy application following frontal lobe damage in man. Brain, 114, 727741.

Shallice, T. & Burgess, P.W. (1996). Domains of supervisory control and the temporal organisation of behaviour. Philosophical Transactions of the Royal Society B, 351, 14051412. [Reprinted in A.C. Roberts, T.W. Robbins, & L. Weiskrantz (Eds.), The prefrontal cortex: Executive and cognitive functions, pp. 22–35. Oxford, U.K.: Oxford University Press, 1998.]

Shallice, T., Burgess, P.W., Schon, F., & Baxter, D.M. (1989). The origins of utilisation behaviour. Brain, 112, 15871598.

Shallice, T. & Evans, M.E. (1978). The involvement of the frontal lobes in cognitive estimation. Cortex, 14, 294303. (Reprinted in J. Davidoff (Ed.), Brain and behaviour: Critical concepts in psychology, Vol. 4, London: Routledge.)

Simons, J.S., Gilbert, S.J., Owen, A.M., Fletcher, P.C., & Burgess, P.W. (2005a). Distinct roles for lateral and medial anterior prefrontal cortex in contextual recollection. Journal of Neurophysiology, 94, 813820.

Simons, J.S., Owen, A.M., Fletcher, P.C., & Burgess, P.W. (2005b). Anterior prefrontal cortex and the recollection of internally-generated thoughts. Neuropsychologia, 43, 17741783.

Stroop, J.R. (1935). Studies of interference in serial verbal reaction. Journal of Experimental Psychology, 18, 643662.

Stuss, D.T., Binns, M.A., Murphey, K.J., & Alexander, M.P. (2002). Dissociations within the anterior attentional system: Effects of task complexity and irrelevant information on reaction time speed and accuracy. Neuropsychology, 16, 500513.

Stuss, D.T., Floden, D., Alexander, M.P., Levine, B., & Katz, D. (2001). Stroop performance in focal lesion patients: Dissociation of processes and frontal lobe lesion location. Neuropsychologia, 39, 771786.

Stuss, D.T., Levine, B., Alexander, M.P., Hong, J., Palumbo, C., Hamer, L., Murphy, K.J., & Izukawa, D. (2000). Wisconsin Card Sorting Test performance in patients with focal frontal and posterior brain damage: Effects of lesion location and test structure on separable cognitive processes. Neuropsychologia, 38, 388402.

Stuss, D.T., Shallice, T., Alexander, M.P., & Picton, T.W. (1995). A multidisciplinary approach to anterior attentional functions. Annals of the New York Academy of Sciences, 769, 191212.

Sukantarat, K.T., Burgess, P.W., Williamson, R.C.N., & Brett, S.J. (2005). Prolonged cognitive dysfunction in survivors of critical illness. Anaesthesia, 60, 847853.

van den Broek, M.D., Bradshaw, C.M., & Szabadi, E. (1993). Utility of the Modified Wisconsin Card Sorting Test in neuropsychological assessment. British Journal of Clinical Psychology, 32, 333343.

Welsh, M.C., Revilla, V., Strongin, D., & Kepler, M. (2000). Towers of Hanoi and London: Is the nonshared variance due to differences in task administration?Perceptual and Motor Skills, 90, 562572.

Welsh, M.C., Satterlee-Cartmell, T., & Stine, M. (1999). Towers of Hanoi and London: Contribution of working memory and inhibition to performance. Brain and Cognition, 41, 231234.

Wilson, B.A., Evans, J.J., Emslie, H., Alderman, N., & Burgess, P.W. (1998). The development of an ecologically valid test for assessing patients with a dysexecutive syndrome. Neuropsychological Rehabilitation, 8, 213228.

Zakzanis, K.K. & Young, D.A. (2001). Memory impairment in abstinent MDMA (“Ecstasy”) users: A longitudinal investigation. Neurology, 56, 966969.

Recommend this journal

Email your librarian or administrator to recommend adding this journal to your organisation's collection.

Journal of the International Neuropsychological Society
  • ISSN: 1355-6177
  • EISSN: 1469-7661
  • URL: /core/journals/journal-of-the-international-neuropsychological-society
Please enter your name
Please enter a valid email address
Who would you like to send this to? *



Altmetric attention score

Full text views

Total number of HTML views: 21
Total number of PDF views: 228 *
Loading metrics...

Abstract views

Total abstract views: 707 *
Loading metrics...

* Views captured on Cambridge Core between September 2016 - 26th June 2017. This data will be updated every 24 hours.