Skip to main content Accessibility help

NIH Toolbox Cognition Battery (NIHTB-CB): List Sorting Test to Measure Working Memory

  • David S. Tulsky (a1) (a2), Noelle Carlozzi (a3), Nancy D. Chiaravalloti (a2), Jennifer L. Beaumont (a4), Pamela A. Kisala (a5), Dan Mungas (a6), Kevin Conway (a7) and Richard Gershon (a4)...

The List Sorting Working Memory Test was designed to assess working memory (WM) as part of the NIH Toolbox Cognition Battery. List Sorting is a sequencing task requiring children and adults to sort and sequence stimuli that are presented visually and auditorily. Validation data are presented for 268 participants ages 20 to 85 years. A subset of participants (N=89) was retested 7 to 21 days later. As expected, the List Sorting Test had moderately high correlations with other measures of working memory and executive functioning (convergent validity) but a low correlation with a test of receptive vocabulary (discriminant validity). Furthermore, List Sorting demonstrates expected changes over the age span and has excellent test–retest reliability. Collectively, these results provide initial support for the construct validity of the List Sorting Working Memory Measure as a measure of working memory. However, the relationship between the List Sorting Test and general executive function has yet to be determined. (JINS, 2014, 20, 1–12)

Corresponding author
Correspondence and reprint requests to: David S. Tulsky, Assessment Research and Translation, Rusk Institute/Department of Rehabilitation Medicine, New York University Langone Medical Center, Ambulatory Care Center 240 E. 38th Street, 17th Floor, New York, New York 10016. E-mail:
Hide All
Alloway, T.P., Gathercole, S.E., & Pickering, S.J. (2006). Verbal and visuospatial short-term and working memory in children: Are they separable? Child Development, 77(6), 16981716.
Awh, E., Jonides, J., Smith, E.E., Schumacher, E.H., Koeppe, R.A., & Katz, S. (1996). Dissociation of storage and rehearsal in verbal working memory: Evidence from positron emission tomography. Psychological Science, 7(1), 2531.
Baddeley, A. (1986). Working memory. Oxford, England: Oxford University Press.
Baddeley, A. (1987). Working memory (Vol. 11). Gloucestershire: Clarendon Press.
Baddeley, A. (1992). Working memory. Science, 255(5044), 556559.
Baddeley, A. (1996). Exploring the central executive. Quarterly Journal of Experimental Psychology, 49A, 528.
Baddeley, A. (2000). The episodic buffer: A new component of working memory? Trends in Cognitive Sciences, 4(11), 417423.
Baddeley, A. (2001). Is working memory still working? American Psychologist, 56(11), 851864.
Baddeley, A. (2002). Is working memory still working? European Psychologist, 7(2), 8597.
Baddeley, A. (2010). Working Memory. Current Biology, 20(4), R136R140.
Baddeley, A., & Hitch, G.J. (1974). Working Memory. In G.H. Bower (Ed.), The psychology of learning and motivation: Advances in research and theory (Vol. 8, pp. 4790). San Diego, CA: Academic Press.
Beatty, W.W., Wilbanks, S.L., Blanco, C.R., Hames, K.A., Tivis, R., & Paul, R.H. (1996). Memory disturbance in multiple sclerosis: Reconsideration of patterns of performance on the selective reminding test. Journal of Clinical and Experimental Neuropsychology, 18(1), 5662. doi:10.1080/01688639608408262
Belger, A., Puce, A., Krystal, J.H., Gore, J.C., Goldman-Rakic, P., & McCarthy, G. (1998). Dissociation of mnemonic and perceptual processes during spatial and nonspatial working memory using fMRI. Human Brain Mapping, 6(1), 1432.
Braver, T.S., Cohen, J.D., Nystrom, L.E., Jonides, J., Smith, E.E., & Noll, D.C. (1997). A parametric study of prefrontal cortex involvement in human working memory. Neuroimage, 5, 4962.
Chiaravalloti, N., Hillary, F., Ricker, J., Christodoulou, C., Kalnin, A., Liu, W.C., DeLuca, J. (2005). Cerebral activation patterns during working memory performance in multiple sclerosis using FMRI. Journal of Clinical and Experimental Neuropsychology, 27(1), 3354.
Cohen, J. (1992). A power primer. Psychological Bulletin, 112, 155159.
Conlin, J.A., Gathercole, S.E., & Adams, J.W. (2005). Children’s working memory: Investigating performance limitations in complex span tasks. Journal of Experimental Child Psychology, 90(4), 303317.
Conway, A.R., Kane, M.J., Bunting, M.F., Hambrick, D.Z., Wilhelm, O., & Engle, R.W. (2005). Working memory span tasks: A methodological review and user’s guide. Psychonomic Bulletin & Review, 12(5), 769786.
Courtney, S.M., Ungerleider, L.G., Keil, K., & Haxby, J.V. (1997). Transient and sustained activity in a distributed neural system for human working memory. Nature, 386(6625), 608611.
Cronbach, L.J., & Meehl, P.E. (1955). Construct validity in psychological tests. Psychological Bulletin, 52, 281302.
Crone, E.A., Wendelken, C., Donohue, S., van Leijenhorst, L., & Bunge, S.A. (2006). Neurocognitive development of the ability to manipulate information in working memory. Proceedings of the National Academy of Sciences of the United States of America, 103(24), 93159320.
Crosson, B., Rao, S.M., Woodley, S.J., Rosen, A.C., Bobholz, J.A., Mayer, A., Stein, E.A. (1999). Mapping of semantic, phonological, and orthographic verbal working memory in normal adults with functional magnetic resonance imaging. Neuropsychology, 13(2), 171187.
Crowe, S.F. (2000). Does the letter number sequencing task measure anything more than digit span? Assessment, 7(2), 113117.
Curtis, C.E., & D’Esposito, M. (2003). Persistent activity in the prefrontal cortex during working memory. Trends in Cognitive Sciences, 7(9), 415423.
D’Esposito, M., Aguirre, G.K., Zarahn, E., Ballard, D., Shin, R.K., & Lease, J. (1998). Functional MRI studies of spatial and nonspatial working memory. Cognitive Brain Research, 7(1), 113.
D’Esposito, M., Onishi, K., Thompson, H., Robinson, K., Armstrong, C., & Grossman, M. (1996). Working memory impairments in multiple sclerosis. Neuropsychology, 10, 5156.
de Jong, P.F. (1998). Working memory deficits of reading disabled children. Journal of Experimental Child Psychology, 70(2), 7596.
de Jong, P.F., & Olson, R.K. (2004). Early predictors of letter knowledge. Journal of Experimental Child Psychology, 88(3), 254273.
Demaree, H.A., DeLuca, J., Gaudino, E.A., & Diamond, B.J. (1999). Speed of information processing as a key deficit in multiple sclerosis: Implications for rehabilitation. Journal of Neurology, Neurosurgery, & Psychiatry, 67(5), 661663.
Dempster, F.N. (1981). Memory span - Sources of individual and developmental differences. Psychological Bulletin, 89(1), 63100.
Donders, J., Tulsky, D.S., & Zhu, J. (2001). Criterion validity of new WAIS-II subtest scores after traumatic brain injury. Journal of the International Neuropsychological Society, 7(7), 892898.
Dunn, L.M., & Dunn, D.M. (2007). Peabody Picture Vocabulary Test - Fourth edition. Minneapolis, MN: NCS Pearson.
Edin, F., Klingberg, T., Johansson, P., McNab, F., Tegner, J., & Compte, A. (2009). Mechanism for top-down control of working memory capacity. Proceedings of the National Academy of Sciences of the United States of America, 106(16), 68026807.
Edin, F., Macoveanu, J., Olesen, P., Tegner, J., & Klingberg, T. (2007). Stronger synaptic connectivity as a mechanism behind development of working memory-related brain activity during childhood. Journal of Cognitive Neuroscience, 19(5), 750760.
Engle, R.W. (1996). Working memory and retrieval: An inhibition-resource approach. In J. Richardson (Ed.), Working memory and human cognition (pp. 89119). New York: Oxford University Press.
Fuster, J.M. (1989). The prefrontal cortex. New York: Raven Press.
Gathercole, S.E., Pickering, S.J., Ambridge, B., & Wearing, H. (2004). The structure of working memory from 4 to 15 years of age. Developmental Psychology, 40(2), 177190.
Gershon, R.C., Cella, D., Fox, N.A., Havlik, R.J., Hendrie, H.C., & Wagster, M.V. (2010). Assessment of neurological and behavioural function: The NIH Toolbox. Lancet Neurology, 9(2), 138139.
Gershon, R.C., Wagster, M.V., Hendrie, H.C., Fox, N.A., Cook, K.F., & Nowinski, C.J. (2013). NIH Toolbox for assessment of neurological and behavioral function. Neurology, 80(Suppl. 3), S2S6.
Gold, J.M., Carpenter, C., Randolph, C., Goldberg, T.E., & Weinberger, D.R. (1997). Auditory working memory and Wisconsin Card Sorting Test performance in schizophrenia. Archives of General Psychiatry, 54(2), 159165.
Goldman-Rakic, P.S. (1987). Circuitry of primate prefrontal cortex and regulation of behavior by representational memory. In V.B. Mountcastle (Ed.), Handbook of physiology (pp. 373417). Bethesda, MD: American Psychological Society.
Grigsby, J., Ayarbe, S., Kravcisin, N., & Busenbark, D. (1994). Working memory impairment among persons with chronic-progressive multiple sclerosis. Journal of Neurology, 241(3), 125131.
Grigsby, J., Busenbark, D., Kravcisin, N., Kennedy, P.M., & Taylor, D. (1994). Impairment of the working memory system in relapsing-remitting multiple sclerosis. Archives of Clinical Neuropsychology, 9, 134135.
Gronwall, D.M. (1977). Paced auditory serial-addition task: A measure of recovery from concussion. Perceptual & Motor Skills, 44(2), 367373.
Grossman, M., Armstrong, C., Onishi, K., Thompson, H., Schaefer, B., Robinson, K., Silberberg, D. (1994). Patterns of cognitive impairment in relapsing-remitting and chronic progressive multiple-sclerosis. Neuropsychiatry, Neuropsychology, and Behavioral Neurology, 7(3), 194210.
Hasher, L., Stolzfus, E.R., Zacks, R.T., & Rypma, B. (1991). Age and inhibition. Journal of Experimental Psychology: Learning, Memory, and Cognition, 17(1), 163169.
Haut, M.W., Kuwabara, H., Leach, S., & Arias, R.G. (2000). Neural activation during performance of number-letter sequencing. Applied Neuropsychology, 7(4), 237242.
Hawkins, K.A. (1998). Indicators of brain dysfunction derived from graphic representations of the WAIS-III/WMS-III Technical Manual clinical samples data: A preliminary approach to clinical utility. The Clinical Neuropsychologist, 12(4), 535551.
Heaton, R.K., Taylor, M.J., & Manly, J. (2003). Demographic effects and use of demographically corrected norms with the WAIS-III and WMS-III. In D.S. Tulsky (Ed.), Clinical interpretation of the WAIS-III and WMS-III (pp. 181210). San Diego: Academic Press.
Hitch, G.J., Towse, J.N., & Hutton, U. (2001). What limits children’s working memory span? Theoretical accounts and applications for scholastic development. Journal of Experimental Psychology-General, 130(2), 184198.
Hodes, R.J., Insel, T.R., & Landis, S.C. (2013). The NIH Toolbox: Setting a standard for biomedical research. Neurology, 80(Suppl. 3), S1.
Hofmann, W., Schmeichel, B.J., & Baddeley, A.D. (2012). Executive functions and self-regulation. Trends in Cognitive Sciences, 16(3), 174180. doi:10.1016/j.tics.2012.01.006.
Jacobsen, C.F. (1935). Functions of frontal association area in primates. Archives of Neurology and Psychiatry, 33(3), 558569.
Jonides, J. (1995). Working memory and thinking. In E.E. Smith & D.N. Osherson (Eds.), Invitation to cognitive science: Thinking (2nd ed., Vol. 3, pp. 215265). Cambridge, MA: MIT Press.
Kirasic, K., Allen, G., Dobson, S., & Binder, K. (1996). Aging, cognitive resources, and declarative learning. Psychology and Aging, 11, 658670.
Klingberg, T., Forssberg, H., & Westerberg, H. (2002). Increased brain activity in frontal and parietal cortex underlies the development of visuospatial working memory capacity during childhood. Journal of Cognitive Neuroscience, 14(1), 110.
Klingberg, T., Kawashima, R., & Roland, P.E. (1996). Activation of multi-modal cortical areas underlies short-term memory. European Journal of Neuroscience, 8(9), 19651971.
Kongs, S.K., Thompson, L.L., Iverson, G.L., & Heaton, R. (2000). Wisconsin card sorting test - 64 card version: Professional manual. Odessa, FL: Psychological Assessment Resources.
Kwon, H., Reiss, A.L., & Menon, V. (2002). Neural basis of protracted developmental changes in visuo-spatial working memory Proceedings of the National Academy of Sciences of the United States of America, 99(20), 1333613341.
Kyllonen, P.C., & Christal, R.E. (1990). Reasoning ability is (little more than) working-memory capacity? Intelligence, 14, 389433.
Lehto, J. (1996). Are executive function tests dependent on working memory capacity? The Quarterly Journal of Experimental Psychology, 49A(1), 2950.
Linden, D.E. (2007). The working memory networks of the human brain. Neuroscientist, 13(3), 257267.
Litvan, I., Grafman, J., Vendrell, P., & Martinez, J.M. (1988). Slowed information processing in multiple sclerosis. Archives of Neurology, 45(3), 281285.
Logie, R.H. (1996). The seven ages of working memory. In J. Richardson (Ed.), Working memory and human cognition (pp. 3165). New York: Oxford University Press.
Manoach, D.S., Schlaug, G., Siewert, B., Darby, D.G., Bly, B.M., Benfield, A., Warach, S. (1997). Prefrontal cortex fMRI signal changes are correlated with working memory load. Neuroreport, 8(2), 545549.
Martin, T. A., Donders, J., & Thompson, E. (2000). Potential of and problems with new measures of psychometric intelligence after traumatic brain injury. Rehabilitation Psychology, 45(4), 402408.
McAllister, T.W., Flashman, L.A., Sparling, M.B., & Saykin, A.J. (2004). Working memory deficits after traumatic brain injury: Catecholaminergic mechanisms and prospects for treatment -- A review. Brain Injury, 18(4), 331350.
McAllister, T.W., Saykin, A.J., Flashman, L.A., Sparling, M.B., Johnson, S.C., Guerin, S.J., Yanofsky, N. (1999). Brain activation during working memory 1 month after mild traumatic brain injury: A functional MRI study. Neurology, 53(6), 13001308.
McAuley, T., & White, D.A. (2011). A latent variables examination of processing speed, response inhibition, and working memory during typical development. Journal of Experimental Child Psychology, 108(3), 453468.
McCabe, D.P., Roediger, H.L., McDaniel, M.A., Balota, D.A., & Hambrick, D.Z. (2010). The relationship between working memory capacity and executive functioning: Evidence for a common executive attention construct. Neuropsychology, 24, 222243.
Messick, S. (1980). Test validity and the ethics of assessment. American Psychologist, 35(11), 10121027.
Messick, S. (1991). Validity of test interpretation and use. In M.C. Alkin (Ed.), Encyclopedia of educational research (6th ed.), New York: Macmillan.
Miller, G.A. (1956). The magical number seven plus or minus two: Some limits on our capacity for processing information. Psychological Review, 63(2), 8197.
Milner, B. (1964). Some effects of frontal lobectomy in man. In J.M. Warren & K. Akert (Eds.), The frontal granular cortex and behavior (pp. 313334). New York: McGraw-Hill.
Miyake, A., Friedman, N.P., Emerson, M.J., Witzki, A.H., Howerter, A., & Wager, T.D. (2000). The unity and diversity of executive functions and their contributions to complex “frontal lobe” tasks: A latent variable analysis. Cognitive Psychology, 41, 49100.
Miyake, A., & Shah, P. (Eds.) (1999). Models of working memory: Mechanisms of active maintenance and executive control. New York: Cambridge University Press.
Mungas, D., Reed, B.R., Marshall, S.C., & Gonzalez, H.M. (2000). Development of psychometrically matched English and Spanish language neuropsychological tests for older persons. Neuropsychology, 14(2), 209223.
Newell, A. (1973). Productions systems: Models of control structures. In W.G. Chase (Ed.), Visual information processing (pp. 463526). New York: Academic Press.
Newell, A., & Simon, H.A. (1972). Human problem solving. Englewood Cliffs, NJ: Prentice-Hall.
Orellana, G., & Slachevsky, A. (2013). Executive functioning in schizophrenia. Frontiers in Psychiatry, 4, 35. doi:10.3389/fpsyt.2013.00035
Owen, A.M., Evans, A.C., & Petrides, M. (1996). Evidence for a two-stage model of spatial working memory processing within the lateral frontal cortex: A positron emission tomography study. Cerebral Cortex, 6(1), 3138.
Paulesu, E., Frith, C.D., & Frackowiak, R.S. (1993). The neural correlates of the verbal component of working memory. Nature, 362(6418), 342345.
Pelphrey, K.A., & Reznick, J.S. (2003). Working memory in infancy. Advances in Child Development and Behavior, 31, 173227.
Rao, S.M., Grafman, J., DiGiulio, D., Mittenberg, W., Bernardin, L., Leo, G.J., Unverzagt, F. (1993). Memory dysfunction in multiple sclerosis: Its relation to working memory, semantic encoding and implicit learning. Neuropsychology, 7(3), 364374.
Repovs, G., & Bresjanak, M. (2006). Cognitive neuroscience of working memory. Neuroscience, 139, 1413.
Riggs, K., McTaggart, J., & Simpson, A. (2006). Changes in the capacity of visual working memory in 5- to 10-year-olds. Journal of Experimental Child Psychology, 95, 1826.
Salmon, E., Van der Linden, M., Collette, F., Delfiore, G., Maquet, P., Degueldre, C., Franck, G. (1996). Regional brain activity during working memory tasks. Brain, 119(Pt 5), 16171625.
Salthouse, T. (1994). The aging of working memory. Neuropsychology, 8(4), 535543.
Salthouse, T., Mitchell, D.R., Skovronek, E., & Babcock, R.L. (1989). Effects of adult age and working memory on reasoning and spatial abilities. Journal of Experimental Psychology: Learning, Memory, and Cognition, 15(3), 507516.
Scherf, K.S., Sweeney, J.A., & Luna, B. (2006). Brain basis of developmental change in visuospatial working memory. Journal of Cognitive Neuroscience, 18(7), 10451058.
Schmiedek, F., Hildebrandt, A., Lovden, M., Lindenberger, U., & Wilhelm, O. (2009). Complex span versus updating tasks of working memory: The gap is not that deep. Journal of Experimental Psychology: Learning, Memory, and Cognition, 35(4), 10891096.
Seidman, L.J., Breiter, H.C., Goodman, J.M., Goldstein, J.M., Woodruff, P.W.R., & Rosen, B.R. (1998). A functional magnetic resonance imaging study of auditory vigilance with low and high information processing demands. Neuropsychology, 12(4), 505518.
Smith, E.E., Jonides, J., & Koeppe, R.A. (1996). Dissociating verbal and spatial working memory using PET. Cerebral Cortex, 6(1), 1120.
Strauss, E., Sherman, E.M.S., & Spreen, O. (2006). A compendium of neuropsychological tests: Administration, norms, and commentary (3rd ed.). New York: Oxford University Press.
Tulsky, D.S., Carlozzi, N.E., Chevalier, N., Espy, K., Beaumont, J., & Mungas, D. (2013). NIH Toolbox Cognitive Function Battery (CFB): Measuring working memory. Society For Research In Child Development, Monograph, 78(4), 7087.
Tulsky, D.S., Saklofske, D.H., & Zhu, J. (2003). Revising a Standard: Evaluation of the Origin and Development of the WAIS-III. Clinical Interpretation of the WAIS-III and WMS-III (pp. 4392). San Diego: Elsevier Science.
Van Snellenberg, J.X. (2009). Working memory and long-term memory deficits in schizophrenia: Is there a common substrate? Psychiatry Research, 174(2), 8996.
Vugs, B., Hendriks, M., Cuperus, J., & Verhoeven, L. (2014). Working memory performance and executive function behaviors in young children with SLI. Research in Developmental Disabilities, 35(1), 6274.
Wechsler, D. (1952). The range of human capacities. Baltimore: The Williams & Wilkins Company.
Wechsler, D. (1997). WAIS-III Administration and Scoring Manual. San Antonio: The Psychological Corporation.
Wechsler, D. (2008). Wechsler Adult Intelligence Scale IV. San Antonio: Harcourt Assessment Inc.
Weintraub, S., Dikmen, S.S., Heaton, R.K., Tulsky, D.S., Zelazo, P.D., Bauer, P.J., Gershon, R.C. (2013). Cognition assessment using the NIH Toolbox. Neurology, 80(11 Suppl. 3), S54S64.
Willmott, C., Ponsford, J., Hocking, C., & Schönberger, M. (2009). Factors contributing to attentional impairments after traumatic brain injury. Neuropsychology, 23(4), 424432.
Woods, S.P., Moore, D.J., Weber, E., & Grant, I. (2009). Cognitive neuropsychology of HIV-associated neurocognitive disorders. Neuropsychology Review, 19(2), 152168.
You, S.C., Geschwind, M.D., Sha, S.J., Apple, A., Satris, G., Wood, K.A., Possin, K.L. (2013). Executive functions in premanifest Huntington’s disease. Movement Disorders, 29(Suppl. 3), 405409.
Zelazo, P.D., Andersen, J., Richler, J., Wallner-Allen, K., Beaumont, J., & Weintraub, S. (2013). NIH Toolbox Cognitive Function Battery (CFB): Measuring executive function and attention. Society For Research In Child Development, Monograph, 78(4), 1633.
Recommend this journal

Email your librarian or administrator to recommend adding this journal to your organisation's collection.

Journal of the International Neuropsychological Society
  • ISSN: 1355-6177
  • EISSN: 1469-7661
  • URL: /core/journals/journal-of-the-international-neuropsychological-society
Please enter your name
Please enter a valid email address
Who would you like to send this to? *



Altmetric attention score

Full text views

Total number of HTML views: 0
Total number of PDF views: 0 *
Loading metrics...

Abstract views

Total abstract views: 0 *
Loading metrics...

* Views captured on Cambridge Core between <date>. This data will be updated every 24 hours.

Usage data cannot currently be displayed