Skip to main content Accessibility help
  • Cited by 11
  • Print publication year: 2018
  • Online publication date: May 2018

12 - Capturing Expert Thought with Protocol Analysis: Concurrent Verbalizations of Thinking during Experts’ Performance on Representative Tasks

from Part III - Methods for Studying the Structure of Expertise
Abernethy, B., Neal, R. J., & Koning, P. (1994). Visual-perceptual and cognitive differences between expert, intermediate, and novice snooker players. Applied Cognitive Psychology, 18, 185211.
Afonso, J., Garganta, J., McRobert, A., Williams, A. M., & Mesquita, I. (2014). Visual search behaviours and verbal reports during film-based and in situ representative tasks in volleyball. European Journal of Sport Science, 14, 177184.
Anderson, J. R. (1983). The architecture of cognition. Cambridge, MA: Harvard University Press.
Anzai, Y., & Simon, H. A. (1979). The theory of learning by doing. Psychological Review, 86, 124140.
Arsal, G., Eccles, D. W., & Ericsson, K. A. (2016). Cognitive mediation of putting: Use of a think-aloud measure and implications for studies of golf-putting in the laboratory. Psychology of Sport and Exercise, 27, 1827.
Austin, J., & Delaney, P. F. (1998). Protocol analysis as a tool for behavior analysis. The Analysis of Verbal Behavior, 15, 4156.
Azevedo, R., Faremo, S., & Lajoie, S. P. (2007). Expert–novice differences in mammogram interpretation. In McNamara, D. S. & Trafton, J. G. (eds.), Proceedings of the 29th Annual Cognitive Science Society (pp. 6570). Nashville, TN: Cognitive Science Society.
Balslev, T., Jarodzka, H., Holmqvist, K., de Grave, W., Muijtjens, A. M. M., Eika, B., … & Scherpbier, A. J. J. A. (2011). Visual expertise in paediatric neurology. European Journal of Paediatric Neurology, 16, 161166.
Binet, A. (1966). Mnemonic virtuosity: A study of chess players (trans. Simmel, M. L. and Barron, S. B.). Genetic Psychology Monographs, 74, 127162. (Original work published 1893)
Boring, E. B. (1950). A history of experimental psychology. New York: Appleton-Century Crofts.
Boshuizen, H. P. A., & Schmidt, H. G. (1992). On the role of biomedical knowledge in clinical reasoning by experts, intermediates and novices. Cognitive Science, 16, 153184.
Bryan, W. L., & Harter, N. (1899). Studies on the telegraphic language: The acquisition of a hierarchy of habits. Psychological Review, 6, 345375.
Burns, B. D. (2004). The effects of speed on skilled chess performance. Psychological Science, 15, 442447.
Calderwood, R., Klein, G. A., & Crandall, B. W. (1988). Time pressure, skill, and move quality in chess. American Journal of Psychology, 101, 481493.
Calmeiro, L., & Tenenbaum, G. (2011). Concurrent verbal protocol analysis in sport: Illustration of thought process during a golf-putting task. Journal of Clinical Sport Psychology, 5, 223236.
Carayon, P., Kianfar, S., Li, Y., Xie, A., Alyousef, B., & Wooldridge, A. (2015). A systematic review of mixed methods research on human factors and ergonomics in health care. Applied Ergonomics, 51, 291321.
Chabris, C. F., & Hearst, E. S. (2003). Visualization, pattern recognition, and forward search: Effects of playing speed and sight of the position on grandmaster chess errors. Cognitive Science, 27, 637648.
Chaffin, R., Imreh, G., Lemieux, A. F., & Chen, C. (2003). “Seeing the big picture”: Piano practice as expert problem solving. Music Perception, 20, 465490.
Charness, N. (1981). Search in chess: Age and skill differences. Journal of Experimental Psychology: Human Perception and Performance, 7, 467476.
Chase, W. G., & Ericsson, K. A. (1981). Skilled memory. In Anderson, J. R. (ed.), Cognitive skills and their acquisition (pp. 141189). Hillsdale, NJ: Erlbaum.
Chase, W. G., & Ericsson, K. A. (1982). Skill and working memory. In Bower, G. H. (ed.), The psychology of learning and motivation (Vol. 16, pp. 158). New York: Academic Press.
Collins, D., & Dunn, M. (2011). Problem-solving strategies and processes in musical composition: Observations in real time. Journal of Music, Technology and Education, 4, 4776.
Cooke, L. (2010). Assessing concurrent think-aloud protocol as a usability test method: A technical communication approach. IEEE Transactions on Professional Communication, 53, 202215.
Crovitz, H. F. (1970). Galton’s walk: Methods for the analysis of thinking, intelligence, and creativity. New York: Harper & Row.
Crowley, R. S., Naus, G. J., Stewart, J., & Friedman, C. P. (2003). Development of visual diagnostic expertise in pathology: An information-processing study. Journal of the American Medical Informatics Association, 10, 3951.
Dansereau, D. F., & Gregg, L. W. (1966). An information processing analysis of mental multiplication. Psychonomic Science, 6, 7172.
Darker, C. D., & French, D. P. (2009). What sense do people make of a theory of planned behaviour questionnaire? A think aloud study. Journal of Health Psychology, 14, 861871.
de Groot, A. (1978). Thought and choice in chess (2nd edn.). The Hague: Mouton. (Original work published 1946)
Den Hartigh, R. J. R., Van Der Steen, S., De Meij, M., Van Yperen, N. W., Gernigon, C. & Van Geert, P. L. C. (2014). Characterising expert representations during real-time action: A Skill Theory application to soccer. Journal of Cognitive Psychology, 26, 754767.
Duncker, K. (1945). On problem solving. Psychological Monographs, 58, 1113.
Ericsson, K. A. (1988). Concurrent verbal reports on reading and text comprehension. Text, 8, 295325.
Ericsson, K. A. (1996). The acquisition of expert performance: An introduction to some of the issues. In Ericsson, K. A. (ed.), The road to excellence: The acquisition of expert performance in the arts and sciences, sports, and games (pp. 150). Mahwah, NJ: Erlbaum.
Ericsson, K. A. (2002). Toward a procedure for eliciting verbal expression of nonverbal experience without reactivity: Interpreting the verbal overshadowing effect within the theoretical framework for protocol analysis. Applied Cognitive Psychology, 16, 981987.
Ericsson, K. A. (2003). Valid and non-reactive verbalization of thoughts during performance of tasks: Toward a solution to the central problems of introspection as a source of scientific data. Journal of Consciousness Studies, 10, 118.
Ericsson, K. A. (2004). Deliberate practice and the acquisition and maintenance of expert performance in medicine and related domains. Academic Medicine, 79, S70S81.
Ericsson, K. A. (2015). Acquisition and maintenance of medical expertise: A perspective from the expert-performance approach with deliberate practice. Academic Medicine, 90, 14711486.
Ericsson, K. A., & Charness, N. (1994). Expert performance: Its structure and acquisition. American Psychologist, 49, 725747.
Ericsson, K. A., Chase, W., & Faloon, S. (1980). Acquisition of a memory skill. Science, 208, 11811182.
Ericsson, K. A., Cheng, X., Pan, Y., Ku, Y., Ge, Y., & Hu, Y. (2017). Memory skills mediating superior memory in a world-class memorist. Memory, 25, 12941302.
Ericsson, K. A., Delaney, P. F., Weaver, G., & Mahadevan, R. (2004). Uncovering the structure of a memorist’s superior “basic” memory capacity. Cognitive Psychology, 49, 191237.
Ericsson, K. A., & Kintsch, W. (1995). Long-term working memory. Psychological Review, 102, 211245.
Ericsson, K. A., Patel, V. L., & Kintsch, W. (2000). How experts’ adaptations to representative task demands account for the expertise effect in memory recall: Comment on Vicente and Wang (1998). Psychological Review, 107, 578592.
Ericsson, K. A., & Polson, P. G. (1988). Memory for restaurant orders. In Chi, M., Glaser, R., & Farr, M. (eds.), The nature of expertise (pp. 2370). Hillsdale, NJ: Erlbaum.
Ericsson, K. A., & Simon, H. A. (1980). Verbal reports as data. Psychological Review, 87, 215251.
Ericsson, K. A., & Simon, H. A. (1984). Protocol analysis: Verbal reports as data. Cambridge, MA: Bradford Books/MIT Press.
Ericsson, K. A., & Simon, H. A. (1993). Protocol analysis: Verbal reports as data (revised edn.). Cambridge, MA: Bradford Books/MIT Press.
Ericsson, K. A., & Smith, J. (1991). Prospects and limits in the empirical study of expertise: An introduction. In Ericsson, K. A. and Smith, J. (eds.), Toward a general theory of expertise: Prospects and limits (pp. 138). Cambridge University Press.
Ericsson, K. A., & Staszewski, J. (1989). Skilled memory and expertise: Mechanisms of exceptional performance. In Klahr, D. and Kotovsky, K. (eds.), Complex information processing: The impact of Herbert A. Simon (pp. 235267). Hillsdale, NJ: Erlbaum.
Ericsson, K. A., & Ward, P. (2007). Capturing the naturally occurring superior performance of experts in the laboratory: Toward a science of expert and exceptional performance. Current Directions in Psychological Science, 16, 346350.
Ericsson, K. A., & Williams, A. M. (2007). Capturing naturally-occurring superior performance in the laboratory: Translational research on expert performance. Journal of Experimental Psychology: Applied, 13, 115123.
Fayena-Tawil, F., Kozbelt, A., & Sitaras, L. (2011). Think global, act local: A protocol analysis comparison of artists’ and nonartists’ cognitions, metacognitions, and evaluations while drawing. Psychology of Aesthetics, Creativity, and the Arts, 5, 135145.
Fox, M. C., Ericsson, K. A., & Best, R. (2011a). Do procedures for verbal reporting of thinking have to be reactive? A meta-analysis and recommendations for best reporting methods. Psychological Bulletin, 137, 316344.
Fox, M. C., Ericsson, K. A., & Best, R. (2011b). Supplemental materials.
French, K. E., Nevett, M. E., Spurgeon, J. H., Graham, K. C., Rink, J. E., & McPherson, S. L. (1996). Knowledge representation and problem solution in expert and novice youth baseball players. Research Quarterly for Exercise and Sport, 67, 386395.
Gagné, R. H., & Smith, E. C. (1962). A study of the effects of verbalization on problem solving. Journal of Experimental Psychology, 63, 1218.
Galton, F. (1879). Psychometric experiments. Brain, 2, 148162.
Gilhooly, K. J., McGeorge, P., Hunter, J., Rawles, J. M., Kirby, I. K., Green, C., & Wynn, V. (1997). Biomedical knowledge in diagnostic thinking: The case of electrocardiogram (ECG) interpretation. European Journal of Cognitive Psychology, 9, 199223.
Hu, Y., & Ericsson, K. A. (2012). Memorization and recall of very long lists accounted for within the Long-Term Working Memory framework. Cognitive Psychology, 64, 236266.
Jaarsma, T., Jarodzka, H., Nap, M., Merriënboer, J., & Boshuizen, H. (2014). Expertise under the microscope: Processing histopathological slides. Medical Education, 48, 292300.
Karpov, A. (1995). Grandmaster musings. Chess Life (November), 3233.
Keller, F. S. (1958). The phantom plateau. Journal of the Experimental Analysis of Behavior, 1, 113.
Koltanowski, G. (1985). In the dark. Coraopolis, PA: Chess Enterprises.
Larkin, J., McDermott, J., Simon, D. P., & Simon, H. A. (1980). Expert and novice performance in solving physics problems. Science, 208, 13351342.
Lesgold, A., Rubinson, H., Feltovitch, P., Glaser, R., Klopher, D., & Wang, Y. (1988). Expertise in a complex skill: Diagnosing X-ray pictures. In Chi, M. T. H., Glaser, R., & Farr, M. J. (eds.), The nature of expertise (pp. 311342). Hillsdale, NJ: Erlbaum.
Lundgrén-Laine, H., & Salanterä, S. (2010). Think-aloud technique and protocol analysis in clinical decision making research. Qualitative Health Research, 20, 565575.
MacDonald, S., Edwards, H. M., & Zhao, T. (2012). Exploring think-alouds in usability testing: An international survey. IEEE Transactions on Professional Communication, 55, 219.
Mann, D. T. Y., Williams, A. M., Ward, P., & Janelle, C. M. (2007). Perceptual-cognitive expertise in sport: A meta analysis. Journal of Sport & Exercise Psychology, 29, 457478.
McLaughlin, K., Novac, K., Rikers, R., & Schmidt, H. (2010). Does applying biomedical knowledge improve diagnostic performance when solving electrolyte problems? Canadian Medical Education Journal, 1, e4e9.
McPherson, S., & Kernodle, M. (2007). Mapping two new points on the tennis expertise continuum: Tactical skills of adult advanced beginners and entry-level professionals during competition. Journal of Sports Sciences, 25, 945959.
McRobert, A. P., Causer, J., Vassiliadis, J., Watterson, L., Kwan, J., & Williams, M. A. (2013). Contextual information influences diagnosis accuracy and decision making in simulated emergency medicine emergencies. BMJ Quality & Safety, 22, 478484.
Meissner, C. A., & Brigham, J. C. (2001). A meta-analysis of the verbal overshadowing effect in face identification. Applied Cognitive Psychology, 15, 603616.
Miglioretti, D. L., Gard, C. C., Carney, P. A., Onega, T. L., Buist, D. S., Sickles, E. A., … & Elmore, J. G. (2009). When radiologists perform best: The learning curve in screening mammogram interpretation. Radiology, 253, 632640.
Miller, G. A., Galanter, E., & Pribram, K. H. (1960). Plans and the structure of behavior. New York: Holt, Rinehart, & Winston.
Moxley, J. H., Ericsson, K. A., Charness, N., & Krampe, R. T. (2012). The role of intuition and deliberative thinking in experts’ superior tactical decision-making. Cognition, 124, 7278.
Neuman, Y., & Schwarz, B. (1998). Is self-explanation while solving problems helpful? The case of analogical problem-solving. British Journal of Educational Psychology, 68, 1524.
Newell, A. (1990). Unified theories of cognition. Cambridge, MA: Harvard University Press.
Newell, A., & Simon, H. A. (1972). Human problem solving. Englewood Cliffs, NJ: Prentice-Hall.
Nielsen, S. G. (1999) Learning strategies in instrumental music practice. British Journal of Music Education, 16, 275291.
Nisbett, R. E., & Wilson, T. D. (1977). Telling more than we can know: Verbal reports on mental processes. Psychological Review, 84, 231259.
Nodine, C. F., Kundel, H. L., Mello-Thoms, C., Weinstein, S. P., Orel, S. G., Sullivan, D. C., & Conant, E. F. (1999). How experience and training influence mammography expertise. Academic Radiology, 6, 575585.
Norman, G. R., Trott, A. D., Brooks, L. R., & Smith, E. K. M. (1994). Cognitive differences in clinical reasoning related to postgraduate training. Teaching and Learning in Medicine, 6, 114120.
Patel, V. L., Arocha, J. F., & Kaufmann, D. R. (1994). Diagnostic reasoning and medical expertise. In Medin, D. (ed.), The psychology of learning and motivation (Vol. 30, pp. 187251). New York: Academic Press.
Patel, V. L., & Groen, G. J. (1991). The general and specific nature of medical expertise: A critical look. In Ericsson, K. A. & Smith, J. (eds.), Toward a general theory of expertise: Prospects and limits (pp. 93125). Cambridge University Press.
Roca, A., Ford, P., McRobert, A., & Williams, A. M. (2011). Identifying the processes underpinning anticipation and decision-making in a dynamic time-constrained-task. Cognitive Processing, 12, 301310.
Roca, A., Williams, A. M., & Ford, P. R. (2013). Capturing and testing perceptual-cognitive expertise: A comparison of stationary and movement response methods. Behavior Research Methods, 46, 173177.
Saariluoma, P. (1991). Aspects of skilled imagery in blindfold chess. Acta Psychologica, 77, 6589.
Saariluoma, P. (1992). Error in chess: The apperception-restructuring view. Psychological Research/Psychologische Forschung, 54, 1726.
Saariluoma, P. (1995). Chess players’ thinking. London: Routledge.
Schmidt, H. G., & Boshuizen, H. (1993). On acquiring expertise in medicine. Educational Psychology Review, 5, 205221.
Schooler, J. W., & Engstler-Schooler, T. Y. (1990). Verbal overshadowing of visual memories: Some things are better left unsaid. Cognitive Psychology, 22, 3671.
Schraagen, J. M. (2006). Task analysis. In Ericsson, K. A., Charness, N., Hoffman, R. R., & Feltovich, P. J. (eds.), The Cambridge handbook of expertise and expert performance (pp. 185201). Cambridge University Press.
St. Germain, J., & Tenenbaum, G. (2011) Decision-making and thought processes among poker players. High Ability Studies, 22, 317.
Thompson, C. P., Cowan, T. M., & Frieman, J. (1993). Memory search by a memorist. Hillsdale, NJ: Erlbaum.
Tuffiash, M., Roring, R. W., & Ericsson, K. A. (2007). Expert word play: Capturing and explaining reproducibly superior verbal task performance. Journal of Experimental Psychology: Applied, 13, 124134.
van der Maas, H. L. J., & Wagenmakers, E. J. (2005). A psychometric analysis of chess expertise. American Journal of Psychology, 118, 2960.
Watson, J. B. (1913). Psychology as the behaviorist views it. Psychological Review, 20, 158177.
Watson, J. B. (1920). Is thinking merely the action of language mechanisms? British Journal of Psychology, 11, 87104.
Whitehead, A. E., Taylor, J. A., & Polman, R. C. J. (2015). Examination of the suitability of collecting in event cognitive processes using Think Aloud protocol in golf. Frontiers in Psychology, 6, 1083.
Whitehead, A. E., Taylor, J. A., & Polman, R. C. (2016). Evidence for skill level differences in the thought processes of golfers during high and low pressure situations. Frontiers in Psychology, 6, 112.
Willis, G. (2005). Cognitive interviewing: A tool for improving questionnaire design. Thousand Oaks, CA: Sage Publications.
Willis, G. (2015). Analysis of the cognitive interview in questionnaire design: Understanding qualitative research. Oxford University Press.