No CrossRef data available.
Article contents
Assessment centers do not measure competencies: why this is now beyond reasonable doubt
Published online by Cambridge University Press: 09 May 2024
Abstract
Although assessment centers (ACs) are usually designed to measure stable competencies (i.e., dimensions), doubt about whether or not they reliably do so has endured for 70 years. Addressing this issue in a novel way, several published Generalizability (G) theory studies have sought to isolate the multiple sources of variance in AC ratings, including variance specifically concerned with competencies. Unlike previous research, these studies can provide a definitive answer to the AC construct validity issue. In this article, the historical context for the construct validity debate is set out, and the results of four large-scale G-theory studies of ACs are reviewed. It is concluded that these studies demonstrate, beyond reasonable doubt, that ACs do not reliably measure stable competencies, but instead measure general, and exercise-related, performance. The possibility that ACs measure unstable competencies is considered, and it is suggested that evidence that they do so may reflect an artefact of typical AC design rather than a “real” effect. For ethical, individual, and organizational reasons, it is argued that the use of ACs to measure competencies can no longer be justified and should be halted.
- Type
- Focal Article
- Information
- Copyright
- © The Author(s), 2024. Published by Cambridge University Press on behalf of Society for Industrial and Organizational Psychology
References
Ansbacher, H. L. (1941). German military psychology. Psychological Bulletin, 38, 370–392. Doi: 10.1037/h0056263.CrossRefGoogle Scholar
Arthur, W. (2012). Dimension-based assessment centers: Theoretical perspectives. In Jackson, D. J. R., Lance, C. E., & Hoffman, B. J. (Eds.), The psychology of assessment centers (pp. 95–120). Routledge.Google Scholar
Arthur, W., Day, E. A., McNelly, T. L., & Edens, P. S. (2003). A meta-analysis of the criterion-related validity of assessment center dimensions. Personnel Psychology, 56(1), 125–154. Doi: 10.1111/j.1744-6570.2003.tb00146.x.CrossRefGoogle Scholar
Arthur, W., Day, E. A., & Woehr, D. J. (2008). Mend it, don’t end it: An alternate view of assessment center construct-related validity evidence. Industrial and Organizational Psychology-Perspectives on Science and Practice, 1, 105–111.CrossRefGoogle Scholar
Arthur, W., Woehr, D. J., & Maldegan, R. (2000). Convergent and discriminant validity of assessment center dimensions: A conceptual and empirical reexamination of the assessment center construct-related validity paradox. Journal of Management, 26(4), 813–835. Doi: 10.1016/S0149-2063(00)00057-X.Google Scholar
Austin, J. T., & Crespin, T. R. (2006). From ‘criterion problem’ to problems of criteria in industrial and organizational psychology: Progress, pitfalls, and prospects. In Bennett, W. Jr., Lance, C. E., & Woehr, D. J. (Eds.), Performance Measurement: Current Perspectives and Future Challenges. Lawrence Erlbaum Associates.Google Scholar
Barrett, G. V., & Depinet, R. L. (1991). A reconsideration of testing for competence rather than for intelligence. American Psychologist, 46, 1012–1024.CrossRefGoogle ScholarPubMed
Bowler, M. C., & Woehr, D. J. (2006). A meta-analytic evaluation of the impact of dimension and exercise factors on assessment center ratings. Journal of Applied Psychology, 91, 1114–1124.CrossRefGoogle ScholarPubMed
Bowler, M. C., & Woehr, D. J. (2009). Assessment center construct-related validity: Stepping beyond the MTMM matrix. Journal of Vocational Behavior, 75, 173–182. Doi: 10.1016/j.jvb.2009.03.008.CrossRefGoogle Scholar
Boyatzis, R. E. (1982). The competent manager: A model for effective performance. Wiley.Google Scholar
Brannick, M. T. (2008). Back to basics of test construction and scoring. Industrial and Organizational Psychology, 1(1), 131–133. Doi: 10.1111/j.1754-9434.2007.00025.x.CrossRefGoogle Scholar
Bray, D. W., Campbell, R. J., & Grant, D. L. (1974). Formative years in business: A long-term AT&T study of managerial lives. Wiley.Google Scholar
Bray, D. W., & Grant, D. L. (1966). The assessment center in the measurement of potential for business management. Psychological Monographs: General and Applied, 80, 1–27.CrossRefGoogle ScholarPubMed
Brennan, R. L. (2000). Performance assessments from the perspective of generalizability theory. Applied Psychological Measurement, 24, 339–353. Doi: 10.1177/01466210022031796.CrossRefGoogle Scholar
Brennan, R. L. (2001b). Manual for urGENOVA. Iowa Testing Programs, University of Iowa.Google Scholar
Buckett, A., Becker, J. R., & Melchers, K. (2020). How different indicator-dimension ratios in assessment center ratings Affect evidence for dimension factors. Frontiers in Psychology, 11, 511636.Google Scholar
Buckett, A., Becker, J. R., & Roodt, G. (2021). The impact of item parceling ratios and strategies on the internal structure of assessment center ratings: A study using confirmatory factor analysis. Journal of Personnel Psychology, 20(1), 1–16. Doi: 10.1027/1866-5888/a000266.CrossRefGoogle Scholar
Byham, W. C. (1970). Assessment centers for spotting future managers. Harvard Business Review, 48, 150–160.Google Scholar
Byham, W. C. (1977). Application of the assessment center method. In J. L. Moses & W. C. Byham (Eds.), Applying the assessment center method (pp. 31–43). New York: Pergamon.Google Scholar
Campbell, D. T., & Fiske, D. W. (1959). Convergent and discriminant validation by the multitrait-multimethod matrix. Psychological Bulletin, 56(2), 81–105. Doi: 10.1037/h0046016.CrossRefGoogle ScholarPubMed
Campion, M. A., Fink, A. A., Ruggeberg, B. J., Carr, L., Phillips, G. M., & Odman, R. B. (2011). Doing competencies well: Best practices in competency modeling. Personnel Psychology, 64(1), 225–262. Doi: 10.1111/j.1744-6570.2010.01207.x.CrossRefGoogle Scholar
Chan, D. (1996). Criterion and construct validation of an assessment centre. Journal of Occupational and Organizational Psychology, 69, 167–181. Doi: 10.1111/j.2044-8325.1996.tb00608.x.CrossRefGoogle Scholar
Connelly, B. S., Ones, D. S., Ramesh, A., & Goff, M. (2008). A pragmatic view of assessment center exercises and dimensions. Industrial and Organizational Psychology-Perspectives on Science and Practice, 1, 121–124.CrossRefGoogle Scholar
Cronbach, L. J., Gleser, G. C., Nanda, H., & Rajaratnam, N. (1972). The dependability of behavioral measurements: Theory of generalizability for scores and profiles. John Wiley.Google Scholar
Cronbach, L. J., Rajaratnam, N., & Gleser, G. C. (1963). Theory of generalizability: A liberalization of reliability theory. British Journal of Statistical Psychology, 16(2), 137–163. Doi: 10.1111/j.2044-8317.1963.tb00206.x.CrossRefGoogle Scholar
Deming, D. (2016). Do Extraordinary Claims Require Extraordinary Evidence? Philosophia, 44(4), 1319–1331. Doi: 10.1007/s11406-016-9779-7.CrossRefGoogle ScholarPubMed
Donahue, L. M., Truxillo, D. M., Cornwell, J. M., & Gerrity, M. J. (1997). Assessment center construct validity and behavioral checklists: Some additional findings. Journal of Social Behaviour and Personality, 12, 85–108.Google Scholar
Eurich, T. L., Krause, D. E., Cigularov, K., & Thornton, G. C. (2009). Assessment centers: Current practices in the United States. Journal of Business and Psychology, 24, 387–407. Doi: 10.1007/s10869-009-9123-3.CrossRefGoogle Scholar
Fitts, P. M. (1946). German applied psychology during World War II. The American Psychologist, 1(5), 151–161. Doi: 10.1037/h0059674.CrossRefGoogle ScholarPubMed
Gaugler, B. B., Rosenthal, D. B., Thornton, G. C., & Bentson, C. (1987). Meta-analysis of assessment center validity. Journal of Applied Psychology, 72, 493–511. Doi: 10.1037/0021-9010.72.3.493.CrossRefGoogle Scholar
Goodge, P. (1988). Task-based assessment. Journal of European Industrial Training, 12, 22–27.CrossRefGoogle Scholar
Guenole, N., Chernyshenko, O. S., Stark, S., Cockerill, T., & Drasgow, F. (2013). More than a mirage: A large-scale assessment centre with more dimension variance than exercise variance. Journal of Occupational and Organizational Psychology, 86, 5–21. Doi: 10.1111/j.2044-8325.2012.02063.x.CrossRefGoogle Scholar
Handler, L. (2001). Assessment of men: Personality assessment goes to war by the office of strategic services assessment staff. Journal of Personality Assessment, 76(3), 558–578.CrossRefGoogle Scholar
Handyside, J. D., & Duncan, D. C. (1954). Four years later: A follow-up of an experiment in selecting supervisors. Occupational Psychology, 28, 9–23.Google Scholar
Hayes, G. (1995). Science and the magic eye: Innovations in the selection of Canadian army officers, 1939-1945. Armed Forces and Society, 22(2), 275–295.CrossRefGoogle Scholar
Highhouse, S. (2002). Assessing the candidate as a whole: A historical and critical analysis of individual psychological assessment for personnel decision making. Personnel Psychology, 55(2), 363–396.CrossRefGoogle Scholar
Highhouse, S., & Nolan, K. P. (2012). One history of the assessment center. In Jackson, D. J. R., Lance, C. E., & Hoffman, B. J. (Eds.), The psychology of assessment centers (pp. 25–44). Routledge/Taylor & Francis Group.Google Scholar
Hoffman, B. J., Melchers, K. G., Blair, C. A., Kleinmann, M., & Ladd, R. T. (2011). Exercises and dimensions are the currency of assessment centers. Personnel Psychology, 64, 351–395. Doi: 10.1111/j.1744-6570.2011.01213.x.CrossRefGoogle Scholar
Howard, A. (2008). Making assessment centers work the way they are supposed to. Industrial and Organizational Psychology: Perspectives on Science and Practice, 1, 98–104. Doi: 10.1111/j.1754-9434.2007.00018.x.CrossRefGoogle Scholar
Hoyt, W. T., & Kerns, M. (1999). Magnitude and moderators of bias in observer ratings: A meta-analysis. Psychological Methods, 4, 403–424. Doi: 10.1037/1082-989X.4.4.403.CrossRefGoogle Scholar
International Taskforce on Assessment Center Guidelines (2015). Guidelines and ethical considerations for assessment center operations. Journal of Management, 41(4), 1244–1273. Doi: 10.1177/0149206314567780.Google Scholar
Jackson, D. J. R. (2007, April). Task-specific assessment centers: Evidence of predictive validity and fairness. Society for Industrial and Organizational Psychology.Google Scholar
Jackson, D. J. R. (2012). Task-based assessment centers: Theoretical perspectives. In Jackson, D. J. R., Lance, C. E., & Hoffman, B. J. (Eds.), The psychology of assessment centers (pp. 173–189). Routledge/Taylor & Francis Group.Google Scholar
Jackson, D. J. R., Ahmad, M. H., Grace, G. M., & Yoon, J. (2011). An alternative take on assessment center research and practice: Task-based assessment centers. In Povah, N., & Thornton, G. C. (Eds.), Assessment centres and global talent management (pp. 33–46). Gower Publishing.Google Scholar
Jackson, D. J. R., Michaelides, G., Dewberry, C., Nelson, J., & Stephens, C. (2022). Reliability in assessment centers depends on general and exercise performance, but not on dimensions. Journal of Occupational and Organizational Psychology, 95, 739–757.CrossRefGoogle Scholar
Jackson, D. J. R., Michaelides, M., Dewberry, C., & Kim, Y. (2016). Everything that you have ever been told about assessment center ratings is confounded. Journal of Applied Psychology, 101(7), 976–994. Doi: 10.1037/apl0000102.CrossRefGoogle ScholarPubMed
Jackson, D. J. R., Stillman, J. A., & Atkins, S. G. (2005). Rating tasks versus dimensions in assessment centers: A psychometric comparison. Human Performance, 18(3), 213–241. Doi: 10.1207/s15327043hup1803_2.CrossRefGoogle Scholar
Jaffee, C. L. (1965). Assessment centers help find managment potential. Bell Telephone Magazine, 44(3), 18–25.Google Scholar
Jones, R. G., & Klimoski, R. J. (2008). Narrow standards for efficacy and the research playground: Why either-or conclusions do not help. Industrial and Organizational Psychology: Perspectives on Science and Practice, 1, 137–139.CrossRefGoogle Scholar
Krause, D. E., & Thornton, G. C. (2009). A cross-cultural look at assessment center practices: Survey results from Western Europe and North America. Applied Psychology: An International Review, 58, 557–585.CrossRefGoogle Scholar
Kudisch, J. D., Ladd, R. T., & Dobbins, G. H. (1997). New evidence on the construct validity of diagnostic assessment centers: The findings may not be so troubling after all. Journal of Social Behavior and Personality, 12, 129–144.Google Scholar
Kuncel, N. R., & Sackett, P. R. (2014). Resolving the assessment center construct validity problem (as we know it). Journal of Applied Psychology, 99(1), 38–47. Doi: 10.1037/a0034147.CrossRefGoogle ScholarPubMed
Lance, C. E. (2008). Why assessment centers do not work the way they are supposed to. Industrial and Organizational Psychology: Perspectives on Science and Practice, 1(1), 84–97. Doi: 10.1111/j.1754-9434.2007.00017.x.CrossRefGoogle Scholar
Lance, C. E., Foster, M. R., Nemeth, Y. M., Gentry, W. A., & Drollinger, S. (2007). Extending the nomological network of assessment center construct validity: Prediction of cross-situationally consistent and specific aspects of assessment center performance. Human Performance, 20(4), 345–362. Doi: 10.1080/08959280701522031.Google Scholar
Lance, C. E., Lambert, T. A., Gewin, A. G., Lievens, F., & Conway, J. M. (2004). Revised estimates of dimension and exercise variance components in assessment center postexercise dimension ratings. Journal of Applied Psychology, 89(2), 377–385. Doi: 10.1037/0021.9010.89.2.377.CrossRefGoogle ScholarPubMed
Lievens, F. (1998). Factors which improve the construct validity of assessment centers: A review. International Journal of Selection and Assessment, 6, 141–152. Doi: 10.1111/1468-2389.00085.CrossRefGoogle Scholar
Lievens, F. (2001a). Assessor training strategies and their effects on accuracy, interrater reliability, and discriminant validity. Journal of Applied Psychology, 86(2), 255–264. Doi: 10.1037/0021-9010.86.2.255.CrossRefGoogle ScholarPubMed
Lievens, F. (2001b). Assessors and use of assessment centre dimensions: A fresh look at a troubling issue. Journal of Organizational Behavior, 22(3), 203–221. Doi: 10.1002/job.65.CrossRefGoogle Scholar
Lievens, F. (2002). Trying to understand the different pieces of the construct validity puzzle of assessment centers: An examination of assessor and assessee effects. Journal of Applied Psychology, 87, 675–686. Doi: 10.1037/0021-9010.87.4.675.CrossRefGoogle ScholarPubMed
Lievens, F. (2008). What does exercise-based assessment really mean? Industrial and Organizational Psychology-Perspectives on Science and Practice, 1, 112–115.CrossRefGoogle Scholar
Lievens, F., Chasteen, C. S., Day, E. A., & Christiansen, N. D. (2006). Large-scale investigation of the role of trait activation theory for understanding assessment center convergent and discriminant validity. Journal of Applied Psychology, 91, 247–258.CrossRefGoogle ScholarPubMed
Lievens, F., & Christiansen, N. D. (2012). Core debates in assessment center research: Dimensions ‘versus’ exercises. In Jackson, D. J. R., Lance, C. E., & Hoffman, B. J. (Eds.), The psychology of assessment centers (pp. 68–91). Routledge.Google Scholar
Lievens, F., & Conway, J. M. (2001). Dimension and exercise variance in assessment center scores: A large-scale evaluation of multitrait-multimethod studies. Journal of Applied Psychology, 86, 1202–1222.CrossRefGoogle ScholarPubMed
Lievens, F., & Klimoski, R. J. (2001). Understanding the assessment center process: Where are we now?. In Cooper, C. L., & Robertson, I. T. (Eds.), International Review of Industrial and Organizational Psychology. vol. 16, p. 245–286). John Wiley & Sons.Google Scholar
Lievens, F., Sanchez, J. I., & de Corte, W. (2004). Easing the inferential leap in competency modeling: The effects of task-related information and subject matter expertise. Personnel Psychology, 57, 881–904.CrossRefGoogle Scholar
Lopez, F. M., Kesselman, G. A., & Lopez, F. E. (1981). An empirical test of a trait-oriented job analysis technique. Personnel Psychology, 34, 479–502.CrossRefGoogle Scholar
Lowry, P. E. (1997). The assessment center process: New directions. Journal of Social Behavior and Personality, 12, 53–62.Google Scholar
McClelland, D. C. (1973). Testing for competence rather than for ‘intelligence. American Psychologist, 28, 1–14.CrossRefGoogle ScholarPubMed
Melchers, K. G., & Konig, C. J. (2008). It is not yet time to dismiss dimensions in assessment centers. Industrial and Organizational Psychology: Perspectives on Science and Practice, 1, 125–127.CrossRefGoogle Scholar
Meriac, J. P., Hoffman, B. J., & Woehr, D. J. (2014). A conceptual and empirical review of the structure of assessment center dimensions. Journal of Management, 40, 1269–1296. Doi: 10.1177/0149206314522299.CrossRefGoogle Scholar
Merkulova, N., Melchers, K. G., Kleinmann, M., Annen, H., & Tresch, T. S. (2016). A test of the generalizability of a recently suggested conceptual model for assessment center ratings. Human Performance, 29, 226–250. Doi: 10.1080/08959285.2016.1160093.CrossRefGoogle Scholar
Monahan, E. L., Hoffman, B. J., Lance, C. E., Jackson, D. J. R., & Foster, M. R. (2013). Now you see them, now you do not: The influence of indicator-factor ratio on support for assessment center dimensions. Personnel Psychology, 66, 1009–1047. Doi: 10.1111/peps.12049.CrossRefGoogle Scholar
Moses, J. L. (2008). Assessment centers work but for different reasons. Industrial and Organizational Psychology: Perspectives on Science and Practice, 1, 134–136.CrossRefGoogle Scholar
Murray, M., & MacKinnon, D. W. (1946). Assessment of OSS Personnel. Journal of Consulting Psychology10, 10, 76–80.CrossRefGoogle Scholar
Prahalad, C. K., & Hamel, G. (1990). The core competence of the corporation. Harvard Business Review, 68, 79–91.Google Scholar
Putka, D. J., & Hoffman, B. J. (2013). Clarifying the contribution of assessee-, dimension-, exercise-, and assessor-related effects to reliable and unreliable variance in assessment center ratings. Journal of Applied Psychology, 98(1), 114–133. Doi: 10.1037/a0030887.CrossRefGoogle ScholarPubMed
Reilly, R. R., Henry, S., & Smither, J. W. (1990). An examination of the effects of using behavior checklists on the construct validity of assessment center dimensions. Personnel Psychology, 43(1), 71–84. Doi: 10.1111/j.1744-6570.1990.tb02006.x.CrossRefGoogle Scholar
Robie, C., Osburn, H. G., Morris, M. A., Etchegaray, J. M., & Adams, K. A. (2000). Effects of the rating process on the construct validity of assessment center dimension evaluations. Human Performance, 13, 355–370.CrossRefGoogle Scholar
Rupp, D. E., Thornton, G. C., & Gibbons, A. M. (2008). The construct validity of the assessment center method and usefulness of dimensions as focal constructs. Industrial and Organizational Psychology: Perspectives on Science and Practice, 1, 116–120. Doi: 10.1111/j.1754-9434.2007.00021.x.CrossRefGoogle Scholar
Sackett, P. R., & Dreher, G. F. (1982). Constructs and assessment center dimensions: Some troubling empirical findings. Journal of Applied Psychology, 67(4), 401–410. Doi: 10.1037/0021-9010.67.4.401.CrossRefGoogle Scholar
Sackett, P. R., & Lievens, F. (2008). Personnel selection. Annual Review of Psychology, 59, 419–450. Doi: 10.1146/annurev.psych.59.103006.093716.CrossRefGoogle ScholarPubMed
Sackett, P. R., Zhang, C., Berry, C. M., & Lievens, F. (2021). Revisiting meta-analytic estimates of validity in personnel selection: Addressing systematic overcorrection for restriction of range. Journal of Applied Psychology, 107(11), 2040–2068. Doi: 10.1037/apl0000994.CrossRefGoogle ScholarPubMed
Sakoda, J. M. (1952). Factor analysis of OSS situational tests. Journal of Abnormal and Social Psychology, 47, 843–852. Doi: 10.1037/h0062953.CrossRefGoogle ScholarPubMed
Schmidt, F. L., & Hunter, J. E. (1998). The validity and utility of selection methods in personnel psychology: Practical and theoretical implications of 85 years of research findings. Psychological Bulletin, 124, 262–274.CrossRefGoogle Scholar
Schneider, J., & Schmitt, N. (1992). An exercise design approach to understanding assessment center dimension and exercise constructs. Journal of Applied Psychology, 77(1), 32–41. Doi: 10.1037/0021-9010.77.1.32.CrossRefGoogle Scholar
Schuler, H. (2008). Improving Assessment Centers by the Trimodal Concept of Personnel Assessment. Industrial and Organizational Psychology-Perspectives on Science and Practice, 1(1), 128–130. Doi: 10.1111/j.1754-9434.2007.00024.x.CrossRefGoogle Scholar
Shippmann, J., Ash, R., Battista, M., Carr, L., Eyde, L., Hesketh, B., Kehoe, J., Pearlman, K., & Prien, E. (2000). The practice of competency modeling. Personnel Psychology, 53(3), 703–740. Doi: 10.1111/j.1744-6570.2000.tb00220.x.CrossRefGoogle Scholar
Spychalski, A. C., Quinones, M. A., Gaugler, B. B., & Pohley, K. (1997). A survey of assessment center practices in organizations in the United States. Personnel Psychology, 50(1), 71–90. Doi: 10.1111/j.1744-6570.1997.tb00901.x.CrossRefGoogle Scholar
Stevens, G. (2013). A critical review of the science and practice of competency modeling. Human Resource Development Review, 12(1), 86–107. Doi: 10.1177/1534484312456690.CrossRefGoogle Scholar
Tett, R. P., & Guterman, H. A. (2000). Situation trait relevance, trait expression, and cross-situational consistency: Testing a principle of trait activation. Journal of Research in Personality, 34, 397–423. Doi: 10.1006/jrpe.2000.2292.CrossRefGoogle Scholar
Tett, R. P., Guterman, H. A., Bleier, A., & Murphy, P. J. (2000). Development and content validation of a ‘hyperdimensional’ taxonomy of managerial competence. Human Performance, 13(3), 205–251. Doi: 10.1207/S15327043HUP1303_1.CrossRefGoogle Scholar
Tett, R., Toich, M., & Ozkum, S. (2021). Trait activation theory: A review of the literature and applications to five lines of personality dynamics research. In Morgeson, F. (Eds.), Annual Review of Organizational Psychology and Organizational Behavior (WOS 000614614100009;. vol. 8, p. 199–233).Google Scholar
Thornton, G. C., & Byham, W. C. (1982). Assessment centers and managerial performance. Academic Press.Google Scholar
VandenBos, G. R. (2007). APA Dictionary of Psychology. American Psychologiacl Association.Google Scholar
Viswesvaran, C., Schmidt, F. L., & Ones, D. S. (2005). Is there a general factor in ratings of job performance? A meta-analytic framework for disentangling substantive and error influences. Journal of Applied Psychology, 90, 108–131. Doi: 10.1037/0021-9010.90.1.108
CrossRefGoogle Scholar
Woehr, D. J., & Arthur, W. (2003). The construct-related validity of assessment center ratings: A review and meta-analysis of the role of methodological factors. Journal of Management, 29, 231–258. Doi: 10.1177/014920630302900206.CrossRefGoogle Scholar