Hostname: page-component-6766d58669-vgfm9 Total loading time: 0 Render date: 2026-05-22T02:20:00.641Z Has data issue: false hasContentIssue false
  • English
  • Français

“Life Doesn't Happen at the Between-Person Level,” or a Cautionary Note on Generating Scientific Inferences Through Meta-Analyses

Published online by Cambridge University Press:  30 August 2017

Hannah L. Samuelson ,
Jessica R. Fernandez  and
James A. Grand
Hannah L. Samuelson*
Affiliation:
Department of Psychology, University of Maryland
Jessica R. Fernandez
Affiliation:
Department of Psychology, University of Maryland
James A. Grand
Affiliation:
Department of Psychology, University of Maryland
*
Correspondence concerning this article should be addressed to Hannah L. Samuelson, Department of Psychology, 4094 Campus Dr., 3143 Biology-Psychology Building, University of Maryland, College Park, MD 20742. E-mail: hsamuels@umd.edu
Get access Rights & Permissions [Opens in a new window]

Extract

The implicit philosophy for how research and practice in industrial and organizational (I-O) psychology has pursued inferences about our field's core phenomena has largely been based on a nomothetic, variable-based, and aggregate/“large-sample” ideal. As Tett, Hundley, and Christiansen (2017) expertly highlight, there are more insightful means for drawing inferences about the nature of such aggregate relationships based on meta-analytic techniques than the current practice in the organizational sciences. However, the motivating force behind our commentary has less to do with the issues raised by Tett et al. (2017) concerning the practice of using meta-analysis for purposes of validity generalization and more to do with the practice of using meta-analysis for purposes of scientific inference. Between-person philosophies in which the end-goal is to identify general conclusions that apply to the aggregate (cf. Hanges & Wang, 2012) have historically guided our scientific inferences and have supported the proliferation of meta-analytic techniques (including what Tett et al. describe as tertiary analyses based on such findings). These philosophies have led to a dearth of understanding at the within-person and social system levels—the levels at which most of our meaningful phenomena exist (e.g., Hamaker, 2012; Von Bertalanffy, 1950). Learning, performing, decision making, communicating, sense-making, feeling/expressing emotion: These are the concepts that drive the lived experiences of individuals both inside and outside of the workplace, and all are vulnerable to being misunderstood or misinterpreted by focusing only on aggregate evidence at the between-person level. Consequently, we wish to first supplement Tett et al.’s recommendations for drawing generalizability inferences in meta-analysis and suggest a “pre-emptive” question (i.e., Question 0) to the list of four they advance in their focal article.

Information

Type
Commentaries
Information
Industrial and Organizational Psychology , Volume 10 , Issue 3 , September 2017 , pp. 459 - 464
Copyright
Copyright © Society for Industrial and Organizational Psychology 2017 

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

Article purchase

Temporarily unavailable

References

Axelrod, R. (1997). Advancing the art of simulation in the social sciences. In Conte, R., Hegselmann, R., & Terna, P. (Eds.), Simulating social phenomena (pp. 21–40). Berlin: Springer.CrossRefGoogle Scholar
Bechtel, W. (1988). Philosophy of science: An overview for cognitive science. Hillsdale, NJ: Lawrence Erlbaum.Google Scholar
Cronin, M. A., Weingart, L.R., & Todorova, G. (2011). Dynamics in groups: Are we there yet? The Academy of Management Annals, 5, 571–612.CrossRefGoogle Scholar
Davis, J. P., Eisenhardt, K. M., & Bingham, C. B. (2007). Developing theory through simulation methods. Academy of Management Review, 32 (2), 480–499.CrossRefGoogle Scholar
Grand, J. A., Braun, M. T., Kuljanin, G., Kozlowski, S. W., & Chao, G. T. (2016). The dynamics of team cognition: A process-oriented theory of knowledge emergence in teams. Journal of Applied Psychology, 101 (10), 1353–1385.CrossRefGoogle ScholarPubMed
Hamaker, E. L. (2012). Why researchers should think “within-person”: A paradigmatic rationale. In Mehl, M. R. & Conner, T. S. (Eds.), Handbook of research methods for studying daily life (pp. 43–61). New York: Guilford.Google Scholar
Hanges, P. J., & Wang, M. (2012). Seeking the holy grail in organizational science: Uncovering causality through research design. In Kozlowski, S. W. J. (Ed.), The Oxford handbook of organizational psychology (Vol. 1, pp. 79–116). New York: Oxford University Press.CrossRefGoogle Scholar
Harrison, J. R., Lin, Z., Carroll, G. R., & Carley, K. M. (2007). Simulation modeling in organizational and management research. Academy of Management Review, 32 (4), 1229–1245.CrossRefGoogle Scholar
Hunter, J. E., & Schmidt, F. L. (2004). Methods of meta-analysis: Correcting for error and bias in research findings. Thousand Oaks, CA: Sage.CrossRefGoogle Scholar
Kozlowski, S. W., Chao, G. T., Grand, J. A., Braun, M. T., & Kuljanin, G. (2013). Advancing multilevel research design: Capturing the dynamics of emergence. Organizational Research Methods, 16 (4), 581–615.CrossRefGoogle Scholar
Kozlowski, S. W., Chao, G. T., Grand, J. A., Braun, M. T., & Kuljanin, G. (2016). Capturing the multilevel dynamics of emergence: Computational modeling, simulation, and virtual experimentation. Organizational Psychology Review, 6 (1), 3–33.CrossRefGoogle Scholar
McDaniel, M. A., Whetzel, D. L., Schmidt, F. L., & Maurer, S. D. (1994). The validity of employment interviews: A comprehensive review and meta-analysis. Journal of Applied Psychology, 79, 599–616.CrossRefGoogle Scholar
Schmidt, A. M., & DeShon, R. P. (2007). What to do? The effects of discrepancies, incentives, and time on dynamic goal prioritization. Journal of Applied Psychology, 92, 928–941.CrossRefGoogle Scholar
Schmidt, F. L., & Hunter, J. E. (1998). The validity and utility of selection methods in personnel psychology: Practical and theoretical implications of 85 years of research findings. Psychological Bulletin, 124 (2), 262–274.CrossRefGoogle Scholar
Tett, R. P., Hundley, N. A., & Christiansen, N. D. (2017). Meta-analysis and the myth of generalizability. Industrial and Organizational Psychology: Perspectives on Science and Practice, 10 (3), 421–456.CrossRefGoogle Scholar
Vancouver, J. B., Thompson, C. M., Tischner, E. C., & Putka, D. J. (2002). Two studies examining the negative effect of self-efficacy on performance. Journal of Applied Psychology, 87, 506–516.CrossRefGoogle ScholarPubMed
Vancouver, J. B., Thompson, C. M., & Williams, A. A. (2001). The changing signs in the relationships among self-efficacy, personal goals, and performance. Journal of Applied Psychology, 86, 605–620.CrossRefGoogle ScholarPubMed
Vancouver, J. B., Weinhardt, J. M., & Schmidt, A. M. (2010). A formal, computational theory of multiple-goal pursuit: Integrating goal-choice and goal-striving processes. Journal of Applied Psychology, 95 (6), 985–1008.CrossRefGoogle ScholarPubMed
Von Bertalanffy, L. (1950). An outline of general systems theory. British Journal of for the Philosophy of Science, 1, 136–164.Google Scholar
Wiesner, W. H., & Cronshaw, S. F. (1988). A meta‐analytic investigation of the impact of interview format and degree of structure on the validity of the employment interview. Journal of Occupational Psychology, 61, 275–290.CrossRefGoogle Scholar