Hostname: page-component-77f85d65b8-v2srd Total loading time: 0 Render date: 2026-03-27T13:42:06.609Z Has data issue: false hasContentIssue false
  • English
  • Français

Don't let perfect be the enemy of better: In defense of unparameterized megastudies

Published online by Cambridge University Press:  05 February 2024

Wei Li  and
Joshua K. Hartshorne Open the ORCID record for Joshua K. Hartshorne [Opens in a new window]
Wei Li
Affiliation:
Department of Psychology & Neuroscience, Boston College, Chestnut Hill, MA, USA liacz@bc.edu, https://www.weiiir.xyz/ joshua.hartshorne@hey.com, https://l3atbc.org
Joshua K. Hartshorne*
Affiliation:
Department of Psychology & Neuroscience, Boston College, Chestnut Hill, MA, USA liacz@bc.edu, https://www.weiiir.xyz/ joshua.hartshorne@hey.com, https://l3atbc.org
*
*Corresponding author.
Get access Rights & Permissions [Opens in a new window]

Abstract

The target article argues researchers should be more ambitious, designing studies that systematically and comprehensively explore the space of possible experiments in one fell swoop. We argue that while “systematic” is rarely achievable, “comprehensive” is often enough. Critically, the recent popularization of massive online experiments shows that comprehensive studies are achievable for most cognitive and behavioral research questions.

Information

Type
Open Peer Commentary
Information
Behavioral and Brain Sciences , Volume 47 , 2024 , e53
Check for updates
Copyright
Copyright © The Author(s), 2024. Published by Cambridge University Press

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

Article purchase

Temporarily unavailable

References

Bleidorn, W., Klimstra, T. A., Denissen, J. J., Rentfrow, P. J., Potter, J., & Gosling, S. D. (2013). Personality maturation around the world: A cross-cultural examination of social-investment theory. Psychological Science, 24(12), 2530–2540.CrossRefGoogle ScholarPubMed
Breithaupt, F., Li, B., & Kruschke, J. K. (2022). Serial reproduction of narratives preserves emotional appraisals. Cognition and Emotion, 36(4), 581–601.CrossRefGoogle ScholarPubMed
Brysbaert, M., Stevens, M., Mandera, P., & Keuleers, E. (2016). How many words do we know? Practical estimates of vocabulary size dependent on word definition, the degree of language input and the participant's age. Frontiers in Psychology, 7, 1116.CrossRefGoogle ScholarPubMed
Chomsky, N. (2014). The minimalist program. MIT Press.CrossRefGoogle Scholar
Clark, H. H. (1973). The language-as-fixed-effect fallacy: A critique of language statistics in psychological research. Journal of Verbal Learning and Verbal Behavior, 12(4), 335–359.CrossRefGoogle Scholar
De Deyne, S., Navarro, D. J., Perfors, A., Brysbaert, M., & Storms, G. (2019). The “small world of words” English word association norms for over 12,000 cue words. Behavior Research Methods, 51, 987–1006.CrossRefGoogle Scholar
Erb, C. D., Germine, L., & Hartshorne, J. K. (2023). Fractionating cognitive control: Congruency tasks reveal divergent developmental trajectories. Journal of Experimental Psychology: General. https://doi.org/10.1037/xge0001429CrossRefGoogle Scholar
Goldberg, A. E. (1995). Constructions: A construction grammar approach to argument structure. University of Chicago Press.Google Scholar
Goldberg, A. E. (2009). The nature of generalization in language. Cognitive Linguistics, 20(1), 93–127.CrossRefGoogle Scholar
Gosling, S. D., & Mason, W. (2015). Internet research in psychology. Annual Review of Psychology, 66, 877–902.CrossRefGoogle ScholarPubMed
Hampshire, A., Highfield, R. R., Parkin, B. L., & Owen, A. M. (2012). Fractionating human intelligence. Neuron, 76(6), 1225–1237.CrossRefGoogle ScholarPubMed
Hartshorne, J. K., Bonial, C., & Palmer, M. (2014). The VerbCorner project: Findings from phase 1 of crowd-sourcing a semantic decomposition of verbs. In Proceedings of the 52nd annual meeting of the association for computational linguistics (vol. 2: Short Papers, pp. 397–402). Baltimore, Maryland, USA, 23–25 June 2014.Google Scholar
Hartshorne, J. K., & Snedeker, J. (2013). Verb argument structure predicts implicit causality: The advantages of finer-grained semantics. Language and Cognitive Processes, 28(10), 1474–1508.CrossRefGoogle Scholar
Hartshorne, J. K., Tenenbaum, J. B., & Pinker, S. (2018). A critical period for second language acquisition: Evidence from 2/3 million English speakers. Cognition, 177, 263–277.CrossRefGoogle ScholarPubMed
Henrich, J., Heine, S. J., & Norenzayan, A. (2010). The weirdest people in the world? Behavioral and Brain Sciences, 33(2–3), 61–83.CrossRefGoogle ScholarPubMed
Hopcroft, J. E., Motwani, R., & Ullman, J. D. (2001). Introduction to automata theory, languages, and computation. ACM Sigact News, 32(1), 60–65.CrossRefGoogle Scholar
Judd, C. M., Westfall, J., & Kenny, D. A. (2012). Treating stimuli as a random factor in social psychology: A new and comprehensive solution to a pervasive but largely ignored problem. Journal of Personality and Social Psychology, 103(1), 54.CrossRefGoogle ScholarPubMed
Kuhn, T. S. (2012). The structure of scientific revolutions. University of Chicago Press.CrossRefGoogle Scholar
Li, W., Germine, L. T., Mehr, S. A., Srinivasan, M., & Hartshorne, J. K. (2022). Developmental psychologists should adopt citizen science to improve generalization and reproducibility. Infant and Child Development, e2348.CrossRefGoogle Scholar
Long, B., Simson, J., Buxó-Lugo, A., Watson, D. G., & Mehr, S. A. (2023). How games can make behavioural science better. Nature, 613(7944), 433–436.CrossRefGoogle Scholar
Nosek, B. A., Banaji, M. R., & Greenwald, A. G. (2002). Harvesting implicit group attitudes and beliefs from a demonstration web site. Group Dynamics: Theory, Research, and Practice, 6(1), 101.CrossRefGoogle Scholar
O'Donnell, T. J. (2015). Productivity and reuse in language: A theory of linguistic computation and storage. MIT Press.CrossRefGoogle Scholar
Peterson, J. C., Bourgin, D. D., Agrawal, M., Reichman, D., & Griffiths, T. L. (2021). Using large-scale experiments and machine learning to discover theories of human decision-making. Science (New York, N.Y.), 372(6547), 1209–1214.CrossRefGoogle ScholarPubMed
Riley, E., Okabe, H., Germine, L., Wilmer, J., Esterman, M., & DeGutis, J. (2016). Gender differences in sustained attentional control relate to gender inequality across countries. PLoS ONE, 11(11), e0165100.CrossRefGoogle ScholarPubMed
Soto, C. J., John, O. P., Gosling, S. D., & Potter, J. (2011). Age differences in personality traits from 10 to 65: Big five domains and facets in a large cross-sectional sample. Journal of Personality and Social Psychology, 100(2), 330.CrossRefGoogle Scholar
Spiers, H. J., Coutrot, A., & Hornberger, M. (2023). Explaining world-wide variation in navigation ability from millions of people: Citizen science project Sea Hero Quest. Topics in Cognitive Science, 15(1), 120–138.CrossRefGoogle ScholarPubMed
Steedman, M. (2001). The syntactic process. MIT Press.CrossRefGoogle Scholar
Ullman, M. T. (2001). The declarative/procedural model of lexicon and grammar. Journal of Psycholinguistic Research, 30, 37–69.CrossRefGoogle ScholarPubMed
Yarkoni, T. (2022). The generalizability crisis. Behavioral and Brain Sciences, 45, e1.CrossRefGoogle Scholar