Hostname: page-component-77f85d65b8-2tv5m Total loading time: 0 Render date: 2026-03-28T12:16:15.090Z Has data issue: false hasContentIssue false
  • English
  • Français

Measurement practices exacerbate the generalizability crisis: Novel digital measures can help

Published online by Cambridge University Press:  10 February 2022

Brittany I. Davidson Open the ORCID record for Brittany I. Davidson [Opens in a new window] ,
David A. Ellis ,
Clemens Stachl ,
Paul J. Taylor  and
Adam N. Joinson
Brittany I. Davidson
Affiliation:
School of Management, University of Bath, Claverton Down, Bath BA2 7AY, UK bid23@bath.ac.uk, https://www.brittanydavidson.co.uk/; dae30@bath.ac.uk, http://www.davidaellis.co.uk/; aj266@bath.ac.uk, http://www.joinson.com/home/Welcome.html Department of Engineering, University of Bristol, Bristol BS1 5DD, UK
David A. Ellis
Affiliation:
School of Management, University of Bath, Claverton Down, Bath BA2 7AY, UK bid23@bath.ac.uk, https://www.brittanydavidson.co.uk/; dae30@bath.ac.uk, http://www.davidaellis.co.uk/; aj266@bath.ac.uk, http://www.joinson.com/home/Welcome.html
Clemens Stachl
Affiliation:
Institute of Behavioral Science & Technology, University of St. Gallen, CH-9000, Switzerland cstachl@stanford.edu, https://www.clemensstachl.com
Paul J. Taylor
Affiliation:
Department of Psychology, Lancaster University, Bailrigg, Lancaster LA1 4YW, UK p.j.taylor@lancaster.ac.uk, https://pauljtaylor.com/
Adam N. Joinson
Affiliation:
School of Management, University of Bath, Claverton Down, Bath BA2 7AY, UK bid23@bath.ac.uk, https://www.brittanydavidson.co.uk/; dae30@bath.ac.uk, http://www.davidaellis.co.uk/; aj266@bath.ac.uk, http://www.joinson.com/home/Welcome.html
Get access Rights & Permissions [Opens in a new window]

Abstract

Psychology's tendency to focus on confirmatory analyses before ensuring constructs are clearly defined and accurately measured is exacerbating the generalizability crisis. Our growing use of digital behaviors as predictors has revealed the fragility of subjective measures and the latent constructs they scaffold. However, new technologies can provide opportunities to improve conceptualizations, theories, and measurement practices.

Information

Type
Open Peer Commentary
Information
Behavioral and Brain Sciences , Volume 45 , 2022 , e10
Check for updates
Copyright
Copyright © The Author(s), 2022. Published by Cambridge University Press

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

Article purchase

Temporarily unavailable

References

Baumeister, R. F., Vohs, K. D., & Funder, D. C. (2007). Psychology as the science of self- reports and finger movements: Whatever happened to actual behavior? Perspectives on Psychological Science, 2(4), 396–403. https://doi.org/10.1111/j.1745-6916.2007.00051.x.CrossRefGoogle ScholarPubMed
Boyd, R. L., Pasca, P., & Lanning, K. (2020). The personality panorama: Conceptualizing personality through Big behavioural data. European Journal of Personality, 34(5), 599–612. https://doi.org/10.1002/per.2254.CrossRefGoogle Scholar
Cattell, R. B. (1958). What is “objective” in “objective personality tests”? Journal of Counseling Psychology, 5(4), 285. https://doi.org/10.1037/h0046268.CrossRefGoogle Scholar
Davidson, B. I. (2020). The crossroads of digital phenotyping. General Hospital Psychiatry. https://doi.org/10.1016/j.genhosppsych.2020.11.009.Google ScholarPubMed
DeMasi, O., Kording, K., & Recht, B. (2017). Meaningless comparisons lead to false optimism in medical machine learning. PLoS ONE, 12(9), e0184604. https://doi.org/10.1371/journal.pone.0184604.CrossRefGoogle ScholarPubMed
Eisenberg, I. W., Bissett, P. G., Zeynep Enkavi, A., Li, J., MacKinnon, D. P., Marsch, L. A., & Poldrack, R. A. (2019). Uncovering the structure of self-regulation through data-driven ontology discovery. Nature Communications, 10(1), 2319. https://doi.org/10.1038/s41467-019-10301-1.CrossRefGoogle ScholarPubMed
Ellis, D. A. (2020). Smartphones within psychological science. Cambridge University Press.CrossRefGoogle Scholar
Geyer, K., Ellis, D. A., Shaw, H., & Davidson, B. I. (2020). Open source smartphone app and tools for measuring, quantifying, and visualizing technology use [Preprint]. PsyArXiv. https://doi.org/10.31234/osf.io/eqhfa.CrossRefGoogle Scholar
Glewwe, P., & van der Gaag, J. (1990). Identifying the poor in developing countries: Do different definitions matter? World Development, 18(6), 803–814. https://doi.org/10.1016/0305-750X(90)90003-G.CrossRefGoogle Scholar
Guttman, R., & Greenbaum, C. W. (1998). Facet theory: It's development and current status. European Psychologist, 3, 13–36. https://doi.org/10.1027/1016-9040.3.1.13.CrossRefGoogle Scholar
Harari, G. M., Müller, S. R., Stachl, C., Wang, R., Wang, W., Bühner, M., … Gosling, S. D. (2020). Sensing sociability: Individual differences in young adults’ conversation, calling, texting, and app use behaviors in daily life. Journal of Personality and Social Psychology, 119(1), 204–228. https://doi.org/10.1037/pspp0000245.CrossRefGoogle ScholarPubMed
Hayes, S. C., & Brownstein, A. J. (1986). Mentalism, behavior-behavior relations, and a behavior-analytic view of the purposes of science. The Behavior Analyst, 9(2), 175–190. https://doi.org/10.1007/BF03391944.CrossRefGoogle Scholar
Hedge, C., Powell, G., & Sumner, P. (2018). The reliability paradox: Why robust cognitive tasks do not produce reliable individual differences. Behavior Research Methods, 50(3), 1166–1186.CrossRefGoogle Scholar
Hinds, J., & Joinson, A. (2019). Human and computer personality prediction from digital footprints. Current Directions in Psychological Science, 28(2), 204–211.CrossRefGoogle Scholar
Howard, M. C., & Jayne, B. S. (2015). An analysis of more than 1,400 articles, 900 scales, and 17 years of research: The state of scales in cyberpsychology, behavior, and social networking. Cyberpsychology, Behavior, and Social Networking, 18(3), 181–187.CrossRefGoogle ScholarPubMed
Kagan, J. (2001). The need for new constructs. Psychological Inquiry, 12(2), 84–103. https://doi.org/10.1207/S15327965PLI1202_03.CrossRefGoogle Scholar
Kaplan, A. (1964). The conduct of inquiry: Methodology for behavioral science. Chandler Publishing Company.Google Scholar
Lazer, D. M. J., Pentland, A., Watts, D. J., Aral, S., Athey, S., Contractor, N., … Wagner, C. (2020). Computational social science: Obstacles and opportunities. Science, 369(6507), 1060–1062. https://doi.org/10.1126/science.aaz8170.CrossRefGoogle ScholarPubMed
Miller, G. (2012). The smartphone psychology manifesto. Perspectives on Psychological Science, 7(3), 221–237. https://doi.org/10.1177/1745691612441215.CrossRefGoogle ScholarPubMed
Mischel, W. (2004). Toward an integrative science of the person. Annual Review of Psychology, 55(1), 1–22. https://doi.org/10.1146/annurev.psych.55.042902.130709.CrossRefGoogle Scholar
Parry, D. A., Davidson, B. I., Sewall, C., Fisher, J. T., Mieczkowski, H., & Quintana, D. (2020). A systematic review and meta-analysis of discrepancies between logged and self-reported digital media use. PsyArXiv. doi:10.31234/osf.io/f6xvz.CrossRefGoogle Scholar
Piwek, L., Ellis, D. A., Andrews, S., & Joinson, A. (2016). The rise of consumer health wearables: Promises and barriers. PLoS Medicine, 13(2), e1001953. https://doi.org/10.1371/journal.pmed.1001953.CrossRefGoogle ScholarPubMed
Sen, I., Floeck, F., Weller, K., Weiss, B., & Wagner, C. (2019). A total error framework for digital traces of humans. ArXiv:1907.08228 [Cs]. http://arxiv.org/abs/1907.08228.Google Scholar
Silberzahn, R., Uhlmann, E. L., Martin, D. P., Anselmi, P., Aust, F., Awtrey, E., … Carlsson, R. (2018). Many analysts, one data set: Making transparent how variations in analytic choices affect results. Advances in Methods and Practices in Psychological Science, 1(3), 337–356.CrossRefGoogle Scholar
Skinner, B. F. (1963). Behaviorism at fifty. Science, 140(3570), 951–958.CrossRefGoogle ScholarPubMed
Skinner, B. F. (1976). About behaviorism. Vintage Books.Google Scholar
Stachl, C., Au, Q., Schoedel, R., Gosling, S. D., Harari, G. M., Buschek, D., … Bühner, M. (2020). Predicting personality from patterns of behavior collected with smartphones. Proceedings of the National Academy of Sciences, 117(30), 17680–17687. https://doi.org/10.1073/pnas.1920484117.CrossRefGoogle ScholarPubMed
Szot, T., Specht, C., Specht, M., & Dabrowski, P. S. (2019). Comparative analysis of positioning accuracy of Samsung Galaxy smartphones in stationary measurements. PLoS ONE, 14(4), e0215562. https://doi.org/10.1371/journal.pone.0215562.CrossRefGoogle ScholarPubMed
Taylor, P. J., Banks, F., Jolley, D., Ellis, D. A., Watson, S. J., Weiher, L., … Julku, J. (2021). Oral hygiene effects verbal and nonverbal displays of confidence. Journal of Social Psychology, 161(2), 182–196. doi: 10.1080/00224545.2020.1784825.CrossRefGoogle Scholar
Towse, J. N., Ellis, D. A., & Towse, A. S. (2020). Opening Pandora's Box: Peeking inside psychology's data sharing practices, and seven recommendations for change. Behavior Research Methods, 53(4), 1455–1468. https://doi.org/10.3758/s13428-020-01486-1.CrossRefGoogle Scholar
Uttal, W. R. (2001). The new phrenology: The limits of localizing cognitive processes in the brain. MIT Press.Google Scholar
Zulueta, J., Piscitello, A., Rasic, M., Easter, R., Babu, P., Langenecker, S. A., … Leow, A. (2018). Predicting mood disturbance severity with mobile phone keystroke metadata: A biaffect digital phenotyping study. Journal of Medical Internet Research, 20(7), e241.CrossRefGoogle ScholarPubMed