Hostname: page-component-5d84bcc8dc-whph2 Total loading time: 0 Render date: 2026-08-10T13:10:27.105Z Has data issue: false hasContentIssue false
  • English
  • Français

Placebo Selection in Survey Experiments: An Agnostic Approach

Published online by Cambridge University Press:  14 June 2021

Ethan Porter  and
Yamil R. Velez Open the ORCID record for Yamil R. Velez [Opens in a new window]
Ethan Porter
Affiliation:
School of Media and Public Affairs, The George Washington University, 805 21st Street NW, Washington, DC 20052, USA. Email: evporter@gwu.edu Institute for Data, Democracy & Politics, The George Washington University, 805 21st Street NW, Washington, DC 20052, USA Department of Political Science, The George Washington University, 805 21st Street NW, Washington, DC 20052, USA
Yamil R. Velez*
Affiliation:
Department of Political Science, Columbia University, 741 International Affairs Building, New York, NY 10027, USA. Email: yrv2004@columbia.edu
*
Corresponding author Yamil R. Velez
Get access Rights & Permissions [Opens in a new window]

Abstract

Although placebo conditions are ubiquitous in survey experiments, little evidence guides common practices for their use and selection. How should scholars choose and construct placebos? First, we review the role of placebos in published survey experiments, finding that placebos are used inconsistently. Then, drawing on the medical literature, we clarify the role that placebos play in accounting for nonspecific effects (NSEs), or the effects of ancillary features of experiments. We argue that, in the absence of precise knowledge of NSEs that placebos are adjusting for, researchers should average over a corpus of many placebos. We demonstrate this agnostic approach to placebo construction through the use of GPT-2, a generative language model trained on a database of over 1 million internet news pages. Using GPT-2, we devise 5,000 distinct placebos and administer two experiments (N = 2,975). Our results illustrate how researchers can minimize their role in placebo selection through automated processes. We conclude by offering tools for incorporating computer-generated placebo text vignettes into survey experiments and developing recommendations for best practice.

Keywords

survey experimentsplacebos

Information

Type
Article
Information
Political Analysis , Volume 30 , Issue 4 , October 2022 , pp. 481 - 494
Copyright
© The Author(s) 2021. Published by Cambridge University Press on behalf of the Society for Political Methodology

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

Article purchase

Temporarily unavailable

Footnotes

Edited by Sunshine Hillygus

*

Authors' names appear in alphabetical order.

References

Broockman, D., and Kalla, J.. 2020. “When and Why Are Campaigns’ Persuasive Effects Small? Evidence from the 2020 US Presidential Election.” OSF Preprints. doi:10.31219/osf.io/m7326.CrossRefGoogle Scholar
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., et al. 2020. “Language Models Are Few-Shot Learners.” Preprint, arXiv:2005.14165.Google Scholar
Colloca, L., and Barsky, A. J.. 2020. “Placebo and Nocebo Effects.” New England Journal of Medicine 382(6):554–561. http://doi.org/10.1056/NEJMra1907805.CrossRefGoogle ScholarPubMed
Colloca, L., and Benedetti, F.. 2005. “Placebos and Painkillers: Is Mind as Real as Matter?” Nature Reviews Neuroscience 6(7):545–552.CrossRefGoogle ScholarPubMed
Coppock, A., Hill, S. J., and Vavreck, L.. 2020. “The Small Effects of Political Advertising Are Small Regardless of Context, Message, Sender, or Receiver: Evidence from 59 Real-Time Randomized Experiments.” Science Advances 6(36):1–6.CrossRefGoogle ScholarPubMed
Dickersin, K., Chan, S., Chalmers, T., Sacks, H., and Smith, H. Jr. 1987. “Publication Bias and Clinical Trials.” Controlled Clinical Trials 8(4):343–353.CrossRefGoogle ScholarPubMed
Franco, A., Malhotra, N., and Simonovits, G.. 2014. “Publication Bias in the Social Sciences: Unlocking the File Drawer.” Science 345(6203): 1502–1505.CrossRefGoogle ScholarPubMed
Geersa, A. L., and Miller, F. G.. 2014. “Understanding and Translating the Knowledge About Placebo Effects: The Contribution of Psychology.” Current Opinion Psychiatry 27(5):326–331.CrossRefGoogle Scholar
Gerber, A. S., Green, D. P., Kaplan, E. H., and Kern, H. L.. 2010. “Baseline, Placebo, and Treatment: Efficient Estimation for Three-Group Experiments.” Political Analysis 18(3):297–315. http://doi.org/10.1093/pan/mpq008.CrossRefGoogle Scholar
Gilbert, C., and Eric, H.. 2014. “Vader: A Parsimonious Rule-Based Model for Sentiment Analysis of Social Media Text.” In Eighth International Conference on Weblogs and Social Media (ICWSM-14), 81–82. Palo Alto, CA: AAAI Press.Google Scholar
Harden, J. J., Sokhey, A. E., and Runge, K. L.. 2019. “Accounting for Noncompliance in Survey Experiments.” Journal of Experimental Political Science 6(3):199–202.CrossRefGoogle Scholar
Litman, L., Robinson, J., and Abberbock, T.. 2017. “Turkprime.com: A Versatile Crowdsourcing Data Acquisition Platform for the Behavioral Sciences.” Behavior Research Methods 49(2):433–442.CrossRefGoogle ScholarPubMed
Montgomery, G. H., and Kirsch, I.. 1997. “Classical Conditioning and the Placebo Effect.” Pain 72(1–2):107–113.CrossRefGoogle ScholarPubMed
Mullinix, K. J., Leeper, T. J., Druckman, J. N., and Freese, J.. 2015. “The Generalizability of Survey Experiments.” Journal of Experimental Political Science 2(2):109–138.CrossRefGoogle Scholar
Mummolo, J., and Peterson, E.. 2019. “Demand Effects in Survey Experiments: An Empirical Assessment.” American Political Science Review 113(2):517–529. http://doi.org/10.1017/S0003055418000837.CrossRefGoogle Scholar
Nelson, T. E., Clawson, R. A., and Oxley, Z. M.. 1997. “Media Framing of a Civil Liberties Conflict and Its Effect on Tolerance. American Political Science Review 91(3): 567–583.CrossRefGoogle Scholar
Nickerson, D. W. 2005. “Scalable Protocols Offer Efficient Design for Field Experiments.” Political Analysis 13(3):233–252. http://doi.org/10.1093/pan/mpi015.CrossRefGoogle Scholar
Nosek, B., and Open Science Framework. 2015. “Estimating the Reproducibility of Psychological Science.” Science 349(6251): aac4716.Google Scholar
Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., and Sutskever, I.. 2019. “Language Models Are Unsupervised Multitask Learners.” OpenAI Blog 1(8):9.Google Scholar
Rubin, D. B. 2005. “Causal Inference Using Potential Outcomes: Design, Modeling, Decisions.” Journal of the American Statistical Association 100(469):322–331.CrossRefGoogle Scholar
Snowsill, T., Flaounas, I., De Bie, T., and Cristianini, N.. 2010. “Detecting Events in a Million New York Times Articles.” In Joint European Conference on Machine Learning and Knowledge Discovery in Databases, 615–618. Berlin: Springer.CrossRefGoogle Scholar
Vambheim, S. M., and Flaten, M. A.. 2017. “A Systematic Review of Sex Differences in the Placebo and the Nocebo Effect.” Journal of Pain Research 10:1831–1839.CrossRefGoogle ScholarPubMed
Velez, Y., and Porter, E.. 2021. “Replication Data for: Placebo Selection in Survey Experiments: An Agnostic Approach.” https://doi.org/10.7910/DVN/4PYOXP, Harvard Dataverse, V1, UNF:6:tg08CtjaYGUVLsprGUDh1A== [fileUNF].Google Scholar
Wells, G. L., and Windschitl, P. D.. 1999. “Stimulus Sampling and Social Psychological Experimentation.” Personality and Social Psychology Bulletin 25(9):1115–1125.CrossRefGoogle Scholar
White, A., Strezhnev, A., Lucas, C., Kruszewska, D., and Huff, C.. 2018. “Investigator Characteristics and Respondent Behavior in Online Surveys.” Journal of Experimental Political Science 5(1):56–67.CrossRefGoogle Scholar
Woolf, M. 2020. “Gpt-2-Simple, a Python Package.” https://github.com/minimaxir/gpt-2-simple.Google Scholar
Zellers, R., Holtzman, A., Rashkin, H., Bisk, Y., Farhadi, A., Roesner, F., and Choi, Y.. 2019. “Defending Against Neural Fake News.” In Advances in Neural Information Processing Systems 32, edited by H. Wallach, H. Larochelle, A. Beygelzimer, F. d’Alché-Buc, E. Fox, and R. Garnett, 9054–9065.Google Scholar
Supplementary material: Link

Porter and Velez Dataset

Dataset

https://doi.org/10.7910/DVN/4PYOXP
Link
Supplementary material: PDF

Porter and Velez Supplementary Material

Porter and Velez Supplementary Material Appendix

Download Porter and Velez Supplementary Material(PDF)
PDF 457.1 KB