Skip to main content
×
Home
    • Aa
    • Aa

Is There a Cost to Convenience? An Experimental Comparison of Data Quality in Laboratory and Online Studies

  • Scott Clifford (a1) and Jennifer Jerit (a2)
Abstract
Abstract

Increasingly, experimental research is being conducted on the Internet in addition to the laboratory. Online experiments are more convenient for subjects and researchers, but we know little about how the choice of study location affects data quality. To investigate whether respondent behavior differs across study location, we randomly assign subjects to participate in a study in a laboratory or in an online setting. Contrary to our expectations, we find few differences between participants in terms of the level of attention and socially desirable responding. However, we find significant differences in two areas: the degree of self-reported distractions while completing the questionnaire and the tendency to consult outside sources for answers to political knowledge questions. We conclude that when the greater convenience (and higher response rates) of online experiments outweighs these disadvantages, Internet administration of randomized experiments represent an alternative to laboratory administration.

Copyright
References
Hide All
BarghJ., and ChartrandT. L. 2000. The Mind in the Middle: A Practical Guide to Priming and Automaticity Research. In Handbook of Research Methods in Social and Personality Psychology eds. Reis H. T. and Judd C. M.. New York: Cambridge University Press, 253–85.
BerinskyA. J. 1999. Can We Talk? Self-Presentation and the Survey Response. Political Psychology 25 (4): 643–59.
BerinskyA. J., HuberG. A., and LenzG. S. 2012. Using Mechanical Turk as a Subject Recruitment Tool for Experimental Research. Political Analysis 20 (3): 351–68.
BerinskyA. J., MargolisM., and SancesM. 2014. Separating the Shirkers from the Workers? Making Sure Respondents Pay Attention on Internet Surveys. American Journal of Political Science Forthcoming.
BerryD. T. R., WetterM. W., BaerR. A., LarsenL., ClarkC., and MonroeK. 1992. MMPI-2 Random Responding Indices: Validation Using a Self-Report Methodology. Psychological Assessment 4 (3): 340–45.
BosterF., and ShulmanH. 2014. Political Knowledge Test Performance as a Function of Venue, Time Pressure, and Performance Norms. Working Paper, North Central College.
CasseseE. C., HuddyL., HartmanT. K., MasonL., and WeberC. R. 2013. Socially Mediated Internet Surveys: Recruiting Participants for Online Experiments. PS: Political Science and Politics 46 (4): 775–84.
ChangL., and KrosnickJ. A. 2009. National Surveys via RDD Telephone Interviewing versus the Internet. Public Opinion Quarterly 73 (4): 641–78.
ChangL., and KrosnickJ. A. 2010. Comparing Oral Interviewing with Self-Administered Computerized Questionnaires. Public Opinion Quarterly 74 (1): 154–67.
CoppockA., and GreenD. P. 2013. Assessing the Correspondence Between Experimental Results Obtained in the Lab and Field: A Review of Recent Social Science Research. Working Paper, Columbia University.
EvansD. C., GarciaD. J., GarciaD. M., and BaronR. S. 2003. In the Privacy of Their Own Homes: Using the Internet to Assess Racial Bias. Personality and Social Psychology Bulletin 29 (2): 273–84.
GerberA. 2011. Field Experiments in Political Science. In Handbook of Experimental Political Science eds. Druckman J., Green D. P., Kuklinski J. H., and Lupia A.. New York: Cambridge University Press, 115–38.
GoodmanJ. K., CryderC. E., and CheemaA. 2013. Data Collection in a Flat World: The Strengths and Weaknesses of Mechanical Turk Samples. Journal of Behavioral Decision Making 26 (3): 213–24.
HuangJ. L., CurranP. G., KeeneyJ., PoposkiE. M., and DeShonR. P. 2012. Detecting and Deterring Insufficient Effort Responding to Surveys. Journal of Business Psychology 27: 99114.
JeritJ., BarabasJ., and CliffordS. 2013. Comparing Contemporaneous Laboratory and Field Experiments on Media Effects. Public Opinion Quarterly 77 (1): 256–82.
LelkesY., KrosnickJ. A., MarxD. M., JuddC. N., and ParkB. 2012. Complete Anonymity Compromises the Accuracy of Self-Reports. Journal of Experimental Social Psychology 48: 1291–99.
LodgeM., and TaberC. S. 2013. The Rationalizing Voter. New York: Cambridge University Press.
MeadeA. W., and CraigS. B. 2012. Identifying Careless Responses in Survey Data. Psychological Methods 17 (3): 437–55.
McConahayJ. G. 1986. Modern Racism, Ambivalence, and the Modern Racism Scale. In Prejudice, Discrimination, and Racism eds. Dovidio J. F. and Gaertner S. L.. New York: Academic Press, 91125.
McDermottR. 2002. Experimental Methods in Political Science. Annual Review of Political Science 5: 3161.
MortonR. B., and WilliamsK. C. 2010. Experimental Political Science and the Study of Causality. New York: Cambridge University Press.
MutzD. 2011. Population-Based Survey Experiments. Princeton, NJ: Princeton University Press.
OppenheimerD., MeyvisT., and DavidenkoN. 2009. Instructional Manipulation Checks: Detecting Satisficing to Increase Statistical Power. Journal of Experimental Social Psychology 45: 867–72.
PriorM. 2009. Improving Media Effects Research Through Better Measurement of News Exposure. Journal of Politics 71 (3): 893908.
SargisE. G., SkitkaL. J., and McKeeverW. 2014. The Internet as Psychological Laboratory Revisited: Best Practices, Challenges, and Solutions. In The Social Net: The Social Psychology of the Internet ed. Amichai-Hamberger Y.. Oxford: Oxford University Press. In Press.
TourangeauR., and SmithT. W. 1996. Asking Sensitive Questions: The Impact of Data Collection Mode, Question Format, and Question Context. Public Opinion Quarterly 60 (2): 275304.
TourangeauR., and YanT. 2007. Sensitive Survey Questions. Psychological Bulletin 133 (5): 859–83.
VavreckL. 2012. The Myth of Cheating on Self-Completed Surveys. http://today.yougov.com/news/2012/04/17/myth-cheating-self-completed-surveys/ (Accessed June 10, 2013).
WarrenJ. 2012. Fake Orgasms and the Tea Party: Just Another Political Science Convention. The Atlantic Online. http://www.theatlantic.com/politics/archive/2012/04/fake-orgasms-and-the-tea-party-just-another-political-science-convention/255909/ (Accessed June 10, 2013).
WeigoldA., WeigoldI. K., and RussellE. J. 2013. Examination of the Equivalence of Self-Report Survey-Based Paper-and-Pencil and Internet Data Collection Methods. Psychological Methods 18 (1): 5370.
WeinbergerJ., and WestenD. 2008. RATS, We Should Have Used Clinton: Subliminal Priming in Political Campaigns. Political Psychology 29 (5): 631–51.
Recommend this journal

Email your librarian or administrator to recommend adding this journal to your organisation's collection.

Journal of Experimental Political Science
  • ISSN: 2052-2630
  • EISSN: 2052-2649
  • URL: /core/journals/journal-of-experimental-political-science
Please enter your name
Please enter a valid email address
Who would you like to send this to? *
×

Keywords:

Type Description Title
PDF
Supplementary Materials

Clifford and Jerit supplementary material
Appendix

 PDF (910 KB)
910 KB
PDF
Supplementary Materials

Clifford and Jerit Supplementary Material
Appendix

 PDF (910 KB)
910 KB

Metrics

Altmetric attention score

Full text views

Total number of HTML views: 12
Total number of PDF views: 167 *
Loading metrics...

Abstract views

Total abstract views: 504 *
Loading metrics...

* Views captured on Cambridge Core between September 2016 - 19th October 2017. This data will be updated every 24 hours.