Skip to main content

Does Survey Mode Still Matter? Findings from a 2010 Multi-Mode Comparison

  • Stephen Ansolabehere (a1) and Brian F. Schaffner (a2)

In this article, we present data from a three-mode survey comparison study carried out in 2010. National surveys were fielded at the same time over the Internet (using an opt-in Internet panel), by telephone with live interviews (using a national Random Digit Dialing (RDD) sample of landlines and cell phones), and by mail (using a national sample of residential addresses). Each survey utilized a nearly identical questionnaire soliciting information across a range of political and social indicators, many of which can be validated with government data. Comparing the findings from the modes using a Total Survey Error approach, we demonstrate that a carefully executed opt-in Internet panel produces estimates that are as accurate as a telephone survey and that the two modes differ little in their estimates of other political indicators and their correlates.

Corresponding author
e-mail: (corresponding author)
Hide All
AAPOR Opt-In Online Panel Task Force. 2010. AAPOR Report on Online Panels.
Ansolabehere, Stephen. 2009. Guide to the 2008 Cooperative Congressional Election Survey.
Ansolabehere, Stephen, and Schaffner, Brian F. 2013. Replication Data for: Does Survey Mode Still Matter? Findings from a 2010 Multi-Mode Comparison. Dataverse Network [Distributor] V1 [Version].
Atkeson, Lonna Rae, Bryant, Lisa A., Adams, Alex N., Zilberman, Luciana, and Saunders, Kyle L. 2011. Considering mixed mode surveys for questions in political behavior: Using the Internet and mail to get quality data at reasonable costs. Political Behavior 33: 161-78.
Atkeson, Lonna Rae, and Tafoya, Lori. 2008. Surveying political activists: An examination of the effectiveness of a mixed mode (Internet and mail) survey design. Journal of Elections, Public Opinion, and Parties 18 (4): 367-86.
Biemer, Paul P. 2010. Total survey error: Design, implementation, and evaluation. Public Opinion Quarterly 74 (5): 817-48.
Blumberg, Stephen J., and Luke, Julian V. 2009. Wireless Substitution: Early Release of Estimates from the National Health Interview Survey, January-June 2011. Centers for Disease Control.
Chang, Linchiat, and Krosnick, Jon A. 2009. National surveys via RDD telephone interviewing versus the Internet: Comparing sample representativeness and response quality. Public Opinion Quarterly 73 (4): 641-78.
Chang, Linchiat. 2010. Comparing oral interviewing with self-administered computerized questionnaires: An experiment. Public Opinion Quarterly 74 (1): 154-67.
Chatt, Cindy, Michael Dennis, J., Li, Rick, and Pulliam, Paul. 2003. Data collection mode effects controlling for sample origins in a panel survey: Telephone versus Internet. Paper presented at the Annual Meeting of the Midwest Chapter of the American Association for Public Opinion Research, Chicago.
Curtin, R., Presser, S., and Singer, E. 2005. Changes in telephone survey nonresponse over the past quarter century. Public Opinion Quarterly 69: 8798.
Carpini, Delli, Michael, X., and Keeter, Scott. 1997. What Americans know about politics and why it matters. New Haven, CT: Yale University Press.
Duncan, W. Jack. 1979. Mail questionnaires in survey research: A review of response inducement techniques. Journal of Management 5: 3955.
Groves, Robert M., Fowler, Floyd J. Jr., Couper, Mick P., Lepkowski, James M., and Singer, Eleanor. 2009. Survey Methodology. 2nd ed. Hoboken, NJ: J. Wiley.
Groves, Robert M., and Lyberg, Lars. 2010. Total survey error: Past, present, and future. Public Opinion Quarterly 74 (5): 849-79.
Hill, Seth J., Lo, James, Vavreck, Lynn, and Zaller, John. 2007. The opt-in Internet panel: Survey mode, sample methodology, and the implications for political research. Unpublished manuscript.
Kenski, Kate, and Jomini Stroud, Natalie. 2006. Connections between Internet use and political efficacy, knowledge, and participation. Journal of Broadcasting & Electronic Media 50 (2): 173-92.
Link, Michael W., and Mokdad, Ali H. 2005. Alternative models for health surveillance surveys: An experiment with web, mail, and telephone. Epidemiology 16 (5): 701-4.
Malhotra, Neil, and Krosnick, Jon A. 2007. The effect of survey mode and sampling on inferences about political attitudes and behavior: Comparing the 2000 and 2004 ANES to Internet surveys with nonprobability samples. Political Analysis 15: 286323.
Pasek, Josh, and Krosnick, Jon A. 2010. Measuring intent to participate and participation in the 2010 census and their correlates and trends: Comparisons of RDD telephone and non-probability sample Internet survey data. Statistical Research Division of the US Census Bureau (#2010–15).
Rivers, Douglas. 2006. Sample matching: Representative sampling from Internet panels.
Rivers, Douglas. 2007. Sampling for web surveys. Paper prepared for the 2007 Join Statistical Meetings, Salt Lake City, UT.
Rivers, Douglas. 2009. Second thoughts about Internet surveys. (September 6).
Sanders, David, Clarke, Harold D., Stewart, Marianne C., and Whiteley, Paul. 2007. Does mode matter for modeling political choice? Evidence from the 2005 British Election Study. Political Analysis 15: 257-85.
Smith, Tom W. 2011. Refining the total survey error perspective. International Journal of Public Opinion Research 23 (4): 464-84.
Yeager, David S., Krosnick, Jon A., Chang, LinChiat, Javitz, Harold S., Levendusky, Matthew S., Simpser, Alberto, and Wang, Rui. 2009. Comparing the accuracy of RDD telephone surveys and Internet surveys conducted with probability and non-probability samples. Unpublished manuscript.
Recommend this journal

Email your librarian or administrator to recommend adding this journal to your organisation's collection.

Political Analysis
  • ISSN: 1047-1987
  • EISSN: 1476-4989
  • URL: /core/journals/political-analysis
Please enter your name
Please enter a valid email address
Who would you like to send this to? *


Altmetric attention score