Skip to main content
×
×
Home

Self-Selection and Misreporting in Legislative Surveys

  • Pirmin Bundi, Frédéric Varone, Roy Gava and Thomas Widmer
Abstract

This article discusses the methodological challenges of legislative surveys. Following an overview of different types of survey biases, the article argues that self-selection and misreporting are the most critical problems for legislative surveys. In order to identify the self-selection and misreporting biases, we compare the answers with a survey from Swiss members of parliament with their observed behavior in the parliament. The empirical analysis shows that the survey sample has a substantial misreporting bias. We conclude that the parliamentarian’s attitude toward the survey object is strongly linked to their response. These issues should be addressed to improve the design and quality of legislative surveys.

Copyright
Footnotes
Hide All
*

Pirmin Bundi, PhD Student, Department of Political Science, University of Zurich, Affolternstr. 56, 8050 Zurich (bundi@ipz.uzh.ch). Frédéric Varone, Professor, Department of Political Science and International Relations, University of Geneva, Bvd du Pont d’Arve 40, 1211 Geneva (frederic.varone@unige.ch). Roy Gava, Lecturer and Postdoctoral Researcher, Department of Political Science and International Relations, University of Geneva, Bvd du Pont d’Arve 40, 1211 Geneva (roy.gava@unige.ch). Thomas Widmer, Professor, Department of Political Science, University of Zurich, Affolternstr. 56, 8050 Zurich (thow@ipz.uzh.ch). A previous version of this article was presented at the ESRA Conference 2015 in Reykjavik. The authors thank all the participants for their feedback. In addition, the authors would like to thank Thomas De Rocchi, Daniela Eberli, Kathrin Frey, Fabrizio Gilardi, Andreas C. Goldberg, Simon Hug, Simon Lanz, Augusta Prorok and Marco Steenbergen for their helpful comments. The authors are grateful to Kenneth Benoit and two anonymous reviewers for their valuable remarks and to the Swiss National Science Foundation for financial support.

Footnotes
References
Hide All
Aelst, Peter V., Sehata, Adam, and Dalen, Arjen Van. 2010. ‘Members of Parliament: Equal Competitors for Media Attention? An Analysis of Personal Contacts Between MPs and Political Journalists in Five European Countries’. Political Communication 27(3):310325.
Ambwani, Suman, and Chmielewski, Jennifer F.. 2013. ‘Weighing the Evidence: Social Desirability, Eating Disorder Symptomatology, and Accuracy of Self-Reported Body Weight Among Men and Women’. Sex Roles 68(7):474483.
Ansolabehere, Stephen, and Hersh, Eitan. 2012. ‘Validation: What Big Data Reveal About Survey Misreporting and the Real Electorate’. Political Analysis 20(3):437459.
Arnold, Jason Ross. 2012. ‘Political Awareness, Corruption Perceptions and Democratic Accountability in Latin America’. Acta Politica 47(1):6790.
Bailer, Stefanie. 2011. ‘People’s Voice or Information Pool? The Role of, and Reasons for, Parliamentary Questions in the Swiss parliament’. The Journal of Legislative Studies 17(3):302314.
Bailer, Stefanie. 2014. ‘Surveys and Interviews in the Study of Legislatures’. In Shane Martin, Thomas Saalfeld and Karre W. Strom (eds), Oxford Handbook of Legislative Studies, 167193. New York, NY: Oxford University Press.
Bailer, Stefanie, Meissner, Peter, Ohmura, Tamaki, and Selb, Peter. 2013. Seiteneinsteiger im Deutschen Bundestag. Wiesbaden: Springer-Verlag.
Behnke, Joachim. 2005. ‘Lassen Sich Signifikanztests auf Vollerhebungen anwenden? Einige Essayistische Anmerkungen’. Politische Vierteljahresschrift 46(1):O1O15.
Beichelt, Timm. 2012. ‘Recovering Space Lost? The German Bundestag’s New Potential in European Politics’. German Politics 21(2):143160.
Belli, Robert F., Traugott, Michael W., Young, Margaret, and McGonagle, Katherine A.. 1999. ‘Reducing Vote Overreporting in Surveys: Social Desirability, Memory Failure, and Source Monitoring’. Public Opinion Quarterly 63(1):90108.
Bethlehem, Jelke. 2010. ‘Selection Bias in Web Surveys’. International Statistical Review 78(2):161188.
Bowler, Shaun, and Farrell, David M.. 2011. ‘Electoral Institutions and Campaigning in Comparative Perspective: Electioneering in European Parliament Elections’. European Journal of Political Research 50(5):668688.
Brick, J. Michael, and Williams, Douglas. 2013. ‘Explaining Rising Nonresponse Rates in Cross-Sectional Surveys’. The ANNALS of the American Academy of Political and Social Science 645(1):3659.
Brun, Mathias E., and Siegel, John Philipp. 2006. ‘What Does Appropriate Performance Reporting for Political Decision Makers Require? Empirical Evidence from Switzerland’. International Journal of Productivity and Performance Management 55(6):480497.
Bundi, Pirmin. 2016. ‘What Do We Know About the Demand for Evaluations? Insights from the Parliamentary Arena’. American Journal of Evaluation, doi:10.1177/1098214015621788.
Bundi, Pirmin, Eberli, Daniela, Frey, Kathrin, and Widmer, Thomas. 2014. Befragung Parlamente und Evaluationen. Methodenbericht. Zürich: Universität Zürich.
Bütikofer, Sarah. 2014. Das Schweizer Parlament: Eine Institution auf dem Pfad der Moderne. Baden-Baden: Nomos Verlagsgesellschaft.
Bütikofer, Sarah, and Hug, Simon. 2015. ‘Strategic Behaviour in Parliament’. The Journal of Legislative Studies 21(3):295322.
Chung, Janne, and Monroe, Gary S.. 2003. ‘Exploring Social Desirability Bias’. Journal of Business Ethics 44(4):291302.
Comşa, Mircea, and Postelnicu, Camil. 2013. ‘Measuring Social Desirability Effects on Self-Reported Turnout Using the Item Count Technique’. International Journal of Public Opinion Research 25(2):153172.
Corstange, Daniel. 2009. ‘Sensitive Questions, Truthful Answers? Modeling the List Experiment With LISTIT’. Political Analysis 17(1):4563.
Crowne, Douglas P., and Marlowe, David. 1960. ‘A New Scale of Social Desirability Independent of Psychopathology’. Journal of Consulting Psychology 24(4):349354.
DeMaio, Theresa J. 1984. ‘Social Desirability and Survey Measurement: A Review’. In Charles F. Turner and Elizabeth Martin (eds), Surveying Subjective Phenomena, 257282. Thousand Oaks, CA: Sage.
Deschouwer, Kris, and Depauw, Sam (eds) 2014. Representing the People: A Survey Among Members of Statewide and Substate Parliaments. Oxford: OUP.
Dubin, Jeffrey A., and Rivers, Douglas. 1989. ‘Selection Bias in Linear Regression, Logit and Probit Models’. Sociological Methods & Research 18(2–3):360390.
Eberli, Daniela, Bundi, Pirmin, Frey, Kathrin, and Widmer, Thomas. 2014. Befragung Parlamente und Evaluationen. Ergebnisbericht. Zürich: Universität Zürich.
Feh Widmer, Antoinette. 2015. Parlamentarische Fluktuation auf Subnationaler Ebene in der Schweiz. Baden-Baden: Nomos Verlagsgesellschaft.
Fowler, Floyd J. Jr. 2013. Survey Research Methods. Thousand Oaks, CA: Sage Publications.
Fowler, James H., Heaney, Michael T., Nickerson, David W., Padgett, John F., and Sinclair, Betsy. 2011. ‘Causality in Political Networks’. American Politics Research 39(2):437480.
Groves, Robert M., and Couper, Mick P.. 2012. Nonresponse in Household Interview Surveys. Hoboken, NJ: John Wiley & Sons.
Groves, Robert M., Dillman, Don A., Eltinge, John L., and Little, Roderick J.A.. 2002. Survey Nonresponse. New York, NY: John Wiley & Sons.
Groves, Robert M., Fowler, Floyd J. Jr, Couper, Mick P., Lepkowski, James M., Singer, Eleanor, and Tourangeau, Roger. 2011. Survey Methodology, vol. 561. Hoboken, NJ: John Wiley & Sons.
Groves, Robert M., and Peytcheva, Emilia. 2008. ‘The Impact of Nonresponse Rates on Self-Selection Bias a Meta-Analysis’. Public Opinion Quarterly 72(2):167189.
Groves, Robert M., Singer, Eleanor, and Corning, Amy. 2000. ‘Leverage-Saliency Theory of Survey Participation: Description and an Illustration’. Public Opinion Quarterly 64(3):299308.
Hebert, James R., Ma, Yunsheng, Clemow, Lynn, Ockene, Ira S., Saperia, Gordon, Stanek, Edward J. III, Merriam, Philip A., and Ockene, Judith K.. 1997. ‘Gender Differences in Social Desirability and Social Approval Bias in Dietary Self-Report’. American Journal of Epidemiology 146(12):10461055.
Heckman, James J. 1979. ‘Sample Selection Bias as a Specification Error’. Econometrica 47(1):153161.
Jackman, Simon. 1999. ‘Correcting Surveys for Non-Response and Measurement Error Using Auxiliary Information’. Electoral Studies 18(1):727.
Jacob, Steve, Speer, Sandra, and Furubo, Jan-Eric. 2015. ‘The Institutionalization of Evaluation Matters: Updating the International Atlas of Evaluation 10 Years Later’. Evaluation 21(1):631.
Johnson, Kelli, Greenseid, Lija O., Toal, Stacie A., King, Jean A., Lawrenz, Frances, and Volkov, Boris. 2009. ‘Research on Evaluation Use: A Review of the Empirical Literature From 1986 to 2005’. American Journal of Evaluation 30(3):377410.
Kam, Christopher, Bianco, William T., Sened, Itai, and Smyth, Regina. 2010. ‘Ministerial Selection and Intraparty Organization in the Contemporary British Parliament’. American Political Science Review 104(2):289306.
Kriesi, Hanspeter. 2001. ‘The Federal Parliament: The Limits of Institutional Reform’. West European Politics 24(2):5976.
Krumpal, Ivar. 2013. ‘Determinants of Social Desirability Bias in Sensitive Surveys: A Literature Review’. Quality & Quantity 47(4):20252047.
Little, Roderick J.A. 1988. ‘Missing-Data Adjustments in Large Surveys’. Journal of Business & Economic Statistics 6(3):287296.
Little, Roderick J.A., and Rubin, Donald B.. 2014. Statistical Analysis with Missing Data. Hoboken, NJ: John Wiley & Sons.
Martin, Elizabeth. 1999. ‘Who Knows Who Lives Here? Within-Household Disagreements as a Source of Survey Coverage Error’. Public Opinion Quarterly 63(2):220236.
Martin, Shane. 2010. ‘Electoral Rewards for Personal Vote Cultivation Under PR-STV’. West European Politics 33(2):369380.
Massey, Douglas S., and Tourangeau, Roger. 2013. ‘Where Do We Go from Here? Nonresponse and Social Measurement’. The ANNALS of the American Academy of Political and Social Science 645(1):222236.
Murphy, Pamela R. 2012. ‘Attitude, Machiavellianism and the Rationalization of Misreporting’. Accounting, Organizations and Society 37(4):242259.
Russell, Meg, Gover, Daniel, and Wollter, Kristina. 2015. ‘Does the Executive Dominate the Westminster Legislative Process?: Six Reasons for Doubt’. Parliamentary Affairs 69(2):286308.
Sartori, Anne E. 2003. ‘An Estimator for Some Binary‐Outcome Selection Models Without Exclusion Restrictions’. Political Analysis 11(2):111138.
Sciarini, Pascal, and Goldberg, Andreas C.. 2015. ‘Lost on the Way: Nonresponse and its Influence on Turnout Bias in Postelection Surveys’. Journal of Survey Statistics and Methodology. doi:10.1093/ijpor/edv049.
Sciarini, Pascal, and Goldberg, Andreas C.. 2016. ‘Turnout Bias in Postelection Surveys: Political Involvement, Survey Participation and Vote Overreporting’. Journal of Survey Statistics and Methodology. doi:10.1093/jssam/smv039.
Scully, Roger, Hix, Simon, and Farrell, David M.. 2012. ‘National or European Parliamentarians? Evidence from a New Survey of the Members of the European Parliament’. JCMS: Journal of Common Market Studies 50(4):670683.
Selb, Peter, and Munzert, Simon. 2013. ‘Voter Overrepresentation, Vote Misreporting, and Turnout Bias in Postelection Surveys’. Electoral Studies 32(1):186196.
Squire, Peverill. 1992. ‘Legislative Professionalization and Membership Diversity in State Legislatures’. Legislative Studies Quarterly 17(1):6979.
Squire, Peverill. 2007. ‘Measuring State Legislative Professionalism: The Squire Index Revisited’. State Politics & Policy Quarterly 7(2):211227.
Strebel, Michael. 2014. Exekutivföderalismus in der Schweiz? Einbezug der Parlamente bei interkantonalen Vereinbarungen. Baden-Baden: Nomos.
Tourangeau, Roger, Shapiro, Gary, Kearney, Anne, and Ernst, Lawrence. 1997. ‘Who Lives Here? Survey Undercoverage and Household Roster Questions’. Journal of Official Statistics 13:118.
Tourangeau, Roger, and Yan, Ting. 2007. ‘Sensitive Questions in Surveys’. Psychological Bulletin 133(5):859883.
Traugott, Michael W., and Katosh, John P.. 1979. ‘Response Validity in Survey of Voting Behavior’. Public Opinion Quarterly 14(3):359377.
Winzen, Thomas. 2013. ‘European Integration and National Parliamentary Oversight Institutions’. European Union Politics 14(2):297323.
Zucco, Cesar Jr., and Lauderdale, Benjamin E.. 2011. ‘Distinguishing Between Influences on Brazilian Legislative Behavior’. Legislative Studies Quarterly 36(3):363396.
Recommend this journal

Email your librarian or administrator to recommend adding this journal to your organisation's collection.

Political Science Research and Methods
  • ISSN: 2049-8470
  • EISSN: 2049-8489
  • URL: /core/journals/political-science-research-and-methods
Please enter your name
Please enter a valid email address
Who would you like to send this to? *
×
Type Description Title
UNKNOWN
Supplementary materials

Bundi et al. Dataset
Dataset

 Unknown

Metrics

Altmetric attention score

Full text views

Total number of HTML views: 0
Total number of PDF views: 0 *
Loading metrics...

Abstract views

Total abstract views: 0 *
Loading metrics...

* Views captured on Cambridge Core between <date>. This data will be updated every 24 hours.

Usage data cannot currently be displayed