Skip to main content

A New Multinomial Accuracy Measure for Polling Bias

  • Kai Arzheimer (a1) and Jocelyn Evans (a2)

In this article, we propose a polling accuracy measure for multi-party elections based on a generalization of Martin, Traugott, and Kennedy's two-party predictive accuracy index. Treating polls as random samples of a voting population, we first estimate an intercept only multinomial logit model to provide proportionate odds measures of each party's share of the vote, and thereby both unweighted and weighted averages of these values as a summary index for poll accuracy. We then propose measures for significance testing, and run a series of simulations to assess possible bias from the resulting folded normal distribution across different sample sizes, finding that bias is small even for polls with small samples. We apply our measure to the 2012 French presidential election polls to demonstrate its applicability in tracking overall polling performance across time and polling organizations. Finally, we demonstrate the practical value of our measure by using it as a dependent variable in an explanatory model of polling accuracy, testing the different possible sources of bias in the French data.

    • Send article to Kindle

      To send this article to your Kindle, first ensure is added to your Approved Personal Document E-mail List under your Personal Document Settings on the Manage Your Content and Devices page of your Amazon account. Then enter the ‘name’ part of your Kindle email address below. Find out more about sending to your Kindle. Find out more about sending to your Kindle.

      Note you can select to send to either the or variations. ‘’ emails are free but can only be sent to your device when it is connected to wi-fi. ‘’ emails can be delivered even when you are not connected to wi-fi, but note that service fees apply.

      Find out more about the Kindle Personal Document Service.

      A New Multinomial Accuracy Measure for Polling Bias
      Available formats
      Send article to Dropbox

      To send this article to your Dropbox account, please select one or more formats and confirm that you agree to abide by our usage policies. If this is the first time you use this feature, you will be asked to authorise Cambridge Core to connect with your <service> account. Find out more about sending content to Dropbox.

      A New Multinomial Accuracy Measure for Polling Bias
      Available formats
      Send article to Google Drive

      To send this article to your Google Drive account, please select one or more formats and confirm that you agree to abide by our usage policies. If this is the first time you use this feature, you will be asked to authorise Cambridge Core to connect with your <service> account. Find out more about sending content to Google Drive.

      A New Multinomial Accuracy Measure for Polling Bias
      Available formats
Corresponding author
e-mail: (corresponding author)
Hide All

Authors' note: We are grateful to two anonymous reviewers and to the editors of Political Analysis for their extremely helpful comments and suggestions. We would also like to thank Gilles Ivaldi (CNRS/University of Nice) for providing data on the French election surveys. Replication materials can be found in Arzheimer and Evans (2013). surveybias.ado, a Stata add-on that facilitates the computation of B and Bw , is available from the Boston College Statistical Software Components (SSC) archive. Supplementary materials for this article are available on the Political Analysis Web site.

Hide All
Agresti, Alan. 2002. Categorical data analysis. 2nd ed. Hoboken, NJ: John Wiley.
Aitchison, John. 1982. The statistical analysis of compositional data (with discussion). Journal of the Royal Statistical Society, Series B 44(2): 139–77.
Alvarez, R. Michael, and Nagler, Jonathan. 1998. When politics and models collide: Estimating models of multiparty elections. American Journal of Political Science 42: 5596.
Arzheimer, Kai, and Evans, Jocelyn. 2013. Replication data for: A new multinomial accuracy measure for polling bias. IQSS Dataverse Network/Distributor V1 [Version]. (accessed August 6, 2013).
Bacon-Shone, John. 2011. A short history of compositional data analysis. In Compositional data analysis: Theory and applications, eds. Pawlowsky-Glahn, Vera and Buccianti, Antonella, 111. Chichester, UK: Wiley.
Callegaro, Mario, and Gasperoni, Giancarlo. 2008. Accuracy of pre-election polls for the 2006 Italian parliamentary election: Too close to call. International Journal of Public Opinion Research 20(2): 148–70.
Cheng, Simon, and Scott Long, J. 2007. Testing for IIA in the multinomial logit model. Sociological Methods & Research 35(4): 583600.
Dow, Jay K., and Endersby, James W. 2004. Multinomial probit and multinomial logit: A comparison of choice models for voting research. Electoral Studies 23: 107–22.
Durand, Claire. 2008. The polls of the 2007 French presidential campaign: Were lessons learned from the 2002 catastrophe? International Journal of Public Opinion Research 20(3): 275–98.
Evans, Jocelyn. 2012. The sound foundations of a socialist victory. Renewal 20 (2–3): 123–28.
Fisher, Stephen D., Ford, Robert, Jennings, Will, Pickup, Mark, and Wlezien, Christopher. 2011. From polls to votes to seats: Forecasting the 2010 British general election. Electoral Studies 30(2): 250–57.
Jackman, Simon. 2005. Pooling the polls over an election campaign. Australian Journal of Political Science 40(4): 499517.
Kropko, Jonathan. 2010. A comparison of three discrete choice estimators (unpublished dissertation chapter). University of North Carolina-Chapel Hill.∼kropko/paper1.pdf (accessed August 6, 2013).
Laakso, Markku, and Taagepera, Rein. 1979. “Effective” number of parties: A measure with application to West Europe. Comparative Political Studies 12(1): 327.
Long, J. Scott. 1997. Regression models for categorical and limited dependent variables. Thousand Oaks, CA: Sage.
Luce, R. Duncan. 1959. Individual choice behavior. A theoretical analysis. New York: John Wiley & Sons.
Martin, Elizabeth A., Traugott, Michael W., and Kennedy, Courtney. 2005. A review and proposal for a new measure of poll accuracy. Public Opinion Quarterly 69(3): 342–69.
McFadden, Daniel. 1973. Conditional logit analysis of qualitative choice behaviour. In Frontiers of Econometrics, ed. Zarembka, Paul, 105–42. New York: Academic Press.
McLean, Iain. 1995. Independence of irrelevant alternatives before Arrow. Mathematical Social Sciences 30(2): 107–26.
Mosteller, Frederick, Hyman, Herbert, McCarthy, Philip, Marks, Eli, and Truman, David. 1949. The pre-election polls of 1948: Report to the committee on analysis of pre-election polls and forecasts. New York: Social Science Research Council.
Pawlowsky-Glahn, Vera, and Buccianti, Antonella, eds. 2011. Compositional data analysis: Theory and applications. Chichester, UK: Wiley.
Pearson, Karl. 1897. Mathematical contributions to the theory of evolution: On a form of spurious correlation which may arise when indices are used in the measurements of organs. Proceedings of the Royal Society of London 60: 489502.
Ray, Paramesh. 1973. Independence of irrelevant alternatives. Econometrica 41(5): 987–91.
Royston, Patrick, and Sauerbrei, Willi. 2008. Multivariable model-building: A pragmatic approach to regression analysis based on fractional polynomials for modelling continuous variables. Chichester, UK: Wiley.
Schaffer, Lena-Maria, and Schneider, Gerald. 2005. Die Prognosegte von Wahlbörsen und Meinungsumfragen zur Bundestagswahl 2005. Politische Vierteljahresschrift 46(4): 674–81.
Train, Kenneth. 2009. Discrete choice methods with simulation. 2nd ed. Cambridge, UK: Cambridge University Press.
Whitten, Guy D., and Palmer, Harvey D. 1996. Heightening comparativists' concern for model choice: Voting behavior in Great Britain and the Netherlands. American Journal of Political Science 40(1): 231–60.
Recommend this journal

Email your librarian or administrator to recommend adding this journal to your organisation's collection.

Political Analysis
  • ISSN: 1047-1987
  • EISSN: 1476-4989
  • URL: /core/journals/political-analysis
Please enter your name
Please enter a valid email address
Who would you like to send this to? *
Type Description Title
Supplementary materials

Arzheimer and Evans supplementary material
Supplementary Material

 Unknown (23 KB)
23 KB


Altmetric attention score

Full text views

Total number of HTML views: 0
Total number of PDF views: 0 *
Loading metrics...

Abstract views

Total abstract views: 0 *
Loading metrics...

* Views captured on Cambridge Core between <date>. This data will be updated every 24 hours.

Usage data cannot currently be displayed