Skip to main content
×
×
Home

Classification Accuracy as a Substantive Quantity of Interest: Measuring Polarization in Westminster Systems

  • Andrew Peterson (a1) and Arthur Spirling (a2)
Abstract

Measuring the polarization of legislators and parties is a key step in understanding how politics develops over time. But in parliamentary systems—where ideological positions estimated from roll calls may not be informative—producing valid estimates is extremely challenging. We suggest a new measurement strategy that makes innovative use of the “accuracy” of machine classifiers, i.e., the number of correct predictions made as a proportion of all predictions. In our case, the “labels” are the party identifications of the members of parliament, predicted from their speeches along with some information on debate subjects. Intuitively, when the learner is able to discriminate members in the two main Westminster parties well, we claim we are in a period of “high” polarization. By contrast, when the classifier has low accuracy—and makes a relatively large number of mistakes in terms of allocating members to parties based on the data—we argue parliament is in an era of “low” polarization. This approach is fast and substantively valid, and we demonstrate its merits with simulations, and by comparing the estimates from 78 years of House of Commons speeches with qualitative and quantitative historical accounts of the same. As a headline finding, we note that contemporary British politics is approximately as polarized as it was in the mid-1960s—that is, in the middle of the “postwar consensus”. More broadly, we show that the technical performance of supervised learning algorithms can be directly informative about substantive matters in social science.

  • View HTML
    • Send article to Kindle

      To send this article to your Kindle, first ensure no-reply@cambridge.org is added to your Approved Personal Document E-mail List under your Personal Document Settings on the Manage Your Content and Devices page of your Amazon account. Then enter the ‘name’ part of your Kindle email address below. Find out more about sending to your Kindle. Find out more about sending to your Kindle.

      Note you can select to send to either the @free.kindle.com or @kindle.com variations. ‘@free.kindle.com’ emails are free but can only be sent to your device when it is connected to wi-fi. ‘@kindle.com’ emails can be delivered even when you are not connected to wi-fi, but note that service fees apply.

      Find out more about the Kindle Personal Document Service.

      Classification Accuracy as a Substantive Quantity of Interest: Measuring Polarization in Westminster Systems
      Available formats
      ×
      Send article to Dropbox

      To send this article to your Dropbox account, please select one or more formats and confirm that you agree to abide by our usage policies. If this is the first time you use this feature, you will be asked to authorise Cambridge Core to connect with your <service> account. Find out more about sending content to Dropbox.

      Classification Accuracy as a Substantive Quantity of Interest: Measuring Polarization in Westminster Systems
      Available formats
      ×
      Send article to Google Drive

      To send this article to your Google Drive account, please select one or more formats and confirm that you agree to abide by our usage policies. If this is the first time you use this feature, you will be asked to authorise Cambridge Core to connect with your <service> account. Find out more about sending content to Google Drive.

      Classification Accuracy as a Substantive Quantity of Interest: Measuring Polarization in Westminster Systems
      Available formats
      ×
Copyright
Corresponding author
* Email: andrew.peterson@unige.ch
Footnotes
Hide All

Authors’ note: We are grateful to Niels Goet, Justin Grimmer and Ben Lauderdale for comments on an earlier draft. Audiences at the European Political Science Association meeting and the American Political Science Association meeting provided helpful feedback. Comments from two anonymous referees and the editor at Political Analysis improved our manuscript considerably. Our replication materials are as described in Peterson (2017).

Contributing Editor: Justin Grimmer

Footnotes
References
Hide All
Addison, Paul. 1994. The road to 1945: British politics and the second world war . London: Pimlico.
Bai, Jushan, and Perron, Pierre. 2003. Computation and analysis of multiple structural change models. Journal of Applied Econometrics 18:122.
Barber, Michael, and McCarty, Nolan. 2015. Causes and consequences of polarization. In Solutions to polarization in America , ed. Nathaniel, Persily. Cambridge: Cambridge University Press, pp. 1559.
Bottou, Léon. 2004. Stochastic learning. In Advanced lectures on machine learning: ML summer schools 2003, Canberra, Australia, February 2–14, 2003, Tübingen, Germany, August 4–16, 2003, revised lectures , ed. Bousquet, O., von Luxburg, U., and Rätsch, G.. Berlin, Heidelberg: Springer, pp. 146168.
Chen, Tianqi, and Guestrin, Carlos. 2016. XGBoost: A scalable tree boosting system. In KDD ’16 proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining . New York: ACM, pp. 785794.
Crammer, Koby, Dekel, Ofer, Keshet, Joseph, Shalev-Shwartz, Shai, and Singer, Yoram. 2006. Online passive-aggressive algorithms. Journal of Machine Learning Research 7(1):551585.
Diermeier, Daniel, Godbout, Jean-Franois, Yu, Bei, and Kaufmann, Stefan. 2012. Language and ideology in congress. British Journal of Political Science 42:3155.
D’Orazio, Vito, Landis, Steven, Palmer, Glenn, and Schrodt, Philip. 2014. Separating the wheat from the chaff: Applications of automated document classification using support vector machines. Political Analysis 22(2):224242.
Fraser, Duncan. 2000. The postwar consensus: A debate not long enough. Parliamentary Affairs 53(2):347362.
Freund, Yoav, and Schapire, Robert E.. 1999. Large margin classification using the perceptron algorithm. Machine Learning 37(3):277296.
Gentzkow, Matthew, Shapiro, Jesse M., and Taddy, Matt. 2016. Measuring polarization in high-dimensional data: Method and application to congressional speech. NBER Working Paper. http://www.nber.org/papers/w22423.
Hastie, Trevor, Tibshirani, Robert, and Friedman, Jerome. 2009. The elements of statistical learning: Data mining, inference, and prediction . New York: Springer.
Hopkins, Daniel, and King, Gary. 2010. A method of automated nonparametric content analysis for social science. American Journal of Political Science 54(1):229247.
Kam, Christopher J. 2009. Party discipline and parliamentary politics . Cambridge: Cambridge University Press.
Kavanagh, Dennis, and Morris, Peter. 1994. Consensus politics from Attlee to Major . Hoboken: Wiley Blackwell.
Kellermann, Michael. 2012. Estimating ideal points in the British House of commons using early day motions. American Journal of Political Science 56(3):757771.
Lauderdale, Benjamin, and Herzog, Alexander. 2016. Measuring political positions from legislative speech. Political Analysis 24(2):121.
Lehmann, Pola, Matthieß, Theres, Merz, Nicolas, Regel, Sven, and Werner, Annika. 2016. Manifesto corpus. Version: 2016-6.
Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., and Blondel, M. et al. . 2011. Scikit-learn: Machine learning in python. Journal of Machine Learning Research 12:28252830.
Peterson, Andrew Jerel. 2017. Replication data for: Classification accuracy as a substantive quantity of interest: Measuring polarization in westminster systems. Harvard Dataverse, UNF:6:iRJ1F7aydu3LemeJ2gjS1A==. doi:10.7910/DVN/YTPJ1N.
Rheault, L., Beelen, K., Cochrane, C., and Hirst, G.. 2016. Measuring emotion in parliamentary debates with automated textual analysis. PLOS ONE 11(12). URL: https://doi.org/10.1371/journal.pone.0168843.
Rhodes, Rod, and Patrick, Weller. 2005. Westminster transplanted and westminster implanted: Exploring political change. In Westminster legacies: Democracy and responsible government in Asia and the Pacific , ed. Haig, Patapan, Wanna, John, and Weller, Patrick. University of New South Wales: University of New South Wales Press.
Schmidt, Mark, Roux, Nicolas Le, and Bach, Francis. 2013. Minimizing finite sums with the stochastic average gradient. Preprint arXiv:1309.2388.
Seldon, Anthony. 1994. The consensus debate. Parliamentary Affairs 47(4):501514.
Slapin, Jonathan B., and Proksch, Sven-Oliver. 2008. A scaling model for estimating time-series party positions from texts. American Journal of Political Science 52(3):705722.
Spirling, Arthur, and McLean, Iain. 2007. UK OC OK? Political Analysis 15(1):8596.
Volkens, Andrea, Lehmann, Pola, Theres, Matthieß, Merz, Nicolas, and Regel, Sven. 2016. The Manifesto Data Collection. Manifesto Project (MRG/CMP/MARPOR). Version 2016b. Berlin: Wissenschaftszentrum Berlin für Sozialforschung (WZB). URL: https://doi.org/10.25522/manifesto.mpds.2017a.
Zeileis, Achim, Leisch, Friedrich, Hornik, Kurt, and Kleiber, Christian. 2002. Strucchange: An R package for testing for structural change in linear regression models. Journal of Statistical Software 7(2):138.
Recommend this journal

Email your librarian or administrator to recommend adding this journal to your organisation's collection.

Political Analysis
  • ISSN: 1047-1987
  • EISSN: 1476-4989
  • URL: /core/journals/political-analysis
Please enter your name
Please enter a valid email address
Who would you like to send this to? *
×
MathJax

Keywords

Type Description Title
UNKNOWN
Supplementary materials

Peterson and Spirling supplementary material 1
Peterson and Spirling supplementary material

 Unknown (200 KB)
200 KB

Metrics

Altmetric attention score

Full text views

Total number of HTML views: 0
Total number of PDF views: 0 *
Loading metrics...

Abstract views

Total abstract views: 0 *
Loading metrics...

* Views captured on Cambridge Core between <date>. This data will be updated every 24 hours.

Usage data cannot currently be displayed