Skip to main content Accessibility help
×
×
Home

Testing the Validity of Automatic Speech Recognition for Political Text Analysis

  • Sven-Oliver Proksch (a1), Christopher Wratil (a1) (a2) and Jens Wäckerle (a1)
Abstract

The analysis of political texts from parliamentary speeches, party manifestos, social media, or press releases forms the basis of major and growing fields in political science, not least since advances in “text-as-data” methods have rendered the analysis of large text corpora straightforward. However, a lot of sources of political speech are not regularly transcribed, and their on-demand transcription by humans is prohibitively expensive for research purposes. This class includes political speech in certain legislatures, during political party conferences as well as television interviews and talk shows. We showcase how scholars can use automatic speech recognition systems to analyze such speech with quantitative text analysis models of the “bag-of-words” variety. To probe results for robustness to transcription error, we present an original “word error rate simulation” (WERSIM) procedure implemented in $R$ . We demonstrate the potential of automatic speech recognition to address open questions in political science with two substantive applications and discuss its limitations and practical challenges.

Copyright
Corresponding author
Footnotes
Hide All

Authors’ note: We are grateful to Leonie Diffené, Felix Reich and Pit Rieger for their excellent research assistance. We are also very thankful for helpful comments on earlier versions of this work by two anonymous reviewers as well as Jeff Gill. Christopher Wratil would like to acknowledge funding by the Fritz Thyssen Stiftung (20.16.0.045WW). All remaining errors are our own. The replication files for this article are available on the Political Analysis Dataverse (Proksch, Wratil, and Wäckerle 2018).

Contributing Editor: Jeff Gill

Footnotes
References
Hide All
Abou-Chadi, T. 2016. “Niche Party Success and Mainstream Party Policy Shifts—How Green and Radical Right Parties Differ in their Impact.” British Journal of Political Science 46(2):417436.
Accornero, G., and Ramos Pinto, P.. 2015. “Mild Mannered? Protest and Mobilisation in Portugal Under Austerity, 2010–2013.” West European Politics 38(3):491515.
Baum, M. A. 2005. “Talking the Vote: Why Presidential Candidates Hit the Talk Show Circuit.” American Journal of Political Science 49(2):213234.
Baum, M. A., and Jamison, A. S.. 2006. “The Oprah Effect: How Soft News Helps Inattentive Citizens Vote Consistently.” Journal of Politics 68(4):946959.
Benoit, K., Laver, M., and Mikhaylov, S.. 2009. “Treating Words as Data with Error: Uncertainty in Text Statements of Policy Positions.” American Journal of Political Science 53(2):495513.
Cook, J. R., and Stefanski, L. A.. 1994. “Simulation-Extrapolation Estimation in Parametric Measurement Error Models.” Journal of the American Statistical Association 89(428):13141328.
Daku, M., Soroka, S., and Young, L.. 2015. “Lexicoder, version 3.0.” www.lexicoder.com.
de Vries, E., Schoonvelde, M., and Schumacher, G.. 2018. “No Longer Lost in Translation. Evidence that Google Translate Works for Comparative Bag-of-Words Text Applications.” Political Analysis 26(4):417430.
Denny, M. J., and Spirling, A.. 2018. “Text Preprocessing for Unsupervised Learning: Why It Matters, When It Misleads, and What to Do About It.” Political Analysis 26(2):168189.
Dietrich, B. J., Hayes, M., and O’Brien, D. Z.. 2017. “Pitch Perfect: Vocal Pitch and the Emotional Intensity of Congressional Speech on Women.” Working Paper.
Dietrich, B. J., Enos, R. D., and Sen, M.. 2018. “Emotional Arousal Predicts Voting on the U.S. Supreme Court.” Political Analysis , https://doi.org/10.1017/pan.2018.47.
Dolezal, M., Hutter, S., and Becker, R.. 2016. “Protesting European Integration: Politicisation from Below? In Politicising Europe: Integration and Mass Politics , edited by Hutter, Swen, Grande, Edgar, and Kriesi, Hanspeter, 112136. Cambridge: Cambridge University Press, chapter 5.
Dresing, T., Pehl, T., and Schmieder, C.. 2015. “Manual (on) Transcription. Transcription Conventions, Software Guides and Practical Hints for Qualitative Researchers.” 3rd English Edition: Marburg. http://www.audiotranskription.de/english/transcription-practicalguide.htm.
Dür, A., and Mateo, G.. 2010. “Bargaining Power and Negotiation Tactics: The Negotiations on the EU’s Financial Perspective, 2007–2013.” Journal of Common Market Studies 48(3):557578.
Edmondson, M.2017. “Package ‘googleLanguageR’.” Technical report.
European Council 2006. “An Overall Policy on Transparency.” http://www.consilium.europa.eu/ueDocs/cms_Data/docs/pressData/en/misc/90112.pdf.
Fernandez-Vazquez, P. 2014. “And Yet it Moves: The Effect of Election Platforms on Party Policy Images.” Comparative Political Studies 47(14):19191944.
Gibson, R. K., and McAllister, I.. 2011. “Do Online Election Campaigns Win Votes? The 2007 Australian YouTube Election.” Political Communication 28(2):227244.
Giugni, M., and Grasso, M.. 2015. Austerity and Protest: Popular Contention in Times of Economic Crisis . Routledge.
Grimmer, J. 2010. “A Bayesian Hierarchical Topic Model for Political Texts: Measuring Expressed Agendas in Senate Press Releases.” Political Analysis 18(1):135.
Grimmer, J., and Stewart, B. M.. 2013. “Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts.” Political Analysis 21(3):267297.
Haynes, A. A., and Pitts, B.. 2009. “Making an Impression: New Media in the 2008 Presidential Nomination Campaigns.” PS—Political Science and Politics 42(1):5358.
Herzog, A., and Benoit, K.. 2015. “The Most Unkindest Cuts: Speaker Selection and Expressed Government Dissent During Economic Crisis.” The Journal of Politics 77(4):11571175.
Hinton, G., Deng, L., Yu, D., Dahl, G. E., Mohamed, A.-R., Jaitly, N., Senior, A., Vanhoucke, V., Nguyen, P., Sainath, T. N., and Kingsbury, B.. 2012. “Deep Neural Networks for Acoustic Modeling in Speech Recognition.” IEEE Signal Processing Magazine (November):8297.
Hochreiter, S., and Schmidhuber, J.. 1997. “Long Short-Term Memory.” Neural Computation 9(8):17351780.
Këpuska, V., and Bohouta, G.. 2017. “Comparing Speech Recognition Systems (Microsoft API, Google API and CMU Sphinx).” International Journal of Engineering Research and Applications 7(3):2024.
Knox, D., and Lucas, C.. 2018. “A Dynamic Model of Speech for The Social Sciences.” Working Paper.
Kriesi, H., Grande, E., Dolezal, M., Helbling, M., Hoglinger, D., Hutter, S., and Wueest, B.. 2012. Political Conflict in Western Europe . Cambridge: Cambridge University Press.
Lauderdale, B. E., and Herzog, A.. 2016. “Measuring Political Positions from Legislative Speech.” Political Analysis 24(3):374394.
Lecun, Y., Bengio, Y., and Hinton, G.. 2015. “Deep Learning.” Nature 521(7553):436444.
Levenshtein, V. I. 1966. “Binary Codes Capable of Correcting Deletions, Insertions, and Reversals.” Soviet Physics Doklady 10(8):707710.
Liao, H., McDermott, E., and Senior, A.. 2013. “Large Scale Deep Neural Network Acoustic Modeling with Semi-Supervised Training Data for YouTube Video Transcription.” In 2013 IEEE Workshop on Automatic Speech Recognition and Understanding, ASRU 2013—Proceedings , 368373.
Martin, A. D., and Quinn, K. M.. 2002. “Dynamic Ideal Point Estimation via Markov Chain Monte Carlo for The U.S. Supreme Court, 1953–1999.” Political Analysis 10(2):134153.
McKibben, H. E. 2016. “To Link or Not to Link? Agenda Change in International Bargaining.” British Journal of Political Science 46(2):371393.
Meeker, M.2017. “Kleiner Perkins Internet Trends 2017.” Technical report. http://www.kpcb.com/internet-trends.
Meguid, B. M. 2005. “Competition Between Unequals: The Role of Mainstream Party Strategy in Niche Party Success.” American Political Science Review 99(3):347359.
Meguid, B. M. 2008. Party Competition between Unequals: Strategies and Electoral Fortunes in Western Europe . Cambridge: Cambridge University Press.
Plummer, M. 2003. “JAGS: A Program for Analysis of Bayesian Graphical Models Using Gibbs Sampling.” In Proceedings of the 3rd International Workshop on Distributed Statistical Computing (DSC 2003) , 2022.
Polk, J., Rovny, J., Bakker, R., Edwards, E., Hooghe, L., Jolly, S., Koedam, J., Kostelka, F., Marks, G., Schumacher, G., Steenbergen, M., Vachudova, M., and Zilovic, M.. 2017. “Explaining the Salience of Anti-Elitism and Reducing Political Corruption for Political Parties in Europe with the 2014 Chapel Hill Expert Survey Data.” Research & Politics 19.
Proksch, S.-O., Lowe, W., Wäckerle, J., and Soroka, S.. 2018. “Multilingual Sentiment Analysis: A New Approach to Measuring Conflict in Parliamentary Speeches.” Legislative Studies Quarterly , https://doi.org/10.1111/lsq.12218.
Proksch, S.-O., Wratil, C., and Wäckerle, J.. 2018. “Replication Data for: ‘Testing the Validity of Automatic Speech Recognition for Political Text Analysis’ by Sven-Oliver Proksch, Christopher Wratil and Jens Wäckerle.” https://doi.org/10.7910/DVN/PGQY2F, Harvard Dataverse, V1.
Proksch, S. O., and Slapin, J. B.. 2012. “Institutional Foundations of Legislative Speech.” American Journal of Political Science 56(3):520537.
Proksch, S. O., and Slapin, J. B.. 2015. The Politics of Parliamentary Debate: Parties, Rebels, and Representation . Cambridge: Cambridge University Press.
Proksch, S.-O., Lowe, W., Wäckerle, J., and Soroka, S.. 2018. “Multilingual Sentiment Analysis: A New Approach to Measuring Conflict in Parliamentary Speeches.” Legislative Studies Quarterly, https://doi.org/10.1111/lsq.12218.
Roberts, M. E., Stewart, B. M., Tingley, D., Lucas, C., Leder-Luis, J., Gadarian, S. K., Albertson, B., and Rand, D. G.. 2014. “Structural Topic Models for Open-Ended Survey Responses.” American Journal of Political Science 58(4):10641082.
Schnakenberg, K. E., and Fariss, C. J.. 2014. “Dynamic Patterns of Human Rights Practices.” Political Science Research and Methods 2(1):131.
Schneider, C. J. 2011. “Weak States and Institutionalized Bargaining Power in International Organizations.” International Studies Quarterly 55(2):331355.
Schneider, C. J. 2013. “Globalizing Electoral Politics: Political Competence and Distributional Bargaining in the European Union.” World Politics 65(3):452490.
Slapin, J. B., and Proksch, S.-O.. 2008. “A Scaling Model for Estimating Time-Series Party Positions from Texts.” American Journal of Political Science 52(3):705722.
Soltau, H., Liao, H., and Sak, H.. 2017. “Neural Speech Recognizer: Acoustic-to-Word LSTM Model for Large Vocabulary Speech Recognition.” In Proceedings of the Annual Conference of the International Speech Communication Association, INTERSPEECH, August , 37073711.
Sood, G.2018. “Package ‘tubr’.” Technical report.
Stenbæk, J., and Jensen, M. D.. 2016. “Evading the Joint Decision Trap: The Multiannual Financial Framework 2014–20.” European Political Science Review 8(4):615635.
Thomson, R., Royed, T., Naurin, E., Artés, J., Costello, R., Ennser-Jedenastik, L., Ferguson, M., Kostadinova, P., Moury, C., Pétry, F., and Praprotnik, K.. 2017. “The Fulfillment of Parties’ Election Pledges: A Comparative Study on the Impact of Power Sharing.” American Journal of Political Science 61(3):527542.
Wratil, C., and Hobolt, S. B.. Forthcoming. “Public Deliberations in The Council of the European Union: Introducing and Validating the DICEU Approach.” European Union Politics 127.
Young, L., and Soroka, S.. 2012. “Affective News: The Automated Coding of Sentiment in Political Texts.” Political Communication 29(2):205231.
Yu, D., and Deng, L.. 2015. Automatic Speech Recognition . Signals and Communication Technology London. London: Springer.
Zittel, T. 2015. “Do Candidates Seek Personal Votes on the Internet? Constituency Candidates in the 2009 German Federal Elections.” German Politics 24(4):435450.
Recommend this journal

Email your librarian or administrator to recommend adding this journal to your organisation's collection.

Political Analysis
  • ISSN: 1047-1987
  • EISSN: 1476-4989
  • URL: /core/journals/political-analysis
Please enter your name
Please enter a valid email address
Who would you like to send this to? *
×
MathJax

Keywords

Type Description Title
UNKNOWN
Supplementary materials

Proksch supplementary material
Proksch supplementary material 1

 Unknown (253 KB)
253 KB

Metrics

Altmetric attention score

Full text views

Total number of HTML views: 0
Total number of PDF views: 0 *
Loading metrics...

Abstract views

Total abstract views: 0 *
Loading metrics...

* Views captured on Cambridge Core between <date>. This data will be updated every 24 hours.

Usage data cannot currently be displayed