Hostname: page-component-669899f699-7xsfk Total loading time: 0 Render date: 2025-05-04T07:28:16.581Z Has data issue: false hasContentIssue false
  • English
  • Français

Multi-Label Prediction for Political Text-as-Data

Published online by Cambridge University Press:  14 June 2021

Aaron Erlich ,
Stefano G. Dantas ,
Benjamin E. Bagozzi Open the ORCID record for Benjamin E. Bagozzi [Opens in a new window] ,
Daniel Berliner  and
Brian Palmer-Rubin
Aaron Erlich
Affiliation:
Department of Political Science, McGill University, Montreal, QC, Canada Centre for the Study of Democratic Citizenship, QC, Canada
Stefano G. Dantas
Affiliation:
Department of Electrical and Computer Engineering, McGill University, Montreal, QC, Canada
Benjamin E. Bagozzi*
Affiliation:
Department of Political Science and International Relations, University of Delaware, Newark, DE, USA. Email: bagozzib@udel.edu
Daniel Berliner
Affiliation:
Department of Government, London School of Economics and Political Science, London, UK
Brian Palmer-Rubin
Affiliation:
Department of Political Science, Marquette University, Milwaukee, WI, USA
*
Corresponding author Benjamin E. Bagozzi
Get access Rights & Permissions [Opens in a new window]

Abstract

Political scientists increasingly use supervised machine learning to code multiple relevant labels from a single set of texts. The current “best practice” of individually applying supervised machine learning to each label ignores information on inter-label association(s), and is likely to under-perform as a result. We introduce multi-label prediction as a solution to this problem. After reviewing the multi-label prediction framework, we apply it to code multiple features of (i) access to information requests made to the Mexican government and (ii) country-year human rights reports. We find that multi-label prediction outperforms standard supervised learning approaches, even in instances where the correlations among one’s multiple labels are low.

Keywords

text-as-datamulti-labelmachine learningclassificationprediction
Type
Article
Information
Political Analysis , Volume 30 , Issue 4 , October 2022 , pp. 463 - 480
Copyright
© The Author(s) 2021. Published by Cambridge University Press on behalf of the Society for Political Methodology

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

Article purchase

Temporarily unavailable

Footnotes

Edited by Jeff Gill

References

Almanzar, T., Aspinwall, M., and Crow, D.. 2018. “Freedom of Information in Times of Crisis: The Case of Mexico’s War on Drugs.” Governance 31(2):321–339.CrossRefGoogle Scholar
Bagozzi, B. E., and Berliner, D.. 2018. “The politics of Scrutiny in Human Rights Monitoring: Evidence from Structural Topic Models of US State Department Human Rights Reports.” Political Science Research and Methods 6(4):661–677.CrossRefGoogle Scholar
Baker, S. and Korhonen, A.-L.. 2017. “Initializing Neural Networks for Hierarchical Multi-Label Text Classification.” In 16th Biomedical Natural Language Processing Workshop, 307–315. Association for Computational Linguistics.CrossRefGoogle Scholar
Barberá, P., Boydstun, A. E., Linn, S., McMahon, R., and Nagler, J.. 2020. “Automated text classification of News Articles: A Practical Guide.” Political Analysis 29(1):19–42.CrossRefGoogle Scholar
Berliner, D., Bagozzi, B. E., and Palmer-Rubin, B.. 2018. “What Information do Citizens Want? Evidence from One Million Information Requests in Mexico.” World Development 109:222–235.CrossRefGoogle Scholar
Berliner, D., Bagozzi, B. E., Palmer-Rubin, B., and Erlich, A.. 2021. “The Political Logic of Government Disclosure: Evidence from Information Requests in Mexico.” Journal of Politics 83(1):229–245.CrossRefGoogle Scholar
Blagus, R., and Lusa, L.. 2013. “Smote for High-Dimensional Class-Imbalanced Data.” BMC Bioinformatics 14(1):64.CrossRefGoogle ScholarPubMed
Boutell, M. R., Luo, J., Shen, X., and Brown, C. M.. 2004. “Learning Multi-Label Scene Classification.” Pattern Recognition 37(9):1757–1771.CrossRefGoogle Scholar
Brinker, K., Fürnkranz, J., and Hüllermeier, E.. 2006. “A Unified Model for Multilabel Classification and Ranking.” In Proceedings of the 2006 Conference on ECAI 2006: 17th European Conference on Artificial Intelligence August 29–September 1, 2006, Riva del Garda, Italy. Amsterdam, the Netherlands: IOS Press.Google Scholar
Cantú, F., and Saiegh, S. M.. 2011. “Fraudulent Democracy? An Analysis of Argentina’s Infamous Decade using Supervised Machine Learning.” Political Analysis 19(4):409–433.CrossRefGoogle Scholar
Chang, C., and Masterson, M.. 2020. “Using Word Order in Political Text Classification with Long Short-Term Memory Models.” Political Analysis 28(3):395–411.CrossRefGoogle Scholar
Chen, J., Pan, J., and Xu, Y.. 2016. “Sources of Authoritarian Responsiveness: A Field Experiment in China.” American Journal of Political Science 60(2):383–400.CrossRefGoogle Scholar
Cheng, W., and Hüllermeier, E.. 2009. “Combining Instance-Based Learning and Logistic Regression for Multilabel Classification.” Machine Learning 76(2–3):211–225.CrossRefGoogle Scholar
Cingranelli, D. L., and Richards, D. L.. 2010. “The Cingranelli and Richards (CIRI) Human Rights Data Project.” Human Rights Quarterly 32(2):401.CrossRefGoogle Scholar
Clare, A. and King, R. D.. 2001. “Knowledge Discovery in Multi-Label Phenotype Data.” In European Conference on Principles of Data Mining and Knowledge Discovery, 42–53. Berlin: Springer.CrossRefGoogle Scholar
D’Orazio, V., Landis, S. T., Palmer, G., and Schrodt, P.. 2014. “Separating the Wheat from the Chaff: Applications Of Automated Document Classification Using Support Vector Machines.” Political Analysis 22(2):224–242.CrossRefGoogle Scholar
deHaan, E., Lawrence, A., and Litjens, R.. 2019. “Measurement Error in Dependent Variables in Accounting: Illustrations Using Google Ticker Search and Simulations.” Available at SSRN: https://ssrn.com/abstract=3398287 or http://doi.org/10.2139/ssrn.3398287.CrossRefGoogle Scholar
Dietrich, B. J., Hayes, M., and O’Brien, D. Z.. 2019. “Pitch Perfect: Vocal Pitch and the Emotional Intensity of Congressional Speech.” American Political Science Review 113(4):941–962.CrossRefGoogle Scholar
Dudani, S. A. 1976. “The Distance-Weighted K-Nearest-Neighbor Rule.” IEEE Transactions on Systems, Man, and Cybernetics 8(4):311–313.Google Scholar
Elisseeff, A. and Weston, J.. 2002. “A Kernel Method for Multi-Labelled Classification.” In Advances in Neural Information Processing Systems 14, 681–687. Cambridge, MA: MIT Press.Google Scholar
Erlich, A., Dantas, S., Bagozzi, B., Berliner, D., and Palmer-Rubin, B.. 2021. “Replication Data for: Multi-label Prediction for Political Text-as-Data.” https://doi.org/10.7910/DVN/SOVPA4, Harvard Dataverse V1.1.CrossRefGoogle Scholar
Fariss, C. J., et al. 2015. “Human Rights Texts: Converting Human Rights Primary Source Documents into Data.” PLOS One 10(9):e0138935.CrossRefGoogle ScholarPubMed
Goncalves, E. C., Plastino, A., and Freitas, A. A.. 2013. “A Genetic Algorithm for Optimizing the Label Ordering in Multi-Label Classifier Chains.” In Machine Learning and Knowledge Discovery in Databases, edited by Calders, T., Esposito, F., Hüllermeier, E., and Meo, R., 453–468. Berlin: Springer.Google Scholar
Greene, K. T., Park, B., and Colaresi, M.. 2019. “Machine Learning Human Rights and Wrongs: How the Successes and Failures of Supervised Learning Algorithms Can Inform the Debate about Information Effects.” Political Analysis 27(2):223–230.CrossRefGoogle Scholar
Grimmer, J., and Stewart, B. M.. 2013. “Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts.” Political Analysis 21(3):267–297.CrossRefGoogle Scholar
Ji, S., Tang, L., Yu, S., and Ye, J.. 2008. “Extracting Shared Subspace for Multi-Label Classification.” In Proceedings of the 14th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. New York: Association for Computing Machinery.Google Scholar
Kostyuk, N., and Zhukov, Y. M.. 2019. “Invisible Digital Front: Can Cyber Attacks Shape Battlefield Events?” Journal of Conflict Resolution 63(2):317–347.CrossRefGoogle Scholar
Laver, M., Benoit, K., and Garry, J.. 2003. “Extracting Policy Positions from Political Texts using Words as Data.” American Political Science Review 92(2):311–331.Google Scholar
Madjarov, G., Kocev, D., Gjorgjevikj, D., and Džeroski, S.. 2012. “An Extensive Experimental Comparison of Methods for Multi-Label Learning.” Pattern Recognition 45(9):3084–3104.CrossRefGoogle Scholar
McCallum, A. K. 1999. “Multi-Label Text Classification with a Mixture Model Trained by EM.” In AAAI 99 Workshop on Text Learning. https://mimno.infosci.cornell.edu/info6150/readings/multilabel.pdf.Google Scholar
Miller, B., Linder, F., and Mebane, W. R.. 2020. “Active Learning Approaches for Labeling Text: Review and Assessment of the Performance of Active Learning Approaches.” Political Analysis 28(4):532–551.CrossRefGoogle Scholar
Mitts, T. 2019. “From Isolation to Radicalization: Anti-Muslim Hostility and Support for ISIS in the West.” American Political Science Review 113(1):173–194.CrossRefGoogle Scholar
Murdie, A., Davis, D. R., and Park, B.. 2020. “Advocacy Output: Automated Coding Documents from Human Rights Organizations.” Journal of Human Rights 19(1):83–98.CrossRefGoogle Scholar
Park, B., Greene, K., and Colaresi, M.. 2020a. “How to Teach Machines to Read Human Rights Reports and Identify Judgments at Scale.” Journal of Human Rights 19(1):99–116.CrossRefGoogle Scholar
Park, B., Greene, K., and Colaresi, M.. 2020b. “Human Rights are (Increasingly) Plural: Learning the Changing Taxonomy of Human Rights from Large-Scale Text Reveals Information Effects.” American Political Science Review 114(3):888–910.CrossRefGoogle Scholar
Pedregosa, F., et al. 2011. “Scikit-Learn: Machine Learning in Python.” Journal of Machine Learning Research 12:2825–2830.Google Scholar
Probst, P., Au, Q., Casalicchio, G., Stachl, C., and Bischl, B.. 2017. “Multilabel Classification with R Package MLR.” The R Journal 9(1):352–369.CrossRefGoogle Scholar
Qi, G.-J., Hua, X.-S., Rui, Y., Tang, J., Mei, T., and Zhang, H.-J.. 2007. “Correlative multi-label video annotation.” In Proceedings of the 15th ACM International Conference on Multimedia. New York: Association for Computing Machinery.Google Scholar
Read, J., Martino, L., and Luengo, D.. 2014. “Efficient Monte Carlo Methods for Multi-Dimensional Learning with Classifier Chains.” Pattern Recognition 47(3):1535–1546.CrossRefGoogle Scholar
Read, J., Pfahringer, B., Holmes, G., and Frank, E.. 2009. “Classifier Chains for Multi-Label Classification.” In Joint European Conference on Machine Learning and Knowledge Discovery in Databases. Berlin: Springer.Google Scholar
Smith, C., and Jin, Y.. 2014. “Evolutionary Multi-Objective Generation of Recurrent Neural Network Ensembles for Time Series Prediction.” Neurocomputing 143:302–311.CrossRefGoogle Scholar
Szymański, P. and Kajdanowicz, T.. 2017. “A Scikit-Based Python Environment for Performing Multi-Label Classification.” ArXiv e-prints, https://arxiv.org/abs/1702.01460.Google Scholar
Tang, L., Rajan, S., and Narayanan, V. K.. 2009. “Large Scale Multi-Label Classification via Metalabeler.” In Proceedings of the 18th International Conference on World Wide Web, 211–220. New York: Association for Computing Machinery.CrossRefGoogle Scholar
Tawiah, C. and Sheng, V.. 2013. “Empirical Comparison of Multi-Label Classification Algorithms.” Proceedings of the AAAI Conference on Artificial Intelligence 27(1). https://ojs.aaai.org/index.php/AAAI/article/view/8521.CrossRefGoogle Scholar
Torres, M. and Cantú, F.. 2020. “Learning to see: Visual Analysis for Social Science Data.” Working Paper.Google Scholar
Tsoumakas, G. and Vlahavas, I.. 2007. “Random k-Labelsets: An Ensemble Method for Multilabel Classification.” In European Conference on Machine Learning. Berlin: Springer.Google Scholar
Ueda, N. and Saito, K.. 2002. “Parametric Mixture Models for Multi-Labeled Text.” In Advances in Neural Information Processing Systems 15, edited by Becker, S., Thrun, S., and Obermayer, K., 737–744.Google Scholar
Wansbeek, T., and Meijer, E.. 2000. Measurement Error and Latent Variables in Econometrics. Amsterdam: North Holland.Google Scholar
Williams, N. W., Casas, A., and Wilkerson, J. D.. 2020. Images as Data for Social Science Research: An Introduction to Convolutional Neural Nets for Image Classification. Cambridge: Cambridge University Press.CrossRefGoogle Scholar
Xu, D., Shi, Y., Tsang, I. W., Ong, Y.-S., Gong, C., and Shen, X.. 2019. “Survey on Multi-Output Learning.” IEEE Transactions on Neural Networks and Learning Systems 24(9):2409–2429.Google Scholar
Yan, R., Tesic, J., and Smith, J. R.. 2007. “Model-Shared Subspace Boosting for Multi-Label Classification.” In Proceedings of the 13th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. New York: Association for Computing Machinery.Google Scholar
Younes, Z., Abdallah, F., Denoeux, T., and Snoussi, H.. 2011. “A Dependent Multilabel Classification Method Derived from the k-Nearest Neighbor Rule.” EURASIP Journal on Advances in Signal Processing 2011:645964.CrossRefGoogle Scholar
Zhang, M.-L. and Zhang, K.. 2010. “Multi-Label Learning by Exploiting Label Dependency.” In Proceedings of the 16th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. New York: Association for Computing Machinery.Google Scholar
Zhang, M.-L., and Zhou, Z.-H.. 2007. “ML-KNN: A Lazy Learning Approach to Multi-Label Learning.” Pattern Recognition 40(7):2038–2048.CrossRefGoogle Scholar
Zhang, M.-L., and Zhou, Z.-H.. 2013. “A Review on Multi-Label Learning Algorithms.” IEEE Transactions on Knowledge and Data Engineering 26(8):1819–1837.CrossRefGoogle Scholar
Supplementary material: PDF

Erlich et al. supplementary material

Appendix

Download Erlich et al. supplementary material(PDF)
PDF 261.1 KB
Supplementary material: Link

Erlich et al. Dataset

Dataset

https://doi.org/10.7910/DVN/SOVPA4
Link