Skip to main content Accessibility help

Measuring the Significance of Policy Outputs with Positive Unlabeled Learning



Identifying important policy outputs has long been of interest to political scientists. In this work, we propose a novel approach to the classification of policies. Instead of obtaining and aggregating expert evaluations of significance for a finite set of policy outputs, we use experts to identify a small set of significant outputs and then employ positive unlabeled (PU) learning to search for other similar examples in a large unlabeled set. We further propose to automate the first step by harvesting “seed” sets of significant outputs from web data. We offer an application of the new approach by classifying over 9,000 government regulations in the United Kingdom. The obtained estimates are successfully validated against human experts, by forecasting web citations, and with a construct validity test.


Corresponding author

Radoslaw Zubek, Associate Professor, Department of Politics and International Relations, University of Oxford,
Abhishek Dasgupta, Research Software Engineer, Department of Computer Science, University of Oxford,
David Doyle Associate Professor, Department of Politics and International Relations, University of Oxford,


Hide All

We thank Ken Benoit, Shaun Bevan, Jack Blumenau, Ruth Dixon, Andy Eggers, Mikolaj Firlej, Diego Garzia, Felix Krawatzek, Ben Lauderdale, Innar Liiv, Marcin Matczak, Nolan McCarty, Sharon Meraz, Scot Peterson, Ulrich Sieberer, Jayne Sunley, Mariusz Zubek, and three anonymous reviewers for helpful comments and suggestions. We acknowledge the financial support from BA/Leverhulme Small Research Grant (SG152293) and John Fell OUP Research Fund (151/025). Abhishek Dasgupta acknowledges support from the Centre for Technology and Global Affairs. David Doyle acknowledges support from the Jackie Lambert Research Fund at St Hugh’s College. Our thanks to Kwok Cheung, Tom Fleming, Vladimir Mikulik, Paul Moore, Edward Percarpio, Hugh Thomas, Zoe Harrison, Neerja Gurnani, Atrayee De, and Jacob Yu for excellent research assistance. Replication files are available at the American Political Science Review Dataverse:



Hide All
Angelova, Mariyana, Hanna, Bäck, Wolfgang C., Müller, and Daniel, Strobl. 2018. “Veto Player Theory and Reform Making in Western Europe.” European Journal of Political Research 57 (2): 282307.
Becher, Michael. 2010. “Constraining Ministerial Power: The Impact of Veto Players on Labor Market Reforms in Industrial Democracies, 1973-2000.” Comparative Political Studies 43 (1): 3360.
Chang, Yin-Wen, and Lin, Chih-Jen. 2008. “Feature Ranking Using Linear SVM.” In Proceedings of the Workshop on Causation and Prediction Challenge at WCCI 3: 5364.
Clinton, Joshua D., and Lapinski, John S.. 2006. “Measuring Legislative Accomplishment, 1877–1994.” American Journal of Political Science 50 (1): 232249.
Conley, Richard S., and Bekafigo, Marija A.. 2010. “No Irish Need Apply? Veto Players and Legislative Productivity in the Republic of Ireland, 1949-2000.” Comparative Political Studies 43 (1): 91118.
Döring, Herbert. 2001. “Parliamentary Agenda Control and Legislative Outcomes in Western Europe.” Legislative Studies Quarterly 26 (1): 145165.
Epstein, Lee, and Segal, Jeffrey A.. 2000. “Measuring Issue Salience.” American Journal of Political Science 44 (1): 6683.
Howell, William, Scott, Adler, Charles, Cameron, and Charles, Riemann. 2000. “Divided Government and the Legislative Productivity of Congress, 1945-94.” Legislative Studies Quarterly 25 (2): 285312.
Huber, John D., and Shipan, Charles R.. 2002. Deliberate Discretion? The Institutional Foundations of Bureaucratic Autonomy. New York: Cambridge University Press.
Junge, Dirk, Koenig, Thomas, and Luig, Bernd. 2015. “Legislative Gridlock and Bureaucratic Politics in the European Union.” British Journal of Political Science 45 (4): 777797.
Li, Xiaoli, and Liu, Bing. 2003. “Learning to Classify Texts Using Positive and Unlabeled Data.” In Proceedings of the 18th International Conference on Artificial intelligence, 587–592.
Liu, Bing. 2011. Web Data Mining. New York: Springer.
Mayhew, David R. 1991. Divided We Govern. New Haven, CT: Yale University Press.
Page, Edward C. 2001. Governing by Numbers: Delegated Legislation and Everyday Policymaking. Oxford: Hart Publishing.
Schultz, Mike, Doerr, John E., and Frederiksen, Lee. 2013. Professional Services Marketing: How the Best Firms Build Premier Brands, Thriving Lead Generation Engines, and Cultures of Business Development Success. Hoboken, NJ: John Wiley & Sons.
Seely, Antony. 2020. The Budget and Annual Finance Bill. Briefing Paper 813. Westminister: House of Commons Library.
Tsebelis, George. 1999. “Veto Players and Law Production in Parliamentary Democracies: An Empirical Analysis.” American Political Science Review 93 (3): 591608.
Warber, Adam L., Ouyang, Yu, and Waterman, Richard W.. 2018. “Landmark Executive Orders.” Presidential Studies Quarterly 48 (1): 110126.
Zhang, Bangzuo, and Zuo, Wanli. 2008. “Learning from Positive and Unlabeled Examples: A Survey.” In 2008 International Symposiums on Information Processing, 650–654.
Type Description Title
Supplementary materials

Zubek et al. Dataset

Supplementary materials

Zubek et al. supplementary material
Online Appendix

 PDF (6.7 MB)
6.7 MB

Measuring the Significance of Policy Outputs with Positive Unlabeled Learning



Altmetric attention score

Full text views

Total number of HTML views: 0
Total number of PDF views: 0 *
Loading metrics...

Abstract views

Total abstract views: 0 *
Loading metrics...

* Views captured on Cambridge Core between <date>. This data will be updated every 24 hours.

Usage data cannot currently be displayed.