Skip to main content Accessibility help

Assessing the validity and reliability of measurements when evaluating public policy

  • Michele Crepaz (a1) and Raj Chari (a2)


A substantial aspect of scientific research involves linking concepts to observations using measurements. This exercise has raised questions among researchers of whether or not measurements “truly” and “reliably” capture ideas and observations. We address this question by setting out a methodological standard on how to assess the validity and reliability of measurements. We do this by examining measurements that evaluate public policy, arguing that this topic is gaining increasing attention from political science researchers and policymakers. The analysis concerns measurements of the level of transparency and accountability of lobbying laws, central to recent regulatory policy research. We conduct convergent validation, content validation and reproducibility tests on four indices applied to 13 regulations found worldwide. By doing so, the article provides scholars with an evaluation of measurements of lobbying laws’ robustness, while offering methodological and theoretical lessons of value to larger regulatory and public policy scholarship.



Hide All
Adcock, R. and Collier, D. (2001) Measurement Validity: A Shared Standard for Qualitative and Quantitative Research. American Political Science Review 95(3): 529546.
Banisar, D. (2006) Freedom of Information Around the World 2006: A Global Survey of Access to Government Information Laws. London, UK: Privacy International.
Bartlett, J. W. and Frost, C. (2008) Reliability, Repeatability and Reproducibility: Analysis of Measurement Errors in Continuous Variables. Ultrasound in Obstetrics and Gynecology 31(4): 466475.
Benoit, K., Conway, D., Lauderdale, B. E., Laver, M. and Mikhaylov, S. (2016) Crowd-Sourced Text Analysis: Reproducible and Agile Production of Political Data. American Political Science Review 110(2): 279285.
Boräng, F., Eising, R., Klüver, H., Mahoney, C., Naurin, D., Rasch, D. and Rozbicka, P. (2014) Identifying Frames: A Comparison of Research Methods. Interest Groups & Advocacy 3(2): 188201.
Bowman, K., Lehoucq, F. and Mahoney, J. (2005) Measuring Political Democracy: Case Expertise, Data Adequacy, and Central America. Comparative Political Studies 38(8): 939970.
Brining, M., Holcombe, R. and Schwartzstein, L. (1993) The Regulation of Lobbyists. Public Choice 77(2): 377384.
Chambers, S. A. and Carver, T. (eds.), (2008) William E. Connolly: Democracy, Pluralism and Political Theory. London, UK: Routledge.
Chari, R., Hogan, J. and Murphy, G. (2010) Regulating Lobbying: A Global Comparison. Manchester, UK: Manchester University Press.
Clarke, B. (1979) Eccentrically Contested Concepts. British Journal of Political Science 9(1): 122126.
Collier, D. F., Hidalgo, D. and Maciuceanu, O. A. (2006) Essentially Contested Concepts: Debates and Applications. Journal of Political Ideologies 11(3): 211246.
Connolly, W. E. (1974) The Terms of Political Discourse. Lexington, MA: Heath D.C. and Co.
Crepaz, M. (2016a) Investigating the Robustness of Lobbying Laws: Evidence from the Austrian Case. Interest Groups & Advocacy 5(1): 524.
Crepaz, M. (2016b) Why Do We Have Lobbying Rules? Investigating the Introduction of Lobbying Laws in EU and OECD Member States. Paper presented at the 87th SPSA Annual Conference in San Juan, January, 8th 2016, Puerto Rico.
Crepaz, M. and Chari, R. (2014) The EU’s Initiatives to Regulate Lobbyists: Good or Bad Administration? Cuadernos Europeos de Deusto 51(1): 7197.
Gallie, W. B. (1955) Essentially Contested Concepts. Proceedings from the Aristotelian Society 56(1): 167198.
Greenwood, J. (1998) Regulating Lobbying in the European Union. Parliamentary Affairs 51(4): 587599.
Greenwood, J. and Dreger, J. (2013) The Transparency Register: A European Vanguard of Strong Lobby Regulation. Interest Groups & Advocacy 2(2): 139162.
Greenwood, J. and Thomas, C. S. (1998) Introduction: Regulating Lobbying in the Western World. Parliamentary Affairs 51(4): 487488.
Hamm, K., Weber, A. and Anderson, B. (1994) The Impact of Lobbying Laws and Their Enforcement: A Contrasting View. Social Sciences Quarterly 75(2): 378381.
Holman, C. and Luneburg, W. (2012) Lobbying and Transparency: A Comparative Analysis of Regulatory Reform. Interest Groups & Advocacy 1(1): 75104.
Holyoke, T. T. (2015) The Ethical Lobbyist: Reforming Washington’s Influence Industry. Washington, DC: Georgetown University Press.
Hrebenar, R. J., Nakainura, A. and Nakamura, A. (1998) Lobby Regulation in the Japanese Diet. Parliamentary Affairs 51(4): 551552.
Jordan, G. (1998) Towards Regulation in the UK: From General Good Sense “to Formalized Rules”. Parliamentary Affairs 51(4): 524525.
King, G., Keohane, R. O. and Verba, S. (1994) Designing Social Inquiry: Scientific Inference in Qualitative Research. Princeton, NJ: Princeton University Press.
Krippendorff, K. (2004) Reliability in Content Analysis: Some Common Misconceptions and Recommendations. Human Communication Research 30(3): 411433.
Landis, R. and Koch, G. (1977) The Measurement of Observer Agreement for Categorical Data. Biometrics 33(1): 159174.
Lombard, M., Snyder-Dutch, J. and Bracken Campanella, C. (2002) Content Analysis in Mass Communication Assessment and Reporting of Intercoder Reliability. Human Communication Research 28(4): 587604.
Lowery, D. and Grey, V. (1997) How Some Rules Just Don’t Matter: The Regulation of Lobbyists. Public Choice 91(2): 139147.
Mahoney, J. and Goertz, G. (2006) A Tale of Two Cultures: Contrasting Quantitative and Qualitative Research. Political Analysis 14(3): 227249.
Maggetti, M. and Gilardi, F. (2016) Problems (and Solutions) in the Measurement of Policy Diffusion Mechanisms. Journal of Public Policy 36(1): 87107.
McMenamin, I. (2004) Varieties of Capitalist Democracy: What Difference Does East-Central Europe Make? Journal of Public Policy 24(3): 259274.
Mikhaylov, S., Laver, M. and Benoit, K. (2012) Coder Reliability and Misclassification in the Human Coding of Party Manifestos. Political Analysis 20(1): 7991.
Munck, G. L. and Verkuilen, J. (2002) Conceptualizing and Measuring Democracy. Comparative Political Studies 35(1): 534.
Newmark, A. J. (2005) Measuring State Legislative Lobbying Regulation, 1990–2003. State Politics & Policy Quarterly 5(2): 182191.
Organisation for Economic Co-Operation and Development (OECD) (2008) Lobbyists, Government and Public Trust: Building a Legislative Framework for Enhancing Transparency and Accountability in Lobbying. Paris, France: OECD Publisher.
Organisation for Economic Co-operation and Development (OECD) (2012) Lobbyists, Government and Public Trust, Volume 2. Paris, France: OECD Publisher.
Opheim, C. (1991) Explaining the Differences in State Lobby Regulation. The Western Political Quarterly 44(2): 405421.
Ozymy, J. (2010) Assessing the Impact of Legislative Lobbying Regulations on Interest Group Influence in US State Legislatures. State Politics & Policy Quarterly 10(4): 397420.
Ozymy, J. (2013) Keepin’ on the Sunny Side: Scandals, Organized Interests and the Passage of Legislative Lobbying Laws in the American States. American Politics Research 41(1): 323.
Paxton, P. (2000) Women in the Measurement of Democracy: Problems and Operationalization. Studies in Comparative International Development 35(3): 92111.
Rechtman, R. E. and Larsen-Ledet, J. P. (1998) Regulation of Lobbyists in Scandinavia: A Danish Perspective. Parliamentary Affairs 51(4): 579586.
Rocco, P. and Thurston, C. (2014) From Metaphors to Measures: Observable Indicators of Gradual Institutional Change. Journal of Public Policy 34(1): 3562.
Rogers, M. Z. and Weller, N. (2014) Income Taxation and the Validity of State Capacity Indicators. Journal of Public Policy 34(2): 183206.
Ronit, K. and Schneider, V. (1998) The Strange Case of Regulating Lobbying in Germany. Parliamentary Affairs 51(4): 559567.
Rooduijn, M. and Pauwels, T. (2011) Measuring Populism: Comparing Two Methods of Content Analysis. West European Politics 34(6): 12721283.
Rosenson, B. A. (2003) Against Their Apparent Self-Interest: The Authorization of Independent State Legislative Ethics Commissions, 1973–96. State Politics & Policy Quarterly 3(1): 4265.
Rush, M. (1998) The Canadian Experience: The Lobbyists Registration Act. Parliamentary Affairs 51(4): 516523.
Seawright, J. and Collier, D. (2014) Rival Strategies of Validation: Tools for Evaluating Measures of Democracy. Comparative Political Studies 47(1): 111138.
Stone, D. (2004) Transfer Agents and Global Networks in the Transnationalization of Policy. Journal of European Public Policy 11(3): 545566.
Thomas, C. S. (1998) Interest Group Regulation Across the United States: Rationale, Development and Consequences. Parliamentary Affairs 58(4): 500515.
True, J. and Mintrom, M. (2001) Transnational Networks and Policy Diffusion: The Case of Gender Mainstreaming. International Studies Quarterly 45(1): 2757.
Veksler, A. (2015) Diluted Regulations: A Need to Review the Theoretical Classification of the Different Lobbying Regulatory Environments. Journal of Public Affairs 15(1): 5664.
Wagenaar, A. C., Harwood, E. M., Silianoff, C. and Toomey, T. L. (2005) Measuring Public Policy: The Case of Beer Keg Registration Laws. Evaluation and Program Planning 28(4): 359367.
Warhurst, J. (1998) Locating the Target: Regulating Lobbying in Australia. Parliamentary Affairs 58(4): 538550.
Witko, C. (2007) Explaining Increases in the Stringency of State Campaign Finance Regulation, 1993–2002. State Politics & Policy Quarterly 7(4): 369393.
Woll, C. (2012) The Brash and the Soft-Spoken: Lobbying Styles in a Transatlantic Comparison. Interest Groups & Advocacy 1(2): 193214.
Yishai, Y. (1998) Regulation of Interest Groups in Israel. Parliamentary Affairs 51(4): 568578.
Recommend this journal

Email your librarian or administrator to recommend adding this journal to your organisation's collection.

Journal of Public Policy
  • ISSN: 0143-814X
  • EISSN: 1469-7815
  • URL: /core/journals/journal-of-public-policy
Please enter your name
Please enter a valid email address
Who would you like to send this to? *


Type Description Title
Supplementary materials

Crepaz and Chari supplementary material
Crepaz and Chari supplementary material

 Unknown (30 KB)
30 KB


Altmetric attention score

Full text views

Total number of HTML views: 0
Total number of PDF views: 0 *
Loading metrics...

Abstract views

Total abstract views: 0 *
Loading metrics...

* Views captured on Cambridge Core between <date>. This data will be updated every 24 hours.

Usage data cannot currently be displayed