Hostname: page-component-76fb5796d-vfjqv Total loading time: 0 Render date: 2024-04-25T14:41:43.224Z Has data issue: false hasContentIssue false
  • English
  • Français

Assessing the validity and reliability of measurements when evaluating public policy

Published online by Cambridge University Press:  09 January 2017

Michele Crepaz  and
Raj Chari
Michele Crepaz
Affiliation:
Department of Political Science, Trinity College Dublin, Ireland E-mail: crepazm@tcd.ie
Raj Chari
Affiliation:
Department of Political Science, Trinity College Dublin, Ireland E-mail: charir@tcd.ie
Get access Rights & Permissions [Opens in a new window]

Abstract

A substantial aspect of scientific research involves linking concepts to observations using measurements. This exercise has raised questions among researchers of whether or not measurements “truly” and “reliably” capture ideas and observations. We address this question by setting out a methodological standard on how to assess the validity and reliability of measurements. We do this by examining measurements that evaluate public policy, arguing that this topic is gaining increasing attention from political science researchers and policymakers. The analysis concerns measurements of the level of transparency and accountability of lobbying laws, central to recent regulatory policy research. We conduct convergent validation, content validation and reproducibility tests on four indices applied to 13 regulations found worldwide. By doing so, the article provides scholars with an evaluation of measurements of lobbying laws’ robustness, while offering methodological and theoretical lessons of value to larger regulatory and public policy scholarship.

Keywords

lobbyingmeasurementregulationreliabilityvalidity
Type
Research Article
Information
Journal of Public Policy , Volume 38 , Issue 3 , September 2018 , pp. 275 - 304
Copyright
© Cambridge University Press, 2017 

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

References

Adcock, R. and Collier, D. (2001) Measurement Validity: A Shared Standard for Qualitative and Quantitative Research. American Political Science Review 95(3): 529–546.Google Scholar
Banisar, D. (2006) Freedom of Information Around the World 2006: A Global Survey of Access to Government Information Laws. London, UK: Privacy International.Google Scholar
Bartlett, J. W. and Frost, C. (2008) Reliability, Repeatability and Reproducibility: Analysis of Measurement Errors in Continuous Variables. Ultrasound in Obstetrics and Gynecology 31(4): 466–475.Google Scholar
Benoit, K., Conway, D., Lauderdale, B. E., Laver, M. and Mikhaylov, S. (2016) Crowd-Sourced Text Analysis: Reproducible and Agile Production of Political Data. American Political Science Review 110(2): 279–285.Google Scholar
Boräng, F., Eising, R., Klüver, H., Mahoney, C., Naurin, D., Rasch, D. and Rozbicka, P. (2014) Identifying Frames: A Comparison of Research Methods. Interest Groups & Advocacy 3(2): 188–201.Google Scholar
Bowman, K., Lehoucq, F. and Mahoney, J. (2005) Measuring Political Democracy: Case Expertise, Data Adequacy, and Central America. Comparative Political Studies 38(8): 939–970.Google Scholar
Brining, M., Holcombe, R. and Schwartzstein, L. (1993) The Regulation of Lobbyists. Public Choice 77(2): 377–384.Google Scholar
Chambers, S. A. and Carver, T. (eds.), (2008) William E. Connolly: Democracy, Pluralism and Political Theory. London, UK: Routledge.Google Scholar
Chari, R., Hogan, J. and Murphy, G. (2010) Regulating Lobbying: A Global Comparison. Manchester, UK: Manchester University Press.Google Scholar
Clarke, B. (1979) Eccentrically Contested Concepts. British Journal of Political Science 9(1): 122–126.Google Scholar
Collier, D. F., Hidalgo, D. and Maciuceanu, O. A. (2006) Essentially Contested Concepts: Debates and Applications. Journal of Political Ideologies 11(3): 211–246.Google Scholar
Connolly, W. E. (1974) The Terms of Political Discourse. Lexington, MA: Heath D.C. and Co.Google Scholar
Crepaz, M. (2016a) Investigating the Robustness of Lobbying Laws: Evidence from the Austrian Case. Interest Groups & Advocacy 5(1): 5–24.Google Scholar
Crepaz, M. (2016b) Why Do We Have Lobbying Rules? Investigating the Introduction of Lobbying Laws in EU and OECD Member States. Paper presented at the 87th SPSA Annual Conference in San Juan, January, 8th 2016, Puerto Rico.Google Scholar
Crepaz, M. and Chari, R. (2014) The EU’s Initiatives to Regulate Lobbyists: Good or Bad Administration? Cuadernos Europeos de Deusto 51(1): 71–97.Google Scholar
Gallie, W. B. (1955) Essentially Contested Concepts. Proceedings from the Aristotelian Society 56(1): 167–198.Google Scholar
Greenwood, J. (1998) Regulating Lobbying in the European Union. Parliamentary Affairs 51(4): 587–599.Google Scholar
Greenwood, J. and Dreger, J. (2013) The Transparency Register: A European Vanguard of Strong Lobby Regulation. Interest Groups & Advocacy 2(2): 139–162.Google Scholar
Greenwood, J. and Thomas, C. S. (1998) Introduction: Regulating Lobbying in the Western World. Parliamentary Affairs 51(4): 487–488.Google Scholar
Hamm, K., Weber, A. and Anderson, B. (1994) The Impact of Lobbying Laws and Their Enforcement: A Contrasting View. Social Sciences Quarterly 75(2): 378–381.Google Scholar
Holman, C. and Luneburg, W. (2012) Lobbying and Transparency: A Comparative Analysis of Regulatory Reform. Interest Groups & Advocacy 1(1): 75–104.Google Scholar
Holyoke, T. T. (2015) The Ethical Lobbyist: Reforming Washington’s Influence Industry. Washington, DC: Georgetown University Press.Google Scholar
Hrebenar, R. J., Nakainura, A. and Nakamura, A. (1998) Lobby Regulation in the Japanese Diet. Parliamentary Affairs 51(4): 551–552.Google Scholar
Jordan, G. (1998) Towards Regulation in the UK: From General Good Sense “to Formalized Rules”. Parliamentary Affairs 51(4): 524–525.Google Scholar
King, G., Keohane, R. O. and Verba, S. (1994) Designing Social Inquiry: Scientific Inference in Qualitative Research. Princeton, NJ: Princeton University Press.Google Scholar
Krippendorff, K. (2004) Reliability in Content Analysis: Some Common Misconceptions and Recommendations. Human Communication Research 30(3): 411–433.Google Scholar
Landis, R. and Koch, G. (1977) The Measurement of Observer Agreement for Categorical Data. Biometrics 33(1): 159–174.Google Scholar
Lombard, M., Snyder-Dutch, J. and Bracken Campanella, C. (2002) Content Analysis in Mass Communication Assessment and Reporting of Intercoder Reliability. Human Communication Research 28(4): 587–604.Google Scholar
Lowery, D. and Grey, V. (1997) How Some Rules Just Don’t Matter: The Regulation of Lobbyists. Public Choice 91(2): 139–147.Google Scholar
Mahoney, J. and Goertz, G. (2006) A Tale of Two Cultures: Contrasting Quantitative and Qualitative Research. Political Analysis 14(3): 227–249.Google Scholar
Maggetti, M. and Gilardi, F. (2016) Problems (and Solutions) in the Measurement of Policy Diffusion Mechanisms. Journal of Public Policy 36(1): 87–107.Google Scholar
McMenamin, I. (2004) Varieties of Capitalist Democracy: What Difference Does East-Central Europe Make? Journal of Public Policy 24(3): 259–274.Google Scholar
Mikhaylov, S., Laver, M. and Benoit, K. (2012) Coder Reliability and Misclassification in the Human Coding of Party Manifestos. Political Analysis 20(1): 79–91.Google Scholar
Munck, G. L. and Verkuilen, J. (2002) Conceptualizing and Measuring Democracy. Comparative Political Studies 35(1): 5–34.Google Scholar
Newmark, A. J. (2005) Measuring State Legislative Lobbying Regulation, 1990–2003. State Politics & Policy Quarterly 5(2): 182–191.Google Scholar
Organisation for Economic Co-Operation and Development (OECD) (2008) Lobbyists, Government and Public Trust: Building a Legislative Framework for Enhancing Transparency and Accountability in Lobbying. Paris, France: OECD Publisher.Google Scholar
Organisation for Economic Co-operation and Development (OECD) (2012) Lobbyists, Government and Public Trust, Volume 2. Paris, France: OECD Publisher.Google Scholar
Opheim, C. (1991) Explaining the Differences in State Lobby Regulation. The Western Political Quarterly 44(2): 405–421.Google Scholar
Ozymy, J. (2010) Assessing the Impact of Legislative Lobbying Regulations on Interest Group Influence in US State Legislatures. State Politics & Policy Quarterly 10(4): 397–420.Google Scholar
Ozymy, J. (2013) Keepin’ on the Sunny Side: Scandals, Organized Interests and the Passage of Legislative Lobbying Laws in the American States. American Politics Research 41(1): 3–23.Google Scholar
Paxton, P. (2000) Women in the Measurement of Democracy: Problems and Operationalization. Studies in Comparative International Development 35(3): 92–111.Google Scholar
Rechtman, R. E. and Larsen-Ledet, J. P. (1998) Regulation of Lobbyists in Scandinavia: A Danish Perspective. Parliamentary Affairs 51(4): 579–586.Google Scholar
Rocco, P. and Thurston, C. (2014) From Metaphors to Measures: Observable Indicators of Gradual Institutional Change. Journal of Public Policy 34(1): 35–62.Google Scholar
Rogers, M. Z. and Weller, N. (2014) Income Taxation and the Validity of State Capacity Indicators. Journal of Public Policy 34(2): 183–206.Google Scholar
Ronit, K. and Schneider, V. (1998) The Strange Case of Regulating Lobbying in Germany. Parliamentary Affairs 51(4): 559–567.Google Scholar
Rooduijn, M. and Pauwels, T. (2011) Measuring Populism: Comparing Two Methods of Content Analysis. West European Politics 34(6): 1272–1283.Google Scholar
Rosenson, B. A. (2003) Against Their Apparent Self-Interest: The Authorization of Independent State Legislative Ethics Commissions, 1973–96. State Politics & Policy Quarterly 3(1): 42–65.Google Scholar
Rush, M. (1998) The Canadian Experience: The Lobbyists Registration Act. Parliamentary Affairs 51(4): 516–523.Google Scholar
Seawright, J. and Collier, D. (2014) Rival Strategies of Validation: Tools for Evaluating Measures of Democracy. Comparative Political Studies 47(1): 111–138.Google Scholar
Stone, D. (2004) Transfer Agents and Global Networks in the Transnationalization of Policy. Journal of European Public Policy 11(3): 545–566.Google Scholar
Thomas, C. S. (1998) Interest Group Regulation Across the United States: Rationale, Development and Consequences. Parliamentary Affairs 58(4): 500–515.Google Scholar
True, J. and Mintrom, M. (2001) Transnational Networks and Policy Diffusion: The Case of Gender Mainstreaming. International Studies Quarterly 45(1): 27–57.Google Scholar
Veksler, A. (2015) Diluted Regulations: A Need to Review the Theoretical Classification of the Different Lobbying Regulatory Environments. Journal of Public Affairs 15(1): 56–64.Google Scholar
Wagenaar, A. C., Harwood, E. M., Silianoff, C. and Toomey, T. L. (2005) Measuring Public Policy: The Case of Beer Keg Registration Laws. Evaluation and Program Planning 28(4): 359–367.Google Scholar
Warhurst, J. (1998) Locating the Target: Regulating Lobbying in Australia. Parliamentary Affairs 58(4): 538–550.Google Scholar
Witko, C. (2007) Explaining Increases in the Stringency of State Campaign Finance Regulation, 1993–2002. State Politics & Policy Quarterly 7(4): 369–393.Google Scholar
Woll, C. (2012) The Brash and the Soft-Spoken: Lobbying Styles in a Transatlantic Comparison. Interest Groups & Advocacy 1(2): 193–214.Google Scholar
Yishai, Y. (1998) Regulation of Interest Groups in Israel. Parliamentary Affairs 51(4): 568–578.Google Scholar
Supplementary material: File

Crepaz and Chari supplementary material

Crepaz and Chari supplementary material

Download Crepaz and Chari supplementary material(File)
File 29.7 KB