Skip to main content
    • Aa
    • Aa

Assessing the validity and reliability of measurements when evaluating public policy

  • Michele Crepaz (a1) and Raj Chari (a2)

A substantial aspect of scientific research involves linking concepts to observations using measurements. This exercise has raised questions among researchers of whether or not measurements “truly” and “reliably” capture ideas and observations. We address this question by setting out a methodological standard on how to assess the validity and reliability of measurements. We do this by examining measurements that evaluate public policy, arguing that this topic is gaining increasing attention from political science researchers and policymakers. The analysis concerns measurements of the level of transparency and accountability of lobbying laws, central to recent regulatory policy research. We conduct convergent validation, content validation and reproducibility tests on four indices applied to 13 regulations found worldwide. By doing so, the article provides scholars with an evaluation of measurements of lobbying laws’ robustness, while offering methodological and theoretical lessons of value to larger regulatory and public policy scholarship.

Linked references
Hide All

This list contains references from the content that can be linked to their source. For a full set of references and notes please see the PDF or HTML where available.

R. Adcock and D. Collier (2001) Measurement Validity: A Shared Standard for Qualitative and Quantitative Research. American Political Science Review 95(3): 529546.

J. W. Bartlett and C. Frost (2008) Reliability, Repeatability and Reproducibility: Analysis of Measurement Errors in Continuous Variables. Ultrasound in Obstetrics and Gynecology 31(4): 466475.

K. Benoit , D. Conway , B. E. Lauderdale , M. Laver and S. Mikhaylov (2016) Crowd-Sourced Text Analysis: Reproducible and Agile Production of Political Data. American Political Science Review 110(2): 279285.

F. Boräng , R. Eising , H. Klüver , C. Mahoney , D. Naurin , D. Rasch and P. Rozbicka (2014) Identifying Frames: A Comparison of Research Methods. Interest Groups & Advocacy 3(2): 188201.

K. Bowman , F. Lehoucq and J. Mahoney (2005) Measuring Political Democracy: Case Expertise, Data Adequacy, and Central America. Comparative Political Studies 38(8): 939970.

M. Brining , R. Holcombe and L. Schwartzstein (1993) The Regulation of Lobbyists. Public Choice 77(2): 377384.

B. Clarke (1979) Eccentrically Contested Concepts. British Journal of Political Science 9(1): 122126.

D. F. Collier , D. Hidalgo and O. A. Maciuceanu (2006) Essentially Contested Concepts: Debates and Applications. Journal of Political Ideologies 11(3): 211246.

M. Crepaz (2016a) Investigating the Robustness of Lobbying Laws: Evidence from the Austrian Case. Interest Groups & Advocacy 5(1): 524.

W. B. Gallie (1955) Essentially Contested Concepts. Proceedings from the Aristotelian Society 56(1): 167198.

J. Greenwood (1998) Regulating Lobbying in the European Union. Parliamentary Affairs 51(4): 587599.

J. Greenwood and J. Dreger (2013) The Transparency Register: A European Vanguard of Strong Lobby Regulation. Interest Groups & Advocacy 2(2): 139162.

J. Greenwood and C. S. Thomas (1998) Introduction: Regulating Lobbying in the Western World. Parliamentary Affairs 51(4): 487488.

C. Holman and W. Luneburg (2012) Lobbying and Transparency: A Comparative Analysis of Regulatory Reform. Interest Groups & Advocacy 1(1): 75104.

R. J. Hrebenar , A. Nakainura and A. Nakamura (1998) Lobby Regulation in the Japanese Diet. Parliamentary Affairs 51(4): 551552.

G. Jordan (1998) Towards Regulation in the UK: From General Good Sense “to Formalized Rules”. Parliamentary Affairs 51(4): 524525.

K. Krippendorff (2004) Reliability in Content Analysis: Some Common Misconceptions and Recommendations. Human Communication Research 30(3): 411433.

R. Landis and G. Koch (1977) The Measurement of Observer Agreement for Categorical Data. Biometrics 33(1): 159174.

M. Lombard , J. Snyder-Dutch and C. Bracken Campanella (2002) Content Analysis in Mass Communication Assessment and Reporting of Intercoder Reliability. Human Communication Research 28(4): 587604.

D. Lowery and V. Grey (1997) How Some Rules Just Don’t Matter: The Regulation of Lobbyists. Public Choice 91(2): 139147.

J. Mahoney and G. Goertz (2006) A Tale of Two Cultures: Contrasting Quantitative and Qualitative Research. Political Analysis 14(3): 227249.

M. Maggetti and F. Gilardi (2016) Problems (and Solutions) in the Measurement of Policy Diffusion Mechanisms. Journal of Public Policy 36(1): 87107.

S. Mikhaylov , M. Laver and K. Benoit (2012) Coder Reliability and Misclassification in the Human Coding of Party Manifestos. Political Analysis 20(1): 7991.

G. L. Munck and J. Verkuilen (2002) Conceptualizing and Measuring Democracy. Comparative Political Studies 35(1): 534.

A. J. Newmark (2005) Measuring State Legislative Lobbying Regulation, 1990–2003. State Politics & Policy Quarterly 5(2): 182191.

C. Opheim (1991) Explaining the Differences in State Lobby Regulation. The Western Political Quarterly 44(2): 405421.

J. Ozymy (2010) Assessing the Impact of Legislative Lobbying Regulations on Interest Group Influence in US State Legislatures. State Politics & Policy Quarterly 10(4): 397420.

J. Ozymy (2013) Keepin’ on the Sunny Side: Scandals, Organized Interests and the Passage of Legislative Lobbying Laws in the American States. American Politics Research 41(1): 323.

P. Paxton (2000) Women in the Measurement of Democracy: Problems and Operationalization. Studies in Comparative International Development 35(3): 92111.

R. E. Rechtman and J. P. Larsen-Ledet (1998) Regulation of Lobbyists in Scandinavia: A Danish Perspective. Parliamentary Affairs 51(4): 579586.

P. Rocco and C. Thurston (2014) From Metaphors to Measures: Observable Indicators of Gradual Institutional Change. Journal of Public Policy 34(1): 3562.

M. Z. Rogers and N. Weller (2014) Income Taxation and the Validity of State Capacity Indicators. Journal of Public Policy 34(2): 183206.

K. Ronit and V. Schneider (1998) The Strange Case of Regulating Lobbying in Germany. Parliamentary Affairs 51(4): 559567.

M. Rooduijn and T. Pauwels (2011) Measuring Populism: Comparing Two Methods of Content Analysis. West European Politics 34(6): 12721283.

B. A. Rosenson (2003) Against Their Apparent Self-Interest: The Authorization of Independent State Legislative Ethics Commissions, 1973–96. State Politics & Policy Quarterly 3(1): 4265.

M. Rush (1998) The Canadian Experience: The Lobbyists Registration Act. Parliamentary Affairs 51(4): 516523.

J. Seawright and D. Collier (2014) Rival Strategies of Validation: Tools for Evaluating Measures of Democracy. Comparative Political Studies 47(1): 111138.

D. Stone (2004) Transfer Agents and Global Networks in the Transnationalization of Policy. Journal of European Public Policy 11(3): 545566.

J. True and M. Mintrom (2001) Transnational Networks and Policy Diffusion: The Case of Gender Mainstreaming. International Studies Quarterly 45(1): 2757.

A. Veksler (2015) Diluted Regulations: A Need to Review the Theoretical Classification of the Different Lobbying Regulatory Environments. Journal of Public Affairs 15(1): 5664.

A. C. Wagenaar , E. M. Harwood , C. Silianoff and T. L. Toomey (2005) Measuring Public Policy: The Case of Beer Keg Registration Laws. Evaluation and Program Planning 28(4): 359367.

C. Witko (2007) Explaining Increases in the Stringency of State Campaign Finance Regulation, 1993–2002. State Politics & Policy Quarterly 7(4): 369393.

C. Woll (2012) The Brash and the Soft-Spoken: Lobbying Styles in a Transatlantic Comparison. Interest Groups & Advocacy 1(2): 193214.

Y. Yishai (1998) Regulation of Interest Groups in Israel. Parliamentary Affairs 51(4): 568578.

Recommend this journal

Email your librarian or administrator to recommend adding this journal to your organisation's collection.

Journal of Public Policy
  • ISSN: 0143-814X
  • EISSN: 1469-7815
  • URL: /core/journals/journal-of-public-policy
Please enter your name
Please enter a valid email address
Who would you like to send this to? *


Type Description Title
Supplementary Materials

Crepaz and Chari supplementary material
Crepaz and Chari supplementary material

 Unknown (30 KB)
30 KB


Full text views

Total number of HTML views: 6
Total number of PDF views: 61 *
Loading metrics...

Abstract views

Total abstract views: 444 *
Loading metrics...

* Views captured on Cambridge Core between 9th January 2017 - 19th September 2017. This data will be updated every 24 hours.