Quality of legislation and compliance: a natural language processing approach

Several disciplines, such as economics, law, and political science, emphasize the importance of legislative quality, namely well-written legislation. Low-quality legislation cannot be easily implemented because the texts create interpretation problems. To measure the quality of legal texts, we use information from the syntactic and lexical features of their language and apply these measures to a dataset of European Union legislation that contains detailed information on its transposition and decision-making process. We find that syntactic complexity and vagueness are negatively related to member states ’ compliance with legislation. The finding on vagueness is robust to controlling for member states ’ preferences, administrative resources, length of texts, and discretion. However, the results for syntactic complexity are less robust.


Introduction
High-quality legislation is written simply, clearly, concisely, and unambiguously (Butt, 2013;Xanthaki, 2014).Well-written legislation is more predictable; it allows policy-makers to specify clear regulatory messages, which are necessary to implement regulatory goals, and ensures that bureaucrats understand the legislation correctly (Xanthaki, 2014).National and international institutions such as the Italian Senate of the Republic and the European Union (EU) incorporate these standards into their guidelines for legal drafters (e.g., Senato della Repubblica, 2001;European Union, 2015).
In this paper, we study how legislative quality affects compliance with EU directives, which specify goals that EU member states must transpose into national law.Compliance refers to the correct and timely transposition of directives.High-quality directives reduce the chances that legal documents will be misinterpreted as well as the costs of transposition (e.g., Chayes and Chayes, 1995;Börzel, 2021).We propose two natural language processing (NLP) measures of legislative quality and apply them to a dataset on EU legislation.We find that syntactic complexity is negatively related to compliance.However, this relationship is not robust to controlling for the length of the legislative texts, as syntactically complex legislation tends to be long.Furthermore, our analysis reveals that vague legislation is less likely to be correctly transposed.Importantly, our results suggest that vagueness has an effect even after controlling for important drivers of compliance, such as member states' preferences, and administrative resources, as well as the length of the texts.
The case of the EU is particularly relevant for two reasons.First, good data and measures are available.We build upon the data of König and Mäder (2014), who worked with graduate law students to assess the extent to which member states' national legislation complies with EU directives.Hence, our data do not rely on reports of the European Commission or member states, which are incomplete and can thus bias the results.The data also include detailed information on EU legislation, including about member states' preferences, amongst others (Thomson et al., 2006).Second, the quality of legislation is especially important at the EU level because legislation needs to be transposed in 27 member states with different legal traditions and institutions.Hence, EU institutions invest significant resources in monitoring legislative quality.For example, the Legal Service of the European Commission has an entire directorate working on the quality of legislation (directorate K).
We make two contributions to the literature.First, we provide novel evidence of how the legislative quality shapes compliance with legislation.Qualitative research provides evidence that the quality of legislation is crucial for understanding how specific directives are transposed in the EU (Falkner et al., 2005), which is in line with the assessment of legal experts (Barrata, 2014).Prior quantitative studies often acknowledge the importance of legislative quality, but ours systematically analyzes the explanatory power of this factor (Tallberg, 2002;Mastenbroek, 2003;Borghetto et al., 2006;Börzel, 2021).
Second, we propose two quantitative measures of legislative quality, which complement previous measures.Prior theoretical work emphasizes the importance of legislative quality (e.g., Hirsch and Schotts, 2015;Hitt et al., 2017;Gratton et al., 2021) but most previous empirical analyses rely on the number of words or pages in their measurement (Gratton et al., 2021). 1 We instead use information on the syntactic and lexical features of the legal texts to measure legislative quality. 2

Explaining compliance with legislation
Two main approaches explain compliance with legislation.First, rational or strategic explanations highlight the role of preferences and strategic incentives (Tallberg, 2002;Carrubba, 2005;Thomson et al., 2007;König and Luetgert, 2009).König and Luig (2014) suggest that noncompliance is a strategic choice of member states, particularly of the minister in charge of transposing the directive.Related explanations concentrate on the power of member states and national institutions as they highlight a rational explanation for non-compliance (Börzel, 2021).Multiple papers highlight the important role of domestic factors in influencing incentives to comply (e.g., Dai, 2005).
Second, public administration approaches emphasize administrative capacity and efficiency (e.g., Chayes and Chayes, 1995).According to this perspective, some states have more efficient or capable administrations, which increases their compliance with international law.Nordic member states are thought to have more efficient administrations, and Southern European states are generally considered less efficient.Administrative capacity might also depend on the adaptation pressures resulting from EU legislation (e.g., Knill, 1998).Finally, legislative quality and policy complexity may decrease states' capacity to implement directives (Zhelyazkova, 2013;Börzel, 2021).
In this paper, we focus on the relationship between legislative quality and compliance in the EU.The quality of legislation has been on the EU agenda since the Birmingham declaration of 1992, when the European Council highlighted the need to make legislation "simpler and clearer" (European Council, 1992, 5).Multiple Council meetings and inter-institutional agreements have since reiterated the importance of legislative quality.The European Commission, the European Parliament, and national administrations have legal services to monitor the quality of legal documents.Appendix A further discusses the role of legislative quality in the EU.

1
We acknowledge that the language is only one aspect of legislative texts.Recent work, for instance, has used a network approach, by looking at the citation network of laws (e.g., Giommoni et al., 2022;Senninger, 2023).

2
A related literature studies policy complexity and uses text readability measures, word entropy, references, the number of recitals, and the number of words as measures (e.g., Zhelyazkova, 2013;Hurka and Haag, 2020).Our focus does not lie on the complexity of the policy as such but on the drafting quality.
Legislative quality can influence compliance via two mechanisms (e.g., Falkner et al., 2005).First, low-quality legislation may delay the transposition of EU law in member states due to interpretation problems.National administrators may need more time to transpose legislation that is unclear and difficult to interpret.For example, national administrators may need to coordinate with the Commission to transpose the directive.Second, low-quality legislation can lead to incomplete transposition. 3  We therefore expect low-quality legislation to be associated with low levels of compliance: Hypothesis: The lower the quality of the legislation, the less likely it is to be correctly transposed.

Measuring legislative quality
We measure the legislative quality of 21 directives by focusing on two features of legislative texts -syntactic complexity and vagueness.We use two NLP methods that work well for small corpora.Appendix D assesses the construct validity of the measures by examining a case study and illustrating the procedure using example sentences.In Appendix F, we illustrate the variation of these variables by policy area.

Syntactic complexity
In line with recent political economy and political science research (Vannoni et al., 2020;Ash et al., 2022), we look at the syntactic features of the text to extract important information.We measure syntactic complexity by focusing on the modifiers of nouns, which make legal texts more complex because it is harder for the reader to identify the subject-verb-object structure (e.g., Xanthaki, 2014, chapter 5).
Our measure of complexity indicates the percentage of words that modify nouns.We therefore focus on adjectival modifiers that modify the meaning of a noun or a pronoun.An example of a sentence with many modifiers (in italics) is "Particularly cruel actions, even if committed with an allegedly political objective, may be classified as serious non-political crimes" (see Table A1 for more examples).These modifiers make the sentence longer and more difficult to read and understand.
We calculate the number of modifiers using a dependency parser (the python module spaCy) to extract the syntactic dependencies that link the words and phrases in the text of the legislative acts in our sample (Honnibal and Montani, 2017).The parser is trained on annotated syntactic dependencies, so-called dependency treebanks (Goldberg and Nivre, 2012;Honnibal and Johnson, 2015).For example, in the sentence "the Member State shall enforce the act", the dependency parser identifies the following syntactic dependencies: determiner "the", compound "Member", nominal subject "State", auxiliary "shall", root "enforce", determiner "the", and direct object "act."

Vagueness
Words are vague when their boundaries are unclear (Xanthaki, 2014).Legal writing experts recommend using concrete rather than vague words because the latter "tend to make sentences ineffective" (Garner, 2002, 31).We measure vagueness using the Linguistic Inquiry and Word Count (LIWC) dictionary (Pennebaker et al., 2015), which includes 178 vague words and (or word 3 We provide more details on the infringement procedure in Appendix B. Note that the quality of legislation can be partly a result of bargaining in the EU decision-making process (e.g., Falkner et al., 2005;Borghetto et al., 2006).The Council and Parliament must approve the proposal and can make amendments (e.g., Osnabrügge, 2015).The bargaining process can thus generate more complex and vague legislative texts due to the divergence of preferences across EU institutions.Hence, we study the influence of legislative quality on compliance holding preference divergence constant.stems) (e.g., "approximat*", "may", "or", "perhaps", and "unknown") and 113 concrete words (e.g., "all", "everyone*", "must", "never", and "totally").
Our measure equals the percentage of vague minus the percentage of concrete words.Previous political science studies have used it to measure the vagueness of party manifestos (Eichorst and Lin, 2019), and the LIWC has been extensively validated across different corpora (Pennebaker et al., 2015, 8). 4 For example, our measure assesses the following sentence as very vague because it includes three vague words (in italics) and zero concrete words: "Such sharing or coordination arrangements may include rules for apportioning the costs of facility or property sharing."

Data
To analyze how legislative quality influences compliance, we build upon the data from König and Mäder (2014), who focus on directives studied by the "Decision making in the European Union" (DEU) project (Thomson et al., 2006).Their data include information on 299 compliance cases related to 21 directives adopted between 1999 and 2003 when the EU had 15 member states.These directives were not affected by procedural changes and were politically important (Thomson et al., 2006, 26).
Our dependent variable equals 1 if a country transposed all the goals of the directive into national law in a timely manner, and 0 otherwise.This measure focuses on the issues covered by the DEU project (König and Mäder, 2014, 251).
König and Mäder's dataset has two advantages.First, they employed graduate law students from the different member states to identify cases of non-compliance.These students studied the text of national legislation and determined the extent to which it implements the directive's goals (König and Mäder, 2014, 251).This approach overcomes a common problem in compliance research that the Commission and member states do not report all cases of non-compliance (see also Zhelyazkova and Yordanova, 2015)-which can bias the analysis.A second advantage is that their data include legislation-specific data on actors' preferences, which helps control for alternative explanations.
We expand this dataset by incorporating the legal texts and our two measures of legislative quality-syntactic complexity and vagueness.Figure 1 (panel a) displays the density of the variable syntactic complexity; the dashed vertical line represents the mean value of the variable.The figure illustrates that legal acts contain, on average, 6.6 percent noun modifiers.Panel (b) displays the density of the variable vagueness.The vertical dashed line represents the mean value of the variable, 1.9, which indicates that the percentage of vague words is 1.9 points higher than the percentage of concrete words.The minimum value of this variable is 0, and the maximum value is 4.6.
We consider multiple control variables that capture factors that might be related to both compliance and the quality of directives.Following König and Mäder (2014), our analysis controls for preferences, administrative capacity, interest group patterns, and the type of directive.Furthermore, we take into account the length of texts, amending legislation, government changes, and member state's discretion.
First, we include variables on preferences computed from the DEU project data (Thomson et al., 2006), which is based on expert surveys.Member states' disagreement is measured as the distance between each country's ideal point and the outcome of the decision-making process.If a legislative act has multiple issues, the distances for all issues are aggregated.The diversity of member states' preferences related to a directive is measured as the sum of standard deviations of their preferences for each issue.
Second, we use a variable on administrative resources, which relies on the index of Kaufmann et al. (2006).We also control for the corporatist interest group system using Lijphart's index of interest group pluralism (Lijphart, 1999): high values referring to member states with a pluralistic interest group system.We create an indicator variable that equals 1 if the Council and European Parliament adopted the proposal, and 0 if only the Council adopted it.
Furthermore, we control for the number of words in a legislative act and create an indicator variable on amending legislation.We also include an indicator variable for government changes during the period between the law's adoption and the implementation deadline.
Finally, we measure the delegation of authority to member states using two measures.More specifically, we use the discretion index computed by Anastasopoulos and Bertelli (2020), and as an alternative a measure of the delegation ratio (Thomson et al., 2007).Appendix F provides descriptive statistics of all variables.

Results
We use descriptive statistics and logistic regression models to study the relationship between legislative quality and compliance with EU legislation at the national level.First, we inspect the correlation between the percentage of countries that correctly implemented the directive and our measures of legislative quality.Second, we run logistic models that include both explanatory variables (syntactic complexity and vagueness).Our preferred model controls for preference divergence among member states, member states' disagreement with the policy, administrative resources, interest group patterns, type of directives, length of texts, amending legislation, government changes, and discretion.Appendix G includes the complete regression outputs.Third, we implement further analysis and robustness tests.
Figure 2 displays the relationship between legislative quality and the percentage of countries correctly implementing the directives.Each dot refers to one of the 21 directives.As panels (a) and (b) illustrate, the correlation between the percentage of countries implementing the directive and syntactic complexity is −0.37.The correlation equals -0.45 when examining the percentage of countries that correctly implemented the directive and vagueness.The evidence suggests that less well-written legislation tends to be correctly implemented in fewer countries.
We continue analyzing the effects using a logistic regression model.In Figure 3, we present the average marginal effects of syntactic complexity and vagueness estimated using a model without control variables and our preferred model that includes all control variables.The average marginal effect captures how much a one-unit change in a variable influences the predicted probability, on average, holding the other variables at their default values (Arel-Bundock, 2023).Figure 3 displays the effects and the 95 percent confidence intervals.In line with Figure 2, panel (a) shows that both complexity and vagueness are negatively related to compliance.Both effects are statistically significant at the 0.05 level.However, panel (b) illustrates that the average marginal effect of the syntactic complexity variable is not robust to including all control variables.On closer inspection, this occurs when controlling for the length of texts as this measure is correlated with syntactic complexity.In models including all control variables except length, the effect of syntactic complexity is statistically significant (see Table A5, column 2).Panel (b) also indicates that a one-unit increase in the ambiguity dictionary scale decreases the probability of compliance by 0.08.This average marginal effect is statistically significant at the 0.01 level.In line with this result, the probability decreases by 0.11 if the vagueness score increases from the 20th to the 80th percentile, which corresponds to a change in the vagueness variable from 1.13 to 2.50.
We also study the interaction effects between disagreement with the policy and our measures of legislative quality to understand whether the effect is stronger for member states that oppose the legislation.We find that the interaction effects are not statistically significant, which suggests that vagueness tends to reduce the probability of compliance regardless of whether states opposed the legislation.To confirm this result, we demonstrate that our result holds when we re-run the analysis on the ten least controversial directives (see Appendix H for details).A5) and panel (b) is based on a model with control variables (model 3 in Table A5).
We assess the robustness of our results using various tests, which are described in Appendices G and H. First, we incorporate an alternative measure of delegation (Thomson et al., 2007).Second, we use a different pre-processing approach.In the main text, we report the results on the corpus of texts excluding recitals and citations, which come before the provisions and are not legally binding.In Appendix G, we run the same analysis but include the recitals and citations.Third, we re-run the models with time-fixed effects.Fourth, we add additional control variables on time-varying factors.Finally, we re-run the models clustering the standard errors by legislation.The main results are robust to all of these tests.
In sum, our analysis establishes that syntactic complexity is negatively related to compliance, but that this effect is not robust to controlling for the length of texts.However, vagueness matters for compliance, even when controlling for other potential explanatory factors.

Concluding remarks
We study how legislative quality shapes compliance, focusing on the transposition of EU directives.High-quality legislation is expected to increase compliance because it reduces interpretation costs and misunderstandings.We propose two NLP measures of legislative quality-syntactic complexity and vagueness.Our analysis builds on the data of König and Mäder (2014), which include detailed information on compliance cases.
The paper makes two main contributions.First, it presents quantitative evidence of how legislative quality affects compliance with EU directives.Previous qualitative research has increased our understanding (e.g., Falkner et al., 2005), but our systematic quantitative evidence controls for alternative explanations.In a second contribution, the paper introduces two simple NLP measures of legislative quality that examine the language of the legislative texts.These measures complement prior research, which focused on the length of sentences and the number of provisions or other formal features (e.g., Gratton et al., 2021).We use the case study of the EU, where compliance is a key part of the rule of law.
Our analysis reveals that the quality of legislation, measured as syntactic complexity and vagueness, is positively related to compliance with EU directives.The results for the former are not robust to controlling for the length of legislative texts since longer texts tend to be syntactically more complex.However, the effect of the latter is robust when including control variables on member states' preferences, administrative resources, interest group structure, type of directive, length of texts, amending legislation, and government changes, as well as discretion.
We hope our paper encourages further work on the causes and consequences of legislative quality.Future research may apply unsupervised and supervised learning techniques to larger corpora to develop new measures of legislative quality.In addition, theoretical work may study the extent to which strategic interaction between actors influences the quality of legislation.Furthermore, it would be promising to evaluate the consequences of legislative quality in different political systems.

Figure 1 .
Figure 1.Density of explanatory variables, (a) syntactic complexity, (b) vagueness.Notes: The figures illustrate the density of the two measures of legislative quality.The vertical dashed lines denote the mean values.The rug plot at the bottom illustrates the data distribution.

Figure 2 .
Figure 2. Percentage of countries that implemented the directive and legislative quality, (a) syntactic complexity, (b) vagueness.Notes:The figures show the correlation between the measures of legislative quality (x-axis) and the percentage of countries complying with a legislation (y-axis).Lines of best fit are computed based on OLS regression models.

Figure 3 .
Figure 3. Average marginal effects, (a) without control variables, (b) with control variables.Notes: The figures show the average marginal effects of syntactic complexity and vagueness and 95 percent confidence intervals.Panel (a) illustrates the effects of a model without control variables (model 1 in TableA5) and panel (b) is based on a model with control variables (model 3 in TableA5).