Introduction
In the field of legal and political philosophy, the rule of law is recognized as an “essentially contested” concept, as described by Jeremy Waldron (Reference Waldron2002). The definition of the rule of law is inherently subject to debate, with enduring and substantial disagreements regarding the concept’s appropriate application. While most of the debates about the definition of the rule of law are scholarly in nature,Footnote 1 the challenges posed by the conceptual contestability of the rule of law also manifest themselves in practical applications, particularly within development programs. Substantial resources are invested in promoting and institutionalizing particular interpretations of the rule of law in development and governance programs (Erbeznik Reference Erbeznik2011). A recurring concern in this literature is that externally promoted understandings of the rule of law may not align with how the concept is articulated and contested by political actors within target countries. Such mismatches may limit the effectiveness of reforms and, in some cases, generate unintended consequences (Kleinfeld Belton Reference Kleinfeld Belton2005; Carothers Reference Carothers2006). To achieve social and economic development goals through the rule of law, it is thus crucial to work toward a better understanding of what the rule of law means for different audiences or in specific contexts.
Some of the most crucial disagreements around the definition of the rule of law focus on the predominance of thin (formal/procedural) and thick (substantive) interpretations of this concept (Craig Reference Craig and Bellamy2017). In simple terms, the disagreement is about whether the rule of law means only fair legal procedures (a thin, procedural approach) or whether it must also guarantee human rights (a thick, substantive approach). This distinction between thin and thick conceptualizations of the rule of law is crucial to this article as it highlights the multifaceted nature of the rule of law, which comprises several discernible layers (Bedner Reference Bedner2010). What is crucial is that these layers carry varying degrees of significance across different legal traditions, as evidenced in various legal philosophical debates (Loughlin Reference Loughlin and Loughlin2010; Meierhenrich Reference Meierhenrich, Meierhenrich and Loughlin2021). For instance, while the common law tradition emphasizes the procedural aspects of the rule of law along with the importance ascribed to the judiciary, the German (especially after the Second World War) and French legal traditions are more compatible with the thick conceptualization of the rule of law as they underscore the human rights aspects.
Against this background, this article first aims to provide a proof of concept for measuring the meaning and understanding of the rule of law across countries or different contexts more generally. Its second goal is to offer a principled way to derive the weights for the rule-of-law indicators to enable a reevaluation of country rankings in terms of adherence to the rule-of-law principles. The goal of the latter approach is to test the sensitivity of existing indicators—that is, to examine whether country rankings would change if certain components of the rule of law were emphasized while others were de-emphasized.
This article’s innovation stems from applying cutting-edge computational social science methodology—namely, word embeddings (vectors) to capture the meaning of the rule of law. Word embeddings model semantic relationships between words by representing them as vector relationships in a high-dimensional space (Mikolov et al. Reference Mikolov, Chen, Corrado and Dean2013a, Reference Mikolov, Sutskever, Chen, Corrado, Dean, Burges, Bottou, Welling, Ghahramani and Weinberger2013b). The distance between vectors in this high-dimensional space captures the semantic similarity of words. By measuring and contrasting similarity across word embeddings trained on texts from different jurisdictions, we can get a representation of the meaning of the rule of law across different countries. In practical terms, this means that we can see how close or distant the rule-of-law term is from its constituent (procedural and substantive) dimensions. In other words, this technique lets us see which ideas of the rule of law are talked about together most often. Furthermore, the similarity scores of word embeddings can be used as weights assigned to constituent parts of the rule-of-law indicators and, therefore, provide a principled way with which to calculate such indicators.
To model word embeddings, we use corpora of parliamentary debates in selected countries with different legal traditions, such as common law, French, German, and post-socialist traditions. Our choice to focus on parliamentary debates is driven by the observation that the rule of law is frequently invoked, contested, and politicized in legislative discourse, even when it is not formally defined in statutory or constitutional texts. We recognize that in some jurisdictions—particularly, common law systems—judicial opinions constitute an important site of rule-of-law reasoning. Parliamentary debates, however, remain a central and comparable arena in which political elites articulate, contest, and strategically deploy the concept. Furthermore, the rule of law is a concept of state governance that is not only legal but also philosophical and political in nature. Martin Loughlin (Reference Loughlin and Loughlin2010, 336) explicitly refers to the rule of law as a political concept that aims to specify “the conditions of legitimate political rule.” Therefore, analyzing how political leaders in parliaments discuss and interpret the rule of law offers an appropriate methodological approach (for a similar approach, see Schwartz Reference Schwartz2022).
It is of note that this article is explicitly descriptive in nature. It examines how the concept of the rule of law is associated with particular procedural and substantive elements in parliamentary discourse. The aim is to capture patterns in elite-level political language rather than societal consensus or normative correctness of the rule of law. The analysis does not evaluate the desirability of specific conceptualizations but maps discursive emphases and explores how these can be applied to existing measurement frameworks. The article proceeds as follows. The next section briefly accounts for various legal traditions toward the rule of law. Based on this, we derive a set of conjectures as to what conceptualization of the rule of law should be prevalent across selected countries. We then follow with sections on data and methodology as well as results. The latter provides a proof-of-concept application of word embeddings to study the semantic meaning of the rule of law across jurisdictions and provides a novel way for (re-)calculating the rule-of-law indicators in order to examine their sensitivity. The last section concludes the article and provides an outlook for future research.
Rule of law and different legal traditions
The English and German legal traditions are often credited with first defining the concept of the rule of law. According to the English (common law) tradition, the purpose of the rule of law is to constrain—or, as Martin Krygier (Reference Krygier2019, Reference Krygier2024) puts it, to temper—the unlimited power of the state over the individual. The English legal tradition assigns a significant role to the judiciary, which is tasked with creating common law (judge-made law), which contrasts with other legal traditions where the rule of law stems from written (de jure) constitutions. Some equate such a judge-centered approach toward the rule of law with “rule by reason” (Loughlin Reference Loughlin and Loughlin2010, 317). The crucial role of the judiciary in conceptualizing the rule of law is also emphasized by Joseph Raz (Reference Raz1979), who highlights judicial independence as a precondition for impartial governance of rules. Beyond the judge-made law aspect, the English approach to the rule of law, particularly as it is conceptualized by Alfred Dicey (Reference Dicey1915), emphasizes two other procedural components. First, individuals can only be punished for breaches of law that must be certain and prospective. Second, the law must apply equally to all, including those in power, who are also subject to the law. Overall, in the English (common law) tradition, although the rule of law is recognized for its moral value, such as constraining and tempering the executive, its primary focus is procedural. In this vein, the judiciary and the quality of the rules receive special attention, while substantive aspects, such as the protection of human rights, are not as prominently emphasized.
The German tradition of the rule of law—Rechtsstaat—stands in stark contrast to the common law tradition, particularly in terms of its origins and initial conceptual emphasis. While the goal of the rule of law in the English tradition is to limit the power of the executive through judge-made laws, the Rechtsstaat tradition prioritizes the state and the rule of law, emanating from the written constitution as a mode of governance, whereby people’s lives are regulated by laws enacted by the legislature. As such, initially, the Rechtsstaat concept in Germany was closer to a “rule by law” (rule by means of law) or “state of reason” approach (Loughlin Reference Loughlin and Loughlin2010, 318), whereby the qualities inherent in the concept of law were most cherished. The Rechtsstaat conceptualization resonates well with Lon Fuller’s (Reference Fuller1969, 106) approach to the rule of law, characterized as “the enterprise of subjecting human conduct to the governance of rules.” According to this logic, the governments adhere to the rule of law as defined because it is in their own interest. Such governance by means of law anchors citizens’ expectations and fosters their allegiance to the state, which by itself is power enhancing (Loughlin Reference Loughlin and Loughlin2010, 340). However, the concept underwent a significant transformation following the Second World War (and Germany’s postwar experience). Substantive aspects, particularly those related to human rights, gained prominence and became perceived as integral to the rule of law. In summary, although originally procedural (thin) at its inception, the Rechtsstaat approach is now characterized by a blend of procedural and substantive aspects of the rule of law, thus embodying a more comprehensive (thick) conceptualization of the rule of law.
The transformed (thick) conceptualization of the Rechtsstaat echoes the French (État de droit) approach to the rule of law. From the outset, the French tradition emphasized the state’s role as the guarantor of constitutionally embedded fundamental rights (as enshrined in the 1789 Declaration of the Rights of Man and of the Citizen) against encroachment by those in power. A crucial role in this system is played by the judiciary, which is tasked with the constitutional review of ordinary legislation. This brief description suggests that the rule of law in the French legal tradition is inherently thick as substantive aspects are prominent. Regarding the procedural dimension, it is likewise important to highlight the judiciary’s significant role in this tradition.
The French tradition of the rule of law starkly contrasts with the socialist or Russian (pravovoe gosudarstvo) approach to the rule of law. In the latter, the state is central, and the law serves primarily as an instrument for projecting power. Within this framework, the rule of law is not valued for its own sake but, rather, is seen as a mode of governance where the state rules by law and laws are expected to be strictly enforced. This approach is often referred to as “socialist legality,” where procedural aspects are predominant (Venice Commission 2011), and it resembles the “rule-under-law” tradition, where law is nothing else than an expression of the state’s will (Palombella Reference Palombella, Gianluigi and Neil2009). Some could deduce from this discussion that such a defined post-socialist tradition provides a fertile ground to populist governments in countries such as Hungary and Poland, which use the law to legitimatize their rule and dismantle the rule of law and democracy more broadly. Kim Scheppele (Reference Scheppele2018, 548) characterized such tactics as autocratic legalism, which is defined as the use of “electoral mandates plus constitutional and legal change … in the service of an illiberal agenda.”
Based on this succinct account of various legal traditions toward the rule of law, we derive the following conjectures, which we put to test in the following sections:
-
• Conjecture 1: Countries adhering to common/English law tradition will stress procedural (quality of rules and judiciary) aspects of the rule of law.
-
• Conjecture 2: Countries adhering to German law (Rechtsstaat) tradition will stress procedural (especially quality of rules) and substantive (human rights) aspects of the rule of law.
-
• Conjecture 3: Countries adhering to French law (État de droit) tradition will stress procedural (especially judiciary) and substantive (human rights) aspects of the rule of law.
-
• Conjecture 4: Countries adhering to (post-)socialist tradition will stress procedural (quality of rules and judiciary) aspects of the rule of law.
It is further important to note that the conceptualizations of the rule of law are not static but, rather, subject to constant reinvention (Meierhenrich Reference Meierhenrich, Meierhenrich and Loughlin2021, 66). Although histories of the rule of law vary across legal traditions, some argue that, in the course of the twentieth century and even more so the twenty-first century, we observe a convergence of conceptual approaches to the rule of law (66). If that is indeed the case, we should expect minor, if any, differences in the rule-of-law conceptualizations among countries.
Data and methods
From the discussion presented, it is clear that the rule of law encompasses various dimensions, with different legal traditions emphasizing varying procedural (formal) and substantive elements as crucial to the rule-of-law understanding. In this article, we do not commit to a single definitive concept of the rule of law. Instead, we draw on parliamentary discourse about the rule of law, acknowledge its multifaceted nature, and empirically test conjectures derived from different legal traditions that emphasize distinct components and characteristics of the rule of law. Specifically, we distinguish between elements pertaining to a procedural (thin) conceptualization (Fuller Reference Fuller1969; Tamanaha Reference Tamanaha2004) and those relating to a substantive (thick) conceptualization of the rule of law (Bedner Reference Bedner2010; Bingham Reference Bingham2011). Within the procedural elements, we separate those that pertain to the quality of the legal rules and the legal system at large from those concerning the judiciary and judicial system.
Regarding the substantive elements, we equate them with those related to fundamental and other rights. This approach consequently allows us to define three conceptual dimensions of the rule of law to analyze across jurisdictions: (1) procedural rules; (2) procedural judiciary; and (3) substantive rights dimensions. In our previous work, we furthermore specifically differentiated the substantive democracy dimension within the rule of law (Kantorowicz and González-Bustamante Reference Kantorowicz and González-Bustamante2025a). However, for the purpose of this article, we have chosen to omit this dimension as none of the legal traditions we examine conflate the rule of law with democratic principles.
Table 1 presents n-grams (terms) used to operationalize each of the three dimensions of the rule of law. This list was validated by ten legal and rule-of-law experts. Besides validating our choices, the experts have expanded the initial set of terms (n-grams). While, in the main body text, we present the results for the initial and validated list of terms, in the Appendix, we present the results for an extended list of terms. The terms for the procedural rules dimension were mainly chosen according to Fuller’s (Reference Fuller1969) definition of the rule of law, which outlines eight desired characteristics of law;Footnote 2 our list includes these and some closely related synonyms, nouns, and adjectives. The procedural judiciary dimension uses n-grams frequently associated with the qualities of the judicial system. For the substantive rights dimension, we selected commonly occurring n-grams that encompass various rights.
The rule of law attributes (dimensions) and the corresponding n-grams
To provide for a more focused test of conjectures, we selected eight country cases for the baseline empirical analysis. All chosen countries are either current or former European Union (EU) members. Such regional focus is dictated by the fact that the rule of law is of central value for the EU, and the EU by itself engages in the promotion of a particular vision of the rule of law. By focusing on the EU member states, we keep this aspect constant across analyzed cases. The countries chosen for the analysis and the legal traditions to which they adhere are presented in Table 2. Consistent with Rafael La Porta, Florencio Lopez-de-Silanes, and Andrei Shleifer (Reference La Porta, Lopez-de-Silanes and Shleifer2008), we classify France, Italy, and Spain within the French legal tradition, Great Britain within the common law tradition, and Germany within the German legal tradition. While La Porta, Lopez-de-Silanes, and Shleifer (Reference La Porta, Lopez-de-Silanes and Shleifer2008) group Poland and Hungary together under the German legal tradition, we observe that both of these countries also exhibit post-socialist characteristics. Furthermore, we classify the Netherlands as part of the German legal tradition, as the term Rechtsstaat is also prevalent in Dutch. This classification differs from La Porta, Lopez-de-Silanes, and Shleifer (Reference La Porta, Lopez-de-Silanes and Shleifer2008), who categorize the Netherlands with countries of French legal origin.
Country cases selected for the analysis
We used the ParlaMint 4.0 dataset that comprises parliamentary debates of twenty-six European countries and three autonomous regions from Spain between the 1990s, depending on the case, and 2023 (Erjavec et al. Reference Erjavec, Kopp, Ogrodniczuk, Osenova, Agirrezabal, Agnoloni and Aires2023b). The dataset not only offers the original speeches in plenary sessions from the relevant legislative bodies but also corpora linguistically annotated in the original languages and machine-translated texts (Erjavec et al. Reference Erjavec, Kopp, Ogrodniczuk, Osenova, Agerri, Agirrezabal and Agnoloni2023a; Kuzman et al. Reference Kuzman, Ljubešić, Erjavec, Kopp, Ogrodniczuk, Osenova and Rayson2023). The latter constitutes a valuable resource for our empirical approach and cross-lingual validation. Despite the advantages of the ParlaMint dataset, one considerable drawback is that they did not offer data for Germany, which is the flagship case of the Rechtsstaat tradition. To address this issue, we complemented ParlaMint with Open Discourse data to have German speeches in the original language (Richter et al. Reference Richter, Koch, Franke, Kraus, Kuruc, Thiem, Högerl, Heine and Schöps2020). Our empirical strategy is based on natural language processing (NLP) and a text-as-data approach, specifically word vectors. In this context, we relied on cross-lingual word representations in both monolingual and bilingual word embeddings (BWEs). The latter implies shared embedding space for words across two languages, while multilingual word embeddings apply to more than two languages (Klementiev, Titov, and Bhattarai Reference Klementiev, Titov, Bhattarai, Kay and Boitet2012; Chen and Cardie Reference Chen, Cardie, Riloff, Chiang, Hockenmaier and Tsujii2018).Footnote 4
In the case of BWEs, it is usual to map two different languages using a fixed target, generally English (Chen and Cardie Reference Chen, Cardie, Riloff, Chiang, Hockenmaier and Tsujii2018; Lample et al. Reference Lample, Conneau, Ranzato, Denoyer and Jégou2018). This process involves machine translation to the target language; therefore, the implementation could be called a BWE pivot, while learning embeddings without translation by using languages of the same family—for example, German and Dutch, which are both West Germanic languages—is BWE-direct implementation (Chen and Cardie Reference Chen, Cardie, Riloff, Chiang, Hockenmaier and Tsujii2018). Although using a fixed language as a pivot should guarantee cross-lingual comparison, there are some drawbacks. For example, BWE-pivot applications tend to be costly in terms of time and resources. However, a direct application between two similar languages could take advantage of linguistic interdependence and improve performance.
We applied monolingual and BWE applications to eight countries of ParlaMint and Open Discourse data between 2015 and 2022—namely, Germany, Spain, France, Great Britain, Hungary, Italy, Netherlands, and Poland. This selection comprises around 2.6 million speeches, almost 380 million words for the original languages and around 368 million for the machine-translated corpora, with the exception of Germany, for which we only have monolingual corpora. The translation was performed using Opus-MT models based on the Marian Neural Machine Translation (NMT) framework (Tiedemann and Thottingal Reference Tiedemann, Thottingal, Martins, Moniz, Fumega, Martins, Batista, Coheur and Parra2020). In general, the quality of machine translation was between average and fluent (between forty and sixty in its Bilingual Evaluation Understudy [BLEU] score), with some languages standing out with quality that is probably better than an average human translator (above sixty in its BLEU score). Further details about the number of utterances and words per country and translation indicators are available in the Appendix.
We pre-processed the corpora by applying tokenization. The original corpora were processed with specific Spanish, French, Italian, Dutch, and Polish tokenizers based on pre-trained Punkt models (Kiss and Strunk Reference Kiss and Strunk2006). Hungarian was processed using a generic multilingual Punkt tokenizer. We also applied lowercase conversion and removed punctuation, symbols, and spaces. Then, we applied a number of recodifications for relevant multi-word collocations and typo corrections in English. Our recodification list consisted of 222 changes.Footnote 5 We opted for a monolingual implementation and some BWE-pivot applications using machine-translated BWEs and Procrustes-aligned BWEs with English as an anchor. This alignment implied an orthogonal matrix alignment to line up different data slices to the chosen anchor (Rodman Reference Rodman2020). Procrustes alignment helps deal with spatial non-comparability; however, the vocabulary is limited to the anchor language.
Both monolingual and BWE implementations used word2vec based on skip-gram with negative sampling (SGNS), a vector of three hundred dimensions, five epochs, and a windows size for training of ten words. This implies using neural network architecture to optimize the prediction of words based on surrounding words to a specific word, which, in our case, is the rule of law (Kozlowski, Taddy, and Evans Reference Kozlowski, Taddy and Evans2019; Rodman Reference Rodman2020; van Breen et al. Reference Van Breen, Rijcken, Kantorowicz and Liem2025). We also pruned low-frequent words and dropped noisy ones using a threshold of ten. In this way, we obtained cosine similarity estimates that allowed us to quantify the semantic relationship between concepts based on the underlying assumption that the direction of the embedded vectors is more relevant than their magnitude to determine semantic linkages (Steck, Ekanadham, and Kallus Reference Steck, Ekanadham and Kallus2024). In simple words, cosine similarity gives us a number between –1 and 1 that tells us how closely related two ideas are in political speech. The cosine similarity measure is the most common metric for capturing the distance between the terms of interest (here, the rule of law and its corresponding dimensions)—hence, the similarity between these terms in a multidimensional embedding space.
Therefore, for Great Britain, for example, the machine-translated BWE is not only equivalent to the English-aligned embeddings but also to the monolingual one, considering slight differences because of the different number of multi-word collocations recodifications during our pre-processing stage. In addition, in order to estimate cosine similarity estimates considering a confidence interval of 95 percent, we included thirty bootstraps per model following the approach used by Maria Antoniak and David Mimno (Reference Antoniak and Mimno2018) and Alex Schwartz (Reference Schwartz2022). In simple terms, we repeated the analysis many times to make sure the results were stable rather than due to chance. This allowed us to average cosine similarity estimates over our bootstrap samples, enhancing the results’ reliability, especially in cases of smaller corpora. Figure 1 summarizes our pipeline.
Cross-lingual word embeddings data pipeline
Figure 1. Long description
The flowchart illustrates the stages of the cross-lingual word embeddings data pipeline. It begins with Phase 1, Data sources and case selection, which includes ParlaMint 4.0 data from 26 European countries plus 3 Spanish regions, and Open Discourse data for Germany. Main cases selected are Great Britain, Spain, France, Hungary, Italy, Netherlands, and Poland. Phase 2, Text processing, involves tokenization, standard pre-processing and cleaning, and recodifications for multiword collocations. Phase 3, Embeddings, uncertainty and replication, includes fitting embeddings per language or case, using BWE-pivot on English MT corpora for all except Germany, word2vec-SGNS on monolingual corpora, and Procrustes-aligned BWEs for all except Germany. The process involves 30 bootstraps per model for confidence intervals and replication of embedding space using mBERT/XLM-RoBERTa.
Finally, to assess the robustness of our semantic association measures in a jointly pretrained multilingual representation space, we replicated our core embedding-based approach in the original languages using multilingual transformer encoders, specifically the classic multilingual BERT (mBERT uncased) and XLM-RoBERTa, both base and large models. For each of the main three dimensions, we extracted a type-level vector by inserting the rule-of-law n-gram into a short neural template sentence and representing the term as the mean of its sub-word-token hidden states, emulating the three dimensions measured with our word2vec and SGNS implementation.Footnote 6
Rule of law in (selected) jurisdictions
Figures 2, 3, 4, and 5 display the main results—specifically, the aggregated cosine similarity scores, including their bootstrapped means and confidence intervals—for each of the three identified dimensions of the rule of law across eight selected jurisdictions. These cosine similarity scores quantify the association between the rule-of-law concept and its three dimensions; thus, a higher score indicates a stronger association with the specified dimension. Since the various presented models (monolingual, aligned BWE, and machine-translated BWE) provide similar results for most countries, we do not distinguish among these models for the purposes of describing the results. However, noticeable differences in the values of cosine similarity scores are observed for Hungary and Poland. It should be noted that these differences pertain to the absolute values of the scores, not to the relative ordering of the dimensions, which is of crucial importance for the ensuing analysis.
Aggregated word embeddings capturing the three dimensions of the rule of law across selected jurisdictions: Great Britain and Germany
Figure 2. Long description
A scatter plot compares cosine similarity of rule of law dimensions between Great Britain and Germany. The plot features three categories: Procedural - Judiciary, Procedural - Rules, and Substantive - Rights. Each category contains data points for Aligned BWE, Machine-Translated BWE, and Monolingual. The x-axis represents cosine similarity values ranging from 0.4 to 0.8. The y-axis lists the three categories. Great Britain’s data shows varying similarity values across categories, with some clustering around 0.5 and 0.7. Germany’s data points are more concentrated around 0.7 and 0.8. Different markers represent Aligned BWE, Machine-Translated BWE, and Monolingual data. The plot highlights differences in the interpretation of the rule of law between the two jurisdictions.
Aggregated word embeddings capturing the three dimensions of the rule of law across selected jurisdictions: Netherlands and France
Figure 3. Long description
A scatter plot compares cosine similarity of rule of law dimensions in Netherlands and France. The x-axis represents cosine similarity ranging from 0.4 to 0.8. The y-axis lists three dimensions: Procedural - Judiciary, Procedural - Rules, and Substantive - Rights. Data points are color-coded: black for Aligned BWE, dark gray for Machine-Translated BWE, and light gray for Monolingual. Each dimension shows clusters of data points with varying degrees of similarity. The plot highlights differences in legal traditions between the two countries, emphasizing procedural and substantive aspects of the rule of law. All values are approximated.
Aggregated word embeddings capturing the three dimensions of the rule of law across selected jurisdictions: Italy and Spain
Figure 4. Long description
A scatter plot compares cosine similarity values for Italy and Spain across three dimensions of the rule of law: Procedural - Judiciary, Procedural - Rules, and Substantive - Rights. The x-axis represents cosine similarity values ranging from 0.4 to 0.8, while the y-axis lists the three dimensions. Data points are color-coded and shaped differently to represent Aligned BWE, Machine-Translated BWE, and Monolingual categories. Each category shows distinct clusters for Italy and Spain, with varying levels of cosine similarity. The plot highlights differences in how the rule of law is conceptualized in these countries, with Aligned BWE generally showing higher similarity values. The visual representation underscores the complexity and context-dependent nature of the rule of law, reflecting the debates and practical challenges in its application.
Aggregated word embeddings capturing the three dimensions of the rule of law across selected jurisdictions: Hungary and Poland
Figure 5. Long description
The image contains two sets of graphs comparing cosine similarity of rule of law dimensions in Hungary and Poland. Each set includes three vertical scatter plots representing procedural-judiciary, procedural-rules, and substantive-rights dimensions. The x-axis measures cosine similarity ranging from 0.4 to 0.8, while the y-axis lists the three dimensions. Each plot contains three data points representing aligned BWE, machine-translated BWE, and monolingual embeddings, distinguished by different colors and markers. The graphs show variations in cosine similarity across the dimensions for both countries, highlighting differences in how the rule of law is conceptualized. All values are approximated.
Panel A of Figure 2 presents the model’s results for Great Britain, which is representative of the common law tradition. We find that the rights dimension is strongly associated with the concept of the rule of law (highest cosine similarity scores), indicating that substantive aspects of the rule of law are highly significant. It should be noted, however, that the judiciary dimension records cosine similarity values only slightly lower than those for the rights dimension. This suggests that substantive and procedural elements related to the judiciary are nearly as important to the concept. The procedural rules dimension, although historically significant, scores lower than the other two dimensions. Overall, the results are mixed regarding Conjecture 1; while the procedural judiciary component scores highly (as conjectured), the unexpectedly high importance of substantive elements is also notable.
Panels B (Figure 2) and C (Figure 3) illustrate the Rechtsstaat tradition of the rule of law in Germany and the Netherlands, respectively. In accordance with Conjecture 2, we observe the relatively high importance of the substantive rights dimension in both cases, as indicated by the high cosine similarity scores. Regarding the procedural dimensions, they diverge more from the substantive rights dimension (and the rule of law concept itself) than was observed in the case of Great Britain. Of the two procedural dimensions, the judiciary dimension appears more closely related to the rule of law than the procedural rules dimension. Although this contradicts what Conjecture 2 initially proposed, it mirrors the findings within the common law tradition. Therefore, we find tentative evidence supporting the statement that “conceptions of the Rechtsstaat and the rule of law have, for all intents and purposes, converged” (Meierhenrich Reference Meierhenrich, Meierhenrich and Loughlin2021, 66).
The results for the État de droit tradition of the rule of law are presented in Panels D (France in Figure 3), E (Italy in Figure 4), and F (Spain in Figure 4). The findings for France and Italy are remarkably similar and consistent with Conjecture 3, indicating that the substantive rights dimension is the most significant component of the rule of law in these countries. Regarding procedural dimensions, both the rules and judiciary aspects perform similarly. However, we initially conjectured that the judiciary dimension would be more significant in this legal tradition; thus, the empirical insights are not entirely in line with Conjecture 3. While Spain deviates from the pattern observed in France and Italy, its results align more closely with Conjecture 3. Specifically, the procedural judiciary aspect is identified as the most crucial, followed by the substantive rights dimension. In contrast, the procedural rules dimension is the least associated with the rule of law in Spain.
Lastly, Panels G and H of Figure 5 display the results for two post-socialist countries—Hungary and Poland—respectively. Among the two, Hungary’s results align more closely with Conjecture 4, which posits that countries with post-socialist traditions are likely to emphasize procedural aspects of the rule of law. In Hungary, the procedural judiciary dimension emerges as the most critical component of the rule of law, followed equally by procedural rules and substantive rights dimensions. Overall, Hungary appears to place greater importance on procedural, rather than substantive, aspects when its members of parliament discuss the rule of law and related concepts. Poland, in contrast, emphasizes the substantive rights dimension significantly more. Depending on the model, this rights component ranks either first or second, alternating with the procedural judiciary dimension. Unambiguously, the procedural rules dimension is deemed the least important, contradicting our initial conjecture for countries with post-socialist legal traditions. Given these findings, and considering that Hungary better aligns with Conjecture 4, one could argue that this predominantly procedural treatment of the rule of law in Hungary corresponds well with the concept of “autocratic legalism,” where the rules are manipulated in a principled and legal manner to undermine democracy and the rule of law itself.
Overall, while some conjectures based on legal traditions were confirmed with selected cases, our analysis suggests a general trend toward convergence in the meaning and understanding of the rule of law across various systems. This convergence was particularly evident within the traditions of common law, Rechtsstaat, and the État de droit (notably in France and Italy). In post-socialist countries, the understanding of the rule of law appears to be less established and more variable. Based on our methodology (word vectors) and results, it seems that, while traditional distinctions persist to some extent, there is a noticeable shift in parliamentary discourse toward a more unified understanding of the rule of law globally or at least in Europe. This finding aligns with our previous work, which shows that the meaning of the rule of law is quite similar in the United Kingdom and the United States (Kantorowicz and González-Bustamante Reference Kantorowicz and González-Bustamante2025a). While the finding that the meaning of the rule of law is rather similar in two common law countries is noteworthy—but not surprising—we also show in this article that a similar trend of convergence can be observed in countries belonging to diametrically different legal traditions.
In addition, in the Appendix, it is possible to find the following material: (1) replication using the extended list of terms that shows the same patterns with slight variations; (2) additional transformer-based estimates and rank-concordance analyses comparing the ordering of associations because each word2vec and SGNS have a different dimensionality than BERT models; and (3) an exploratory extension to the entire list of ParlaMint countries, including a principal component analysis (PCA). Concerning the transformer-based alternatives, the results are heterogeneous: Hungary and the Netherlands show strong robustness, while, for Germany and Great Britain, the BERT models also tend to reproduce our method’s ordering, especially XLM-RoBERTa-large. A similar pattern, but with a slightly weaker concordance, is observed for Spain, France, and Italy in the pair-wise associations. Poland presents the most mixed evidence; however, XLM-RoBERTa-base matches our main ordering exactly. With respect to the PCA, we did not observe any distinct country clusters, providing additional evidence for the convergence in how the rule of law is understood across jurisdictions.
Assessing the sensitivity of existing rule-of-law indicators
Having shown how word embeddings can be used to map discursive associations surrounding the rule of law, we now demonstrate how these associations can serve as a sensitivity check for existing rule-of-law indicators. Rather than proposing a new indicator, we use similarity scores derived from word embeddings to generate weights for the constituent components of existing indicators. This provides a principled and conceptually, as well as empirically, grounded way of aggregating various components of the rule-of-law indicators. Several widely used rule-of-law indicators are produced by, among others, Bertelsmann Stiftung, Freedom House, the World Bank, and—most prominently—the World Justice Project (WJP). These indicators differ substantially in how they conceptualize and measure the rule of law, particularly with respect to which elements they treat as constitutive of the concept (Skaaning Reference Skaaning2010; Ginsburg Reference Ginsburg2011; Voigt Reference Voigt2012; Versteeg and Ginsburg Reference Versteeg and Ginsburg2017; Ginsburg and Versteeg Reference Ginsburg, Versteeg, Meierhenrich and Loughlin2021). Crucially, however, once an element/component is included in an indicator, it is typically assigned equal weight to all other components. An alternative approach is to assign weights in a more principled manner, reflecting the relative importance of different components to the concept of the rule of law. Our aim, therefore, is to derive such weights from word embeddings and apply them to existing indicators in order to assess how robust their country rankings are to the application of alternative, conceptually and empirically informed, weighting schemes.
To this end, we employed the WJP indicators, which measure adherence to the rule of law as an arithmetic average of eight components: (1) constraints on government powers; (2) absence of corruption; (3) open government; (4) fundamental rights; (5) order and security; (6) regulatory enforcement; (7) civil justice; and (8) criminal justice. The WJP indicator is based primarily on large-scale surveys of the general population and legal experts, capturing perceptions and experiences related to both de facto and de jure aspects of the rule of law. Its eight components are derived from questionnaire-based indicators rather than textual or discursive data.Footnote 7 Importantly, the WJP indicator does not rely on parliamentary debates or elite discourse, nor does it attempt to capture how political actors conceptually frame the rule of law. Instead, it is designed as a normative governance indicator that assesses institutional performance and outcomes. This distinction is crucial for interpreting the analysis that follows. Our word-embedding results do not measure adherence to the rule of law, institutional quality, or legal effectiveness. Rather, they capture patterns in how the concept of the rule of law is articulated and associated with particular themes in parliamentary discourse. The reweighting exercise below therefore does not propose an alternative rule-of-law index, nor does it claim to improve upon the WJP’s normative judgments. Instead, it serves as an exploratory sensitivity check, examining whether the equal-weighting assumption embedded in the WJP index is robust to an alternative weighting, which is derived based on the salience of different rule-of-law components in elite political discourse.
The fact that the WJP uses an arithmetic average to compute the indicator means that each of the eight components is assigned equal weight when creating the overall rule-of-law index. While this approach is valid and transparent, it invites consideration of an alternative approach where different components of the rule of law vary in their importance and, thus, in their contribution to the overall index. For instance, the consistently high semantic proximity between the rule of law and rights-related terms in parliamentary discourse raises the question of whether existing indicators should prioritize similar components. We use this observation solely to explore the sensitivity of the WJP index to alternative, discursively informed, weighting schemes.
Table 3 enumerates all eight components of the rule of law (column 1) as defined by the WJP, along with the closest n-gram representing these components (column 2). It is important to note that the WJP’s civil and criminal justice components are combined under the single n-gram “justice.” In column 3, we present the cosine similarity scores between the selected n-grams and the “rule_of_law” n-gram. These reported cosine similarities represent the arithmetic average of scores obtained from a monolingual model across eight selected countries (that is, France, Germany, Hungary, Italy, Netherlands, Poland, Spain, and Great Britain). Finally, in column 4, we display the recalculated weights for the WJP’s rule-of-law components. These weights were derived by rescaling the cosine similarity scores so that their total sum equals 1.
The WJP’s rule-of-law components and new weights
The WJP’s recalculated rule-of-law indicators for all eight selected countries are displayed in Figure 6. Interestingly, applying tailored weights results in only minor upward adjustments to the indicators. The ranking of the countries is largely preserved. However, a notable interpretative difference can be observed for Spain. While the WJP’s original indicator positioned Spain below France, the recalculated indicator classifies these two countries as being nearly equal in terms of adherence to the rule of law. In Figure 7, we also report on ten countries that experienced the most significant adjustments in their rule-of-law indicators after implementing new weights. Iran, China, and Armenia exhibit the largest absolute adjustments among these ten countries. For these three countries, the adjustment is downward, primarily due to the increased weight assigned to the human rights component (substantive rule-of-law element). Overall, it is important to note that the changes observed based on recalculated indicators are relatively small (lack of tectonic changes), implying that the existing indicators, which do not implement the varying weights, are quite robust.
The WJP’s original and recalculated rule-of-law indicators across eight selected countries
Figure 6. Long description
A scatter plot compares the WJP’s original and recalculated rule-of-law indicators across eight selected countries. The x-axis represents the Rule of Law Index, ranging from 0.00 to 1.00. The y-axis lists the countries: Netherlands, Germany, Great Britain, France, Spain, Italy, Poland, and Hungary. Each country has two data points: one for the WJP’s original indicator (represented by gray circles) and one for the recalculated indicator (represented by black diamonds). The Netherlands and Germany have the highest rule-of-law index values, while Hungary has the lowest. The data points for each country are close to each other, indicating minor differences between the original and recalculated indicators. All values are approximated.
The WJP’s original and recalculated rule-of-law indicators across ten countries with the most significant adjustments
Figure 7. Long description
A scatter plot compares the WJP’s original and recalculated rule-of-law indicators across ten countries. The x-axis represents the Rule of Law Index, ranging from 0.00 to 1.00. The y-axis lists the countries: Portugal, Costa Rica, United Arab Emirates, Croatia, Georgia, Montenegro, Panama, China, Iran, and Venezuela. Each country has two data points: one for the WJP’s original indicator (gray circles) and one for the recalculated indicator (black diamonds). Portugal and Costa Rica show the highest values, while Venezuela shows the lowest. The data points for each country are close but not identical, indicating adjustments in the recalculated indicators. All values are approximated.
Although the adjustments in the rule-of-law indicators are not significant, this should not detract from the utility of applying cosine similarity scores as weights for constituent dimensions. Such an approach should be encouraged in the future not only as a means of demonstrating the sensitivity of the indicators but also, potentially, for providing confidence intervals around existing indicators. This would improve the interpretability and reliability of the measurements, thereby offering a more nuanced understanding of adherence to the rule of law across different jurisdictions. We emphasize that discursive salience should not be interpreted as normative importance. Components that feature prominently in parliamentary debate may do so because they are politically contested, institutionally weak, or strategically invoked rather than because they are well protected in practice. Consequently, any divergence between the discursively informed weights and the WJP’s original weights should be understood as analytically informative rather than normatively corrective.
Conclusions and outlook
Across Europe, politicians increasingly talk about the rule of law in similar ways. In this article, we aimed to demonstrate a proof of concept for disentangling the meaning of the concept of the rule of law across various jurisdictions using NLP and, more specifically, word vectors. We explored several conjectures based on discussions of legal traditions, which emphasized different aspects of the rule of law as being essential to its understanding. While some conjectures were confirmed, particularly those highlighting the substantive rights dimension of the rule of law (a thick conceptualization), others found no supporting evidence, especially those related to the common and post-socialist legal traditions. Overall, and as stated above, it is important to note that the models used in this study indicate a general convergence in the European definition of the rule of law. This supports the scholarly view that, despite the differences in origins and values that they aim to uphold, contemporary approaches to the rule of law are broadly similar in nature, at least as represented in parliamentary speeches by political elites.
We furthermore tested the utility of word vectors and the resulting cosine similarity scores between the “rule_of_law” n-gram and various components of the rule of law as weights to recalibrate some well-known rule-of-law indicators. Currently, all existing rule-of-law indicators uniformly apply the same weights to different constitutive elements of the rule of law. Our goal was to assign these weights in a more principled manner, aligning them with the perceived importance of these various components to the overall concept as captured in parliamentary debates. The recalculations of the WJP’s rule-of-law indicators, which originally consisted of eight components, resulted in only minor modifications to the rule-of-law indices, attesting that the current application of the equal-weighting system is robust. The general trend involved upward adjustments in scores for countries performing well in human rights and downward adjustments for countries performing poorly in this aspect as this component was given the greatest weight. While no “tectonic” changes were identified, we believe that this principled approach to deriving weights for constitutive parts could improve current indicators. This approach could be used to either assess their sensitivity or compute confidence intervals around the indicators by applying a variety of weights to the constitutive parts of the rule-of-law index.
This article has demonstrated how word embeddings can be used to map the multidimensional meaning of the rule of law as articulated in parliamentary discourse across different countries. Importantly, the analysis highlights both the potential and the limits of discursive data for understanding the rule of law. While such data provide insight into elite conceptualizations, they cannot substitute for normative theory or institutional analysis. Their primary value lies in complementing existing approaches and enabling systematic sensitivity checks of widely used indicators. As this article has served as a proof of concept, future research should expand upon our analysis to include a broader array of countries, particularly outside the European context, and extend the analyzed corpus beyond the scope of parliamentary speeches. An especially valuable expansion would be to incorporate court judgment analyses or other legal documents. We also encourage further efforts to refine existing rule-of-law indicators by exploring alternative methods for systematically calculating the weights of their constituent elements. A potential method could involve surveying a diverse group of experts (such as in Albert et al. Reference Albert, Corren, Brinks, Chilton, Dixon, Elkins, Ginsburg, Ran Hirschl, Moran, Tew and Versteeg2024) and the general public (see, for example, Gutmann, Kantorowicz, and Voigt Reference Gutmann, Kantorowicz and Voigt2024) to examine how the importance of different components of the rule of law varies.
Supplementary material
To view supplementary material for this article, please visit https://github.com/RoL-project/unravelling-rule-of-law
Acknowledgments
Bastián González-Bustamante and Jarosław Kantorowicz would like to thank the participants of the 2024 COMPTEXT conference held in Amsterdam for their useful comments.
Funding Statement
Funding for this project was provided by the Netherlands Organization for Scientific Research Open Competition XS Grant no. 406.XS.01.013. Neither author has competing interests to declare.
Open access
