6.1 Introduction: Varieties of Data in EU Legal Scholarship
The present chapter offers an overview of the varieties of data that are used in EU law scholarship. The chapter is based on a systematic analysis of data sources used in 248 academic articles on EU law, particularly on EU courts.Footnote 1 Each article was analysed with respect to its geographical coverage, empirical approach, qualitative or quantitative focus, methods, data, as well as the opportunities and limitations arising from the data and method employed. The aim of this chapter is threefold. First, it offers an overview of the types of data in EU legal scholarship on courts. Second, it engages critically with the possibilities and limitations of different data sources and the associated research methods and how the understanding of these can improve the study of EU law. Finally, the chapter seeks to stimulate a critical discussion on to what extent the emerging and non-traditional data sources complement or challenge the traditional understandings of what counts as law.
The outcome of the systematic literature review was unsurprising: doctrinal legal research continues to dominate the study of EU courts. Of the 248 articles analysed,Footnote 2 142 (57 per cent) involve doctrinal legal research based on, most commonly, courts’ case lawFootnote 3 and other legal documents.Footnote 4 This is expected, considering that legal research dominated scholarship on EU law throughout the twentieth century.Footnote 5 Other non-empirical contributions, although only a small number (12 in total), are from disciplines such as public administration,Footnote 6 sociology,Footnote 7 or finance.Footnote 8 The remainder of this chapter is based on an analysis of the 82 papers from the literature review (33 per cent) that are of empirical nature: either based on qualitative, quantitative, or mixed data. In Section 6.1.1, I start by examining the most common source of data: case law, both from the Court of Justice of the EU (CJEU) and national courts (62 per cent). Next, in Section 6.1.2, I explore data deriving from other official sources, such as reports, statistics, and other documents (11 per cent), which often complement the data based on the case law. Section 6.1.3 looks at expert data from other sources: interviews, surveys, experiments, newspapers, media, and courts’ websites (27 per cent). The distribution of data sources discussed in this chapter is illustrated in Figure 6.1.
6.1.1 Data from Case Law
More than 60 per cent of the studies analysed rely on case law as a primary data source (see Figure 6.1). However, it is important to distinguish between studies relying on data based on the CJEU’s case law (29 per cent) and those based on national courts’ case law (33 per cent).
6.1.1.1 The CJEU’s Case Law
The CJEU’s Curia database is a goldmine for researchers across the disciplines interested in EU law. At the time of writing, EU courts have delivered over 40,000 decisions, with almost 3,000 cases currently pending.Footnote 9 Early work on the CJEU written during the 1980s and 1990s came primarily from legal scholars such as Stein,Footnote 10 Snyder,Footnote 11 Shapiro,Footnote 12 Rasmussen,Footnote 13 Lenaerts,Footnote 14 Arnull,Footnote 15 and Weiler.Footnote 16 They focused on the interpretation of the CJEU’s judgments, explaining the origin and formal consequences of EU law doctrines such as supremacy, direct effect, or state liability,Footnote 17 as well as on the impact of the CJEU on the European integration processes and the development of a ‘symbiotic relationship between national courts and the Court of Justice’Footnote 18 via the preliminary ruling procedure.Footnote 19 Other scholars soon joined the debate, contributing new insights and data. In the mid-1990s, the literature on European legal integration expanded dramaticallyFootnote 20 and, by the turn of the century, American political scientists had published more work on the CJEU than on any other court except for the US Supreme Court.Footnote 21
Based on the systematic literature review, the most prevalent method used to analyse the CJEU case law after doctrinal legal analysis (66 per cent; see Figure 6.2) is qualitative content analysis (12 per cent), which involves analysing and interpreting the content of textual data in order to ‘provide a systematic way to identify patterns, concepts, and larger themes within the data to gain insight into the meaning and context of the content’.Footnote 22 For example, Passalacqua analysed 291 CJEU rulings in the field of EU migration law to map migrants’ rights defenders before the Luxembourg court,Footnote 23 while Pavelek and Zajíčková analysed 98 Court decisions in the area of data protection to map the trends and actors involved in these proceedings.Footnote 24 Other methods include descriptive statistics,Footnote 25 critical discourse analysis,Footnote 26 and natural language processing.Footnote 27
Methods of EU scholarship to analyse the CJEU’s case law.
Most studies use EurLex or Curia as primary data sources, either by hand-coding or web scraping the data. In 1999, Stone Sweet and Brunnel published the largest data collection study in EU law of the time,Footnote 28 with ample information about the preliminary ruling procedure across time, Member States (MS), and subject matter areas.Footnote 29 This dataset was later updated by Naurin et al. and Dyevre et al.Footnote 30 Similar research efforts were made with respect to CJEU’s infringement proceduresFootnote 31 and actions for annulment.Footnote 32 Scholars have, however, raised questions regarding the reliability of the metadata available on Curia, arguing, for example, that the subject matter of preliminary references is manually assigned by the CJEU’s legal service and does not completely capture the topic of the judgment.Footnote 33
The majority of similar datasets are smaller and produced for a specific MS,Footnote 34 policy area,Footnote 35 or time period.Footnote 36 Although of a smaller scale, these datasets offer important insights into certain country-specific drivers of referral rates, such as a country’s legal cultureFootnote 37 or the behaviour of the CJEU under the threat of override.Footnote 38 They are often based on qualitative content analysis, offering more detailed insights into specific cases and the broader context surrounding them. Yet, the question of the extent to which such insights can be generalised remains.
Large datasets of the CJEU’s case law are rare because they require a long and tedious process of hand-coding references submitted to the CJEU – which, at the time of the writing, totals 12,550 cases.Footnote 39 Besides being time-consuming, hand-coding requires ample resources. Some scholars have resorted to web scraping, text mining, and network analysis to derive insights. For example, Dyevre et al.’s analysis of more than 200,000 legislative acts produced by the EU since the 1950s showed that the most prevalent topics in EU legislation are related to Common Agricultural Policy (CAP) and the single market, and not constitutional issues as one would conclude from legal journals.Footnote 40 Meanwhile, Derlén and Lindholm used network analysis of the CJEU’s citations to challenge a traditional picture of EU law that portraits van Gend en Loos as a central case.Footnote 41
Thus, while we see an increase in the availability of data on the CJEU’s case law (see Table 6.1), ‘these data-collection efforts [are] overlapping, uncoordinated, and not always easily accessible’.Footnote 42 The IUROPA Project, founded in 2018, has tried to address this issue, together with many other difficulties associated with large-scale data collections based on the CJEU’s case law.Footnote 43 The project’s CJEU Database is the most complete collection of data on the CJEU to date and, importantly, it is regularly updated.Footnote 44 This cannot be said for other similar databases where data collection efforts are often discontinued when funding ends. One exception is the Judicial Proceedings in the European Union dataset by Ovádek, which is updated automatically.Footnote 45
| Project/author | Scope | Publicly available | Time frame |
|---|---|---|---|
| Evolution of European Union Law (EvoEU) Database Fjelstul (Reference Fjelstul2023)Footnote 48 | Connections between EU law, the CJEU’s case law, national case law that applies EU law, and national law that implements EU law | Yes (https://jfjelstul.github.io/evoeu/) | 1951–2015 |
| The European Union case law corpus (EUCLCORP) Trklja and McAuliffe (2018)Footnote 49 | Standardised, multidimensional and multilingual corpus of terminological and phraseological phrases in the CJEU case law | Yes, only to registered users (https://llecj.karenmcauliffe.com/euclcorp/) | 1952–2018 |
| IUROPA Database PlatformFootnote 50 | 9 datasets, each with a different unit of observation: cases, proceedings, decisions, parties, procedures, assignments, submissions, citations, and judges | Yes, upon request (https://www.iuropa.pol.gu.se/) | 1953–ongoing |
| Centrality Over Time of CJEU Decisions Derlén and Lindholm (Reference Derlén and Lindholm2014)Footnote 51 | Case citations in all CJEU judgments | Yes (https://data.mendeley.com/datasets/c8tg89hrbr/1) | 1954–2011 |
| Project/author | Scope | Publicly available | Time frame |
|---|---|---|---|
| The Archives project Database on Cases and Lawyers of the CJEU Avril and Brissaud (2021)Footnote 52 | All parties in the cases before the CJEU | No | 1954–2020 |
| Judicial Proceedings in the European Union Ovádek (2021)Footnote 53 | All CJEU judicial proceedings | Yes (https://michalovadek.github.io/eucourt/) | 1957–ongoing |
| EUTHORITY Project Preliminary References Dataset Dyevre et al. (Reference Dyevre, Glavina, Lampach, Ovádek and Wijtvliet2018)Footnote 54 | Preliminary ruling procedure | Yes (https://euthority.eu/?page_id=795) | 1958–2020 |
| EUTHORITY Project GEOCOURT Dataset Dyevre and Lampach (Reference Dyevre and Lampach2021)Footnote 55 | Geographic coordinates of the courts that have submitted preliminary references | Yes (https://euthority.eu/?page_id=795) | 1958–2020 |
| Stone Sweet and McCrown (2004)Footnote 56 | Infringement proceedings and preliminary ruling procedure – free movement of goods | No | 1961–1998 |
| Carrubba and Gabel (Reference Carrubba and Gabel2015)Footnote 57 | Infringement proceedings | Yes (https://polisci.emory.edu/home/people/biography/carrubba-clifford.html) | 1961–1999 |
| Project/author | Scope | Publicly available | Time frame |
|---|---|---|---|
| Data Set on Preliminary References in EC Law (Art. 234) Brunell and Stone Sweet (1998)Footnote 58 | Preliminary ruling procedure | Yes (https://glpn.la.utexas.edu/2018/05/19/18-0024-data-set-on-preliminary-references-in-ec-law-art-234-1961-2006/) | 1961–2006 |
| Larsson and Naurin (Reference Larsson and Naurin2016)Footnote 59 | Infringement proceedings | No | 1961–2016 |
| Berlin Infringement Database Börzel (2021)Footnote 60 | Infringement proceedings | Yes (https://www.polsoz.fu-berlin.de/en/polwiss/forschung/international/europa/bid/index.html) | 1978–2019 |
| Castro-Montero et al. (2018)Footnote 61 | Infringement proceedings | No | 2000–2017 |
Large-scale datasets such as these are mostly used to explore the drivers of national courts’ referral behaviour – very often complemented by other data sources including official statistics, reports, and other documents (see Section 6.1.2) – or the extent to which different parties (MSFootnote 46 or litigantsFootnote 47) impact the decision-making of the Court. Some studies have tried to uncover individual voting preferences of the CJEU’s judges based on the Court’s case law, in an attempt to address the ‘black box’ problem of European judicial politics. For example, Malecki showed that judges do not share uniform preferences in favour of EU integration: instead, their preferences lie on a continuum ranging from Europhilia to Euroscepticism.Footnote 62 Wijtvliet produced similar findings for judges of the General Court.Footnote 63
The field keeps evolving. Machine learning – probably the most contested method used to analyse the courts – is now being used on case law data to predict future decision-making. While this has so far been more successful in the case of the European Court of Human Rights (ECtHR),Footnote 64 EU law scholars have resorted to using Advocate General (AG) opinions to predict the Court’s decisions; the Court follows the AG’s opinion in 67–88 per cent of cases, depending on the study.Footnote 65 A recent study by Fjelstul uses a case-space model and machine learning to generate predictions on the impact of the panel size on the outcome of the CJEU’s infringement cases.Footnote 66 However, in 2019, France passed a controversial law that criminalised research on the behaviour of individual judicial behaviour in order to protect judges’ independence and reputation.Footnote 67 Developments such as these are particularly damaging for big data-driven projects that use machine learning and prediction algorithms.Footnote 68
6.1.1.2 National Courts’ Case Law
EU law scholarship relying on national courts’ case law as the main data source often tries to account for the limitations of existing EU law datasets. One of the limitations is the fact that for the last three decades legal scholars and political scientists have focused largely on preliminary ruling procedure as the most visible aspect of EU law use.Footnote 69 Hübner’s 2015 study is one of the few attempts to estimate the bulk of national case law that relates to EU law outside of the preliminary ruling procedure.Footnote 70 She introduced the Dec.Nat. Database (National Decisions Database) which contains 36,400 national decisions involving EU law.Footnote 71 The database, however, suffers from a couple of weaknesses. First, selection bias: the database covers only those national decisions that the CJEU, its administrators, and/or national courts have deemed important for the purpose of monitoring national case law relating to EU law,Footnote 72 whereby the selection process is not motivated by systematic criteria.Footnote 73 Second, since 2003 and under resource pressure, the database has been limited to decisions issued by national courts of the last instance and to those decisions that consider ‘innovative points of law and legal reasoning’.Footnote 74 Other comparable data collection efforts such as the JuriFast, Caselex, or CODICES databases suffer from similar shortcomings.Footnote 75
Other notable large-scale databases of national case law include the European Commission for the Efficiency of Justice (CEPEJ) and the EU Justice Scoreboard data. The CEPEJ was established by the Council of Europe (CoE) in 2002 with the aim of improving the efficiency and functioning of justice in CoE MS. Following the establishment of the CEPEJ, in 2013 the Commission launched its own evaluation instrument called the EU Justice Scoreboard, with the aim of helping EU MS to improve the efficiency of their justice systems. Since its launch, the CEPEJ has been contributing data on several indicators: courts’ caseload, length of proceedings, and courts’ staff. When assessing the efficiency of justice of CoE and EU MS, courts’ workload and the timeframe to solve cases take centre stage.Footnote 76 While the CEPEJ and EU Justice Scoreboard data has been extremely valuable in practice, sparking policy changes at the national levelFootnote 77 and being increasingly used in academicFootnote 78 and political debates, scholars suggest extreme caution ‘when using such data for scholarly, legislative or policy discourses’.Footnote 79 Limitations include historical data series inconsistencies, as well as inconsistencies in the category, nature, and status of cases. Similar to the Dec.Nat database, divergences in MS’ methodologies and definitions, as well as the lack of systematic selection criteria, make data difficult to compare robustly between the MS.
Other scholarly efforts in collecting national courts’ case law are of a smaller scale, focusing primarily on a limited number of MSFootnote 80 or a specific policy area.Footnote 81 For example, Conant collected French, German, and British courts’ citations of the CJEU’s case law and EU law sources and reported that ‘national judges who do invoke European provisions in their decisions are likely to interpret the treaties, regulations, and directives without any explicit reliance on ECJ case law’.Footnote 84 Furthermore, Obermaier explored the role of national courts in the implementation and enforcement of EU law, focusing primarily on MS’ acceptance of the Kohll/Decker doctrine, to explore how far the national courts’ rulings were decisive in the implementation of the CJEU’s doctrines into national social security legislation. Also looking at France, Germany, and the UK, he found that implementation of the CJEU’s rulings takes place faster and smoother when national courts accept and apply CJEU doctrine. In all three MS, national courts successfully influenced the implementation of the Kohll/Decker doctrine by forcing the legislator to incorporate the doctrine into national social law.Footnote 85
Chalmers, too, warned that EU scholarship on legal integration focuses too narrowly on ‘those minority of judgments referred to the ECJ from national jurisdictions’.Footnote 86 He collected all reported judgments between 1971 and 1998 in which British judges addressed a question of EU law or British accession to the EU (n = 1,088). He concluded that British judges’ engagement with EU law is very narrow both with respect to legal areas it covers and the change it provoked: only in 8 per cent of cases have the litigants successfully invoked economic freedoms governed by EU law.Footnote 87 Chalmers concluded that ‘it is difficult to believe that the British experience on the narrow application of EC law is unique’.Footnote 88 Dyevre subsequently compiled a new dataset coding doctrinal positions of 34 national courts at the highest level (supreme and constitutional courts) to capture courts’ doctrinal responses to European integration.Footnote 89 These data collection efforts have been further extended by EUTHORITY’s Domestic Judicial Response (DJR) Dataset, which collects and hand-codes information on the decisions issued by 64 highest-level national courts in the EU on the relationship between EU law and domestic law – although official publications from this project are still pending.Footnote 90
Nyikos went further and examined the enforcement of the preliminary ruling procedure, exploring how often the referring court complies with the preliminary ruling issued by the CJEU. She found that the level of compliance among national courts is extremely high: the CJEU’s ruling was implemented in 96 per cent of cases. The referring court evaded or did not implement the CJEU’s ruling in less than 4 per cent of cases – either by reinterpreting the facts of the case or claiming that the CJEU had exceeded its jurisdiction.Footnote 91 The latter situation was documented in the famous Arsenal v. Reed case where the British judge refused to follow part of the CJEU’s ruling under the rationale that the ruling concerned the facts of the case which, in this judge’s view, was outside of the CJEU’s jurisdiction.Footnote 92 This example, according to Davies, ‘serves as a reminder that the Court is dependent upon national judges and has good reasons to be sensitive to the division of functions between them’.Footnote 93
The majority of these studies focus on (Western) MS, with a handful of them examining the impact of EU law and the CJEU’s case law on EU candidate states or neighbouring countries. For example, Reich and Micklitz collected and analysed citations of the CJEU’s judgments by national courts in Armenia, Azerbaijan, Georgia, Israel, Jordan, Russia, Switzerland, Tunisia, Turkey, Ukraine, and the Eurasian Economic Union.Footnote 94
While studies focusing on national courts’ case law offer valuable information on the actual enforcement and reception of EU law on the ground, they suffer from several shortcomings. First – and similar to CJEU case law databases – data collection efforts on national case law involving EU law have been equally overlapping, uncoordinated, and even more inaccessible (see Table 6.2). Furthermore, collecting data from tens of thousands of courts across the EU – who adjudicate hundreds of thousands of cases each year – by manual coding or web scraping would be a mammoth effort and create noise in the data. More importantly, many courts do not publish their judgments online; this is especially true for lower national courts. Researchers interested in obtaining this data should, therefore, be prepared to enter a court’s archives and manually analyse judgments of interest.Footnote 95 Because of limited access to this data, many studies narrow their focus to specific MS (France, Germany, and the UK are particularly popular), a specific type of court (e.g., supreme and/or constitutional courts, whose workload is not as heavy and whose decisions are generally accessible online), or a specific policy area. To circumvent the fundamental problem of access, some studies even employ surveys or experiments in the classroom, where students act as judges.Footnote 96 Again, this raises the issue of the generalisability of findings to the entire EU.
| Project/author | Scope | Publicly available | Time frame |
|---|---|---|---|
| EUTHORITY Project Domestic Judicial Response (DJR) Dataset | Doctrinal positions of 34 highest national courts towards European integration | No | 1958–2020 |
| Dec.Nat. Database (National Decisions Database) | National courts’ decisions involving EU law (since 2003 only highest courts’ decisions) | Yes (https://www.aca-europe.eu/index.php/en/dec-nat-en) | 1959–ongoing |
| Nyikos | Enforcement of preliminary ruling procedure by referring court (Germany, France, Netherlands, Belgium, UK) | No | 1961–1994 |
| CODICES database Council of Europe (CoE) | Constitutional courts’ decisions (EU MS and other countries) | Yes (https://codices.coe.int/) | 1965–ongoing |
| Conant (Reference Conant2002)Footnote 82 | French, German and British courts’ citation of the CJEU’s case law and EU law sources | No | 1983–1999 |
| Chalmers | Judgments of British courts that address a question of EU law or British accession to the EU | No | 1971–1998 |
| Obermaier (2008)Footnote 83 | French, German and UK courts’ implementation and enforcement of EU law | No | 1998–2006 |
| Project/author | Scope | Publicly available | Time frame |
|---|---|---|---|
| JuriFast Database ACA Europe | Most important cases in which national highest courts apply EU law | Yes (https://www.aca-europe.eu/index.php/en/jurifast-en) | 2000–ongoing |
| Ceselex database | National courts’ case law linked to EU law | No | 2000–2010 |
| CEPEJ (European Commission for the Efficiency of Justice) Council of Europe (CoE) | Data related to judicial systems of CoE MS | Yes (https://www.coe.int/en/web/cepej/cepej-stat) | 2010–ongoing (every two years) |
| EU Justice Scoreboard | Overview of indicators on the efficiency, quality and independence of justice systems in EU MS | Yes (https://commission.europa.eu/strategy-and-policy/policies/justice-and-fundamental-rights/upholding-rule-law/eu-justice-scoreboard_en) | 2015–ongoing |
6.1.2 Data from Official Statistics, Reports, and Other Documents
Case law data is often complemented by data based on official statistics, court or county official reports, and other documents. The preliminary ruling procedure is a prime example of a topic where these two data sources meet. The doctrinal debate on what drives a national court’s preliminary questions to the CJEU revolved primarily around the obligation versus discretion of national courts to refer questions to Luxembourg, yet legal scholarship had failed to offer convincing empirical evidence to account for the referral activity of national courts.Footnote 97 This is where the empiricists came in: some of the empirical factors put forth by the political science and economics literature include macro-level factors such as divergences in MS intra-EU trade,Footnote 98 GDP,Footnote 99 population size,Footnote 100 or years of EU membership,Footnote 101 as well as meso-level factors such as and type and level of the courtFootnote 102 or court’s workload,Footnote 103 and micro-level factors such as judges’ knowledge of, experiences with, and attitudes towards EU law.Footnote 104
To explore this topic many studies have combined preliminary reference data based on the CJEU’s case law (see Section 6.1.1.1) with other external datasets and data sources such as Eurobarometer, Eurostat, or the Commission’s Annual Macro-Economic database (AMECO)Footnote 105 (see also Figure 6.1). This data is, however, much more suited to exploring macro-level variables such as the effect of intra-EU trade, GDP, or population size, for which there is also easily accessible data, than it is for exploring meso- and micro-level variables such as judges’ knowledge of EU law, experiences with EU law, and their attitudes towards the EU.Footnote 106 The ability to draw inferences about the distinct influences of micro- and meso-level variables is severely restricted by the lack of data on national judiciaries. To explore these factors, scholars have often resorted to proxies. For example, Hornuf and Voight use the number of graduates from the College of Europe in Bruges as a proxy for the familiarity with EU law,Footnote 107 while Dyevre and Lampach rely on the years of a country’s EU membership.Footnote 108 Such use of proxies has not been without criticism.Footnote 109
Other sources of data present in EU scholarship on courts include a range of reports and official documents, including official documents and statements of EU Presidencies,Footnote 110 Commission’ decisions and speeches of Commissioners,Footnote 111 the European Court of Auditors annual reports,Footnote 112 the CJEU’s annual reports on judicial activity,Footnote 113 and European Ombudsman decisions.Footnote 114 These are often combined with judicial’ case law. For example, Kameoka combines the qualitative content analysis of the CJEU’s case law with the analysis of the Commission’s official documents to explore how legal professional privilege is interpreted in EU competition law,Footnote 115 while Stylianou and Iacovides supplement the CJEU’s case law and AG’s opinions with the Commission’s decisions and speeches from Commissioners for Competition to reveal the underlying goals of EU competition law.Footnote 116 Regarding the preliminary ruling procedure, Pavone combines data based on local courts’ referrals to the CJEU with interview results, newspaper records, and public opinion data to demonstrate the transformation of Italian port law following the ‘Port of Genoa’ case.Footnote 117
While such data sources are useful for data triangulation, thereby raising the reliability and validity of results, they suffer from a key weakness: access. Some official statistics may not be available for a desired time period. Examining limited access to documents, Leino-Sandberg argued that, while EU institutions, offices, and bodies are in possession of various documents that are of high importance for legislative and policy outcomes in the EU (and, therefore, for EU law scholarship), ‘beyond leaks that sometimes appear in Politico or the Financial Times, access to legal opinions remains highly restricted and largely relies on individuals making public access requests under the EU’s public access legislation’.Footnote 118 While the EU courts and the European Ombudsman played an important role in facilitating public access to such documents, Leino-Sandberg remains sceptical that future research employing such data ‘is threatened by an increasing problem of empty archives’, with institutions ‘actively engaged in destroying existing records to avoid public scrutiny’.Footnote 119
6.1.3 Data from Experts
The two data sources explored in the section above (case law and official statistics) tell us little about the role of individual actors in shaping the decision-making of the EU courts, or about the acceptance and perception of EU law by different actors at the national and EU level. As discussed in detail above, nor is this data suitable for exploring meso- or micro-level variables. Additionally, there is growing criticism that EU scholarship focuses too much on CJEU case law that is itself too narrowly focuses on the minority of cases which end up at the CJEU, ignoring the fact that the majority of enforcement of EU law happens at the national level by national courts and other authorities.Footnote 120 For instance, regarding the preliminary ruling procedure, scholars have been criticised for ignoring the fact that preliminary references do not originate from MS or their courts but are rather a work of individual judges.Footnote 121
To account for this criticism, a new generation of scholars has started looking at court composition and the role of individual actors and their ideologies, attitudes, knowledge, and experiences in the process of the application of EU law. However, instead of using proxies (see Section 6.1.1.1), they resort to different types of data: secondary data on judges’ biographies, interviews, and survey data.
6.1.3.1 Interviews
The second-most used data after judicial’ case law is interview data (13 per cent; see Figure 6.1). Until the early 2000s, EU law scholarship focused primarily on MS’ and national courts’ acceptance of EU law, ignoring the role of individual judges, référendaires, law clerks, and litigants. This changed in the mid-2010s.
In 2013, Nowak et al. launched a study on the knowledge experiences and attitudes of Dutch and German judges towards the EU and EU law.Footnote 122 Interviews on the incentives and constraints in the application of EU law were later undertaken by Jaremba in PolandFootnote 123 and Glavina in Slovenia and Croatia.Footnote 124 All three studies show striking similarities: judges do not think they encounter EU law cases very often, are not so sure about their knowledge of EU law, and often do not apply EU law because of a lack of time, knowledge, and resources to engage with ‘such a complicated topic’.Footnote 125 Tatham interviewed Hungarian judges regarding their experiences with EU law and – similar to GlavinaFootnote 126 – he identified foreign language skills and EU law training as two key elements that would encourage the application of EU law by national judges.Footnote 127 Krommendijk’s work tackled the reasons that Dutch and Irish judges tend not to refer preliminary questions,Footnote 128 and Leijon’s study on Swedish judges further explored judges’ reasoning for providing (or not) their own opinions in their references for a preliminary ruling.Footnote 129 Only one study in the literature review involved interviews with référendaires.Footnote 130
Interviews with judges are a rewarding tool for exploring how EU law is applied in practice and how it is perceived. They help to shed light on the reality of EU law and uncover factors that could not be detected through purely legal or quantitative analysis. However, interview-based research suffers from several, very often practical, shortcomings. The most visible issue is once again: access. No study has employed in-depth interviews to explore the attitudes and experiences of CJEU judges. Indeed, this is a difficult task: political elites such as judges are unlikely to participate in studies or offer (fully) honest answers,Footnote 131 making any conclusions unreliable. Popular interviews with CJEU judges, such as Judge Sasha Prechal’s interview with the European Law Blog,Footnote 132 are not scientific and are light on content. This is why scholars interested in judicial politics have resorted to using votes,Footnote 133 or other proxies such as the party affiliation of the appointing presidentFootnote 134 or the appointing government,Footnote 135 as explanations for judges’ behaviour. National supreme or constitutional court justices are also rarely included in studies because – with these highest courts hosting only a handful of justices – it is difficult to grant full anonymity. Lower court judges can more easily preserve anonymity and, ideally, speak openly. Furthermore, unlike judges sitting on constitutional or supreme courts, lower court judges are rarely considered political elites or celebrities.Footnote 136 Worth mentioning, however, is that it is difficult to ensure a participant’s anonymity when conducting an interview: the process inevitably involves the exchange of private information, such as name and contact details, and often involves a face-to-face meeting. The best the researcher has to offer is pseudonymisation of the research data (preceded by obtaining a strict ethical approval).Footnote 137
Access to lower-court’ judges also presents a challenge because judges must be approached through the president of a court.Footnote 138 Furthermore, interviewing judges often requires permission from relevant authorities, such as the ministry of justice, the council of the judiciary, or the (supreme) court presidents, which can take time. Glavina referenced two court presidents who refused to forward the call for research to judges because their ‘judges are too busy to participate in the study’.Footnote 139
Another methodological issue with interview data is the problem of self-selection bias and the generalisability of results. First, studies on the application of EU law at the national level are naturally more likely to attract more Europhilic judges or judges who more often deal with EU law in their daily work.Footnote 140 Is the sample then representative of the population? This problem may to some extent be addressed with a sampling strategy that includes different levels and types of courts, different regions, or MS, generating a more representative sample.Footnote 141 Finally, researchers may encounter potential struggles for control of the data, whereby the interviewee attempts to manage the interview process or data interpretation.Footnote 142 Interviewees, of course, have the full right to withdraw their consent to participate in the study at any point, even after the interview has been conducted.Footnote 143 Yet, the question of usability of such data remains.
Judges are, however, not the only actors of EU law that have been covered by interviews. In his seminal work on the transformation of Italian port law,Footnote 144 Pavone conducted interviews with Genoese legal professionals, twenty lawyers, seven judges, and ten law professors. Pavone followed the work of two Eurolawyers – Giuseppe Conte and Giuseppe Giacomini – who used EU competition law as a basis to challenge a centuries-long monopoly over the Genova port by the dockworkers’ union.Footnote 145 Another example includes the work of Everson and Eisner who interviewed lawyers and judges of the High Court of England and Wales to uncover the leading role of lawyers in accepting the supremacy of EU law.Footnote 146 Kilpatrick’s study similarly uncovered that a high number of gender equality preliminary questions from North West Germany was driven by a small number of Europhilic lawyers.Footnote 147 Interviews with lawyers, too, pose challenges that are not sufficiently addressed by empirical legal research. Here, we also encounter issues of access and confidentiality, as well as potential struggles for control of the data.
Recent research efforts have started looking at the role of those who mobilise EU law. After all, without litigants there are no cases that could reach the CJEU.Footnote 148 The literature has focused on NGOs, equality bodies, trade unions, and human rights associations in EU non-discrimination law,Footnote 149 migration law,Footnote 150 and environmental law.Footnote 151 While many of these studies relied on the CJEU’s case law as their main data source,Footnote 152 other studies use interview data with NGOs and equality bodies involved in initiating some of the landmark EU law cases: Muir and Kolf uncovered the crucial role of the Belgian equality body CGKR (‘Centre for equal opportunities and opposition to racism’) in the development of EU racial discrimination law.Footnote 153 Similarly, Miller combined interviews with case law, documents, and newspaper analyses to trace the transformation of Danish disability law through legal mobilisation of EU law and the preliminary ruling procedure.Footnote 154 Passalacqua mapped collective actors in the field of migration law (so called ‘migrant rights defenders’) that helped to change restrictive national migration laws and policies,Footnote 155 while, in a very recent study, van der Pas uncovered the conditions under which actors mobilise EU law.Footnote 156
6.1.3.2 Surveys and Experiments
Survey data can reflect public opinion or expert opinion. Studies of public opinion on courts are a more recent development. Consider the Eurobarometer public opinion surveys: questions about the European Parliament (EP) and the Commission were already a part of these in the 1980s;Footnote 157 questions about the EU courts (the CJEU and the Court of Auditors) were added much later.Footnote 158 Once this data became available, scholars started looking into the levels and drivers of public support towards the CJEU via a range of related concepts: trust, confidence, public support, diffuse support, or legitimacy.Footnote 159
Caldeira and Gibson’s Reference Caldeira and Gibson1995 study building on the Eurobarometer data did not find much diffuse support for the CJEU among the public, arguably because the CJEU was still a young institution at the time; they did expect the Court’s visibility and support to increase in the future. Their findings further highlighted that public attitudes towards the CJEU are partly shaped by public attitudes towards the EU.Footnote 160 Also relying on Eurobarometer’s data, Kelemen’s 2012 study showed that that the CJEU is consistently and by far the most trusted institution in the EU.Footnote 161 Finally, Arnold et al. used the Eurobarometer data to explore the drivers of citizens’ trust in EU political institutions and found that the MS’ corruption levels are the most important drivers of the relationship between national and EU institutional trust: lower levels of perceived national corruption go hand in hand with more trust in the CJEU.Footnote 162 Furthermore, their findings suggested that women are less inclined to trust the CJEU compared to men, which is in line with earlier research that important CJEU decisions in favour of women’s interest had had no impact on women’s support for the Court.Footnote 163
Other studies go beyond the large-scale Eurobarometer data. Grosfeld et al., for example, collected survey data on 1,180 individuals from Finland, France, Germany, Italy, Netherlands, and Poland via the online participant platform Prolific. They demonstrated that value alignment plays an important role in the perceived legitimacy of the CJEU and the EU.Footnote 164 Other uses of large-scale surveys on public opinion include the European Social Survey (ESS), which captures Europhilic/Eurosceptic attitudes, and Google Trends data to capture public interest in the CJEU.Footnote 165 Studies using experiment-based data are not uncommon either. Cheruvu, for example, relies on a vignette experiment with the German public to uncover public support for EU law. His results show that when a local court decides an issue of EU law based on the CJEU’s preliminary ruling, this increases citizens’ support for EU law.Footnote 166
Other types of survey data on EU law capture experts’ opinions, most commonly national judges. Already in 2007, the European Parliament published a report based on a survey of more than 2,300 judges on the role of national judges in the European judicial system.Footnote 167 The report revealed significant disparities in the knowledge of EU law among judges, a limited awareness of EU law, a lack of knowledge of the preliminary ruling procedure, difficulties with accessing information on EU law, a perception of EU law as excessively complex and opaque, and the need to enhance judicial knowledge of foreign languages.Footnote 168 Following on from this report, Nowak et al. emphasised the disparity between the academic literature and the reality that judges face when applying EU law.Footnote 169 They identified three general clusters of problems that could affect national judges’ ability to fulfil the EU’s expectations: (a) judicial experiences with the application of EU law in their daily work, (b) judicial knowledge of EU law, and (c) attitudes of individual judges towards the EU, EU law, or their new role as European judges.Footnote 170 Similar results were later obtained by Jaremba on Polish judges,Footnote 171 Mayoral on Spanish judges,Footnote 172 and Glavina on Slovenian and Croatian judges.Footnote 173 Their research efforts resulted in several combined empirical outcomes.Footnote 174
Other studies have surveyed other court staff such as law clerks and assistants to, for example, assess the training needs with regard to EU law;Footnote 175 policy-makers, civil servants, companies, interest groups, and scientific experts to assess the level of (non-)compliance of MS with EU law;Footnote 176 and experts on EU law to assess the strength of the European Court of AuditorsFootnote 177 or the ideology of judges serving on the General Court.Footnote 178
Survey-based research on EU law has been extremely valuable in uncovering the reality of EU law application at the national level and all the challenges that come with EU membership. Yet, similar challenges as with the interview-based research arise. As already discussed above, the principal practical challenge is access to respondents – and, thus, data (see Section 6.3.2). Another problem is the response rate: for example, a 2012 European Parliament study received 6,000 responses from judges and prosecutors, representing only 5 per cent of all judges and prosecutors in the EU.Footnote 179 Mayoral’s study on Spanish judges received a response rate of only 2.3 per cent;Footnote 180 Jaremba’s study on Polish judges 8 per cent;Footnote 181 Nowak et al.’s study on German judges 10 per cent.Footnote 182 Glavina’s response rate from Croatian and Slovenian judges was 16.6 and 14.7 per cent respectively.Footnote 183 Only in the Netherlands has the response rate been higher – 32 per cent – which the authors attribute to Dutch judges already being acquainted with participating in online surveys.Footnote 184 This raises the question of what level of response rate is sufficient to make valid and generalisable results? Of course, a higher response rate is preferable because missing data is never random.Footnote 185 However, boosting response rates within such an inaccessible group as judges poses a considerable challenge.
Self-assessment reports such as surveys have further been characterised as problematic as the data is often not able to be verified by other means. Furthermore, respondents such as judges may answer in a way that portrays themselves in a better light. Thus, the validity of the data depends very much on the sincerity of the respondents. In contrast, some of the novel literature defends the idea of using self-assessment reports with the argument that individual judges are the best experts on their own attitudes.Footnote 186 Respondents are also more likely to fill in a survey when it is about themselves.Footnote 187 In fact, motivation was found to be the strongest predictor of the validity of self-assessment reports.Footnote 188 Further methodological problems may arise when both dependent and independent variables are collected from the same source, which is known as the ‘problem of common method variance’.Footnote 189 This is common to studies on EU law that rely on regression analysis to analyse survey results, such as Glavina’s study on the drivers of referral behaviour among national judgesFootnote 190 or Grosfeld et al.’s study on public opinion about the EU,Footnote 191 although many other studies of survey results rely on descriptive statistics.Footnote 192
While it is important to acknowledge the problems associated with the use of self-reporting, there is no foolproof method to explore perceptions of judges and other experts in EU law. Focusing on the CJEU’s case law and the preliminary ruling procedure, for example, overlooks the bulk of cases that never reached Luxembourg; using judicial votes is not possible in many cases as they are not made public (in the case of the CJEU); or there may be a limited access to the case law (in the case of national courts). Using proxies, such as judges’ appointments and statements, is not ideal either, which will be discussed in more detail in the following section.
Some pitfalls may be addressed by data triangulation or adopting a mixed-method research design. Such a research design was adopted by 16.5 per cent of the empirical studies included in the literature review. Despite the advantages, mixed-method research remains rare in EU law scholarship (see Figure 6.3). This perhaps reflects the complexity of mastering multiple research methods, especially due to a lack of training of legal scholars in empirical methods.Footnote 193 But quantitative scholars should equally not rush to dismiss qualitative methods or data as ‘an easy way of conducting research’, because qualitative data has the potential to enhance their research in its own way.
6.1.3.3 Judges’ Biographies, Websites, and Newspapers
Because judges are such a hard group to gain access to, many scholars have resorted to proxies. This has introduced a whole new source of data: newspapers, social media, and other online information. While the use of data on judicial appointments,Footnote 194 party affiliations,Footnote 195 pre-nomination speeches by high-ranking justices,Footnote 196 or their statements in newspaper editorialsFootnote 197 has a prominent standing in the US, using this type of data in EU law scholarship is still relatively new.
Judicial Biographies.
Because of the secrecy of decision-making in European courts, particularly the CJEU, scholars interested in the voting preferences of judges have had to resort to secondary data. The most commonly used source of data in this respect is judicial biographies. For example, Malecki’s studyFootnote 198 on ideological preferences of the ECJ’s judges used a measure for the left–right composition of the appointing government based on the manifesto scores dataset by Kim and Fording.Footnote 199 Favale then looked at the practice of assigning copyright cases at the CJEU based on judges’ prior specialisation. To do so, she relied on judges’ biographical data from the Court’s website.Footnote 200 The CJEU’s website may, however, lack a lot of background data – whether because it is missing or deliberately omitted (e.g., a judge’s party affiliation).
To account for the limited data, Zhang et al. collected a whole range of background characteristics of judges to explore their potential effect on the outcome of competition law cases.Footnote 201 This included data on the judges’ country of origin, prior work experience (academic, civil servant, private practitioner, and judge), last job before joining the CJEU, country of legal training, age, and gender. In addition, they used the political ideology of the appointing governments as a proxy for the judge’s ideology. However, the country of origin is an imperfect proxy as it tells us little about whether a judge has lived or studied abroad and so been shaped by other legal traditions. The authors acknowledge that data from the CJEU’s website or annual reports is incomplete and supplement it with information from the European Council: whenever MS appoint a judge to the CJEU, the European Council publishes their resumes. When resumes were missing, the researchers made a formal information request to the European Council.Footnote 202 Yet, even there, crucial information on the judges’ background can be missing.
Comparable research efforts were conducted by the aforementioned IUROPA Project (see Section 6.3.1.1) where the JUDGES Dataset contains the name of the judges, their MS of origin, gender, positions at the Court, and professional background (judge, academic, civil servant, practicing lawyer, politician).Footnote 203 Although limited, similar research on collecting biographies has also been conducted on Advocates GeneralFootnote 204 and national judges.Footnote 205
News Articles.
Another source of data in EU law scholarship is news articles, based either on traditional or modern news sources. For example, Pavone’s seminal work on the port of Genoa made use of local and national papers, ranging from labour-friendly to more liberal outlets, and showed how Eurolawyers have strategically mobilised local press to sensitise public opinion and increase public awareness of EU law and the CJEU.Footnote 206 Miller used a similar approach on data triangulation, combining interview data, case law, and Danish newspaper analysis to illustrate the transformation of Danish anti-discrimination law,Footnote 207 while Schenk relied on newspaper articles, semi-structured expert interviews, and policy documents to explore the development of EU students’ free movement and cross-border welfare rights in England.Footnote 208 Finally, Glavina’s recent study on the emergence of dissenting opinions in the Croatian Constitutional Court relies on online newspaper data to track the appointing government as a proxy for judicial ideologies, and newspaper coverage of judges’ private lives as a proxy for judicial temperament.Footnote 209
Turning to the salience of the CJEU’s decisions, Dedeke explored the conditions under which CJEU decisions are subject to discussions in the news based on newspaper reports in eight broadsheets in six MS. His results showed that national media plays an important role in the way the CJEU’s decisions are observed and evaluated by the public.Footnote 210
Data based on secondary sources such as websites and newspapers offer a way to place legal developments in (historical) context. However, this data source also has several limitations. Chiefly, again, the problem of access. For example, older newspapers that are not fully digitalised may not be easily accessible and researchers may need to be prepared to enter physical archives to collect the data of interest. Furthermore, researchers using this type of data may easily find themselves in danger of selection bias, selecting news sources which lean more towards one side of the political spectrum, including that on European integration.
6.2 Concluding Remarks
This chapter has offered an overview of the varieties of data used in EU law scholarship and reflected on the opportunities and limitations of different data sources and their associated research methods.
This concluding section reflects on the sources of data which are underexplored in EU law research with the aim of stimulating a critical discussion of how such emerging and non-traditional data sources either complement or challenge traditional understandings of what counts as law. Underexplored sources of data include national courts’ case law, which remains very fragmented and limited to a handful of MS (Germany, France, and the UK). Research efforts that do cover all EU MS, such as the EUTHORITY Project’s DJR Dataset or the Dec.Nat. Dataset cover only the highest national courts. The extent to which lower national courts engage with EU law is still a vastly unexplored research area.
Another gap remains in respect to legal mobilisation in the EU. The legal construction of Europe via the preliminary ruling procedure has long been seen as an interaction among litigants, national courts, and the CJEU.Footnote 211 Yet, the questions of how, why, and when EU law is mobilised, and by which actors (public interest groups, businesses, individuals), remain unanswered. Recent research efforts by Glavina uncovered the leading role of businesses (private-interest actors) in the construction of Europe, critiquing EU legal mobilisation scholarship for its limited focus on public interest groups.Footnote 212 Glavina’s effort to theorise legal mobilisation of private-interest actors alongside the inclusion of information on the parties in the IUROPA CJEU Database Platform is a start, but more research is needed into who these private-interest actors are, their strategies for mobilising EU law, and how they differ from those developed by public-interest actors.
Finally, as noted by Brekke et al., there is surprisingly little research on the role of judges’ characteristics in the decision-making of European and national courts.Footnote 213 How does a judge’s MS of origin, gender, appointing party ideology, and professional background influence the working of the CJEU? The IUROPA dataset, which contains information on the background of judges, again, serves as a useful starting point in this regard.
A final word on whether legal and empirical research can be reconciled: throughout this chapter, I have critically reflected on what empiricists and legal scholars can learn from each other. To date, there have been five AG opinions on the referral behaviour of national judges, yet none of them makes references to any of the empirical literature on the topic.Footnote 214 This once again confirms the reality that the two disciplines – law and empirical perspectives – continue to operate in isolation. Legal doctrinal research has long been criticised for lacking any (empirical) evidence to support its normative claims; empirical legal studies (ELS), as Davies noted, can offer EU lawyers a new academic home – a place of asylum – offering empirical data and knowledge of methods to make their findings applicable beyond doctrine and to have a greater impact on society and policy-making.Footnote 215 On the other hand, legal scholars bring to the table a deep understanding of complex legal structures, without which an empirical scholar ‘has no idea whether what they are measuring has anything to do with the law’.Footnote 216 As argued by Naurin et al., deciphering judicial reasoning without legal training is at least as hard as crafting credible empirical research designs without proper training in research methods.Footnote 217 EU lawyers can, in that sense, be seen as natural partners to empiricists.Footnote 218 Yet, despite this, the two disciplines continue to operate in isolation: Epstein has shown that the percentage of empirical legal articles in traditional law journals is remarkably low – 5–8 per cent, depending on the study.Footnote 219 In contrast, journals with an ELS focus are much more open to papers from other disciplines, including law (66 per cent of covered European journals fall into the ELS category).
It seems that while ELS is ready to embrace legal scholars as their natural partners, law scholars continue to act as gatekeepers of the field, whether because of ignorance or an ambition to protect the traditional art of studying law. One thing is certain: legal scholarship is becoming more open to ELS. The question is who will board the boat and who will miss it.
Open access