5.1 Introductory Notes
From various accounts of the evolution of empirical legal studies (ELS), a common story emerges – and it unfolds in three chapters; Legal Realism, Expansion, and Globalisation.
Chapter 1 starts in the 1920s with the legal realists – a group of US academics, lawyers, and policy-makers who decided to collect ‘facts’ (data) to help solve a range of societal problems (Adler and Simon Reference Adler and Simon2014; Kalman Reference Kalman1986; Kritzer Reference Kritzer2009, Reference Kritzer, Cane and Kritzer2010). The idea was to convince judges, policy-makers, legislators, and the like to reach evidence-based – not speculative-driven – decisions.
Among the original realists were some very prominent people, including Felix Frankfurter – a Harvard Law professor and later a US Supreme Court justice – and Roscoe Pound – dean of the Harvard Law School. Together they produced a landmark report, Criminal Justice in Cleveland (Pound and Frankfurter Reference Pound and Frankfurter1922), which sought to render a statistical accounting of the criminal justice system with an eye towards identifying its defects. Not surprisingly, the report is chock-full of data and recommendations based on the data.
Yet another famous team of legal realists consisted of William O. Douglas and J. Howard Marshall. Douglas had been a professor at Yale, chair of the Securities and Exchange Commission, and ultimately, like Frankfurter, a US Supreme Court justice. Marshall was a lawyer in the US Department of the Interior and later co-founder of the Great Northern Oil Co. Together they wrote an influential article in the Columbia Law Review, ‘A Factual Study of Bankruptcy’. As its title suggests, the piece houses lots and lots of data on the bankruptcy system (Douglas and Marshall Reference Douglas and Marshall1932).
After these auspicious starts, chapter 2 of the ELS story suggests the project expanded in two ways. One was the spread of ELS beyond the US borders, especially to the UK. Rather early on, for example, Radzinowicz (Reference Radzinowicz1939) deployed data to analyse sentencing patterns in England. And around the same time, Jackson (Reference Jackson1937, Reference Jackson1938) published two articles on English juries, one on the decline in civil jury trials and the other, on juror eligibility – both published in law journals.Footnote 1 Other studies soon followed.
A second expansion came in the number of topics treated. Work continued on criminal and civil justice (e.g., Rosenberg Reference Rosenberg1964; Zeisel and Callahan Reference Zeisel and Callahan1963). But studies also began to cover the legal profession and education (e.g., Carlin Reference Carlin1962; Johnstone and Hopson Reference Johnstone and Hopson1967), judicial staffing (e.g., Gadbois Reference Gadbois1968; Jacob Reference Jacob1964), the judges’ behaviour (e.g., Pritchett Reference Pritchett1948; Schubert Reference Schubert1965), and on and on.
The story of ELS’s evolution concludes in chapter 3 with ELS spreading worldwide and to most law-related subjects (e.g. Cane and Kritzer Reference Cane and Kritzer2010; van Dijck, Sverdlov, and Buck Reference van Dijck, Sverdlov and Buck2018; Wulf Reference Wulf2016).Footnote 2 Hence the title of this chapter: ‘The Globalisation of Empirical Legal Studies’. Or one might say: We’re all data analysts now.
I leave the veracity of chapters 1 and 2 to historians. My interest is in chapter 3, specifically in the question of whether, in fact, ELS has spread across the US and Europe in particular, such that it’s now an accepted, even mainstream, approach to the analysis of law and legal institutions. Hence the question mark at the end of the chapter’s title.
To develop an answer, I begin by asking what we mean by ‘empirical legal studies’; specifically, how can we characterise projects falling into this genre? The answer, I propose, focuses on the goals of the research rather than on (contentious matters of) data and methods (Section 5.2). Based on this characterisation of ELS, I inventoried academic journals to gauge the spread of ELS in leading US and European law-related journals (Section 5.3). Because the results show that the percentage of ELS articles in traditional law journals is quite small, Section 5.4 considers why ELS hasn’t gone mainstream (as the standard story suggests) and offers proposals for forward movement.
5.2 What Exactly Is ELS?
This volume centres on empirical legal studies but doesn’t that seem one word too many? Why call it ‘empirical legal studies’, and not simply ‘empirical studies’? After all, regardless of whether empirical researchers are addressing a legal question or any other, they follow the same rules – the rules that allow them to extract meaning from the information they’ve collected (Epstein and King Reference Epstein and King2002; King, Keohane, and Verba Reference King, Keohane and Verba1994). What’s more, because empirical research in law has methodological concerns that overlap with those in biology, chemistry, economics, medicine and public health, political science, psychology, and sociology, empirical legal researchers can – and do – adopt methods from these other disciplines to suit their own purposes.
Even so, in virtually every discipline that has developed a serious empirical research programme – law not excepted – scholars discover methodological problems that are unique to the special concerns in that area. Each new data source often requires at least some adaptation of existing methods, and sometimes the development of new methods altogether. There is bioinformatics within biology, biostatistics and epidemiology within medicine and public health, econometrics within economics, chemometrics within chemistry, political methodology within political science, psychometrics within psychology, sociological methodology within sociology, and so on (Epstein and Martin Reference Epstein, Martin, Cane and Kritzer2010).
As of the time of writing, there’s no ‘legalmetrics’ but there is ELS. It surely has features in common with methodological programmes in adjacent disciplines (especially when it comes to executing the studies), yet it’s also presumably distinct too. In what ways? How might we characterise ELS?
Well, substantively ELS centres on concerns distinctive to the legal community (broadly defined): law and legal institutions. In response to a prompt about ELS, ChatGPT put it this way: ‘Overall, empirical legal studies aim to bring a more rigorous and evidence-based approach to the study of law, offering insights that can inform legal practice, policymaking, and the development of legal theory.’ That sounds about right and certainly distinguishes ELS from other disciplines.
Beyond that, the various definitions of ELS offered by scholars and organisations don’t help much. Some seem way too inclusive, including, regrettably, one my co-author and I offered: research ‘based on observations of the world – in other words, data, which is just a term for facts about the world’ (Epstein and King Reference Epstein and King2002: 3). To the extent that many (most?) articles published in law journals cite some ‘facts’ to support their claims (Diamond and Mueller Reference Diamond and Mueller2010), this definition is too broad.Footnote 3 Citing a supporting fact and conducting the research to develop that fact are two different things.
Other definitions seem too narrow and exclusive, equating ELS with quantified data and ‘statistical techniques and analyses’ (Heise Reference Heise2011: 821),Footnote 4 when there are alternatives to both.
To wit (again according to ChatGPT): ‘Empirical legal studies use various quantitative and qualitative research methods to gather and analyse data related to legal questions. This can include surveys, experiments, observational studies, case studies, and statistical analysis.’
Like the other definitions, though, the Chat is focused on data and methods. But ‘data are data and methods are methods’ (Patty Reference Patty2015). None is more empirical than another, making any definition based on them actually worse than no definition, for battles will (actually always do) ensue over whether a particular study is sufficiently empirical to be deemed ‘ELS’.
Far more relevant in characterising ELS, I think, are the aims of the research. In service of, say, answering a research question or assessing a hunch related to law and legal institutions, these aims typically entail summarising systemically the information (facts) the researcher has collected and/or drawing descriptive or causal inferences (i.e., using facts we know or can learn to learn about facts we do not know).Footnote 5 Using AI and other approaches to forecast outcomes of interest at the individual or collective level, to identify patterns in law, and so on may soon join this list of ‘typical’ goals, though presently with many cautionary notes and downsides (see generally Chau and Livermore Reference Chau, Livermore and Epstein2024; Forecasting Collaborate 2023; Niblett Reference Niblett and Epstein2024).
Note that this approach to characterising ELS is agnostic about the type of information researchers use to pursue their goals or how they gather it. Whether the information is quantified or not is irrelevant, as is the plan for generating it (e.g., analysing text, surveying, observing, etc.). Where the approach is not agnostic is over the aims of the research. To the extent that the goals of summarising and drawing inferences from information cannot be met willy nilly, researchers ought to adhere to tried-and-true procedures regardless of the type of facts they intend to collect and their plan for collecting them (see, e.g., Baude, Chilton, and Malani Reference Baude, Chilton and Malani2017; Epstein and King Reference Epstein and King2002).
It is well beyond the scope of this chapter to set out those procedures, but an example from traditional doctrinal legal research should suffice to make the point. In some, perhaps most, of these studies researchers (and judges too) look over court cases to make empirical claims about whatever doctrine is under analysis. This may seem to amount to summarising the state of the case law but only in the loosest sense. For, as Baude, Chilton and Malani (Reference Baude, Chilton and Malani2017: 40–41) demonstrate, the ‘summaries’ are often just a description of a few (cherry-picked) cases listed in string cites. Only if authors conduct a more ‘systematic review’ – for example, by taking the four steps outlined in Baude, Chilton, and Malani – would this type of study fall under the heading of ELS.Footnote 6
More generally, characterising ELS based on the aims of the research has the benefit of tying ELS to the illumination of law and legal institutions and so differentiates it from bioinformatics, biostatistics, econometrics, and all the rest. And because the characterisation emphasises summarising systematically and drawing inferences from data, it differentiates ELS from traditional academic legal work, whether doctrinal, theoretical, or historical (see, e.g., van Boom, Desmet, and Mascini Reference van Boom, Desmet, Mascini, van Boom, Desmet and Mascini2018). (What is not a point of differentiation: the capacity of both ELS and traditional work to generate normative, practical, and policy implications. I return to this point in Section 5.4.)
5.3 Has ELS ‘Globalised’ (At Least in the US and Europe)?
With this characterisation of ELS in mind, has ELS indeed globalised across the US and Europe – such that more scholars are pursuing one or more of the goals of ELS and more law and legal institutions are coming under analysis?
To be sure, it’s not hard to locate evidence in support of the globalisation story. Take my corner of ELS – judicial behaviour, defined as the analysis of the choices judges make and the consequences of their choices for society (Epstein and Weinshall Reference Epstein and Weinshall2021). Globalisation on all dimensions has been just astonishing. In two decades, the field went from a niche US political science project to a true worldwide enterprise (Epstein, Šadl, and Weinshall Reference Epstein, Šadl and Weinshall2021). Virtually no apex domestic court has escaped systematic attention. And judges serving on international and supranational courts, long of interest to scholars, are now more than ever the focus of ELS research; the Court of Justice of the European Union, in particular, has generated a substantial number of interesting studies (e.g., Frankenreiter Reference Frankenreiter2017; Larsson et al. Reference Larsson, Naurin, Derlén and Lindholm2017; Ovàdek Reference Ovádek2021; Šadl and Olsen Reference Šadl and Olsen2017; Schroeder and Lindholm Reference Schroeder and Lindholm2023).Footnote 7
In a nutshell, ELS work on judicial behaviour worldwide is now so expansive that we recently edited an Oxford Handbook on the subject, with 40+ chapters covering the range of topics and showing off the key aims of ELS (Epstein et al. Reference Epstein, Grendstad, Šadl and Weinshall2024): from summarising vast amounts of information (see, e.g., Naurin and Voeten Reference Naurin, Voeten and Epstein2024; Smyth Reference Smyth and Epstein2024; Solberg and Waltenburg Reference Solberg, Waltenburg and Epstein2024) to drawing descriptive or causal claims using a multitude of data and methods (see, e.g., Bricker, Carrubba, and Gabel Reference Bricker, Carrubba, Gabel and Epstein2024; Engel Reference Engel and Epstein2024; Epperly Reference Epperly and Epstein2024) (though, happily, many gaps remain for future work).
Looking more broadly and systematically the picture is decidedly mixed – or at least that’s the takeaway from an inventory I made of articles published in academic journals.Footnote 8 The inventory is of two types of journals: those in which legal academics have traditionally published (‘conventional’ law journals), and those a bit more outré because they specialise in empirical legal studies or are discipline-adjacent to law but nonetheless are attractive outlets for legal scholars (‘less conventional’ journals). Both categories include (non-randomly selected) US-based and European-based journals (see Sections 5.3.1 and 5.3.2 below).
For each articleFootnote 9 published in 2023, I determined whether or not the author collected the information (‘data’) with an eye towards systematically summarising, drawing inferences, or both. Making the determination wasn’t especially difficult. Judgment calls chiefly came in articles making use of (unstructured) elite interviews. If the content was summarised in any way, I counted it as ELS; if the interviews were used mainly for background or insights, as in journalism, I didn’t count it as ELS.
Note that my aim here is obviously not to draw inferences about all articles in all journals. It is rather to summarise the information I’ve collected.
5.3.1 The ‘Less Conventional’ Journals
Turning to the results, let’s start with the good news for those desiring the spread of ELS: publications in the less conventional journals. I selected four based in the US: the Journal of Empirical Legal Studies, Journal of Law & Courts, Journal of Legal Analysis, and Journal of Legal Studies. The volume’s editors pointed me to European journals that are attractive outlets for empirical legal scholars specialising in EU law: European Union Politics, Journal of Common Market Studies, Journal of European Integration, and Journal of European Public Policy. Almost needless to write, the US and European journals are different in kind, with the latter less oriented towards law than the four US journals, perhaps making any comparisons apples-to-oranges. And yet in reviewing the articles in the European journals, it is striking how many are sufficiently focused on law and legal institutions that they could have been published in one of the US journals – including investigations of citizens’ attitudes towards the EU COVID-19 recovery package (Unger et al. Reference Unger, Sirsch, Stockemer and Niemann2023), the use of law in the manifestos of right-wing Eurosceptic parties (Castillo-Ortiz Reference Castillo-Ortiz2023), and the creation of the European Public Prosecutor’s Office (Schmeer Reference Schmeer2023), among many others.Footnote 10
Figure 5.1 displays the results of my small study, showing the percentage of articles in each US/Europe-based journal that could be fairly characterised as ‘ELS’ based on the approach outlined in Section 5.2.
Percentage of articles published in 2023 in less conventional law-related journals in line with the aims of ELS: systematically summarising information and/or drawing inferences. The numbers in parentheses are the total number of articles.
Figure 5.1 Long description
The X-axis represents the percentage of articles in line with aims of E L S, ranging from 0 to 100. Left. The graph titled U S based journals plots bars for J Emp Legal Studies 28, J Law and Courts 20, J Legal Studies 13, and J Legal Analysis 8. The values are 100, 90, 38, and 13, respectively. Right. The graph titled Europe-based journals plots bars for E U Politics 26, J Eur Public Policy 13, J Common Market 75, and J Eur Integration 24. The values are 100, 77, 60, and 42, respectively. Note, all values are approximated.
Begin with the US journals. All in all, 75 per cent of the 69 total articles, across all four journals, could be characterised as ELS – though differences emerge among the four. Considering its name, the Journal of Empirical Legal Studies is something of a ringer: All the articles embrace one or more of the aims of empirical research. The percentage is also comparatively high for the Journal of Law and Courts, but far lower for the Journal of Legal Studies, which publishes a good deal of (formal) theoretic work lacking an empirical component, and the Journal of Legal Analysis, which blends traditional law articles and interdisciplinary studies.
Likewise, a healthy percentage of articles published in the four European journals falls into the ELS category: 66 per cent of the 138 articles across the four journals – a percentage that isn’t significantly different from the US journals. Again, there’s some variation among the four, but even the least frequent publisher of ELS articles, the Journal of European Integration, bests two of the US-based journals.
The upshot is this: looking at some important European and US-based non-traditional journals, the globalisation story receives some support. Though the data do not allow for over-time comparisons, at least in 2023 well over a majority of the studies across the eight journals could be characterised as ELS.
But whether the ELS articles themselves reflect globalisation on other potentially interesting dimensions is a matter of interpretation, as Figure 5.2 suggests. Starting with the US bars, note that of the 52 ELS articles, 29 per cent are set in locales/cover topics other than the US. Likewise, of the 123 total authors of ELS articles, 28 per cent are affiliated with universities outside the US. Both data points, I suppose, could seem impressive for US journals.
Various (potential) dimensions of ELS globalisation in 2023 articles published in the ‘less conventional’ law-related journals by whether the journal is based in the US or Europe. See Figure 5.1 for the journals. The last three sets of bars pertain only to ELS articles.
Notes:
1. % ELS Articles = Per cent of articles published across the journals in line with the aims of ELS (systematically summarising information and/or drawing inferences). Total N of articles = 69 (US) and 138 (Europe).
2. % ELS Articles Set Outside … = Per cent of ELS articles set in countries/cover topics other than the US (US bar) or Europe (Europe bar). Total N of articles = 52 ELS (US) and 91 ELS (Europe).
3. % ELS Authors with Affiliation … = Per cent of authors with an institutional affiliation outside the US (US bar) or outside of Europe (Europe bars). Total N of authors = 123 ELS (US bar) and 194 ELS (Europe bar).
4. % ELS Articles with an International Team = Per cent of articles (with more than one author) with a US co-author and non-US coauthor (US bar) or with a European co-author and a co-author from another country (in Europe or otherwise) (Europe bar). Total N of articles with more than one author = 36 ELS (US bar) and 58 ELS (Europe bar).
Figure 5.2 Long description
The X-axis represents the percentage, ranging from 0 to 80. Left. The graph titled U S based journals plots bars for percent of E L S articles, percent of E L S articles set outside the U S, percent of E L S authors with affiliation outside the U S, and percent of E L S articles with an international team. The values are 75, 29, 28, and 14, respectively. Right. The graph titled Europe-based journals plots bars for the percent of E L S articles, percent of E L S articles set outside Europe, percent of E L S authors with affiliation outside Europe, and percent of E L S articles with an international team. The values are 66, 7, 15, and 48, respectively. Note, all values are approximated.
Then again of the 36 total empirical articles with more than one author, in only five (14 per cent)Footnote 11 did a US researcher collaborate with a non-US researcher.11 Aside from the implications for ELS’s globalisation, the low number of internationally diverse teams may bode poorly for the impact of US-ELS studies. Research shows that international teams, to be sure, have their costs (e.g., ‘communication and cultural challenges’ resulting from personal diversity that can reduce productivity) (Specht and Crowston Reference Specht and Crowston2022: 15).Footnote 12 But the benefits of cross-country partnerships (e.g., enhanced creativity and overall better decisions) may well outweigh the costs (Krammer et al. Reference Krammer, Dahlin, Doh and Potočnik2024; Stahl and Tung Reference Stahl and Tung2015; Taras et al. Reference Taras, Baak, Caprar, Dow, Froese, Jimenez and Magnusson2019).
In contrast come the journals based in Europe. Of the 91 ELS articles, published across the four journals, the focus is, yes, on the EU (40 per cent) or European countries (or some combination thereof, e.g., the EU and UK) (53 per cent): only six studies (7 per cent) are set outside of Europe (no surprise considering that the journals are not only European-based but Europe-focused). And, yes, most of the authors are from Europe (85 per cent). But, perhaps reflecting funding conditions and geography (among other factors), international collaboration between/among one Europe-based author(s) and an author(s) from another European country (or elsewhere) is nearly the norm: Of 58 ELS articles with more than one author, 48 per cent were cross-border collaborations.Footnote 13
All in all, differences emerge between Europe- and US-based non-traditional journals. But, to reiterate, the overall picture generally supports the globalisation story: The majority of articles published in the eight journals could be characterised as ELS by almost any definition or metric.
5.3.2 The ‘Conventional’ Law Journals
Where the ELS story starts to unravel is with the conventional law reviews, as Figure 5.3 shows. Paralleling Figure 5.1, the data depict the percentage of 2023 articles embracing the aims of ELS in 10 traditional law reviews: five based in the US and five in Europe.
Percentage of articles published in 2023 in traditional law journals in line with the aims of ELS: systematically summarising information and/or drawing inferences. The numbers in parentheses are the total number of articles.
Figure 5.3 Long description
The X-axis represents the percentage of articles in line with aims of E L S, ranging from 0 to 40. Left. The graph titled U S based journals plots bars for the standard L Rev 16, S Calif L Rev U S C 22, Wash U L Rev 23, Harvard L Rev 9, and Yale L J 16. The values are 43.8, 22.8, 21.7, 11.1, and 6.3, respectively. Right. The graph titled Europe-based journals plots bars for Modern L Rev 34, European Con L Rev 14, European L Open 17, Common Market L Rev 26, and European L Rev 19. The values are 8.8, 7.1, 5.9, 3.9, and 0, respectively. Note, all values are approximated.
Starting with the US-based journals, I selected two that would be considered ‘top’ publication outlets by any definition: the Harvard Law Review and the Yale Law Journal. The remaining three are also leading journals but, unlike Harvard and Yale, they come out of law schools that embrace ELS (Stanford, University of Southern California (USC), and Washington University-St. Louis (WashU)).
True to their reputed disinterest in ELS, the percentages are infinitesimal for Harvard and Yale (collectively only 2 of 25 articles, or 8 per cent).Footnote 14 This is so even though many of the articles lend themselves, even beg for, empirical analysis, such as articles on search and seizure law, public nuisance, and precedent.
Moving from the staid East Coast schools of Harvard and Yale to the more progressive (Mid)West and ELS-friendly institutions (Stanford, USC, and WashU), not surprisingly the percentages jump considerably. Overall, 17 of the 61 articles published in their law reviews (28 per cent) are ELS-oriented – with Stanford leading the way at 44 per cent (7/16).
The findings for the European-journal analysis (the right panel of Figure 5.3), which includes five journals thought to be important outlets for European and especially EU studies,Footnote 15 look a lot more like the US East Coast: Only six out of 110 articles (5 per cent) published across the five journals bear any resemblance to an ELS study – even though, like their US-East Coast counterparts – many might benefit from empirical analysis (or, at the least, some attention to the empirical literature).
Obviously, drawing any inferences about the traditional law journals from the small (non-random) sample in Figure 5.3 isn’t possible. Still, I hasten to note, the percentages in the underlying data are not altogether different from other inventories of ELS articles in the law reviews. For example, Diamond and Mueller (Reference Diamond and Mueller2010) find ‘substantial’ empirical content in about 14 per cent of US law reviews; the percentage here, for the five US law reviews, is somewhat higher, at 22 per cent. Van Dijck, Shahar, and Sverdlov’s (Reference van Dijck, Sverdlov and Buck2018) figure of 5 per cent ELS for traditional European legal journals is about the same as the 8 per cent in my data.
These are the numbers. What about the content of the articles that could be characterised as empirical? There are so few, especially in the European journals, that I can’t provide much more than a couple of impressions. Beginning with substance, parallels exist between the US and European publications; for example, articles in both, naturally enough, focus on the law community, including a small survey in the Modern Law Review of where law professors publish their articles (Manley Reference Manley2023) and an analysis in the Harvard Law Review of how judges select their clerks (Fogel, Hopes, and Liu Reference Fogel, Hopes and Liu2023). In terms of the goals of the articles, they are strikingly similar, with almost all seeking to summarise systematically the data they have collected. For example, a Stanford Law Review article inventoried references in US federal courts to the ‘political question doctrine’,Footnote 16 displaying the results in a bar chart with the references organised by year (Bradley and Posner Reference Bradley and Posner2023: 1050). In contrast, work attempting to draw inferences, whether descriptive or causal, is relatively rare in the US law reviews (but see, e.g., Fisch and Robertson Reference Fisch and Robertson2023; Nunna, Price, and Tietz Reference Nunna, Price and Tietz2023) and almost non-existent in the European journals. Whether this reflects the state of ELS scholarship in the law schools and departments, the taste of law students (who select the articles in the US), the desire of faculty to publish in conventional versus less conventional publications, or some combination thereof, who knows?
5.4 Why Hasn’t ELS Gone Mainstream? And How It Can
Leeuw and Schmeet (Reference Leeuw and Schmeet2017: 256) write that ‘empirical legal research is blossoming, if not booming’. Well, that seems to depend on the corner of ELS under scrutiny. Based on the analysis here, ELS does indeed seem to be booming in the non-traditional journals, what with about 75 per cent of the articles fairly characterised as empirical studies. But articles in the ten traditional law reviews are a different story with only 13 per cent falling into the ELS category; removing the Stanford Law Review reduces the percentage to 10. The 8 per cent figure for Europe alone is paltry.
To put it simply, ELS is hardly mainstream, at least not in the law schools. This is a sobering finding because it quite likely means that legal academics are not keeping pace with the needs of larger law-and-policy communities. Way back in the 1920s, the legal realists recognised the importance of evidence-based decision-making for the development of law and policy. The need for data and ELS studies among lawyers, judges, policy-makers, and commissioners has likely increased exponentially ever since (see, e.g., Epstein, Friedman, and Stone Reference Epstein, Friedman and Stone2015; Epstein and Martin Reference Epstein and Martin2014; Lawless, Robbennolt, and Ulen Reference Lawless, Robbennolt and Ulen2016; Leeuw and Schmeet Reference Leeuw and Schmeet2017). But you couldn’t tell as much from the traditional law reviews.
What can be done? How can ELS truly globalise or at least move to the ‘blossoming’, if not ‘booming’ phase? To me, there are a series of related challenges blocking true progress but, happily, all present real opportunities for forward movement. Below I describe three: Open the Gates (Section 5.4.1), Make ELS (More) Relevant (Section 5.4.2), and Work Collaboratively (Section 5.4.3).
5.4.1 Open the Gates
With little doubt, traditional law scholars both in the US and Europe – the ‘gatekeepers’ – have slowed the advancement of ELS by devising various ways to prevent it from going mainstream. Even in this day and age when many conventional law review articles cite some data (see Diamond and Mueller Reference Diamond and Mueller2010), the gatekeepers write articles with titles that border on the hilarious: for example, ‘Is a Science of Comparative Constitutionalism Possible?’ and ‘Empirical Research in Comparative Constitutional Law: The Cool Kid on the Block or All Smoke and Mirrors?’ – chock-full of warnings about the dangers of data and methods.
These words are not meant to belittle informed critiques of ELS, for in executing their studies, ELS researchers make scores of decisions, many of which amount to (educated) judgment calls that can (and should) be questioned. To deny that ELS blends science and art is to misunderstand the nature of the enterprise.
The problem with the gatekeepers’ bashing is that it’s not informed. It comes mostly from traditional scholars who don’t know much if anything about executing ELS research. By way of example, consider the task of defining and measuring concepts in concrete terms, which has long been targeted by the gatekeepers because it’s so easy to poke holes! You need to know next to nothing about methods or even the topic at hand to say: ‘Your measures are too simple.’ It’s also a complaint that has the ring of truth because all measurement schemes are susceptible to the critique of oversimplification; understanding the real world always requires a certain level of abstraction. But ELS researchers not only acknowledge the problem; they have developed a principled response that stresses the importance of abstracting the right dimensions for their purposes, of measuring enough dimensions of each subject to capture all the parts that are essential to their research, and, most crucially of all, carefully evaluating their choices and procedures.
The fundamental problem of causal inference provides yet another point of vulnerability that the gatekeepers are all too willing to exploit. Because this is indeed a problem without a solution it opens the door for the traditionalists to assert that ELS scholars shouldn’t bother trying to make causal claims, that they should talk only in the language of correlations or associations. Or better yet that they shouldn’t conduct data studies at all. Nonsense. Lawyers, judges, and legislators often depend on, if not make, causal inferences, and so throwing in the towel isn’t an option. Moreover, generating useful, policy-relevant research is among the things that ELS-minded legal academics do best. And, on top of it all, scholars have hardly ignored the difficulties of performing causal inference. To the contrary: they’ve spent decades devising various fixes.
I could go on, but you get the drift. A set of traditional US and European legal academics who don’t think that ELS plays to their own competitive advantage have developed all kinds of ways to prevent ELS from going mainstream – writing overly-simplistic, naive evaluations of data studies of course, but also trying to keep ELS work out of the journals altogether and lobbying against hiring empiricists on their faculties.
How can we open the gates? A couple of prospects present themselves, but education is key. Because van Dijck, Sverdlov, and Buck (Reference van Dijck, Sverdlov and Buck2018) describe in some detail various approaches for enhancing the skills necessary to undertake or even simply interpret ELS projects, let me emphasise but one here: the development of training programmes. For over a decade my colleague Andrew D. Martin and I taught a weekend course on ELS once or twice a year to law professors, lawyers, and judges – thousands of them. It’s not that we were able to teach them how to do empirical research in a compressed course. But I do think we helped them to appreciate the work. Along similar lines, Daniel Naurin and Urška Šadl have been running more targeted workshops for early-career researchers.
This is all to the good. But, obviously, given the law review percentages in Figure 5.3 and the presence of gatekeepers, there’s only so much a handful of workshops can accomplish. We need to scale up.
Many avenues exist for so doing, but surely one is to train the next generation of students by pushing law faculties to incorporate into their curriculum at least one course on empirical research – as many top US law schools now do (including USC, Stanford, Michigan, and Virginia). Increased use of online programming suggests that teaching these courses to very large numbers of students, as well as to judges, lawyers, and policy-makers, is well within the realm of the possible.
Whatever the approach, developing and expanding courses could have many happy byproducts. For one, students with ELS skills will be more marketable than those without them, considering the increased demand for data analysis and data science. Virginia Law justifies its (quantitative) ELS course in these terms: ‘Legal practice and research increasingly involve analysis of big data to resolve legal questions, and the importance of quantitative analysis is likely to continue growing in the coming years.’ Second, faculty would benefit. Offering empirical courses will require law schools to hire ELS scholars (even if to teach solely online) who, in turn, could serve as resources for the faculty (even if from a distance).
5.4.2 Make ELS (More) Relevant
Creating these opportunities would be great. But we can’t stop there; scholars need incentives to shift their mode of research. And that takes me to my second recommendation: make ELS more relevant.
Many scholars, but perhaps especially legal academics, hope that their research will have both academic impact and policy consequence. But a non-trivial fraction of law professors just don’t seem to see the relevance of ELS for the analysis of law and legal institutions. And for this reason, the so-what question looms large. Or to put it in terms of Zglinski, Chapter 1 in this volume, ‘What is the point of empirical legal research’?
Our responsibility is to address this – the so-what – question by pointedly and deliberately demonstrating the relevance of our work for law, as well as its real-world implications. The legal realists of the 1920s may be “‘distant relatives” to those presently engaged in empirical legal scholarship’ in terms of their data and methods (Heise Reference Heise2011: 822; see also Schlegel Reference Schlegel1995), but the realists were really good at articulating the importance of their projects for the betterment of society; and my hope is that today’s ELS scholars will become equally trained on the real-world implications of their work with the twin goals of increasing its policy relevance and, I believe, its scientific impact.
How? Let me start with how not to proceed: We shouldn’t reorient our research questions to address transient policy concerns. So doing is a bad idea because it’s patently obvious that career payoffs in law (and the social sciences for that matter) come not from asking narrow questions about fleeting phenomena; the payoffs follow from theoretical, substantive, and, yes, methodological breakthroughs. If, and only if, we ELS scholars think we can make these sorts of contributions will (should) we undertake research with the potential to influence law and public policy.
So, if we ELS scholars don’t need to revise our research questions, how can we heighten both the scientific impact of the work and its importance for policy and society? I propose three paths; (1) clarify the real-world implications of our research, (2) assess speculation in the law reviews and court decisions, and (3) develop collaborations with journalists.
First and most obvious, we should take the extra step and state the contemporaneous importance of our research. This should be easy because there’s almost no ELS study lacking in real-world implications and potential contributions to society. It’s just that we academics have failed to notice, much less to develop them ourselves.
To me, there’s a simple fix. Instead of concluding our articles with the almost never illuminating call for ‘more research’, how about a section that contemplates the consequential nature of our findings? This recommendation should not be altogether unfamiliar because funding agencies often consider the potential ‘broader impacts’ of proposals when deciding whether to fund them. Here’s the US National Science Foundation’s policy:
When evaluating NSF proposals, reviewers will be asked to evaluate … the Broader Impacts of the study, [including] the potential to benefit society and contribute to the achievement of specific, desired societal outcomes.
Sometimes, perhaps often, the ‘impacts’ or implications will centre on institutional arrangements; for example, should we change the method of selecting judges? Others might focus on improving public policy. Whatever, developing implications should not be hard because often it involves little more than thinking through the findings, although more systematic approaches are possible (and perhaps preferable) too (see Bartels and Bonneau Reference Bartels and Bonneau2015).
A second path to making our work more relevant is to assess rigorously the speculation and (unsubstantiated) empirical claims – call them hypotheses – so rampant in the law journals, policy reports, and even judicial decisions. For example, based on little more than intuition and a few anecdotes, Scheppele (Reference Scheppele2013: 559) not only makes empirical claims but suggests we should jettison time-tested rule of law (governance) indicators in favour of ‘forensic legal analysis’:
Around the world, organizations are developing indicators of the rule of law that measure whether countries have more or less of it. However, such indicators often miss something crucial: interaction effects among the elements that, if recognized appropriately, would send a country from the top to the bottom of the scale …
Although checklist-based rule-of-law indicators may seem an advance over fuzzy definitions, only forensic legal analysis can tell how a constitutional order works in practice. And ultimately that is the rule-of-law indicator that really matters.
Is Scheppele right? Would an ELS project support her claims? This is but one example. There are so many others that it’s hard to pick and choose (see Epstein, Friedman, and Stone Reference Epstein, Friedman and Stone2015), but here are a few untested empirical claims in US court decisions.
Campaign Finance. When it ruled that corporations and other outside groups could spend unlimited funds on elections, the Supreme Court declared ‘The appearance of influence or access … will not cause the electorate to lose faith in our democracy.’Footnote 17 Really?
Abortion. In upholding restrictions on abortion, the Court observed ‘While we find no reliable data to measure the phenomenon, it seems unexceptionable to conclude some women come to regret their choice to abort the infant life they once created and sustained.’Footnote 18 True?
Diversity Programmes. In expressing his agreement with the Court’s decision to invalidate university admission programmes that consider race, Justice Thomas had this to say: ‘Large racial preferences in college admissions stamp Blacks and Hispanics with a badge of inferiority.’Footnote 19 Is he right?
At least in the US, these are big topics that could lead to terrific studies – even theoretical and methodological breakthroughs. And it’s probably just as easy to identify equally important but unsubstantiated ‘hypotheses’ in the decisions of the CJEU, not to mention in the many law review articles on the Court and the EU more generally.
The final path to increasing the policy relevance of ELS is to develop collaborations with journalists. Such partnerships can perform an educative function, teaching the public, its elected representatives, and even the judges themselves about how law and legal institutions work on the ground, which, in turn, could promote positive societal change. Working with journalists also helps to promote and publicise ELS research thereby legitimising it among traditional law scholars (so many of whom desire to produce impactful work). In other words, partnering with journalists is yet another way to answer the so-what question.
How can we ELS scholars initiate such collaborations? One approach is to contact the relevant reporters and email them our studies. If the journalist is empirically oriented, we might get lucky and the journalist will write a story about our project. That happened to us recently. We wrote a paper about the US Supreme Court’s lack of deference to the president (eventually published as Brown and Epstein Reference Brown and Epstein2023), which was rather ‘scientific’ – full of graphs and even statistical results. But the reporter, Adam Liptak, read it anyway, asked us some questions, and wrote stories for the New York Times that received a fair amount of attention (Liptak Reference Liptak2022, Reference Liptak and Parlapiano2023).
Honestly, though, I don’t know if there are a lot of reporters like Liptak – reporters willing to take the time to read and understand dense ELS papers. For this reason, I offer a second recommendation: write a digestible, easy-to-understand version of your study in the form of a short report. You don’t have to dumb it down, but you do have to eliminate the jargon so that people without training can grasp and appreciate your major findings.
For the New York Times, we write such reports, chock-full of data and narrative, at the end of each US Supreme Court term. Sometimes our analysis drives the story of the term and even occasionally makes the front page.Footnote 20
Another example are studies for journalists summarising the results of research papers. One I’m particularly proud of is our report on free speech,Footnote 21 which led to an article in the Times that generated over 1,500 comments (Liptak Reference Liptak2018) – that’s about 1,500 more than the modal number of citations to an article published in a social science or law journal.
Do these reports take time and effort; and are they stressful to produce because of the tight deadlines? Yes and yes. But I have no doubt that more translational work for ELS studies will yield equally high rewards.
To summarise: clarifying the real-world implications of our work, assessing speculation in the law reviews and court decisions, and developing collaborations with journalists are three specific suggestions aimed at making our research more relevant. The more general point is that with a little effort on our part, not only can our ELS work make a difference in academic circles but in the wider world. We have the skills and tools to do just that. We should.
5.4.3 Work Collaboratively
So far, my proposals focus mostly on our connection with external audiences. My last suggestion is aimed at us: the ELS community. Widespread agreement exists among ELS scholars that data infrastructure for analysing law and legal institutions has not kept pace with interest in these subjects (see, e.g., Weinshall and Epstein Reference Weinshall and Epstein2020). From there, consensus breaks down – with deep divisions over exactly what form the infrastructure should take, for what purposes it should be developed, and on and on.
These differences and debates matter to some people, but, honestly, what a waste of time: as I wrote earlier, data are data, methods are methods. As long as infrastructure can advance knowledge and accelerate discovery, these are differences without meaning. Put another way: moving forwards with ELS requires us to put aside our disagreements, to work together for the common good. Because suggestions for so doing have been offered in other places (Epstein, Šadl, and Weinshall Reference Epstein, Šadl and Weinshall2021; Staton Reference Staton and Epstein2024; Weinshall and Epstein Reference Weinshall and Epstein2020), suffice it to note that IUROPA’s CJEU project is a model.Footnote 22 And contributors to this volume have equally great ideas for forwards movement along these lines.
If we can put these ideas into effect and collaborate as a community of scholars, I have no doubt that we will rewrite chapter 3 of the ELS story, eliminating the question mark and realising the true potential of empirical legal studies.
Open access