Reflecting its political salience, European migration law (EML) has grown significantly as a research field. So far, however, the field has been dominated by doctrinal approaches and methodological debates centred on discussions about more teleological versus more statist orientations.Footnote ¹ At its outset, however, EML presents certain methodological challenges for traditional legal approaches. EML is a composite field spanning multiple different regimes, most notably international refugee law, international and European human rights law, EU law, but also ancillary bodies of law such as the law of the sea, aviation law, and transnational labour law. Moreover, EU law in this area is dependent on legislative harmonisation between Member States over immigration. The latter has proven difficult to secure with respect to EU external borders and policies for third-country nationals, given that these are areas where states have traditionally sought to preserve sovereign discretion.Footnote ² EML has thus moved from earlier hopes for EU supranationalism to face ‘reluctant harmonisation’,Footnote ³ with a range of issues simultaneously governed by EU, national, and international law. Whether due to attempts by Member States to recoup national sovereignty through national measures, or the relative ineffectiveness of regional rules in some areas, EML could almost be characterised as ‘organised hypocrisy’, with long-standing norms or competence decisions that are frequently undermined, as an enduring attribute of the EU legal order.Footnote ⁴

Whilst extending from a regional framework of EU law, the lack of effective harmonisation and interaction with other international and regional legal regimes have created spill-over effects, whereby EML come to be characterised by indeterminacy, which undermines its systemic qualities, and makes it difficult to disentangle.Footnote ⁵ On the one hand, EML is horizontally fragmented as the legal protection of migrants in the European space now depends, for a large part, not on EU law but on the rights afforded by the European Court of Human Rights (ECtHR) and national courts.Footnote ⁶ On the other hand, EML is vertically fragmented, as despite decades of attempts to encourage trans-national judicial dialogue, judges have difficulty aligning on similar questions of EML.Footnote ⁷ It can thus be difficult to adequately capture the inter-operations and machinations of EML by focusing on one regime alone, particularly as EU courts promote a discourse that champions the autonomy of the legal order.

Whilst EML is hardly the only area of European law subject to this type of multi-level implementation problem, the issue is particularly acute considering the politicisation of migration and asylum law across virtually all European countries. Asylum law is perhaps the most salient example in this regard. The EU legal framework presumes judicial harmonisation between national judges and decision-makers of EU Member States, but this has never materialised in practice.Footnote ⁸ This reflects not only the residual sovereignty left to Member States in terms of how to design and populate national asylum institutions, but also the fact that legal decision-making in this area hinges on somewhat ambiguous level provisions and incomplete evidence.Footnote ⁹ Research has thus shown that legal decision-making on asylum is subject to inter-subjective assessments and bias, which enable ‘local yardsticks’ to determine who is entitled to the protections of EU law.Footnote ¹⁰ This has resulted in a high level of variability between Member States which contributes to gross outcome variations, such that ‘a person can have 90 per cent chance being accepted as a refugee in one EU country, whilst her chances are virtually nil next door’.Footnote ¹¹

EML scholarship has long recognised and lamented this situation, yet dominant doctrinal methods rarely, if ever, engage these issues head on. On the one hand, approaches that focus on EML as deriving from the broader operational structure of EU lawFootnote ¹² or the EU legislative framework tend to neglect or downplay normative developments that take place under adjacent regimes of law.Footnote ¹³ On the other hand, research that adopts an inter-regime focus easily loses sight of the diversity or incongruence of approaches at the national level.Footnote ¹⁴ Whilst each of these approaches has merits, they assume that that legal decision-making will take place in accordance with the rule of law, but legal methods themselves cannot address the behaviour of the norm user, particularly when decision-making inconsistency is endemic to the law.

At a more general level, EU legal method has been criticised for its myopic focus because it derives its standards as ‘mimesis of judicial discourse’, and thus tends to reproduce certain narratives.Footnote ¹⁵ This is, in a sense, a classic criticism of doctrinal legal method as a system of research set with certain epistemological limitations as it focuses principally on finding what is the valid law.Footnote ¹⁶ As de Witte notes, ‘writing about EU law means exposing which norms of EU law exist on a given subject, how they interact, and how they are put into practice by the legal system.’Footnote ¹⁷ To properly heed that call in an area such as EML may ultimately require one to move beyond doctrinal research. To this end, a recent ‘empirical turn’ is slowly taking foothold and opens up new ways of researching EML law, creating different opportunities and epistemological commitments to the mainstream of EU law scholarship.Footnote ¹⁸ Scholars have employed socio-legal,Footnote ¹⁹ ethnographic,Footnote ²⁰ and political science methodsFootnote ²¹ to open up space for thinking on what can be known about EML, how it is applied, and individuals that are subjected to it. Yet, for the most part, this literature still adopts an external approach for analysing EML, which does not offer means for disentangling the broader normative framework, nor its inherent legal problems.

In this contribution, we thus seek to outline an empirical research agenda for EML that puts the law first in its analysis but enhances analytical capacity through the power of computation.Footnote ²² To do so, we build from the existing computational turn in legal research, which in an EU law context has been employed to analyse large corpora of EU case law for well over a decade.Footnote ²³ Pioneering such an approach in EML, however, requires further methodological considerations.Footnote ²⁴ Reflecting the particularities of the field, it needs to be attuned to both multi-sited case law at the international level, and the vertical production of case law across international, regional, and national levels. Moreover, reflecting the existing state of the art, we seek to develop a computational legal research agenda that can carry novel insights for a largely doctrinally oriented audience. That is, our ambition is both normative and empirical and has policy implications by helping to enhance protections for individual migrants and asylum seekers and foster more equitable asylum systems.Footnote ²⁵ The stakes are certainly high, and we remain wary to promote computational legal methods as a panacea, and thus in Section 19.4 we articulate some of the limits of computational legal research when it is projected as a stand-alone method absent of cross-methodological and interdisciplinary exchange.

The chapter is structured as follows. Section 19.1 introduces the computational turn in the legal discipline before outlining its developing applications in EU law studies. Section 19.2 investigates the use of case citation networks to analyse the lack of judicial dialogue between national judges on questions of migration law. Section 19.3 explores the potential for forms of machine learning to open the black box of asylum law judging. Section 19.4 concludes by noting some of the benefits and limitations of our approach and canvasses some further possible opportunities in going forward.

19.1 Computational Methods for European Migration Law

Computational analysis when applied to law treats law as data. Computational law forms part of the field of text analytics, ‘the discovery of knowledge that can be found in text archives’, or more specifically, legal analytics, the process of ‘deriving of substantively meaningful insight from some sort of legal data’.Footnote ²⁶ The predominant mode of computational analysis of legal materials now focuses on ‘translation of legal texts into machine-readable data capable of quantitative analysis to determine the content, causes, or consequences of legal decisions’.Footnote ²⁷ Legal scholars have been increasingly drawn to new opportunities arising from computational methods by enabling tasks that would simply not be possible without the processing power of computation, and using machine learning to develop the scale and scope of legal research.

Computational law aims to identify recursive patterns in legal materials by coding legal texts as data and analysing these texts with algorithms. In principle, no forms of ‘text’ are off limits for this line of research.Footnote ²⁸ The majority of this work has thus far taken the form of coding written legal texts, such as judicial decisions, statutes, and international treaties, into a machine readable form and then analysing the texts on the basis of texts which are contained in the documents.Footnote ²⁹ Computational law thereby allows researchers to develop efficiencies, synergies, and more in-depth insights than traditional methods, with a focus that can be extended from the practice of law and its concern for the normative application of legal principles, to the study of the nature and operation of the legal system itself.Footnote ³⁰ In this way, a computational approach can offer fresh insights on traditional legal questions but also questions which have traditionally been approached through philosophical or social scientific forms of analysis.

The computational turn as emerging in the legal discipline draws from rich traditions of empirical method in political science, the digital humanities, and empirical legal research.Footnote ³¹ In reflection of a wider turn to interdisciplinary methods, both EU law and international law scholarship is warming up to treating law as data. In one respect, this represents an extension of the ‘empirical turn’ in legal scholarship’,Footnote ³² which has sought to ‘provide scholars with tools to gain new facts, see existing ideas through a different lens, and engage in a more nuanced analysis’ of law and its practices.Footnote ³³ At the same time, it speaks to more pragmatic objectives of helping lawyers to find ‘what works’, as an essential motivation for the turn to more realist agendas.Footnote ³⁴ In this sense, the computational turn in law is ultimately a product of, and responsive to, profound changes in the way we receive and process legal information, as law has itself become ‘a “big data” phenomenon’.Footnote ³⁵ However, computational legal analysis is not necessarily at odds with more traditional and practice-oriented approaches. It can also operate in tandem with doctrinal methods, serving to enhance ‘the validity, reliability, and transparency’ of doctrine, whilst preserving its capacity for conceptual analysis and attention to legal detail.Footnote ³⁶

What computational law adds to existing doctrinal and empirical legal research is then the ability to broaden analysis in a way that permits greater ability to ‘test, validate, and refine existing frameworks’.Footnote ³⁷ It enables researchers to ‘provide empirical evidence to validate hunches and prove legal intuitions correct’,Footnote ³⁸ or to ‘ask new and more ambitious questions about how international law works’.Footnote ³⁹ Computational law further renders legal research scalable and empowers research to study law with unprecedented depth and breadth’,Footnote ⁴⁰ and gain ‘a more stable and comprehensive quantitative basis’ for incisive analysis of a research question.Footnote ⁴¹ Computational law also further promotes efficiency in research, as computers replace assistants for repetitive tasks thus enabling ‘faster, methodical, more efficient leafing through case law’.Footnote ⁴²

These gains are being rapidly realised in EU law scholarship, in part, because EU law documents come in machine readable form and are highly structured, and are thus an ideal testing ground for these technologies. Computational text analysis requires text in digital form, and the large corpora of EU law is freely available online. These documents can be downloaded or ‘web scraped’ using Python and R. With coding, computational methods can thereafter be employed to investigate simple features such as the length of texts, to more advanced analyses of discursive elements that can be identified through instruction or more autonomous forms of machine learning.Footnote ⁴³ The methods develop with such pace that a literature review by time of publication will be out of date, and in particular now with the increasing utility of Large Language Models (LLMs).Footnote ⁴⁴

Existing computationally driven research on EU, however, has overwhelmingly focused on the role and production of the Court of Justice of the European Union (CJEU). Consider, for example, the pioneering analysis of precedent information by Mattias Derlén and Johan Lindholm,Footnote ⁴⁵ Urška Šadl’s use of network analytics to measure how precedents change and evolve,Footnote ⁴⁶ or Artur Dyvre’s explorations of the role of institutional actors in shaping EU law.Footnote ⁴⁷ This body of scholarship has foundationally shown how the use of computational legal methods can challenge conventional wisdoms in EU law, not least, on how EU legal institutions build their legal authority, and how the legal system grows.Footnote ⁴⁸ A new generation is joining this fold by using computational methods to explore fundamental issues of EU constitutional law and in particular, relations arising from the role of national courts.Footnote ⁴⁹ National courts are of course not a new field of research for empirical legal studies, though work in this area often remains similarly siloed as it tends to focus exclusively on legal issues of EU law. There remains unrealised potential to leverage the power of computational methods to amplify research scale and scope, thereby integrating multiple different sites or sources of law at a level that would previously have been thought impossible.Footnote ⁵⁰ Further inroads for exploring the dynamic nature of EU law could open up by using the same methods to analyse the practices of administrative and quasi-judical bodies that apply EU law as the frontline of engagement with national legal orders. Building on recent advances in this regard, we seek to demonstrate the purchase of computational methodologies specifically addressing the traits of EML law and practice.

19.2 Citation Networks and the Lack of Judicial Dialogue of Migration Law Judges

EML is composed of a patchwork of laws arising at regional and national levels, which together comprise a loosely knit normative framework pertaining to a legal subject. It is not a legal system in the traditional sense of the word, nor can it be properly conceived as its own legal order. Rather, it operates as a transnational field of practice that combines norms from a number of different regimes, in particular, with the significant overlap of migration, asylum, and human rights normative bases. It is perhaps better thought of as a ‘legal space’ characterised by ‘topologies’ where multiple legalities co-exist, as they are multiple overlapping norms which may be relevant to any legal problem.Footnote ⁵¹ But it is also a ‘topographical’ space because its terrain is not flat, nor is it necessarily neutral subject to networks and hierarchies that informally comprises varied corpora of EU law, regional human rights, and national immigration law – which in any given case may move to ascendance, to name but a only few core elements.Footnote ⁵²

But this can also make EML difficult to navigate for scholars and practitioners alike. There is no central court for interpreting and implementing EML, simply because there is no court which holds sole jurisdiction over such a complicated legal arrangement. Despite early hopes that the CJEU could become a ‘new refugee court’Footnote ⁵³ following foundational cases like Y and Z,Footnote ⁵⁴ and X, Y, and Z,Footnote ⁵⁵ the court has tended to adopt a minimalist approach to fundamental issues, such as the right to seek asylum enshrined in Article 18 of the Charter of European Union.Footnote ⁵⁶ In addition, on several key juridical issues the court has simply shied away from establishing jurisdiction, such as the conclusion of migration agreements with third countries,Footnote ⁵⁷ or the responsibility of the Union’s own border agency in the context of migration control.Footnote ⁵⁸

The lack of a centralised mechanism for authoritative judicial interpretation over commonly worded legal protections has not meant an absence of international and legal case law production, however. Instead, EML has largely developed through processes of ‘surrogate judicialisation’ that functions like a ‘work around’, whereby, for example, the UN treaty bodies at international levelFootnote ⁵⁹ and the ECtHR at regional levelFootnote ⁶⁰ have taken up a large portion of rights complaints from immigrants in the EU regional area – both from the vantage points of regimes originally not intended to focus on migrant rights. Migration law scholars have tended to celebrate and embrace this development, insofar that is has extended a new terrain of international fora for migrants and refugees, who traditionally have lacked access to redress beyond national avenues.Footnote ⁶¹ Others have warned that the judicialisation of EML through human rights risks becoming a cause célèbre, pointing to how the expansion of human rights case law for migrants coincide with increasing political backlash.Footnote ⁶²

From a more theoretical vantage point, these developments show how EML has become constitutively entangled – to the point where it is practically difficult to isolate its constituent parts because the proliferation of cross-cutting case law connections has significantly reshaped interpretation in the underlying regimes.Footnote ⁶³ The epistemological challenge of cognising a ‘European’ migration law as a such has been recognised by other scholars, insofar as it privileges a certain way of looking at migration law at the expense of other legible paradigms.Footnote ⁶⁴ However, the broader epistemological challenge also translates to a methodological problem. If we tend to isolate only parts of EML through a singular regime optic, we may fail to properly undergird its normative potential. In practical terms, it may also be unduly time consuming for migration lawyers to keep abreast with normative developments across multiple different regimes, and to comprehensively scope the connection points most conducive to a given case.

One way in which computational methods can assist EML is thus by helping create ‘coherence out of chaos’Footnote ⁶⁵ with less selective or other types of bias than traditional doctrinal legal methods. In our previous research, we have used computational methods to map the human rights turn in international migration law using case citation network analysis. Network analysis is a tool for representing connections between ‘nodes’ that capture the relationship between types of information in a network and has been applied to analyse large bodies of case law by tracing connections through citations between legal decisions.Footnote ⁶⁶ Whereas doctrinal analysis typically focuses on ‘leading cases’ and controversial decisions, network analysis enables researchers to reveal less obvious links, and to examine their significance as well as the network’s overall characteristics.

For our analysis we identified the penetration of migration issues in the ECtHR’s jurisprudence by scraping the data for a set of keywords: Refugee; Asylum; Refouler; [^-n]refoulement; Non-refoulement; Nonrefoulement; Immigration; Deport; Migrant. This gave us a set of 3,273 cases (nodes). The cases were then connected through case-to-case citations (edges), but many of these were never cited and are thus peripheral to the network (unless they are very recent cases). The main body of cases that made up the citation network consisted of 1,174 cases. The network had a total of 4,613 edges (connections). We then calculated the most important network features of each node in the network, and through a number of different metrices calculated high network scores. High network scores in this instance are driven by the frequency of court citations; the more the case is cited, the higher the network score. For this purpose, we then used the ‘Betweenness’ and ‘Eigenvector’ methods to detect network centrality as the measure of influence of a node (i.e., a case) in the network. We then used this data to calculate the ECtHR’s most cited decisions in network of migration decisions and represent them graphically.

Some of the cases in Figure 19.1 may be more familiar to EML researchers than others. Saadi v. Italy is likely commonly cited by the ECtHR for its well-known findings on the absolute nature of non-refoulement. Čonka v. Belgium might also be familiar for the proposition that effective remedies on deportation must hold a suspensive effect under the terms of the Convention. M.S.S. v. Belgium is a cardinal case of EML for its examination of the compatibility of transfers under the Dublin Regulation, whilst Salah Sheekh was an important case on internal protection alternatives, whilst Üner v. The Netherlands affirmed importance principles for cases concerning the expulsion of long-term immigrants who had been convicted of criminal offences.

Figure 19.1 Case citation network of the ECtHR’s migration law jurisprudence.

Figure 19.1Long description

Each node represents a specific legal case, with the case name and involved parties labeled. The size of the nodes varies. The cases involve various countries, including Italy, the United Kingdom, Russia, Belgium, Austria, the Czech Republic, the Netherlands, Switzerland, Latvia, Turkey, Georgia, and Bulgaria. The cases include Saadi versus Italy, El-Masri versus The Former Yugoslav Republic of Macedonia, Othman Abu Qatada versus The United Kingdom, Maslov versus Austria, Muminov versus Russia, D H and Others versus The Czech Republic, Uner versus The Netherlands, Na. versus The United Kingdom, Shamayev and Others versus Georgia and Russia, Al-Skeini and Others versus The United Kingdom, M S S versus Belgium and Greece, Abdolkhani and Karimnia versus Turkey, Nasrulloyev versus Russia, Hirsi Jamaa and Others versus Italy, Nada versus Switzerland, A. and Others versus The United Kingdom, Conka versus Belgium, Saadi versus The United Kingdom, and Stanev versus Bulgaria.

Identifying case law on the basis of citations can serve theoretical and practical aims. In our study, we further showed how this type of research can be used to assess how much weight the court ascribes to a migration-specific case but also to identify case law which might be relevant for a legal argument. By constructing networks for the Court’s Article 1, 3, and 8 case law, we further showed how migration-related decisions are amongst the most cited of the Court’s case law on these articles, thus evincing their role in building wider jurisprudence. Other scholars have analysed cross-citation on migration issues between the ECtHR and CJEU.Footnote ⁶⁷ Further research could explore the interaction of EML with regimes outside the European spaceFootnote ⁶⁸ as well as emerging normative issues associated with climate change.Footnote ⁶⁹ In this way, network analysis can tackle horizontal problems of entanglement by helping to identify the extent of normative spread, coherence, and conflict beyond single regime analysis.

First, network analysis provides empirical evidence of authoritative judicial practice. Networks can make visible the relative importance of precedent by identifying centrality in the branches of the web in a network,Footnote ⁷⁰ or its density across similar legal issue areas.Footnote ⁷¹ Network analysis can also identify arbitrary authority as ‘the rich get richer’ – citation accrues subsequent citations as a highly cited precedent.Footnote ⁷² In this way, the technology can be used to ascertain if citation has increased or decreased and the ‘tipping points’ where precedent becomes ‘good law’ in a network.Footnote ⁷³ Mapping the web of case law thus also provides greater ability to deride outlier interpretations. For instance, do restrictive decisions on extra-territorial jurisdiction, such as the ECtHR’s decision in Bankovic v. Belgium, reflect widespread judicial practice across jurisdictions, or is Al-Skeini v. the UK now the leading precedent? EML scholars and lawyers gain a stable empirical basis from which cases can be argued across jurisdictions and a more encompassing overview of areas of law that are ripe for further doctrinal analysis.

Second, network analysis can be used to map the evolution of legal principles over time and thus help to identify the factors or events that seem to drive the process of judicial cross-citation.Footnote ⁷⁴ It can be used to identify the incremental reach of law over specific factual issue areas, such as the extension of legal protection over specific classes of specially protected persons.Footnote ⁷⁵ For example, network analysis could reveal the scope of the ‘particular social group’ legal criteria for asylum protection across European countries. Network analysis could also be coupled with techniques of natural language processing to extract accompanying text surrounding citation of a precedent in order to assess the factors that might constrain a court in following a precedent. Such an analysis could help to identify erroneous restraints that inhibit the transposition of precedents due to perceived differences in national legal traditions.Footnote ⁷⁶

Finally, network analysis can be used to detect whether the legal precedents that are cited in legal scholarship reflect their actual importance in a given area of jurisprudence. For instance, Olsen and Frese have used network analysis to identify the cases most frequently cited in the ECtHR’s Article 14 discrimination cases, and compared the findings to British, French, and German textbooks.Footnote ⁷⁷ The authors found significant discrepancies in what is cited in the textbooks with what is actually cited by the Court, thus reflecting an incomplete or outmoded picture of current law. This problem is arguably accentuated in EML as the discourse is heavily dependent on a small number of seminal textbooks and key case law data bases sponsored by EU institutions.Footnote ⁷⁸

Network analysis thus offers significant potential to build a more cohesive EML through doctrinal analysis or to highlight cross-jurisdictional differences and patterns in interpretation in order to narrow targets for litigation. In a field where national courts remain the principal drivers of interpretative developments in international law this is particularly important.Footnote ⁷⁹ Language and procedural differences, as well as the lack of trans-judicial dialogue, means that national judges rarely talk to each other on many core aspects of the law, largely because there are often inherent cultural – and constitutional – differences on issues that broach questions of national sovereignty.Footnote ⁸⁰ Enhancing scale through computational methods will of course not solve this but could make a significant step in helping to promote a better understanding of EML. Other methods, moreover, can also be invoked to examine decision-making heuristics amongst and across different national law judges, which we now turn to in Section 19.3 below.

19.3 Data Driven Analysis of Adjudicator Bias in the Common European Asylum System

A large part of doctrinal research on EML is framed towards elaborating the legal norms which are impugned in migration and asylum cases, such as the cardinal principle of non-refoulement, which constitutes a quintessential focus of EML. However, the design of asylum and expulsion procedures are largely unregulated by international law,Footnote ⁸¹ and even in states with similar legal traditions asylum procedures can be very different.Footnote ⁸² The Asylum Procedures Directive has proven difficult to implement, reflected in a widening gap between EU acquis prescriptions and national laws.Footnote ⁸³ Implementing effective asylum procedures is furthermore constrained by epistemic uncertainties, which enable high degrees of discretion in determining who is entitled to protection.Footnote ⁸⁴

Thus, despite decades of moves towards regional harmonisation, the chance of receiving asylum in Europe often appears so arbitrary that it is an ‘asylum lottery’.Footnote ⁸⁵ The UN Refugee Agency (UNHCR) states that within the EU ‘recognition rates for the same asylum-seeking populations can vary from between 1 per cent to over 50 per cent’,Footnote ⁸⁶ with studies suggesting that the EU has had only a limited impact on asylum outcomes,Footnote ⁸⁷ and it is one of the principal causes of ‘asylum shopping’, which tends to exacerbate the problem by encouraging a ‘race to the bottom on asylum policy’.Footnote ⁸⁸ What makes refugee status determination (RSD) as the last great enigma of EML scholarship so recalcitrant? Refugee lawyers have been troubled by outcome variation for quite some time, and despite a rich literature investigating its origins, no clear solutions have emerged for the problem.Footnote ⁸⁹ Legal scholars suggest inconsistency may arise through differences in institutional design, procedures, and legal interpretations across or within national asylum systems.Footnote ⁹⁰ Others point to differences in standards of evidenceFootnote ⁹¹ or problematic use of fact-finding reports.Footnote ⁹² Some propose that variation can arise from a lack of financial resourcesFootnote ⁹³ or interference to judges.Footnote ⁹⁴ The issue is thus not easily resolved within the internally focused confines of doctrinal legal scholarship.

Social scientific research suggests that the law is just but one factor in outcome variation. Some scholars argue that rules and standards can be applied in an ad hoc fashionFootnote ⁹⁵ or that the RSD is innately intersubjective.Footnote ⁹⁶ Asylum procedures are thus prone to forms of explicit and implicit bias,Footnote ⁹⁷ gendering dynamics,Footnote ⁹⁸ performative scripts,Footnote ⁹⁹ and cultural misrecognitions between applicants and decision-makers.Footnote ¹⁰⁰ Political science research suggests that asylum decisions might be impacted by legal extraneous factors such judge’s party affiliation and political events,Footnote ¹⁰¹ whilst medical studies emphasise the impact of stress amongst decision-makers and asylum seekers.Footnote ¹⁰² In sum, as Gorlick rightly notes, ‘perhaps there is no standard answer for differential recognition rates among like-minded states’.Footnote ¹⁰³ However, existing disciplinary perspectives from social sciences and law on this problem are rarely integrated, which limits the findings and prevents wider theorisation of the issues at stake.Footnote ¹⁰⁴

Computational research on outcome variation in RSD, however, holds significant potential to broaden our understanding of the problem on a more stable empirical basis. Early steps have been taken in this direction by empirical legal researchers working in a number of jurisdictions. In a seminal study, Ramji-Nogales et al. found significant variance in asylum decision-making by US judges even within the same sub-national courts, with evidence that ‘one judge is 1,820 per cent more likely to grant an application for important relief than another judge in the same courthouse’.Footnote ¹⁰⁵ Similarly, Rehaag’s study of Canadian asylum found one judge 50 times more likely than other judges in the same court to grant in favour of asylum applicants,Footnote ¹⁰⁶ whilst in Australia, Ghezelbash, Dorostkar, and Walsh found that one judge had a ‘success relative to the median’ that was 185 per cent higher than judges at the mean level (0), that is, a success rate of 23 per cent compared to the success rate of 7 per cent average.Footnote ¹⁰⁷ However, the research is currently empirically divided across national contexts, which makes it difficult to explain outcome variation in asylum decision-making between countries.

In our previous research, we have undertaken a comparative analysis of some of the factors which may be driving outcome variations through a dataset of 15,535 asylum cases tried by the Danish Refugee Appeals Board in the period 1995–2020 (RAB).Footnote ¹⁰⁸ We analysed these case files using Natural Language Processing (NLP) models to extract information from the text through regular expressions (a technique to define and match textual patterns that we were required to use because Danish language NLP is underdeveloped.) The case files are in the form of a judicial decision and stipulate not only the legal reasoning underpinning the decision, but also the procedural history, relevant facts and documents, as well as interview transcripts with the applicant. This is a rich dataset, which allows us to ask complex and sophisticated research questions concerning recurrent argumentation grounds, tracing the process and steps of asylum adjudication from the time when the applicant enters Denmark until the final decision, understanding why the RAB overturns first instance decisions, and inspecting the presence of systemic biases.

The RAB was established in 1983 as a quasi-judicial body and is composed of a chairperson who is a judge and two other members representing the Danish Immigration Ministry and the Danish Bar Association.Footnote ¹⁰⁹ First-instance decisions are made by the Immigration Service and Decisions rejected at first instance are automatically appealed to the RAB. The RAB as a quasi-judicial body has full legal competence to assess questions of fact and law, which means that there is no possibility to appeal its decisions to the ordinary courts. The RAB’s decisions are based on Danish immigration law, but relevant legal criteria remain largely based on international and EU law. Although the Danish opt out means that Denmark is not bound by certain EU law regulations in the asylum domain, it remains part of the Common European Asylum System (CEAS) and is also bound by certain directives due to its participation in the Schengen legal framework, and RAB decision-makers routinely follow developments in the EU legal sphere.Footnote ¹¹⁰

The information extracted first allowed us to analyse some factors that extant literature has deemed important to explain differences in asylum recognition rates, such as the marital status of the applicant, the sex and the age of the applicant, and the sex of the judge. In our analysis, we found no statistical difference for these factors. This suggests that previous studies on outcome variations might not be generalisable across all jurisdictions.Footnote ¹¹¹ In our case, the fact that we did not observe a difference in recognition rates based on the sex of the person presiding over the asylum decision might reflect both differences in national legal systems, training, and appointments, but it could also be due to the fact that in the Danish system appeal decisions are taken by a panel rather than by a single judge. This underscores the importance of taking into account differences in procedural and institutional architecture when moving to analyse and compare administrative decision-making at the national level, as these factors may significantly impact how EU law is understood and implemented.

We further used this data as a basis for comparing outcome variation between countries where data was available from previous research. We did this by employing statistical models. The results of our analysis of relative rates of dispersion are displayed in Figure 19.2.

Figure 19.2 Comparative analysis of recognition rates of asylum decision-makers.

Figure 19.2Long description

Top left. The bar graph titled, Denmark, plots recognition rate ranging from 0.1 to 0.4 versus rank ranging from 0 to 30. The graph follows an increasing trend from (0, 0) to (0.42, 38). Top right. The bar graph titled, Australia plots recognition rate ranging from 0 to 0.20 versus rank, ranging from 0 to 30. The graph increases from (0, 0) to (30, 0.22). Middle left. The bar graph titled, Canada from 2008 to 2011. It plots recognition rate ranging from 0 to 0.30 versus ranks from 0 to 40. The graph increases from (0, 0) to (39, 0.35). Middle right. The bar graph titled, Canada from 2013 to 2016. It plots recognition rate, ranging from 0.05 to 0.15, versus rank, ranging from 0 to 40. The graph increases from (0, 0) to (46, 0.23). Bottom left. The bar graph titled, U S A, plots recognition rate ranging from 0 to 0.8 versus rank ranging from 0 to 500. The graph increases from (0, 0) to (499, 1). Bottom right. The multi-line graph titled, Comparison plots density ranging from 0 to 8 versus recognition rate from 0 to 1. It plots curves for Denmark, Australia, Canada from 2008 to 2011, Canada from 2013 to 2016, and U S A. They peak between the recognition rate of 0 to 0.4. Note, all data are approximated.

We can now broaden our analysis to include a comparison with other jurisdictions outside of the EU jurisdictional area. Here, we conducted a similar analysis on our data as well as data for other jurisdictions: the rates of the US judges for the period 2016–2021,Footnote ¹¹² the rates of the Australian judges,Footnote ¹¹³ and the overall rates of the Canadian leave judges in the judicial review stage for the periods 2008–2011 and 2013–2016.Footnote ¹¹⁴ Both EU law and European human rights law introduce more specific procedural standards and additional bases for protection compared to Australia, Canada, and the United States, who at the international level are primarily orientated towards the 1951 Convention Relating to the Status of Refugees. Whilst it would thus make little sense to compare outcomes or overturn rates directly, it is still relevant to compare internal coherence and variation amongst individual decision-makers.

Figure 19.2 shows the density plot of individual judges/panel chairs within each jurisdiction. Turning to the normalised comparison in the bottom-right panel, it immediately becomes clear that each jurisdiction (as expected) exhibits different overturn rates – the peak of each line representing the mode, that is, the overturn rate occurring most often across the individual judges. Looking at the distribution patterns, it is further clear that most jurisdictions exhibit a clear peak. Looking at both of the Canadian curves, the higher peak for the early period suggests a somewhat closer distribution of overturn rates amongst the majority of judges prior to the reform; yet what the reform achieved was an overall more unimodal outcome distribution, without the previous second peak representing a smaller cluster of judges with a higher turnover rate. The Danish plot is similarly unimodal, but with a wider and more symmetrical bell-shaped curve around the peak – suggesting slightly more individual but normal-distributed variation around the mean. The Australian curve exhibits several peaks, indicating different clusters of individual judges with similar overturn rates. Last, but not least, for the US data, the distribution does not have a well-defined mode or peaks, such that the US data exhibits the highest degree of dispersion.

The dispersion of the distribution tells us something about consistency amongst decision-makers. If cases are randomly assigned to judges, the variation observed in countries that exhibit a bell-shaped distribution are less likely to result from individual bias. Adjudicators that exhibit overly large or overly low recognition rates can be understood as outliers not descriptive of the overall system, although they impact the overall system. Under this understanding of consistency, the US system appears to be less consistent because of the variation pattern. However, two caveats apply. First, if the allocation of cases to judges is non-random, this type of variation may be explained on other grounds. Second, a certain degree of variation is to be expected given the specific circumstances of individual cases, and some variation will be inevitable, as legal scholars have noted, because it reflects ‘degrees of specialization, complexity, dynamism, emotional or ideological content, and spectrum of choice’.Footnote ¹¹⁵

Thus, on the one hand, the analysis seems to show that not all apparently random legal processes are equally random, on the contrary, some appear to exhibit a greater deal of decision-making consistency than others. However, it is important to not draw overly strong conclusions without considering all of the other variables which may shape the results. Domestic legal cultures can strongly affect national peculiarities, and it is relevant to consider that Danish society functions with a strong pull towards consensus.Footnote ¹¹⁶ The Danish experience is further instructive as the results are likely skewed by what in US empirical legal studies is known as ‘panel effects’, the impact judges have on each other with typical regard to such factors as race, gender, and ideology.Footnote ¹¹⁷ As previously mentioned, Danish panels are constituted with mixed expertise (i.e., judges, civil society, and lawyers)Footnote ¹¹⁸ which may mute the role of preference, whereas in the US the decisions of politically appointed judges often show a strong correlation with certain outcomes – that is, more ‘progressive’ judges are more likely to grant relief whilst more ‘conservative’ judges are more likely to deny applications.Footnote ¹¹⁹ Institutional design can also affect variation, and the Canadian data is interesting in this regard because the Canadian distribution in 2013–2016 is less skewed than 2008–2011, which indicates that inequality was reduced. This was precisely the result of Rehaag’s first major study finding inconsistency amongst Canadian asylum law judges and its recommendations, as judicial reform measures were subsequently undertaken by the state to ameliorate inequalities.Footnote ¹²⁰ Due to the confidential nature of this data, the data richness also comes with other challenges related to the need to anonymise data sources, which can further inhibit effective comparison on asylum data.

19.4 Potentials and Pitfalls of Staging Computational Interventions in EML

To briefly conclude, it is necessary to circle back to some critical points which have emerged from our discussion. First, the analysis has clearly shown some of the problems of large-scale quantitative analysis, and ultimately, what it can teach us. Quantification is a representation, and at best, relies on a high degree of interpretation. At worst, it reifies particular aspects of the world, selected because they happen to lend themselves more to statistical analysis. Computational research is inherently biased towards written materials, and will never be able to tell us what happened, for example, in a court room, which is the domain of qualitative forms of inquiry.Footnote ¹²¹ Whilst this is hardly a new problem, existing literature often underestimates the extent to which these dynamics are accentuated by algorithmic analysis, which may introduce additional ‘black box’ challenges of interpretation and in that sense entail a double removal of human subjectivities within the research. This underscores the importance of approaching computational research with proper domain legal expertise, but also being open to mixed-method approaches that draw in more critical perspectives as a means for interrogating and questioning research findings and data practices.Footnote ¹²²

This leads to another important reflection for researchers undertaking computational analysis of EML in a field such as EU law, which is overwhelmingly doctrinally orientated. Computational research often receives pushback from doctrinal scholars who claim that it devalues the normative aspect of law.Footnote ¹²³ The experience of researchers working on the Supra Legem Project (2017) in France is instructive here. Using a simply predictive algorithm, the project found significant disparities in asylum decision-making associated with the identity of asylum judges. Yet, the project also faced significant backlash from judges and the legislature, who subsequently introduced a general ban on research on individual judicial behaviour.Footnote ¹²⁴ Policy audiences may not always welcome hearing evidence that is unsettling, and interventions must be crafted with due regard for practices for implementing policy reforms.Footnote ¹²⁵ It must always be recalled that in contrast to computational discourse, law is a ‘slow hermeneutic’ that requires reasons, deliberation, and dialogue.Footnote ¹²⁶ In other words, connecting the empirical to the normative through computational research on EML is no easy task, but we have hoped in this contribution to lay some possible ways to bring this about.