Scholars in search of quality standards for traditional legal scholarship could well end up disappointed. By answering the question concerning what standards legal academics use for evaluating such works—through reviewing the international literature on evaluative standards and interviews with forty law professors—this Article aims at filling this gap. This Article recommends that traditional legal scholarship is judged by using the following criteria: (1) the conceptual design—a clearly formulated research question that is both original and significant and the adequacy of the methods proposed to answer that question; (2) the composition of a particular line of reasoning—does the researcher adhere to principles of accountability, accuracy, balance, and credibility?; and (3) the overall characteristics of a work of scholarship—the readability and persuasiveness of the whole and the extent to which the researcher managed to identify and clarify the presuppositions that may have potentially affected her inquiry.
Marnix Snel is an associate professor in private law and legal research methods at the University of Curaçao, a research fellow of the Tilburg Institute for Private Law, and a visiting lecturer at Nyenrode Business University. Email: firstname.lastname@example.org.
1 See Lee Epstein & Gary King, The Rules of Inference, 69 U. Chi. L. Rev. 1, 18–19 (2002) (“There seems to be a long tradition of legal academics denigrating articles published in their journals. Over the years, they have referred to the content in these journals as ‘junk stream,’ ‘manure,’ ‘[not] readable by humans,’ ‘fuzzy wuzzy,’ ‘junk science,’ ‘spinach,’ ‘boring, too long,’ rife with ‘assertions unconnected to an empirical basis,’ dependent on ‘anecdotes,’ ‘opaque’ and an ‘open scandal.’”).
2 See Mary I. Coombs, Outsider Scholarship: The Law Review Stories, 63 U. Colo. L. Rev. 683, 706 (1992).
3 A search for publications on quality standards or quality evaluation in the social science databases quickly resulted in hundreds of relevant hits, while I only identified about thirty relevant contributions by performing a similar search in the international legal databases. See Alain L. Verbeke, Beyond Quantity—Classifying and Evaluating Legal Research in a Trusting Environment, Leuven L. Res. Classification & Evaluation Model 1, 1 (2013); Theunis Roux, Judging the Quality of Legal Research: A Qualified Response to the Demand for Greater Methodological Rigour, 24 Legal Educ. Rev. 173, 186–87 (2014).
4 Rubin goes as far as to argue that the absence of an evaluative theory has led to a wide range of conceptual and practical difficulties that “have festered to produce confusion and malaise throughout the field.” He argues that many of the most significant debates in legal scholarship—for instance, about the value of critical legal studies and its relationship to traditional scholarship, or the contribution that other fields such as economics, literature, and social science can make to legal thought—involve evaluation, and that many of these are repetitive and unproductive for a lack of an evaluation theory. Edward L. Rubin, On Beyond Truth: A Theory for Evaluating Legal Scholarship, 80 Calif. L. Rev. 889, 889—94 (1992).
5 Information on the methods followed is presented in paragraph D.
6 See generally Philip C. Kissam, The Evaluation of Legal Scholarship, 63 Wash. L. Rev. 221 (1998); Rubin, supra note 4; Rachel J. Anderson, From Imperial Scholar to Imperial Student: Minimizing Bias in Article Evaluation by Law Reviews, 20 Hastings Women’s L.J. 197 (2009); Roux, supra note 3.
7 When conducting socio-legal research, for example, the scholar has access to a body of sociological literature on quality standards that, at least partly, apply.
8 For contributions that address the standards that apply to other types of legal scholarship, see generally Epstein & King, supra note 1 (empirical legal research); William Nelson, Standards of Criticism, 60 Tex. L. Rev. 447 (1982) (legal history); Nancy Levit, Defining Cutting Edge Scholarship: Feminism and Criteria of Rationality, 71 Chicago-Kent L. Rev. 947 (1996) (jurisprudence); Susan Bartie, The Impact of Legal Meta-Scholarship: Love Thy Navel, 18 Griffith L. Rev. 727 (2009) (meta-scholarship); Coombs, supra note 2 (feminism and critical race theory); Gijs van Dijck, Kwaliteit van de Juridische Annotatie: Een Empirische Studie naar Kenmerken en Kwaliteitsindicatoren (2011) (case notes).
9 Of course, there is much more to say about the nature and characteristics of traditional legal scholarship. As the purpose of this Article is to focus on the debate on quality standards, I hope the reader accepts that I subscribe to definitions provided by, for instance, Rubin, supra note 4, at 902–10.
10 A few philosophers of science have even contested the possibility of defining quality standards. See generally Roderick Chisholm, Theory of Knowledge (3d ed. 1989); Thomas F Gieryn, Boundary-Work and the Demarcation of Science from Non-Science: Strains and Interests in the Professional Ideologies of Scientists, 48 Am. Soc. Rev. 781 (1983). I, however, start from the assumption that it is possible to define quality standards, an assumption that is present among nearly all scholars that contributed to the debate on quality evaluation criteria in several disciplines.
11 See Yvonne S. Lincoln & Egon G. Guba, Naturalistic Inquiry 329 (1985); Margaret A. Eisenhart & Kenneth R. Howe, Validity in Educational Research, in The Handbook of Qualitative Research in Education 4 (Margaret D. LeCompte, et al. eds., 1992); Coombs, supra note 2, at 705. Abrams remarked that understandings developed even within a particular subdiscipline—such as traditional legal scholarship—are probably not even unitary nor uncontested. See Kathryn Abrams, Hearing the Call of Stories, 79 Calif. L. Rev. 971, 1018–1019 (1991).
12 Recent empirical research has confirmed that—for one or a combination of the reasons presented below—the majority of the members of the Dutch and Swiss legal scholarly communities by now seem to consider the current situation—in which our quality standards are largely implicit and mysterious—untenable. See Rob A. J. van Gestel, Karin Byland & Adreas Lienhard, Evaluation of Legal Research: Comparison of the Outcomes of a Swiss and Dutch National Survey, 23 Tilburg L. Rev. 3 (2018).
13 See Rubin, supra note 4, at 889; Kissam, supra note 6, at 221–22.
14 See Levit, supra note 8, at 947; Richard A. Posner, The Future of the Student-Edited Law Review, 47 Stan. L. Rev. 1131, 1133 (1995). According to Posner, such criteria may be the reputation of the author, the author’s relation to the editors, the prestige of the author’s law school, the length of the article, and the number and length of the footnotes in it.
A logical exception that might be raised to this point is that the quality of inferences can be judged by knowing the reputation of the investigator…. This kind of appeal-to-authority, however, is wholly irrelevant to valid inference. Even if it were true that famous authors are wrong less often than obscure authors (a hypothetical with little supporting evidence!), they would still sometimes be wrong. Only the evidence—the process by which the data came to be observed, not the investigator’s fame, job, status, or income—is the stuff of serious scientific inference.
Epstein & King, supra note 1, at 34.
15 This argument has been discussed in some depth by Rubin, supra note 4, at 894–902, 963. From a somewhat different perspective:
Because the criteria are under-articulated, the test may in effect become one of familiarity. If we cannot say exactly what good scholarship is, we may use, as an implicit paradigm, something we wrote, or tried to write, or wished we could write. Work that is different than anything we can imagine wanting to write will not qualify.
Coombs, supra note 2, at 707.
16 They are, for instance, used to decide questions of lateral hiring and—more crucially—tenure, when a judgement must be passed on the performance of a faculty, department, research group, program, or institute, or to decide about the distribution of research funds.
17 Carter argues:
At least it is a test that would put all arguments on the same footing; if one wants to say that this article by Scholar Black is better than that article by Scholar White, one at least now has rules within which the battle can be fought, and the possibility of agreed rules is a considerable improvement over the current situation, in which there are as many versions of the rules for evaluation as there are evaluators.
Stephan L. Carter, Academic Tenure and “White Male” Standards: Some Lessons from the Patent Law, 100 Yale L.J. 2065, 2084–85 (1991). I remark that this idea has also been refuted within the literature. See Pierre Schlag, Pre-Figuration and Evaluation, 80 Calif. L. Rev. 965 (1992) (arguing that a theory of evaluation does not free an evaluator from the bias inherent in her own pre-figurations. On the contrary, the belief that better evaluative criteria will advance the cause of fairer evaluation is itself an effect of flawed and unrationalized pre-figurations of conventional legal thought); Richard Delgado, On Telling Stories in School: A Reply to Farber and Sherry, 46 Vand. L. Rev. 665, 765 (1993) (“When standards take hold in the academy, the result may be to legitimate an unfair and biased system.”).
18 Compare Coombs, supra note 2, at 703–04; Jayne W. Barnard, Reflections on Britain’s Research Assessment Exercise, 48 J. Legal Educ. 467, 490 (1998).
19 See also Verbeke, supra note 3, at 1–2 (“Especially our young doctoral and post-doctoral researchers are in demand of more clarity and guidance about what is research and what is not.”); Edward G. White, The Text, Interpretation, and Critical Standards, 60 Tex. L. Rev. 569, 584–85 (1982) (“To characterize the evaluative standards of academics or other professional elites as radically indeterminate is therefore either to abdicate educational responsibility or to advocate an alternative approach to education that is profoundly misguided.”).
20 It should be noted here, though, that scholars have—I believe rightly—warned that when used aspirationally, criteria can only indicate what is minimally required to achieve certain quality levels. What makes a work of scholarship truly outstanding, extraordinary, or even path-breaking, is dependent on many different factors that, moreover, are largely recognizable only in retrospect. See Coombs, supra note 2, at 705; Aviam Soifer, MuSings, 37 J. Legal Educ. 20, 23 (1987).
21 See Barnard, supra note 18, at 491.
22 See Rob A. J. van Gestel & Jan B. M. Vranken, Assessing Legal Research: Sense and Nonsense of Peer Review Versus Bibliometrics and the Need for a European Approach, 12 German L.J. 901, 905 (2011).
23 See Verbeke, supra note 3, at 1–2. (“The outsiders, other sciences, do not take us serious. They wonder whether law is a science and how what is done there could be considered research.”); Roux, supra note 3, at 173; Coombs, supra note 2, at 703; Van Gestel & Vranken, supra note 22, at 906.
24 Robert Cryer et al., Research Methodologies in EU and International Law 10 (2011); Terry Hutchinson & Nigel Duncan, Defining and Describing What We Do: Doctrinal Legal Research, 17 Deakin L. Rev. 83, 84 (2012); Jan M. Smits, What Is Legal Doctrine? On the Aims and Methods of Legal-Dogmatic Research 1, 7 (Maastricht European Private Law Institute, Working Paper No. 2015/06, 2015).
25 Some examples are the reports produced by the German Wissenschaftsrat in Perspektiven Der Rechtswissenschaft in Deutschlands: Situation, Analysen, Empfehlungen (Wissenschaftsrat 2012); reports by the Belgian Vlaamse Interuniversitaire Raad in Model Voor Integralte Kwaliteitsevaluatie Van Het Onderzoek in De Rechtswetenschappen (Vlaamse Interuniversitaire Raad 2004); and reports by the Dutch Evaluatiecommissie Rechtswetenschappelijk Onderzoek, Kwaliteit & Diversiteit: Rechtswetenschappelijk Onderzoek in Nederland (Evaluatiecommissie Rechtswetenschappelijk Onderzoek 2009), but also the reviews resulting from the British Research Assessment Exercise and the Excellence in Research for Australia.
26 See generally Van Gestel & Vranken, supra note 22 (considering protocols developed in the United Kingdom, Belgium, the Netherlands, Australia, and Denmark); Evaluating Academic Legal Research in Europe: The Advantage of Lagging Behind (Rob A. J. van Gestel & Andreas Lienhard eds., forthcoming Apr. 2019) (providing an overview of assessment practices in twelve European countries).
27 See generally Van Gestel & Lienhard, supra note 26 (Europe); Jian Zhang, Evaluating Chinese Legal Scholarship in Journals: Are Journal Rankings and Elite Law Journals Representing the Best Quality, https://ssrn.com/abstract=2706585 (Dec. 15, 2015) (China); Epstein & King, supra note 1, at 45–47 (United States).
28 Van Gestel & Vranken, supra note 22, at 908–11.
29 Although debates within the discipline of law have mainly revolved around the means by which other scholars can determine a scholarly publication’s quality—a large body of literature addresses the “sense and nonsense” of peer review, bibliometrics, journal rankings, and a wide variety of other means that could be helpful in getting some grip on the quality of a legal academic’s contribution—there are some that have tried to define more substantive quality standards for evaluating traditional legal scholarship. Without pretending to be exhaustive, I provide a brief overview of some of the more elaborate and thoughtful accounts I came across.
30 Also remarked by Terrill Pollman & Linda H. Edwards, Scholarship by Legal Writing Professors: New Voices in the Legal Academy, 11 Legal Writing: J. Legal Writing Inst. 3, 46 (2005). See also Erwin Chemerinsky & Catherine Fisk, In Defense of the Big Tent: The Importance of Recognizing the Many Audiences for Legal Scholarship, 34 Tulsa L.J. 667, 676 (1999).
31 Rubin states:
Sometimes, this can be accomplished in a sentence or a paragraph. A constitutional argument might be based on the norm that we should interpret the Constitution as an evolving system of structural relationship; a common law argument might begin by declaring that the law should be predictable by private parties. In other cases, stating one’s normative starting point might require a more elaborate argument.
Rubin, supra note 4, at 916.
32 Rubin argues:
For purposes of this criterion, it is unimportant whether the author chooses to justify his starting premise or merely identify it, whether he engages other scholars who adopt different premises or ignores those scholars and proceeds with his own discussion. These issues will become relevant for other criteria, but with respect to normative clarity, the crucial question is whether the reader can engage the author.
33 Rubin states:
For example, the scholar might argue that the policy of avoiding accidents can be implemented by holding manufacturers strictly liable for injuries caused by their products, or that the First Amendment should be interpreted in accordance with a principle of personal autonomy. The first argument threats a descriptive statement—accidents can be avoided through strict liability—as problematic and claims validity on the basis of the statement’s ability to implement the stated norm. The second argument presents an expressive claim; it asks the reader to recognize, as a matter of personal experience, that particular way of reading the Constitution is authentic…. Given the pragmatic quality of legal scholarship (not truth, positivism), the author is entitled to assume the premise, as long as it is clearly stated, and proceed to consider means for its implementation. The real test of persuasiveness, then, is whether readers would be convinced if they accepted the initial premise. In other words, if we wanted to avoid accidents, would we impose strict liability on manufactures to implement that policy? If we agree that personal autonomy is a valid norm, can the First Amendment be authentically read to achieve it?
Id. at 920.
34 Id. at 923. According to Rubin, “[t]his judgement is made according to standards that the interpreters would recognize as valid ways of judging what they themselves mean when they write.”
35 Id. at 923–24. As an example, Rubin refers to a study in which the authors tried to demonstrate that a right to privacy is already ensconced in the common law—while that was simply not the case—rather than devoting their argument to arguing for the attractiveness of such a right.
36 Rubin notes that:
As a practical matter, significance may seem to be a difficult criterion to use for evaluative purposes because it requires knowledge of the work’s subsequent history …[b]ut the question of significance is not limited to subsequent effect; it also concerns the work’s relationship to the developing historical context at the time it is written.
Id. at 934.
37 Id. at 937.
38 Kissam, supra note 6, at 228.
40 Id. at 228–29. Thus, similar to Rubin’s criteria of significance and applicability.
41 As clearly shown by Van Gestel & Lienhard, supra note 26.
42 Verbeke, supra note 3.
43 Id. at 5. As he states it, “what is analytical … will also be critical.”
44 Jan B. M. Vranken, Mr. C. Asser’s Handleiding Tot De Beoefening Van Het Nederlands Burgerlijk Recht: Algemeen Deel: Een Synthese (2014). For an English summary, see Jan B. M. Vranken, Methodology in Legal Research and Legal Practice: A Comparison of Judge and Legal Academic, in Recht in Geding II (Marc S. Groenhuijsen, Ewoud Hoondius & Arend Soeteman eds., 2016).
45 Jan B. M. Vranken, Mr. C. Asser’s Handleiding Tot De Beoefening Van Het Nederlands Burgerlijk Recht: Algemeen Deel: Een Synthese (2014) at 50–51. According to Vranken, that newness may be a contribution to theory, or—more modest—adding a new aspect to the discussion in the literature, pointing to the fact that a particular decision of the Supreme Court should be interpreted differently than the scholarly community generally does, or a comparison with lower court decisions or other jurisdictions.
46 Id. at 55–56.
47 Id. at 56–57.
48 Id. at 107–45.
49 Meaning that the data the scholar used is not clandestinely massaged in a particular direction; the composition of the answer must be “imitable,” requiring the scholar to explicate and justify the norms or perspectives that may have consciously or unconsciously influenced the way in which the problem definition is answered.
50 Id. at 145–50. According to Vranken, there may exist conflict between the standards of persuasiveness and readability and the other standards, which means that a balance must be sought between the different criteria.
51 Moreover, they are not without criticism. The standards developed by Rubin have been criticized by Schlag, supra note 17, and Levit, supra note 8, at 953–55. The standards developed by Vranken were critiqued by Willem H. Van Boom, Kritiek En Methode in Juridisch-Dogmatisch Privaatrechtelijk Onderzoek—Een Bespreking Van Asser-Vranken Algemeen Deel****, 32 Nederlands Tijdschrift voor Burgerlijk Recht 123 (2015). What personally struck me is that the different contributions to the debate on quality standards almost appear in isolation. I hardly found any cross-referencing among these and other—less elaborate and therefore undiscussed—contributions.
52 See generally Ginevra Peruginelli, Evaluating Research: The Case of Legal Scholarly Outputs, 15 Legal Info. Mgmt. 50 (2015); Sven E. Hug & Michael Ochsner, How Legal Scholars Could Change the Landscape of Research Evaluation, in L’evaluation De La Recherche En Droit: Enjeux Et Méthodes (Thierry Tanquerel & Alexander Flückiger eds., 2015); Wolfgang Kaltenbrunner & Sarah De Rijcke, Quantifying ‘Output’ for Evaluation: Administrative Knowledge Politics and Changing Epistemic Cultures in Dutch Law Faculties, 44 Sci. & Pub. Pol’y 284 (2016).
53 I chose to focus on law professors as this group of scholars is generally the most experienced, regularly involved in the evaluation of scholarly work—as peer reviewer, editor, or supervisor—and to a certain extent defining the standards of a disciplines because of their function as exemplar. See Thomas Kuhn, The Structure of Scientific Revolutions 187 (3d ed. 1996); Sébastien Pimont, Á Propos De L’activité Doctrinale Civiliste (Quelques Questions Dans L’air Du Temps), Revue Trimestrielle de droit civil 707 (2006); Coombs, supra note 2, at 711. The sample was composed by taking into consideration variations in sociodemographic characteristics such as age, experience, gender, research institutes, and areas of the law—private law, administrative and constitutional law, criminal law, and international law. To assure that the choice to consult Dutch scholars does not result in findings that may only be relevant to Dutch legal scholarship, I made sure that all of the respondents publish(ed) internationally, act(ed) as peer reviewer or editor for international or European journals, and/or were part of dissertation committees in other countries.
54 I decided to use this qualitative approach because prior research conducted within other scientific disciplines revealed that empirical investigations with qualitative approaches towards research habits can yield rich and useful data. See generally Barbara Blummer, Sara L. Watulak & Jeffrey Kenton, The Research Experience for Education Graduate Students: A Phenomenographic Study, 17 Internet Reference Serv. 117 (2012); Rosemary Green & Peter Macauley, Doctorol Students’ Engagement with Information: An American-Australian Perspective, 7 Portal 317 (2007); Vanessa J. Earp, Information Source Preferences of Education Graduate Students, 27 Behav. & Soc. Sci. Libr. 73 (2008); Hug & Ochsner, supra note 52, at 361–62. Some legal scholars have also specifically called for a qualitative approach in order to arrive at a more nuanced and deepened understanding of this particular research activity. See Christopher G. Wren and Jill R. Wren, Reviving Legal Research: A Reply to Berring and Vanden Heuvel, 82 Law Libr. J. 463, 466 (1990); Stephanie Davidson, Way Beyond Legal Research: Understanding the Research Habits of Legal Scholars, 102 Law Libr. J. 561, 570 (2010). Further information on the methods and techniques followed—on sampling, the way in which the semi-structured interviews were setup, the way in which the data was coded, and so on—can be requested from the author.
55 One of the respondents stated:
If I have to evaluate a journal article, the first thing I do is see whether a clear problem definition is provided. Sometimes you might question the content of the problem definition, but the fact that a journal article possesses a problem definition alone is—unfortunately—already quite something within contemporary traditional legal scholarship.
56 Suppose a researcher intends to answer the question: “Is or should obligatory supervision of adolescents that are at risk of derailing because of psychological problems (be) legally legitimized?” Without a description of what the indeterminate concepts of adolescents, derailing, psychological problems, and obligatory supervision mean, this question is not answerable. See Jan B. M. Vranken, Mr. C. Asser’s Handleiding Tot De Beoefening Van Het Nederlands Burgerlijk Recht: Algemeen Deel: Een Synthese (2014) at 56–57.
57 See Carter, supra note 17, at 2080: “The standard that work must be new is simply a matter of what the law reviews sometimes call preemption: Did the writer think of this idea first or is it already in the literature?”
One must further notice a double-edged sword at work, with originality caught between preemption and citation. An article must simultaneously be well supported and not preempted … An insistence on knowing where we have come from will make it more likely that we and our readers will know when we are saying something new.
Note, Originality, 115 Harv. L. Rev. 1989, 2007 (2002). For a somewhat different view:
Our main insight was that doctrinal research does not proceed on the back of a research question in the traditional sense, i.e. a question aimed at filling a gap in a defined body of scholarly knowledge. Instead, doctrinal research is directed at addressing an alleged lack of coherence, disputed issue of application or normative shortcoming in a defined area of law. While there are similarities between the two types of research question, the body of knowledge in the case of doctrinal research is legal doctrine itself.
Roux, supra note 3, at 186.
58 Or, in the words of Epstein and King:
(1) asking a question that the legal community may view as important but that no other scholars has tackled; (2) attempting to settle a question that has evoked conflicting responses; (3) raising an “old” question but addressing it in a unique way; (4) collecting new data on the same observable implications or different implications altogether; or (5) applying better methods to reanalyze existing data.
Epstein & King, supra note 1, at 59. The most elaborate publication I encountered on being original within legal scholarship, although taking a somewhat broader perspective, is the contribution of Siems, proposing several techniques to achieve originality. Mathias M. Siems, Legal Originality, 28 Oxford J. Legal Stud. 147 (2008).
59 Meeting this criterion is usually not problematic within legal scholarship. See Epstein & King, supra note 1, at 60 (“Our survey of law review articles suggest that it is the rare piece of legal scholarship that does not pose a question that has at least a potential implication—normative, policy, or otherwise—for the real word.”).
60 The minimal level of scientific significance will be met when the work belongs to the ongoing development of the field in some fashion. The highest levels of significance are met when a contribution, for instance, helps rethink contract law. In between, we could place a work that, for instance, develops a new theory of damages within contract law. Rubin provides several—quite random—examples of articles that he finds to meet the significance criterion. Rubin, supra note 4, at 930–35.
61 Such a preliminary inquiry is referred to in other academic disciplines as a “systematic literature review.” See generally Helen Aveyard, Doing a Literature Review in Health and Social Care: A Practical Guide (Open University Press 3d. rev. ed. 2010); Chris Hart, Doing a Literature Search: A Comprehensive Guide for the Social Sciences (2001); Arlene Fink, Conducting Research Literature Review: From the Internet to Paper (3d ed. 2010). Epstein and King highlighted another advantage of engaging existing scholarship: It “ensures that someone will be interested in the results,” as “if a body of literature—however slim and underdeveloped it may be—is on hand, it indicates that the question is important to at least some others.” Epstein & King, supra note 1, at 59. For the exact techniques that could be followed, see Marnix V. R. Snel & Janaína G. Garcia de Moraes, Doing a Systematic Literature Review in Legal Scholarship (2018).
62 As has been remarked by many before, these criteria are far from always explicated in legal publications. See Rubin, supra note 4, at 917; Roux, supra note 3. Some respondents even stated that, because of the often-absent frameworks, “the normative part [of] a traditional legal research project is often the least scientific part,” or at least “a vulnerable section.”
63 Evidently, the researcher can also opt for other research frameworks, like a more general framework construed on the basis of the doctrine of tort law, or a framework on the basis of one or more legal principles that dominate the field of civil and/or public law. According to a few respondents, the best of traditional legal scholarship combines all these different frameworks in one single publication, creating a “matrix-like presentation of different insights, recommendations, or solutions.”
64 While some respondents mentioned that, once a scholar has a research question, it may be constructive to begin theorizing—developing a reasoned and precise speculation about the answer to a research question—about possible answers that she can use to generate observable implications—things that we would expect to detect in the real world if our theory is right—others did not in any way refer to the importance hereof. With the few respondents who mentioned such a criterion and Epstein and King, I agree that the importance of theorizing is underestimated within traditional legal scholarship. For more detail on what this criterion may mean, see Epstein & King, supra note 1, at 61–80.
65 That does not, however, mean that every single sentence needs substantiation. See Note, supra note 57, at 2006 (“In other academic journals, by contrast, many facts are considered sufficiently well known to need no citation. But law reviews give citations for everything under the sun, as if every proposition, and some things that are not propositional at all, needed ‘authority.’”).
66 Within the legal literature, three different functions have been assigned to the employment of footnotes: (1) A positioning function—to position the research within its context and to suggest further readings on a particular issue; (2) a legitimizing function—to indicate the authority of the assertion made; and (3) a control function—informing the reader about the information on which assertions are made. See Elizabeth Fajans and Mary R. Falk, Scholarly Writing for Law Students: Seminar Papers, Law Review Notes and Law Review Competition Papers 99 (4th ed. 2011). For publications in other academic disciplines on the functions of footnotes, and their perverse effects, see generally Robert N. Broadus, An Investigation of the Validity of Bibliographic Citations, 34 J. Am. Soc’y Info. Sci. 132 (1983); Terrance A. Brooks, Evidence of Complex Citer Motivations, 37 J. Am. Soc’y Info. Sci. 34 (1986).
67 While the importance of precise and correct referencing might sound obvious, it causes problems in research practice. See Jeffrey L. Harrison & Amy R. Mashburn, Citations, Justifications, and the Troubled State of Legal Scholarship: An Empirical Study (University of Florida Levin College of Law Legal Studies Research Paper Series 2015).
68 The respondents argued that choices made that lie in the peripheral areas of an investigation require less elaborate justification than choices made that relate to the main line of reasoning followed by the author.
69 Epstein and King also write: “In many, perhaps most, instances, legal academics conducting these sorts of investigations rarely provide even a tracing of how they collected the evidence.” Epstein & King, supra note 1, at 42–45.
We rarely learn: (1) How authors canvassed the relevant case law and what precisely was the population from which they sampled; (2) How authors selected their cases and how many they read; (3) How authors distinguished “key” or “a few … exemplary cases” from those that are not central or not typical.
Law scholars may be fastidious about documenting textual sources of information via the omnipresent footnote, but they are not particularly attentive to the need to document their data procedures, nor, we hasten to note, have they established procedures for ensuring the requisite attention to or repositories (private or public) for their data.
70 See also Hutchinson & Duncan, supra note 24, at 98–99.
71 In this respect, Emden and Sandelowski argued for “the potential value of including among one’s criteria for goodness a ‘criterion of uncertainty’; that is, an open acknowledgement that claims about our research outcomes are at best tentative and that there may indeed be no way of showing otherwise.” Carolyn Emden & Margarete Sandelowski, The Good, the Bad and the Relative, Part Two: Goodness and the Criterion Problem in Qualitative Research, 5 Int’l J. Nursing Prac. 1, 5 (1999).
72 Epstein & King, supra note 1, at 208.
73 Epstein and King have a somewhat different view:
It may be that, after amassing the evidence for which the design calls, the scholar finds an imperfect fit among it, the main research question, and the theory. Rather than erasing months or even years of work, the investigator certainly should return to the drawing board, design more appropriate procedures, or even recast the original research question.
Epstein & King, supra note 1, at 54.
74 Questions that may be asked in this respect are: Whether the structure of the argumentation is comprehensible, whether the author takes the reader by the hand, or whether the reader is dragged along with the thoughts expressed by the author.
75 For example, new technological developments may bring to light novel questions of liability for harm caused, or the implications of the introduction of a partly new statutory framework for current legal practice.
76 In this context Roux stated:
The best doctrinal researchers are able to draw on a vast field of reference, and are familiar with a lot of positive law (both local and foreign) that potentially has a bearing on the questions they are addressing. In this way, they are able to see system-wide implications of new legal developments that less experienced researchers may miss. They are also able to resolve questions arising in ways that promote greater coherence in the affected body of law.
Roux, supra note 3, at 188.
77 The reader might live under the impression that it should not be too difficult to comply with the principle of accuracy. I found, however, sound evidence—both in the literature and from examples provided by the respondents—that even the most experienced traditional legal scholars make errors in this respect. For an empirical investigation, see generally Rob A. J. van Gestel, et al., Rechtswetenschappelijke Artikelen: Naar Criteria Voor Methodologische Verantwoording, 82 Nederlands Juristenblad 1448 (2007).
78 Within the empirical data, views somewhat differed on what qualifies as a limited or focused enough problem definition to be able to consult all relevant materials.
79 For instance, one of the respondents with an international and European law expertise argued that “any inquiry towards the European law faces the challenge that there are twenty-seven member states where that law applies. That means that there are probably twenty-seven countries in which relevant literature commentaries, case law, and legislation exist.”
80 The consulted material thus needs not only to be relevant to your inquiry, but also “functional.”
81 Achieving completeness and a thorough consideration of alternative interpretations, arguments, and solution is, however—self-evidently—not imperative for every claim made within a doctrinal legal contribution. That is both impossible and would substantively detract from the readability of a publication. In general, “being complete, balanced and honest is more crucial with regard to the main line of reasoning within an inquiry than in relation to its peripheral components.” For instance, that means that, with respect to the main line of reasoning, the easy way out—for example, only providing a footnote that says “for a different view, see author X”—is not considered sufficient by the respondents.
82 A scholar that searches for reasons to extend constitutional rights, to incorporate more safeguards for the victim into criminal procedures, or to intensify regulation to remedy environmental pollution may most certainly find arguments and evidence that such might indeed be necessary. Haack has labelled such research practice as “fake inquiry”; the scholar is not concerned with finding how things really are, but to advance herself by making a case for some proposition to the truth-value of which she is indifferent. Susan Haack, Manifesto of a Passionate Moderate: Unfashionable Essays 8–10 (1999).
83 See also Roux, supra note 3, at 188 (“In pure doctrinal research, however, the style is permissible—indeed it is desirable—provided that opposing arguments are giving their due.”). One of the respondents provided clarification by presenting a hypothetical example:
Suppose there exist two contradictory points of view within the literature, X and Y. The researcher may either chose to aim her research at the identification of different viewpoints regarding a certain topic or to defend a particular position. If the researcher opts for the former, she has to make sure that all viewpoints are discussed on an equal footing. If the researcher opts for the latter, she may attach more value to viewpoint X, as long as she also pays sufficient attention to viewpoint Y, motivating why viewpoint X is to be preferred over viewpoint Y.
84 Often referred to as “validity” in other academic disciplines. In relation to qualitative interpretative research, the validity criterion is often subdivided in “descriptive validity,” “interpretative validity,” “evaluative validity,” “theoretical validity,” and “generalizability.” See generally Joseph A. Maxwell, Understanding and Validity in Qualitative Research, 62 Harv. Educ. Rev. 279 (1992). Comparable concepts were encountered sporadically within the legal literature. See Hutchinson & Duncan, supra note 24; Charles Tremper, Sue Thomas & Alexander C. Wagenaar, Measuring Law for Evaluation Research, 34 Evaluation Rev. 242 (2010).
85 See also Epstein & King, supra note 1, at 3 (“There is a good deal of description in most legal scholarship, and this can be evaluated in terms of truth.”).
86 Compare Bardley Wendel, Explanation in Legal Scholarship: The Inferential Structure of Doctrinal Legal Analysis, 96 Cornell L. Rev. 101, 105 (2011). Because of the textuality of its sources, descriptive validity is less a concern within legal scholarship than it may be in disciplines where sources are non-textual. Where a researcher observing what happens in a classroom needs to make sure that what happened in that classroom is correctly described, the correctness of what happened in, for instance, a judicial opinion is achieved more easily as it is already written down. Still, achieving descriptive validity, even though not hard, remains highly important.
87 For the debate between Mazzarese and Niiniluoto, see Tecla Mazzarese, Norm-Propositions: Epistemic and Semantic Queries, 22 Rechtstheorie 39 (1991); Ilkka Niiniluoto, Norm-Propositions Defended, 4 Ratio Juris 367 (1991).
88 An often-cited publication that addresses these issues is the article of William W. Fisher III, Texts and Contexts: The Application to American Legal History of the Methodologies of Intellectual History, 49 Stan. L. Rev. 1065 (1997).
89 In line with Rubin, supra note 4, at 923–24. The issue here is not the appropriateness of the concept of the account forwarded in the interpretation, but its accuracy as applied to the perspective of the person or institution included in the account, referred to as “authenticity” by Sally Thorne, The Art (and Science) of Critiquing Qualitative Research, in Completing a Qualitative Project: Details and Dialogue (Janice M. Morse ed., 1997); Robin Whittemore, Susan K. Chase & Carol L. Mandle, Validity in Qualitative Research, 11 Qualitative Health Res. 533 (2001); Margarete Sandelowski, The Problem of Rigor in Qualitative Research, 8 Advances in Nursing Sci. 27 (1986).
90 See also Roux, supra note 3, at 188 (“A statement of the law that appears to a person familiar with the area of law concerned to be more persuasive and compelling than the other contending interpretations.”).
91 That does not mean that the researcher always has to add much more information. From the wording of an interpretation, it is often quite clear that the researcher, for instance, applied a teleological or historical approach.
92 If a legal scholar, for example, intends to discover a line in the Supreme Court’s case law with regard to a particular aspect of the duty of care in vendor-buyer relationships and the scholar therefore selects several exemplary or key cases in that area, generalizability would of course also be concerned with the question whether the selected cases were indeed exemplary or key.
93 Maxwell, supra note 84, at 295.
94 Id. at 291–92.
95 See supra note 63.
96 Compare Epstein & King, supra note 1, at 87–88.
97 See also Epstein & King, supra note 1, at 77–80.
98 Such requires that—to a certain extent—alternative explanations for a causal association are systematically tested and dismissed. The greater the number of relevant alternative variables considered and eliminated, the greater the confidence that the relationship is causal. Preferably, the causal relation established is also grounded in existing knowledge. In the legal example above, for instance, the claim made may be substantiated by the fact that judicial opinions of the European Court in Human Rights result in valid high Dutch law and therefore need to be followed by the Dutch Supreme Court. One should, however, be aware that “no matter how perfect the research design, no matter how much data we collect, and no matter how much time, effort, and research resources we expend, we will never be able to make causal inferences with certainty.” Epstein & King, supra note 1, at 37.
99 One of the respondents, for instance, argued that “if you manage to word the matter you are investigating in an exceptional manner, your publication is probably rated better than if you do not.” That such is not always the case is made very clear by Fred Rodell, Goodbye to Law Reviews, 1 J. Contemp. L. 4 (1936).
100 One of the respondents said that “a publication becomes more convincing if it only addresses real arguments and omits the incorporation of non-feasible or unrealistic arguments.” “The fact that argumentation is important in (especially) normative inquiry,” according to one of the respondents, “makes the estimation of a publication’s quality difficult”; “the ordering of arguments in a sense always has a subjective element in it.”
101 For instance, if I intrinsically believe that consumer protection is much more important than complete autonomy of contracting parties, that would—or at least could—presumably influence my research project that addresses the doctrine of contract termination.
102 Also proposed within the literature, see Michael Pendleton, Non-Empirical Discovery in Legal Scholarship—Choosing, Researching and Writing a Traditional Scholarly Article, in Research Methods for Law 164–65 (Mike McConville & Wing H. Chui eds., 2007); Rob A. J. Van Gestel, Hans W. Micklitz & Miguel P. Maduro, Methodology in the New Legal World 1, 23 (European University Institute, Working Paper No. EUI LAW 2012/13, 2012); Roux, supra note 3, at 189–90. Theory of knowledge acknowledges the effect of a researcher’s position and perspectives on any research endeavor, and disputes the belief of neutrality. It is the always limited perspective of the researcher that determines the particularities of a research process. See generally Kirsti Malterud, Qualitative Research: Standards, Challenges, and Guidelines, 358 Lancet 483 (2001); Thomas Nagel, The View from Nowhere (1986); Ludwig Wittgenstein, On Certainty (1969). Absolute objectivity probably cannot be achieved. If I understood the respondents correctly, trying to achieve objectivity can only mean to recognize that knowledge is indeed partial and situated, and to account adequately for the effects of the positioned researcher; bias should thus be accounted for, though not completely eliminated, and subjectivity arises when the effect of the researcher is ignored.
103 Compare Epstein & King, supra note 1, at 42.
* Marnix Snel is an associate professor in private law and legal research methods at the University of Curaçao, a research fellow of the Tilburg Institute for Private Law, and a visiting lecturer at Nyenrode Business University. Email: email@example.com.
Email your librarian or administrator to recommend adding this journal to your organisation's collection.
* Views captured on Cambridge Core between <date>. This data will be updated every 24 hours.
Usage data cannot currently be displayed