2. A closer look: Vignettes and probe questions

The current studies surveying folk attributions of epistemic justification are very rich. Owing to limitations of space, I cannot do justice to their complexity here. Instead, I will focus directly on the details that matter for my purposes, namely the choice of vignettes and probe questions.Footnote ³

In their two experimental studies, Friedman and Turri are interested in the folk epistemology of lotteries: they aim to investigate whether lay people attribute knowledge and justification to lottery propositions (i.e., that particular ticket of a fair big enough lottery will lose). Epistemologists have often claimed that one cannot know that a particular ticket is a loser solely on the basis of the odds involved and that knowledge of lottery propositions requires testimonial evidence – for example, hearing on the evening news the announcement of the winning numbers (Pritchard Reference Pritchard2005; Williamson Reference Williamson2000). These claims play an important role also for theorising about epistemic justification, as some (though not all) epistemologists have claimed that one cannot justifiably believe that a particular ticket is a loser solely on the basis of the odds involved (Nelkin Reference Nelkin2000; Smith Reference Smith2010). Friedman and Turri’s experimental studies aim to examine whether folk judgements about lottery beliefs lend support to these claims often made by epistemologists. To achieve such aim, they presented the participants with these vignettes:

The vignettes introduce the contrast of interest between statistical and testimonial evidence. To survey folk attributions of epistemic justification, the following were used as dichotomous test questions in Lotto and News: (1) Abigail is [justified/unjustified] in thinking that the ticket is a loser (Friedman & Turri Reference Friedman and Turri2015: 1066) and (2) Ellen is [justified/unjustified] in thinking that the ticket is a loser (Turri & Friedman Reference Turri, Friedman and Beebe2014: 49). Notably, these probe questions did not draw the attention to the evidence available to the protagonist of the vignette nor did they include additional qualifications specifying the relevant sense of epistemic justification.

Ebert et al. (Reference Ebert, Smith and Durbach2018) are also interested in the folk epistemology of lotteries. To build the contrast between statistical and testimonial evidence, they presented participants with the following case-pairs:

It’s immediately worth noting that Ebert and colleagues’ study shows significant differences from the two studies on lottery beliefs run by Turri and Friedman. Firstly, these vignettes are more precise. The probabilities are made explicit, and it is clear that a winning ticket must match all the numbers. Moreover, since these vignettes do not specify that the ticket is in fact a loser, concerns about truth effects on belief evaluation and outcome bias are plausibly avoided (Gerken Reference Gerken2020). Secondly, and perhaps more importantly, Ebert and colleagues are more cautious. They explicitly refrain from drawing strong conclusions from their experimental surveys, and they do not reject any normative theory of epistemic justification on the basis of empirical data (Ebert et al. Reference Ebert, Smith and Durbach2018: 134).Footnote ⁵

Yet, despite these improvements and differences, the probe questions asked to the participants remain the same in both Probabilistic Evidence and Testimonial Evidence: (3) Suppose Harry believes that the ticket has lost. Is Harry justified in believing that the ticket has lost? Yes/No (Ebert et al. Reference Ebert, Smith and Durbach2018: 117). Participants could express confidence in their answer on a five-point Likert scale.Footnote ⁶ Once again, the probe question on (epistemic) justification did not include additional qualifications or mention the available evidence.

The study by Nolte, Rose and Turri (Reference Nolte, Rose, Turri, Lombrozo, Nichols and Knobe2021) has a different focus: they investigated whether folk attributions of knowledge and justification support the view that knowledge entails justified belief. Predictably, they presented a series of vignettes to the participants and ‘simply asked for both knowledge and justification’ (Nolte et al. Reference Nolte, Rose, Turri, Lombrozo, Nichols and Knobe2021: 32). Here’s one vignette of interest:

Two test statements followed, and participants could express agreement with each statement on a six-point Likert scale. The first statement concerned whether Andy knows that his daughter is innocent. The second statement was meant to survey attributions of epistemic justification: (4) Andy is justified in believing that his daughter is innocent (Nolte et al. Reference Nolte, Rose, Turri, Lombrozo, Nichols and Knobe2021: ${\rm{\S}}$ 1.4). In this experimental study too, the justification question did not include any additional qualification.Footnote ⁷

Let’s take stock. Each of these experimental studies is supposed to target folk judgements about epistemic rather than pragmatic or moral justification, but surprisingly enough none of these studies actually qualifies or mentions such target explicitly. To fully appreciate this point, here’s a list of the probe questions for epistemic justification used in each study:

1. 1) Abigail is [justified/unjustified] in thinking that the ticket is a loser.

2. 2) Ellen is [justified/unjustified] in thinking that the ticket is a loser.

3. 3) Is Harry justified in believing that the ticket has lost? Yes/No.

4. 4) Andy is justified in believing that his daughter is innocent.

These unqualified occurrences of justification should give us pause, as well as reasons to raise the alarm about possible equivocations. Why should we think that philosophically untrained participants correctly understood what they were asked? The validity of these experimental studies essentially depends on a satisfactory answer to this question and on further substantial assumptions. I tackle this question and bring out these assumptions in the next section.

3. The assumptions outlined

The success of these experimental surveys of folk attributions of epistemic justification is conditional on some substantial assumptions: if we are to take these surveys at face value, then these assumptions have to be granted. While these assumptions have never been made explicit, it is important to bring them out clearly in order to best assess the validity and the relevance of these experimental studies. The assumptions are the following:

A few comments on each. Firstly, Folk Understanding must be granted in order for the data to have any significance. For example, an interesting result replicated by each experimental study on lottery propositions is that lay people generally tend to ascribe justification to these propositions on the basis of both statistical and testimonial evidence (Ebert et al. Reference Ebert, Smith and Durbach2018: §5.1). This result can inform epistemological theorising on epistemic justification only if participants correctly understand the question and have a grasp of what it is for a belief to be epistemically rather than morally or prudentially justified. Without this assumption, lay people would be assessing a different type of justification, and the data would be about something else.Footnote ⁸

Secondly, and relatedly, Immediate Testability must also be granted; otherwise, the choice of vignettes and probe questions would be inapt: as highlighted in the previous section, neither the vignettes nor the probe questions make any attempt to introduce the concept of epistemic justification. As a result, it must be granted that folk attributions of epistemic justification can be tested without having to provide any preliminary explication of epistemic justification.

Finally, Theoretical Neutrality captures the ambitions of generality of these experimental studies. As Nolte et al. admit (Reference Nolte, Rose, Turri, Lombrozo, Nichols and Knobe2021: 46), these studies are not testing whether folk attributions of epistemic justification favour any particular theory of the nature or structure of justification (internalism or externalism, and coherentism or foundationalism). Instead, they are surveying folk attributions of epistemic justification in general and regardless of the implicit folk notion of epistemic justification that lay people operate with. Accordingly, we must also take for granted that folk attributions of epistemic justification can be surveyed neutrally, and without making any reference in the vignettes to substantive theories of epistemic justification.

Summing up, these are fairly minimal but ultimately necessary requirements for the validity and relevance of these experimental studies. First, that participants can plausibly be taken to understand and assess the epistemic justification of beliefs. Second, that the concept of epistemic justification needs no previous explication in the vignettes. Third, that lay people attribute epistemic justification independently of any substantial theory of epistemic justification.

Unfortunately, these assumptions will not hold up to scrutiny, and I will provide specific considerations against each of them. But before doing so, I want to be charitable and pause to make the following observation: there is a seemingly good reason why these assumptions were made in the first place. All these experimental studies were designed to survey attributions of both knowledge and epistemic justification, and these assumptions are innocent – perhaps even natural – for the purpose of surveying lay attributions of knowledge. Of course lay people understand the question when they are asked to attribute knowledge. Of course there is no need to previously explicate the concept of knowledge in order to survey knowledge attributions. And of course, lay people can attribute knowledge independently of any specific theory of knowledge: intuitively enough, knowledge attributions require no prior grasp of anti-Gettier conditions.Footnote ⁹

Yet, what is true of knowledge need not apply to epistemic justification, and in this case, it plausibly does not. As I show next, there’s an important asymmetry between lay attributions of knowledge and folk attributions of epistemic justification, and the assumptions made to investigate the folk epistemology of the former do not automatically carry over to the latter. The assumptions underlying these experimental studies neglect the following key point: lay people cannot be expected to already possess or immediately grasp and apply the epistemologist’s concept of epistemic justification. This reveals a folk conceptual gap between the subject matter of these experiments and the conceptual repertoire available to the participants. Rejecting each assumption in turn will bring out this gap more clearly; this is the task of the next section.

4. The assumptions rejected

The significance of the data concerning folk attributions of epistemic justification depends on whether the participants correctly understand the probe questions. However, none of the experimental studies explicitly qualify the kind of justification which is supposed to be surveyed. To evaluate the lottery vignettes, participants were asked whether the protagonist is ‘justified’ in thinking (or believing) that their ticket is a loser. Similarly, in Accusation, participants were asked if the protagonist is ‘justified’ in believing that their daughter is innocent.

The absence of a suitable qualifier is a problem for Folk Understanding . Participants may have reasonably picked up on the more ordinary sense of ‘justified’ and interpreted it as pragmatically or morally instead of epistemically justified. This diagnosis gains further support from empirical data recently gathered through corpus analysis. Broadly speaking, corpus linguistics aims to gain insight into ordinary language by surveying large collections of texts in plain English such as the Corpus of Contemporary American English (COCA), the British National Corpus (BNC), or the EnTenTen20 Corpus. Evidence from a recent corpus study on epistemic justification suggests that the expression ‘justified belief’ occurs almost exclusively in philosophical discourse: in the EnTenTen Corpus alone, 91% of the occurrences of justified belief belong to a philosophical context (Grindrod Reference Grindrod2022: 19). And when ‘justified’ and ‘justify’ occur in non-philosophical contexts, they are used to evaluate actions or emotions, not beliefs (Grindrod Reference Grindrod2022: 21). These results jointly indicate that the probe questions employ unusual terminology: in ordinary discourse, justification talk does not normally take belief as an object. This problem is compounded by the salient moral features in Accusation: the protagonist forms a belief about the guilt of his very own daughter. Participants may have reasonably attributed justification for reasons of partiality and thought that the protagonist is morally justified in believing that his daughter is innocent because he’s her father.Footnote ¹⁰ These considerations against Folk Understanding are however limited in the following important sense. They apply more clearly to the study run by Nolte and colleagues, but it remains unclear whether a similar problem affects the surveys of lottery beliefs. Despite the often noted connection between lottery beliefs and action (e.g., Hawthorne Reference Hawthorne2004: 29–30), it’s difficult to characterise with precision how exactly participants may interpret justification in a pragmatic sense when asked about a losing lottery ticket. Moreover, some sensitivity to epistemic cues may suggest that participants are picking up on the epistemic sense of justification. In support of this point, Ebert and colleagues found that folk attributions of epistemic justification are influenced by changes in probabilities due to the size of the lottery and also by further contextual information on the type of evidence in the vignettes (Ebert et al. Reference Ebert, Smith and Durbach2018: 126; 133). Relatedly, it’s hard to maintain that participants are confused about different senses of justification given the high confidence rates replicated in each of the three studies on folk attributions of justification in lottery cases. For these reasons, the problems I raised with Folk Understanding are particularly pressing for Nolte and colleagues’ study, but they are less troublesome for Ebert and colleagues and Turri and Friedman’s surveys.

Despite this acknowledged limitation, the assumption remains at the very least questionable. Taken together, the facts that the ordinary use of justification tracks pragmatic or moral propriety and that the vignette emphasises moral features cast some doubt on Folk Understanding : overall, participants could not obviously assess epistemic justification.

Immediate Testability , the assumption that folk attributions of epistemic justification can be tested immediately without providing any explication of the concept at issue, is also untenable. Technical concepts seem especially unsuited to be surveyed directly without first being introduced or explained, and epistemic justification is often taken to be a technical concept denoted by a philosophical term of art.Footnote ¹¹ In fact, one point of agreement in the otherwise divided literature on epistemic justification is exactly that epistemic justification is a technical concept. In support of this, textual evidence abounds: as epistemologists have emphasised, epistemic justification is ‘a term from the philosophy classroom’ (Hawthorne & Stanley Reference Hawthorne and Stanley2008: 578) an instance of ‘philosophical jargon’ (Reynolds Reference Reynolds1998: 533), ‘a term of art in contemporary epistemology’ (Kvanvig Reference Kvanvig, Bernecker and Pritchard2010: 25) and ‘manifestly technical terminology’ (Williamson Reference Williamson, Dorsch and Dutantforthcoming: 3). For this reason, we should be sceptical of the possibility of surveying folk judgements about a distinctively technical notion that is not in any way introduced or explained.Footnote ¹²

Crucially, the problems with Immediate Testability will be more or less pressing depending on the exact dialectical role played by the empirical data. The more technical aspects of the concept of epistemic justification are particularly problematic when folk attributions of epistemic justification are taken to count against a normative theory of epistemic justification (Turri & Friedman Reference Turri, Friedman and Beebe2014: 64), or to support the more substantive thesis that knowledge entails justification (Nolte et al. Reference Nolte, Rose, Turri, Lombrozo, Nichols and Knobe2021: §5). These technical aspects may be less problematic when the empirical data are more modestly used to corroborate various armchair predictions on folk attributions of epistemic justification (Ebert et al. Reference Ebert, Smith and Durbach2018 pursue this markedly different aim).

Bracketing such specific dialectical subtleties, the problems with Immediate Testability also affect Theoretical Neutrality . Since the concept of epistemic justification needs some introduction and explanation, folk attributions of epistemic justification can hardly be theory-neutral: these judgements will inevitably be constrained by the specific and substantive theory of epistemic justification which needs to be introduced. I return to this point in more detail below; now I just want to note that both generality and theoretical neutrality seem unfeasible given the theory-laden subject matter of these experimental studies.

Summing up, the design of all these experimental studies is problematic on the following fronts. Firstly, because it fails to specify the epistemic rather than the moral, pragmatic or prudential sense of justification, it employs ambiguous probe questions. Secondly, it purports to survey directly folk judgements on a technical notion which is neither introduced nor explained to the participants. Finally, it aims for a kind of theoretical neutrality which seems hard to achieve with the notion of epistemic justification.

With these observations in play, we can appreciate the more fundamental problem. These experimental studies neglect an important gap between their subject matter, folk attributions of epistemic justification, and the conceptual repertoire that we can reasonably expect lay people to possess. It is unreasonable to expect lay people to possess and master the concept of epistemic justification routinely employed by epistemologists, or to grasp it without previous explanations. It is equally unreasonable to survey folk judgements on justified belief directly using terms that are not ordinarily used to assess beliefs. Such is the folk conceptual gap, which crucially went unnoticed. Each experimental study trades on assumptions that overidealise the conceptual repertoire of the participants, and this overidealisation is reflected in the choice of the vignettes and probe questions.

It’s important to note that the folk conceptual gap does not necessarily point at a performance error nor does it predict that lay people make obvious mistakes in assessing the vignettes. The gap is in place because the probe questions employ ambiguous and technical terminology which led the participants to reasonably pick up on different aspects of the justification question. Participants are not being irrational because they should but nonetheless fail to appreciate the epistemic sense of justification: rather than highlighting the irrationality of the participants, the folk conceptual gap may underscore more subtle instances of dysrationalia, a term coined by Stanovich (Reference Stanovich2009) to denote the inability to think accurately despite adequate levels of intelligence and rational capacities. Dysrationalia is caused by mindware gaps (Stanovich Reference Stanovich2009: 129–130; 150–151; West Reference West2010: 102), broadly conceived as the lack of relevant concepts and the knowledge required for accurate responses to specific cognitive tasks. A strikingly similar analogy applies to folk attributions of epistemic justification: participants confidently assess the vignettes but nonetheless lack the relevant concepts, knowledge, and training to evaluate whether a belief is epistemically (as opposed to morally, or pragmatically) justified. Mindware gaps of this kind might constitute the cognitive bases of the folk conceptual gap.Footnote ¹³ Unless this gap can be bridged, a major challenge to survey folk attributions of epistemic justification remains unaddressed.

5. Folk attributions of epistemic justification: the way forward

The folk conceptual gap raises a considerable challenge. Can this challenge be addressed? If so, how? I will argue that while there are compelling reasons for a strongly pessimistic answer to these questions, different strategies to survey folk attributions of epistemic justification may warrant some cautious optimism. My aim is not to offer the perfect strategy to survey folk attributions of epistemic justification: more modestly, my main goal is to emphasise that the challenge raised by the folk conceptual gap is hard to address and thus worth taking seriously. I shall do so by pointing out the merits of pessimism, as well as the limitations of optimism.