2 Overview of the terms coherence and correspondence

The terms C&C have a long history in philosophy, and Reference Dawson and GregoryDawson and Gregory (2009) address some of their historic roots. These terms stem from philosophy, where they are considered competing theories of knowledge attainment, or truth (Reference Gigerenzer and KurzKirkham, 1992). Both of these theories of truth answer the same question; which is, how do we know that a statement/belief/judgment is true? The different theories of truth offer different criteria for answering this question and hence, assessing the truth value of a statement/belief/judgment.

The correspondence theory of truth is the oldest of the philosophical theories of truth due to intuitive appeal. Simply put, a belief can be said to be true if it corresponds with the facts. The philosopher Bertrand Russell advocates the correspondence approach stating that “although truth and falsehood are properties of beliefs, they are properties dependent upon the relations of the beliefs to other things, not upon any internal quality of the beliefs. This leads us to the view that truth consists in some form of correspondence between belief and fact” (quoted in Velasquez, Reference Velasquez2005, p. 446). This view was dominant with no real opposition until the nineteenth century when it came under attack from pragmatists and idealists (Reference SchmittSchmitt, 2004). The idealists objected to the notion of fact. They argued that the apprehension of a fact was itself an act of judgment and perception and consequently, a belief held by an individual. Because objects in the world are not directly knowable, but mediated by our senses, they are only representations of objects in the world. Correspondence of beliefs with facts, they argued, was impossible to assess since facts were also beliefs. This argument is what led to the coherence theory of truth, the idea that truth is assessed via consistency of belief (Reference SchmittSchmitt, 2004).

The coherence theory of truth states that “a set of two or more beliefs are said to cohere if and only if (1) each member of the set is consistent with any subset of the others and (2) each is implied … by all of the others taken as premises or, according to some coherence theories, each is implied by each of the others individually” (Reference Gigerenzer and KurzKirkham, p. 104, 1992). In other words, internal consistency and logical standards are the heart of the coherence theory of truth. This can be seen in Blanshard’s statement that, “It is perhaps in such systems as Euclidean geometry that we get the most perfect examples of coherence that have been constructed” (cited in Kirkham, p. 106, Reference Kirkham1992).

The coherence theory of truth is not limited to beliefs held by one person; what I will refer to as intrapersonal coherence. Intrapersonal coherence requires simply that a statement or belief held by an individual be consistent with other statements or beliefs held by the same individual. Within the field of JDM, the assessment of transitivity is an assessment of intrapersonal coherence. The coherence theory of truth extends beyond intrapersonal coherence to interpersonal coherence. That is, beliefs held by an individual are true if they are consistent with widely accepted beliefs of other individuals. This interpersonal coherence is what gives the coherence theory of truth normative standing. Hammond’s use of the term includes this broader view of interpersonal coherence as he includes the assessment of judgment against normative standards, such as Bayes’s Theorem, under the coherence metatheory (Reference HammondHammond, 1996; Reference Hammond2007).

Philosophers acknowledge that it is possible to have two internally coherent belief systems, meaning that individually they are not self-contradictory, that may contradict each other. The assumption of coherence philosophers is that reality cannot be self-contradicting. So, while the coherence of a set of beliefs does not necessitate that the beliefs are true, coherence is a necessary feature of true beliefs. A related weakness of the coherence theory of truth is the criticism that it “seems possible for a coherent system of beliefs, even an ideal coherent system, to be false. We can imagine a system of beliefs that describes a fantasy world, one substantially different from the actual world” (Reference SchmittSchmitt, 2004, p. 15). Likewise, Hammond (Reference Hammond1996; Reference Hammond2007) has argued that coherence of judgment does not guarantee correspondence of judgment with facts. Baron (2004) uses the terms coherence and calibration, rather than correspondence, to describe these approaches. Baron (2008) writes, “judgments can be coherent without being calibrated. For example, I can say that the probability of heads is .90 and the probability of tails is .10. These two are consistent with each other, but not with the facts” (p. 119).

Because the coherence of a set of beliefs does not guarantee truth, some correspondence of the beliefs with facts is needed. Despite the limitations of C&C theories of truth, and their historic opposition, they are inseparable. If nature is necessarily lacking in self-contradictions, as the coherence theorists argue, then beliefs that correspond with nature should ultimately be coherent as well. Coherence is seen as a necessary, but insufficient condition for truth. Because coherent beliefs systems can be widely accepted and false, such as the previously accepted view that the earth was flat, correspondence is inescapable as a criterion of truth.

Modern scientific reasoning advocates using both coherence in the form of rationalism and correspondence in the form of empiricism. Coherence is used to organize data through the creation and modification of theories, and in reasoning about specific hypotheses. Theories must be coherent. That is, theories cannot be self-contradictory and generally, they must be consistent with other widely held beliefs within that scientific community. It was for this very reason that Einstein never accepted the probabilistic nature of quantum physics and the notion of complementarity. Einstein rejected the idea that a quantum element could act as both a particle and a wave and assumed that there must be some coherent explanation that has yet to be discovered. New theories that contradict widely held beliefs face an up-hill battle because they challenge normative beliefs. In time, such theories can achieve normative standing as the field reexamines its core beliefs and engages in a paradigm shift (Reference KuhnKuhn, 1962/1996; Reference ZimanZiman, 1984).

From theory, specific hypotheses are developed and then tested empirically. Hypotheses are evaluated for how well they correspond to empirical facts. If the hypothesized data are found to correspond to the observed data, then we claim to have supported the coherent theory. If not, then we try and make sense of the data and the theory and engage in rationalism to modify the theory or explain the discrepancy. In order for scientific theories to be considered true, they must: 1) be internally consistent, 2) be consistent with other widely held beliefs and 3) correspond with empirical facts. Internal consistency and consistency with other beliefs are examples of coherence criteria while a match with empirical facts is an example of a correspondence criterion. In the search for scientific truth, JDM researchers, embrace the strategies of both C&C.

Despite the embrace of C&C concepts in our practice as scientists, it is rare for researchers in JDM to explicitly use the terms C&C and even rarer for researchers to assess both the C&C of human judgment (for a notable exception, see Reference Adam and ReynaAdam & Reyna, 2005). Hammond’s claim is that research on the competence of human judgment often paints a contradictory picture because some researchers, implicitly or explicitly, adopt one criterion while other researchers adopt the other. Without the recognition that they are using different kinds of criteria, the conclusions cannot be synthesized. Research on base rate neglect provides an example of Hammond’s claim.

3 Base rate neglect: An example of Hammond’s claim

Research on our ability to use base rate information is a good example of how dominant the coherence approach is in evaluating judgment. The vast majority of evaluations of base rate usage utilize the criterion of Bayes’s Theorem. Bayes’s Theorem is a normative standard based on logic which makes it a coherence criterion.

Kahneman and Tversky published “On the Psychology of Prediction” in 1973, which included the now famous lawyer and engineer problem. In their experiments, subjects were presented with a variety of word problems and asked to estimate probabilities and likelihoods. The correct answer was established by the use of Bayes’s Theorem. When provided with word problems that include base rate information and specific information from a test, “graduate students relied on a description derived from such tests and ignored the base rates” (p. 239). When base rates were manipulated in the lawyer and engineer problem, they had “a minimal effect on subjective probability” (p. 242).

Conclusions drawn from research on base-rate neglect argued that base-rate neglect was an inevitable bias and a result of either heuristics, incompetence, or both (see Koehler, 1996 for a review of this work). In an often quoted statement, Bar-Hillel (1980) wrote that, “The genuineness, the robustness, and the generality of the base-rate fallacy are matters of established fact” (p. 215). Reference Gigerenzer and HoffrageGigerenzer and Hoffrage (1995) would later show that base rate usage could be improved by changing the information format of the word problems from probability to a frequency format. Although there has been debate about whether or not this result is due to the information format itself or a confound in the presentation of the summary statistics (see Reference Neace, Michaud, Bolling, Deer and ZecevicNeace, Michaud, Bolling, Deer, & Zecevic, 2008 for a recent example), the approach is still fundamentally that of the coherence variety. A word problem is presented and judgment is assessed against the normative standard of Bayes’s Theorem.

In Koehler’s review of this literature, he points out that there are few instances in the real world where Bayes’s Theorem can be unambiguously mapped to provide a clear criterion. He questions the conclusions from this literature by stating, “We have been oversold on the base-rate fallacy in probabilistic judgment from an empirical, normative, and methodological standpoint” (Reference KoehlerKoehler, 1996, p. 1). In other words, Koehler argues that the coherence criterion of Bayes’s Theorem has few instances where a clear correspondence criterion is available. Indeed, base rate neglect has seen very little in the way of investigation using correspondence criteria.

Goodie and Fantino (Reference Goodie and Fantino1995; Reference Goodie and Fantino1996; Reference Goodie and Fantino1999a) introduced a correspondence criterion to the study of base rate neglect when they provided subjects direct experience with base rates and individuating information and required subjects to make predictions in simulated environments. Simulations such as these create micro-worlds where the correspondence of judgments with an ecological criterion (that is, the ecology of the micro-world) can be assessed. Although Goodie and Fantino did observe greater sensitivity to individuating information than base rate information, they noted that there might be some environments where the strategy of base rate neglect would produce better correspondence. They write, “… when base rates change relatively often and cue accuracy relatively seldom isn’t it just as well to underweight base rates, since they’re liable to change at any moment?” (1999b, p. 327). In examining base rate neglect with a correspondence criterion, they opened the door to considerations of how base rate usage might be shaped by correspondence.

Reference Dunwoody, Goodie and MahanDunwoody, Goodie, and Mahan (2005) sought to examine how manipulating correspondence influenced base rate usage. Using a design based on that of Goodie and Fantino (Reference Goodie and Fantino1995; Reference Goodie and Fantino1996) they created several different simulated environments in which subjects had to learn to accumulate points by accurately predicting binary events. Subjects directly experienced base rates and the accuracy of individuating information over a series of learning trials. Base rates were defined by the probability that a particular event was the correct response and cue accuracy (the individuating information) was defined as the probability of the cue, given the correct response. Subjects were randomly assigned to either a condition where cue accuracies were stable and base rates varied or a condition where cue accuracies varied and base rates were stable. Correspondence was assessed as a match between the choice of the subject and the empirical outcome in the micro-world. In this task, correspondence was equivalent to reinforcement because subjects received a point for every correct prediction. After 200 training trials with outcome feedback, subjects completed 100 additional without feedback. Base rate usage was evaluated in these last 100 trials. Although the long-term reinforcement of base rates and cue accuracies was equal, correspondence would be more stable if subjects relied on the information source that was stable. Subjects who had experienced variable base rates chose a response that matched the base rate 37% of the time while those that experienced a stable base rate chose a response that matched the base rate 56% of the time. Experienced correspondence had a significant impact on base rate usage in both studies 1 and 2 despite the fact that long-term correspondence was identical between conditions, only short-term correspondence varied.

In study 3, Dunwoody et al. expanded the investigation by manipulating long-term correspondence as well. Short-term correspondence (information consistency) was manipulated as it was in studies 1 and 2 and orthogonally crossed with long-term correspondence (the reinforcement rate using a particular information source). Would subjects still prefer consistent information if it resulted in a 10% lower reinforcement rate? Subjects switched from utilizing the consistent information source in studies 1 and 2 to utilizing the information source with the highest long-term correspondence in study 3. When base rates were inconsistent but resulted in 10% higher correspondence, subjects chose base rate responses 69% of the time. When base rates were consistent but resulted in 10% lower correspondence, subjects chose base rates only 26% of the time.

The combined results of the studies by Dunwoody et al. (2005) show that base rate usage varies as a function of experienced correspondence. Subjects will use, or neglect, base rates depending on how base rates help them predict events. They are sensitive to both short-term manipulations of correspondence consistency that result in no long-term gains and long-term manipulations of correspondence that cause more short-term losses. These studies demonstrate that subjects have a sophisticated sensitivity to base rate information when it is experienced in a way that allows them to evaluate the correspondence value of using base rate information. This conclusion is strikingly different from the more pessimistic conclusions drawn when base rate usage is evaluated only with coherence criteria.

4 Hammond’s claim applied to the field of JDM

Coherence based theories of judgment focus on evaluating the competence of judgment using logic and mathematics as criteria. For example, do judgments follow the laws of probability? Do decisions violate transitivity? Are probability judgments consistent with Bayes’s Theorem? Do judges make conjunction probability errors? Correspondence based theories of judgment focus instead on evaluating the predictive ability of a judge based on some ecological criteria. For example, do judgments accurately predict events in the world? How often is weather prediction accurate? How often are medical diagnoses accurate? How well calibrated are judges? These two standards for evaluating judgment have produced different conclusions about the competence of human judgment aside from the base rate example provided above.

Specifically, Hammond (Reference Hammond1996; Reference Hammond2007) argues that the Heuristics and Biases (H&B) program of Kahneman and Tversky emphasizes coherence criteria and a pessimistic view of human judgment while the Fast and Frugal Heuristics (F&FH) program of Gigerenzer and colleagues emphasizes correspondence criteria and an optimistic view of human judgment. Although research in both traditions focuses on heuristics, their conclusions about the competence of human JDM differ.

Tversky and Kahneman write that “people rely on a limited number of heuristic principles which reduce the complex tasks of assessing probabilities and predicting values to simpler judgmental operations. In general, these heuristics are quite useful, but sometimes they lead to severe and systematic errors” (1982, p. 3). Their program is widely known as the H&B program and this title emphasizes their main findings: heuristics produces biases, or systematic errors, in judgment. Research in this tradition regularly adopts coherence criteria in assessing competence and human judgment is often shown to deviate widely from these coherence criteria. According to much of the research in this tradition, human judgment fails the coherence assessment and therefore humans are not rational. For example, Reference Kahneman and TverskyKahneman and Tversky (1972) stated, “man is apparently not a conservative Bayesian: he is not Bayesian at all” (p. 450).

Reference HammondGoldstein and Hogarth (1997) point out that the H&B program has often been criticized. They state, “experiments were conducted so that the word problems set up a ‘trap’ that subjects would fall into if they were using a particular heuristic. In this way, biased behavior was merely a device used to provide (dramatic) evidence that hypothesized psychological processes were at work.” This research agenda emphasizes, “that behavioral deviations from a presumed standard of rationality are the ‘interesting’ phenomena requiring explanation” (p. 26). In other words, the presumed standard of rationality from which people deviate is typically a coherence criterion.

Much, but certainly not all, of the research in the correspondence tradition is rooted in the work of Egon Brunswik. Brunswik was a functionalist, influenced heavily by the work of Darwin and Tolman and was concerned primarily with how organisms adapt to their environments (Reference Doherty and KurzDoherty & Kurz, 1996; Reference Dunwoody, Goodie and MahanHammond & Stewart, 2001; Reference Tolman and BrunswikTolman & Brunswik, 1935/2001). Brunswik was interested in issues of perceptual constancy (Reference BrunswikBrunswik, 1944/2001; Reference Doherty and KurzDoherty & Kurz, 1996). Perceptual constancy is fundamentally focused on correspondence; the correspondence between a percept and the distal object. Brunswik (1957/2001) believed that the proper focus for psychology was the understanding of how the organism achieves this correspondence. Hammond is credited with demonstrating the relevance of Brunswik’s arguments for understanding human judgment (Reference BrunswikCooksey, 1996; Reference HammondGoldstein & Hogarth, 1997). While Hammond extended Brunswik’s initial ideas, he retained the strong focus on achievement, defined as the correspondence between judgment and empirical event, as the central problem of psychology. As such, most JDM researchers in this tradition focus on how well judgments correspond to empirical events in the world.

Correspondence based research also has a long history in what is known as multi-cue-probability-learning (MCPL) (Reference BrunswikCooksey, 1996; Reference Hammond and SummersHammond & Summers, 1965; Reference HolzworthHolzworth, 2001). Studies investigating MCPL often create micro-worlds where empirical accuracy can be measured as correct predictions within the micro-world. Cooksey (1996) states that:

Achievement is a correlation between an individual’s judgments and some ecological criterion and is a measure of correspondence. While this research has shown many limitations on human judgment, it has often demonstrated that human judgment is capable of achieving the level of accuracy allowed by the environment (Reference BrunswikCooksey, 1996).

Gigerenzer and colleagues have taken a similar approach to evaluating human judgment by emphasizing the correspondence of human judgment within a particular ecology. Reference Goodie and FantinoGigerenzer and Todd (1999) explicitly state that their research agenda “dispenses with the focus on coherence criteria …. Instead, we study correspondence-based performance of heuristics in real-world environments” (p. 28). They introduce the term “ecological rationality” to distinguish their correspondence-based rationality from the more traditional coherence-based rationality. Their findings are consistent with the title of their 1999 book, Simple heuristics that make us smart. In short, they argue that F&FH are essential for adaptive human judgment. They argue that the H&B approach has focused on artificial environments where heuristics are likely to produce biases. However, their own approach is explicitly biased in the opposite direction. They state that they “focus on the ways and settings in which simple heuristics lead to accurate and useful inferences” (p. 28).

Gigerenzer and colleagues’ research on F&FH has shown that given the right environment, intuitive use of heuristics that may lack coherence can work well in achieving correspondence. Gigerenzer and Kurz write:

Although both the H&B program and the F&FH program study heuristics, they have reached different conclusions about the competence of human JDM. While much of the research in the H&B program has focused on demonstrating a lack of coherence in human judgment, and hence a lack of logic and rationality, the F&FH program of Gigerenzer has focused on demonstrating correspondence in human judgment, and hence ecological rationality. Much of the research in the H&B program ignores correspondence criteria in defining “rational judgment” while the F&FH program explicitly rejects coherence criteria in favor of correspondence criteria. Similarly, research in the Brunswikian tradition, including MCPL, typically ignores coherence criteria in assessing human JDM.

In the philosophy literature, the coherence of a judgment is considered a necessary but insufficient feature of truth. True beliefs are those that are both coherent and correspondent. It may be likewise true in the field of JDM that increases in the coherence of judgment are (imperfectly) associated with increases in the correspondence of judgment. Baron (2008), using the term calibrated in place of correspondent, writes, “If my judgments are perfectly calibrated, however, they must also be coherent” (p. 119). However, few studies have empirically examined the relationship between C&C. Dunwoody et al. (2005) found that, while manipulations of experienced correspondence influenced base rate usage under conditions of direct experience, the same manipulation failed to improve Bayesian responses to word problems in a related task. Reference Adam and ReynaAdam and Reyna (2005) examined the coherence and correspondence of experts’ judgments about sexually transmitted infections and noted a disconnect between coherence and correspondence measures. They write:

Reference Weiss, Brennan, Thomas, Kirlik and MillerWeiss, Brennan, Thomas, Kirlik, and Miller (2009) examine performance (not judgment) in a golf-putting task and found that a coherence based measure (CWS) correlated with a correspondence based measure (the mean absolute distance between the golf ball and target) at .676. This is a strong correlation and indicates that at least under some conditions, coherence is correlated with correspondence. It is conceivable that task differences between studies might account for the differing relationships observed between coherence and correspondence. The research findings suggest that it might be best to conceive of C&C assessments as orthogonal dimensions. When assessed jointly it is possible to be high on both dimensions, only one dimension, or neither dimension.

5 Critiquing Hammond’s claim

Hammond’s claim is essentially correct; researchers in JDM either adopt coherence or correspondence criteria and the adoption of different criteria has lead to different conclusions about the competence of human JDM. Although I support Hammond’s goal of greater recognition of C&C in the field of JDM and am in agreement with his main claim, I have two criticisms of his claim. First, Hammond’s focus on research programs paints with too broad a brush and second, his framework is incomplete. I address each point below.

While there is undeniable benefit in examining schools of thought, or programs of research, important connections between the programs, and subtleties within the programs, can be lost when painting with such a broad brush. Although in general Hammond is correct in his categorizations, one does not have to look far to find notable exceptions. Calibration research, which focuses on the correspondence of judged and actual probabilities, is a main area of research in the H&B program (see Reference Lichtenstein, Fischhoff and PhillipsLichtenstein, Fischhoff & Phillips, 1982 for an early review of this literature). Similarly, a traditional measure in Brunswikian research is cognitive control, written as R_s in the lens model equation (see Reference BrunswikCooksey, 1996; Reference Hammond and SummersHammond & Summers, 1972), which measures the degree to which an individual judge consistently applies his or her policy. Research in the Brunswikian tradition also examines judgments of repeated cases to examine the consistency of judgment. Both cognitive control and consistency focus on intrapersonal coherence, not correspondence (see Reference Beckstead and StampBeckstead & Stamp, 2007 for a recent example utilizing measures of both cognitive control and consistency).

It is therefore an oversimplification to discuss the entirety of the H&B program as coherence oriented and the entirety of the Brunswikian program as correspondence oriented. While each program may emphasize one type of criterion over the other, C&C are used by both research traditions. It is more accurate to use the C&C distinction to classify the criteria used to assess judgment, than to use it to classify research programs, which are much broader and more nebulous. Classifying assessment criteria, rather than research programs, may also make it more likely that researchers examining a specific topic do so via multiple criteria. To achieve a more complete understanding of human competence, it is necessary to examine topics via multiple criteria, as demonstrated in the base rate neglect example above.

My second critique, that the C&C framework is incomplete, is based on two points. First, it ignores the distinction I introduced earlier in this paper between intrapersonal coherence and interpersonal coherence. Intrapersonal coherence is the examination of consistency within a person. Assessing judgments and decisions for transitivity or consistency are good example of intrapersonal coherence. The phenomenon of cognitive dissonance shows that people are motivated to achieve intrapersonal coherence when inconsistencies are made salient. Interpersonal coherence is the examination of consistency in belief or judgments among people. Specifically, assessing judgments and decisions of a subject against the normatively held beliefs of others, such as Bayes’s Theorem, is an assessment of interpersonal coherence. Although both appeal to logic, assessing judgments for internal consistency is different from assessing judgments against normative standards of which the judge may be unaware. It is easy to see that measures of interpersonal coherence need not also measure intrapersonal coherence but the reverse is not true. Consistency within a person’s beliefs/judgments is a normative standard of logic and therefore assesses both intra and interpersonal coherence.

The second aspect of Hammond’s C&C framework that is incomplete is the absence of pragmatism. Coherence and correspondence are two of the three main philosophical theories of truth, pragmatism is the third. In adopting only two of the three main theories of truth for use as a categorization scheme, Hammond’s C&C framework neglects an important class of assessment criteria, goal-oriented assessment. I will first address pragmatism as a philosophical theory of truth and then explain why it is a necessary criterion for human JDM.