I. Testimony and Knowledge before Facebook

Humans learn useful things by communicating with one another without having to directly experiment and learn from the world. When a speaker makes an assertion with the intent that it be accepted as true, listeners will use that testimony to adjust their understanding of the world. Although the speaker could be lying, the speaker’s reputation will suffer if listeners discover the deceit. Unintentional mistruths can also occur, but the speaker’s reputation has some disciplining effect on this too. The reputational sanctions are particularly relevant for assertions based on “harder” information—that is, factual claims that can be independently verified by listeners without a good deal of interpretation.

Thus, even though a listener may continue to have doubt and uncertainty about a proposition, her probabilistic predictions about its truth are refined and improved with the help of testimony. This ability to learn from the testimonial reports of others is one of our species’ superpowers as it relieves us from having to start every epistemic quest with only the raw materials of our firsthand observations.

In principle, having access to more testimony should bring epistemic benefits by increasing the evidence available to individuals and hence producing more accurate beliefs. In fact, we show that advances in communications technology that increase our access to testimony also increase the likelihood of selection effects, causing beliefs to become more radicalized and less true. To understand why this is so, let’s first explore the relationship between testimony and knowledge in the pre-Internet era using an illustration.

Suppose Simone is a rational alien from planet Bayesianus.Footnote ⁴ He, like everyone on his planet, is an epistemic agent. He is motivated exclusively by a desire to ascertain the truth and he is able to think in an ordered, logical way. His beliefs about the truth of a proposition at any given time can be expressed as a credence between 0 (false) and 1 (true). His credences are updated according to the axioms of probability as he acquires new information over time.

Simone is not an expert of American politics, but has become focused on the question $ \varphi $ : “Is the president of the United States a bad one, who makes worse-than-average policy decisions?” Being completely ignorant at the beginning, Simone starts with a prior of $ cr\left(\varphi =T\right)=\frac{1}{2} $ .Footnote ⁵ He then teleports to Earth to investigate the answer.

Shortly after arrival, let’s assume that Simone receives a piece of evidence about the president’s performance. For clarity, we provide a highly stylized and constrained example, though the ideas generalize. Suppose Simone can receive one of three possible signals, $ a,b, $ or $ c $ , by chatting with somebody on the street ( $ a $ ), reading an op-ed in the first newspaper he finds ( $ b $ ), or casually listening to talk radio ( $ c $ ). Each of these pieces of evidence is a signal that Simone can use to update his credence, where $ s\in \left\{a,b,c\right\} $ .

However, Simone also knows that the evidence in $ a,b, $ $ c $ is merely suggestive on its own and cannot alone give him confidence. This is because the likelihood distribution functions of receiving these signals under conditions where $ \unicode{x03C6} $ is true or $ \varphi $ is false are not very informative. Simone knows, for example, that an expression of disappointment about the president from a single nonexpert’s opinion on talk radio is quite likely to occur whether the president is doing a good job or not. Moreover, the information contained in these signals is likely to be softer in nature—the product of someone else’s interpretation rather than hard data that Simone can independently verify. Further, even if Simone could independently verify the harder facts that testifiers used as the basis for their conclusions, he knows he lacks the expertise to independently process and interpret this evidence.

Nevertheless, Simone is at least able to update his initial credence to $ cr\left(\varphi =T|s\right) $ , where this credence will depend on the likelihood distribution functions of receiving each signal under conditions where $ \varphi $ is true or $ \varphi $ is false and in accordance with the Bayes rule.Footnote ⁶ Suppose that, depending on the signal he receives, Simone will update his prior as follow: $ cr\left(\varphi =T|a\right)=0.391 $ , $ cr\left(\varphi =T|b\right)=0.483 $ , and $ cr\left(\varphi =T|c\right)=0.605 $ .Footnote ⁷ The precision of these calculations and the likelihood distribution functions that produce them, like the hypothetical as a whole, is far-fetched. But if our illustration works under the demanding and constrained assumptions of objective Bayesianism (which assumes that likelihood distribution functions are objective and known), it will also work a fortiori under more relaxed assumptions. This includes assuming that (i) agents are Bayesians but have only a subjective representation of likelihood distribution functions (that is, can conceive of, but do not know for sure, certain states of the world, together with their likelihood, that might apply to a certain object of investigation),Footnote ⁸ and (ii) individuals are unable to engage in formal complex computations but can efficiently use heuristics in its place (for example, one can know how to throw a ball without necessarily knowing the laws of physics governing this action).

Now, imagine Simone receives signal $ c $ , by listening to the radio, so that he updates his credence to $ cr\left(\varphi =T|c\right)=0.605 $ —a moderate belief that the president is bad. Because the Internet has not yet become popularized, Simone has limited options for what to do next. Media markets have high entry barriers because printing presses and journalism staff are expensive, and competition from television networks has captured the demand for smaller, local papers.Footnote ⁹ And access to broadcast media is restricted to some extent by limited usable frequency bandwidths and resulting licensing schemes. Like most other people, Simone can only turn to a handful of “prestige” national newspapers and network televisions, and maybe one local newspaper, to look for further evidence.Footnote ¹⁰

While pondering his options, Simone turns on the television in his hotel room and watches an interview with Dave, a respected political philosophy professor. Dave can help Simone learn whether $ \varphi $ is true because Dave, too, is an immigrant from planet Bayesianus who initially came to Earth with a clean slate of knowledge (namely, $ cr\left(\varphi =T\right)= $ ½). But unlike Simone, Dave has spent years studying American politics. Hence, the evidence available to Dave is more informative than the evidence independently available to Simone, both because this evidence will tend to be harder in nature and because Dave has the skills to verify the veracity of the information he receives. Let’s then assume that the signals Dave can receive about $ \varphi $ , the quality of the president’s policy decisions, are qualified signals, $ q\in \left\{d,e,f\right\} $ , where the likelihood distribution functions of these signals is much more informative than those of the set of signal $ s\in \left\{a,b,c\right\} $ about $ \varphi $ being true or false.Footnote ¹¹ Before doing the interview, Dave has received $ d $ . His updated credence on $ \varphi $ under the Bayes rule is $ cr\left(\varphi =T|d\right)=0.068 $ , strongly convincing Dave that the president’s policy is in fact a good one. If Dave had received one of the other signals, his updated credences would have been $ cr\left(\varphi =T|e\right)=0.524 $ , and $ cr\left(\varphi =T|\hskip0.3em ,f\right)=0.912 $ .

What will happen to Simone’s moderate belief that the president is bad after he listens to Dave, the expert with a high credence that the president is good?Footnote ¹² By applying the Bayes rule, Simone’s updated credence will be $ 0.101 $ ,Footnote ¹³ meaning that Simone will also come to be convinced that the president is, in fact, good.

Simone wanders the streets a little while longer, picking up more signals, but because none have the same authority and epistemic influence as Dave’s signal, Simone’s updating never again veers very far from $ 0.101 $ . He returns to his planet fairly confident that the president is making good policy decisions. His beliefs are informed by the well-honed testimony of Dave.

II. Testimony and Knowledge under the Influence of Facebook

Now, suppose we move Simone forward in time by a couple decades so that he visits Earth during the modern Internet era. His first testimony is the same as before—signal $ \mathrm{c} $ that updates his prior to $ cr\left(\varphi =T|c\right)=0.605 $ .

This time, however, it is going to be much easier for Simone to access evidence about $ \varphi $ . The Internet and social networks can provide virtually infinite sources of testimony, while many of the entry barriers to the media and broadcast markets have been removed to promote increased competition. In this changed environment, the opportunity costs to look for further evidence have considerably declined. One would expect this to translate into access to more total evidence and hence better beliefs. Instead, it will translate into polarization. Where do things go wrong?

Let’s go back to our illustration to try to answer this question. Imagine that, like a majority of adults in the United States these days,Footnote ¹⁴ Simone decides to use social networks to get more evidence on $ \varphi $ . After his initial exposure in the real world to the signal $ c $ and before watching the interview with Dave, Simone joins a social networking site and asks, “How has the president been doing? I hear $ c $ .”Footnote ¹⁵ Simone’s post is viewed by Jane and Saura, who are interested in the comment, or maybe even searched for something like it, because they, too, each independently observed $ c $ under similar likelihood distribution functions as Simone.

In this case, one could think that Simone’s update credence after talking with Jane and Saura—who share his same moderate belief that the president is bad—is unlikely to change much. After all, unlike Dave, Jane and Saura are not expert with epistemic authority and do share the same belief as Simone. Yet, when the three of them confer and verify that their observations of $ c $ were independent, each will wind up with a stronger posterior credence that the president is bad— $ cr\left(\varphi =T|c,c,c\right)=0.782 $ .Footnote ¹⁶ And if the next search or post from either of the three results in another, seemingly independent, discovery of $ c $ , and then another, and another (which could happen, for example, if other people who independently received the same signal $ c $ join the conversation), it would take only ten independent opinions for Simone’s, Jane’s, and Saura’s priors to be updated to a radical extreme, i.e., $ cr\left(\varphi =T|c\times 10\right)=0.986 $ .Footnote ¹⁷

To borrow from Jason Brennan, Simone and his social network friends have turned from “Hobbits” to “Hooligans,” the rabid sports fans of politics, simply by interacting in good faith to learn from each other.Footnote ¹⁸ Note that this is not a case of “post-truth,” although the effects are similar. The concept of “post-truth” has come to denote circumstances in which objective facts are less influential—if not altogether irrelevant—in shaping public opinion than appeals to emotion and personal beliefs.Footnote ¹⁹ Likewise, for Simone, the exposure to evidence that radicalizes him is likely to eclipse the value of harder evidence. But this does not happen because Simone will be biased and value how “he feels” more than hard evidence. Instead, Simone will rationally discount the value of that additional evidence based on the other evidence he already possesses.

To better see this, suppose that by the time Simone watches the interview with Dave after he has already engaged with his newfound social network friends. While Dave’s testimony will still have some influence on Simone’s belief, this influence will be much attenuated, $ cr\left(\varphi =T|c\times 10,d\right)=0.840 $ ,Footnote ²⁰ in spite of Dave’s greater expertise and access to harder evidence. Simone would have to get respite from the information circulating on his social network and encounter several random signals in order to reverse the effects of his social networks’ conversations.

Thus, while “post-truth” is often associated with a form of ideological purity that discounts the value of evidence tout court,Footnote ²¹ we claim that Simone’s transformation is more akin to a form of “rational dogmatism.” Unlike pure dogmatists, Simone does not a priori exclude the relevance of Dave’s testimony.Footnote ²² Yet, after he has interacted with his social network friends, Simone will come to be convinced—based on the evidence he has gathered in that interaction—that Dave’s evidence does not matter that much given the weight and consistency of his other evidence.Footnote ²³

Note that polarization would still have been the outcome if Simone had seen the interview with Dave as soon as he got to Earth. If this had occurred, Simone’s social networking post would have shared different content (namely, “I learned $ d $ —it does seem like the president is doing a good job.”) Then, instead of hearing from Jane and Saura, he would have heard from Jean and Sara, who also, independently, received a similar signal and have a similar starting credence. Jean, Sara, and Simone would then push each other rapidly to the far opposite extreme. Similarly, things would not have changed much had Simone received another signal, namely, $ a $ . In this case, he would have had a credence that $ \varphi $ is true, $ cr\left(\varphi =T|a\right)=0.391 $ Footnote ²⁴ before interacting on social media. If his search or his post led him then to meet another couple of individuals who also independently received $ a $ , his credence would have become $ 0.208 $ ,Footnote ²⁵ and if he had communicated with ten more individuals, $ 0.012 $ ,Footnote ²⁶ another radical credence.

Moreover, even the expert, Dave, is not immune from these effects. If he had conferred with friends—other experts like him—on social media about $ d $ , his own updated beliefs after meeting others with the same signal, independently derived, would lead to an even more extreme credence that would not be easily dislodged, even if he later received the signal $ f $ from a different, respected expert. Concededly, there is a qualitative difference between expert and laymen polarization, as long as we have reason to expect experts to behave like scientists and to value countervailing evidence as some of the most informative. But if academics have mixed motives that interfere with their quest for truth—if, for example, professional reputations are less often advanced by disproving the consensus opinion than they once were—then this difference might become increasingly slim. In the digital era, we cannot presume experts to be immune from what Lee McIntyre has dubbed “the dark side of interactive group effects.”Footnote ²⁷

The abundance of available signals has reduced the opportunity costs of looking for additional evidence, but the search and customization functions have also removed the role of randomness and constraint in epistemic journeys. Counterintuitively, the result is exposure to less, rather than more, total evidence. Easy access to a wide range of sources of testimony has made the subtle art of selection a determinative step in any unwary traveler’s quest for truth.Footnote ²⁸

With an abundance of testimony, epistemic discovery is better served when search functions are tempered by aggregation across the largest possible body of evidence. A further modification of our example helps illustrate. Assume that the society Simone visits on Earth is only composed of twenty-one people: ten individuals who independently received signal $ c $ , ten individuals who independently received signal $ a $ , and Dave. Now further assume that an individual who received $ c $ and an individual who received $ a $ fall in love and then exchange their signals (omnia vincit amor, after all). These individuals’ update credence on $ \varphi $ would then be 0.496.Footnote ²⁹ Assume now that our two individuals decide to get married and invite to their wedding the remaining population of polarized agents, so that all ten individuals who received $ c $ and the ten who received $ a $ get to communicate their evidence with each other. Our little society’s updated credence on $ \varphi $ would be 0.457.

Dave is also invited to the wedding, but he is late. By the time he arrives, the guests have already exchanged their evidence and updated their credence. After talking to Dave and hearing $ d $ , they would further adjust their credence on $ \varphi $ to 0.058. Yet, if Dave had received $ f $ , rather than $ d, $ our society would end up with a credence of 0.897. Therefore, neutralizing (or, more realistically, mitigating) selection effects also neutralizes post-truth and rational dogmatism effects. Once two groups of moderates talk to each other, Hobbits remain Hobbits, avoiding that intensively selected aggregation of evidence that takes place on the Internet. This is because when evidence is aggregated widely across individuals, what matters in the epistemic process is the kind of evidence one has. The aggregation of softer information no longer produces radical changes in one’s belief when that information comes from many heterogeneous sources of testimony. At the same time, the aggregation across a representative sample of people would increase the value of harder evidence when those facts are easily accessible from many, nonpolarized individuals, (that is, Dave’s well-honed testimony would reach a much larger number of people) for a net epistemic gain. Under the current state of affairs, however, the dark-side effect of Internet interactions based on soft information and selection mechanisms dominates the bright-side effect of increased access to harder information, producing an overall informational loss.

The question then is how one should proceed toward correcting selection effects. At first blush, the automated filtering of content by Facebook, Google, and other tech giants would seem the natural place to start studying and treating the disorder. Perhaps these platforms can devise new algorithms to expose their users to more representative (or, at least, less nonrandom) sets of messages. But we suspect that the information ecosystem as a whole, and the ways people use it, will be unusually resistant to policy intervention. The next two sections explore why this is so.

III. What Has Knowledge Done for Me Lately?

There is a question we have not yet asked—the proverbial elephant in the room: Why don’t people recognize the epistemic impairment that arises from selection effects? And why don’t we actively correct for their adverse impact on truth-seeking?

Simone, being Vulcan-like in his single-minded quest for logic and truth, and his knowledge about the likelihood distributions for different sorts of signals, should have worked out that the probability of receiving another signal $ c $ after reporting his own testimony would have been great regardless of whether the proposition $ \varphi $ were true or false. Simone should proactively search for evidence in a way that eliminates path dependency—that is independent from the signals he has received before. In time, however, as Simone learns about the customized nature of Internet searches and news feed algorithms, an alien like him will adjust his understanding of likelihood ratios for signals. But what about the rest of us, who, presumably, have seen countless times that Internet content is highly curated and path-dependent? Why do some of us, at least, ignore the power of content selection?

If people were motivated only by epistemic goals, they would search for people who are different from them to exchange evidence, like scientists do.Footnote ³⁰ But most people do not behave like scientists in everyday life, even when they are motivated by truth-seeking. While some have hypothesized that our inherent cognitive bias makes us ripe for manipulation and exploitation by those with an agenda,Footnote ³¹ we suspect selection will occur even absent a cognitive bias. For although we have epistemic reasons to engage in Internet discourse, we also have pragmatic reasons that are truth-independent. The main (but not only) pragmatic goal that can explain much of the epistemic failing in selection is the social one. We don’t want to fastidiously survey a random sample of information on the Internet; we want to find our friends and engage with them. But these are precisely the people who are most likely to give us a biased stream of cues about the way the world works.

In real life, our social circles are also, to some extent, nonrandom. Our first social circle—the family—shares genetic similarities in addition to geographic and cultural similarities; and school friends and neighbors, too, tend to sort themselves in ways where social peace, continuity, and shared values can take precedence over factual truth. Even before the Internet, social sorting was becoming more intense. Education and religiosity gaps emerged between urban and rural neighborhoods, and marriages bonded people who were more and more similar to each other. Thus, the evidence and signals we exchanged with each other for epistemic purposes have always suffered from some selection effects. But the Internet has vastly increased these effects, making it virtually costless, and therefore more likely, for people who are similar in ways that matter socially (and, therefore, likely to already share similar beliefs) to find and engage with each other.

Those who love the Internet (and we confess we are among them) usually believe that it has two awesome features: you can connect with people, and you can exchange information. But these two goals are often in tension for the reasons we explained above. Worse still, they cannot be neatly separated. Friends want to talk politics, learn from each other, maintain their good standing with each other, and converge on a shared understanding of facts. As the Internet makes it easy to share signals about complex topics like policy, economics, or public health, epistemic goals are more frequently at odds with pragmatic goals, and the latter can predominate the former. And so, there are sometimes pragmatic reasons sounding in social harmony and the desire for group acceptanceFootnote ³² to ignore selection effects in information, even when the selection problem is obvious. The digital era has less room for John Stuart Mill’s harm principle, in which truth emerges through exposure to opposing views, because our exposure is largely determined by social phenomena.Footnote ³³

There are also times, though, that epistemic goals are well aligned with pragmatic goals. This is most immediately observable when new knowledge will have a direct and highly consequential bearing on the learner. Suppose, for example, that Jane is a highly social agent who often prefers to agree with her friends even at the expense of knowing the truth. If Jane’s friend Saura told her to drink bleach in order to avoid catching a deadly virus, Jane is likely to stop and think. She will see that her pragmatic interest in social bonding may be diverging from her pragmatic interest in staying safe and healthy. Jane’s behavior on the Internet and elsewhere will look much more like the purely epistemic agent since the stakes of being wrong, in either direction, are so great.

Thus, we would predict that the variable mix of pragmatic and epistemic motives will cause people to seek information and update their beliefs in different ways depending on how directly consequential the particular proposition may be.

IV. Polarized People Polarize the Media

Individual Internet users are not the only actors with mixed motives. Media companies of every sort and, though we don’t like to admit it, academics too, have a mix of pressures and goals. They want to pursue truth and epistemic authority, but the pragmatic need to attract readers (or students) and turn a reliable profit is also a concern. When media markets had significant entry barriers, media companies could simultaneously strive for epistemic truth and fill their coffers. Since consumers had limited choice, every media “niche” was well populated, so that any strategic motives only partly jeopardized their epistemic mission.Footnote ³⁴

The Internet dramatically lowered the costs for new media entrants, making competition for people’s attention quite fierce. At the same time, for the reasons explained above, the users that media companies are in competition for are increasingly polarized. Media companies know that they will not be successful if they provide contrarian evidence in a market of polarized readers. After all, a news agency, for example, can anticipate that readers with an unlimited supply of accessible and inexpensive testimony that is curated based on prior beliefs will eventually abandon the news agency as a reliable source of testimony because they believe—with good reason—that the company routinely provides “misinformation.”

Thus, even traditional media organizations whose tenure and business models predate the Internet will have to supply cherry-picked arguments and true facts (which is easy enough to do in the age of abundant information) so that it can maintain its place as a reliable source of information for an audience. Alternatively, they can resort to models where a contrarian view is always presented to foster a sense of objectivity, regardless of whether this view has any evidentiary basis and often at the expense of further inquiry.Footnote ³⁵ One way or the other, epistemic motives are sidelined—it is a matter of survival.

Just as social sorting was intensified but certainly not caused by the Internet, the competitive pressure for media companies to compromise truth-seeking was also underway well before social networking websites became popular. In the book Network Propaganda, Yochai Benkler, Robert Faris, and Hal Roberts show compelling evidence that the strategy to stratify and pitch information to a self-radicalizing audience (particularly, but not exclusively, on the right wing of the ideological spectrum) was the playbook for the Fox News channel, which learned from the talk radio shows that came before it.Footnote ³⁶ Rush Limbaugh was the pioneer of “community” talk shows, where followers were allowed to call in, but only after vetting them and making sure they shared Limbaugh’s views. As put by Tom Nichols in his book the Death of Expertise, “[d]ebate … was not the point: the object was to create a sense of community among people who already were inclined to agree with each other.”Footnote ³⁷ MSNBC and Democracy Now! radio use the same strategies for the ideological far left. Every media company is constrained, to some degree, by the risk that it will lose its base if it veers too far from wherever its base currently lands on an issue.

There is little reason to believe that Internet media companies, including Facebook and Google, are immune from the same competitive pressures. There is room to wonder whether these companies became dominant because they model their algorithms based on the behavior and revealed preferences of their users. If so, aggressive attempts to undo selection effects could drive users to other platforms that better serve their niches.

Consider, for example, Richard Sorabji’s diagnosis and recommendations for the treatment of filter bubbles. Sorabji suggests that a change in business model would improve epistemic outcomes without badly undermining profits. Specifically, he suggests that social networking platforms should use subscriptions or contextual advertising to fund their operations rather than tailored advertising that profiles each participant. As a practical matter, Sorabji may underestimate the impact that this change would have on revenues. Since behavioral advertising has a “click-through” rate several times higher than contextual advertising,Footnote ³⁸ we would expect overall revenues to plummet without the help of tailored matching. But even assuming that a shift in business model is possible, it will not help if users are actively seeking out content based on some pragmatic objective that biases the flow of testimony they encounter.

Two empirical observations suggest there is evidence that Sorabji’s recommendations could backfire. First, even individuals who primarily use Facebook as a source of news are exposed primarily to news content that their friends share, not content shared by advertisers or paying propagandists. Their individual choices in both choosing “friends” and selecting from the content that appears on the news feed contribute much more to selection effects than Facebook’s algorithms do.Footnote ³⁹ Second, a restriction on behavioral advertising could reward polarizing content producers and penalize the more objective ones. In Europe, when the Data Protection Directive made targeted advertising much more difficult, advertisers were forced to rely on contextual cues based on the content of a website to decide where to place their ads. The biggest losers were general interest news websites, and the winners were websites that produced focused content that helped stratify Internet users into pigeonholes for the advertisers.Footnote ⁴⁰ Thus, the policies that Sorabji recommends could backfire by sending both users and revenue streams to more fragmented social media platforms.

This unearths two counterintuitive facts that complicate prior assumptions about sound policy. First, media concentration can have a positive influence on epistemic authority and social trust. Perhaps in time, with the right cultural or political shifts, people will change their behavior to better reward media companies when they prioritize truth and open-minded inquiry. In the short-term, though, media competition will have a negative effect on knowledge, introducing incentives for distortionary information practices at the expense of epistemic goals. The conventional wisdom of antitrust that competition enhances quality is strained. It may not be that consumers are better off when media companies compete, at least in the context of political knowledge, if consumers choose among media platforms based on pragmatic preferences (such as social goals) that come in direct conflict with truth.

Second, and relatedly, the selection effects that accelerate polarization among social participants on the Internet will be replicated in traditional media and even, to some degree, at universities and other educational institutions. Any authority that depends to some degree on the patronage of readers or listeners will have to compromise their epistemic mission to get closer to one or the other camp of a bimodal distribution. (At least, this is so for sources of information about issues like politics that have an indirect and ambiguous impact on the listener, and require interpretation to make sense of observations.) This is significant because conventional advice to get off social media and use a traditional, trusted source for news has missed the breadth of our epistemic affliction: even traditional sources of authority are catering to a social clique.

V. What To Do?

The dysfunction we have attempted to explain here is vexing and unusually hardened to policy intervention. We know this because we authors have different ideas about the best (no, the least-bad) solution. All three of us want to reduce selection effects and influence social factors so that seeking out opposing evidence is rewarded instead of punished. But there are no obvious interventions that have a high chance of success. So we will conclude with two rough sketches of policy implications that can hint at where the levers for reasonable government action might be.