It is an axiom older than the nation itself that the federal judiciary has “no influence over either the sword or the purse … [having] neither FORCE nor WILL, but merely judgment.”Footnote 1 (Hamilton Reference Hamilton, Madison, Hamilton and Jay1966). In other words, their authority depends on observers’ perceptions of their fairness and legitimacy. This raises an important question in political science, political and social psychology, and jurisprudence: can specific approaches to judicial decision-making or opinion writing impact public perceptions of courts’ fairness and legitimacy (Braman Reference Braman2009; Segal and Spaeth Reference Segal and Spaeth2002; Tyler Reference Tyler2006)?
Justices have, at times, explicitly alluded to these considerations in their opinions. For example, in Planned Parenthood v. Casey (1992), Justice O’Connor’s plurality opinion declining to overrule Roe v. Wade expressed concern that overruling a controversial decision under political pressure might create the perception that the Court’s decisions are unprincipled and vulnerable to political influence, compromising its legitimacy.Footnote 2 Justice O’Connor’s account suggests one theory as to how courts can maintain legitimacy — namely, that people are likelier to perceive the Court as a legitimate authority if they perceive its decisions as being non-partisan, principled, and not outcome-driven. By contrast, in Dobbs v. Jackson Women’s Health, overruling Roe v. Wade, Justice Alito’s opinion suggested that these concerns about legitimacy were misguided, and that the judiciary’s legitimacy derives primarily from reaching the most accurate or faithful interpretations of the Constitution — that is, the “correct” outcome, according to Justice Alito’s view of the Constitution.Footnote 3 Justice Alito’s account of legitimacy dismisses concerns about the Court appearing unprincipled or partisan, and instead emphasizes the outcome of the decision.
Research on what generates perceptions of fair judicial decision-making and the legitimacy of courts often divides along similar lines. One literature argues that such perceptions depend primarily on outcomes, not process. Under these accounts, members of the public perceive legal decisions as fair and legitimate when the outcome aligns with their ideologies or worldviews. Using mental processes such as motivated reasoning or cultural cognition, people automatically process information in a way that is consistent with their cultural and social identities, discounting evidence that threatens deeply held cultural and social identities while crediting evidence that conforms to them (Kahan, Jenkins-Smith, and Braman Reference Kahan, Jenkins-Smith and Braman2011; Taber and Lodge Reference Taber and Lodge2006). Individuals thus unconsciously assess fairness and legitimacy by reasoning backwards from the decision, viewing decisions they agree with as being decided fairly and vice versa.
Another set of theories suggests that people’s perceptions of judicial fairness and legitimacy also depend on whether the decision maker employed fair procedures. A large body of research in the procedural justice literature suggests that if a decision maker successfully conveys that their decision-making process was fair, that is, unbiased, neutral, impartial, and evenhanded, people will be likelier to regard the decision as legitimate, regardless of whether the outcome favors them (Tyler Reference Tyler2006).
Our study thus contributes to both bodies of work, one suggesting that incorporating elements of procedural fairness can improve support for judicial decisions (Gallagher and Ashford Reference Gallagher and Ashford2021; Strother and Glennon Reference Strother and Glennon2021; Tyler Reference Tyler2006), and one suggesting that perceptions of legitimacy are primarily tied to whether someone agrees with a decision’s outcome (Bartels and Johnston Reference Bartels and Johnston2020; Parker and Woodson Reference Parker and Woodson2020; Simon and Scurich Reference Simon and Scurich2011; Badas Reference Badas2016). Many studies in procedural fairness test the impact of fair procedures in in-person interactions with police, judges, and other government agencies. Our work tests the impact of procedural fairness in a different context — where language in a judicial opinion conveys elements of procedural fairness when justifying a court’s decision-making process. Specifically, we assessed whether a judicial opinion conveying elements of procedural fairness improves perceptions of fairness and legitimacy, even when the outcome is antithetical and potentially threatening to the reader’s worldview or cultural identity. These questions are especially pressing in the increasing partisan polarization that characterizes American politics generally (Hasen Reference Hasen2019; Mason Reference Mason2018; Abramowitz and Webster Reference Abramowitz and Webster2016) and the judiciary specifically, which has driven the Supreme Court’s approval ratings to historic lows (Brenan Reference Brenan2023).
In two survey experiments with samples drawn by Survey Sampling International, participants watched video clips in scenarios involving topics that are culturally polarizing. The first was a dashcam of a police chase where police ran a driver off the road; the second contained footage of an anti-abortion protest outside an abortion clinic, which police would later be called to shut down. We chose these two video vignettes because previous studies have shown that people’s cultural worldviews influence their perceptions of the facts and appropriate outcomes in these specific scenarios, making them well-suited for generating motivated reasoning and specifically identity-protective cognition (Kahan, Hoffman, and Braman Reference Kahan, Hoffman and Braman2009; Kahan et al. Reference Kahan, Hoffman, Braman and Evans2012).
After watching the video vignette, participants read an excerpt from a simulated judicial opinion explaining why the court ruled in favor of or against the authorities’ actions in these scenarios. Because previous studies had shown that these specific cases were culturally polarizing and generated identity-protective cognition (Kahan, Hoffman, and Braman Reference Kahan, Hoffman and Braman2009; Kahan et al. Reference Kahan, Hoffman, Braman and Evans2012), we expected that people would be divided — largely along ideological and identity-based lines — as to the correct outcome in these cases. Because we expected these decisions to be polarizing, they provided a good test for whether an opinion conveying elements of procedural fairness might mitigate some of the tendency to credit or discredit a decision depending on its alignment with one’s own worldviews.
In this 2 x 2 design, participants were randomly assigned to one of four conditions, which varied (1) which side won and (2) whether the written judicial opinion employed elements of procedural fairness in its reasoning (for example, evenhandedly weighing the views of both sides, respecting both parties’ interests, etc.). We examined to what extent both factors impacted participants’ perceptions of the judge’s fairness and the legitimacy of courts.
We found that agreeing with the outcome of an opinion had a large, positive impact on participants’ perceptions of fairness and legitimacy in both studies, while assignment to the procedural fairness condition had a smaller positive impact on perceptions of fairness but not legitimacy. The differences in effect sizes between receiving the procedural fairness treatment and outcome agreement were substantial, with outcome agreement having four times the positive impact on perceptions of fairness than assignment to the procedural fairness condition in our first study and ten times in the second. Overall, our results suggest that procedural fairness matters less to perceptions of fairness and legitimacy than substantive agreement with the outcome.
Fairness, Legitimacy, Outcome Agreement, and Procedural Justice
By fairness, we refer to the perception that judges have made an impartial decision without favoritism, meaning, for example, the judge has no personal stake in the issue, has no meaningful relationship with one of the parties, or has not decided a case according to their own ideological beliefs (Geyh Reference Geyh2013). Here, we mean not so much the use of court procedures associated with due process, such as notice, an opportunity to be heard, or present evidence, as these formal procedural mechanisms may be present in a proceeding in which the judge is nonetheless perceived to be biased or unfair. Instead, we refer to a sense of fair treatment, or that the judge was impartial, treated both parties with respect, and did not reach an outcome motivated by the judge’s own ideological priors. Our operationalization of judicial legitimacy, by contrast, relies on the well-studied concept of “diffuse support,” for example, Caldeira and Gibson (Reference Caldeira and Gibson1992), which relies on broader support for courts as decision-making institutions, distinct from agreement with any particular decision or set of decisions they make.Footnote 4
Our study is designed to further investigate two well-developed literatures that promote different answers as to what drives the perception that judges have made fair decisions and that courts are legitimate. One literature holds that people base their assessments of fairness and legitimacy on whether they agree with the decision outcome(s), rather than on anything judges do during the decision-making process. As Bartels and Johnson put it, “procedural perceptions do not drive the acceptance of decision outcomes … rather, policy agreement drives procedural perceptions” (Bartels and Johnston Reference Bartels and Johnston2020, 237). Specific court decisions and judicial institutions are fair and legitimate, respectively, only as long as they are in line with the views of most citizens (or elites).
This seemingly backwards reasoning process may be driven by motivated reasoning or cultural cognition, in which individuals tend to interpret evidence and perceive risks in accordance with their worldviews, or downgrade evidence that might threaten their cultural biases or group identities (for example, Kahan and Braman Reference Kahan and Braman2006; Kahan et al. Reference Kahan, Braman, Gastil, Slovic and Mertz2007; Redlawsk Reference Redlawsk2002; Taber and Lodge Reference Taber and Lodge2006). There is strong evidence that outcome agreement affects how judicial decisions are evaluated, particularly when the subject matter of the dispute involves what Skitka calls “moral mandates,” where objectionable decisions invite not only disagreement but also pose some threat “to people’s sense of both private and public personal identity” (Skitka Reference Skitka2002, 589). This evidence includes findings that laypeople found judicial decisions unacceptable when they disagreed with them (Simon and Scurich Reference Simon and Scurich2011); that individuals who perceive they are in disagreement with the Court’s decisions “ascribe lower legitimacy to the Court” than those who agree with them (Bartels and Johnston Reference Bartels and Johnston2013, 196); that research participants who read about a trial where the outcome was contrary to their cultural identity or commitments were more likely to express anger, judge the outcome as unfair, and steal a borrowed pen (Mullen and Nadler Reference Mullen and Nadler2008); that research participants were more likely to find state courts impartial if they agreed with their decisions (Barwick and Dawkins Reference Barwick and Dawkins2020); that survey participants found constitutional experts more credible when they agreed with their determinations (Braman Reference Braman2023); that citizens are more likely to support limiting a court’s power when doing so advances their political goals (Bartels and Johnston Reference Bartels and Johnston2020); and that support for both national and European Union courts varies alongside agreement with outcomes, rather than the venue (Madsen et al. Reference Madsen, Mayoral, Strezhnev and Voeten2022). The through-line for these studies is clear: individuals base their assessments of judicial fairness and legitimacy on whether they agree with the decision made.
The second body of research does not deny that individuals may reason this way, but is more optimistic about how legal actors can improve positive responses to their decisions even in the face of disagreement. These studies suggest that perceptions of fairness and legitimacy are at least somewhat dependent on the manner in which judges make or justify their decisions. The very existence of judicial opinions suggests some limit on our ability to accept decisions based only on their coherence with our own beliefs. Beyond this low baseline, studies have found evidence that judges can act in other ways to increase perceptions of fairness and legitimacy. For example, a large literature on “procedural justice” suggests that procedural fairness — defined as showing respect for all parties, being transparent and even-handed in the application of rules, and making participants in the legal system feel they have a voice in the process — is associated with perceptions of legitimacy, voluntary compliance, and respect for institutions (Tyler Reference Tyler2006; Tyler and Huo Reference Tyler2002). These findings have been replicated in a number of legal and criminal justice settings (for example, Gallagher and Ashford Reference Gallagher and Ashford2021; Shook et al. Reference Shook, Goodkind, Kolivoski and Ballentine2021; Elliott, Thomas, and Ogloff Reference Elliott, Thomas and Ogloff2011; Murphy and Barkworth Reference Murphy and Barkworth2014; Beijersbergen et al. Reference Beijersbergen, Dirkzwager, Eichelsheim, Van der Laan and Nieuwbeerta2015; Gottfredson et al. Reference Gottfredson, Kearley, Najaka and Rocha2007; Skogan, Van Craen, and Hennessy Reference Skogan, Van Craen and Hennessy2015; Wales, Hiday, and Ray Reference Wales, Hiday and Ray2010).
Consistent with procedural justice theory, Strother and Glennon (Reference Strother and Glennon2021), for example, found that a justice’s off-bench speeches can positively influence perceptions of judicial legitimacy in a nationally representative survey, while Gallagher and Ashford (Reference Gallagher and Ashford2021) discovered that Veteran’s Courts that explicitly honored or demonstrated respect for a defendant’s military service were more likely to be perceived as making fair decisions by a sample of veterans before those courts. These studies suggest that showing respect for all parties and conveying impartiality or even-handedness may improve public perceptions of fairness and legitimacy.
While not directly connected to the procedural fairness literature, other research on judicial legitimacy has found that support for courts decreases when decisions are framed as partisan rather than principled (Strother and Gadarian Reference Strother and Kushner Gadarian2022; Magalhães et al. Reference Magalhães, Skiple, Pereira, Arnesen and Bentsen2023; Woodson Reference Woodson2015). Individuals can accept a court with some level of sincere legal views different from their own, this argument goes, but not ones they feel are motivated by partisan ends (Gibson, Caldeira, and Spence Reference Gibson, Caldeira and Kenyatta Spence2005), such as in vignettes that made participants aware of the campaign financing of judicial elections (Gibson Reference Gibson2008). Similar results were seen in Woodson’s study of support for state supreme courts, which found that in states with highly visible elections, elected judges were viewed as less legitimate than appointed judges (though not in states where campaign activity was low or less visible) (Woodson Reference Woodson2017). While these studies do not specifically test the impact of procedural fairness on legitimacy, their findings suggest that legitimacy is dependent on avoiding being seen as one-sided or partial in one’s decision-making.
Our research thus adds to both the motivating reasoning and procedural fairness literatures. We take as given that decision agreement has some impact on how individuals assess decision making and decision makers; we seek to assess when and whether procedural fairness can mitigate its impacts. Our study also expands the scope of the procedural fairness literature by examining whether it can operate outside the person-to-person context in which it is usually theorized to work, that is, interpersonal interactions between government authorities and civilians.
Specifically, our investigation focuses on the impact of a judicial opinion that conveys that the court adhered to components of procedural fairness while making its decision. How a judge conveys their reasoning is an important aspect of procedural fairness, apart from the formal processes often associated with “Due Process,” such as adequate notice, an opportunity to be heard, and to introduce evidence, etc. (Meares and Tyler Reference Meares and Tyler2014). These formal processes may be in place, but a decision-maker may nonetheless make decisions, statements, or issue an opinion in a manner that suggests (or is perceived as suggesting) they are partial toward one outcome, and ideologically motivated to rule that way. If this happens, formal processes are not sufficient to ensure the decision-making process is fair or perceived as such. It is also critical that the judge (or decision maker) demonstrate — in their interactions with the parties, and in either their oral announcement of the decision or their written opinion — that their decision-making process adhered to the tenets of procedural fairness, such as evenhandedness, neutrality, respect for both parties’ interests, etc. This assures the parties that the decision maker is trustworthy, benevolent, evenhanded, and respectful of both parties. Without any assurance that the judge’s decision-making process was fair, the formalities of due process (notice, the hearing, etc.) are of lesser value.
Opinions are central to the legitimacy of judicial decisions, offering what some scholars refer to as reasoned elaboration, or justifying decisions in reference to the arguments made by the parties (Simon and Scurich Reference Simon and Scurich2011; Waldron Reference Waldron2023). Without such justification, holdings become little more than arbitrary exercises of power. Importantly, judges have considerable freedom in how they craft their justifications, providing a space where different choices in argument or rhetoric might lead to different reactions from the parties before the court, political elites, or the public.
We conclude this review by noting that attempts to improve public perceptions of legitimacy may elide deeper concerns over whether that authority is actually behaving in a normatively legitimate or just manner (Bell Reference Bell2017). Bell, for example, argues that the cynicism and distrust many Black residents have towards police have deep historical and structural roots that procedural justice cannot adequately address. Thacher similarly notes that a focus on procedural justice may lead reformers to prioritize superficial tactics to enhance public perceptions while ignoring harder questions of when coercive authority should be employed (Thacher Reference Thacher, Anthony and Weisburd2019). These critiques about police reform could easily be applied to any argument for employing elements of procedural fairness in opinion writing as a method of conveying fairness and improving perceptions of legitimacy. To the extent our study finds this is an effective tactic, this does not address concerns about the risk of using elements of procedurally fair language as a tactic to assuage or pacify public grievances about real injustices or abuses of power. However, to the extent our study finds the opposite — that procedural fairness in a judicial opinion is relatively insignificant, compared to how the outcome aligns with the reader’s identity — this study suggests that, at least in the opinion writing context, incorporating language of procedural fairness will not be an effective tactic for buying legitimacy when readers perceive the decision as being substantively unjust or threatening to their identity.
Hypotheses and Research Design
Given that written opinions are central to the exercise of judicial power and that judges have considerable freedom in how they craft decision justifications, we feel it worthwhile to examine whether judicial opinions employing procedural fairness principles can have similar impacts to what other studies have found when examining personal interactions. Here, we tested the impact of a judicial opinion conveying that the decision maker adhered to certain tenets of procedural fairness, compared to opinions conveying that the decision maker elided the values of procedural fairness by engaging in reasoning that was one-sided and conclusory.
Survey experiments are well-suited for our goal, as they allow researchers to randomly assign participants to read and respond to legal vignettes that vary along the variables of interest (for example, Gibson, Caldeira, and Spence 2005; Simon and Scurich Reference Simon and Scurich2011; Strother and Glennon Reference Strother and Glennon2021), better isolating the impact of our treatment (the reasoning style) on perceptions of fairness and legitimacy. Our two studies each operate on a 2 x 2 design, varying by which party wins and the type of justification the judge employs in the opinion excerpt participants read.
Based on the foregoing literature review, we formulated two hypotheses. First, based on procedural justice theory and related findings outlined above, we expect that the vignette conditions where the opinion excerpt conveys the judge employed elements of procedural fairness (discussed in more detail below) will generate higher perceptions of fairness and legitimacy, compared to vignettes where the judge’s reasoning is one-sided.Footnote 5 Going forward, we refer to these experimental conditions as “procedurally fair” and “one-sided,” respectively. Second, we expect that, in line with the literature on outcome agreement, survey participants will be more likely to rate decisions as fair and courts as legitimate when they agree with the substantive outcome of the decision. Given prior research (Simon and Scurich Reference Simon and Scurich2011), we also examine whether an opinion’s reasoning style (fair vs. one-sided) interacts with outcome agreement (such as disliking one-sided arguments when you disagree with the outcome but accepting them when you agree) as well as other factors, including demographics, ideology, and partisanship.
To test the impact of procedurally fair opinions on perceptions of decision fairness and judicial legitimacy, we used a 2 x 2 between-subjects design, where we manipulated (1) the reasoning style (procedurally fair vs. one-sided) and (2) the opinion outcome. Participants were assigned to one of four conditions, representing each combination of these factors. In both studies, participants watched a video that provided much of the factual context for the legal dispute and then read vignettes styled as newspaper articles describing legal opinions on controversial topics (discussed below) while also including an excerpt of those opinions. These excerpts varied in terms of both factors listed above: reasoning style (procedurally fair vs. one-sided) and the winner.
In our vignettes, opinions using “procedurally fair” reasoning convey that the judge’s decision-making process incorporated elements associated with procedural fairness: voice and participation, respect, neutrality, and trustworthy motives (Tyler Reference Tyler2006). To operationalize these components, the “procedurally fair” opinion contained five reasoning elements that the “one-sided” opinion did not: (1) acknowledging that the case was difficult; (2) recognizing that there were good arguments on both sides of the case; (3) explicitly restating the losing party’s argument and describing it as a “good point”; (4) explicitly stating that the winning party exaggerated their claims to some degree; and (5) stating that the Court engaged in “much deliberation”— that is, thought hard about the case.
These elements convey that the court’s decision-making process incorporated the elements of procedural fairness in the following ways. Considering both sides carefully and recognizing that both sides have good arguments demonstrates that the court considered both parties’ interests and took their claims seriously. This conveys fairness and impartiality, and also conveys respect for both parties by showing that the court was concerned about each party’s claims and interests. The court explicitly discussing both sides’ arguments could potentially provide a vicarious sense of voice or participation, as the parties see their own perspectives explicitly recognized in the court’s discussion. Finally, by acknowledging the difficulty of the case and explaining how much it deliberated, the court might convey trustworthy (or benevolent) motives — that is, that the court tried to take everyone’s interests into account and reach the fairest outcome. Benevolent motives are a component of procedural fairness that is associated with trust and legitimacy (Tyler Reference Tyler2006).
In contrast, our one-sided opinions were designed to convey that the court’s decision-making process was antithetical to these elements of procedural fairness. The opinions conveyed certitude and confidence in the outcome and cursorily dismissed the losing parties’ claims. As a contrast to the “procedurally fair” opinions, the one-sided versions contained the following reasoning elements, all dismissive of the losing party: (1) stating that the correct outcome is “clear” and not debatable; (2) stating that the losing party’s argument was inaccurate/incorrect; (3) stating that the right outcome is obvious, and (4) stating that anyone would see the facts the same way that the court did. These elements conveyed that the court’s decision-making was antithetical to components of procedural justice. Rejecting the losing side’s arguments cursorily, without considering or weighing them, conveys partiality and disrespect for the losing party. Furthermore, dismissing the losing party’s argument as a misstatement of the facts may convey a lack of respect for their view. Stating that the outcome is obvious and that no one could see it differently denies anyone with a different perspective a vicarious voice. All of this might lead a reader to infer that the court’s motives are not trustworthy — that is, that the court is motivated to reach a particular outcome, rather than carefully weighing the evidence to reach the fairest conclusion.
We designed each study’s vignettes to share as much common text as possible (swapping out phrases when possible) and to be roughly the same length, to avoid introducing confounding factors that could influence how the reasoning is perceived. Relatedly, we contrasted procedurally fair opinions with one-sided opinions, rather than including a “neutral” opinion as a control condition because we did not find it feasible to create a realistic “neutral” opinion with neither fair nor one-sided reasoning, short of a decision merely announcing an outcome without any reasoning at all. Such a justification would be considerably shorter and potentially less persuasive than either of our conditions, and both these differences could confound the effects of our experimental variables. Samples of the fair and one-sided conditions from Survey 1 can be seen in Table 1, while both survey instruments in their entirety appear in Online Appendix A.
Opinion excerpts from newspaper article vignettes in Study 1
We highlight the manipulated text in bold here. Such emphasis was not present in the vignettes shown to participants.
We note that these contrasting styles of justification have real-life analogues. For example, in Kennedy v. Louisiana (2008), the Supreme Court held its prior ban on execution for crimes other than homicide covered a death sentence for the aggravated rape of a child. In his majority opinion, Justice Kennedy wrote that “it must be acknowledged that there are moral grounds to question a rule barring capital punishment for a crime against an individual that did not result in death” (435). Such a statement attempts to show respect for both sides by recognizing moral disagreement over the issue. By contrast, Justice Scalia’s opinion in Scott v. Harris (2007) provides a useful example of the one-sided approach. Reviewing a dashcam video of a police chase in an attempt to determine whether police were justified in running the plaintiff off the road, Scalia wrote that “no reasonable jury” would have disagreed with the officer’s decision (380).
We ran two separate survey experiments with this general design. We discuss the design and results of each study separately below.
Study 1: Police Chase Vignette
Design and Measures
Our first survey experiment was hosted by Qualtrics and distributed by Survey Sampling International (SSI) to a nationally representative sample in November 2017.Footnote 6 SSI operates opt-in online panels for participant recruitment, which are thereafter balanced along relevant demographic factors to ensure a diverse panel. For this study, SSI recruited 861 completed surveys.
As noted above, we designed each study around a legal conflict within a policy area that prior research shows may elicit motivated reasoning (Kahan, Hoffman, and Braman Reference Kahan, Hoffman and Braman2009; Kahan et al. Reference Kahan, Hoffman, Braman and Evans2012). The vignette for Study 1 centered on whether a police officer had acted constitutionally when running the plaintiff off the road — causing permanent paralysis — during a high-speed pursuit. The facts of the case are based on Scott v. Harris (2007), in which the Supreme Court held that the officer’s conduct was reasonable, making them entitled to summary judgment. This holding meant that a central factual issue in the case, whether the plaintiff’s conduct posed a “substantial and immediate risk” to others, would not go to a jury. The Court made the unusual move of referencing a dashcam video of the chase as the basis for its conclusion (Kahan, Hoffman, and Braman Reference Kahan, Hoffman and Braman2009).
All vignettes in Study 1 were presented as newspaper stories that describe the facts of the case, relay the outcome, embed a (shortened) video of that same dashcam footage (making clear the court relied on this footage when making its decision), and include an excerpt of the opinion’s reasoning. As with Scott v. Harris, we used the United States Supreme Court as the deciding court in this vignette. Participants were assigned to one of four conditions, varying by whether the court sides with “Grey” (our name for the plaintiff) or the police, as well as whether the opinion excerpt uses procedurally fair or one-sided language (operationalized as described above).
To determine whether participants were sensitive to the differences in reasoning style — and the specific elements we described above — we ran two sets of manipulation checks. First, after reading the opinion, we asked participants (on a six-point Likert agreement scale) whether the court’s reasoning reflected several elements presented in the “fairness” condition, but not in the “one-sided” condition. These items were: Whether the court found the decision difficult, whether the court recognized that the losing party had strong arguments, and whether the court recognized that the winning party exaggerated some of their claims. Participants answered these questions correctly two-thirds of the time on average, with the question means being significantly different under t-tests in the expected direction.Footnote 7 This indicates that participants in the procedurally fair condition are overall more likely to recognize the opinion as containing more elements that might be associated with procedural fairness (such as considering both sides, articulating both parties’ points of view), compared to those who read the one-sided opinion. Though the proportion of participants passing the manipulation check was lower than desired, this may reflect the difficulty in manipulating participants with specific rhetorical styles, and thus still provides a useful test of the treatment’s effect — that is, whether members of the public are, in fact, sensitive to elements of reasoning style in a written opinion.
Second, SSI provided data on how long it took for participants to complete the survey. We retested our models, excluding participants who took less than 300 seconds (a time frame in which we believe participants could not have reasonably watched the video, read the excerpt, and taken the survey) to complete the experiment, we found no substantive difference in the direction, size, or statistical significance of our model variables after doing so. Given this, and given that excluding participants who fail manipulation checks is increasingly disfavored (for example, Aronow, Baron, and Pinson Reference Aronow, Baron and Pinson2019), we opted to maintain all participants who completed the surveys.
After reading the story setup, watching the video, and reading the opinion excerpt, participants answered a battery of questions, from which we created our model variables. To measure our dependent variables — perceptions of fairness and judicial legitimacy — we asked two series of questions on a six-point Likert scale that we later scaled into two variables (each variable scaled well, with an alpha of 0.82 for the fairness measure and 0.88 for the legitimacy measure). Questions on fairness included items such as whether the subject agreed the court was impartial, considered both sides of the case, and if the judge appeared to be biased. Our legitimacy measures are based on items developed by Gibson, Caldeira, and Spence (2005), including questions such as whether the power of the courts should be reduced or whether courts can generally be trusted to do the right thing. For Study 1, we inverted the usual polarity of the Gibson questions, meaning that higher scores indicate lower perceived legitimacy and greater support for constraining the power of courts to make important decisions.
Our independent variables track our hypotheses. We included dummy variables for the experimental condition, effectively controlling for the treatment, the winner, and their interaction. We measured subject agreement with the decision outcome on a six-point Likert scale. Given our hypotheses, we expected perceptions of fairness and legitimacy to be lower for participants in the one-sided condition relative to the procedural fairness condition, as well as when they disagreed with the outcome.
We also asked participants a series of demographic questions, including ideology, partisanship, age, education, race/ethnicity, and gender. For race and ethnicity, we employed the two-step procedure employed by the Census Bureau, afterwards combining the two answers. For gender, we collapsed our categorical variable into a binary variable of “male and “not male” for ease of interpretation and because the number of answers in categories other than male and female was small in number.Footnote 8 The full survey instruments for both studies are in Online Appendix A, while the demographic breakdown of the samples for each study is in Online Appendix B.
To test our hypotheses, we first regressed a dummy variable for our four experimental conditions on our dependent variables (the fairness and constraining courts scales), aiming to provide a comparison of the mean effect and confidence intervals for each condition and variable (with the plaintiff wins/fairness treatment as the reference category). We next added whether the respondent liked or disliked the judicial decision to this model. For ease of interpretation, we converted outcome agreement into a binary variable, where scores of 1–3 on the “outcome agreement” scale are coded as disagreement (0) and scores of 4–6 as agreement (1).
For each model, we present the regression results for our experimental conditions on our fairness and legitimacy measures, as well as the predicted mean outcome for each condition. We then model and present the results from our full OLS regression for both dependent variables, which includes all aforementioned independent and control variables, enabling us to test for interactions between these variables.
Results
We present the regression of experimental condition on fairness and legitimacy in Table 2, and the comparison of predicted mean outcomes by condition from that regression in Figure 1.
Regression of experimental condition on fairness and legitimacy measures, police chase survey
* p < 0.05, ** p < 0.01, *** p < 0.001
Standard errors in parentheses. Fairness treatment and victory for plaintiff Grey is the reference category.
Comparison of predicted mean outcomes by experimental condition, police chase survey
Model 1 in Table 2 shows that police victory, rather than the use of a fairness reasoning style, is associated with increased perceptions of fairness. Figure 1A also illustrates this point. As for the constraining courts scale — both Model 2 in Table 2 and Figure 1B show that police victories decrease participants’ willingness to constrain courts, with a decision employing one-sided reasoning on behalf of a ruling for police creating the least support (contrary to expectations) for constraining courts.
We next present Table 3 and Figure 2, which adds whether the respondent liked the judicial decision into the previous regression and the comparison of means illustration.
Regression of experimental condition and decision agreement on fairness and legitimacy measures, police chase survey
* p < 0.05, ** p < 0.01, *** p < 0.001
Standard errors in parentheses. Fairness treatment and victory for plaintiff Grey is the reference category.
Comparison of predicted mean outcomes by experimental condition and like-dislike decision, police chase survey
Table 3 shows the impact of decision agreement on assessments of fairness and legitimacy. A police victory is no longer statistically significant in either model, suggesting that in Table 2, such a victory is a proxy for decision agreement for the majority of participants who preferred a police victory. Reasoning style, by contrast, is now significant in one comparison: both Table 3 and Figure 2 demonstrate that participants found the decision less fair when Grey’s victory is accompanied by a one-sided, as opposed to fair reasoning style.
Given that treatment effects under random assignment should be resistant to the inclusion of covariates, and that our treatment effect appears sensitive to at least one covariate (decision agreement), we employ random inference testing to assess our results. Here, we use Simon Heß’s ritest package for Stata (Heß Reference Heß2017), which, in essence, estimates the treatment effect under all possible random assignments and then compares the distribution of these simulated effects to the observed ones. We find only slight differences between the p-values of the simulated treatment effects and our observed ones, implying that our model results are based on real effects. Full results for our random inference testing can be seen in Online Appendix C.
We next estimate our full OLS models. We plot regression coefficients and their confidence intervals in Figure 3, using binary variables for the treatment and winner conditions (while also including an interaction term in the model) for ease of interpretation. Full model results for both surveys can be found in Online Appendix D.
Effect of model variables on fairness and legitimacy measures, police chase survey
Figure 3 plots the point estimates (dots) and 95% confidence intervals (lines) of the OLS coefficients on the support for the dependent variable. Figure 3A presents the results for the fairness scale, while Figure 3B shows results for the constraining courts scale. Ideology and partisanship are measured on seven-point scales where higher numbers are more conservative and more Republican, respectively. Education and age are ordinal groupings. The reference category for the race and ethnicity variables is White. The pacific islander category was dropped from the figure because it contained only a single participant. We also omit the interaction term between treatment and winner.
Echoing Table 3, we again find that reasoning style does have a small but statistically significant effect on perceptions of fairness: procedurally fair reasoning is positively associated with perceptions of fairness, relative to one-sided reasoning. Reasoning style has no statistically significant impact on support for constraining courts. In comparison, the outcome’s favorability has a much stronger impact on perceptions of fairness and legitimacy. Liking the decision’s outcome increases perceptions of fairness by 20%, on average, relative to the 5% gain associated with procedural fairness. Liking the decision’s outcome also significantly increases perceptions of legitimacy (that is, reduced support for constraining courts), in contrast to the treatment’s lack of impact on the legitimacy measure.
These findings provide support for both of our hypotheses, albeit much more so for the outcome agreement hypothesis. As for control variables, conservatives are slightly less likely to find the court made a fair decision regardless of experimental condition. Male and Black participants are more likely to support constraining court power, while older and more educated participants are less likely to do so.
An investigation of potential interactions between the experimental conditions and other model variables uncovers two additional findings of note. First, the most liberal participants are significantly more impacted by the fairness treatment (in terms of it increasing their perceptions of fairness) relative to other participants. We interact the treatment with both participants’ self-reporting of ideology and whether they like the decision. Figure 4 shows the predicted probabilities of the fairness assessment, with Panel A simulating when participants did not like the decision, and Panel B when they did.
Predicted probabilities of fairness scale under the fairness condition by ideology, broken down by agreement or disagreement with the decision, police chase survey
Figure 4 plots the predicted probabilities of the fairness measure under the procedural fairness condition, varying by the self-reported ideology of the survey participants from 1 (extremely liberal) to 7 (extremely conservative). Panel A presents the predictions for a simulation where participants did not like the decision; Panel B presents the predictions for when they did. The black line reports the estimates; the grey dashed lines show the 95% confidence intervals.
Figure 4 illustrates that the most liberal participants react differently to the fairness condition relative to conservatives when faced with an unfavorable outcome — that is, a decision with which they are inclined to disagree. As panel A shows, when the outcome is unfavorable, procedural fairness had more impact on perceptions of fairness among participants who identify as liberal, and less impact on perceptions of fairness among participants who identify as conservative. By contrast, in Panel B, which simulates predicted probabilities of perceived fairness and legitimacy among participants faced with a favorable outcome — that is, a decision with which they are inclined to agree, there is no interaction between procedural fairness, ideology, and participants’ perceptions of fairness and legitimacy. This suggests that, when a decision’s outcome is unfavorable to the participant, procedural fairness may improve perceptions of fairness among more liberal participants, but not among more conservative ones. While this interaction does not occur for our legitimacy measure (or in our second study), this suggests that procedural fairness may have differential impacts based on ideology or factors that might underlie ideology, such as personality (Graham et al. Reference Graham, Nosek, Haidt, Iyer, Koleva and Ditto2011). This potential interaction merits further study.
Finally, when the police win, Black participants are significantly more likely to support constraining courts relative to other racial and demographic groups. Given the lower rates of confidence in police as an institution and a long history of police brutality in Black communities, this finding is not surprising.
Study 2: Clinic Protest Vignette
Design and Measures
Our second survey experiment was also hosted by Qualtrics and distributed by Survey Sampling International (SSI) to a diverse national sample in July 2018.Footnote 9 SSI delivered 1785 completed surveys for this study, which took place over two days in July 2018.
This survey design mirrors the first, with a few important changes. Most importantly, the subject area of the vignette involves a city police department’s decision to shut down a pro-life protest of a clinic. As with Study 1, Study 2 presented a news story about the protest, embedded footage from a 2010 abortion protest (Kahan et al. Reference Kahan, Hoffman, Braman and Evans2012, ftnt 87), and then excerpted a judicial opinion ruling on whether the protestors — whom we named “People for Life” in the vignette text — obstructed the entrance to the clinic or intimidated individuals seeking to use it. A ruling that they did obstruct or threaten meant a victory for the city, while a ruling that they did not obstruct or threaten meant the city violated the protestors’ First Amendment rights. As before, we created a 2 x 2 vignette that varies both along who wins — the city or the protestors — as well as whether the opinion employs procedurally fair or one-sided rhetoric in their decision justification.
We made a few other alterations in the design relative to Study 1. First, as we did not use a question that asked the participant’s response to the video prior to reading the opinion in Study 1, we omitted that question here. Second, rather than flip the legitimacy items to serve as support for constraining courts, here we employed a more straightforward use of Gibson, Caldeira, and Spence’s legitimacy questions (2005), creating a scale measure for supporting courts rather than constraining them. Third, rather than measure agreement outcome on a six-point scale that we later convert to a binary measure to ease interpretation, here we made an agreement with the vignette judge’s decision a binary outcome in the survey itself.
Otherwise, Study 2 employed the same measures and design as Study 1. We measured and scaled the assessment of judicial fairness as before. As with Study 1, our dependent variables for fairness and legitimacy/support for courts scaled well, improving on the prior study with alphas of 0.88 for the fairness scale and 0.91 for the supporting courts measure. Our manipulation checks also generated comparable results to Study 1. We again used experimental condition and agreement with the vignette outcome as our independent variables, while maintaining all the control variables from Study 1. Finally, our hypotheses remained the same, theorizing that perceptions of both fairness and legitimacy will be higher in the procedural fairness conditions, as well as when participants like the outcome of the vignette they read.
We present our results for the second study in the same manner as the first, displaying regression results and predicted mean outcomes when regressing experimental condition on our dependent variables in Table 4 and Figure 5, presenting regression results and predicted mean outcomes when adding outcome agreement to that model in Table 5 and Figure 6, and showing the regression coefficients and confidence intervals for our full model in Figure 7. We also run random inference checks on this data, again presented in Online Appendix C. As with the first study, our randomization checks do not reveal any meaningful discrepancies.
Regression of experimental condition on fairness and legitimacy measures, clinic protest survey
* p < 0.05, ** p < 0.01, *** p < 0.001
Standard errors in parentheses. Fairness treatment and victory for the city is the reference category.
Comparison of mean outcomes by experimental condition, clinic protest survey
Regression of experimental condition and decision agreement on fairness and legitimacy measures, police chase survey
* p < 0.05, ** p < 0.01, *** p < 0.001
Standard errors in parentheses. Fairness treatment and victory for the city is the reference category.
Comparison of mean outcomes by experimental condition and like-dislike decision, clinic protest survey
Effect of model variables on fairness and legitimacy measures, clinic protest survey
Results
Table 4 and Figure 5 show that participants’ views of fairness are to some extent influenced by the reasoning style: In the condition where the city won, they are more likely to find the decision fair when the opinion used evenhanded reasoning, relative to when the opinion used one-sided reasoning. However, in the condition where People for Life won, the reasoning style did not make a difference in fairness perceptions. For the legitimacy measure, participants had higher perceptions of legitimacy (that is, supported courts more) when the city won and the reasoning style was procedurally fair, compared to the condition in which the plaintiff won and the reasoning style was one-sided. However, when holding the outcome constant (that is, city wins or plaintiff wins), reasoning style made no significant difference in support for courts.
Table 5 and Figure 6 make clear how much more outcome agreement impacts perceptions of fairness and support for courts compared to the reasoning style, substantially increasing mean outcomes for both scales in every experimental condition. Holding constant agreement with the outcome, we find that procedurally fair reasoning makes a significant difference only where the city wins. The magnitude of the effect, however, is much smaller than that of outcome agreement.
Figure 7 plots the point estimates (dots) and 95% confidence intervals (lines) of the OLS coefficients on the support for the dependent variables. Figure 7A presents the results for the fairness scale model, while Figure 7B shows results for the supporting courts scale. Ideology and partisanship are measured on seven-point scales where higher numbers are more conservative and more Republican, respectively. Education and age are ordinal groupings. The reference category for the race and ethnicity variables is White. The interaction between treatment and winner is omitted.
The regression results in Figure 7 are similar to those in Figure 3. The procedural fairness intervention has a modest positive impact on perceptions that the court’s decision is fair, but no impact on legitimacy. By contrast, and as in Study 1, agreement with the decision has a much stronger impact on perceptions of fairness relative to reasoning style (here, a 33% increase for agreement versus only a 3% increase for reasoning style), while also having a substantial and positive impact on perceived legitimacy.
As for demographic correlations in our second study, education has a positive impact on perceptions of fairness, while support for courts is again positively associated with higher education and age, and lower among Black participants. It’s unclear, exactly, why Black participants assess the court as less legitimate in this vignette. Racial identity may be a proxy for some other variable, such as religiosity, which could impact results given the subject matter. Alternatively, this may reflect a more general distrust of courts among Black participants. Finally, unlike Study 1, no interactions among model variables, including ideology, are statistically significant.
Discussion
This study tested two theories regarding how individuals evaluate the credibility and legitimacy of government decisions. The first theory holds that individuals perceive policy outcomes and the institutions that proffer them as fair and legitimate primarily when the decision’s outcome aligns with their social identities and cultural worldviews (Bartels and Johnston Reference Bartels and Johnston2020; Kahan, Jenkins-Smith, and Braman Reference Kahan, Jenkins-Smith and Braman2011). The second hypothesis contends that procedural fairness also impacts perceptions of fairness and legitimacy, distinct from agreement with the outcome (Tyler and Huo 2002; Tyler Reference Tyler2006). Our study expands on prior procedural justice research by testing procedural fairness outside its usual context of person-to-person interactions, instead examining whether a written judicial opinion conveying elements of procedural fairness may influence perceptions of fairness and legitimacy.
Our findings provide strong evidence for the first theory discussed above, that is, that outcome agreement is more strongly associated with perceptions of fairness and legitimacy. We found this in every analysis in both studies, and outcome agreement had a greater impact on perceptions of fairness (four and ten times the effect size, respectively) relative to the procedural fairness treatment. We find only mixed support for the second theory, that is, that procedural fairness impacts perceptions of fairness and legitimacy, independent of outcome favorability. We found that the procedural fairness condition was associated with slightly higher perceptions of fairness but not with perceptions of legitimacy.
What can we make of these results? A large body of research suggests that procedural fairness can improve how legal decisions are received. Some studies even suggest that procedural fairness is relatively more important to legitimacy than the favorability of the outcome (Tyler and Huo 2002; Tyler Reference Tyler2007; Thibaut Reference Thibaut1975). Our results were not consistent with these studies’ findings. On the contrary, participants’ perceptions of judicial fairness and legitimacy depended much more on whether they agreed with the outcome. This raises interesting questions about when, and under what circumstances, procedural justice can influence people’s perceptions of legitimacy, particularly in decisions involving issues upon which people are culturally (and politically) polarized, meaning that one’s stance is a marker of one’s social and cultural identity. Our study provides robust evidence that perceptions of fairness and legitimacy are more strongly influenced by the favorability of the decision, at least when that decision implicates one’s social and cultural identity, and is therefore likely to trigger motivated (that is, identity-protective) cognition.
There are several ways our findings might be reconciled with existing procedural justice research. One is that procedural justice is less effective or relevant to people’s perceptions of legitimacy when the dispute involves a culturally charged issue that touches on one’s group identity, as opposed to an individualized matter such as a benefits determination, which holds no broader cultural significance. It may be that in circumstances where the decision implicates group identity, a threat to one’s group identity is relatively more important than the nature/process/reasoning style of the decision. Where participants’ cultural priors lead them to believe a decision is self-evidently right or wrong, procedural fairness may come across as pandering or bad-faith justification and fail to increase legitimacy.
Another possibility is that including elements of procedural fairness in a written judicial opinion, where the study participant is not a party to the case, was less salient or effective than it would be in a personal interaction with a judge or a police officer. Many studies of procedural fairness are observational studies surveying actual litigants about their lived experience with courts (for example, Bradford Reference Bradford2011; Casper, Tyler, and Fisher Reference Casper, Tyler and Fisher1988; Kitzmann and Emery Reference Kitzmann and Emery1993), whereas our study attempted to manipulate or manufacture fairness in a written opinion. Because procedural fairness is most often employed through interpersonal relations, experiencing it through a written opinion, rather than firsthand, may be less powerful. In our view, this suggests an important limitation on the utility of procedural fairness, as judges (particularly higher court judges) regularly decide questions that have broad public applications in cases to which most members are not personally involved, and these decisions are conveyed to the public via written opinion only.
A third possibility is that our rhetorical manipulation of procedural fairness and one-sidedness was not sufficiently effective. We note above that about a third of our participants failed manipulation checks attempting to ascertain whether they recalled their treatment category included fairness rhetoric. Rerunning our models with only the participants that passed these specific manipulation checks (as opposed to the time of competition checks) in Study 1 and Study 2 meaningfully increased the effect size of the procedurally fair opinions on participants’ assessments of fairness. Analyses using these truncated samples did not provide evidence that the treatment impacted legitimacy measures as theorized, nor did they change the strong impact of outcome agreement on our dependent variables. Nevertheless, that participants who correctly remembered whether their vignette contained fairness language more strongly responded to that treatment in their assessment of judicial fairness suggests that a different operationalization of procedural fairness within judicial opinions might be more successful. Alternatively, this may suggest an inherent limitation in the ability to convey procedural fairness via written opinion. It may be that interested members of the public do not read opinions with the level of awareness or sensitivity to take note of these nuances in reasoning style.
To the extent that our findings do indicate that outcomes matter more than procedural fairness, they suggest that written opinions may be mostly evaluated based on the desirability or congeniality of the outcomes they reach. This is not necessarily a bad thing. One critique of procedural justice, mentioned previously, is that it might enable legal authorities such as police, judges, etc., to use rhetoric of fairness to placate people into perceiving decisions as legitimate and accepting them, even when they are substantively unjust or reinforce deeper social hierarchies (Bell Reference Bell2017; Thacher Reference Thacher, Anthony and Weisburd2019). Our findings suggest that this is unlikely to work. If people substantively disagree with the outcome, using fair or even-handed language is unlikely to convince them that a decision is fair or that the authoring court is legitimate.
Finally, our research presents some interesting findings about how different groups responded to the experimental conditions. Black respondents were more likely to support constraining courts’ policymaking powers in experimental conditions where the police won in the first survey, and less likely to support the courts in the second survey, regardless of experimental condition. Our results from Study 1 also suggest there may be an interaction between approval of the decision, reasoning style, and ideology, in that our most liberal participants were more likely to rate a disagreeable opinion as being fairer if it had elements of procedural justice, as compared to moderates and conservatives. That procedural fairness interventions might appeal differently to individuals of different ideologies or worldviews seems reasonable given existing work connecting ideology to values (Piurko, Schwartz, and Davidov Reference Piurko, Schwartz and Davidov2011) or personality traits (Gerber et al. Reference Gerber, Huber, Doherty and Dowling2011). If these results are replicated elsewhere, they could have significant consequences for the utility of procedural justice across different groups, and thus merit further exploration.
Like any experimental vignette design, our survey experiments contained limitations. First and most importantly, the intervention effects we demonstrate here result from participants reading an excerpt of a judicial opinion, something they might not be likely to do in real life. While including a video and placing the vignette within a journalistic framework should increase verisimilitude, questions about the likelihood of treatment outside an experimental framework remain. That said, we have chosen the sort of issues most likely to break through a media filter, and research has shown that individuals learn not just from media accounts, but from more knowledgeable individuals within their social networks who are reading such media (Carlson Reference Carlson2019; Strother and Glennon Reference Strother and Glennon2021), or even the opinion itself.
Second, our study faces the familiar problem of providing a single intervention followed by immediate measurement, making the durability of that intervention unclear. As a third point, we were also unable to generate a viable control or “neutral” opinion excerpt, as doing so tended to create an opinion that was either shorter in length or contained less reasoning or both, features that research shows have their own negative impact on credibility or perceptions of fairness (Simon and Scurich Reference Simon and Scurich2011). It may be that the modest impact from our experimental treatment is driven more by negative responses to one-sided reasoning than positive responses to the fairness condition — though this alternative interpretation still provides useful information regarding the impact of reasoning styles on reactions to a decision outcome. Either way, our experiment provided a relatively easy test for procedural fairness, contrasting it with an explicitly one-sided justification rather than something more nuanced. This suggests that procedural fairness impacts on perceptions of fairness might be smaller when compared to a less one-sided alternative.
Fourth and finally, while we are confident our vignettes are well-chosen in terms of the documented cultural polarization over these issues — and hence, participants’ likelihood of being ideologically polarized along predictable lines — it’s possible that our findings would not generalize to other legal issues.
In sum, our findings suggest that perceptions of fairness and legitimacy of judicial decisions depend significantly on whether the outcome is agreeable, and less so on incorporating procedural fairness into the Court’s reasoning. Thus, our study lends support to a body of research suggesting that people’s perceptions of a decision’s credibility and legitimacy (at least on culturally polarizing topics) depend primarily on whether the outcome aligns with and affirms their cultural worldview. At least in the context of a written judicial opinion on a culturally polarizing topic, we found less support for the procedural fairness theory — that is, that decisionmakers may bolster perceptions of legitimacy by conveying that their decision-making process was even-handed, that they considered both parties’ views, that they worked hard to find the best answer for society, rather than imposing their own values. Whether and to what extent these findings may differ depending on participants’ ideology, personality, and other demographic variables is worthy of future study.
Supplementary material
To view supplementary material for this article, please visit https://doi.org/10.1017/lsi.2025.10114
Author Statements
Rob Robinson is Associate Professor of Political Science at California State University, Fullerton, and can be reached by email at rorobinson@fullerton.edu. Robinson would like to thank Dan Kahan for his advice at the early stages of this project, as well as the reviewers at Law & Social Inquiry.
Danieli Evans is an Assistant Professor of Law at the University of Washington School of Law, and can be reached by email at devans6@uw.edu. Evans is grateful to Dan Kahan and Tom Tyler for feedback on the theory and design of this study, and to the reviewers and editors at Law & Social Inquiry for their helpful suggestions.
Funding Statement
Funding for this project was provided by the American Psychology-Law Society. IRB Compliance was approved by California State University, Fullerton, HSR-16-0042. Neither author has competing interests to declare.
Open access
