Understanding judicial motives: The search for judicial ideology

Since Pritchett (Reference Pritchett1942) and Murphy (Reference Murphy1964), judicial scholarship has placed preeminent importance on reliability and accuracy in measuring the ideological positions of judges.Footnote ³ It is generally accepted within political science that judges’ policy preferences play an important role in judicial behavior and decision-making, both in the United States and comparatively abroad. However, measuring judicial preferences and the extent to which they are influenced by both internal and external factors has been the dominant point of contention for expanding our understanding of judicial behavior, as well as critiquing the limits of judicial ideology to explain decision-making.Footnote ⁴

Researchers have been implementing various measures of judicial ideology for several decades. Bonica and Sen (Reference Bonica and Sen2021) provide a comprehensive overview of this literature, which – as they describe – has exhibited considerable change in the breadth of methodological rigor. Understandably, the bulk of this research concerns the Supreme Court,Footnote ⁵ though scholars are often keen to warn against over-extrapolation of inferences from the Justices’ behaviors because it represents a small sample of decision-making and “places considerable importance on nine idiosyncratic individuals who are relatively unconstrained in their position atop the American judicial hierarchy (Bailey, Reference Bailey2007)” (Bonica and Sen, Reference Bonica and Sen2021, 98). Nonetheless, ideology – particularly when measured through judge-level features of partisanship – has proven to be reliable predictors of the Justices’ decision-making. We provide an overview for some of these various methodologies in Table 1, which includes both the source of observable data, as well as the approaches’ strengths and weaknesses posited by Bonica and Sen (Reference Bonica and Sen2021).Footnote ⁶

Table 1.

Overview of Existing Measures of Judicial Ideology (Bonica and Sen, Reference Bonica and Sen2021)

¹ See Segal and Cover (Reference Segal and Cover1989).

² See Bonica and Sen (Reference Bonica and Sen2017).

³ See Epstein et al. (Reference Epstein, Martin, Quinn and Segal2007); Giles et al. (Reference Giles, Hettinger and Peppers2001); Segal and Spaeth (Reference Segal and Spaeth2002), among others.

⁴ See Bonica and Sen (Reference Bonica and Sen2017).

While political science has embraced the role of ideology in judicial decision-making, some legal scholarship and other observers remain unconvinced. As noted by Fischman and Law (Reference Fischman and Law2009), “there is little reason to expect those who practice or teach the craft of legal argument to embrace a body of research that questions the extent to which judicial decision-making is actually driven by legal argument” (134).Footnote ⁷ For as long as judicial scholars have amassed evidence that a principal component of judicial behavior is the sincere policy preferences of the judge (or judges), legal scholars have labeled their findings overtly political (Edwards, Reference Edwards1984), “born in a congeries of false beliefs,” (Tamanaha, Reference Tamanaha2009, 687-688), or unfortunately, “innocently ignorant” (Cross, Reference Cross1997, 251). The primary concern among these and other legal scholars is two-fold: First, either by methodological ignorance or biases against theoretical approaches that would minimize the role of “the law” in judging, legal research often contends that judicial scholars have claimed ideological biases in judges without appropriately measuring or explaining what is meant by “ideology” (Fischman and Law, Reference Fischman and Law2009). To be sure, methodological rigor by judicial scholars attempting to measure judicial ideology often minimizes the value of clear conceptualization of the subject, choosing instead to state simply that Justice X votes the way they do because they are very conservative (liberal). Ideology is a “highly flexible conceptual tool” (Gerring, Reference Gerring1997, 957) but often boils down to how actors organize and express their opinions and how those opinions are formed by values. For judicial ideology, this baseline is often used to operationalize ideology by focusing on outward actions such as votes, making it appear that courts are “just another political institution,” which has dangerous implications for judicial legitimacy (Gibson and Nelson, Reference Gibson and Nelson2017).

Second, legal scholarship contends that the measurement of ideology itself misses the point of the law by vastly ignoring the role of precedent and legal opinion in decision-making. Part of this, we expect, is born from longstanding notions of how common law principles emerge in judicial behavior. Students of the law are often schooled on the ethics of judicial decision-making, particularly as it relates to how stare decisis applies to synthesizing and formulating jurisprudence. Even in most senior legal institutions like supreme courts, a considerable majority of caseloads will consider routine (or even mundane) applications of the law. In these circumstances, especially when the role of a judge is to simply apply codified standards, elements of ideology and interpretation are effectively restrained. An opinion piece published by The New York Times entitled “The Supreme Court Is Not as Politicized as You May Think” (Donnelly and Loeb, Reference Donnelly and Loeb2023) draws further on this point by arguing that painting the Supreme Court as invariably political requires observers to ignore the vast majority of statutory litigation reviewed by the Justices, almost all of which end in total (or near) unanimity. Yet, while it may be true that much of the Court’s docket lacks definitive political salience (as do most dockets across the federal and state judiciaries),Footnote ⁸ this contention itself is seemingly the result of self-selecting observations to fit a broader thesis.Footnote ⁹ Unanimous opinions can certainly constrain ideological preferences, though often when ideology can be exchanged for some level of legal certainty in the case (Corley et al., Reference Corley, Steigerwalt and Ward2013). In addition, the Supreme Court’s discretionary powers grant the Justices the ability to think far more about policy goals than law (Baum, Reference Baum1997; Segal and Spaeth, Reference Segal and Spaeth2002), as it is “easy for them to find legal justification for whatever position they prefer” (Baum, Reference Baum1997, 64). And while the vast majority of scholars would agree that personal preference and policy are not the only thing that matters (Hansford and Spriggs, Reference Hansford and Spriggs2006; Richards and Kritzer, Reference Richards and Kritzer2002; Songer and Lindquist, Reference Songer and Lindquist1996), ideology often dictates which precedents are recognized and promulgated over time and across courts (Fix and Kassow, Reference Fix and Kassow2020; Hinkle, Reference Hinkle2015).

Understanding ideology with words: Text-based methods for ideological measurement

As inferred by Table 1, the breadth of measures for judicial ideology are considerably varied, both as it relates to sources of observable data and the inferences that can reliably be drawn from them. However, what remains an emerging and largely unexplored element of measurement is data originating from written (or spoken) text. As Bonica and Sen (Reference Bonica and Sen2021) observe, research leveraging tools for automated text analysis have emerged as a way of studying ideology across other actors and institutions, and especially as it relates to discerning individual and policy-level positions in elected legislatures like the United States Congress. At their core, these studies assume that language retains special significance as cues of underlying preferences. Given that ideology is understood as a system of belief organization and an attempt at understanding the world around us, language is central to its understanding since language provides us with meaning (Diermeier et al., Reference Diermeier, Godbout, Yu and Kaufmann2012; Thompson, Reference Thompson1987). “Intuitively,” according to Diermeier et al. (Reference Diermeier, Godbout, Yu and Kaufmann2012), “a political ideology specifies which issue positions go together, the ‘knowledge of what-goes-with-what’” (31).Footnote ¹⁰ While it is correct to say that language can evolve, and with it subtly change the ideological nature of words and their meaning, we presume that this is part of a holistic process of idea generation and refinement over time. This knowledge helps us better discern variance in the framing of particular issues and their overarching sentiment, and broadens considerations for understanding how to take language seriously as it changes,Footnote ¹¹ as well as offering a key to understanding elite discourse and how it is promulgated in the mass public. Indeed, given the progressive development of automated tools and machine learning, a wealth of literature has developed or implemented comprehensive analyses of ideologically oriented Congressional speech, such as Wordscores (Laver et al., Reference Laver, Benoit and Garry2003), Wordfish (Slapin and Proksch, Reference Slapin and Proksch2008), and Wordshoal (Lauderdale and Herzog, Reference Lauderdale and Herzog2016).

Yet, applications of these approaches to the courts, and particularly the Supreme Court, remain few. Until recently, Lauderdale and Clark (Reference Lauderdale and Clark2014) remained the most pivotal development in this field, as they were able to use an autoregressive preference model to scale case-level ideology for Justices between 1946 and 2005. Others, including Hausladen et al. (Reference Hausladen, Schubert and Ash2020), similarly approached case-level positions using data from the Federal Courts of Appeals.Footnote ¹² Even then, both of these approaches scale ideology as a reflection of the cases themselves, rather than judicial actors. More recently, Cope (Reference Cope2024) developed Jurist-Derived Judicial Ideology Scores (JuDJIS), which leverage “information…collected by professional survey firms commissioned by the Almanac of the Federal Judiciary, a [triennially] published initiative which surveys a stratified sample of qualified experts for each judge” (2).Footnote ¹³ The result is a dynamic measure of judicial ideology recovered from a hierarchical n-gram analysis conditioned on how lawyers familiar with each judge would review their “ability; demeanor; trial practice/oral argument; settlement/opinion quality; and ideology.”

Using text to determine judicial ideology

To better understand how opinion language acts as a signal of judicial ideology, we must first imagine that the outcome in any case exists within a relatively confined “case space” (Lax, Reference Lax2011). As part of determining case outcomes, judges write opinions in attempts to justify and persuade others to the “correctness” of their decision (Romano and Curry, Reference Romano and Curry2019). Within each case, there exist a specific number of topics that judges can choose from to frame their argument. Judges choose their words carefully when crafting opinions (Romano and Curry, Reference Romano and Curry2019), but different judges will choose language to explain their argument. Choices in topic selection, framing, and language are conditioned upon an individual judge’s views and beliefs concerning what the right course of action should be in determining the outcome of any case – that is, on their ideology.

As we noted in an earlier section, attempts to map legal concepts and broader elements of lexical variation onto ideology remain few in studies of judicial politics. To date, the primary contributions to this literature remain Lauderdale and Clark (Reference Lauderdale and Clark2014), Hausladen et al. (Reference Hausladen, Schubert and Ash2020), and Cope (Reference Cope2024). Even then, none are directly applicable to our efforts. Our goal is to disassociate dichotomous voting behaviors or alternative proxies to instead place the burden of mapping ideology using the text of written opinions on the law. Decisions by the Supreme Court represent a unique case study to accomplish these ends.

However, important questions remain concerning if cues of ideology emerge in legal text, and more importantly how we might go about retrieving it. The answer to the first question must be yes. Irrespective of the underlying facts and merits of each case, opinions authored by the Justices represent policy positions. Particularly when the Court is divided, these positions can manifest across multiple opinions, and the Justices are able to use these opportunities to articulate their perspectives (Brace and Hall, Reference Brace and Hall1993; Hall, Reference Hall1987; Hettinger et al., Reference Hettinger, Lindquist and Martinek2004; Romano and Curry, Reference Romano and Curry2019; Songer, Reference Songer1982) Alternatively, when the Court is unified, we can assume they are speaking as one voice and thus conveying some measure of certainty that minimizes ideological cues (Corley et al., Reference Corley, Steigerwalt and Ward2013). In essence, given that the Justices are unconstrained in providing concurrences and dissents, we can derive comprehensive elements of their perspectives in a dimension that is more robust than whether they voted to affirm or reverse.Footnote ¹⁴

The second question is clearly more complicated, though research in ideology as a function of articulated positions in legislative settings provides an intuitive path forward. In particular, we leverage work by Lauderdale and Herzog (Reference Lauderdale and Herzog2016) by applying Wordshoal to opinions by the Supreme Court between the 2005 and 2022 terms. At its core, this methodology employs a two-stage estimation strategy to retrieve ideal points from expressed positions offered during debates in the House of Representatives. The Justices’ opinions lend themselves well to this approach, given that the motivations for their decision-making are deterministic of their preferences.Footnote ¹⁵

The focus of our work toward yielding inferences from broader lexical variance further lends itself well to this methodology. The Wordshoal approach places greater emphasis on how text found within documents can be used to articulate position-taking by its author (or authors). In a legislative setting, this processes addresses relative lexical variance from individual subsets (i.e., issue-specific debates) to draw generalizations of the individuals’ (legislators) broader positions. Further, given how attribution of equal predictive weight to specific words (or phrases) across debates of varying substance is sure to yield dubious results, the implications of particular word choice are sensibly restricted to individual debates. In essence, while certain words or phrases may inform us where to draw distinctions between competing coalitions on specific issues, assuming their influence is constant irrespective of subject matter would bias our results. Instead, Wordshoal offers a practical alternative that (1) addresses distinct lexical variance as emblematic for expressing “stated positions” (Lauderdale and Herzog, Reference Lauderdale and Herzog2016, 375) within reduced (debate) settings and subsequently (2) considers how that variance contributes to drawing broader discrimination in a legislator’s latent preferences. We discuss our amended implementation of Wordshoal below.

Implementing Wordshoal

Wordshoal is implemented using a two-stage hierarchical process, which uses a Poisson scaling model to retrieve debate-level estimates from Wordfish (Slapin and Proksch, Reference Slapin and Proksch2008) and subsequently aggregate to a general latent position for each individual actor. We review this process below and maintain notation from Lauderdale and Herzog (Reference Lauderdale and Herzog2016), though we indicate where substitutions are made for the Supreme Court versus their original legislative observations using brackets. In this substitution, our notation follows that the Justices [legislators] are indexed as i $ \in $ 1, 2… , N, cases [debates] as j $ \in $ 1, 2… , M, and words as k $ \in $ 1, 2… , K.

$$ {\omega}_{ijk}\sim \rho \left({\mu}_{ijk}\right) $$

$$ \rho \left({\mu}_{ij k}\right)=\exp \left({\nu}_{ij k}+{\lambda}_{jk}+{\kappa}_{jk}{\psi}_{ij}\right) $$

Being that “the frequency that [Justice] i will use word k in [case] j depends on a general rate parameter $ {\nu}_{ijk} $ for [Justice] i’s word usage in [case] j, word-[case] usage parameters $ {\lambda}_{jk} $ , $ {\kappa}_{jk} $ and the individual’s [case]-specific position $ {\psi}_{ij} $ . The $ {\nu}_{ijk} $ parameters capture the baseline rate of word usage in a given [opinion], which is simply a function of the length of the [opinion]. The $ {\lambda}_{jk} $ capture variation in the rate at which certain words are used. The $ {\kappa}_{jk} $ capture how word usage is correlated with the [Justice]’s [case]-specific position $ {\psi}_{ij} $ ” (Adapted from Lauderdale and Herzog, Reference Lauderdale and Herzog2016, 377).

With these estimates from case-level dimensions, we can subsequently use Wordshoal for aggregating to a (in this instance) single latent dimension with normally distributed error.

$$ {\displaystyle \begin{array}{c}{\psi}_{ij}\sim N\left({\alpha}_j+{\beta}_j{\theta}_i,{\tau}_i\right)\\ {}{\theta}_i\sim N\left(0,1\right)\\ {}{\alpha}_j,{\beta}_j\sim N\left(0,{\left(\frac{1}{2}\right)}^2\right)\\ {}{\tau}_i\sim \mathcal{G}\left(1,1\right)\end{array}} $$

Again, drawing from Lauderdale and Herzog (Reference Lauderdale and Herzog2016), “this specification means that the primary dimension of word usage variation in individual [cases] can be more or less strongly associated with the aggregate latent dimension $ \theta $ being estimated across all [cases], with either positive or negative polarity for any particular [case]. Essentially, this allows the model to select out those debate-specific dimensions that reflect a common dimension (larger estimates values of $ {\beta}_j $ ), while down-weighting the contribution of debates where the word usage variation across individuals seems to be idiosyncratic ( $ {\beta}_j\approx $ 0). The priors on $ {\theta}_i $ and $ {\theta}_j $ allow the model to remain agnostic about the relative polarity of individual [case] dimensions, while constraining the common latent dimension of interest to a standard normal scale” (378).

However, our methodology departs slightly from Lauderdale and Herzog (Reference Lauderdale and Herzog2016) in how we consider the relative responsibility associated with opinion authorship. Unlike Congressional debates, where we can directly associate speeches with particular legislators, opinion authorship on the Supreme Court dictates that a majority (or at least a plurality) of the Justices will coalesce to a single opinion. Similar concerns emerge when we recognize that Justices frequently join concurrences or dissents. It might be problematic to assume that Justices choosing to join opinions – whether they be majorities, concurrences, or dissents – should be granted the same weight in estimating latent ideal points as those who actually authored them. In one sense, we can infer that given the Justices’ capacity to author their own concurrences and dissents, failure to deviate from the language of the opinions they are joining is indicative of it being their own words, as well. However, it clearly lacks the same degree of authenticity that structured assumptions in Lauderdale and Herzog (Reference Lauderdale and Herzog2016) of directly tying each speech to individual legislators. To account for this, we introduce a procedure that assigns relative weights reflective of opinion authorship. We can assume that in any circumstance, authorship of a decision can be weighed fully as a representation of the writer’s voice, while choosing not to join an opinion bears no weight. Alternatively, joining majority opinions, dissents, or (special) concurrences might require additional considerations.Footnote ¹⁶ Lacking sufficient guidance from existing literature, we implement a combination of weighting arrangements to establish whether degrees of associative responsibility induce significant variance in the estimates.Footnote ¹⁷ We subsequently amend the second set of equations to include parameter $ {\phi}_{ij} $ ,Footnote ¹⁸ representing the relative responsibility of an opinion for Justice i in case j. Footnote ¹⁹

Scaling Supreme Court Justices between the 2005 and 2022 terms

To this point, we have introduced an amended application of Wordshoal as a methodology to estimate latent ideology as an expression of lexical variance without actually considering whether a Justice voted to affirm or reverse. That is, it draws on the expectation that the variance in word choice signals their relative positions and reflects separability among individuals. Granted, the notion of coalition structures raises potential concerns. Apart from our inability to draw the same assumptions of invariably associating opinions to single authors, it is important to further recognize that the Justices have a tendency to coalesce among like-minded partisans – particularly in response to divisive cases. With respect to our attempts to remove voting and ideology from our procedure, we are cognizant that these behaviors encapsulate aspects of both. However, given our application of different weighting schemes and the capacity for the Justices to voice divergent perspectives to any opinion, we are confident that we took the most appropriate steps to map the Justices’ ideal points reflective of the language used in their opinions.

We provide estimates of individual Justices reflective of 1,972 opinions authored in 678 non-unanimous cases between the 2005 and 2022 terms below (Figure 1),Footnote ²⁰ which coincides with the Roberts Court (2005–present). Apart from the scheme that only applies to those who authored a particular opinion (‘Sole Weight’),Footnote ²¹ we observe noticeable consistency in our estimates. Furthermore, the breadth of error – which represents 95 percent confidence intervals – is shown to only be spacious for those whose presence in the data is more limited.Footnote ²² Alternatively, those present for longer periods of observation tended to display more consistency in their estimated positions.

Figure 1.

Wordshoal Estimates (High Weight) by Justice and Weighting Scheme.

Note: Left-Right axis ( $ {\theta}_i $ ) scaled to represent progressive values of Liberalism to Conservatism. For more information regarding the associated weighting schemes, see the Supplemental Appendix (Table A1).

At face value, our estimates retain discernible validity. Insofar as we can provide normative assessments of the Justices’ categorical positions on the ideological spectrum, we know that Justices Sotomayor and Ginsburg should populate divergent positions from Thomas and Scalia. Further, the nuance of those we would expect to populate the center – for example, Justices Kennedy and O’Connor – is likewise reflected in our estimates. Alternatively, the ordinal rankings of our estimated positions require careful consideration. Notwithstanding measures of potential error, our rankings place Justices Ginsburg and Alito in the most polarized positions. The exact locations of the Justices relative to each other on our unbounded scale might lead to some debate concerning whether the ordinal and cardinal distances are, in a sense, the most accurate representation.Footnote ²³ What we can say is that the relative location of the Justices on this scale lends credibility to its accuracy, particularly given the clustering of objectively like-minded Justices.

Further, another test of our estimates’ validity is to compare them to established methodologies. For this, we draw on ideal points estimated by Martin and Quinn (Reference Martin and Quinn2002), from which most measures of judicial ideology that consider the Justices’ voting behaviors draw their inspiration. Given the dynamic nature of their scaling procedure, we plot the average position for each Justice against those recovered using Wordshoal (Figure 2). However, while the directionality of both measures is symmetric, their relative magnitudes differ. To account for this, we standardize both using z-score normalization, ensuring they share a common scale with a mean of zero and a standard deviation of one. We then plot the relative variance of Justice-level estimates (Figure 3).

Figure 2.

Wordshoal Estimates (High Weight) versus Martin and Quinn (Reference Martin and Quinn2002).

Note: This figure compares Wordshoal estimates using High Weights with the dynamic ideal point estimates by Martin and Quinn (Reference Martin and Quinn2002). Both represent Justice-level means across the 2005–2021 observation terms, where averages for Martin-Quinn were recovered from their post_mn variable. While the scales are not equal or normalized, the (left-right) progression of Liberalism to Conservatism is observed in both. Correlation = 96.6%.

Figure 3.

Comparison of Static Wordshoal (High Weight) versus Martin-Quinn.

Note: Both axes represent the absolute values of each Justice’s estimated ideal point using Martin-Quinn and Static Wordshoal, where both scales are standardized using z-score normalization. Values for Martin-Quinn (Static Wordshoal) are measured using the average of post_mn ( $ {\theta}_i $ ) across the observation period. Points nearest to the diagonal segment indicate greater correlation between the relative placement of a Justice. Alternatively, values above (below) the diagonal indicate greater relative ideological placement in Martin-Quinn (Static Wordshoal). Correlation = 86.84%.

Notwithstanding the difference in observable behaviors used to estimate latent ideal points, we observe a strong correlation between our measure and Martin-Quinn. A particular difference is the relative position of Clarence Thomas, who is estimated to be discernibly more conservative in Martin-Quinn, as well as subtle variance in the Justices’ ordinal rankings. However, we are not of the mind that our measure is failing to discern what is arguably the “correct” ordering or placement of each Justice. Instead, it is important to recognize the incomplete nature of the data. Being that, while Martin-Quinn maps the full extent of each Justice’s tenure on the Court, ours only represents the full service of seven.Footnote ²⁴ Accounting for those whose tenures (as of the 2022 term) are fully represented in the data, we observe strong similarity with Martin-Quinn’s ordinal rankings. We imagine that incorporating the absent data for the remaining eight Justices whose tenures are not fully represented could remediate any potential shortcomings in (ordinal) placement. Even then, the fact that we observe such high degrees of correlation with an established and respected measure of judicial ideology underscores the robustness of our methodology. The Justices are keen to express ideology as a principal feature of their authored opinions, and our methodology is able to map these distinct behaviors.