POLITICAL SOPHISTICATION AND FACTUAL KNOWLEDGE

Public opinion researchers routinely incorporate political sophistication in their empirical analysis—either directly as an outcome variable of interest, as a major explanatory factor, or as an important confounder to control for. In order to measure the underlying latent trait, scholars commonly rely on short batteries of standard recall questions on basic facts about the political system.Footnote ² For instance, Delli Carpini and Keeter (Reference Delli Carpini and Keeter1993)—a canonical article proposing such a battery—has been cited more than one thousand times since its publication. In short, political knowledge remains a concept of intense scholarly interest and it is frequently measured using standard off-the-shelf recall questions.

The ubiquity of basic recall questions in public opinion research is accompanied by the frequent findings that many people know too little about politics (Barabas et al. Reference Barabas, Jerit, Pollock and Rainey2014; Delli Carpini and Keeter Reference Delli Carpini and Keeter1996) and that the discrepancies in information levels can result in unequal representation in the political system (Althaus Reference Althaus1998; Gilens Reference Gilens2001; Kuklinski et al. Reference Kuklinski, Quirk, Jerit, Schwieder and Rich2000). The underlying reason why scholars focus on people’s ability to recall factual information about politics is that these items “more directly than any of the alternative measures, capture what has actually gotten into people’s minds” (Zaller Reference Zaller1992, 21; see also Gomez and Wilson Reference Gomez and Wilson2001; Zaller Reference Zaller1991). However, there is some reason to doubt this assertion, both from theoretical and methodological perspectives.

First, the discipline’s exclusive focus on factual political knowledge has been criticized on theoretical grounds. Most importantly, recalling facts about political institutions has little relevance for citizen competence (Cramer and Toff Reference Cramer and Toff2017; Lupia Reference Lupia2006). Given that there is usually no consensus about what information is necessary in the first place, Druckman (Reference Druckman2014) proposes abandoning recall questions as measures of “quality opinion.” Instead, the author advocates “less focus on the content/substance of opinions […] and more on the process and specifically the motivation that underlies the formation of those opinions” (Reference Druckman2014, 478, emphasis in the original). The key distinction should, therefore, be how citizens approach a political issue and whether they are motivated to engage in elaborate reasoning to arrive at their particular decision.

It turns out that such competent decision-making does not necessarily require citizens to hold large swaths of political information in their declarative memory (i.e., what is being measured by conventional knowledge scales). In fact, people can often use heuristics to navigate the realm of politics without having to remember encyclopedic facts about institutions or actors (Lupia Reference Lupia1994). Even simple visual cues have been shown to increase political knowledge levels (Prior Reference Prior2014). While other research suggests that heuristics do require a baseline level of expertise (Lau, Andersen, and Redlawsk Reference Lau, Andersen and Redlawsk2008; Lau and Redlawsk Reference Lau and Redlawsk2001) and that their effectiveness can depend on the political context (Dancey and Sheagley Reference Dancey and Sheagley2013), this body of literature shows that competence cannot be reduced to the capacity of citizens to remember facts alone. It is more important that people possess the skills and resources to find the information required in a specific context (e.g., Bernhard and Freeder Reference Bernhard and Freeder2020). In other words, procedural memory appears to be more integral to political competence than declarative memory (Prior and Lupia Reference Prior and Lupia2008). In a similar vein, Luskin (Reference Luskin1990) suggests that individual motivation and abilities help explain political sophistication more so than the availability of factual information.

Beyond these theoretical critiques, there are several methodological issues that cast doubt on the validity of factual knowledge scores as a measure of political sophistication. One problem frequently discussed in the literature revolves around the question whether or not to offer “don’t know” options in multiple choice recall questions (Miller and Orr Reference Miller and Orr2008; Mondak Reference Mondak2000; Mondak and Davis Reference Mondak and Davis2001). Including such an option can lead to biased estimates of information levels because they are confounded by people’s differential propensity to guess instead of admitting not to know the correct answer (but see Luskin and Bullock Reference Luskin and Bullock2011). Other scholars criticized open-ended factual knowledge questions due to problematic coding rules, which do not capture partial knowledge (DeBell Reference DeBell2013; Gibson and Caldeira Reference Gibson and Caldeira2009; Krosnick et al. Reference Krosnick, Lupia, DeBell and Donakowski2008; Mondak Reference Mondak2001). However, closed-ended recall questions are not without issues either. Conventional item batteries differ with regard to the temporal and topical dimensions of the underlying information—which can have important implications for researcher’s conclusions about the antecedents and consequences of political knowledge (Barabas et al. Reference Barabas, Jerit, Pollock and Rainey2014). In addition to question content, recent research reveals how their format (e.g., true–false vs. multiple choice) can further exacerbate assessed knowledge inequalities in society (Fraile and Fortin-Rittberger Reference Fraile and Fortin-Rittberger2020).

The increasing reliance on online surveys creates additional concerns for recall questions due to people’s tendency to look up answers (Clifford and Jerit Reference Clifford and Jerit2016; Höhne et al. Reference Höhne, Cornesse, Schlosser, Couper and Blom2020). This is particularly problematic if there are systematic differences between respondents’ likelihood to cheat when answering knowledge questions (Style and Jerit Reference Style and Jerit2020). Even if cheating was not an issue in online surveys—because respondents are effectively discouraged from searching for correct answers—factual knowledge scores can still suffer from differential item functioning, since individual recall questions have varying measurement properties across the population (Pietryka and MacIntosh Reference Pietryka and MacIntosh2013). Item batteries that are easier to answer for certain groups can, therefore, exacerbate observed differences in political knowledge—for example, between racial groups (Abrajano Reference Abrajano2014).

PLEASE MIND THE GENDER GAP

Survey researchers not only find that people are not sufficiently informed as a whole, but also attest that women are systematically less knowledgeable than men. For instance, women routinely score lower on political information, interest, and efficacy, which decreases their respective levels of political participation. Since gender differences in political information and interest can only partly be explained by resource-related factors such as individual levels of education, Verba, Burns, and Schlozman (Reference Verba, Burns and Schlozman1997, 1070) diagnose a “genuine difference in the taste for politics” between women and men, which they suspect is driven largely by socialization (see also Wolak and McDevitt Reference Wolak and McDevitt2011). Indeed, Dow (Reference Dow2009, 117) describes the systematic gender differences in knowledge as “one of the most robust findings in the study of political behavior.” While differences between women and men in political interest can certainly be attributed to gendered political socialization (Bos et al. Reference Bos, Greenlee, Holman, Oxley and Lay2022; Wolak Reference Wolak2020), at least part of the disparities in knowledge may simply be an artifact of the measurement approach.

The discussion revolving around the apparent gender gap is, therefore, closely intertwined with the methodological debate about measuring political knowledge. For instance, Mondak and Anderson (Reference Mondak and Anderson2004) suggest that women are more likely to report that they do not know the answer to a recall question, whereas men are more inclined to guess. Correcting for these systematic differences in the propensity to guess mitigates the gender gap in knowledge but does not eliminate it completely (see also Ferrín, Fraile, and García-Albacete Reference Ferrín, Fraile and García-Albacete2017; Lizotte and Sidman Reference Lizotte and Sidman2009). Furthermore, recent research further suggests that open-ended question formats that discourage guessing may diminish observed gender differences altogether (Ferrín, Fraile, and García-Albacete Reference Ferrín, Fraile and García-Albacete2018).

Other aspects of the survey context have been shown to affect gender differences in political knowledge as well. McGlone, Aronson, and Kobrynowicz (Reference McGlone, Aronson and Kobrynowicz2006) present evidence that the gender gap is exacerbated in an environment that induces stereotype threat, such as if women are aware of the fact that the study focuses on gender differences or if they are interviewed by a male interviewer. However, gender differences are not only induced by how researchers ask their questions, but also by the question content. Focusing on gender-relevant political knowledge items such as information about women’s representation in the federal government has been shown to close—or at least reduce—the gap (Barabas et al. Reference Barabas, Jerit, Pollock and Rainey2014; Dolan Reference Dolan2011; Fraile Reference Fraile2014b; Graber Reference Graber2001). Similarly, the gender gap shrinks or disappears when people are asked about specific policies and/or long-standing facts rather than current events (Ferrín, Fraile, and García-Albacete Reference Ferrín, Fraile and García-Albacete2018), practical issues related to the government such as the availability of benefits and services (Stolle and Gidengil Reference Stolle and Gidengil2010), or in political contexts characterized by more equitable representation of women (McAllister Reference McAllister2019; Pereira Reference Pereira2019; Wolak and Juenke Reference Wolak and Juenke2021). Importantly, women’s lower factual knowledge scores can be easily ameliorated by providing additional information (Jerit and Barabas Reference Jerit and Barabas2017) and they do not appear to impede on their political competence. In fact, Dassonneville et al. (Reference Dassonneville, Nugent, Hooghe and Lau2020) find that women are no less likely to vote for candidates who represent their preferences, and are, therefore, able to participate in politics just as effectively as men.

Overall, the gender gap appears to be influenced by how we ask for political information in surveys, as well as the kind of knowledge that is required for a correct response. Indeed, a comprehensive cross-national analysis of election studies in 47 countries between 1996 and 2011 suggests that question format and content account for large portions of the variance of gender disparities in political knowledge (Fortin-Rittberger Reference Fortin-Rittberger2016; Reference Fortin-Rittberger2020). In short, conventional knowledge measures have problematic measurement properties that may exacerbate observed gender differences.

BACK TO THE ROOTS: THE STRUCTURE OF BELIEF SYSTEMS

Despite the discipline’s reliance on off-the-shelf item batteries, factual knowledge about political institutions has little relevance for competent decision-making in politics, which lead some scholars to suggest that we should start considering alternatives to these types of recall questions (Druckman Reference Druckman2014). From a theoretical perspective, knowledge scores are all but a proxy for an underlying latent trait—political sophistication—which is usually conceptualized based on people’s belief systems instead of focusing on isolated pieces of factual information stored in declarative memory. Belief systems are defined as “a configuration of ideas and attitudes in which the elements are bound together by some form of constraint or functional interdependence” (Converse Reference Converse and Apter1964, 207).

Political sophistication can then be characterized by how these ideas and attitudes (or considerations) are structured along three different dimensions (Luskin Reference Luskin1987). The first, and most obvious one, is the size of a belief system, which simply describes the number of distinct considerations that are available for retrieval. Politics, however, is comprised of a diverse set of independent domains—with some people having a deep grasp of a narrow field and others having a broad and potentially more shallow understanding of various issues. Thus, the second dimension describes the range of a belief system across domains—for example, different policy issues or other evaluative categories. The last dimension is a belief system’s constraint, which describes the extent to which considerations are organized in a meaningful way through differentiation and integration of competing cognitions (Luskin Reference Luskin1987). In other words, this dimension captures whether available considerations are perceived as operating in isolation or are rather as part of a more complex interconnected system, for example, by identifying inherent value conflicts (Tetlock Reference Tetlock1983; Reference Tetlock, Iyengar and McGuire1993). To summarize, I conceptualize political sophistication based on the structure of individual belief systems along the following three dimensions:

1. 1. Size: The number of considerations associated with a given category or issue.

2. 2. Range: The dispersion of considerations across different categories or issues.

3. 3. Constraint: The extent to which considerations are interconnected in a meaningful way.

Political sophistication, in turn, is the conjunction of these dimensions: “A person is politically sophisticated to the extent to which his or her [political belief system] is large, wide-ranging, and highly constrained” (Luskin Reference Luskin1987, 860). Similarly, Tetlock (Reference Tetlock1983; Reference Tetlock, Iyengar and McGuire1993) coined the term integrative complexity to describe the degree to which considerations related to an issue are interconnected. In short, sophisticated political reasoning should reflect this notion of complex belief systems.

To what extent does political sophistication defined as a complex system of beliefs ultimately facilitate citizen competence? As discussed above, conventional knowledge questions have been criticized because the required information has little relevance for people’s ability to make high-quality decisions (Lupia Reference Lupia2006). As Cramer and Toff eloquently summarize, conventional measures implicitly focus on “what people do not know” (Reference Cramer and Toff2017, 756, emphasis added) by presupposing pieces of information as necessary for political competence. Examining people’s political beliefs system, on the other hand, allows us to shift the focus back to what they do know and how they use that information. After all, a large, wide-ranging, and highly constrained system of beliefs will help citizens to locate their own interests within the political system, understand the functioning of institutions, assess the performance of the incumbent government, and evaluate the actions of the main political actors (e.g., Converse Reference Converse and Apter1964).Footnote ³

MEASURING DISCURSIVE SOPHISTICATION

Given that recall questions are only an imperfect measure for political sophistication, it is worth considering alternative—and potentially more imminent—observable implications of the underlying latent trait of interest: complex and highly constrained political belief systems. In the following, I propose a framework that leverages the content of open-ended responses in conjunction with the survey structure to evaluate how people discuss their political beliefs and preferences in their own words (see also Kraft Reference Kraft2018, for a related analysis of moral reasoning in open-ended responses). To illustrate my approach in the context of a concrete example, consider a questionnaire that asks respondents to answer the following open-ended item:

Now suppose that this questionnaire includes a whole set of similar prompts on other topics such as abortion, immigration, health cure, and trade policies—each asking respondents for both positive or negative considerations related to specific policy proposals. How would a complex and constrained set of political beliefs manifest itself across such a battery of open-ended responses? I argue that each dimension outlined above has direct observable implications for individual response behavior.

First, the size of a belief system is defined as the number of available considerations associated with a given category or issue. In the context of open-ended survey questions, a large belief system should, therefore, allow people to discuss their views by raising a larger number of distinct topics in response to each query. While this could also be achieved through manual coding, I rely on the structural topic model framework to extract the number of topics mentioned by each respondent in a survey (Roberts et al. Reference Roberts, Stewart, Tingley, Lucas, Leder-Luis, Gadarian and Albertson2014).Footnote ⁴ Let $ {\mathcal{W}}_i $ denote the set of words contained in a response of individual i. Each word $ w\,\in\,{\mathcal{W}}_i $ is assigned to a topic $ {t}^{*}\,\in\,\{1,...,T\} $ , such that $ P({t}^{*}|w,{X}_i)>P(t|w,{X}_i)\forall t\ne {t}^{*} $ .Footnote ⁵ In other words, each unique term in a response is assigned to the topic that has the highest likelihood of having generated that term, given the model. The set of topics that are mentioned by respondent i across all words in $ {\mathcal{W}}_i $ can then be described as $ {\mathcal{T}}_i^{*} $ and the number of considerations can be written as

(1) $$ \begin{array}{rl}{\mathrm{size}}_i=\frac{|{\mathcal{T}}_i^{*}|}{\max\,|{\mathcal{T}}_i^{*}|}.& \end{array} $$

I rescale the measure to range from zero to one by dividing raw count of topics by the maximum number of topics observed across individuals.

Second, the range of a belief system is defined as the dispersion of considerations across categories or issues. Given a set of survey prompts covering various political issues, high levels of sophistication should correspond with people’s ability to respond to each query with comparable levels of elaboration. Therefore, I quantify the consistency in response behavior across items by computing the Shannon entropy in open-ended response lengths:

(2) $$ \begin{array}{rl}{\mathrm{range}}_i=\frac{-{\displaystyle \sum_{j=1}^J}{p}_{ij}\ln {p}_{ij}}{\ln J},& \end{array} $$

where $ {p}_{ij} $ is the proportion of words in the response of individual i to question $ j\,\in\,\{1,...,J\} $ relative to the overall size of the individuals’ response. The variable ranges from 0 (only one question was answered) to 1 (all questions were answered with the same word length per answer).

The last component addresses the level of constraint between considerations. The extent to which considerations are interconnected in a meaningful way should be associated with people’s ability to differentiate and/or integrate them in their reasoning (Tetlock Reference Tetlock, Iyengar and McGuire1993). Following Tausczik and Pennebaker (Reference Tausczik and Pennebaker2010), I rely on specific function words as linguistic markers for these processes. More specifically, differentiating competing considerations in speech is usually accomplished using exclusive words (e.g., but and without), whereas integrating multiple thoughts is accomplished by the use of conjunctions (e.g., and and also). Thus, I measure relative constraint by identifying the number of conjunctions ( $ {\mathrm{CONJ}}_i $ ) and exclusive words ( $ {\mathrm{EXCL}}_i $ ) in each open-ended response using the Linguistic Inquiry and Word Count dictionary (Pennebaker et al. Reference Pennebaker, Boyd, Jordan and Blackburn2015):

(3) $$ \begin{array}{rl}{\mathrm{constraint}}_i=\frac{{\mathrm{CONJ}}_i+{\mathrm{EXCL}}_i}{\max\,\left[{\mathrm{CONJ}}_i+{\mathrm{EXCL}}_i\right]}.& \end{array} $$

As before, I rescale the measure to range from zero to one by dividing all values by the empirical maximum observed across all individuals in the data.

Together, the three measures can be combined in an additive scale of discursive sophistication in political attitude expression:

(4) $$ \begin{array}{rl}{\mathrm{discursive}\ \mathrm{sophistication}}_i={\mathrm{size}}_i+{\mathrm{range}}_i+{\mathrm{constraint}}_i.& \end{array} $$

Overall, a highly sophisticated individual should, therefore, give a more elaborate response across the full range of questions by integrating and/or differentiating multiple considerations. Given each individual input, the resulting metric has a theoretical range between 0 and 3. In order to allow for easier comparisons with conventional additive knowledge scores, I rescale discursive sophistication to mean zero and unit variance. Note that this simple framework makes no assumptions about the direction of people’s attitudes or their specific ideology. Crucially, since it is solely based on how individuals discuss their preferences, it can be directly applied in various settings to target specific political issues or tasks such as choosing between candidates running for election. In other words, we can study discursive sophistication in well-defined (and potentially narrow) areas by using open-ended questions that focus on attitudes and beliefs that are relevant for a specific context. Researchers interested in citizen competence in local politics, for instance, could field a battery of open-ended questions examining relevant topics such as schooling, zoning, or other areas of local administration.

Of course, this is not the first time a framework is developed to assess the complexity of written (or spoken) word. In fact, this task has been the subject of long-standing research in linguistics and educational sciences, resulting in a multitude of alternative metrics. Recently, these measures have been employed by political scientists who study different forms of elite communication. Spirling (Reference Spirling2016), for example, uses a standard readability score based on the ratio of words per sentence and syllables per word to study the linguistic complexity of speeches in the British House of Commons over time. More recently, Benoit, Munger, and Spirling (Reference Benoit, Munger and Spirling2019) expanded on previous metrics to develop a measure of comprehensibility that is more applicable in the realm of politics.

These approaches—and especially the development of metrics specifically suited for political text—are particularly useful when studying elite communication. Yet, in contrast to the framework outlined above, they focus on the comprehensibility as a measure of complexity; elite sophistication is evaluated based on a recipient’s ease to understand the message, which is largely driven by linguistic and syntactic difficulty rather than actual political content. While this is certainly a reasonable approach when studying the effects of elite communication, the inference of interest outlined in this article is markedly different. My focus is to examine verbatim attitude expression to assess the underlying degree of elaborate political reasoning. Pure linguistic style is, therefore, not of central concern so long as it is unrelated to the actual political content.Footnote ⁶ After all, being hard to comprehend does not necessarily imply that someone put a lot of thought into a statement.

DATA AND ANALYTICAL STRATEGY

To evaluate my proposed measure of discursive sophistication, I rely on the battery of open-ended questions described above, which was included in a 2018 wave of the Cooperative Election Study (CES)Footnote ⁷ consisting of a national stratified sample of one thousand respondents. In addition, I illustrate the versatility and robustness of the approach by applying the measure across multiple previously collected surveys that employ a range of alternative open-ended items. Below is a summary of all datasets and items used in the subsequent analysis:Footnote ⁸

• • Cooperative Election Study (CES 2018): 10 open-ended questions targeting policy preferences on gun legislation, abortion, immigration, health care, and trade.

• • American National Election Study (ANES 2012; 2016; 2020): eight open-ended likes–dislikes questions targeting preferences for parties and candidates.

• • YouGov Survey (2015): four open-ended questions targeting policy preferences on gun legislation and health care.

• • Swiss Referendum Surveys (2008–12): two open-ended questions asking respondents to justify their vote choice in various policy referenda. These surveys were conducted in three languages (French, German, and Italian).

I proceed by providing descriptive evidence regarding the face validity of discursive sophistication. Next, I assess its construct validity by comparing it to factual knowledge as a predictor of various relevant outcomes such as political participation and engagement. The last validation step consists of comparing discursive sophistication to manually coded levels of justification in open-ended responses. Each of these steps leverages different subsets of the studies listed above, depending on the availability of necessary items. After validating the measure, I assess gender gaps in discursive sophistication and factual knowledge using the complete set of surveys.

A FIRST LOOK AT DISCURSIVE SOPHISTICATION

While each dimension of discursive sophistication outlined above provides a unique source of variance to the underlying concept (Luskin Reference Luskin1987), all three are positively correlated.Footnote ⁹ Furthermore, exploratory factor analyses confirm that they load on a single factor with all loadings exceeding 0.5 across the CES and ANES data—thus confirming that we can rely on an additive score to measure discursive sophistication (see Table 1).Footnote ¹⁰

Table 1.

Factor Loadings of Discursive Sophistication Components

How does this discursive sophistication score compare to alternative metrics of political knowledge? As discussed, the standard approach to measuring political knowledge in surveys is to ask a set of factual questions about political institutions. The CES and ANES include such a set of basic recall items, inquiring, for example, about presidential term limits or the majority party in either chamber of Congress. Borrowing the classification in Barabas et al. (Reference Barabas, Jerit, Pollock and Rainey2014), the CES items focus on policy-specific facts, whereas the ANES battery tests general institutional knowledge.Footnote ¹¹ I combine responses on these items to form an additive index of factual knowledge about politics. In order to facilitate easier comparisons with discursive sophistication, each factual knowledge measure is rescaled to zero mean and unit variance. As an additional benchmark, I consider interviewer assessments of each respondent’s political sophistication (see Bartels Reference Bartels2005; but cf. Ryan Reference Ryan2011).Footnote ¹²

Figure 1 compares discursive sophistication to conventional knowledge metrics for the CES and ANES. Each figure presents scatterplots between individual measures (lower triangular), univariate densities (diagonal), and correlation coefficients (upper triangular). The measure of discursive sophistication is positively correlated with both conventional metrics while capturing some additional variation. Interestingly, there is a stronger correlation between discursive sophistication and interviewer evaluations than between factual knowledge and interviewer evaluations ( $ r=0.35 $ vs. $ r=0.23 $ in 2016, and $ r=0.45 $ vs. $ r=0.31 $ in 2012), which indicates that the open-ended measure captures characteristics that influence subjective assessments of sophistication. In other words, a respondent’s verbatim answers seem to be more influential for subsequent knowledge assessments by the interviewer than a respondent’s performance on the factual knowledge questions.

Figure 1.

Correlation Matrix of Discursive Sophistication and Conventional Political Knowledge Metrics in the CES and ANES

Note: Each subfigure (a-d) compares discursive sophistication and conventional political knowledge metrics within a given survey. The panels on the diagonal display univariate densities for each variable. The panels on the lower triangular display scatter plots combining both measures as well as a linear fit. The upper triangular displays correlation coefficients. Correlations are statistically significant at *p< 0.05, **p< 0.01, and ***p< 0.001.

While discursive sophistication and the alternative measures are clearly correlated, the relationship between each metric is far from perfect. To provide some intuition as to whether the variation in discursive sophistication is theoretically meaningful, I present an example of open-ended responses from two individuals in the 2018 CES who scored equally on factual knowledge (three out of five correct responses), but varied in discursive sophistication.

The results are presented in Table 2. Each row represents one of the open-ended responses targeting specific policy issues. Column A displays the responses of an individual who scored low on discursive sophistication and column B displays the responses of a high-scoring individual. Even though both individuals have the same factual knowledge score, there are systematic differences in their response behavior that suggest disparity in their political sophistication. Overall, respondent A provided a less elaborate response and only focused on a narrow range of issues. Irrespective of whether one agrees with the specific statements, A’s response pattern is suggestive of a less sophisticated political belief system and a lower level of motivation to engage in in-depth reasoning about each issue. Overall, this initial result suggests that the variation in discursive sophistication captures meaningful differences in response behavior that overlaps with traditional knowledge metrics while displaying some unique variation. The following sections will show that this variation is also politically consequential.

Table 2.

Example of Open-Ended Responses for Low and High Scores on Discursive Sophistication with Equal Factual Knowledge Scores (Three out of Five Correct Responses)

Note: Column A displays the verbatim responses of an individual who scored low on discursive sophistication and column B displays the verbatim responses of an individual who scored high on the open-ended measure. Note that responses are slightly edited for readability.

VALIDATING THE MEASURE

A crucial step in validating any measure of political sophistication is to examine the extent to which it is correlated with political engagement and citizen competence (Lupia Reference Lupia2006; Reference Lupia2015). Accordingly, I consider how discursive sophistication is associated with (1) engagement and participation in politics, (2) the ability to incorporate new information, and (3) well-justified policy preferences. Appendix C of the Supplementary Material contains robustness checks and supplementary analyses showing, for instance, how discursive sophistication is furthermore predictive of (4) reduced uncertainty about ideological placements of parties and politicians, and (5) higher probabilities to vote based on ideological proximity.

Engagement and Participation in Politics

Any measure of political sophistication should be strongly associated with individual engagement and participation in politics. In fact, factual knowledge items have been validated in the past based on their strong relationship with outcomes such as turnout and other forms of participation (Lupia Reference Lupia2015, 230–3). Figure 2 compares the effect of discursive sophistication and factual knowledge on four dependent variables related to political engagement: turnout, political interest, internal efficacy, and external efficacy. The model predicting turnout is estimated via logistic regression, whereas the estimates for the three remaining dependent variables are based on OLS. In addition to both key predictors, each model controls for gender, education, income, age, race, and church attendance.Footnote ¹³

Figure 2.

Effects of Political Sophistication on Turnout, Political Interest, Internal Efficacy, and External Efficacy in the CES and ANES (Including 95% Confidence Intervals)

Note: Estimates are based on logistic (turnout) or linear (political interest, internal efficacy, and external efficacy) regressions. Each model includes controls for sociodemographic variables. Full regression results are displayed in Appendix D.I of the Supplementary Material.

Each panel in Figure 2 compares the estimated effect of increasing either sophistication measure from one standard deviation below the mean to one standard deviation above the mean (holding all other variables constant at their means). Note that the examples previously shown in Table 2 illustrate the substantive meaning of such a two standard deviation increase in discursive sophistication. For factual knowledge, on the other hand, this increase is approximately equivalent to correctly answering three additional knowledge questions.Footnote ¹⁴ Of course, these effects are purely correlational and should not be interpreted causally. Nevertheless, across all four surveys, discursive sophistication and factual knowledge are complementary and similarly sized predictors of turnout, political interest, and internal efficacy.Footnote ¹⁵ Only for external efficacy we find more ambiguous results. Factual knowledge has strikingly inconsistent effects—sometimes predicting higher, lower, or no change in external efficacy. Discursive sophistication, in contrast, is more consistently associated with higher external efficacy (the only exception is the 2018 CES, which uses a shorter battery to measure external efficacy).

Considering these initial results, a potential concern may be that discursive sophistication is confounded by individual characteristics that influence verbatim response patterns as well as engagement. As a robustness check, Appendix C.III of the Supplementary Material provides additional regression results controlling for various factors that might drive verbosity such as personality (extraversion, openness to experience, and being reserved), survey mode (online vs. face-to-face), verbal skills (Wordsum vocabulary test score), and overall verbosity itself (response length). The substantive conclusions remain unchanged.

Incorporation of New Information

In order to replicate and extend this first validation, I rely on a separate nationally representative survey employing an alternative set of open-ended responses. The data were collected by YouGov in December 2015 and contain responses of one thousand U.S. residents.Footnote ¹⁶ As part of this study, respondents were asked four open-ended questions to describe their attitudes toward two salient issues: gun legislation and the Affordable Care Act.

Political sophistication should make it easier for people to incorporate relevant new information about parties, office-holders, and policies. After all, Zaller (Reference Zaller1990; Reference Zaller1992) and others argue that factual knowledge is the best available proxy for political awareness. In this analysis, I explore whether discursive sophistication serves as an accurate predictor of people’s ability to incorporate new information from media sources. As part of the survey, respondents were asked to read a newspaper article about a fictional infectious disease and were subsequently asked to answer questions about information provided in the article (e.g., regarding symptoms and modes of contraction). I compute an additive index counting the pieces of information that were correctly recalled (information retrieval, ranging from 0 to 9) as a measure of the ability to retrieve information from a news article on a nonpartisan issue that is related to public health policies.

Figure 3 displays the relationship between political sophistication and disease information retrieval in the 2015 YouGov study. Estimates are based on a linear regression controlling for education, income, age, church attendance, gender, and race. As a benchmark for discursive sophistication, I again consider the effect of factual knowledge based on a battery of eight items similar to the knowledge questions in the ANES. Recall that both measures are rescaled to zero mean and unit variance to facilitate direct comparisons between them. Both discursive sophistication and factual knowledge are positively correlated with the amount of information individuals are able to recall from a news article discussing a fictional disease. In addition, this analysis reveals how discursive sophistication can help explain important variation at both tails of the distribution. Conventional additive knowledge scales often suffer from ceiling effects since there is no way to differentiate respondents who answer all questions correctly (or incorrectly, although that is less common with standard batteries). Discursive sophistication suffers from no such constraints and, therefore, allows us to better represent the full spectrum of the underlying latent variable. Thus, the degree to which citizens discuss their own political beliefs in a more elaborate manner is not only a strong predictor of political engagement, but also serves as a powerful proxy for the ability to incorporate new information about a nonpartisan issue.

Figure 3.

Expected Information Retrieval in the 2015 YouGov Study as a Function of Political Sophistication (Including 95% Confidence Intervals)

Note: Estimates are based on a linear regression including controls for sociodemographic variables. The predictions are made by setting covariates equal to their mean (continuous covariate) or median (categorical covariate) value. Full regression results are displayed in Appendix D.II of the Supplementary Material.

Well-Justified Policy Preferences

As the last validation step, I examine an additional set of surveys that provide a unique opportunity to compare my proposed measure of discursive sophistication with manually coded open-ended responses across three languages. Colombo (Reference Colombo2018) compiled a dataset of cross-sectional surveys administered in Switzerland after national popular votes on multiple policy propositions. For each referendum, respondents were asked to explain why they voted in favor or against a given proposition in two separate open-ended items. Based on these verbatim responses, I computed discursive sophistication using the same procedure outlined above. Since the survey was conducted in three different languages (German, French, and Italian), I created separate metrics for each group.

Beyond the ability to incorporate new information, political sophistication should enable people to justify their own preferences. Colombo’s (Reference Colombo2018) manual coding of the respondents’ level of justification assessed the content, elaboration, and complexity of open-ended responses. Thus, this study provides an opportunity to directly assess the extent to which high levels of discursive sophistication correspond to well-justified policy preferences in open-ended responses. Any overlap between Colombo’s (Reference Colombo2018) manual coding with my automated measure corroborates the face validity of discursive sophistication.

The results are presented in Figure 4, which displays the distribution of discursive sophistication for each level of justification coded by Colombo (Reference Colombo2018) as well as the correlation coefficients for both respective variables. Across all three language groups, discursive sophistication is systematically higher among respondents with the highest level of justification and both measures are positively correlated ( $ r=0.26,0.33 $ , and $ 0.36 $ ). The proposed measure of discursive sophistication, therefore, shows a high degree of correspondence with individual levels of justification assessed by independent manual coders.

Figure 4.

Discursive Sophistication and Manually Coded Level of Justification (Colombo Reference Colombo2018) in Swiss Post-Referendum Surveys

Note: The plot compares kernel densities of discursive sophistication for each manually coded level of justification.

To summarize, the results presented thus far indicate that while discursive sophistication shares common characteristics with factual political knowledge measures, both capture different dimensions of sophistication. Indeed, the text-based measure and conventional metrics are independent predictors of political participation and engagement. In addition, discursive sophistication provides a better proxy for the ability to incorporate new information from news sources and shares significant overlap with manually coded levels of justification in open-ended responses. Supplementary analyses reveal that respondents who score higher on discursive sophistication display a smaller degree of uncertainty around the ideological placement of politicians and parties (Appendix C.IV of the Supplementary Material)—and are ultimately more likely to vote based on ideological proximity in senatorial races (Appendix C.V of the Supplementary Material). Next, I illustrate how discursive sophistication can help refine previous findings regarding the gender gap in political knowledge.

REASSESSING THE GENDER GAP

How do women and men compare on the different metrics of political sophistication in the surveys analyzed in the present study? Figure 5 displays the distributions of discursive sophistication and conventional metrics comparing both genders. While we observe sizable and statistically significant gender gaps in factual knowledge across the CES, ANES, and YouGov surveys, these differences all but disappear for discursive sophistication. In other words, while women may perform worse than men on political quizzes, there are no gender differences in the level of elaboration when describing their political preferences.

Figure 5.

The Gender Gap in Political Sophistication

Note: The figures display distributions of political sophistication using open-ended or conventional measures comparing women and men (including 95% confidence intervals around the means). Gender differences are statistically significant at $ {}^{*}p<0.05 $ , $ {}^{**}p<0.01 $ , and $ {}^{***}p<0.001 $ .

Of course, we need to make sure that this absence of a gender gap in discursive sophistication is not idiosyncratic to the particular measurement approach proposed here. One way to investigate this question is to examine gender differences in discursive sophistication using data from Colombo (Reference Colombo2018) and comparing them to her manually coded measure. That way, we cannot only determine whether the lack of a gender gap in discursive sophistication replicates in the Swiss survey, but also check whether there is an equivalent lack of gender differences in Colombo’s alternative measure of citizen competence in direct democracies. If discursive sophistication captures a person’s motivation to undertake in-depth reasoning and form quality opinions (and assuming these characteristics do not differ by gender), there should be no difference between women and men on either metric (discursive sophistication and Colombo’s measure).

The bottom row of Figure 5 reveals insignificant gender differences for all but one of the metrics across all three languages in the Swiss referendum surveys.Footnote ¹⁷ Thus, the absence (or at least reduction) of the gender gap remains robust—whether open-ended responses are coded manually or using the discursive sophistication approach.

Next, we have to consider whether the apparent gender gap in factual knowledge is a manifestation of real differences between women and men. Prior research attributes at least part of the gap to actual discrepancies in individual resources and engagement. Accordingly, we need to control for these determinants of political knowledge to provide a more comprehensive examination of the veracity of observed gender differences. In addition, to the extent that we observe significant gender differences in discursive sophistication—such as in the 2020 ANES or among German respondents in the Swiss survey—we need to assess to what extent these differences are substantively meaningful. Figure 6 shows estimated gender differences after controlling for various potential common determinants such as education, income, age, race, and church attendance. Following Rainey (Reference Rainey2014), the figure also displays a range of small effect sizes (equivalent to Cohen’s $ d\le 0.2 $ ; see Sawilowsky Reference Sawilowsky2009), in order to evaluate whether statistically significant differences are indeed substantively meaningful.

Figure 6.

The Gender Gap in Political Sophistication Controlling for Common Determinants

Note: Estimates are OLS regression coefficients with 95% and 90% confidence intervals. Dependent variables are discursive sophistication and factual political knowledge. Estimates are based on linear regressions including controls for sociodemographic variables. Dashed lines indicate a range of small effect sizes equivalent to Cohen’s $ d\le 0.2 $ . Full regression results are displayed in Appendix D.III of the Supplementary Material.

After controlling for common determinants, discursive sophistication only reveals negligible (and almost exclusively statistically insignificant) differences between women and men across the CES, ANES, and YouGov surveys. Indeed, the 90% confidence intervals only contain negligible effects within Cohen’s $ d\le 0.2 $ , which implies that we can reject the null hypothesis of meaningful gender differences in discursive sophistication (Rainey Reference Rainey2014).Footnote ¹⁸ The gender gap in factual political knowledge, however, persists and is substantively as well as statistically significant.Footnote ¹⁹ Thus, a considerable portion of the observed differences in factual knowledge between women and men cannot be attributed to underlying disparities in resource-related factors or engagement. Comparing the confidence intervals across both measures further reveals that the insignificant gender differences in discursive sophistication are estimated with similar precision as the significant differences in factual knowledge. Such a result precludes the possibility that null findings for discursive sophistication are purely driven by measurement error on the dependent variable. It is also worth pointing out in this context that the remaining control variables exhibit effects of similar magnitude (and uncertainty) across both measures, which further suggests that there is no systematic difference in measurement error.Footnote ²⁰ For instance, knowledge and discursive sophistication are significantly higher among respondents who are more educated and have higher income. The finding that core sociodemographic predictors of political sophistication are consistent across models lends additional validity to the open-ended measure.

That said, supplementary analyses included in Appendix C.VII of the Supplementary Material reveal that discursive sophistication and factual knowledge have diverging associations with certain personality characteristics, verbal skills, and survey mode.Footnote ²¹ For instance, while openness to experience has a positive effect on discursive sophistication, it has a negative effect on factual knowledge (at least in the 2012 ANES). Being reserved, on the other hand, shows a negative association with discursive sophistication but no relationship with factual knowledge. Especially interesting, however, is the finding that verbal skills (measured using the Wordsum vocabulary test) have a stronger effect on factual knowledge than discursive sophistication. Furthermore, respondents in online surveys score significantly higher on factual knowledge than in face-to-face interviews. This difference can be attributed to the fact that individuals are able to look up answers for factual knowledge questions while taking an online survey (Clifford and Jerit Reference Clifford and Jerit2016). For discursive sophistication, on the other hand, individuals perform better in the face-to-face survey. Open-ended answers in online surveys may be less elaborate because respondents have to manually type their responses. These results illustrate once again that both measures should be seen as complements rather than competing metrics of political sophistication, as they capture different aspects of the underlying concept of interest.

EXPLAINING THE (LACK OF A) GENDER GAP

To summarize, conventional knowledge measures and discursive sophistication produce diverging conclusions regarding the existence of a gender gap. This naturally raises the question which metric we should ultimately trust? Prior research attributed gender differences in factual knowledge—at least partly—to the format (e.g., availability of “don’t know” options) and content (e.g., focusing on issues that are less relevant to women) of item batteries. This section explores whether these arguments provide a sufficient explanation for the conflicting results for discursive sophistication—namely the complete lack of systematic differences between women and men. In other words, which one is more likely to be an artifact of the respective measurement approach: the existence of a gender gap in factual knowledge or the absence of a gap in discursive sophistication?

The first set of arguments as to why conventional metrics may overstate potential gender differences is based on the finding that women are less likely to guess than men (Mondak and Anderson Reference Mondak and Anderson2004). Arguably, respondents’ differential willingness to admit not knowing the answer to a question is certainly less of an issue when they are simply asked to voice their opinions rather than being quizzed on political facts. Following best practices, however, the surveys presented here omitted “don’t know” options in their recall questions. Differential propensity to guess cannot, therefore, be viewed as a valid explanation for the gender gap in factual knowledge observed here. At the same time, the lack of significant differences between women and men in discursive sophistication may itself be the product of selection biases in women’s willingness to answer open-ended question in the first place. Following this argument, it could be the case that only women who are highly sophisticated provide a response, thereby misleadingly closing the gender gap in the discursive measure. There are two reasons why that is unlikely to be the case. First, as the analyses presented thus far have shown, this potential selection mechanism does not diminish gender differences in factual knowledge. Second, and more importantly, there are no significant differences between men’s and women’s willingness to answer open-ended questions.Footnote ²² In fact, adjusting for potential selection effects when examining determinants of sophistication does not change the substantive conclusions.

The second major explanation for the gender gap in political knowledge focuses on question content. By choosing a specific set of recall questions as a general metric for political knowledge, researchers are making strong assumptions about the information deemed necessary for competent decision-making. As it turns out, these item batteries usually focus on male-dominated topics in politics (Dolan Reference Dolan2011). Open-ended questions, on the other hand, make it possible to directly study the information that is in fact available to citizens and—importantly—to examine how they apply their knowledge when discussing their political preferences.

Accordingly, if it is the case that the gender gap in discursive sophistication is nonexistent simply because open-ended questions allow women to raise political considerations particularly salient to them, then we should be able to observe systematic variation in types of issues discussed by women and men, respectively. Luckily, we can directly examine such gender differences in topic prevalence within the structural topic model framework used to measure discursive sophistication. More specifically, gender is included in the model as one of the covariates that influences how often each topic is discussed by a respondent (see also Roberts et al. Reference Roberts, Stewart, Tingley, Lucas, Leder-Luis, Gadarian and Albertson2014, for details).

Therefore, I explore in this last analysis how women and men differ in topical prevalence across open-ended responses in the 2012, 2016, and 2020 ANES. Note that these open-ended items did not focus on specific issue areas as in the CES, but rather asked respondents to evaluate different political parties and candidates. Thus, they were able to focus on whatever issue they deemed most important. Figure 7 displays the subset of topics that shows the largest absolute gender difference in topic prevalence in both waves. Positive coefficients indicate that women are more likely than men to mention a given topic, and vice versa. The top five topics are more prevalent among men and the bottom five are more likely to be mentioned by women. The label for each coefficient consists of the five most frequent and exclusive (FREX) terms related to the topic to illustrate its content.

Figure 7.

Gender Differences in Topic Proportions in Open-Ended Responses Based on the Structural Topic Model Used to Compute Discursive Sophistication (Including 95% Confidence Intervals)

Note: Coefficients indicate the difference in predicted topic prevalence among women and men; positive values indicate higher prevalence among women. Labels are based on the five most frequent and exclusive (FREX) terms associated with each topic.

Taking the 2012 ANES as an example, the topic consisting of terms such as care, health, and reform is significantly more likely to be mentioned by women. On the other hand, men are more likely to mention the topic revolving around terms like tax, deficit, and cut. Overall, across all three waves of the ANES, women were less likely than men to discuss foreign affairs, economic issues, or the Supreme Court. Instead, they focused on issues related to women’s rights, equality, and health care. The considerations raised by women when discussing their political preferences are, therefore, clearly different from men’s and—crucially—the issues discussed by men happen to be more aligned with the type of questions usually covered in standard political knowledge batteries (i.e., pertaining to the economy, institutions, elites, etc.). For example, men are more likely to mention considerations related to the federal budget in their open-ended responses. At the same time, two of the five knowledge questions included in the 2012 ANES pertain to government spending: one asking respondents to compare the federal deficit to levels in 1990, the other requiring a comparison of federal spending on different programs such as foreign aid, medicare, and national defense.

Overall, the results indicate that gender differences in conventional knowledge metrics are at least partly driven by the fact that the issues women care about are not represented in standard item batteries. When using the alternative measure—discursive sophistication—any evidence for systematic differences between women and men disappears since open-ended questions about political preferences allow respondents to focus on specific considerations that are most salient to them.