Political and Social Discussion Network Survey Items Are Not Interchangeable

Abstract Experimentalists and survey researchers regularly measure the makeup and size of respondent personal discussion networks to learn about the social context in which citizens make political choices. When measuring these personal networks, some scholars use question prompts that specifically ask respondents about whom they discuss “politics” with, while others use more general prompts that ask respondents about whom they discuss “important matters” with. Prior research suggests that “political” discussion network prompts create self-reported networks that are substantively similar to “important matters” prompts. We conduct a nationally representative survey experiment to re-evaluate this question. Our results suggest that, although the size of networks generated by the two questions may be similar on average, the two questions generate different response distributions overall. In particular, respondents interested in politics report larger political discussion networks than general discussion networks, and respondents uninterested in politics report smaller political discussion networks than general discussion networks.


Introduction
Personal social networks play an important role in citizens' political behaviors and attitudes (Pietryka and DeBats, 2017;Reilly, 2017;Ryan, 2011;Santoro and Beck, 2017;Settle, Bond, and Levitt 2011;Sinclair, 2012;Sokhey and McClurg, 2012;Song and Eveland Jr, 2015).As a result, social network measures have been widely used in surveys to measure respondents' personal communication contexts.Among scholars who analyze political discussion, however, there is a long-running debate about whether important differences exist between measuring a citizen's "political" discussion networks and their more general "social" discussion networks.For instance, some scholars suggest that a citizen's political discussion network will be This article has earned badges for transparent research practices: Open Data and Open Materials.For details see the Data Availability Statement.more politically homogenous than their general social discussion network due to political conflict avoidance on the part of citizens (Mutz, 2002).Others, however, suggest that most citizens do not meaningfully distinguish between political and social discussion partners, discussing both political and nonpolitical matters in a single core discussion network of close acquaintances (Huckfeldt and Sprague, 1995).
This debate yields challenges for the construction of survey items.Should researchers ask respondents about their "political" discussion networks or their general "social" discussion networks?Klofstad, McClurg, and Rolfe (2009) argue that the matter is moot: they find that political discussion survey questions prompt largely indistinguishable responses to more general "important matters" social discussion questions on a variety of dimensions, including the size of the discussion network, characteristics of network members, and discussion frequency.
Yet questions remain about the interchangeability of the two discussion measures.Responding to survey questions is a cognitively complex process of memory search, retrieval, consolidation, and finally, mapping judgments to a response option (Krosnick, 1999;Schwarz and Oyserman, 2001).Moreover, many people have different ideas about what constitutes "political" discussion in the first place (Eveland Jr, Morey, and Hutchens 2011;Settle, 2018).Politically interested respondents, in particular, are not only more enthusiastic about discussing politics with others but also have a significantly broader view of what counts as "political" discussion than do the politically uninterested (Fitzgerald, 2013).Simultaneously, increased partisan animosity in recent years (Mason, 2018) has made less interested citizens more inclined to disengage from contentious and rancorous political discussion entirely (Klar and Krupnikov, 2016;Settle and Carlson, 2019).
Accordingly, in the modern political environment, we might expect different kinds of responses from citizens now than in the past.In particular, not only might highly politically interested citizens report larger political discussion networks, when compared to garden-variety social discussion networks, but also the politically uninterested may report smaller political discussion networks.The two survey measures may not be as interchangeable as previously thought.

Study
To evaluate whether political discussion and general social discussion prompts are interchangeable on surveys, we embedded a survey experiment in a module of the 2020 Cooperative Election Study (Reilly, 2022;Schaffner, Ansolabehere, and Luks 2021).In the module, we randomly assigned every respondent to receive one of two network prompts: • Treatment A ("Politics"): How many friends and family members would you say you regularly talk to about politics?(n = 427) • Treatment B ("Social"): How many friends and family members would you say you regularly talk to about important matters in your life?(n = 415) Questions were phrased to be brief but still closely match other survey questions regularly used in political research, including the choice of "politics" and "important matters" as the two key phrases (see, for instance, Huckfeldt, Johnson, and Sprague (2004); Klofstad, McClurg, and Rolfe (2009)).Respondents could indicate that they talked to 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10 or 10 people. 1  On average, "politics" respondents reported 4.3 discussion partners, while "social" respondents reported 4.0 discussion partners.However, while the two resulting response distributions are centered in roughly the same place, they have different shapes.The "politics" group provided more responses on the tails of the distribution (0 or 10 discussion partners) while the "social" group reported comparably fewer, resulting in a distribution with more central clustering, peakedness, and a smaller overall standard deviation (sd pol 3:2, sd soc 2:5; kurtosis pol 2:6, kurtosis soc 4:0; see Figure 1).Furthermore, although a t-test yields a negligible and statistically insignificant difference between sample means, a Kolmogorov-Smirnov test reveals a statistically significant difference in the distribution of overall responses (Table 1).Additional analysis confirms that the probability of response is significantly different in tail categories 0 and 10, where more people from the "politics" condition respond, and in central categories 3 and 4, where more people from the "social" condition respond. 2  To evaluate whether these differences were related to political interest, as speculated, we fit a censored poisson regression model, predicting reported discussion network sizes by the experimental condition, the respondent's level of political interest, and the interaction of the two variables.We operationalized "political interest"  For purposes of analysis, the "10" category was coded as "11".

2
See appendix for statistical tests on this point.
Political and Social Discussion Network Survey Items Are Not Interchangeable using a survey question asking how often a respondent would "follow what's going on in government and public affairs", with four response options varying from "hardly at all" to "most of the time". 3igure 2 plots the predicted network size of respondents based on their experimental condition and level of political interest.Our results reveal that respondents who were most interested in politics responded with higher network sizes when in the "politics" condition than when in the "social" condition.Accordingly, respondents who were least interested in politics were more likely to respond with lower network sizes in the "politics" condition when compared with the "social" condition.Perhaps most tellingly, there was little overall variation in network size across political interest for those in the "social" condition, but significant variation across political interest for those in the "politics" condition.
Figure 1.Distribution of network size across two measures. 1 Figure 2.Predicted discussion network size response varies by political interest for "politics" respondents but not for "social" respondents (95% confidence intervals).

Table 1 .
Distribution of network size across two measures