Skip to main content Accessibility help
×
Home
Hostname: page-component-7ccbd9845f-dzwm5 Total loading time: 0.981 Render date: 2023-01-30T15:51:32.131Z Has data issue: true Feature Flags: { "useRatesEcommerce": false } hasContentIssue true

Political and Social Discussion Network Survey Items Are Not Interchangeable

Published online by Cambridge University Press:  16 January 2023

Jack Lyons Reilly*
Affiliation:
Division of Social Sciences, New College of Florida, USA
Jack K. Belk Jr.
Affiliation:
Duke University, North Carolina, USA
*
*Corresponding author. Email: jreilly@ncf.edu
Rights & Permissions[Opens in a new window]

Abstract

Experimentalists and survey researchers regularly measure the makeup and size of respondent personal discussion networks to learn about the social context in which citizens make political choices. When measuring these personal networks, some scholars use question prompts that specifically ask respondents about whom they discuss “politics” with, while others use more general prompts that ask respondents about whom they discuss “important matters” with. Prior research suggests that “political” discussion network prompts create self-reported networks that are substantively similar to “important matters” prompts. We conduct a nationally representative survey experiment to re-evaluate this question. Our results suggest that, although the size of networks generated by the two questions may be similar on average, the two questions generate different response distributions overall. In particular, respondents interested in politics report larger political discussion networks than general discussion networks, and respondents uninterested in politics report smaller political discussion networks than general discussion networks.

Type
Short Report
Creative Commons
Creative Common License - CCCreative Common License - BY
This is an Open Access article, distributed under the terms of the Creative Commons Attribution licence (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted re-use, distribution and reproduction, provided the original article is properly cited.
Open Practices
Open dataOpen materials
Copyright
© The Author(s), 2023. Published by Cambridge University Press on behalf of American Political Science Association

Introduction

Personal social networks play an important role in citizens’ political behaviors and attitudes (Pietryka and DeBats, Reference Pietryka and DeBats2017; Reilly, Reference Reilly2017; Ryan, Reference Ryan2011; Santoro and Beck, Reference Santoro and Beck.2017; Settle, Bond, and Levitt Reference Settle, Bond and Levitt.2011; Sinclair, Reference Sinclair2012; Sokhey and McClurg, Reference Sokhey and McClurg.2012; Song and Eveland Jr, Reference Song and Eveland2015). As a result, social network measures have been widely used in surveys to measure respondents’ personal communication contexts. Among scholars who analyze political discussion, however, there is a long-running debate about whether important differences exist between measuring a citizen’s “political” discussion networks and their more general “social” discussion networks. For instance, some scholars suggest that a citizen’s political discussion network will be more politically homogenous than their general social discussion network due to political conflict avoidance on the part of citizens (Mutz, Reference Mutz2002). Others, however, suggest that most citizens do not meaningfully distinguish between political and social discussion partners, discussing both political and nonpolitical matters in a single core discussion network of close acquaintances (Huckfeldt and Sprague, Reference Huckfeldt and Sprague.1995).

This debate yields challenges for the construction of survey items. Should researchers ask respondents about their “political” discussion networks or their general “social” discussion networks? Klofstad, McClurg, and Rolfe (Reference Klofstad, McClurg and Rolfe.2009) argue that the matter is moot: they find that political discussion survey questions prompt largely indistinguishable responses to more general “important matters” social discussion questions on a variety of dimensions, including the size of the discussion network, characteristics of network members, and discussion frequency.

Yet questions remain about the interchangeability of the two discussion measures. Responding to survey questions is a cognitively complex process of memory search, retrieval, consolidation, and finally, mapping judgments to a response option (Krosnick, Reference Krosnick1999; Schwarz and Oyserman, Reference Schwarz and Oyserman.2001). Moreover, many people have different ideas about what constitutes “political” discussion in the first place (Eveland Jr, Morey, and Hutchens Reference Eveland, Morey and Hutchens.2011; Settle, Reference Settle2018). Politically interested respondents, in particular, are not only more enthusiastic about discussing politics with others but also have a significantly broader view of what counts as “political” discussion than do the politically uninterested (Fitzgerald, Reference Fitzgerald2013). Simultaneously, increased partisan animosity in recent years (Mason, Reference Mason2018) has made less interested citizens more inclined to disengage from contentious and rancorous political discussion entirely (Klar and Krupnikov, Reference Klar and Krupnikov.2016; Settle and Carlson, Reference Settle and Carlson.2019).

Accordingly, in the modern political environment, we might expect different kinds of responses from citizens now than in the past. In particular, not only might highly politically interested citizens report larger political discussion networks, when compared to garden-variety social discussion networks, but also the politically uninterested may report smaller political discussion networks. The two survey measures may not be as interchangeable as previously thought.

Study

To evaluate whether political discussion and general social discussion prompts are interchangeable on surveys, we embedded a survey experiment in a module of the 2020 Cooperative Election Study (Reilly, Reference Reilly2022; Schaffner, Ansolabehere, and Luks Reference Schaffner, Ansolabehere and Luks.2021). In the module, we randomly assigned every respondent to receive one of two network prompts:

  • Treatment A (“Politics”): How many friends and family members would you say you regularly talk to about politics? (n = 427)

  • Treatment B (“Social”): How many friends and family members would you say you regularly talk to about important matters in your life? (n = 415)

Questions were phrased to be brief but still closely match other survey questions regularly used in political research, including the choice of “politics” and “important matters” as the two key phrases (see, for instance, Huckfeldt, Johnson, and Sprague (Reference Huckfeldt, Johnson and Sprague.2004); Klofstad, McClurg, and Rolfe (Reference Klofstad, McClurg and Rolfe.2009)). Respondents could indicate that they talked to 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10 or 10+ people.Footnote 1

Table 1. Distribution of network size across two measures

On average, “politics” respondents reported 4.3 discussion partners, while “social” respondents reported 4.0 discussion partners. However, while the two resulting response distributions are centered in roughly the same place, they have different shapes. The “politics” group provided more responses on the tails of the distribution (0 or 10+ discussion partners) while the “social” group reported comparably fewer, resulting in a distribution with more central clustering, peakedness, and a smaller overall standard deviation ( $sd_{pol}=3.2$ , $sd_{soc}=2.5$ ; $kurtosis_{pol}=2.6$ , $kurtosis_{soc}=4.0$ ; see Figure 1). Furthermore, although a t-test yields a negligible and statistically insignificant difference between sample means, a Kolmogorov–Smirnov test reveals a statistically significant difference in the distribution of overall responses (Table 1). Additional analysis confirms that the probability of response is significantly different in tail categories 0 and 10+, where more people from the “politics” condition respond, and in central categories 3 and 4, where more people from the “social” condition respond.Footnote 2

Figure 1. Distribution of network size across two measures.

To evaluate whether these differences were related to political interest, as speculated, we fit a censored poisson regression model, predicting reported discussion network sizes by the experimental condition, the respondent’s level of political interest, and the interaction of the two variables. We operationalized “political interest” using a survey question asking how often a respondent would “follow what’s going on in government and public affairs”, with four response options varying from “hardly at all” to “most of the time”.Footnote 3

Figure 2 plots the predicted network size of respondents based on their experimental condition and level of political interest. Our results reveal that respondents who were most interested in politics responded with higher network sizes when in the “politics” condition than when in the “social” condition. Accordingly, respondents who were least interested in politics were more likely to respond with lower network sizes in the “politics” condition when compared with the “social” condition. Perhaps most tellingly, there was little overall variation in network size across political interest for those in the “social” condition, but significant variation across political interest for those in the “politics” condition.

Figure 2. Predicted discussion network size response varies by political interest for “politics” respondents but not for “social” respondents (95% confidence intervals).

Discussion

Our results suggest that “politics” and general “social” prompts are not fully interchangeable when measuring the size of respondent-reported interpersonal discussion networks in surveys. A “politics” generator results in more responses with smaller or larger networks than does an “important matters” social network generator. Furthermore, respondent levels of political interest are strongly correlated with discussion network size in the “politics” generator, but not in the “social” generator, indicating these distributions are generated by different survey response processes in our respondents.

These findings have implications for how we measure social and political discussion networks and for how we think of discussion network formation generally. Methodologically, researchers should consider whether they want to generate estimates of personal networks that are more political in nature or more general in nature. Substantively, our results suggest that political discussion networks are, in fact, different in nature from general social discussion networks, in contrast to earlier findings (Huckfeldt and Sprague, Reference Huckfeldt and Sprague.1995; Klofstad, McClurg, and Rolfe Reference Klofstad, McClurg and Rolfe.2009). Accordingly, some changes may be required in the way we think about social communication in politics in the modern polarized age.

Supplementary material

To view supplementary material for this article, please visit https://doi.org/10.1017/XPS.2022.34

Data availability statement

Support for the 2020 Cooperative Election Study data used in this research was provided by the National Science Foundation (Award #1948863).

The data, code, and any additional materials required to replicate all analyses in this article are available at the Journal of Experimental Political Science Dataverse within the Harvard Dataverse Network, at: https://doi.org/10.7910/DVN/O96MDX. (Reilly and Belk, Reference Reilly and Belk2022).

Acknowledgments

We thank Matthew Pietryka, Debra Leiter, and members of the New College of Florida Political Behavior Lab for comments and advice on earlier versions of this manuscript, as well as the comments of three anonymous reviewers. We also thank the National Science Foundation (Award #1948863) and the New College of Florida for funding support.

Conflicts of interest

The authors declare no conflicts of interest.

Ethics statement

The data collection for this research study was approved by the New College of Florida’s Institutional Review Board (IRB ID: 2016/08/1) and adheres to all relevant aspects of APSA’s Principles and Guidance for Human Subjects Research.

Footnotes

This article has earned badges for transparent research practices: Open Data and Open Materials. For details see the Data Availability Statement.

1 For purposes of analysis, the “10+” category was coded as “11”.

2 See appendix for statistical tests on this point.

3 We treated the ordinal variable for interest categorically in our model. See appendix for further technical details, regression tables, and marginal effects plot.

References

Eveland, W. P. Jr., Morey, A. C., and Hutchens., M. J. 2011. “Beyond Deliberation: New Directions for the Study of Informal Political Conversation from a Communication Perspective.” Journal of Communication 61(6):1082–103.CrossRefGoogle Scholar
Fitzgerald, J. 2013. “What does ‘Political’ Mean to You?Political Behavior 35(3):453–79.CrossRefGoogle Scholar
Huckfeldt, R., Johnson, P. E., and Sprague., J. 2004. Political Disagreement: The Survival of Diverse Opinions within Communication Networks. Cambridge, UK: Cambridge University Press.CrossRefGoogle Scholar
Huckfeldt, R. R., and Sprague., J. 1995. Citizens, Politics and Social Communication: Information and Influence in an Election Campaign. Cambridge, UK: Cambridge University Press.CrossRefGoogle Scholar
Klar, S., and Krupnikov., Y. 2016. Independent Politics. Cambridge, UK: Cambridge University Press.CrossRefGoogle Scholar
Klofstad, C. A., McClurg, S. D., and Rolfe., M. 2009. “Measurement of Political Discussion Networks: A Comparison of Two ‘Name Generator’ Procedures.” Public Opinion Quarterly 73(3):462–83.CrossRefGoogle Scholar
Krosnick, J. A. 1999. “Survey Research.” Annual Review of Psychology 50(1):537–67.CrossRefGoogle ScholarPubMed
Mason, L. 2018. Uncivil Agreement: How Politics Became Our Identity. Chicago, IL: University of Chicago Press.CrossRefGoogle Scholar
Mutz, D. C. 2002. “The Consequences of Cross-Cutting Networks for Political Participation.” American Journal of Political Science 46(4):838–55.CrossRefGoogle Scholar
Pietryka, M. T., and DeBats, D. A. 2017. “It’s Not Just What You Have, But Who You Know: Networks, Social Proximity to Elites, and Voting in State and Local Elections.” American Political Science Review 111(2):360–78.CrossRefGoogle Scholar
Reilly, J. L. 2017. “Social Connectedness and Political Behavior.” Research & Politics 4(3):18.CrossRefGoogle Scholar
Reilly, J. L. 2022. CES 2020, Team Module of the New College of Florida (NCF): Harvard Dataverse, V1. doi: 10.7910/DVN/RWN2MC.Google Scholar
Reilly, J. L., and Belk, J. K. 2022. Replication Data for: “Political and Social Discussion Network Survey Items are Not Interchangeable”. Harvard Dataverse, V1. doi: 10.7910/DVN/O96MDX.Google Scholar
Ryan, J. B. 2011. “Social Networks as a Shortcut to Correct Voting.” American Journal of Political Science 55(4):753–66.CrossRefGoogle Scholar
Santoro, L. R., and Beck., P. A. 2017. Social Networks and Vote Choice. The Oxford Handbook of Political Networks. Oxford: Oxford University Press, 383406.Google Scholar
Schaffner, B., Ansolabehere, S., and Luks., S. 2021. Cooperative Election Study Common Content, 2020. Harvard Dataverse, V4. doi: 10.7910/DVN/E9N6PH.Google Scholar
Schwarz, N., and Oyserman., D. 2001. “Asking Questions about Behavior: Cognition, Communication, and Questionnaire Construction.” The American Journal of Evaluation 22(2):127–60.CrossRefGoogle Scholar
Settle, J. E. 2018. Frenemies: How Social Media Polarizes America. Cambridge, UK: Cambridge University Press.CrossRefGoogle Scholar
Settle, J. E., Bond, R., and Levitt., J. 2011. “The Social Origins of Adult Political Behavior.” American Politics Research 39(2):239–63.CrossRefGoogle Scholar
Settle, J. E., and Carlson., T. N. 2019. “Opting Out of Political Discussions.” Political Communication 36(3):476–96.CrossRefGoogle Scholar
Sinclair, B. 2012. The Social Citizen: Peer Networks and Political Behavior. Chicago, IL: University of Chicago Press.CrossRefGoogle Scholar
Sokhey, A. E., and McClurg., S. D. 2012. “Social Networks and Correct Voting.” The Journal of Politics 74(3):751–64.CrossRefGoogle Scholar
Song, H., and Eveland, W. P. Jr. 2015. “The Structure of Communication Networks Matters: How Network Diversity, Centrality, and Context Influence Political Ambivalence, Participation, and Knowledge.” Political Communication 32(1):83108.CrossRefGoogle Scholar
Figure 0

Table 1. Distribution of network size across two measures

Figure 1

Figure 1. Distribution of network size across two measures.

Figure 2

Figure 2. Predicted discussion network size response varies by political interest for “politics” respondents but not for “social” respondents (95% confidence intervals).

Supplementary material: Link

Reilly and Belk Dataset

Link
Supplementary material: PDF

Reilly and Belk supplementary material

Reilly and Belk supplementary material 1

Download Reilly and Belk supplementary material(PDF)
PDF 27 KB
Supplementary material: PDF

Reilly and Belk supplementary material

Reilly and Belk supplementary material 2

Download Reilly and Belk supplementary material(PDF)
PDF 824 KB
Supplementary material: PDF

Reilly and Belk supplementary material

Reilly and Belk supplementary material 3

Download Reilly and Belk supplementary material(PDF)
PDF 36 KB
You have Access Open access

Save article to Kindle

To save this article to your Kindle, first ensure coreplatform@cambridge.org is added to your Approved Personal Document E-mail List under your Personal Document Settings on the Manage Your Content and Devices page of your Amazon account. Then enter the ‘name’ part of your Kindle email address below. Find out more about saving to your Kindle.

Note you can select to save to either the @free.kindle.com or @kindle.com variations. ‘@free.kindle.com’ emails are free but can only be saved to your device when it is connected to wi-fi. ‘@kindle.com’ emails can be delivered even when you are not connected to wi-fi, but note that service fees apply.

Find out more about the Kindle Personal Document Service.

Political and Social Discussion Network Survey Items Are Not Interchangeable
Available formats
×

Save article to Dropbox

To save this article to your Dropbox account, please select one or more formats and confirm that you agree to abide by our usage policies. If this is the first time you used this feature, you will be asked to authorise Cambridge Core to connect with your Dropbox account. Find out more about saving content to Dropbox.

Political and Social Discussion Network Survey Items Are Not Interchangeable
Available formats
×

Save article to Google Drive

To save this article to your Google Drive account, please select one or more formats and confirm that you agree to abide by our usage policies. If this is the first time you used this feature, you will be asked to authorise Cambridge Core to connect with your Google Drive account. Find out more about saving content to Google Drive.

Political and Social Discussion Network Survey Items Are Not Interchangeable
Available formats
×
×

Reply to: Submit a response

Please enter your response.

Your details

Please enter a valid email address.

Conflicting interests

Do you have any conflicting interests? *