NORMATIVE CONCERNS: ISSUES CAPTURING THE COMPLEXITIES OF POLITICAL LIFE VIA SURVEYS

Surveys predominate in political science research because they are helpful for researchers seeking to gather broad insights about diverse populations. They also are cost-effective and efficient. For these reasons, many quantitative researchers rely on surveys to answer a host of complex research questions. In 2000, Brady (Reference Brady2000, 47) wrote, “No other method for understanding politics is used more, and no other method has so consistently illuminated political science theories with political facts.” However, their predominance has not yet yielded the types of thickness of description and complexity that many qualitative methods produce. What are some of the limitations of surveys? How have survey researchers navigated these concerns? I turn to survey researchers themselves to answer these questions. Many of the canonical studies of identity in political science have relied on quantitative approaches. They also frequently provide critiques of this method. For example, Herbst (Reference Herbst1993, 2; original italics) noted that many researchers rely on surveys due to their “objective” nature “because they seem unprejudiced by ideology and are designed to communicate the general will.” As such, many political scientists have not sought to challenge survey methods and their centrality in the discipline. Zaller (Reference Zaller1992, 32) noted that one of the ways that surveys fail to capture the intricacies of public opinion is through “systematic error,” which often is introduced in the context of the survey itself. Effectively, people’s answers in survey interviews are influenced highly by seemingly innocuous artifacts of the survey environment, such as question order, type, and format. This limitation represents a fundamental problem with surveys because it cloaks the true cause of the potential errors embedded in survey methodology.

In addition to concerns about systematic error, the overall absence of marginalized voices from survey research should be a major concern for political scientists. Structurally, polling via landlines, cell phones, and Internet-based methods all come with risks of missing key portions of the population and leaving out groups that are distributed systematically by race and age (Berinsky Reference Berinsky2017; Walker Reference Walker2020). This disproportionate exclusion of racial minorities, younger Americans, and other vulnerable people misrepresents findings not only for marginalized people but also for white Americans and other normative groups. In their analysis of the 2016 presidential campaign and the role of identity in shaping people’s political attitudes, Sides, Tesler, and Vavreck (Reference Sides, Tesler and Vavreck2018) found that incorporating “racialized politics” in survey research reveals more complex dimensions of identity politics that are not clear when only examining white public opinion. They wrote, “[D]rawing attention to the country’s changing demographics in a survey significantly increased the percentage of whites who identified as Republicans” (Sides, Tesler, and Vavreck Reference Sides, Tesler and Vavreck2018, 32). Not only that, increased contact with racial minorities also shaped conservative political behavior among white Americans (Sides, Tesler, and Vavreck Reference Sides, Tesler and Vavreck2018). Their study, although primarily focused on the mass opinions of white Americans, shows that when surveys are expanded to include the voices and experiences of marginalized people, the findings are more accurate and depthful (Jackson Reference JacksonN.D.).

Recent political science surveys have sought to address the lack of comprehensive representation of marginalized voices in quantitative research. In particular, the Collaborative Multiracial Post-Election Survey (CMPS)—developed at the University of California, Los Angeles—was established in 2008 as a post-presidential election survey in the United States. Whereas the CMPS focuses on traditional groups such as white, Black, Asian, and Latinx respondents, it also features oversamples of marginalized people and a subsample of youth respondents. CMPS researchers also have focused on recruiting what they refer to as “hard-to-reach” groups including “Afro-Latinos, Black immigrants, Native Americans, Native Hawaiians, Muslims, and people who identify as LGBTQ.” In 2024, the CMPS also featured a youth sample of 2,000 16- and 17-year-olds. To address concerns about access, it was available in English, Spanish, Chinese, Korean, Vietnamese, Arabic, Urdu, Farsi, and Haitian Creole. Thus, it represents an important corrective to long-standing national studies such as the American National Election Studies (which oversamples Blacks and Latinos but overrepresents older and highly educated Americans) and the General Social Survey (which also overrepresents older Americans and underrepresents middle-class Americans) (Simmons and Bobo Reference Simmons and Bobo2015). An important aspect of the CMPS data is that the survey questions are collected from researchers themselves. This is a helpful model for ensuring that surveys are reflective of the diverse research agendas and subjects in which we engage. However, the selected questions often continue to rely on the conceptual frameworks of the broader discipline rather than on the language, narratives, and firsthand experiences of respondents. Whereas intentional oversamples of minoritized communities help to mitigate the limitations introduced with this method (Frasure et al. Reference Frasure, Wong, Vargas and Barreto2025), there remains additional depth that can be obtained only through direct engagement with impacted communities. This is especially important when considering the unique political lives of multiply marginalized people whose intersectional identities often traverse many categories of description and analysis. One recent and significant limitation of the CMPS was the COVID-19 pandemic. For example, Smith and Greer (Reference Smith and Greer2025, 328) explained that “Ideally, an oversample for hard-to-reach groups would include on-the-ground and face-to-face interviews in partnership with non-profit organizations. This simply was not possible at the time.” Therefore, the demographic interventions of the CMPS are critical in ensuring that surveys are more representative of the true nature and experiences of the broader population. However, these surveys cannot fully account for the ways that varying identities intersect and overlap in US politics. These challenges are inherent in the survey instrument itself.

ONTOLOGICAL CONCERNS: PROBLEMS WITH THE VITALITY OF “PUBLIC OPINION”

A primary concern in the proliferation of large-N survey research in political science is that these methods, although allowing for the collection of a breadth of data, do not make space for the fullness of how and why individuals develop their attitudes and opinions. A primary ontological concern with surveys is that they provide only a snapshot of the attitudes and perspectives of respondents. In her effort to tell complete stories about rural consciousness, Cramer (Reference Cramer2016, 20) wrote, “I find mass-sample public opinion surveys enormously helpful for capturing what a large population of people think at a given point in time. But for the task of figuring out why people think what they do, I have found no better substitute than listening to them in depth” (emphasis original). Cramer (Reference Cramer2016) underscored that large-N surveys work well in the aggregate specifically because they capture mass opinion for large swaths of people at a given time. Yet, we may know little about how they came to hold those beliefs and attitudes. Even when relying on longitudinal data, political scientists run the risk of making biased inferences due to attrition (Frankel and Hillygus Reference Frankel and Hillygus2014; Olson and Witt Reference Olson and Witt2011). In general, surveys are helpful but not sufficient in ascertaining the full stories of the lived experiences and attitudes of our sample populations, especially multiply marginalized people.

In her multimethodological study of criminal justice contact in Black and Brown communities, Walker (Reference Walker2020, 17) noted, “[S]urveys introduce social desirability bias, where respondents may be less likely to report personal and proximal experiences with criminal justice out of shame or embarrassment.” Her concerns reflect the broader issues with surveys in that it is impossible to verify responses, especially when respondents intentionally mislead survey interviewers (Walker Reference Walker2020). In addition to concerns about the veracity of survey data, there are issues with the timeliness and persistence of the results. Walker (Reference Walker2020, 17) wrote, “[T]he use of cross-sectional survey data, which captures a snapshot in time rather than change over time, prevents a causal analysis absent an exogenous treatment.” In many ways, surveys alone—especially the cross-sectional variant—cannot tell the full story of the experiences of marginalized people because they lack the context and framing that come from in-depth analysis. These ontological concerns illustrate how—even for survey researchers—there are limitations that structurally bias their findings and truncate the applicability of their inferences. When considering research that centers multiply marginalized people, these biases may misrepresent and understate the impacts of structural inequality and other societal problems they face.

CONCEPTUAL CONCERNS: CONSTRUCT VALIDITY IN LARGE-N DATA

Conceptual definitions are critical in the study of political science. “Concepts in political science are a precursor to their application or measurement in analysis” (Holmes et al. Reference Holmes, Guliford, Mendoza-Davé and Jurkovich2024, 52). In many large-N studies, concepts are expected to have a general, commonly understood meaning. One example that emerged from my own research is the term “police contact.” For many people, police contact might mean being pulled over for a broken taillight or speeding through a red light. However, for many young Black Americans, police contact refers to a host of unwanted bodily searches, experiences of harassment, arrests, or confrontations with loaded weapons (Jackson Reference Jacksonforthcoming). For different racial, gender, class, and sexuality groups, their experience with policing varies drastically based on their identity and orientation to power (Jackson Reference Jacksonforthcoming). Black feminist scholarship points to this failure of conceptual clarity in other instances. For example, in her study of Black women who were convicted felons and their reentrance experiences, Middlemass (Reference Middlemass, Jordan-Zachery and Alexander-Floyd2018, 74) found that “[C]riminology and the study of crime tend to be dominated by quantitative methods that do not adequately capture reality. The prevailing methodological approaches make many women invisible in reentry discourse, and Black women are further marginalized because criminal justice scholars tend to focus on race, gender, and criminal involvement as distinct characteristics and separate issues when in fact they are not autonomous.” Middlemass (Reference Middlemass, Jordan-Zachery and Alexander-Floyd2018) was helpful in showing how there is an inherent misalignment between the methodological approaches that political scientists employ and the lived experiences of vulnerable people, especially Black women. In these instances, it is critical to extend survey methods to ameliorate concerns that conceptual meanings are lost in the survey process.

Black politics and feminist scholars have illustrated the complex nature of conceptual development with regard to multiply marginalized populations. In her study of Black women legislators, Brown (Reference Brown2014) emphasized how lived experiences and feminist histories are just as informative of legislative decision making and policy decisions as race, gender, and class background. Furthermore, her findings demonstrate the ways that terms such as “Black” and “women” may function as rigid “containers” wherein social scientists may fail to investigate the differences not only across race and gender groups but also among them (Brown Reference Brown2014). Michener, SoRelle, and Thurston (Reference Michener, SoRelle and Thurston2022) referred to this type of study as starting at the “bottom,” from communities that are impacted most directly by the concerns at the center of our work, rather than a top-down analysis that originates from political science theories and logics. These bottom-up approaches more accurately capture the fluidity and shifts in salience that occur when race, gender, class, and sexuality are introduced into our research.

Concepts also have origin stories. When studying marginalized populations, it is important that political scientists acknowledge how our general theorizations and models are rooted in whiteness as a normative basis (Zuberi and Bonilla-Silva Reference Zuberi and Bonilla-Silva2008). The centering of the experiences of white Americans, often making them the comparison group (hinting at their supposed neutrality), has framed the discipline around white epistemological logics of knowledge production and a biased understanding of what “counts” as political (Herbst Reference Herbst1993; Zuberi and Bonilla-Silva Reference Zuberi and Bonilla-Silva2008). This rootedness in whiteness as a normative identity also shapes the meanings of terms and concepts that lie at the heart of our study. As Entman and Rojecki (Reference Entman and Rojecki2000, 22–23) wrote in their multimethodological study of white Americans’ attitudes about Black Americans:

When terms that have long been associated with white ways of being and knowing become systematized in longitudinal studies, those conceptualizations also become canonical ways of knowing within the discipline. This represents an epistemological as well as a conceptual concern.

DISCUSSION: METHODOLOGICAL RECOMMENDATIONS FOR SURVEY METHODS

It is important to note that this article extends the use of survey methods rather than eradicating them wholesale. As such, I suggest three recommendations for survey research that might assist in telling truer, more comprehensive stories about multiply marginalized people. One approach that may assist with addressing the limitations of survey methods is the adoption of feminist research techniques that have long been rooted in understanding the nature and contexts of political life rather than seeking one best answer. In a general sense, “feminist methodology can help researchers and respondents to generate a unique context in which not only are women’s lives studied, but through the research process, they may unlearn silences, prejudices, and fears of conflict” (Bloom Reference Bloom1997, 119). In political science, there have been efforts to bring Black Feminist research methods into survey research. For example, in her empirical study of Black Feminist consciousness, Simien (Reference Simien2006, 6) wrote, “I wish to show how the omission of black feminist voices causes survey researchers to ask the wrong questions and base their empirical work and conclusions on uninterrogated assumptions—that, for instance, all of the women are white and all of the blacks are men.” The incorporation of feminist theories into survey methods encourages the expansion of conceptual definitions as well as the extension of structural constructs that political scientists study. For example, one area of challenge for political scientists that might be addressed by feminist research methods is the design of questions. For many quantitative researchers, survey questions materialize directly from theoretical claims, variables of interest, measurement concerns, and expected outcomes. Feminist research methods posit that survey questions should come from dialogue and conceptual definitions that emerge from multiply marginalized communities. Rather than rewrite questions that have long dominated political science theory and quantitative study, researchers should take the opportunity to discover new terms and frameworks based on direct observation. These methods help to dismantle existing epistemologies that limit the issues demanding our study.

The second approach is the incorporation of research agendas that combine both qualitative methods and surveys. Qualitative methods including focus groups, in-depth interviews, case studies, and participant observations allow for the development of conceptual meaning. It is important to note that these methods originate from differing epistemological and ontological origins than survey research. Rather than relying on deduction—the process of moving from general theoretical claims to specific conclusions—qualitative methods inductively move through observational data to locate patterns and frameworks that might emerge. These qualitative methods rely on the “principle of hermeneutics, or meaning making, [that] details the idea that the whole must be understood in relation to the parts, which derive their meanings from our understanding of the whole—an interpretation that then furthers our grasp of the meanings of the parts, and so on” (Josselson Reference Josselson2013, 6). Although there are concerns about selection bias and sampling with qualitative methods, there are several approaches that address them (Boas Reference Boas2024). Cyr (Reference Cyr2017, 1038) explained that focus groups are uniquely helpful to political scientists because they simultaneously offer three levels of analysis: individual, group, and interactive. These strengths are not hampered by interviewer bias or involvement because they are “emic data” that are captured with little interference from the researcher (Cyr Reference Cyr2017). As a uniquely comprehensive approach, Cyr (Reference Cyr2017, 1038) noted, “the distinct types of data that focus groups generate allow researchers considerable flexibility regarding how they use the data produced.” These analytical strengths are unique to qualitative and interpretive methods. White (Reference White2018, 26) explained in her analysis of the branding of right-wing activism that “case studies are useful for holistic and in-depth investigations, particularly because they use multiple sources of data and produce findings with real-world anchoring.” Relying on multiple research methods extends the use of surveys so that the political experiences and attitudes of marginalized people are not truncated by the available response options and structural frameworks of the methods.

There are many ways to develop mixed-methods research models to yield more accurate findings. In her multimethodological study of Black immigrants and labor-union members in New York City, Greer (Reference Greer2013) relied on survey data of panethnic identity and union membership that she complemented with in-depth interviews. Although she focused primarily on the quantitative aspects of her analysis, she recognized that conversations with multi-ethnic Black Americans would reveal rich contexts and precision not available through surveys alone. As Walker (Reference Walker2020, 19) stated, “[A] variety of techniques help verify findings. There is, however, no replacement for the voices of those directly impacted by the criminal justice system….Their voices demand that we pay attention, that we not look away, and they instruct us about where we should go next as researchers, policy makers, and democratic citizens.” This process of verification is what some political scientists call “methodological triangulation,” wherein multiple methods are used to study the same phenomenon (Tzagkarakis and Kritas Reference Tzagkarakis and Kritas2023). Fundamentally, there is no one best way to engage in mixed methods. It is critical only that political scientists have many methods at the ready when posing research questions, especially those that concern multiply marginalized people.

A third approach that allows for both breadth and depth in the study of marginalized groups is based on institutional change. Political scientists should be more intentional about providing qualitative methods as required courses for graduate students. Although many political science departments have mandatory quantitative methods courses on their rosters, this has yet to extend to other methods. In their study of the use of alternative methods in political science, Bennett, Barth, and Rutherford (Reference Bennett, Barth and Rutherford2003, 373) wrote, “All of the top 30 departments surveyed offered courses in statistical methods, while only two thirds of the departments offered courses in formal modeling and qualitative methods. Two thirds of the departments required students to take statistics, while formal modeling and qualitative methods were required in only two departments.” The premium placed on the use of survey methods has produced generations of political science scholars who may not be prepared methodologically to embark on multimethodological research. However, mixed-methods research is necessary to answer our most pressing questions about the political lives of multiply marginalized people. Not only that, these failures of qualitative-methods training shape the ways that scholars across the methodological paradigm relate to one another’s work. In American politics, there is an overrepresentation of survey-based research in our top journals that reflects the discipline’s distorted focus (Bennett, Barth, and Rutherford Reference Bennett, Barth and Rutherford2003). In their study of graduate qualitative-methods courses in political science, Emmons and Moravcsik (Reference Emmons and Moravcsik2020) found that the emphasis on qualitative and mixed methods has been on the decline. “Only 60% of the top political science departments offer any focused training in qualitative methods—a decrease during the past 10 to 15 years” (Emmons and Moravcsik Reference Emmons and Moravcsik2020). They referred to this dearth of qualitative-methods training as a “disciplinary crisis,” stating, “[W]e conclude that most essential qualitative research techniques cannot be acquired in traditional research design courses” (Emmons and Moravcsik Reference Emmons and Moravcsik2020, 261). Given the shifting political terrain and the growth of marginalized groups in the United States, it is important that political scientists consider institutional change among the recommendations to address the limitations of survey methods. The more our departments invest in training scholars to perform research via many diverse and complementary methods, the more robust and accurate our findings collectively will become. If political science departments take seriously the training of graduate students in qualitative-methods and if journals encourage the publication of multimethodological studies on an ongoing basis, this pluralist approach will become a reality.