Two Approaches to Measuring Gender Identities

We draw on the 2019 CES (N=37,822) and the 2021 CES (N=20,968) online surveys (Stephenson et al., Reference Stephenson, Harell, Rubenson and Loewen2020, Reference Stephenson, Harell, Rubenson and Loewen2022) for our analysis, capitalizing on the significant differences in measuring gender identity. Traditionally, gender was measured in surveys using binary definitions of male/female and was either assigned by the interviewer or included as a sex/gender demographic question with only two options (for a review, see Bittner and Goodyear-Grant Reference Bittner and Goodyear-Grant2017). These techniques are still used today. As we report elsewhere (Albaugh et al. Reference Albaugh, Harell, Loewen, Rubenson and Stephenson2024, 15), an analysis of the Comparative Study of Election Surveys (CSES) suggests that almost half of the surveys still include only binary gender options. Increasingly, however, surveys are including various forms of “Other” genders to capture the diversity of gender identities. The 2019 CES uses a three-option gender identity question that is becoming increasingly common to replace binary measures of gender identity in survey research. The 2019 CES question’s “Other” option explicitly lumps together trans, nonbinary, Two-Spirit, and other gender minority respondents into a category other than “Man” or “Woman.” At the beginning of the first wave of the 2019 CES online panel, respondents see the following mandatory gender question:

The “Other” category explicitly mentions trans, nonbinary, and Two-Spirit respondents.Footnote ² The only way to identify as trans on the 2019 CES is to select the “Other” response option, effectively forcing trans men and women to choose whether they want to prioritize their gender identity (as men or women) or their trans identity. Although many trans men and women may select the “Man” and “Woman” options (and, therefore, not be identifiably trans in the 2019 CES), we suspect that a substantial number of trans men and women selected the “Other” response in the 2019 CES. If so, the “Other” category would include some individuals who identify as men or women, which would inflate estimates of how many people are nonbinary (and deflate estimates of how many people are women and men). This conflation is likely to be present in other surveys, since many surveys include “transgender” as a separate response option alongside man and woman, and some scholars recommend this approach as a best practice (for example, National Academies of Medicine, Engineering and Science 2022). Analyzing the 2019 and 2021 CES provides an illustrative example of the extent of this mismeasurement given the change in question wording.

The 2021 CES presents respondents with a revised multiple-category gender identity question followed immediately by a transgender identity question:

Importantly, these questions appear on the same page so that trans men and women know when answering the gender identity question that they can also self-identify as transgender. This approach identifies nonbinary people as well as trans men and trans woman as compared to cis men and women, allowing researchers to explore the consequences of transgender and nonbinary identities for political behavior. However, as noted above, including an open-ended response option raises concerns that respondents will provide hostile or noncooperative responses, common in online surveys (Jaroszewski et al. Reference Jaroszewski, Lottridge, Haimson and Quehl2018). We recode open-ended responses into man, woman, nonbinary/another gender identity, or missing (see Appendix A). Our replication syntax are available through Harvard Dataverse (Albaugh et al. Reference Albaugh, Harell, Loewen, Rubenson and Stephenson2025).

Table 1 shows the gender identity distributions from the 2019 CES, the 2021 CES raw gender identity responses, and our recoding of the 2021 CES. Both the unweighted 2019 and 2021 CES over-represent women and underrepresent men relative to the 2016 Census. However, the 2019 and 2021 surveys differ substantially in how gender minority respondents answer the questions. Although 2021 had a larger share of transgender and nonbinary respondents than 2019 (1% vs. 0.8%), many of those respondents selected “Man” or “Woman” in the main gender identity question. As a result, the share of respondents counted as neither men nor women is much higher in 2019 (0.8) than in 2021 (0.5). We suspect that the 2019 question pushes many transgender men and women to select the “Other” response option so that they are visible as transgender to researchers, not because they do not identify as men or women. Among the 99 respondents who identified as neither men nor women in 2021 by selecting either “Non-binary” or “Another Gender, Please Specify:” 90 selected “Non-binary.” As a result, we suspect that nonbinary serves as a useful response option for most respondents who identify as neither men nor women.

Table 1 Gender Identity, 2019 and 2021 CES

To compare the consequences of these two approaches, we construct common variable coding across the two surveys for age, education, income, province/territory of residence, language of questionnaire, mother tongue, country of birth, citizenship status, religious identity, community size, party identification, vote intention, and attitudes toward lesbians and gay men. The 2019 and 2021 CES have different questions about racial and sexual identity, which we recoded into common categories (see online Appendix B).

The shared coding across the 2019 and 2021 CES allows us to conduct two analyses that illustrate the consequences of measurement in the two surveys. First, we compare the correlates of nonbinary identity across the two surveys to look for differences in the demographics of nonbinary respondents. Second, we pool the two surveys and use cross-survey multiple imputation to estimate the percentage of “Other” respondents in 2019 that are actually trans men or women. Researchers have previously used cross-survey multiple imputation to simulate how respondents would have responded to an alternative questionnaire (Eckman Reference Eckman2022).

Correlates of Nonbinary Identity the 2019 and 2021 CES

The 2019 CES question wording not only inflates the number of respondents who do not identify as men or women but also distorts the correlates of identifying as neither a man nor a woman. Figures 1–3 plot the estimated percentage of (1) 2019 “Other” respondents, (2) 2021 trans and nonbinary respondents, (3) 2021 nonbinary respondents only, and (4) 2021 trans men and women respondents only (pooled for sample size) within categories of age, sexual identity, and attitudes toward lesbians and gay men (LG). We estimate these percentages using the 2019 and 2021 CES raked campaign period weights and display Wilson confidence intervals to handle uncertainty around small percentages.

Figure 1 Estimated Percentage of Respondents by Age Category

Figure 1 shows that all four sets of respondents skew young. However, the 2021 nonbinary respondents are much more likely to be 18–29 than both the 2019 “Other” respondents and the 2021 trans men and women respondents. If anything, the 2019 “Other” respondents most resemble the 2021 trans men and women respondents rather than the 2021 nonbinary respondents. The 2019 “Other” respondents also have a higher share of people 65+ than any of the 2021 groups. One possible explanation for these results is that the 2019 “Other” respondents may include a substantial number of trans men and women as well as nonbinary people.

As an additional check, we run Kolmogorov–Smirnov tests to examine whether (a) the 2019 “Other” respondents and the 2021 nonbinary respondents and (b) the 2021 trans men and women respondents and the 2021 nonbinary respondents come from the same underlying distribution on age. The age distribution of the 2021 nonbinary respondents skews younger than the 2019 “Other” respondents (p<0.05). Similarly, the 2021 nonbinary respondents’ age distribution is younger than the 2021 trans men and women respondents (p<0.05). This result fits with evidence that nonbinary people are younger than trans men and women (Statistics Canada, 2022).

Figure 2 shows a similar plot for sexual identity. The 2019 “Other” respondents are almost half straight respondents, and the remaining respondents are evenly split among lesbian or gay, bisexual, or other sexual identities. (The other category includes substantial numbers of respondents who write in queer, pansexual, or asexual identities.) The 2021 trans and nonbinary respondents look similar, though fewer of them are lesbian or gay and more of them choose the other sexual identity option than the 2019 “Other” respondents. The 2021 trans men and women respondents are majority straight, though a substantial number report bisexual or other sexual identities.

Figure 2 Estimated Percentage of Respondents by Sexual Identity

Finally, the 2021 nonbinary respondents overwhelmingly choose “other” sexual identity, in line with past work showing that nonbinary people are overwhelmingly not straight (Bauer Reference Bauer2020). However, the sexual identity results among the 2019 “Other” respondents appear to overestimate the percentage of straight people if this measure is only capturing nonbinary respondents. Kolmogorov–Smirnov tests indicate that (a) the 2019 “Other” respondents and the 2021 nonbinary respondents and (b) the 2021 trans men and women respondents and the 2021 nonbinary respondents come from different underlying distributions (p<0.05 in both cases). Given that trans men and women are much more likely to identify as straight, we take these findings as evidence that the 2019 CES question pushes trans men and women to select the “Other” category, misidentifying them as nonbinary.

Finally, we compare attitudes toward lesbians and gay men by gender identity in each year. In figure 3, we plot each respondent’s feeling thermometer rating of “gays and lesbians” by gender identity and year using pairwise deletion to handle missing data, jittered to make it easier to see individual points with a horizontal line at the mean for each group. Although all four groups are generally positive toward gay men and lesbians, a two-sided t test indicates that the 2021 nonbinary respondents are substantially more positive toward lesbians and gay men than the 2019 “Other” respondents (d=9.3, p=0.03). This discrepancy can be explained by the bottom left panel in figure 3, which shows trans men and women provide lower feeling thermometer responses toward lesbians and gay men than nonbinary people in 2021 (d=-9.9, p=0.03). Overall, the results suggest that 2021 trans men, trans women, and nonbinary people are a better comparison group for the 2019 “Other” respondents than the 2021 nonbinary respondents alone.

Figure 3 Feeling Thermometer Toward “Gays and Lesbians”

Three discrepancies from these results are noteworthy: the 2019 “Other” respondents are older, more likely to be straight, and more negative toward lesbians and gay men than the 2021 nonbinary respondents. When we compare the 2021 trans men and women respondents and the 2021 nonbinary respondents, we find that trans men and women are older, more likely to be straight, and more negative toward lesbians and gay men than nonbinary people. As a result, we have strong reasons to suspect that some of the 2019 “Other” respondents are actually trans men and women who selected the “Other” response, especially since it explicitly mentioned trans people.

In Appendices D and E, we present additional evidence that the 2019 CES does not demographically capture nonbinary people in its “Other” category by comparing both the 2019 and 2021 CES with the Trans PULSE Canada nonbinary subsample (Bauer Reference Bauer2020) and the 2021 Canadian Census release (Statistics Canada 2022). The 2019 data have important discrepancies, while the 2021 data more closely mirror these other data sets.

Estimating the Share of Trans Men and Women in 2019 “Other” Category

Although we have shown that (a) the 2019 gender identity question inflates the “Other” category and (b) 2019 “Other” respondents differ from 2021 nonbinary respondents on age, sexual identity, and attitudes toward LGs, we do not know how many of the 2019 “Other” respondents are trans men and women rather than nonbinary people. We therefore estimate how extensively the 2019 CES overestimated the percentage of nonbinary respondents by constructing counterfactual estimates of how the 2019 “Other” respondents would have responded to the 2021 CES gender identity and trans identity questions if they had received them.

To do this, we treat the ambiguity in 2019 as a missing data problem in which some of the “Other” respondents are nonbinary and others are actually trans men or women. Restricting the data to 2019 “Other” respondents and 2021 trans or nonbinary respondents, we code the quantity of interest (“nonbinary”) as 0 for trans men and women and 1 for nonbinary people. This variable is present for all 2021 respondents and six 2019 respondents who identified as trans men, trans women, or nonbinary in open-ended responses (two trans men or women and four nonbinary respondents) but is missing for the remaining 2019 “Other” respondents. We address this missing data problem using multiple imputation with chained equations. We discuss the details in Appendix D. Our analysis suggests that about 55% of the 2019 “Other” respondents are trans men and women rather than nonbinary individuals. Even the lower bound of the estimates suggests that over 44% of the 2019 “Other” respondents are trans men or women rather than nonbinary people, which is a substantial amount of measurement error for a small group.

The 2019 question wording, by explicitly mentioning “Trans,” may be particularly likely to push trans men and women respondents to select “Other.” However, overestimation likely exists any time trans men and women are forced to choose between identifying as trans or identifying as men and women when confronted with mutually exclusive response options within a single question. Researchers who wish to include trans and/or nonbinary communities in survey research should adopt question formats which make it possible for respondents to be both trans and a man or a woman, as well as neither a man nor a woman.

Examining the 2021 Open-Ended Responses

Our final question is practical: does the revised 2021 question wording lead to more noncompliance and/or withdrawal from the survey? We find no evidence of this. Only 16 respondents dropped out before completing these two questions, versus 21 for the following question on province of residence. Further evidence of a limited reactivity effect is evident in the open-ended gender identity responses. We recode the 33 open-ended responses to the 2021 gender identity question into other categories where possible (see Appendix A.) Of the 33, 14 provide binary gender responses (usually male or female rather than man or woman, sometimes with political commentary on the “political correctness” of asking about gender rather than sex), nine provide recognizable gender identity responses other than man or woman (e.g., “genderfluid,” “agender,” or “bigender”), and 10 provide noncooperative responses that do not allow for coding. We recode the first group as men or women depending on their responses, the second group as nonbinary, and the third group as missing. Very few valid respondents provide noncooperative gender identity responses, which suggests limited risks associated with including an open-ended category.

Importantly, these open-ended responses suggest that researchers should not simply recode respondents who select “Another gender, please specify:” as nonbinary or as a fourth category (people who do not identify as men, women, or nonbinary) as over 40% of the responses indicate a usable binary sex/gender response. Researchers need to prioritize reviewing these open-ended response categories to avoid false positives—taking men and women as if they are neither. However, if researchers exclude the “Another gender, please specify” category entirely, they will lose responses that indicate identities not strictly man or woman, such as agender, bigender, or gender-fluid, and may code as false negatives respondents who do typically fall under the nonbinary umbrella. This can lead to even smaller nonbinary subsamples that are more difficult to include in statistical analyses. In general, our results suggest the 2021 question wordings do not create a substantial proportion of missing data or high levels of noncooperative responses. Including an open-ended write-in category also provides options for a wider variety of gender identities with minimal costs.

Implications for Studying LGBTQ+ Identities and Political Behavior

Our analyses suggest that the 2021 CES made improvements in measuring gender over the 2019 CES, as the 2021 questions appear to have substantially reduced measurement error among trans and nonbinary respondents. There is considerable evidence that many—if not most—2019 respondents in the “Other” category may actually be men and women. The 2021 nonbinary subsample also looks much more plausible on primary demographic variables when compared with recent data on nonbinary people in Canada. Furthermore, the 2021 open-ended gender identity questions did not generate many respondents who provide hostile or noncooperative responses. Providing explicit nonbinary response options and separate transgender identity questions that make it easy for nonbinary and trans individuals to identify themselves is therefore good practice for robust measurement of gender identity. As surveys increasingly move away from binary gender questions, our analysis provides evidence that capturing nonbinary and trans individuals requires allowing people to express their identities in ways that do not make trans identity exclusive from gender identities and requires ensuring that questions leave space for the diversity of nonbinary and other gender identities which people adopt.

We recommend that other large-sample election surveys consider adopting similar two-step questions to identify trans and nonbinary respondents. One of the strengths of large-sample election surveys is that they are useful for measuring attitudes and behaviors among small groups within the population (Stephenson et al. Reference Stephenson, Harell, Rubenson and Loewen2021). Large-sample surveys should, therefore, pay particular attention to making sure their measurement captures small groups especially when these groups are theoretically and substantively important, as is the case for trans, nonbinary, and Two-Spirit people. We can only have an incomplete understanding of gender and political behavior if we do not measure these identities. As we have demonstrated elsewhere (Albaugh et al. Reference Albaugh, Harell, Loewen, Rubenson and Stephenson2024), accurately capturing nonbinary and trans respondents allows us to document important, large gaps in political preferences and behavior, such as the tendency for nonbinary people to prefer and vote for social democratic parties, even among LQBTQ respondents. This is only possible when we provide question formats that allow us to capture and analyse diverse gender communities in our surveys.

Transgender and nonbinary people are important to expanding our understanding of LGBTQ2S+ political behavior beyond lesbians, gay men, and bisexual people. Ultimately, however, these groups are important to study in their own right and from an ethical point of view deserve to be treated with dignity by including more affirming language in survey research. As our study has shown, there are efficient updates that can be made to common gender questions to provide both respectful and reliable measures of these identities in survey research.