2 Research on Gendered Citation Patterns

We must understand two related issues to detect potential gendered patterns in scholarly citations. First, if male scholars publish more journal articles or books than female scholars, a pattern typically observed in many disciplines, this creates greater baseline opportunities for men’s work to capture citations. Second, citations are embedded in academic networks, making it important to examine who is citing whom and whether certain types of scholars are more likely to be cited. In other words, overall citation counts (or counts per article) miss the sociological aspects of how scholars recognize the work of their peers.

We start with a basic question about gendered variance in citations: do women faculty receive fewer citations to their journal articles than male faculty? Several disciplines produce evidence of lower citation levels for female scholars compared to male scholars in the same field including economics (Ferber Reference Ferber1988; Ferber and Brün Reference Ferber and Brün2011), ecology (Cameron, White, and Gray Reference Cameron, White and Gray2016), political science (Dion and Mitchell Reference Dion and Mitchell2012; Maliniak, Powers, and Walter Reference Maliniak, Powers and Walter2013; Mitchell, Lange, and Brus Reference Mitchell, Lange and Brus2013; Roberts, Stewart, and Nielsen Reference Roberts, Stewart and Nielsen2017), library and information sciences (Håkanson Reference Håkanson2005), linguistics and sociology (Leahey, Crockett, and Hunter Reference Leahey, Crockett and Hunter2008), and health and natural sciences (Aksnes et al. Reference Aksnes, Rorstad, Piro and Sivertsen2011; Beaudry and Larivière Reference Beaudry and Larivière2016). This is often described as a “Matthew effect” whereby men as the predominant publishers in a field receive more citations, stature, and influence in their disciplines (Rossiter Reference Rossiter1993; Reece-Evans Reference Reece-Evans2010: 13). Analyses of citation data in other disciplines suggest either that there are no differences between male and female faculty in terms of citations (total or per article) [public administration (Corley and Sabharwal Reference Corley and Sabharwal2010), sociology (Ward, Gast, and Grant Reference Ward, Gast and Grant1992), international relations (Østby et al. Reference Østby, Strand, Nordås and Gleditsch2013), economic history (Di Vaio, Waldenström, and Weisdorf Reference Di Vaio, Waldenström and Weisdorf2012), Spanish science PhDs (Borrego et al. Reference Borrego, Barrios, Villarroya and Ollé2010), criminal justice (Stack Reference Stack2002), forestry and geography (Slyder et al. Reference Slyder, Stein, Sams, Walker, Beale, Feldhaus and Copenheaver2011), in a sample of Danish scientists (Nielsen Reference Nielsen2016)] or that female scholars receive more citations per article than their male peers [biochemistry (Long Reference Long1992), construction studies (Powell et al. Reference Powell, Hassan, Dainty and Carter2009)]. Where gender citation gaps are found, research typically shows that men’s willingness to cite their own work at higher rates accounts for some of these observed gaps (Hutson Reference Hutson2006; Maliniak, Powers, and Walter Reference Maliniak, Powers and Walter2013; Ghiasi, Larivière, and Sugimoto Reference Ghiasi, Larivière, Sugimoto, Rafols, Molas-Gallart, Castro-Martínez and Woolley2016).

Citation counts can also be problematic because they can be heavily affected by outliers, they can increase significantly over scholars’ careers, and they tend to reward scholars who publish a high quantity of articles. Long’s (Reference Long1992) analysis of productivity and citations in biochemistry raised this issue, pointing to the potential gendered bias that could emerge if female scholars publish fewer articles on average than their male peers, yet have higher citations per article. The h-index was created to address some of these problems, capturing the total number of articles that have accrued that number of citations.Footnote ⁶ Yet even the h-index came under scrutiny for being biased against female faculty (Symonds et al. Reference Symonds, Gemmell, Braisher, Gorringe and Elgar2006; Nosek et al. Reference Nosek, Graham, Lindner, Kesebir, Hawkins, Hahn and Schmidt2010; Geraci, Balsis, and Busch Reference Geraci, Balsis and Busch2015). Self-citations can increase the h-index score, behavior that is more likely for male scholars (Fowler and Aksnes Reference Fowler and Aksnes2007; King et al. Reference King, Bergstrom, Correll, Jacquet and West2016). Hopkins et al. (Reference Hopkins, Jawitz, McCarty, Goldman and Basu2013) found that women in biochemistry, water resources, economics, and anthropology had h-indices that were one to four points smaller on average than their male peers. In a study of nearly 1700 social work faculty, a field where 65% of faculty are women, Carter, Smith, and Osteen (Reference Carter, Smith and Osteen2017) showed that men had 3.86 higher h-index scores than women across all ranks, with a larger gap at the full professor rank. Cameron, White, and Gray (Reference Cameron, White and Gray2016) observe in ecology, where women have earned more than 40% of PhDs since 1993, that men have higher h-indices and cite their work more often than women.

Most of these studies seek to account for the variance in scholars’ citation scores by examining the output over a scholarly career and citation patterns over time. Yet while the aggregate differences in citation counts or h-indices tells us that potential gender biases exist, it is difficult to know if this occurs simply because men publish and cite themselves more than women or if scholars systematically fail to cite relevant work by women in their field (or both). In contrast to the “Matthew effect” that raises the stature and influence of research by male scholars, the “Matilda effect” is very pernicious because this dynamic omits work by women from articles,Footnote ⁷ textbooks (Cassese, Bos, and Duncan Reference Cassese, Bos and Duncan2012), and course syllabi (Hardt et al. Reference Hardt, Kim, Meister and Smith2017; Colgan Reference Colgan2017). Knobloch-Westerwick, Glynn, and Huge (Reference Knobloch-Westerwick, Glynn and Huge2013) tested for the “Matilda effect” by randomly assigning male or female names from the U.S. census database to abstracts that were submitted to a national communication studies conference; 243 graduate students in the field assessed the scientific quality of the abstracts. They found that abstracts randomly assigned male authors were scored higher on the quality scale by both male and female graduate students. Furthermore, if the topic of the abstract was a more male-centric topic, the work was perceived to have greater scientific quality. Much of the variance in these research quality scores could be explained by the participants’ underlying gender norms. These kinds of implicit biases can influence the value that scholars place on academic research and hence influence citation practices.

Ferber, an economist, investigated these sociological patterns by focusing on who cites whom in academic research. Her 1986 study examined citations in the Journal of Economic Literature and discovered that scholars were more likely to cite research by scholars that share their biological sex. To see if these findings generalized beyond labor economics, another article (Ferber Reference Ferber1988) expanded the analysis to multiple journals across several academic disciplines, including U.S. and Canadian academic journals in economics, mathematics, psychology, and sociology. Articles published by one or more women were combined with a random sample of articles by authors based on publication lists in the Journal of Economic Literature, Mathematical Abstracts, Psychological Abstracts, and Sociological Abstracts to construct the percentage of citations to work by male and female scholars by discipline. This expanded dataset confirms Ferber’s earlier findings that male scholars are more likely to cite the work of other male scholars, while female scholars also cite the work of female peers more frequently. Among male authors, the gender gap ranged from 15% in development psychology to less than 5% in mathematics. For example, in development psychology, 67% of the citations in articles written by men are to research by men, while 52% of citations in female authored articles cite research by men, generating a 15% gap. Yet in fields in which the gap is small, men and women authors cite mostly male authors. In mathematics, 98% of works cited by men are written by men, while 94% of works cited by female authors are written by men. In development psychology and sociology, on the other hand, Ferber found that research by women constituted 7.5–18.1% of male authors’ citations and 15.8–29.1% of female authors’ citations. Given the relatively larger percentage of female scholars in psychology and sociology,Footnote ⁸ both male and female scholars’ decisions about what work to cite are influenced by the overall representation of women in the discipline. “The ‘citations gap’ appears to decrease as the proportions of women in the field and of articles written by women increase, in part because men are then more likely to cite work by women (Ferber Reference Ferber1988: 86).Footnote ⁹

Mitchell, Lange, and Brus (Reference Mitchell, Lange and Brus2013) utilize Ferber’s approach to match the biological sex of each author with the sex of each author cited in the bibliography entries ( $N=\text{3,414}$ ) for 57 articles published in International Studies Quarterly (ISQ) and International Studies Perspectives (ISP) in 2005. The analyses show that the proportion of citations of women’s research is three times higher for female authors compared to male authors in ISQ and twice as large for female authors in ISP. Dion and Mitchell (Reference Dion and Mitchell2012) conduct similar analyses for 2005 journal issues for Conflict Management and Peace Science (CMPS), Journal of Conflict Resolution (JCR), and Political Analysis. Female authors in CMPS are three times more likely to cite the work of other female scholars in comparison to male authors, an effect that is very close to what Mitchell, Lange, and Brus (Reference Mitchell, Lange and Brus2013) reported for ISQ.Footnote ¹⁰ In Political Analysis, the citation gender gap is even larger with only 3% of total citations in male authored papers referencing research by female authors. This compares to 10% of citations by female authors to the research of other female authors. These analyses demonstrate that female political scientists reference more research by women in the discipline than their male colleagues and that such patterns are especially likely in areas of political science that are populated heavily by male scholars.

While a variety of factors could explain the gender gap in academic citations, in fields where women make up a small minority of the profession, their work may simply be less visible relative to the majority group. Women might need to reach a critical mass before a systemic change in citation patterns can occur. Ferber and Brun’s (Reference Ferber and Brün2011) study suggests that the gender gap in citations has decreased, especially in the field of labor economics, where women constitute close to one third of scholars in the field. In the general field of economics, where women are still a relatively small group, however, the gender gap in citations is larger. Our review of citation studies across disciplines shows some fields where the gap is large (e.g. political science, library and information sciences, natural sciences), others where it is negligible (e.g. public administration, sociology, economic history), and others where women are advantaged (e.g. biochemistry).Footnote ¹¹ Fields where the gender gap has been reduced tend to be populated by higher percentages of female scholars, although some female-dominated disciplines like social work still witness higher citation rates for male scholars. This shows that while the “Matthew effect” can be attenuated as more women scholars populate an academic discipline, the “Matilda effect” can produce a persistent gender gap in citations.Footnote ¹² Therefore, we include in our sample publications from political science subfields with high proportions of men (political methodology) and women (politics and gender) as well as the top methodological publications in sociology, political science, and economics, all social sciences with different gender distributions. In the social sciences, a preoccupation with research methodologies that began with behavioralism in the 1950s continues to shape the development of disciplinary practices and research. Even those critical of political science’s trend toward specialization acknowledge that scholars using sophisticated methodologies serve as the discipline’s gatekeepers and trend setters (Mead Reference Mead2010, 454). By using a larger number of journal articles than previous studies and probabilistic methods for identifying authors’ gender (Teele and Thelen Reference Teele and Thelen2017; Sumner Reference Sumner2018), we shed light on how the representation of women in research areas influences gendered citation patterns.

3 Data and Analyses

We analyze a dataset of article references for all articles published between 2007 and 2016 in three political science journals and three social science methodology journals.Footnote ¹³ These comparisons seek to maximize variation in two ways. First, we compare citation patterns within political science, focusing on the highest-ranked flagship research journal of the American Political Science Association (APSA, American Political Science Review, APSR) and the journals (Political Analysis, PA and Politics & Gender, P&G) sponsored by APSA-affiliated organizations with the lowest and highest levels of gender representation (the Society for Political Methodology [aka, Polmeth] and Women and Politics organized APSA section). As of August 2017, the Political Methodology section (which is a subset of Polmeth membership) had 472 members of whom 79.7% self-identified as male, while the Women and Politics section had 330 members of whom 91.7% self-identified as female (APSA 2017).Footnote ¹⁴ Of the 213 APSA members who identified methodology as their primary field of study, 83.57% also self-identified as male (APSA 2017).

Polmeth governs PA and the APSA Women and Politics section governs Politics & Gender. While the APSR has a strong history of being one of the most cited and leading journals in the discipline, PA and P&G are well-cited and highly ranked journals in their respective fields (see Appendix for impact metrics). The editorial leadership of these journals tracks closely with those of the discipline or field. The APSR has not had any female editors between 2007 and 2017, though Associate Editors or Co-Editors (3–8 per year) have been between 25% and 67% female (see detailed tables in Appendix). Between 2007 and 2016, the Editorial Board was 36–43% female, similar to overall APSA membership. In 2017, APSR had a male editor, six Associate Editors of whom only 33% were female, and 78 Editorial Board members of whom 67% were female. Since 2007, PA has only had one female Editor who was part of a three-member editorial team in the one year in which there were no Associate Editors. Typically, one of the three (or four) Associate Editors is a female scholar, which is greater than the percentage of female methodologists according to APSA section membership or primary field. The PA Editorial Board ranges between 16 and 28% female, which is more consistent with membership or field identification. In contrast to APSR and PA, P&G is led by female scholars, including all Editors, 75–100% of Associate Editors, and 86–89% of Editorial Board members. The distribution of editorial roles by gender in these political science journals is consistent with gender distributions in organizational membership and self-identified primary field of study. This variation, including a flagship research journal as well as the journals associated with the two least gender diverse subfields (one dominated by men, the other by women), will help us evaluate the hypothesis that overall citations of female scholarship will be higher and gendered citation gaps lower in subfields with more female scholars.

Given that women constitute a clear critical mass in the politics and gender area, we expect female, male, and mixed gender authors to be similarly likely to cite work by women in Politics & Gender. Political methodology is the APSA section with the highest proportion of male members and thus we expect to confirm earlier analyses of Political Analysis, which show a clear gender gap in citations (Dion and Mitchell Reference Dion and Mitchell2012). We anticipate that APSR will fall somewhere in the middle, given that it represents political science research broadly; 37.5% of APSA members self-identify as female (APSA 2017). Such journals are charged with publishing works from all fields of political science but they often fail to publish research in topical areas or methodological approaches associated with higher concentrations of female scholars (Teele and Thelen Reference Teele and Thelen2017, 435–440).

Second, we compare citation patterns across the top-ranked methodology journals in three social sciences that share many research methodologies but experience different disciplinary gender distributions: economics (Econometrica), political science (PA), and sociology (Sociological Methods & Research, SMR).Footnote ¹⁵ While methodological subfields are likely to be male-dominated in general, the gender gap is likely smaller in disciplines with greater gender diversity, such as sociology. In 2013, only 24% of economists were female, compared to 32% of political scientists and 52% of sociologists (Hur et al. Reference Hur, Andalib, Maurer, Hawley and Ghaffarzadegan2017).Footnote ¹⁶ The gender distribution within disciplines is also reflected in the gender distributions of editorial team membership across the methodology journals. Although both Econometrica and SMR have had male Editors or Co-Editors between 2007 and 2017, the larger Editorial Board (or Associate Editors, as they are named in Econometrica) is about 7–21% female in SMR and 4–16% female in Econometrica (see Appendix, Table A5).Footnote ¹⁷ Therefore, we expect the odds of female authors to cite work by women to be higher in economics (compared to mixed gender or male authors), moderate in political science (PA), and the difference to be smallest among sociology authors (SMR). These analyses give us a first look at how the distribution of women in a research area influences the sociological processes by which scholars make citation choices.

3.1 Authors’ Genders in Sample Journals

Using bibliographic data from the Web of Science for all articles published between 2007 and the end of 2016 in the journals described in the previous section, we assigned genders to authors of articles.Footnote ¹⁸ Each article’s first five author(s) are coded as being all male, all female, or mixed gender (Table 1). The distribution of gender of author(s) for PA and P&G are consistent with their membership. One or more men wrote approximately 75% of PA articles, and one or more women wrote 78% of P&G articles. The distribution of authors for APSR is not quite so skewed, though it looks more like that of PA than P&G with 70% of articles written by male only authors.Footnote ¹⁹ Among methodology journals, Econometrica has the highest percentage of male only author teams (77%), while SMR has the lowest (65%).

Table 1. Distribution of author genders by article, 2007–2016.

Source: Authors’ coding of Web of Science bibliographic data. One article in Politics & Gender with an anonymous author was not coded. See the Appendix for sample characteristics.

3.2 Analysing Citation Patterns Within and Across Disciplines

To capture who cites whom, we used the Web of Science to gather all references for the articles published in these five journals since 2007.Footnote ²⁰ The result is a dataset of nearly 54,000 cited references, of which we retrieved complete bibliographic information and coded the gender of the first five authors of a reference for 87.2%.Footnote ²¹ We then code our primary dependent variable of interest: whether reference authors are male only, female only, or mixed gender. Table 2 includes the distribution of this variable by journal, which varies in ways consistent with the gendered distribution of scholars in each field or subfield. While 57.8% of references in P&G are to work by all female author teams, only 4.4% and 5.5% of references in Econometrica and PA respectively are to work by women only.

Table 2. Distribution of reference author genders, 2007–2016.

Source: Authors’ coding of Web of Science bibliographic data. See Appendix for distributions by journal and year.

Table 3. Logistic Regression Estimates: Gender of references and author teams.

Robust standard errors in parentheses . The dependent variable equals one when a bibliography citation was authored by females (only).

Following the methodology used elsewhere (Ferber Reference Ferber1988; Mitchell, Lange, and Brus Reference Mitchell, Lange and Brus2013), we regress the gender of the reference authors on the gender of the original article authors. We dichotomize the reference author gender into those references written only by women compared to those by men or mixed gender author teams. To compare the strength of the relationship between article author gender and reference gender by journal, we first estimate separate models by journal to see if the varying representation of women in these political science research areas and social science disciplines influences the degree to which we observe a gender citation gap. These logistic regressions are estimated with robust errors clustered by article. Then, for comparison, we also estimate a model that pools all the references together, with fixed effects by journal and robust errors clustered by article.

The results are presented in Table 3. We see that female authors are significantly more likely to cite work by female scholars in four of the five journals in our sample (and in the pooled model). The journal fixed effects show differences across research areas. Politics & Gender articles cite female authors at the highest rate, followed by APSR (baseline category), and Political Analysis. Among methodology journals, Econometrica articles cite the fewest female authors, followed by Political Analysis, and SMR. Even though all three of these areas are publishing methodological research and draw upon work in other disciplines, there are still noticeable differences across fields, which is consistent with varying degrees of women’s representation in the fields.

Figure 1. Predicted probability (with 95% confidence intervals) of citing a female-only reference. Note: Based on separate models in Table 3.

This is best illustrated in Figure 1, which plots the predicted probabilities of a reference having only female authors by journal and article author team gender. While women have the highest probability of citing work by other women in P&G, there is still a sizable gap in terms of how men and mixed gender teams cite work by women in this research area. Female authors have a 0.63 predicted probability of citing work by women, while the predicted probability of citing work by women for men is 0.50 and for mixed teams is 0.46 in the journal. By comparison, the three methods journals have lower probabilities of citing work by female authors than either a gender journal in a female-dominated field (P&G) or a general disciplinary flagship journal (APSR), due primarily to the predominance of work by men in these research subfields. In PA, the predicted probabilities that male authors and female authors will cite work by women are 0.05 and 0.08 respectively, which situates PA between the predicted probabilities of citing work by women for all male and all female author teams in Econometrica (men: 0.04 and women: 0.12) and SMR (men: 0.08 and women 0.15). In addition, the gender citation gaps are relatively smaller in these subfields with fewer women than the large gap in the probabilities observed in P&G or APSR. Much like Ferber (Reference Ferber1988) observed a small citation gender gap for mathematics, areas of the social sciences with mostly male scholars also produce articles with few citations to women’s research and a smaller citation gender gap. On the other hand, the gap between female authors citing work by women and male/mixed authors is larger for APSR, consistent with “Matilda effects” for the field’s flagship journal. Female authors in the APSR have a 0.25 predicted probability of citing work by women, while that probability is only 0.11 among male authors and 0.13 among mixed gender teams. The proportion of women in a subfield or discipline is clearly associated with gendered citation practices for all scholars working in those fields. While greater gender representation in an academic field improves attribution of ideas to work by women scholars, the citation gender gap persists, providing evidence in favor of the “Matilda effect”. Even when women constitute the majority of scholars in a research area and the editors and editorial board members of a journal like P&G, the probability of men citing work by women scholars is 0.13 lower than their female author peers.

Figure 2. Odds ratios (with 95% confidence intervals) for models of female-only references by journal. Note: Based on models in Table 3.

Figure 2 illustrates the odds ratios of article author genders relative to male only author teams across models in Table 3. The figure shows that female-only author teams have significantly higher odds of citing work written only by women, compared to male only author teams. By comparison, when women co-author with men on mixed gender author teams, they are not significantly more likely to cite references with female-only authors than male only article authors. This pattern mirrors previous results that showed mixed gender teams adopting citation practices like all male author teams (Mitchell, Lange, and Brus Reference Mitchell, Lange and Brus2013). At the same time, the tendency for female authors to be more likely to cite other women is higher among authors of articles in Sociological Methods & Research than female authors in PA, which is expected given the differences in the proportion of women in each discipline. The results show that citation practices are influenced by gender diversity. While methodological approaches in the social sciences share many similarities and build upon research in other disciplines, the centrality of women’s work varies significantly with respect to citations in the flagship methodology journals in economics, political science, and sociology.

Analyses in the Appendix show that our results are robust when controlling for other factors that influence citation decisions. First, we present results for specifications where the dependent variable is recoded to combine female-only and mixed gender authored references, compared to male only references. The results are similar to the analyses reported above. Second, given that our dataset covers a ten-year span, we also test for temporal changes in our results. This is especially important given the emergence of mentoring programs for women in research fields that include authors of the journals that we examine.Footnote ²² In the Appendix, we present the proportion of article authors (first five authors) that are male or female in each journal over time. Of the journals in our sample, only APSR appears to have a steady increasing proportion of authors that are female over the decade of the sample. Next, we examine the percentage of the first five authors of each coded reference that is male or female by journal and year. If women’s representation is increasing in the corpus of available research to be cited, then we might also expect the proportion of reference authors that is female to increase as well. While the proportion of reference authors that are female seem to increase slightly in APSR and Political Analysis in the last few years of the data, we see few dynamic patterns overall. We also estimate the models in Table 3 including fixed effects for each year in our sample. Our primary findings hold; in all journals except Political Analysis, female authors are significantly more likely to cite work by female authors compared to both men and mixed gender author teams. If indeed women were, on average, more likely to be cited with each additional year due to their growing representation in the corpus of available references, we would expect the fixed effects to gradually move away from the baseline category, 2007. Instead, the results indicate that while some years may deviate significantly from 2007, overall there is no trend in the coefficients toward higher probabilities of citing work by female authors in any of the journals. The confidence intervals for the fixed effects mostly overlap as well, suggesting few temporal changes in citation practices. We also estimated the models in Table 3 separately for the first and most recent years for each journal included in the sample. The results are available in the Appendix (Table A11). In most comparisons across years for the same journal, the coefficients are substantively and statistically similar.Footnote ²³ In summary, we have no reason to believe that the pooled results that we present change in any significant way between 2007 and 2016.

Third, we estimated models controlling for the earliest publication by a member of the author(s) to account for potential cohort effects since older authors might have higher baseline probabilities of being cited. We coded the author teams into one of 5 cohorts based on the oldest author: 2007–2016 (reference category in analysis), 1997–2006, 1987–1996, 1977–1986, and 1967–1976. We also estimated our models using time since first publication, measured in years from zero through 50. The coefficients for the gender of the author team (female only, mixed) are not substantively different from those presented in Table 3 when we include cohort or time since first publication variables (see Appendix, Tables A12 and A13). We still find that women cite work by women at significantly higher rates than male or mixed gender author teams.Footnote ²⁴

Finally, we seek to determine if the citation gender gap is likely driven by women overciting work by other women, men underciting work by women, or both. Two tables in the Appendix (A6 and A7) provide information about the proportion of female authors and the proportion of females cited in the references. The percentage of females cited in the references for each journal is lower than the percentage of female authors, suggesting that women’s work in general is not being overcited in these journals. Even in a journal with a majority of female authors like P&G, the gap between female authors (82%) and female references (61%) is fairly large. Given the gender dynamics we see when pairing author to reference citations (Table 3), this suggests that most of the citation gap is driven by men and mixed author teams underciting work by women rather than women overciting work by other female scholars.