The Gender Balance Assessment Tool (GBAT) was introduced in 2016 as a shortcut for researchers and instructors who wanted to quickly determine the gender balance of the authors in their bibliographies and syllabi (Sumner Reference Sumner2018). Although admittedly the GBAT was less accurate than hand-coding, it was promoted by some journals and institutions due to its ease of use. It also was limited by shortcomings, including not being as accurate as hand-coding, providing less information than some users wanted, and—due to being hosted and maintained by only one person—dealing with frequent outages and errors. Although flawed, the GBAT was useful and a reasonable use of the technology available.
However, technology has advanced significantly in recent years including, most notably, the emergence of generative AI models and accompanying systems (e.g., ChatGPT, Microsoft Copilot, and DeepSeek) (Feuerriegel et al. Reference Feuerriegel, Hartmann, Janiesch and Zschech2024). Although there are many concerns about the use of generative AI in classrooms and in academia generally (Ardoin and Hicks Reference Ardoin and Hicks2024; Wu and Wu Reference Wu and Wu2024), the models and systems will improve on the GBAT because they are not subject to the same limitations. Because the training data involve large troves of publicly available data, they can draw from a wider variety of information sources about an author’s gender and race, use more current tools in its identification of names, and provide substantially and increasingly more accurate estimates than a probabilistic name-estimation system alone. This article demonstrates that these systems can be more accurate than the GBAT and can provide additional information that the GBAT cannot. This allows scholars to evaluate their syllabi and bibliographies along other dimensions of interest (e.g., institutional rank). Moreover, and from a usability perspective, because the larger chatbots are better resourced and staffed, they are not limited by the same risk of outages, error messages, and long lag times for bugs to be fixed.
This article is an overview of the arguments for assessing demographic balance in syllabi and bibliographies. It outlines the shortcomings of the GBAT and the potential benefits of instead using generative AI systems. Practical advice for using ChatGPT, Microsoft Copilot, and DeepSeek is presented to assess the demographic characteristics of syllabi. We tested its accuracy in predicting gender compared to both hand-coding and the GBAT. Using a small but representative sample of syllabi and bibliographies, we show that ChatGPT consistently outperformed the GBAT; Copilot and DeepSeek performed somewhat less consistently. The article includes information and demonstrations about how a prompt can be crafted to produce better estimates from all three systems; Copilot benefited most from this additional instruction. Because we analyzed only a small sample due to rate-limiting barriers in AI tools, our findings are only suggestive. Therefore, we encourage further research on the use of generative AI programs in the discipline.
BACKGROUND AND SHORTCOMINGS OF THE GBAT
The interest in gender-balancing syllabi and bibliographies arose from research on gender gaps in both bibliographies (Dion, Sumner, and Mitchell Reference Dion, Sumner and Mitchell2018; Djupe, Smith, and Sokhey Reference Djupe, Smith and Sokhey2019; Maliniak, Powers, and Walter Reference Maliniak, Powers and Walter2013; Teele and Thelen Reference Teele and Thelen2017) and syllabi (Hardt et al. Reference Hardt, Kim, Smith and Meister2019; Smith et al. Reference Smith, Hardt, Meister and Kim2020). Even after controlling for relevant covariates, women authors often are cited and assigned as required readings less frequently than men. Many within the discipline have responded to this by aiming for a greater gender balance in their bibliographies and their syllabi; some journals also have begun to implement policies to achieve this goal. As highlighted in extant scholarship, balancing bibliographies and syllabi along gender lines has many benefits to the discipline, including increasing the diversity of topics, questions, and methodological approaches in political science research; supporting women’s career advancement in academia; and countering implicit gender biases in graduate-student training. It also encourages scholars to expand their own horizons and discover research that is unknown to them.Footnote 1
...balancing bibliographies and syllabi along gender lines has many benefits to the discipline, including increasing the diversity of topics, questions, and methodological approaches in political science research; supporting women’s career advancements in academia; and countering implicit gender biases in graduate-student training.
The GBAT was introduced in 2016 in response to previous research on this topic (Colgan Reference Colgan2017; Maliniak, Powers, and Walter Reference Maliniak, Powers and Walter2013).Footnote 2 The motivation for the GBAT was to make it easier to estimate the gender balance of syllabi and bibliographies for those who wanted to do so but were deterred by the time and effort required to manually evaluate their syllabi (Sumner Reference Sumner2018). The GBAT was and still is an R Shiny site that allows users to either copy and paste or upload a syllabus or a bibliography. It then estimates the percentages of the document that are likely women and likely men authors. As an option, the GBAT also can assess the racial balance, but it does so with caveats about its accuracy.
The GBAT uses fairly simple tools (i.e., word elimination and then regular expressions) to first identify which words in the document were likely to be authors’ names. Second, it probabilistically estimates the gender and race for each likely name. Third, it returns the percentages of the author list likely to be women (Blevins and Mullen Reference Blevins and Mullen2015; Mullen Reference Mullen2021) and, optionally, racial categories from the US Census Bureau (Kaplan Reference Kaplan2023). This process typically requires less than a minute and is reasonably accurate. However, this process had many clear shortcomings because accuracy and breadth were sacrificed for speed and ease.
First, some users found the GBAT to be too narrow in its focus on race and gender because they also wanted to assess other types of demographic balance. However, the GBAT was limited to only the information that it could learn from the uploaded document. For authors, this information was limited to their names. The GBAT had no additional capacity to conduct Internet searches or query databases to learn more about the authors. Therefore, it was limited to what reasonably could be inferred from only names: that is, gender and race.
Second, the GBAT was not always accurate. Two key sources of inaccuracy are built into the GBAT. The first is the problem of both false positives and false negatives in the name-identification stage. That is, when the GBAT uses its simple tools to determine which words in the document are authors’ names, it mistakenly might include words or names that are not authors’ names (i.e., false positive) or exclude those that it assumes are not names at all (i.e., false negative). This means that the denominator might be wrong when calculating the overall proportions and that the aggregated estimates include too much noise.
The second source of inaccuracy built into the GBAT is that it might misgender a name. To estimate gender and race from names, the GBAT used tools built into R that would infer an individual author’s name (Mullen Reference Mullen2021) and race (Kaplan Reference Kaplan2023) from information about the gender and race typically associated with those names.Footnote 3 For most people, this was correct for gender. It was far less accurate for race (as was acknowledged, which is why it was optional) because race predictions become more accurate with the incorporation of additional information about individuals that is not contained in syllabi and bibliographies (e.g., geographic location). However, it was not always correct. Some names are gender neutral; are sufficiently uncommon such that they were not in the databases; vary in gender by nationality (e.g., “Jan” in many European countries is primarily male but in the United States it is typically female); vary by generation (e.g., a Stevie or Kerry born in the early twentieth century was more likely to be male but born later was more likely to be female; older Rileys are more likely than younger Rileys to be male)Footnote 4; or belonged to people who did not identify with the gender most common for their name.
A third problem with the GBAT is its sensitivity to formatting. Because of how it handles document imports and how it detects names, the GBAT occasionally would not work if a document were formatted in a nonstandard way. For instance, a syllabus with readings formatted in a table often would be unreadable and the GBAT would return an error. It also is sensitive to citation style: citation styles that use only an author’s initials and a surname would produce either no estimate or bad estimates. The same is true for syllabi that use only an author’s surname.
Fourth, when the GBAT produced errors or failed to work, there was a lack of technical support. Specifically, the GBAT typically is supported and maintained by only one political science faculty member, who often was not available to fix the problem or quickly respond to errors due to time and resource constraints. Therefore, when the GBAT failed or when errors occurred for individual users, it was not uncommon to take days or weeks for the maintainer to fix the issue, depending on its complexity and when the issue occurred. Because the GBAT had no funding, it was not possible to maintain any support staff. For similar reasons, the maintainer was unable to continue paying for server space or use of some of the original tools, which meant that during busy times of the year, the site might go down entirely or be unable to produce gender estimates.
In summary, the GBAT is a simple tool that was created to solve a problem for a specific subset of the population. It had many shortcomings that, for a period, were acceptable given that they still were an improvement over the primary alternative of hand-coding and hand-counting. However, technology has improved substantially such that better alternatives currently are available and the GBAT’s shortcomings are no longer acceptable in comparison. The next section outlines key benefits of using generative AI and suggested prompts for accessing that information.
USING GENERATIVE AI TO ASSESS DEMOGRAPHIC BALANCE
Generative AI improves on every shortcoming of the GBAT because popular systems (e.g., ChatGPT, Microsoft Copilot, and DeepSeek) are trained on data from the entire Internet. This includes university and department websites, personal faculty websites, and curricula vitae. As such, generative AI models have knowledge of substantially more information than only names from which to estimate the key parameters. This is an improvement over the probabilistic name algorithms because these models can properly gender groups of people who consistently are misgendered using the GBAT approach. These groups include those who are nonbinary, have names more commonly used for the opposite gender, have names not sufficiently common to appear in the databases used for those algorithms, and have gender-ambiguous names.
A first benefit of this approach is that it permits users to learn about more demographic characteristics than only gender and race. For instance, if there is concern about appropriately recognizing scholars who are likely to have fewer financial resources to conduct their research and to attend conferences to promote it (Farris, Key, and Sumner Reference Farris, Key and Sumner2024), generative AI can provide information about the types of institutions at which the authors work by asking the following questions:
“Can you provide percentage breakdowns of the colleges and universities at which each author works, using the Carnegie Research Activity Designations? Can you give a breakdown of author institutions based on the most recent U.S. News & World Report rankings, broken up into ranges such as 0–10, 11–20, and so on?”
A first benefit of this approach is that it allows users to learn about more demographic characteristics than only gender and race.
Many critiques of the GBAT also focus on its inability to assess whether a substantial proportion of scholars were international, were trained outside of the United States, identified as LGBTQIA+, or were first-generation college graduates. These are characteristics that cannot be determined from names; however, to the degree that this information is disclosed by authors on their website, AI can assess it.
Second, the generative AI models (in principle) can provide more-accurate estimates for an individual author because they may “know” information about a specific person, not from those who have the same first or last name. This means that the models can estimate gender from the pronouns on authors’ websites and other materials linked to them. It also can provide continuity and current information about an author’s identity. For instance, if an author transitions to another gender, generative AI systems should be able to provide the current gender if they are properly prompted to do so. They also can provide information about where researchers were employed when their cited pieces were written or where they currently are employed. Generative AI systems also can be prompted to provide a list of how they code each author on each dimension, allowing the user to quickly check the accuracy of the estimation and fix anything that seems incorrect. This allows researchers to use their own knowledge and become part of the estimation process with much less effort.
Third, generative AI systems can provide a more accurate document-level estimate. The GBAT uses a series of formatting rules to identify what “looks like” a name. By contrast, AI uses natural-language processing and therefore can identify names more like a human would. Consequently, generative AI systems are far less prone to both false negatives and false positives. The systems’ ability to export a table of each author’s name and their predicted demographic data allows the user to quickly assess whether anything is missing or included erroneously. This improves the estimation process by allowing the user into the process. Similarly, because these systems do not rely on specific formatting to identify a name, they are not sensitive to a document’s formatting or file type.
Fourth, because popular generative AI systems are created and maintained by large teams of dedicated engineers, they are far less likely to produce errors, break down, or go entirely offline. These systems have many more assets at their disposal to further develop their tools. This also allows them to be faster and suggests that they may become more accurate over time.
Baseline Testing
To assess the performance of the GBAT against generative AI, we collected publicly available syllabi and bibliographies. We collected syllabi through the American Political Science Association (APSA) collection, which features 99 primarily undergraduate course syllabi from different institutions, instructors, and subfields in political science. All syllabi listed on APSA’s main webpage were downloaded; however, 11 links were broken or no longer available, resulting in a total of 88 syllabi. Bibliographies were collected from the four most recent issues of three political science journals: American Journal of Political Science, American Political Science Review, and Journal of Politics. This collection process resulted in a total of 169 bibliographies. Of this sample, we selected 5% (15) of the bibliographies and 10% (nine) of the syllabi to evaluate, for a total of 24 documents (Musgrave and Sumner Reference Musgrave and Sumner2026). Hand-coding is a labor-intensive process—if it were not, the automation would be less attractive—therefore, we opted to hand-code only a small sample to use as a benchmark. In addition, the process of estimation for the three generative AI systems was time-consuming—that is, all three had rate limits, which meant we could check only a few each day. Furthermore, Copilot has a character-entry limit, which meant that each document had to be divided into multiple documents for estimation.
We ran each syllabus and bibliography through the GBAT, ChatGPT, Microsoft Copilot, and DeepSeek platforms. For all three generative AI systems, we used the following prompt:
“Can you estimate what percentage of the authors of assigned readings in the list of readings below are women versus men, and what percentage belong to each of the US Census racial categories? In addition, please report how long it takes to complete this task in secondsFootnote 5 and put all of these results into a single row that I can easily copy into a spreadsheet. The first column should be the file name. The remaining columns should be: % women, % men, % white, % Black, % Hispanic/Latino, % Asian, % unknown, and elapsed time.”
We then copied and pasted the content from the document below the prompt.Footnote 6 For Copilot, which limits how much text can be entered, we added the text: “This may come in multiple lists. Do not begin estimating until I say ‘I am done, go.’” Both ChatGPT and Copilot have the option to upload files—DeepSeek did not have that functionality in its free version when we were conducting this research—but Copilot consistently froze and would not permit us to enter a prompt with the file. Therefore, for consistency, we copied and pasted the documents into each system.Footnote 7
Although AI allows researchers to find or report this information in different ways (e.g., specifically asking it to tally gender by pronouns or to include other genders or different racial categories), we used it in our baseline testing in the way that was most comparable to the existing GBAT. To assess each method, we recorded the percentages of women and men and the percentages in each racial category. We considered the hand-coding to comprise the baseline correct answers because it was conducted with the most diligence and human knowledge.
Baseline Findings
To establish a performance baseline for each method, we evaluated hand-coding against the four estimation processes only for gender. Although the GBAT and all three generative AI systems produced estimates for race, we ultimately found the hand-coding of race to be a poor baseline. Our only options to assess an author’s race from publicly available sources were names and photographs, which can be misleading or uninformative. Moreover, people do not often publicly state their race. Therefore, we evaluated all four methods against hand-coding for gender, and we evaluated them against one another for race.
For our initial test, we evaluated whether each generative AI system produced an estimate that was within the same general range as hand-coding for each of the 20 documents. Because each estimation technique has a certain amount of error built into it, none should be considered especially precise and instead interpreted as the midpoint of a range. Therefore, we coded each document as a match if it was within a +/- 5-point range of the hand-coded estimate. We found that ChatGPT performed the best, arriving at an estimate within that range for 62% of documents. The GBAT was in range for slightly more than half (52.3%) of the documents. DeepSeek and Copilot performed the worst, with 42.9% and 23.8% matching, respectively.
Figure 1 is a graphic depiction of this and also illustrates how the different systems tend to err when they do. Documents with an estimate higher than the hand-coding range are labeled as “over” and those that are less than are labeled “under.” The figure demonstrates that the GBAT has a tendency to overestimate the number of women in the documents, overestimating almost as many as it estimated within range (i.e., 42.9% were overestimates). Copilot, conversely, tended to underestimate the percentage of women: almost 62% of documents were less than the range. Indeed, for 19% of the documents, Copilot returned an estimate of 0, estimating no women in the document. DeepSeek was almost as likely to overestimate as underestimate, and ChatGPT was exactly as likely to overestimate as to underestimate. This reveals that ChatGPT was more accurate than the GBAT for this sample as well as not systematic in its errors, whereas the GBAT was likely to overestimate women and Copilot was likely to underestimate them.
Accuracy of Coding (All Documents)
Percentage of documents within a range of +/-5 of the hand-coded estimate (“correct”), as well as higher than that range (“over”) and less than that range (“under”).
Separating the documents into bibliographies and syllabi provided greater insight. The bars in figure 2 indicate the percentage of documents that were within a +/- 5 point range of hand-coding (“correct”) and how many were higher than (“over”) or less than (“under”) that range. This reveals that ChatGPT was very accurate for bibliographies (85.7%) but considerably less accurate for syllabi (only 14.3% in range). When ChatGPT is not in range, however, it overestimates and underestimates the percentage of women at equal rates. The GBAT also performed reasonably well for bibliographies and poorly for syllabi, but it had a marked tendency in both to overestimate the percentage of women in a document. Although both DeepSeek and Copilot performed poorly, they performed better than both ChatGPT and the GBAT for bibliographies at 50% and 28.6% within range, respectively. Copilot’s tendency to underestimate women in documents was present in both bibliographies and syllabi, whereas DeepSeek was more likely to overestimate in bibliographies and underestimate in syllabi.
Categorization of Bibliographies and Syllabi
Because the choice of a +/- 5-point range was arbitrary, it is worthwhile to inquire exactly how far off the estimates were: estimates that were slightly outside of and 20 points out of range both appeared as out of range in our analysis so far. Table 1 shows the average distance between hand-coding estimates and the estimates of each method, revealing that when Copilot and the GBAT were wrong, they tended to be very wrong. Copilot was under, on average, by more than 10 points for both bibliographies and syllabi (14.48 and 11.85, respectively), whereas the GBAT—although close to hand-coding on bibliographies—was out of range by approximately 20 points for syllabi in our sample.
Mean and Median Distances Between Hand-Coding Estimates and Estimates of Each Type of Generative AI
The GBAT’s difficulty with syllabi is due to how it determines what is and is not a name. Whereas there are standard formats for citations and authors’ names are clearly structured—typically a first and a last name—the ways in which authors are listed on syllabi are varied and often contain only surnames. Because the GBAT uses first names for its information on gender, and because it uses regular expressions to determine what is and is not a name, it is especially vulnerable to incomplete information and variations in formatting. Although the generative AI systems also perform worse for syllabi than for bibliographies, they can outperform the GBAT because they triangulate information to determine an author’s name.
Finally, although we could not compare the race predictions with hand-coding, we could compare them among methods. To do this, we considered the standard deviation of each technique’s estimates of the percentage of white authors for each document. The lowest and highest standard deviations were 6.5 and 50, respectively. The median standard deviation was 21, suggesting that for most documents, the techniques produced significantly different estimates for race. Because we could not compare these with any type of baseline, we conclude that predicting race is difficult with inconsistent results. The choice of AI tool is likely highly consequential for whatever inferences we might draw about racial distributions.
In summary, however, our findings indicate that ChatGPT outperformed the GBAT in predicting aggregate author’s gender for both syllabi and bibliographies. DeepSeek was somewhat reliable. Copilot was the worst of the three and, although it outperformed the GBAT for syllabi, the GBAT outperformed Copilot for bibliographies.
…our findings indicate that ChatGPT outperformed the GBAT in predicting aggregate author’s gender for both syllabi and bibliographies.
GETTING BETTER ESTIMATES FROM GENERATIVE AI PLATFORMS
The previous section discussed using generative AI with the least-specific prompt, asking it to estimate only the quantities of interest. This section considers the syllabus and bibliography with which each generative AI platform performed worse and explores ways to engage differently with them to improve predictions. However, it is noteworthy that because of the nature of how generative AI works, it does not necessarily consistently return the same estimates, even with the same prompt and the same list. In this way, it is similar to running statistical simulations—there is randomness involved—and, moreover, the underlying processes are poorly understood. Therefore, this article reports how the estimates changed when we changed the prompt; however, we cannot guarantee that the differences were due to the different prompts.
To evaluate whether the augmented prompts improved estimates, we chose the worst-performing syllabus and bibliography for each of the three generative AI systems, determined by the distance from the hand-coded estimate. This created a list of six documents: three syllabi and three bibliographies.Footnote 8 For each document, we ran them through the system with the listed prompt and then evaluated whether the new estimate was within, higher than, or less than the +/- 5-point range.
Instructions for Calculation
One difference among the generative AI output, the hand-coding, and the GBAT may be related to the specifics of counting rather than the estimation technique. The GBAT and hand-coding, for instance, counted repeated authors as different entries. For example, if a scholar authored five articles on a syllabus, it would count as five people. It is unclear how the generative AI systems address this concern because they do not always report it in the output.
Additionally, it is not known how the generative AI systems address names that they cannot ascertain. The GBAT drops from the denominator any name for which it cannot obtain a prediction. It does this as part of its efforts to figure out which words in a document are names, assuming that any “name” without a gender probability is probably not a name at all. Even without this assumption, it would need to drop these names because it would be unable to assign it to a gender. For names that do have a gender probability, the GBAT assigns any name with a probability of being a woman greater than 0.5 as a woman and otherwise as a man. By contrast, it is unclear how generative AI handles this.
To circumvent this uncertainty, we added language to the prompt to specify how the generative AI system should handle these issues. Specifically, we instructed it to count every author occurrence as a different author—rather than deciding a repeated author counts only once—and to report any author whose gender cannot be ascertained as “undetermined.” We expected this to improve estimates if the generative AI systems were counting repeated authors as only one person and if they were simply guessing at unknown genders. We used the following prompt:
“Can you estimate what percentage of the authors of assigned readings in the list of readings below are women versus men? Put all of these results into a single row that I can easily copy into a spreadsheet. The first column should be the file name. The remaining columns should be: % women, % men. A final column should be anyone whose gender cannot be determined from the given information and should be labeled % undetermined. Count every author occurrence as a different author in the denominator and for estimation purposes, such that someone who is listed twice will count as two people.”
We again assessed whether each estimate was within a +/- 5-point range of hand-coding. By that metric, this did not lead to improvements overall. Of the 18 estimates (i.e., six documents x three AI systems), seven were in range originally (approximately 39%) and seven were in range using the revised prompt; however, they were not the same seven. The ChatGPT estimates worsened with this prompt: three were originally in range and only one was in range with the revised prompt. In each of those cases, the revised estimate was lower than the original: that is, the three that had been in range fell to less than range but then one that was higher was brought to within range. In two of the cases that fell out of range, the system incorrectly revised upward its estimate of men; in one, it shifted one author into the unknown category. For the one case that was brought within range, the system correctly shifted some of the coded women to being coded as men.
Copilot had one case shift from in range to out of range, and two cases shifted from out of range into range. This was due in part because Copilot tended to guess that there were zero women. One improvement was simply that the system gave an estimate, and one case worsened because it answered zero. In general, Copilot had very high variance, regardless of the prompt, and may have answered correctly simply by chance. DeepSeek had three cases improve, one remained correct, and two fell out of range, for a net gain of one. As with ChatGPT, the revised prompt led DeepSeek to revise downward its estimates of the percentage of women. Overall, we found no evidence that providing the AI system with more detailed instructions improved estimates.
Using More Information
Although ChatGPT noted that it sometimes used publicly available information about scholars, with the basic prompt, both generative AI systems often fell back on the same methods as the GBAT. Copilot did so more often than not, as noted periodically in its output. This improves on the GBAT in its ability to detect what is an author’s name, with tools more flexible than regular expressions (i.e., guessing about what names typically look like to figure out what probably is a name), and Copilot appears to draw from a wider range of name databases than the GBAT. However, this is not using the AI to its full capacity. This section explores options for obtaining better estimates from generative AI systems by leveraging their ability to tap publicly available information about the researchers.
To accomplish this, we added to the previous prompt that specified how to handle repeated authors and undetermined genders by specifying that the AI systems should use “university and professional websites, other publications from the author, and RateMyProfessor” as sources for inferring an author’s gender whenever possible. We expected this approach—that is, being increasingly more specific about how to handle data and which sources to use—to lead to the greatest improvements because—at least in principle—it eliminates the most uncertainty in the process of how the generative AI systems work by providing the clearest instructions. Specifically, we used the following prompt:
“Can you estimate what percentage of the authors of assigned readings in the list of readings below are women versus men? Put all of these results into a single row that I can easily copy into a spreadsheet. The first column should be the file name. The remaining columns should be: % women, % men. A final column should be anyone whose gender cannot be determined from the given information and should be labeled % undetermined. Count every author occurrence as a different author in the denominator and for estimation purposes such that someone who is listed twice will count as two people. Please use university and professional websites, other publications from the author, and RateMyProfessor to infer gender where possible.”
When we combined these two possibilities for improving estimates, we observed modest improvements. For ChatGPT, four of the six estimates were within range: two remained there and two shifted from out of range into range. However, the two that remained out of range had been correct with the original prompt but were revised downward when we introduced more detail into the prompt and fell below range. Being given information sources drastically improved Copilot’s estimates: it brought five within range and, in fact, very close to hand-coding—all within a range of +/- 2.5 points. The sixth estimate was a syllabus that remained incorrect. DeepSeek’s answers changed very little with the revised prompt: the four that were in range remained in range, the two that had been out of range remained so as well. Our primary takeaways were that providing information on sources resulted in Copilot performing quite well in our sample, marginally improving the estimates of ChatGPT, and having no meaningful effect on DeepSeek.
DRAWBACKS AND ETHICAL CONSIDERATIONS
Although generative AI systems can improve on the GBAT and be comparable in accuracy to hand-coding (but at a substantial reduction in user time), they are not problem free. This section addresses key problems that users may have with generative AI systems.
One issue that users may be concerned about and may be able to control is whether a generative AI system subsequently saves their syllabi or citation list after use for training purposes. For practical and ethical reasons, users may have reservations about their work being used to further train or improve AI tools. To address this, we encourage users to determine which generative AI models their institution may have contracts with and whether they preclude the saving of user data. Additionally, ChatGPT reports that through its privacy portal users can opt out of their data being used for training purposes.Footnote 9
It also is noteworthy that AI tools are prone to error, despite the advantages they provide over other analysis methods and tools. For example, when we conducted baseline tests in ChatGPT, there were several instances when we uploaded our prompt and references list and instead of getting the author demographic estimates, we received a weather forecast or transportation options for the local metropolitan area. However, submitting the same prompt and references list at another time or on another device provided the information that we were seeking. It is possible that this was a response that ChatGPT gives when a references list is particularly long; however, we cannot conclude this with any certainty. The main takeaway is that ChatGPT and other generative AI tools are imperfect and can be difficult to troubleshoot when they do not function as expected. We recommend that users ask the AI tool to produce a coded list to allow users to doublecheck the answers rather than accept them at face value.
Finally, some people may have ethical or moral reservations or objections to generative AI. It is known to be environmentally deleterious, for instance, because its servers require both extensive energy and cooling (Feuerriegel et al. Reference Feuerriegel, Hartmann, Janiesch and Zschech2024). In general, generative AI also can be used to displace and replace human labor in ways to which society and governments are not fully able to respond. Moreover, it is trained on information that people and institutions published on the Internet without consent for it being used in this way. In some cases, generative AI systems have been trained on copyrighted or otherwise protected information without compensation of authors. These are all credible arguments against the use of generative AI for assessing syllabus and bibliography demographic balance.
CONCLUSION
Although generative AI has caused many problems and significant concern within academia, this study contends and demonstrates that one area in which it can excel is in the ability of researchers and instructors to quickly and accurately assess the demographic balance of their bibliographies and syllabi. This article presents the major shortcomings of the GBAT, a popular Internet tool used to evaluate gender balance, and explains why and how generative AI systems can improve on it. We tested this idea by evaluating a small random sample of publicly available bibliographies and syllabi against the GBAT and hand-coding.
We found in our sample that ChatGPT performed most similarly to hand-coding and outperformed the GBAT. The GBAT outperformed DeepSeek and Copilot in our tests, although it is prone to overestimating the percentage of women in a document. We then identified two additions to a baseline prompt to improve estimates by providing more detailed instructions on how to code and where to source information. These additions significantly improved the estimates for Copilot. We also recommend future work to further evaluate how these tools can be used in academia to evaluate demographic balancing in syllabi and bibliographies.
In summary, we found in our sample that generative AI performs at least as well as the GBAT, if not often substantially better. We recommend that ChatGPT or other AI tools should be used instead of the GBAT. However, our research has caveats. First, we tested only a small sample of documents and our findings should be interpreted with caution. Although we did select the documents randomly from a fairly representative pool—and we have no reason to believe our findings would not be replicated with other documents—we can make only modest claims with our data. Second, we evaluated the AI only against the GBAT because our aim was to assess whether it could be replaced. The main critiques of the GBAT—it uses simplistic tools to identify names and cannot incorporate additional information—may be unique to the GBAT. Therefore, we only can claim that in our testing, the generative AI performed well against the GBAT, but we cannot claim that it would outperform other tools or algorithms. Third, we were unable to assess systematically whether the generative AI systems varied in their accuracy based on characteristics of the document (e.g., the subfield of the article or the syllabus).
In conclusion, we recommend that future research take this study a step further and interrogate what happens when generative AI is asked not only to estimate the balance of our syllabi but also to suggest replacements. There is an ample literature about the gender citation gap when humans choose what to cite. However, to the best of our knowledge, as of yet there are no studies on whether generative AI—freed from our memories and biases but laden with its own—would make this worse, better, or no difference.
DATA AVAILABILITY STATEMENT
Research documentation and data that support the findings of this study are openly available at the PS: Political Science & Politics Harvard Dataverse at https://doi.org/10.7910/DVN/BDLZ66.
CONFLICTS OF INTEREST
The authors declare that there are no ethical issues or conflicts of interest in this research.
Open access
