What counts as “impactful” research in political science? Scholars want to know whether particular peer-reviewed journals or scholarly presses are relatively more or less prestigious. Knowing the relative stature of journals and presses helps scholars make professional decisions and helps colleagues, department chairs, and administrators make difficult personnel decisions and evaluate entire departments.
Political scientists have engaged in several efforts to gauge the impact of our research. Most of this work counts citations that peer-reviewed articles receive in other peer-reviewed articles (e.g., Garand and Giles Reference Garand and Giles2003; Giles and Garand Reference Giles and Garand2007; Hix Reference Hix2004; Masuoka, Grofman, and Feld Reference Masuoka, Grofman and Feld2007), although I (Reference Samuels2011) recently explored how frequently peer-reviewed articles are also cited in books. Other work uses perceptual data to consider scholars' evaluation of journals' relative stature (e.g., Garand et al. Reference Garand, Giles, Blais and Mclean2009).
Little research, however, has attempted to ascertain the relative impact of books. Moreover, extant efforts have considered subjective data only, using surveys of scholars' assessments of presses' relative stature (Garand and Giles Reference Garand and Giles2011; Goodson, Dillman, and Hira Reference Goodson, Dillman and Hira1999; Moore Reference Moore2010). This article is the first effort to quantify the scholarly influence of published books, counting citations after five years of all political science books published in 2004 and 2005.
In an age when budget cuts and changing norms of scholarly engagement increasingly question and even threaten academic book publishing, this exercise serves several useful purposes. It offers a reasonable apples-to-apples comparison of the relative impact of the average university-press book versus the average Social Science Citation Index (SSCI)-indexed peer-reviewed article, and it reveals which presses are relatively more likely to publish books that measurably influence other scholars' research. Such information is useful to scholars considering publishing books or articles and should prove useful for departmental chairs and other administrators seeking to evaluate scholars' relative productivity. Finally, it offers a reality check on political scientists' subjective perceptions of publishers' relative quality.
Just like peer-reviewed articles (Samuels Reference Samuels2011), books are cited more frequently in (other) books as compared to the number of times that they are cited in peer-reviewed articles. Moreover, the average book published by a university press receives about three times the number of citations that the average peer-reviewed article in an SSCI-indexed journal receives. Books' bibliographies may be longer than articles', but it is not obvious that the ratio of bibliographic entries per published page differs across articles and books: a 250-page monograph typically contains much more original research and can develop an argument in much greater depth and detail than a 25-page article. Counting the citations that books receive in articles as well as in other books provides a comprehensive assessment of book impact and publisher quality and provides a benchmark for judging any book's relative importance.
Counting book citations also reveals important variations in the impact of research across subfields. Not surprisingly given their potentially global scholarly influence, books published in international relations tend to be cited relatively more frequently than books in the other substantive subfields, even considering only citations in English-language publications. Yet the biggest revelation is that books on social-science methodology receive at least twice as many citations on average as books in any substantive subfield.
In sum, the findings in this article broaden the way we think about the effect of research. To the extent that political science remains a field in which publication in books and journal articles matters for scholarly reputation, this article supplements qualitative assessments of publisher quality and quantitative indicators of journal impact with quantitative indicators of books' and book publishers' relative impact.
COUNTING CITATIONS OF BOOKS
To count citations that books receive in articles and in other books, and then compare publisher rankings based on citations against scholars' subjective assessments, I first compiled a list of all publishers with a booth at the 2010 APSA Annual Meeting plus any other publisher listed in Garand and Giles (Reference Garand and Giles2011). I then compiled a list of all books these publishers released and classified as “political science” in 2004 and 2005. To do so, I contacted each publisher and requested a list of said books. Thirty presses sent spreadsheets; I compiled the books from the publishers' websites for the remainder.Footnote 1 This process generated a list of 3,183 books from 71 publishers. For each book, I then manually compiled the number of citations in the year of publication plus five years—that is, through 2009 for books published in 2004, and through 2010 for books published in 2005.
To find the number of citations that a published book subsequently received in peer-reviewed articles, I used the SSCI. Within the SSCI interface, I first clicked on “cited reference search” and then set the desired time-span for each book. The SSCI catalogs authors' names by last name and first initial, or by last name and first and middle initial. For each book, I entered this information in the “cited author” box (e.g., “jones b*”), and clicked “Search.” This returns a page with potentially many authors who share that name and initials.
For example, consider the book Peacemaking in Rwanda, by Bruce D. Jones. If you search for “jones b*” SSCI returns a list of more than 1,000 authors. If you include Jones' middle initial “d” you still get 141 names. Each name is associated with a particular published work; these are listed in alphabetical order. To find the correct book you must review the entire list and click on the particular entries that are associated with the book in question. Because scholars use different citation styles and may cite authors' names slightly differently, a book can enter the SSCI in different ways—meaning there may be multiple entries for the same book.
After clicking on all the relevant entries, I selected “results in English only” at the bottom of the page, and then clicked on “finish search,” which eliminated double counting. On the next page, I clicked on “articles,” which eliminated book reviews and other material indexed in the SSCI. This resulted in the number entered into my database: only citations to published, peer-reviewed articles.
To ascertain the number of times a book is later cited in other books, I used Google Books (www.books.google.com). Google Scholar (www.scholar.google.com) cannot be used for this purpose because it under-counts citations of books that appear in other books (see Samuels Reference Samuels2011). Consider my book Ambition, Federalism, and Legislative Politics in Brazil, published by Cambridge University Press in 2003. Google Scholar reports 146 citations to my book through 2008, but only finds citations in six books (one not in English).
To find the number of citations that a book receives in other books using Google Books, I followed this procedure: I went to http://books.google.com/advanced_book_search and then entered the book's full title in the “with the exact phrase” search box.Footnote 2 I also included the following search parameters that Google Books offers: (1) search for “all books”; (2) limit search results to “books” (eliminating “magazines” from the results); and (3) limit results to books in English. Finally, I limited search results by date as described previously.
After clicking “search,” results appear. However, the raw Google Books results are not useful for citation-counting purposes. There are two problems. First, because Google scanned literally everything on the shelves of libraries that participated in its Google Books project, the Google Books database contains many things that are not books as commonly defined. Following the same procedure as Samuels (Reference Samuels2011), to maintain a reasonable apples-to-apples comparison with SSCI citations, I defined a book as (1) not a serial publication; (2) formally published by an academic or commercial press; and (3) not a mere a compilation of bibliographic information. This excludes journal articles, working papers, dissertations, and reference works such as Current Contents. With these criteria, I entered the full title and subtitle of each book (plus the author's surname, if the title was imprecise), and then excluded from the results journals and other “non-books” and any foreign-language publications that slipped through the filters.
However, this is not the end of the process. The second problem is that, like regular Google searches as well as Google Scholar searches, searches in Google Books return “related” results, even if you specify that you are searching for a precise string of text. Thus, if you are interested in counting your own book's citations in other books to compare with the results I report in this article, follow the method above, but then, within the results that Google Books returns, only count results that clearly indicate a full, verbatim citation (usually in boldface) to your book's full title and subtitle.
Using this method, after removing five journal articles and two PhD dissertations, Google Books reveals that my book was cited 39 times in other books, in contrast to the five English-language citations that Google Scholar reported. (All of the books found in Google Scholar also appeared in the Google Books results.) Clearly, Google Scholar is not up to the task of counting the citations that books receive in other books. Google Books is far from perfect, but does allow a close estimate, albeit through labor-intensive investigation. All methods of counting citations have flaws—as many scholars have noted, even the SSCI is imperfect. Still, to the extent that we are interested in discovering the impact research has in books, Google Books is a useful tool.
RESULTS
How many citations does the average political science book receive five years after publication? Table 1 presents summary statistics for the entire sample and then separates university presses from other presses. One caveat is needed: my sample of books is numerically large, but still only covers two years. Although I have no reason to suspect these two years were unusual, results may vary over time, particularly given that some presses publish relatively few books each year (see table 3). With this in mind, note first that books are cited far more frequently in other books rather than in articles. Adding up total citations, the average is about 23.4 citations per book after five years. However, this number obscures a substantial difference between university presses (34 presses, N = 1,517) and others presses (37 presses, N = 1,666). The average book published by a university press is cited almost 2.5 times as often (33.2 citations) as the average book published by other presses (14.4 citations).
Results
To put books' and articles' relative impact in perspective, consider my findings (Reference Samuels2011) that after five years the average peer-reviewed political science article will have received about 11 total citations in both articles and books. The appropriate apples-to-apples comparison to make with a peer-reviewed article is with books that have been published by a university press, all of which require peer review (some other presses also require peer review, but this is not a universal practice). This means that the average peer-reviewed article indexed in the SSCI receives about one-third the total number of citations as the average university-press book after five years. Even books that have not gone through a process of peer review (i.e., those not published by university presses) are cited more often than the average article published in an SSCI-indexed journal.
The mean number of citations can be misleading, as there is considerable skew in the distribution—relatively few books with many citations drive up the averages. Figure 1 plots the distribution of citations that books received as a percentage of all books for non-university presses, and figure 2 does the same for books published by university presses. This reveals that one-third of books published by non-university presses receive three or few citations, while the same is true of only about 10% of books published by university presses.
Distribution of Citations (Non-University Press Books)
Distribution of Citations (University Press Books)
Moreover, books published by non-university presses are far less likely to receive a relatively high (e.g., >20) number of citations. (To simplify the presentation, the histograms exclude the 115 books (~3.5%) that receive more than 100 total citations. Including these books does not change this point.) Almost half of all university-press books receive more than 20 total citations, while only 13% of SSCI-published articles are cited that frequently (Samuels Reference Samuels2011, table 3). Still, as the histograms suggest, there is considerable inequality in the distribution of citations: the 1,517 university-press books were cited 50,395 times, but the top 10% of books received 44% of this total, while the bottom 50% received only 12.4%.
Let us now explore differences across subfields. I classified every book as fitting in one of political science's four main substantive fields plus methodology, based on the book's title or the publisher's brief description of the book. Table 2 provides the mean number of citations university-press books in different subfields received. Of the four substantive subfields, books published in international relations (IR) have slightly greater impact. Books in American politics and political theory are cited relatively infrequently in peer-reviewed articles compared to books in IR and comparative politics, while books in IR and theory are cited relatively more frequently in other books.
Citations to Books by Subfield (University Presses Only)
The high numbers for the few books published on methodology jumps out of table 2. For example, during the two years for which I gathered book titles, several significant methods books were published, including Event-History Modeling by Box-Steffensmeier and Jones (121 total citations), Politics in Time by Paul Pierson (347 total citations), and Case Studies and Theory Development by Alexander George and Andrew Bennett (371 total citations).
Let us now compare presses against each other. To create a ranking I add together the total citations books receive in articles and books. Before considering the results in table 3, some caveats are necessary. Most obviously, the rankings are sensitive to presses' publishing strategy: some presses care more about profits than citations; some adopt a broad definition of what counts as political science for marketing purposes; and some publish very few books. Each factor may affect a press' average citation count.
Publishers Ranked by Total Citations
For example, Perseus Books published Jean Bethke Elshtain's Just War Against Terror, the Oklahoma University Press (which previously had an academic best-seller in Samuel Huntington's The Third Wave) published Theda Skocpol's Diminished Democracy, and W.W. Norton published Liberalism by Paul Berman, The Great Unraveling by Paul Krugman, and War, Terrorism and Democracy by Benjamin Barber. In contrast, Oxford University Press published the most books of any university press in these two years, and its wide coverage may have hurt its average citation count.
Given these caveats, to what extent do scholars' assessments of publisher quality correlate with the quantitative measures? Although table 1 revealed that books published by university presses are more widely cited on average than books published by non-university presses, table 3 reveals that books published by some university presses do not receive many citations—while books published by non-university presses are sometimes quite influential.
I placed political scientists' subjective rankings of publisher quality, as derived from Garand and Giles (Reference Garand and Giles2011), in the last column of table 3. On the one hand, one might suppose this relationship should be strong—after all, how does a press gain a reputation for quality other than having people frequently cite books that the press publishes? On the other hand, survey-based evaluations could be subject to a selection effect: scholars whose careers depend more on book publishing participate in the survey at greater rates than scholars whose careers depend more on article publishing. Or, perhaps, scholars rank presses they have published with or are more familiar with as “better.”
The Spearman rank correlation statistic between the 48 publishers ranked in Garand and Giles (Reference Garand and Giles2011) that are also listed in table 3 is .42, which is not particularly strong. Although this result could be driven by the particular two-year span chosen, some of the differences in rankings are stark. For example, political scientists only gave W.W. Norton a middling rank in a subjective assessment, but Norton's books were widely cited. Meanwhile, political scientists regard CQ Press fairly highly, but its books were not particularly well cited. As for academic presses, some rise and some fall depending on whether the measure is quantitative or qualitative. For example, Duke University Press performed relatively poorly in Garand and Giles' (Reference Garand and Giles2011) survey, but the few books it published were well cited.
In the end, do the differences in average citation counts clearly distinguish presses from each other? It is hard to say. For nearly every press listed, the standard deviation of the average number of total citations is larger than the average itself. Almost all presses publish books that are infrequently cited, and many publish a few that receive dozens or even hundreds of citations.
The results in table 3 raise questions about what “counts” when assessing “quality” or “impact.” According to one view, the only thing that should matter is whether one's research demonstrably shapes the scholarly conversation. To the extent one accepts that dictum, a subjective assessment of publisher quality is inferior to a citation count. The results in table 3 suggest that there is no cut-and-dry hierarchy of academic publishers; many can help scholars draw attention to their research. If you are interested in having others read, use, and cite your work, you do not necessarily need to publish with Princeton, Cambridge, or Oxford—although not surprisingly, publishing with a university press tends to be a safer bet.
CONCLUSION
Book authors should receive proper credit for their work. Scholarly influence comes in many forms, but—despite its limitations—we often rely on the only quantitative measure we have: the citation count. As such, it is in every scholar's interest to count the citations that books receive.
This article provides an objective assessment of the impact of published books in political science, counting the citations books receive in peer-reviewed articles and in other books. The main findings are as follows. First, the average book published by a university press receives about three times as many citations as the average article indexed in the SSCI. Even books published by non-university presses are cited more often than the average article indexed in the SSCI.
Second, publishing with a university press is more likely, on average, to have an effect on the scholarly conversation, as the average number of citations for a university-press book is about 2.5 times that of a book published by other presses.
Third, counting book citations reveals variations across subfields. Books published in American politics follow the same pattern as articles in American politics (Samuels Reference Samuels2011): they are least-frequently cited relative to books published in other subfields, particularly because books in that subfield are relatively less-frequently cited in peer-reviewed articles. Scholarly communication in American politics is increasingly dominated by article publishing, which seems to be less the case for the other subfields. Meanwhile, the books with the broadest impact on average are those that seek to advance social-science methodology.
Finally, the correlation between scholars' subjective assessment of publisher quality does not closely match a simple citation count. Some presses—both university and other—adopt a strategy of publishing few books with expected high impact and sales, while others more closely hew to the scholarly mission of publishing broadly in a field, even if not all the books have a large medium-term impact.
Counting the citations that books receive in articles as well as in other books provides a comprehensive assessment of book impact and publisher quality and provides a benchmark against which scholars can judge a book's relative importance. The findings in this article suggest that there are many paths to scholarly influence—not just that books receive substantially more citations than articles, but that the quantitative distinction between “top” presses is not as clear-cut as some scholars may imagine.
ACKNOWLEDGMENTS
The author thanks Paul Goren and Kathy Hochstetler for comments, and Claire Atmore, Bethany Birdsall, Alisha Bower, Anna Breen, Annie Gjerde, Laura Modl, and Emma Van Pelt for research assistance.
