Scientometrics is the attempt to better understand the nature of scientific discourse by employing and analyzing quantitative data as it emerges from scientific exchanges (Mingers & Leydesdorff Reference Mingers and Leydesdorff2015). Analysis of citation data has achieved an especially prominent role, as it can be used not only to describe characteristic patterns within scientific conversations, but also serves as a means for assessing the impact and quality of contributions, outlets, authors, or departments.
In this paper we take different ways of utilizing citation data as starting points to better understand the idiosyncratic character of the discipline of economics: more precisely, we compare citation patterns in economics to those of other academic disciplines and ask how different conceptual understandings of citation data also lead to differing interpretations with regard to the nature and comparative development of academic disciplines. Thereby we make use of two basic, but different, ways of interpreting citation data: on the one hand, we follow what is a standard routine in many academic contexts, namely to interpret citations as an indicator for the quality of the cited publication. We call this the evaluative use of citation data (see Moed et al. Reference Moed, Colledge, Reedijk, Moya-Anegon, Guerrero-Bote, Plume and Amin2012, and, for specific examples in economics, Arrow et al. Reference Arrow, Bernheim, Feldstein, McFadden, Poterba and Solow2011, and Diamond Reference Diamond1989). The second approach we suggest is to interpret the act of mentioning one scientific publication in another as an indicator for some form of “communication” between researchers: the act of citing a paper is understood as a key element in scientific conversation. In such an approach, citation analysis allows for assessing the properties of scientific conversation, which amounts to answering questions like “who talks to whom,” “what are the intellectual foundations of a certain field of research,” or “how do scientists distribute their attention across the whole universe of scientific papers?” Such uses of citation data have traditionally been labeled as cognitive scientometrics (De Solla Price Reference De Solla Price1965; Rip & Courtial Reference Rip and Courtial1984; and, for examples in economics, Hamermesh Reference Hamermesh2018 and Kim et al. Reference Kim, Morse and Zingales2006), which serves as a second and alternative frame of interpretation applied to our quantitative findings. While these two approaches are not necessarily contradictory – as citations often serve as both, a means for communication as well as an indicator of (dis)agreement, – they obviously emphasize different aspects of the scholarly process and, hence, can lead to quite different or even conflicting conclusions when applied to actual data.
In what follows, we evaluate and compare citation data from five academic disciplines and analyze the respective citation patterns. We are especially interested in differences between citation patterns in economics and those in other disciplines, since economics is typically understood as a highly idiosyncratic field of research. There are several reasons for this peculiar character of economics: one such reason is that economists often consider their field as most advanced and, hence, superior compared to the other social sciences (e.g. Fourcade et al. Reference Fourcade, Ollion and Algan2015; Lazear Reference Lazear2000). This attitude is at least partly due to the origin-story of modern economics, which is influenced by the archetype of the natural sciences – a fact that had repercussions on both the historical positioning of the discipline (Samuelson Reference Samuelson1972) as well as the evolution of its prevailing practices (Mirowski Reference Mirowski1989). Other aspects of the peculiarities of economics include differences in research style and method (owing to a focus on mathematical and statistical methods [Morgan Reference Morgan1988]); a stronger degree of internal stratification (Fourcade et al. Reference Fourcade, Ollion and Algan2015; Hodgson & Rothman Reference Hodgson and Rothman1999); its prominent role in the context of political decision-making (Beker Reference Beker2010, Hall Reference Hall1989); or its focus on a single, major theoretical approach as summarized in highly standardized textbooks (Graupe Reference Graupe, van Treeck and Urban2016; Skousen Reference Skousen1997).
In what follows, we take three of these alleged peculiarities of economics as a starting point for understanding interdisciplinary differences in citation patterns. Specifically, we relate to findings (a) that modern mainstream economics shows greater signs of stratification than other fields of research (Bayer & Rouse Reference Bayer and Rouse2016; Fourcade et al. Reference Fourcade, Ollion and Algan2015; Han Reference Han2003; Hodgson & Rothman Reference Hodgson and Rothman1999), (b) that economists are more prone to conform to institutional incentives due to their conceptual focus on instrumental rationality (Bauman & Rose Reference Bauman and Rose2011; Etzioni Reference Etzioni2015; Frank et al. 1993; Reference Frank, Gilovich and Regan1996; Frey et al. Reference Frey, Pommerehne and Gygi1993; Yezer et al. Reference Yezer, Goldfarb and Poppen1996), and (c) that mainstream economic research is autonomous in the sense that it only rarely relies on external input (Fourcade et al. Reference Fourcade, Ollion and Algan2015; Pieters & Baumgartner Reference Pieters and Baumgartner2002) and, hence, is largely inaccessible to paradigmatic or disciplinary outsiders (Dobusch & Kapeller Reference Dobusch and Kapeller2009; Fourcade et al. Reference Fourcade, Ollion and Algan2015; King Reference King2003; Leijonhufvud Reference Leijonhufvud1973). We aim to provide an empirical evaluation of these claims on the peculiarity of economics by comparing citation patterns found in economics with those found in some companion social sciences (sociology, political science, and psychology) as well as in different branches of physics. The main rationale for selecting these fields is that they either cover related subjects and questions (the social sciences) or serve as a historically important conceptual archetype for conducting economic research and composing economic theory (physics). In doing so, we hope to shed some light on the question of whether and to what extent these peculiarities of economics are brought forth by a specific disciplinary culture as opposed to being driven by differences in research effort and quality.
Additionally we argue that economics is of special interest when it comes to questions of institutional power-relations, intellectual stratification, or the distribution of attention within a discipline for at least two reasons: first, we find not only that the discipline of economics is coined by stark internal differentiations and a rather rigid internal hierarchy, but also that these properties affect several dimensions. Examples for this specificity include the popularity and gate-keeping power of specific outlets such as the “Big five” (see Heckman & Moktan Reference Heckman and Moktan2018), the recruitment of officers and editors in the discipline’s most important institutions (Fourcade et al. Reference Fourcade, Ollion and Algan2015), the high degree of institutional concentration in economic publishing (Aigner & Glötzl, in this issue) or the influence of PhD-advisers or co-authors from well-respected institutions on publishing prospects (Colussi Reference Colussi2018; D’Ippoliti Reference D’Ippoliti2018).
Second, we argue that the question of how attention, citations, or resources are distributed is especially delicate in economics, as economics is typically understood as a paradigmatically and politically contested field of research (Lee & Elsner Reference Lee and Elsner2011). This view in turn raises the question of whether the alleged “contestedness” of economics has an impact on its discursive properties, that is, on the way it organizes and reproduces scientific conversations. It is this strong degree of internal stratification in conjunction with the character of economics as a “contested discipline” that motivated our main hypotheses on the peculiarity of economics and the economic discourse.
In what follows, we analyze three main aspects of current economic discourse – disciplinary stratification, compliance with incentives, and conceptual isolation – by means of an interdisciplinary comparison. Thereby, we expect our results to reflect the disciplinary character of economics: we assume to find higher degrees of discursive concentration, a higher responsiveness towards incentives, and a greater propensity to engage in inward-looking or exclusionary styles of research and discourse. However, while such findings would lend additional support to the diagnosis that economics is an exceptionally insular and hierarchical research field (Fourcade et al. Reference Fourcade, Ollion and Algan2015), we also hope to provide some clues on which factors – e.g. disciplinary culture, core theoretical ideas, or paradigmatic prejudices – do contribute to the emergence of these peculiarities of economics as an academic discipline.Footnote 1
In this section we subsume the comparatively high level of self-reference and the strongly hierarchical character of economics under the heading of concentration to signify that important authors, contributions, departments, and outlets are (more) strongly concentrated in terms of their spatial distribution and their personal as well as professional relations. We argue that one of the main roots of concentration in economics is the highly stratified tertiary education system comprised of a very small number of high-profile schools in economics, which in turn have a tight grip on the most important outlets and professional associations.Footnote 2 The hierarchical structure within the economic discipline has been confirmed empirically in various ways. Medoff, for instance, finds evidence of a “positive institutional Matthew effect” (Medoff Reference Medoff2006, 485) for Harvard University and the University of Chicago leading to a disproportionate amount of peer recognition devoted to contributions by economists affiliated with these institutions. On a more general level, Baghestanian and Popov find a strong association between the reputation of an economists PhD-granting institution and the probability of publishing in a top economic journalFootnote 3 which they call the “Alma mater”-effect.
The Alma mater effect is large in size: a top-ten graduate has a 30 percent chance of publishing in a top-five journal, but if instead he graduates from a top-thirty, his chances decrease to 17 percent, and lowering his Alma Mater ranking further to 100+ lowers his chances to ten percent. (Baghestanian & Popov Reference Baghestanian and Popov2017, 17)
Another finding on “concentration” in economics is due to Hodgson and Rothman who analyze the institutional background and affiliation of the editors and authors of a set of thirty “top” economic journals. They identify a strong dominance of institutions located in the U.S., and, within that subset, a dominance of a small group of elite universities. For instance, in 1995 more than half (54 percent) of the authors of articles published in these “top thirty” journals obtained their PhD degrees at one of twelve prestigious U.S. universities,Footnote 4 while at the same time more than one fifth (21 percent) of authors in said journals were also affiliated with one of these universities (Hodgson & Rothman Reference Hodgson and Rothman1999).
A partial replication of the analysis of Hodgson & Rothmann (Reference Hodgson and Rothman1999) with data from 2014 suggests that this level of “institutional oligopoly” has remained fairly constant over time: Just looking at the subsample consisting of three major journals, the Quarterly Journal of Economics (QJE), the American Economic Review (AER)Footnote 5 and the Journal of Political Economy (JPE) reveals that a majority of authors (55 percent) publishing in these outlets received their PhD degrees at one of the twelve universities mentioned above. In 2014, more than one third (38 percent) of these authors were affiliated with one of these universities at the time of publication (see also Wu Reference Wu2007).
Similar tendencies of concentration can also be found in large-scale analyses of the economic literature. Laband (Reference Laband2013), for instance, analyzes the representation of journals in the reference sections of the 409 most cited economics papers published between 2001-2005: more than half (55 percent) of the references cited by these top-papers came from a group of only twenty journals. Even more strikingly, the proportion of citations devoted to the three journals mentioned above – QJE, JPE and AER – is about one fifth (20.6 percent) of the total references (cf. Laband Reference Laband2013, F250-F251). In light of these observations, it comes as no surprise that a recent analysis of the development of economic “top-journals,” which has been published in the Journal of Economic Literature, was limited to the study of only five journals, adding Econometrica and the Review of Economic Studies to the set of three journals mentioned above (see Card & DellaVigna Reference Card and DellaVigna2013).
Our empirical application takes a closer look at the “Big 5,” i.e. the five core journals covered by Card & DellaVigna (Reference Card and DellaVigna2013), to gain potential insights on tendencies of concentration within economic discourse similar to those already observed in the context of editor- and authorships in top-journals (Hodgson & Rothman Reference Hodgson and Rothman1999) or the recruitment of officers for academic associations (Fourcade et al. Reference Fourcade, Ollion and Algan2015). Specifically, we measure the concentration of attention within top-journals by assessing the degree of self-referentiality within this group of journals to compare the relative openness of top-journals across fields. Figure 1 plots the share of journal self-citations as well as the share of citation flows originating from within the “Big 5” relative to total citations in these journals for the period 2009-2013. Figure 1 conveys our first finding: more than a quarter of all sources cited by the top five economic journals originate are self-referential – they stem from the very same set of journals.
In addition to this key observation – that one out of four citations made in this journal sample also originates from within this sample – it can also be inferred from Figure 1, that the average share of self-citations within a top-journal plus the share of citations imported from its four “best buddies” is fairly stable across the top five journals with individual values ranging from 23.8 percent (QJE) to 31.7 percent (EMA).
When comparing the citation pattern found across economics’ top journals to the degree of self-reference in top journals located in other fields – e.g. the “top five” journals in psychology, sociology, political science, and physics (fig. 2) – there is the difficulty of selecting appropriate sets of top journals for these fields to advance a sensible comparison across disciplines. To enhance comparability of the respective “top” samples in each discipline we first harmonized our selection criteria across all fields under study. Specifically, we selected every journal that was ranked in the top 20 of Clarivate Analytics’ Footnote 6 annually published Journal Citation Reports (JCR) for ten consecutive years (2006-2015) to account for the relative long-term position of outlets – a condition inspired by the ranking patterns in economics, where the “Big 5” are constantly within the Top 20 of journals as ranked by the JCR. In the case that more than five journals fit those criteria, the five journals with the highest average rank were selected.
In comparison with fig. 1, the results shown in fig. 2 reveal two remarkable findings. Firstly, in the other social science disciplines considered here, the degree of concentration measured as the share of journal self-citations plus the citation flow within top journals is – with average values ranging between 7.9 percent and 12.4 percent of total citations – pending between a third and a half of what we found for economics (27.5 percent). This finding reinforces the diagnosis that economics exhibits a much more pronounced hierarchy, or “pecking-order” (Leijonhufvud Reference Leijonhufvud1973, 328), compared to other social science disciplines. Secondly, this relative high degree of concentration in economics also exceeds the degree of concentration measured within our sample of (multidisciplinary) physics journals (20.3 percent). Hence, although the difference between economics and physics as its conceptual archetype is less pronounced than the one between economics and the other social sciences, there remains a substantial gap even in comparison to the much different and often more narrow discourses in the realm of the natural sciences.
In response to the finding on concentration in physics documented in fig. 2, we acquired data for analyzing a second example in the realm of physics, but, this time, focusing on top journals in very specific research areas to assess the intuition that economics is similar to specialized, instead of general, discourses in physics. In doing so, we found a striking similarity between our novel sample from physics consisting of specifically chosen field journals (Physical Review A-E), see fig. 3, and economics. Although the network structures differ markedly between these two cases – the Physical Reviews are huge journals with an average publication output of more than 3,000 papers per year focusing on very different subfields, the degree of concentration is nearly as high (27.3 percent) as in the economics sample. While the stronger content-based separation of the respective journals into subfields (designated with letters A-EFootnote 7) provides an explanation for a much higher share of self-citations, we also find that the average amount of self-citations within these journals (24.1 percent) is eventually rather close to the average share of attention the “Big 5” in economics devote to themselves (27.5 percent). In sum, the top five journals in economics taken together behave rather similarly to specific field journals within physics in terms of discursive concentration.
The result that the “Big 5” in economics behave similarly to more specialized subfields in physics and exhibit correspondingly high rates of concentration is especially striking in the face of an overall decrease in the concentration of cited sources within the academic literature over the past decades (Larivière et al. Reference Larivière, Gingras and Archambault2009). Beside this anachronism, it is quite remarkable that a multi-faceted discipline such as economics, which deals with mathematical, game-theoretical, sociological, historical, philosophical, and psychological components, behaves in the same concentrated manner as, say, Physical Review B, which focuses on condensed matter and material physics. The fact that the relative citation patterns of top multidisciplinary journals in physics (as documented in the lower right panel of fig. 2) shows much less regularity, gives an indication for the tight conceptual and theoretical commitments established in contemporary economics.
While these findings are interesting in themselves, it is worthwhile to further explore how their actual implications depend on the respective frame of reference: in terms of an evaluative interpretation, for instance, the relevant problem is to explain the comparatively strong agglomeration of high-quality articles in such a small number of outlets. A potential conclusion to be drawn from this perspective is that the editors of these journals do excellent work and manage to concentrate most high-quality research in economics in their outlets. This constellation, in turn, makes it easier to locate said contributions.
In contrast, while a cognitive interpretation could take such reasoning into account, it would eventually be forced to ask a broader set of questions and inspect other variables, like the diversity of authors, readers and institutional backgrounds, the role of reputational concerns, or the impact of “Big 5” publications on tenure decisions and related career concerns (Heckman & Moktan Reference Heckman and Moktan2018). In comparison with physics (fig. 3), such a cognitive perspective would necessarily go beyond asserting that editors in economics are simply superior, but would rather ask which properties make economics discursively similar to subfields in modern physics.
Under such premises, the outcome looks quite different: here the findings above reveal that the economic discourse at the very top is to a large extent self-contained and exhibits a strong hierarchical character, while the discourse in sociology and political science can be characterized as more diverse and, correspondingly, less hierarchical and insular. This example illustrates how the interpretation and associated valuation of citations is guided by theoretical presumptions, which effectively decide how to translate numerical into verbal assessments.
The second aspect in this paper relates to the phenomenon of reactivity, specifically in connection with scientific research evaluation. Our basic argument is that the introduction of journal rankings in economics in particular and in academia in general has led to a change in and a refinement of the institutional incentives governing the academic sphere.
Notwithstanding the fact that rankings have a long tradition in economics (see, for instance, Diamond Reference Diamond1989; Fusfeld Reference Fusfeld1956; Graves et al. Reference Graves, Marchand and Thompson1982; Liebowitz & Palmer Reference Liebowitz and Palmer1984; or the rankings hosted by RePEc on repec.org), we argue that the importance of journal rankings received a significant boost within the last two decades (see also Kalaitzidakis et al. Reference Kalaitzidakis, Mamuneas and Stengos2011; Stern Reference Stern2013; Zimmermann Reference Zimmermann2013). The journal rankings originally produced by the Institute for Scientific Information (ISI), whose database is today widely known as Clarivate Analytics’ Web of Science, proved to be pivotal for the ascension of journal rankings in academic assessment and evaluation, which eventually led to visible changes in the disciplinary trajectory of economics at the national level (Lee Reference Lee2007; Bloch Reference Bloch2010; Corsi et al. Reference Corsi, D’Ippoliti and Lucidi2010). As these rankings are imposed onto the scientific field for means of assessment and evaluations of researchers and institutions, they do not come without an impact on the institutional incentives individual researchers face. “There is a powerful feedback between the ranking systems used to assess scientific productivity and the actions of scientists trying to further their careers via these ranking systems” (Braun et al. Reference Braun, Bergstrom, Frey, Osterloh, West, Pendlebury and Rohn2010, 871). In short, the introduction of journal rankings for purposes of evaluation causes reactive behavior invoked by the “authority of numbers” in the sense of Espeland and Sauder, who argue that “[c]asting reactivity as a threat to validity of measures draws attention to the dynamic nature of quantitative authority and its capacity to intervene, sometimes dramatically, with other forms of authority” (Espeland & Sauder Reference Espeland and Sauder2007, 6-7).
The notion of reactivity is a well-known concept in the methodology of the social sciences and refers to instances in which the application of some instrument of investigation triggers a change in the behavior of the investigated subjects or institutions. Such behavioral changes are conceived as a source of bias, which may arise in the course of data-acquisition (especially in circumstances, where data is collected repeatedly). While it is methodologically correct to speak of a bias here in the sense that such a behavioral change is to be considered as an artifact created in the course of observation, it seems important to note that the change in behavior is, eventually, real: although induced by external observation, such changes in behavior can be profound and have far-reaching effects.
In the case of methodologically guided evaluations in general and evaluative scientometrics in particular, reactivity induces actors to anticipate evaluation criteria. This anticipation in turn affects the behavior of the subjects or institutions evaluated according to these criteria (Ferraro et al. Reference Ferraro, Pfeffer and Sutton2005, Ghoshal Reference Ghoshal2005). Such forms of “evaluation bias” have been confirmed empirically, especially for evaluation procedures such as rankings, where Espeland & Sauder’s (Reference Espeland and Sauder2007) seminal study on the reactivity in the context of the US News’ law school ranking stands out (see also Rafols et al. Reference Rafols, Leydesdorff, O’Hare, Nightingale and Stirling2012; Willmott Reference Willmott2011). Another example are international rankings based on indexes such as the World Bank’s Doing Business index or even the prominent Human Development Index used by the United Nations Development Programme (UNDP). Such country rankings may induce reactive behavior of government institutions in favor of the defined ranking criteria.
Whenever the scores of international index rankings are taken literally, the indexes may be poor guides for policies as each link between indicators and scores is noisy and uncertain, but presented as certain. … The belief in accuracy in the presence of inaccuracy, however, may lead to a shift in focus among reformers from what really counts to what the makers of these rankings count. This is the tyranny of international index rankings. (Høyland et al. Reference Høyland, Moene and Willumsen2012, 12)
Therefore, it does not come as a surprise that journal rankings do trigger similar effects when used for large-scale institutional evaluation: Italy, for instance, has recently introduced a quantitative standard for supporting decisions on appointment and promotion within the academic system. While the official purpose of this reform is to serve as an objective way to decrease the amount of nepotism in hiring, one cornerstone is the number of scientific publications as indexed by either Scopus or Web of Science. This has led to various attempts of Italian scholars to have past publications that are not covered by Scopus/WoS included into Scopus/WoS and created a strong incentive to only publish in journals covered by these two indices (Abatemarco & Dell’Anno Reference Abatemarco and Dell’Anno2012). While we observe an overall “climate of perverse incentives” (Edwards & Roy Reference Edwards and Roy2017, 51) and also more blunt forms of reactivity taking the form of simple gaming strategies in academia (Kapeller Reference Kapeller2010; Wilhite & Fong Reference Wilhite and Fong2012; Necker Reference Necker2014), in this paper we want to ask a more nuanced question: how would the introduction of journal rankings and their rise to institutional authority change existing patterns of scientific recognition to impact citation behavior in economics? Taking the classic Mertonian view (Reference Merton1968) that the future recognition depends on past recognition already received (see also Haucap & Muck Reference Haucap and Muck2015), we argue that the introduction of journal rankings as evaluative devices affects the self-reinforcing routines governing the distribution of academic attention by raising the relative importance of journals as compared to specific authors or contributions. As journals gain in visibility in the course of the introduction and utilization of journal rankings, some attention is redistributed from the actual producers and products of scientific activities to their (most important) outlets. And, indeed, Attema et al. claim to have found that U.S. economists “would sacrifice more than half a thumb for an AER publication” (Attema et al. Reference Attema, Brouwer and Van Exel2014, 495).
So far, we have introduced two hypotheses on reactivity, namely that the introduction of journal rankings redistributes prestige in the favor of journals (see also Mcdonald & Kam Reference Mcdonald and Kam2007; Lariviére & Gingras Reference Lariviére and Gingras2010) and that economists intuitively and/or deliberately consider this redistribution of prestige, which eventually leads to a change in recorded citation patterns. Specifically, our theoretical argument implies that we should see the share of citations received by the top fraction of articles within a top journal decrease, while the share of citations going to the least considered fraction of papers is expected to rise as all papers in a given outlet draw on the latter’s prestige (see fig. 4 for a stylized illustration of the expected effect).
We would assume to find effects as envisaged in fig. 4 – an increase in the share of the bottom two thirds of articles with a corresponding decrease in the share of top-cited papers – across all disciplines. However, we also expect these effects to be most pronounced in economics as we assume a higher degree of conformity to institutional incentives in economics as compared to other fields.
In what follows, figs. 5 to 7 show our empirical findings on this question. In acquiring these results we analyzed the citation history of all (top-)journals sampled so farFootnote 8 and further extended our sample by acquiring corresponding data for Nature and Science. More specifically, we compare the distribution of citations to papers published between 1981 and 1985 (counting received citations up to 1990) to the distribution of citations to all articles published between 2004 and 2008 (counting citations up to 2013) for all analyzed journals to identify changes in aggregate citation patterns.
Figure 5 shows the results so obtained by providing individual estimates for all disciplines under study. The resulting pattern closely corresponds with our theoretical expectations – a decrease in the share of citations dedicated to the upper limit of the distribution is complemented by a corresponding increase in lower deciles. The discipline of economics indeed shows the biggest effect with an effective decline in the share of the top decile of papers by roughly nine percent. Only the generalist top journals Science and Nature (as shown in fig. 6) as well as top journals in psychology come close to this pattern, while the same effect for the other social science journals – as well as the physics journals under study – is much less pronounced.
Figures 6 provides a different perspective on the same data by further aggregating results across disciplines: it shows, that the expected pattern is much more pronounced in economics as compared to the average social science or physics papers in our sample. Indeed, the journals coming closest to the pattern found for economics are multidisciplinary journals, which enjoy an high visibility and, hence, command a considerable amount of prestige. Thus, it seems that, in terms of reactivity induced by journal prestige, economics really plays in the league of the world’s most prolific journals, while leaving the mundane world of ordinary research well behind.
Eventually, the question is what to make of this result? Although the alignment between expected citation behavior and actual patterns retrieved is close and the supposed pattern non-trivial, how can we assure that our inter-temporal comparisons are due to the role of journal rankings and not, let’s say, due to other factors, like different research culturesFootnote 9 or simply the increasing size of the academic literature in general? While such alternative explanations are indeed appealing and, at the same time, the possibilities to quantitatively assess qualitative differences across fields are limited, we can confront the most obvious argument in this context, namely the growth of the academic literature in general.
Figure 7 compares the overall output of scientific publications between the two periods, which has increased significantly in all disciplines under consideration. However, the similarity between the citation patterns in economics and the multidisciplinary journals shown in fig. 6 stands somewhat in contrast to different growth rates in these disciplines (+187 percent in economics vs. +51 percent in multidisciplinary sciences). Conversely, the two remaining disciplines, which experience growth rates more similar to economics (+88 percent in the social sciences and +103 percent in physics), both exhibit a similar citation pattern (see again fig. 6). This pattern is, in turn, quite different to the one in economics. As this comparison is inconclusive, it can hardly be argued that the specific differences noted in figs. 5 and 6 are solely due to the different growth rates affecting these fields.
While our argument – that the difference in responsiveness to incentives across disciplines contributes to the differences in citation patterns – is surely of a cognitive origin, an evaluative approach to scientometrics would be able to offer several alternative explanations. For instance, one could argue that the average quality of papers in top journals has increased more strongly over time than the quality of top-cited papers in these journals – and that the very same process is going on faster in economics than in other fields. Again, the interpretation of the observed empirical patterns depends on our preferred theoretical framing as the data hardly put us in a position to assess the relative appropriateness of a particular perspective.
Our final section deals with an alleged lack of diversity in economics. Diversity is a concept that can be approached from a variety of angles, as, for instance, demographic, disciplinary, or theoretical diversity. A lack of diversity in terms of demography is well documented in economics (e.g. Bayer & Rouse Reference Bayer and Rouse2016). Fourcade et al. recently studied the matter of intellectual diversity in economics from a disciplinary angle (Fourcade et al. Reference Fourcade, Ollion and Algan2015). By analyzing both, attitudes towards interdisciplinary research as well as interdisciplinary citation patterns in economics, sociology, and political science, Fourcade et al. show that economists are comparatively less inclined to import findings from other disciplines and also have less trust towards interdisciplinary approaches and research strategies (ibid.). These findings are highly consistent with past results from data-driven research on citation patterns, which document the solitary character of economics in terms of disciplinary location and age of imported references. Relative to other disciplines from the natural and the social sciences, papers in economics refer with a higher probability to their own specialty (see fig. 8) and the cited contributions are, on average, older than papers cited in other disciplinary contexts (Wallace et al. Reference Wallace, Larivière and Gingras2012). The only discipline in our data exhibiting a higher degree of self-referentiality than economics – astrophysics and astronomy – is coined by a very specific and clearly separable research focus (“outer space”), which may explain the high share of inward-looking citations in this field.Footnote 10 For economics, on the other hand, these results not only point to the comparatively strong imprint of traditional approaches and findings on current research, but also signify a comparatively strong intellectual autarky in current economic discourse.
In a similar vein, also Fourcade et al. (Reference Fourcade, Ollion and Algan2015) summarized their findings under the heading of “insularity” to signal an absence of theoretical and conceptual diversity. And indeed, economists often conceive interdisciplinary interaction as a competition rather than cooperation, where disciplines are to be judged with regard to their relative “performance.”
“What is interesting to think about are the terms of trade between economics and all these other disciplines. We are clearly a net exporter to political science and sociology” (Wolfers Reference Wolfers2010, 30).
In what follows, we want to add to this picture by supplying a perspective on conceptual diversity in economics. This question is especially peculiar as economics can be regarded as a “contested discipline” (Lee et al. Reference Lee, McConnell and Dean2010, and Lee & Elsner Reference Lee and Elsner2011). The character of scientific discourse in economics differs from that found in other social sciences, because economics is dominated by a single paradigm – neoclassical economic theory – which significantly shapes academic teaching and economic research.Footnote 12 A thus resulting exclusion of alternative economic approaches, like institutional microeconomics, evolutionary analysis of economic change, or Keynesian macroeconomics, from the “official” economic discourse in further consequence led to the foundation of – nowadays eminent – heterodox economic journals, like the Cambridge Journal of Economics, the Journal of Post Keynesian Economics or the Journal of Economic Issues in the late 1960s and 1970s (King Reference King2003, 134-136). Notwithstanding these efforts for providing alternative economic approaches with adequate institutional foundations, alternative or “heterodox” schools of thought nowadays only constitute a small fraction inside the economics discipline. While heterodox economists are confronted with a series of exclusionary practices and routines (Dobusch & Kapeller Reference Dobusch and Kapeller2009; King Reference King2013), their works also draw on a broader variety of sources and inputs and, hence, exhibit a distinctive, more interdisciplinary communicative pattern (Glötzl & Aigner Reference Glötzl and Aigner2015). Moreover, Glötzl & Aigner (Reference Glötzl and Aigner2015) provide additional evidence for this contestedness of economics by pointing to a “mainstream core – heterodox periphery structure” (see, also, Dobusch & Kapeller Reference Dobusch and Kapeller2012a) in the citation networks associated with different economics departments at the University of Vienna and the Vienna University of Economics and Business, that differ with respect to their openness for alternative and heterodox approaches in economics.
The resulting lack of diversity within the economic discipline has been studied empirically, in particular by analyzing the relative citation flows between 13 high-ranked mainstream and 13 high-ranked heterodox economic journals in two different periods: before 1989-2008 (Dobusch & Kapeller Reference Dobusch and Kapeller2012b) and after the recent financial and economic crisis, 2009-2013 (Aistleitner et al. Reference Aistleitner, Kapeller and Steinerberger2018). The results indicate that the monistic and closed citation behavior exhibited by (highly-ranked) mainstream economic journals remained stable after the crisis, as the mainstream journals continue to devote only minimal attention to heterodox inputs (as is evidenced by the small share of citations – less than 3 percent – imported from heterodox sourcesFootnote 13 – see fig. 9). On the contrary, the heterodox journals themselves exhibit a more balanced citation pattern with import shares from the mainstream sample of 47.6 percent (pre-crisis) and 24.6 percent (post-crisis), where this decrease in part also reflects the fact that heterodox contributions typically cite older papers from the mainstream literature more strongly than recent ones. In addition, this result lends support to Kuhn’s Reference Kuhn1962 prediction, which stated that a dominant paradigm that faces a significant anomaly (the crisis) will strive for adapting already established theses, models, and methods instead of turning to more fundamental revisions.
To further assess the robustness of these results, fig. 9 additionally shows results derived from data on a control group and a large-scale sample. The first robustness-check uses a control group to ask whether the comparative neglect of heterodox inputs in mainstream outlets can be explained by the lower ranks of heterodox journals, which supposedly signify a lower overall quality. In composing the control group we used a group of mainstream journals with similar ranking positions than the heterodox sample. The second robustness check analyzes data of a large-scale sample acquired from the Web of Science database to scrutinize the robustness of the observed differences.
By comparing these results to other recent findings on the “insularity” of the economics discipline (Fourcade et al. Reference Fourcade, Ollion and Algan2015), one notices several similarities between an interdisciplinary and an intradisciplinary view on conceptual openness in economics. For once, we find that economics is comparatively closed not only to outside disciplines, but also to theoretical outsiders within the discipline. For another, we observe that these theoretical outsiders share close ties to other branches of the social sciences (e.g. Glötzl & Aigner Reference Glötzl and Aigner2015), and thereby do not resemble the disciplinary isolation practiced by the economic mainstream. A possible, tentative conclusion to be drawn from this is that the interdisciplinary insularity of economics as diagnosed by Fourcade et al. (Reference Fourcade, Ollion and Algan2015) or Gingras and Schinckus (Reference Gingras and Schinckus2012) with respect to econophysics is not primarily due to the idiosyncracies and peculiarities of the subject matter in economics, but rather represents an outgrowth of the dominant economic approach which seems to discount external inputs a priori.
Qualitative examples for illustrating the above line of argument are easily found. One instance is provided by the “Economics of Identity” – a stream of research based on a well-received paper by Akerlof & Kranton (Reference Akerlof and Kranton2000), which gathered more than 800 citations in Web of Science, most of them from the field of economics. In this context, it is interesting to note that the conceptual core of “identity” as introduced in economics – that people have ideas about who they are and try to conform to these ideas in their behavior – overlaps strongly with the meaning of identity as used by social psychologists since, at least, the 1950s. While Akerlof & Kranton (Reference Akerlof and Kranton2000) obviously import this idea into economics and thereby attract much interest within their profession, classic contributions on social identity in other fields, e.g. Tajfel (Reference Tajfel1974) or Ashforth & Mael (Reference Ashforth and Mael1989), remain largely neglected in the economic discourse, although they are widely cited in general. In sum, these two papers have more than 3,000 documented citations in Web of Science, where only 34 of those stem from economics.Footnote 15 Hence, Akerlof & Kranton (Reference Akerlof and Kranton2000) have contributed significantly to increasing the prominence of the psychological concept of identity within economics. It, however, remains difficult to say, whether this increase in prominence is to be attributed to an increase in visibility or legitimacy. In the former interpretation, “economists read narrowly” and Akerlof & Kranton (Reference Akerlof and Kranton2000) simply introduce economists to a hitherto unknown concept – personal identity – that is in turn taken up for further investigation. In the latter interpretation, “only economists do it right” and personal identity as a theoretical concept gains credibility and plausibility exactly because it is being taken up by well-known economists in a very respected outlet. While our results point towards the importance of both factors, visibility as well as legitimacy, it is remarkable that the integration of identity in the economics’ discourse does not spill over to the extra-disciplinary origins of said concept.
Against this backdrop, the relative “superiority” of economics derived from inspecting citation-trade balances – as suggested by Wolfers (Reference Wolfers2010) – can be explained not only by a higher quality of research outputs, but also by a higher degree of group-think and inward-looking behavior within the economics profession, which effectively suppresses the import of ‘foreign’ citations. As we have seen, such an interpretation would be well in line with the average economists stance on the value and merit of extra-disciplinary contributions. Yet again, we find that the two ways of looking at and interpreting citation data lead to very different implications.
Most scientometric studies up to now remain committed to a certain perspective – either cognitive or evaluative – which then frames our understanding of the relevant data and has a strong imprint on the interpretation of results and the lessons drawn from these outcomes. In this paper we use the peculiar case of economics as an example for illustrating the partially stark differences arising from these two different points of view.
We have shown how citation patterns in current economics systematically differ from patterns found in other disciplines and thereby further substantiated past results regarding the peculiar disciplinary character of academic economics (e.g. Dobusch & Kapeller Reference Dobusch and Kapeller2009; Fourcade et al. Reference Fourcade, Ollion and Algan2015; Leijonhufvud Reference Leijonhufvud1973; Morgan Reference Morgan1988). We suggested explaining these differences with reference to specific characteristics of economics, namely that economics exhibits a more intense disciplinary stratification (leading to and reinforcing concentration of attention within economics), that economists show greater awareness for institutional incentives (fostering reactive citation behavior) and, finally, that economics is a contested discipline dominated by a single approach (resulting in a lack of openness for external inputs).
These observations thereby align well with core elements of the dominant theoretical approach in economics: in the standard textbook view, competition is objective and leads to superior results when compared to other modes of social organization – hence, a focus on top journals seems natural and legitimate. Similarly, textbooks portray deviations from instrumental rationality and a corresponding neglect of incentives as somehow foolish behavior – hence, reactivity can be easily framed as a rational response to given constraints. Finally, the standard textbook view suggests that scientific progress has, more or less, linear properties – hence, a single, dominant theoretical approach is seen not as a limitation for the field, but, quite on the contrary, as a signifier for the superior status of economics as compared to its neighboring disciplines.
While we found that the data does really lend some support to our main hypotheses, all of our findings can be subjected to different interpretations. Higher concentration of attention might be due to more precise instruments for detecting the quality of manuscripts before publication. What seems like reactivity might be emerging from an otherwise invisible shift in patterns of quality in the underlying literature. And openness and ignorance in economic thinking might simply reflect differences in quality located at the level of more fundamental ideas. Our main message in this regard is to think carefully about one’s preconceptions on what citations really signify, because, as we have shown, these preconceptions eventually determine the implications drawn from seemingly innocent empirical outcomes and results.
Finally, our work also points to several policy implications regarding the link between current practices in research evaluation and the development of the economic discipline. In line with earlier findings on the disciplines’ “contestedness,” our results suggest that the incentives arising from imposing citations as the “currency of our industry” (Coffman et al. Reference Coffman, Niederle and Wilson2017, 3) not only reinforces the risk of marginalizing alternative theoretical approaches within the discipline. It also tends to hamper intellectual exchange of knowledge across its disciplinary borders.
The authors want to thank Chenglin Lu for excellent research assistance, Ernest Aigner and Florentin Glötzl for their generous provision of large-scale data on citations in economics, and Vivienne Baldassare for her insightful comments. We also want to thank Matthew L. Wallace, Vincent Larivière and Yves Gingras for their kind provision of large-scale data on interdisciplinary citation behavior in economics and other disciplines. Financial support from the Institute for New Economic Thinking (INET) under grants INO1500038 & INO1500039 (M.A., J.K. & S.S.), from the National Science Foundation (NSF) under grant DMS-1763179 and the Sloan Foundation (both S.S.) is gratefully acknowledged.
Matthias Aistleitner received his undergraduate degree from the Johannes Kepler University Linz (Austria) and is currently a research associate at the Institute for Comprehensive Analysis of the Economy (ICAE) at the same university. His research interests are focused on political economy, economics and philosophy and sociology of economics.
Jakob Kapeller is a professor for socioeconomics at University Duisburg-Essen (Germany) and heads the Institute for Comprehensive Analysis of the Economy (ICAE) at Johannes Kepler University Linz (Austria). His research interests are focused on socio-economic change from a political economy perspective and relate to the history of ideas, the philosophy of the social sciences, distributional issues and heterodox economics.
Stefan Steinerberger received his undergraduate degree from the Johannes Kepler University Linz (Austria) and a PhD in Mathematics from the University of Bonn (Germany). He is currently an Assistant Professor at Yale University and works in Analysis and its Applications