Skip to main content Accessibility help
×
Home
Hostname: page-component-568f69f84b-5zgkz Total loading time: 0.499 Render date: 2021-09-20T11:17:13.530Z Has data issue: true Feature Flags: { "shouldUseShareProductTool": true, "shouldUseHypothesis": true, "isUnsiloEnabled": true, "metricsAbstractViews": false, "figures": true, "newCiteModal": false, "newCitedByModal": true, "newEcommerce": true, "newUsageEvents": true }

Citation patterns in economics and beyond

Published online by Cambridge University Press:  23 March 2020

Matthias Aistleitner
Affiliation:
Johannes Kepler University Linz, Institute for Comprehensive Analysis of the Economy (ICAE)
Jakob Kapeller*
Affiliation:
University of Duisburg-Essen, Institute for Socioeconomics & Johannes Kepler University Linz, Institute for Comprehensive Analysis of the Economy (ICAE)
Stefan Steinerberger
Affiliation:
Yale University, Department of Mathematics
*Corresponding
*Corresponding author. E-mail: jakob.kapeller@uni-due.de
Rights & Permissions[Opens in a new window]

Argument

In this paper we comparatively explore three claims concerning the disciplinary character of economics by means of citation analysis. The three claims under study are: (1) economics exhibits strong forms of institutional stratification and, as a byproduct, a rather pronounced internal hierarchy; (2) economists strongly conform to institutional incentives; and (3) modern mainstream economics is a largely self-referential intellectual project mostly inaccessible to disciplinary or paradigmatic outsiders. The validity of these claims is assessed by means of an interdisciplinary comparison of citation patterns aiming to identify peculiar characteristics of economic discourse. In doing so, we emphasize that citation data can always be interpreted in different ways, thereby focusing on the contrast between a “cognitive” and an “evaluative” approach towards citation data.

Type
Research Article
Creative Commons
Creative Common License - CCCreative Common License - BY
This is an Open Access article, distributed under the terms of the Creative Commons Attribution licence (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted re-use, distribution, and reproduction in any medium, provided the original work is properly cited.
Copyright
© The Author(s) 2020

1. Introduction

Scientometrics is the attempt to better understand the nature of scientific discourse by employing and analyzing quantitative data as it emerges from scientific exchanges (Mingers & Leydesdorff Reference Mingers and Leydesdorff2015). Analysis of citation data has achieved an especially prominent role, as it can be used not only to describe characteristic patterns within scientific conversations, but also serves as a means for assessing the impact and quality of contributions, outlets, authors, or departments.

In this paper we take different ways of utilizing citation data as starting points to better understand the idiosyncratic character of the discipline of economics: more precisely, we compare citation patterns in economics to those of other academic disciplines and ask how different conceptual understandings of citation data also lead to differing interpretations with regard to the nature and comparative development of academic disciplines. Thereby we make use of two basic, but different, ways of interpreting citation data: on the one hand, we follow what is a standard routine in many academic contexts, namely to interpret citations as an indicator for the quality of the cited publication. We call this the evaluative use of citation data (see Moed et al. Reference Moed, Colledge, Reedijk, Moya-Anegon, Guerrero-Bote, Plume and Amin2012, and, for specific examples in economics, Arrow et al. Reference Arrow, Bernheim, Feldstein, McFadden, Poterba and Solow2011, and Diamond Reference Diamond1989). The second approach we suggest is to interpret the act of mentioning one scientific publication in another as an indicator for some form of “communication” between researchers: the act of citing a paper is understood as a key element in scientific conversation. In such an approach, citation analysis allows for assessing the properties of scientific conversation, which amounts to answering questions like “who talks to whom,” “what are the intellectual foundations of a certain field of research,” or “how do scientists distribute their attention across the whole universe of scientific papers?” Such uses of citation data have traditionally been labeled as cognitive scientometrics (De Solla Price Reference De Solla Price1965; Rip & Courtial Reference Rip and Courtial1984; and, for examples in economics, Hamermesh Reference Hamermesh2018 and Kim et al. Reference Kim, Morse and Zingales2006), which serves as a second and alternative frame of interpretation applied to our quantitative findings. While these two approaches are not necessarily contradictory – as citations often serve as both, a means for communication as well as an indicator of (dis)agreement, – they obviously emphasize different aspects of the scholarly process and, hence, can lead to quite different or even conflicting conclusions when applied to actual data.

In what follows, we evaluate and compare citation data from five academic disciplines and analyze the respective citation patterns. We are especially interested in differences between citation patterns in economics and those in other disciplines, since economics is typically understood as a highly idiosyncratic field of research. There are several reasons for this peculiar character of economics: one such reason is that economists often consider their field as most advanced and, hence, superior compared to the other social sciences (e.g. Fourcade et al. Reference Fourcade, Ollion and Algan2015; Lazear Reference Lazear2000). This attitude is at least partly due to the origin-story of modern economics, which is influenced by the archetype of the natural sciences – a fact that had repercussions on both the historical positioning of the discipline (Samuelson Reference Samuelson1972) as well as the evolution of its prevailing practices (Mirowski Reference Mirowski1989). Other aspects of the peculiarities of economics include differences in research style and method (owing to a focus on mathematical and statistical methods [Morgan Reference Morgan1988]); a stronger degree of internal stratification (Fourcade et al. Reference Fourcade, Ollion and Algan2015; Hodgson & Rothman Reference Hodgson and Rothman1999); its prominent role in the context of political decision-making (Beker Reference Beker2010, Hall Reference Hall1989); or its focus on a single, major theoretical approach as summarized in highly standardized textbooks (Graupe Reference Graupe, van Treeck and Urban2016; Skousen Reference Skousen1997).

In what follows, we take three of these alleged peculiarities of economics as a starting point for understanding interdisciplinary differences in citation patterns. Specifically, we relate to findings (a) that modern mainstream economics shows greater signs of stratification than other fields of research (Bayer & Rouse Reference Bayer and Rouse2016; Fourcade et al. Reference Fourcade, Ollion and Algan2015; Han Reference Han2003; Hodgson & Rothman Reference Hodgson and Rothman1999), (b) that economists are more prone to conform to institutional incentives due to their conceptual focus on instrumental rationality (Bauman & Rose Reference Bauman and Rose2011; Etzioni Reference Etzioni2015; Frank et al. 1993; Reference Frank, Gilovich and Regan1996; Frey et al. Reference Frey, Pommerehne and Gygi1993; Yezer et al. Reference Yezer, Goldfarb and Poppen1996), and (c) that mainstream economic research is autonomous in the sense that it only rarely relies on external input (Fourcade et al. Reference Fourcade, Ollion and Algan2015; Pieters & Baumgartner Reference Pieters and Baumgartner2002) and, hence, is largely inaccessible to paradigmatic or disciplinary outsiders (Dobusch & Kapeller Reference Dobusch and Kapeller2009; Fourcade et al. Reference Fourcade, Ollion and Algan2015; King Reference King2003; Leijonhufvud Reference Leijonhufvud1973). We aim to provide an empirical evaluation of these claims on the peculiarity of economics by comparing citation patterns found in economics with those found in some companion social sciences (sociology, political science, and psychology) as well as in different branches of physics. The main rationale for selecting these fields is that they either cover related subjects and questions (the social sciences) or serve as a historically important conceptual archetype for conducting economic research and composing economic theory (physics). In doing so, we hope to shed some light on the question of whether and to what extent these peculiarities of economics are brought forth by a specific disciplinary culture as opposed to being driven by differences in research effort and quality.

Additionally we argue that economics is of special interest when it comes to questions of institutional power-relations, intellectual stratification, or the distribution of attention within a discipline for at least two reasons: first, we find not only that the discipline of economics is coined by stark internal differentiations and a rather rigid internal hierarchy, but also that these properties affect several dimensions. Examples for this specificity include the popularity and gate-keeping power of specific outlets such as the “Big five” (see Heckman & Moktan Reference Heckman and Moktan2018), the recruitment of officers and editors in the discipline’s most important institutions (Fourcade et al. Reference Fourcade, Ollion and Algan2015), the high degree of institutional concentration in economic publishing (Aigner & Glötzl, in this issue) or the influence of PhD-advisers or co-authors from well-respected institutions on publishing prospects (Colussi Reference Colussi2018; D’Ippoliti Reference D’Ippoliti2018).

Second, we argue that the question of how attention, citations, or resources are distributed is especially delicate in economics, as economics is typically understood as a paradigmatically and politically contested field of research (Lee & Elsner Reference Lee and Elsner2011). This view in turn raises the question of whether the alleged “contestedness” of economics has an impact on its discursive properties, that is, on the way it organizes and reproduces scientific conversations. It is this strong degree of internal stratification in conjunction with the character of economics as a “contested discipline” that motivated our main hypotheses on the peculiarity of economics and the economic discourse.

In what follows, we analyze three main aspects of current economic discourse – disciplinary stratification, compliance with incentives, and conceptual isolation – by means of an interdisciplinary comparison. Thereby, we expect our results to reflect the disciplinary character of economics: we assume to find higher degrees of discursive concentration, a higher responsiveness towards incentives, and a greater propensity to engage in inward-looking or exclusionary styles of research and discourse. However, while such findings would lend additional support to the diagnosis that economics is an exceptionally insular and hierarchical research field (Fourcade et al. Reference Fourcade, Ollion and Algan2015), we also hope to provide some clues on which factors – e.g. disciplinary culture, core theoretical ideas, or paradigmatic prejudices – do contribute to the emergence of these peculiarities of economics as an academic discipline.Footnote 1

2. Concentration

In this section we subsume the comparatively high level of self-reference and the strongly hierarchical character of economics under the heading of concentration to signify that important authors, contributions, departments, and outlets are (more) strongly concentrated in terms of their spatial distribution and their personal as well as professional relations. We argue that one of the main roots of concentration in economics is the highly stratified tertiary education system comprised of a very small number of high-profile schools in economics, which in turn have a tight grip on the most important outlets and professional associations.Footnote 2 The hierarchical structure within the economic discipline has been confirmed empirically in various ways. Medoff, for instance, finds evidence of a “positive institutional Matthew effect” (Medoff Reference Medoff2006, 485) for Harvard University and the University of Chicago leading to a disproportionate amount of peer recognition devoted to contributions by economists affiliated with these institutions. On a more general level, Baghestanian and Popov find a strong association between the reputation of an economists PhD-granting institution and the probability of publishing in a top economic journalFootnote 3 which they call the “Alma mater”-effect.

The Alma mater effect is large in size: a top-ten graduate has a 30 percent chance of publishing in a top-five journal, but if instead he graduates from a top-thirty, his chances decrease to 17 percent, and lowering his Alma Mater ranking further to 100+ lowers his chances to ten percent. (Baghestanian & Popov Reference Baghestanian and Popov2017, 17)

Another finding on “concentration” in economics is due to Hodgson and Rothman who analyze the institutional background and affiliation of the editors and authors of a set of thirty “top” economic journals. They identify a strong dominance of institutions located in the U.S., and, within that subset, a dominance of a small group of elite universities. For instance, in 1995 more than half (54 percent) of the authors of articles published in these “top thirty” journals obtained their PhD degrees at one of twelve prestigious U.S. universities,Footnote 4 while at the same time more than one fifth (21 percent) of authors in said journals were also affiliated with one of these universities (Hodgson & Rothman Reference Hodgson and Rothman1999).

A partial replication of the analysis of Hodgson & Rothmann (Reference Hodgson and Rothman1999) with data from 2014 suggests that this level of “institutional oligopoly” has remained fairly constant over time: Just looking at the subsample consisting of three major journals, the Quarterly Journal of Economics (QJE), the American Economic Review (AER)Footnote 5 and the Journal of Political Economy (JPE) reveals that a majority of authors (55 percent) publishing in these outlets received their PhD degrees at one of the twelve universities mentioned above. In 2014, more than one third (38 percent) of these authors were affiliated with one of these universities at the time of publication (see also Wu Reference Wu2007).

Similar tendencies of concentration can also be found in large-scale analyses of the economic literature. Laband (Reference Laband2013), for instance, analyzes the representation of journals in the reference sections of the 409 most cited economics papers published between 2001-2005: more than half (55 percent) of the references cited by these top-papers came from a group of only twenty journals. Even more strikingly, the proportion of citations devoted to the three journals mentioned above – QJE, JPE and AER – is about one fifth (20.6 percent) of the total references (cf. Laband Reference Laband2013, F250-F251). In light of these observations, it comes as no surprise that a recent analysis of the development of economic “top-journals,” which has been published in the Journal of Economic Literature, was limited to the study of only five journals, adding Econometrica and the Review of Economic Studies to the set of three journals mentioned above (see Card & DellaVigna Reference Card and DellaVigna2013).

Our empirical application takes a closer look at the “Big 5,” i.e. the five core journals covered by Card & DellaVigna (Reference Card and DellaVigna2013), to gain potential insights on tendencies of concentration within economic discourse similar to those already observed in the context of editor- and authorships in top-journals (Hodgson & Rothman Reference Hodgson and Rothman1999) or the recruitment of officers for academic associations (Fourcade et al. Reference Fourcade, Ollion and Algan2015). Specifically, we measure the concentration of attention within top-journals by assessing the degree of self-referentiality within this group of journals to compare the relative openness of top-journals across fields. Figure 1 plots the share of journal self-citations as well as the share of citation flows originating from within the “Big 5” relative to total citations in these journals for the period 2009-2013. Figure 1 conveys our first finding: more than a quarter of all sources cited by the top five economic journals originate are self-referential – they stem from the very same set of journals.

Figure 1. The intra-group citation behavior of five core economic journals. (The authors’ own calculation based on data from Clarivate Analytics.).

In addition to this key observation – that one out of four citations made in this journal sample also originates from within this sample – it can also be inferred from Figure 1, that the average share of self-citations within a top-journal plus the share of citations imported from its four “best buddies” is fairly stable across the top five journals with individual values ranging from 23.8 percent (QJE) to 31.7 percent (EMA).

When comparing the citation pattern found across economics’ top journals to the degree of self-reference in top journals located in other fields – e.g. the “top five” journals in psychology, sociology, political science, and physics (fig. 2) – there is the difficulty of selecting appropriate sets of top journals for these fields to advance a sensible comparison across disciplines. To enhance comparability of the respective “top” samples in each discipline we first harmonized our selection criteria across all fields under study. Specifically, we selected every journal that was ranked in the top 20 of Clarivate Analytics’ Footnote 6 annually published Journal Citation Reports (JCR) for ten consecutive years (2006-2015) to account for the relative long-term position of outlets – a condition inspired by the ranking patterns in economics, where the “Big 5” are constantly within the Top 20 of journals as ranked by the JCR. In the case that more than five journals fit those criteria, the five journals with the highest average rank were selected.

Figure 2. An interdisciplinary comparison on concentration – the intra-group citation behavior of five core journals in (a) psychology (multidisciplinary), (b) sociology, (c) political science, and (d) physics (multidisciplinary). (The authors’ own calculation based on data from Clarivate Analytics.).

In comparison with fig. 1, the results shown in fig. 2 reveal two remarkable findings. Firstly, in the other social science disciplines considered here, the degree of concentration measured as the share of journal self-citations plus the citation flow within top journals is – with average values ranging between 7.9 percent and 12.4 percent of total citations – pending between a third and a half of what we found for economics (27.5 percent). This finding reinforces the diagnosis that economics exhibits a much more pronounced hierarchy, or “pecking-order” (Leijonhufvud Reference Leijonhufvud1973, 328), compared to other social science disciplines. Secondly, this relative high degree of concentration in economics also exceeds the degree of concentration measured within our sample of (multidisciplinary) physics journals (20.3 percent). Hence, although the difference between economics and physics as its conceptual archetype is less pronounced than the one between economics and the other social sciences, there remains a substantial gap even in comparison to the much different and often more narrow discourses in the realm of the natural sciences.

In response to the finding on concentration in physics documented in fig. 2, we acquired data for analyzing a second example in the realm of physics, but, this time, focusing on top journals in very specific research areas to assess the intuition that economics is similar to specialized, instead of general, discourses in physics. In doing so, we found a striking similarity between our novel sample from physics consisting of specifically chosen field journals (Physical Review A-E), see fig. 3, and economics. Although the network structures differ markedly between these two cases – the Physical Reviews are huge journals with an average publication output of more than 3,000 papers per year focusing on very different subfields, the degree of concentration is nearly as high (27.3 percent) as in the economics sample. While the stronger content-based separation of the respective journals into subfields (designated with letters A-EFootnote 7) provides an explanation for a much higher share of self-citations, we also find that the average amount of self-citations within these journals (24.1 percent) is eventually rather close to the average share of attention the “Big 5” in economics devote to themselves (27.5 percent). In sum, the top five journals in economics taken together behave rather similarly to specific field journals within physics in terms of discursive concentration.

Figure 3. Citation patterns in specialized physics journals – the intra-group citation behavior of five core field journals in physics. (The authors’ own calculation is based on data from Clarivate Analytics.).

The result that the “Big 5” in economics behave similarly to more specialized subfields in physics and exhibit correspondingly high rates of concentration is especially striking in the face of an overall decrease in the concentration of cited sources within the academic literature over the past decades (Larivière et al. Reference Larivière, Gingras and Archambault2009). Beside this anachronism, it is quite remarkable that a multi-faceted discipline such as economics, which deals with mathematical, game-theoretical, sociological, historical, philosophical, and psychological components, behaves in the same concentrated manner as, say, Physical Review B, which focuses on condensed matter and material physics. The fact that the relative citation patterns of top multidisciplinary journals in physics (as documented in the lower right panel of fig. 2) shows much less regularity, gives an indication for the tight conceptual and theoretical commitments established in contemporary economics.

While these findings are interesting in themselves, it is worthwhile to further explore how their actual implications depend on the respective frame of reference: in terms of an evaluative interpretation, for instance, the relevant problem is to explain the comparatively strong agglomeration of high-quality articles in such a small number of outlets. A potential conclusion to be drawn from this perspective is that the editors of these journals do excellent work and manage to concentrate most high-quality research in economics in their outlets. This constellation, in turn, makes it easier to locate said contributions.

In contrast, while a cognitive interpretation could take such reasoning into account, it would eventually be forced to ask a broader set of questions and inspect other variables, like the diversity of authors, readers and institutional backgrounds, the role of reputational concerns, or the impact of “Big 5” publications on tenure decisions and related career concerns (Heckman & Moktan Reference Heckman and Moktan2018). In comparison with physics (fig. 3), such a cognitive perspective would necessarily go beyond asserting that editors in economics are simply superior, but would rather ask which properties make economics discursively similar to subfields in modern physics.

Under such premises, the outcome looks quite different: here the findings above reveal that the economic discourse at the very top is to a large extent self-contained and exhibits a strong hierarchical character, while the discourse in sociology and political science can be characterized as more diverse and, correspondingly, less hierarchical and insular. This example illustrates how the interpretation and associated valuation of citations is guided by theoretical presumptions, which effectively decide how to translate numerical into verbal assessments.

3. Reactivity

The second aspect in this paper relates to the phenomenon of reactivity, specifically in connection with scientific research evaluation. Our basic argument is that the introduction of journal rankings in economics in particular and in academia in general has led to a change in and a refinement of the institutional incentives governing the academic sphere.

Notwithstanding the fact that rankings have a long tradition in economics (see, for instance, Diamond Reference Diamond1989; Fusfeld Reference Fusfeld1956; Graves et al. Reference Graves, Marchand and Thompson1982; Liebowitz & Palmer Reference Liebowitz and Palmer1984; or the rankings hosted by RePEc on repec.org), we argue that the importance of journal rankings received a significant boost within the last two decades (see also Kalaitzidakis et al. Reference Kalaitzidakis, Mamuneas and Stengos2011; Stern Reference Stern2013; Zimmermann Reference Zimmermann2013). The journal rankings originally produced by the Institute for Scientific Information (ISI), whose database is today widely known as Clarivate AnalyticsWeb of Science, proved to be pivotal for the ascension of journal rankings in academic assessment and evaluation, which eventually led to visible changes in the disciplinary trajectory of economics at the national level (Lee Reference Lee2007; Bloch Reference Bloch2010; Corsi et al. Reference Corsi, D’Ippoliti and Lucidi2010). As these rankings are imposed onto the scientific field for means of assessment and evaluations of researchers and institutions, they do not come without an impact on the institutional incentives individual researchers face. “There is a powerful feedback between the ranking systems used to assess scientific productivity and the actions of scientists trying to further their careers via these ranking systems” (Braun et al. Reference Braun, Bergstrom, Frey, Osterloh, West, Pendlebury and Rohn2010, 871). In short, the introduction of journal rankings for purposes of evaluation causes reactive behavior invoked by the “authority of numbers” in the sense of Espeland and Sauder, who argue that “[c]asting reactivity as a threat to validity of measures draws attention to the dynamic nature of quantitative authority and its capacity to intervene, sometimes dramatically, with other forms of authority” (Espeland & Sauder Reference Espeland and Sauder2007, 6-7).

The notion of reactivity is a well-known concept in the methodology of the social sciences and refers to instances in which the application of some instrument of investigation triggers a change in the behavior of the investigated subjects or institutions. Such behavioral changes are conceived as a source of bias, which may arise in the course of data-acquisition (especially in circumstances, where data is collected repeatedly). While it is methodologically correct to speak of a bias here in the sense that such a behavioral change is to be considered as an artifact created in the course of observation, it seems important to note that the change in behavior is, eventually, real: although induced by external observation, such changes in behavior can be profound and have far-reaching effects.

In the case of methodologically guided evaluations in general and evaluative scientometrics in particular, reactivity induces actors to anticipate evaluation criteria. This anticipation in turn affects the behavior of the subjects or institutions evaluated according to these criteria (Ferraro et al. Reference Ferraro, Pfeffer and Sutton2005, Ghoshal Reference Ghoshal2005). Such forms of “evaluation bias” have been confirmed empirically, especially for evaluation procedures such as rankings, where Espeland & Sauder’s (Reference Espeland and Sauder2007) seminal study on the reactivity in the context of the US News’ law school ranking stands out (see also Rafols et al. Reference Rafols, Leydesdorff, O’Hare, Nightingale and Stirling2012; Willmott Reference Willmott2011). Another example are international rankings based on indexes such as the World Bank’s Doing Business index or even the prominent Human Development Index used by the United Nations Development Programme (UNDP). Such country rankings may induce reactive behavior of government institutions in favor of the defined ranking criteria.

Whenever the scores of international index rankings are taken literally, the indexes may be poor guides for policies as each link between indicators and scores is noisy and uncertain, but presented as certain. … The belief in accuracy in the presence of inaccuracy, however, may lead to a shift in focus among reformers from what really counts to what the makers of these rankings count. This is the tyranny of international index rankings. (Høyland et al. Reference Høyland, Moene and Willumsen2012, 12)

Therefore, it does not come as a surprise that journal rankings do trigger similar effects when used for large-scale institutional evaluation: Italy, for instance, has recently introduced a quantitative standard for supporting decisions on appointment and promotion within the academic system. While the official purpose of this reform is to serve as an objective way to decrease the amount of nepotism in hiring, one cornerstone is the number of scientific publications as indexed by either Scopus or Web of Science. This has led to various attempts of Italian scholars to have past publications that are not covered by Scopus/WoS included into Scopus/WoS and created a strong incentive to only publish in journals covered by these two indices (Abatemarco & Dell’Anno Reference Abatemarco and Dell’Anno2012). While we observe an overall “climate of perverse incentives” (Edwards & Roy Reference Edwards and Roy2017, 51) and also more blunt forms of reactivity taking the form of simple gaming strategies in academia (Kapeller Reference Kapeller2010; Wilhite & Fong Reference Wilhite and Fong2012; Necker Reference Necker2014), in this paper we want to ask a more nuanced question: how would the introduction of journal rankings and their rise to institutional authority change existing patterns of scientific recognition to impact citation behavior in economics? Taking the classic Mertonian view (Reference Merton1968) that the future recognition depends on past recognition already received (see also Haucap & Muck Reference Haucap and Muck2015), we argue that the introduction of journal rankings as evaluative devices affects the self-reinforcing routines governing the distribution of academic attention by raising the relative importance of journals as compared to specific authors or contributions. As journals gain in visibility in the course of the introduction and utilization of journal rankings, some attention is redistributed from the actual producers and products of scientific activities to their (most important) outlets. And, indeed, Attema et al. claim to have found that U.S. economists “would sacrifice more than half a thumb for an AER publication” (Attema et al. Reference Attema, Brouwer and Van Exel2014, 495).

So far, we have introduced two hypotheses on reactivity, namely that the introduction of journal rankings redistributes prestige in the favor of journals (see also Mcdonald & Kam Reference Mcdonald and Kam2007; Lariviére & Gingras Reference Lariviére and Gingras2010) and that economists intuitively and/or deliberately consider this redistribution of prestige, which eventually leads to a change in recorded citation patterns. Specifically, our theoretical argument implies that we should see the share of citations received by the top fraction of articles within a top journal decrease, while the share of citations going to the least considered fraction of papers is expected to rise as all papers in a given outlet draw on the latter’s prestige (see fig. 4 for a stylized illustration of the expected effect).

Figure 4. Expected effect of a relative increase in journals’ prestige on observed citation patterns.

We would assume to find effects as envisaged in fig. 4 – an increase in the share of the bottom two thirds of articles with a corresponding decrease in the share of top-cited papers – across all disciplines. However, we also expect these effects to be most pronounced in economics as we assume a higher degree of conformity to institutional incentives in economics as compared to other fields.

In what follows, figs. 5 to 7 show our empirical findings on this question. In acquiring these results we analyzed the citation history of all (top-)journals sampled so farFootnote 8 and further extended our sample by acquiring corresponding data for Nature and Science. More specifically, we compare the distribution of citations to papers published between 1981 and 1985 (counting received citations up to 1990) to the distribution of citations to all articles published between 2004 and 2008 (counting citations up to 2013) for all analyzed journals to identify changes in aggregate citation patterns.

Figure 5. Change of citation patterns in top journals: 1980s vs. today. (The authors’ own graph is based on data from Clarivate Analytics.).

Figure 5 shows the results so obtained by providing individual estimates for all disciplines under study. The resulting pattern closely corresponds with our theoretical expectations – a decrease in the share of citations dedicated to the upper limit of the distribution is complemented by a corresponding increase in lower deciles. The discipline of economics indeed shows the biggest effect with an effective decline in the share of the top decile of papers by roughly nine percent. Only the generalist top journals Science and Nature (as shown in fig. 6) as well as top journals in psychology come close to this pattern, while the same effect for the other social science journals – as well as the physics journals under study – is much less pronounced.

Figure 6. Citation patterns in the 1980s and today: An aggregated comparison. (The authors’ own graph is based on data from Clarivate Analytics.).

Note: Published document types such as editorials, book reviews, bibliographies or biographic items were excluded in this analysis.*Includes sociology, political science and psychology (multidisciplinary).**Includes physics multidisciplinary, physics nuclear, physics atomic molecular chemical, physics condensed matter, and physics particles fields.***The Web of Science Category “Multidisciplinary Sciences” Includes journals that cover a broad spectrum of different specialties, such as PLoS ONE, PNAS, Nature, Science etc… The overlap with other categories (e.g. social sciences, physics) is marginal in both periods (8.1 percent and 3.1 percent of all papers respectively).

Figure 7. Publication output in the 1980s and today: An aggregated comparison. (The authors’ own graph based on data from Clarivate Analytics.).

Figures 6 provides a different perspective on the same data by further aggregating results across disciplines: it shows, that the expected pattern is much more pronounced in economics as compared to the average social science or physics papers in our sample. Indeed, the journals coming closest to the pattern found for economics are multidisciplinary journals, which enjoy an high visibility and, hence, command a considerable amount of prestige. Thus, it seems that, in terms of reactivity induced by journal prestige, economics really plays in the league of the world’s most prolific journals, while leaving the mundane world of ordinary research well behind.

Eventually, the question is what to make of this result? Although the alignment between expected citation behavior and actual patterns retrieved is close and the supposed pattern non-trivial, how can we assure that our inter-temporal comparisons are due to the role of journal rankings and not, let’s say, due to other factors, like different research culturesFootnote 9 or simply the increasing size of the academic literature in general? While such alternative explanations are indeed appealing and, at the same time, the possibilities to quantitatively assess qualitative differences across fields are limited, we can confront the most obvious argument in this context, namely the growth of the academic literature in general.

Figure 7 compares the overall output of scientific publications between the two periods, which has increased significantly in all disciplines under consideration. However, the similarity between the citation patterns in economics and the multidisciplinary journals shown in fig. 6 stands somewhat in contrast to different growth rates in these disciplines (+187 percent in economics vs. +51 percent in multidisciplinary sciences). Conversely, the two remaining disciplines, which experience growth rates more similar to economics (+88 percent in the social sciences and +103 percent in physics), both exhibit a similar citation pattern (see again fig. 6). This pattern is, in turn, quite different to the one in economics. As this comparison is inconclusive, it can hardly be argued that the specific differences noted in figs. 5 and 6 are solely due to the different growth rates affecting these fields.

While our argument – that the difference in responsiveness to incentives across disciplines contributes to the differences in citation patterns – is surely of a cognitive origin, an evaluative approach to scientometrics would be able to offer several alternative explanations. For instance, one could argue that the average quality of papers in top journals has increased more strongly over time than the quality of top-cited papers in these journals – and that the very same process is going on faster in economics than in other fields. Again, the interpretation of the observed empirical patterns depends on our preferred theoretical framing as the data hardly put us in a position to assess the relative appropriateness of a particular perspective.

4. Diversity

Our final section deals with an alleged lack of diversity in economics. Diversity is a concept that can be approached from a variety of angles, as, for instance, demographic, disciplinary, or theoretical diversity. A lack of diversity in terms of demography is well documented in economics (e.g. Bayer & Rouse Reference Bayer and Rouse2016). Fourcade et al. recently studied the matter of intellectual diversity in economics from a disciplinary angle (Fourcade et al. Reference Fourcade, Ollion and Algan2015). By analyzing both, attitudes towards interdisciplinary research as well as interdisciplinary citation patterns in economics, sociology, and political science, Fourcade et al. show that economists are comparatively less inclined to import findings from other disciplines and also have less trust towards interdisciplinary approaches and research strategies (ibid.). These findings are highly consistent with past results from data-driven research on citation patterns, which document the solitary character of economics in terms of disciplinary location and age of imported references. Relative to other disciplines from the natural and the social sciences, papers in economics refer with a higher probability to their own specialty (see fig. 8) and the cited contributions are, on average, older than papers cited in other disciplinary contexts (Wallace et al. Reference Wallace, Larivière and Gingras2012). The only discipline in our data exhibiting a higher degree of self-referentiality than economics – astrophysics and astronomy – is coined by a very specific and clearly separable research focus (“outer space”), which may explain the high share of inward-looking citations in this field.Footnote 10 For economics, on the other hand, these results not only point to the comparatively strong imprint of traditional approaches and findings on current research, but also signify a comparatively strong intellectual autarky in current economic discourse.

Figure 8. Citation patterns in economics and other disciplines: a large scale sample. Authors’ own graph based on data from Wallace et al. (Reference Wallace, Larivière and Gingras2012).Footnote 11.

In a similar vein, also Fourcade et al. (Reference Fourcade, Ollion and Algan2015) summarized their findings under the heading of “insularity” to signal an absence of theoretical and conceptual diversity. And indeed, economists often conceive interdisciplinary interaction as a competition rather than cooperation, where disciplines are to be judged with regard to their relative “performance.”

“What is interesting to think about are the terms of trade between economics and all these other disciplines. We are clearly a net exporter to political science and sociology” (Wolfers Reference Wolfers2010, 30).

In what follows, we want to add to this picture by supplying a perspective on conceptual diversity in economics. This question is especially peculiar as economics can be regarded as a “contested discipline” (Lee et al. Reference Lee, McConnell and Dean2010, and Lee & Elsner Reference Lee and Elsner2011). The character of scientific discourse in economics differs from that found in other social sciences, because economics is dominated by a single paradigm – neoclassical economic theory – which significantly shapes academic teaching and economic research.Footnote 12 A thus resulting exclusion of alternative economic approaches, like institutional microeconomics, evolutionary analysis of economic change, or Keynesian macroeconomics, from the “official” economic discourse in further consequence led to the foundation of – nowadays eminent – heterodox economic journals, like the Cambridge Journal of Economics, the Journal of Post Keynesian Economics or the Journal of Economic Issues in the late 1960s and 1970s (King Reference King2003, 134-136). Notwithstanding these efforts for providing alternative economic approaches with adequate institutional foundations, alternative or “heterodox” schools of thought nowadays only constitute a small fraction inside the economics discipline. While heterodox economists are confronted with a series of exclusionary practices and routines (Dobusch & Kapeller Reference Dobusch and Kapeller2009; King Reference King2013), their works also draw on a broader variety of sources and inputs and, hence, exhibit a distinctive, more interdisciplinary communicative pattern (Glötzl & Aigner Reference Glötzl and Aigner2015). Moreover, Glötzl & Aigner (Reference Glötzl and Aigner2015) provide additional evidence for this contestedness of economics by pointing to a “mainstream core – heterodox periphery structure” (see, also, Dobusch & Kapeller Reference Dobusch and Kapeller2012a) in the citation networks associated with different economics departments at the University of Vienna and the Vienna University of Economics and Business, that differ with respect to their openness for alternative and heterodox approaches in economics.

The resulting lack of diversity within the economic discipline has been studied empirically, in particular by analyzing the relative citation flows between 13 high-ranked mainstream and 13 high-ranked heterodox economic journals in two different periods: before 1989-2008 (Dobusch & Kapeller Reference Dobusch and Kapeller2012b) and after the recent financial and economic crisis, 2009-2013 (Aistleitner et al. Reference Aistleitner, Kapeller and Steinerberger2018). The results indicate that the monistic and closed citation behavior exhibited by (highly-ranked) mainstream economic journals remained stable after the crisis, as the mainstream journals continue to devote only minimal attention to heterodox inputs (as is evidenced by the small share of citations – less than 3 percent – imported from heterodox sourcesFootnote 13 – see fig. 9). On the contrary, the heterodox journals themselves exhibit a more balanced citation pattern with import shares from the mainstream sample of 47.6 percent (pre-crisis) and 24.6 percent (post-crisis), where this decrease in part also reflects the fact that heterodox contributions typically cite older papers from the mainstream literature more strongly than recent ones. In addition, this result lends support to Kuhn’s Reference Kuhn1962 prediction, which stated that a dominant paradigm that faces a significant anomaly (the crisis) will strive for adapting already established theses, models, and methods instead of turning to more fundamental revisions.

Figure 9. Diversity and the interaction pattern between economic mainstream and heterodox journals. (a) the discourse between a sample of top 13 orthodox and top 13 heterodox journals (1989-2008; taken from Dobusch & Kapeller Reference Dobusch and Kapeller2012b). (b) replication with a corresponding control group (1989-2008; the authors’ own calculation based on data from Clarivate Analytics). (c) replication for the post-crisis period (2009-2013; the authors’ own calculation based on data from Clarivate Analytics). (d) Analysis of a large-scale journal sample (1969-2013; taken from Aigner & Glötzl, in this issue).Footnote 14

To further assess the robustness of these results, fig. 9 additionally shows results derived from data on a control group and a large-scale sample. The first robustness-check uses a control group to ask whether the comparative neglect of heterodox inputs in mainstream outlets can be explained by the lower ranks of heterodox journals, which supposedly signify a lower overall quality. In composing the control group we used a group of mainstream journals with similar ranking positions than the heterodox sample. The second robustness check analyzes data of a large-scale sample acquired from the Web of Science database to scrutinize the robustness of the observed differences.

By comparing these results to other recent findings on the “insularity” of the economics discipline (Fourcade et al. Reference Fourcade, Ollion and Algan2015), one notices several similarities between an interdisciplinary and an intradisciplinary view on conceptual openness in economics. For once, we find that economics is comparatively closed not only to outside disciplines, but also to theoretical outsiders within the discipline. For another, we observe that these theoretical outsiders share close ties to other branches of the social sciences (e.g. Glötzl & Aigner Reference Glötzl and Aigner2015), and thereby do not resemble the disciplinary isolation practiced by the economic mainstream. A possible, tentative conclusion to be drawn from this is that the interdisciplinary insularity of economics as diagnosed by Fourcade et al. (Reference Fourcade, Ollion and Algan2015) or Gingras and Schinckus (Reference Gingras and Schinckus2012) with respect to econophysics is not primarily due to the idiosyncracies and peculiarities of the subject matter in economics, but rather represents an outgrowth of the dominant economic approach which seems to discount external inputs a priori.

Qualitative examples for illustrating the above line of argument are easily found. One instance is provided by the “Economics of Identity” – a stream of research based on a well-received paper by Akerlof & Kranton (Reference Akerlof and Kranton2000), which gathered more than 800 citations in Web of Science, most of them from the field of economics. In this context, it is interesting to note that the conceptual core of “identity” as introduced in economics – that people have ideas about who they are and try to conform to these ideas in their behavior – overlaps strongly with the meaning of identity as used by social psychologists since, at least, the 1950s. While Akerlof & Kranton (Reference Akerlof and Kranton2000) obviously import this idea into economics and thereby attract much interest within their profession, classic contributions on social identity in other fields, e.g. Tajfel (Reference Tajfel1974) or Ashforth & Mael (Reference Ashforth and Mael1989), remain largely neglected in the economic discourse, although they are widely cited in general. In sum, these two papers have more than 3,000 documented citations in Web of Science, where only 34 of those stem from economics.Footnote 15 Hence, Akerlof & Kranton (Reference Akerlof and Kranton2000) have contributed significantly to increasing the prominence of the psychological concept of identity within economics. It, however, remains difficult to say, whether this increase in prominence is to be attributed to an increase in visibility or legitimacy. In the former interpretation, “economists read narrowly” and Akerlof & Kranton (Reference Akerlof and Kranton2000) simply introduce economists to a hitherto unknown concept – personal identity – that is in turn taken up for further investigation. In the latter interpretation, “only economists do it right” and personal identity as a theoretical concept gains credibility and plausibility exactly because it is being taken up by well-known economists in a very respected outlet. While our results point towards the importance of both factors, visibility as well as legitimacy, it is remarkable that the integration of identity in the economics’ discourse does not spill over to the extra-disciplinary origins of said concept.

Against this backdrop, the relative “superiority” of economics derived from inspecting citation-trade balances – as suggested by Wolfers (Reference Wolfers2010) – can be explained not only by a higher quality of research outputs, but also by a higher degree of group-think and inward-looking behavior within the economics profession, which effectively suppresses the import of ‘foreign’ citations. As we have seen, such an interpretation would be well in line with the average economists stance on the value and merit of extra-disciplinary contributions. Yet again, we find that the two ways of looking at and interpreting citation data lead to very different implications.

5. Conclusion

Most scientometric studies up to now remain committed to a certain perspective – either cognitive or evaluative – which then frames our understanding of the relevant data and has a strong imprint on the interpretation of results and the lessons drawn from these outcomes. In this paper we use the peculiar case of economics as an example for illustrating the partially stark differences arising from these two different points of view.

We have shown how citation patterns in current economics systematically differ from patterns found in other disciplines and thereby further substantiated past results regarding the peculiar disciplinary character of academic economics (e.g. Dobusch & Kapeller Reference Dobusch and Kapeller2009; Fourcade et al. Reference Fourcade, Ollion and Algan2015; Leijonhufvud Reference Leijonhufvud1973; Morgan Reference Morgan1988). We suggested explaining these differences with reference to specific characteristics of economics, namely that economics exhibits a more intense disciplinary stratification (leading to and reinforcing concentration of attention within economics), that economists show greater awareness for institutional incentives (fostering reactive citation behavior) and, finally, that economics is a contested discipline dominated by a single approach (resulting in a lack of openness for external inputs).

These observations thereby align well with core elements of the dominant theoretical approach in economics: in the standard textbook view, competition is objective and leads to superior results when compared to other modes of social organization – hence, a focus on top journals seems natural and legitimate. Similarly, textbooks portray deviations from instrumental rationality and a corresponding neglect of incentives as somehow foolish behavior – hence, reactivity can be easily framed as a rational response to given constraints. Finally, the standard textbook view suggests that scientific progress has, more or less, linear properties – hence, a single, dominant theoretical approach is seen not as a limitation for the field, but, quite on the contrary, as a signifier for the superior status of economics as compared to its neighboring disciplines.

While we found that the data does really lend some support to our main hypotheses, all of our findings can be subjected to different interpretations. Higher concentration of attention might be due to more precise instruments for detecting the quality of manuscripts before publication. What seems like reactivity might be emerging from an otherwise invisible shift in patterns of quality in the underlying literature. And openness and ignorance in economic thinking might simply reflect differences in quality located at the level of more fundamental ideas. Our main message in this regard is to think carefully about one’s preconceptions on what citations really signify, because, as we have shown, these preconceptions eventually determine the implications drawn from seemingly innocent empirical outcomes and results.

Finally, our work also points to several policy implications regarding the link between current practices in research evaluation and the development of the economic discipline. In line with earlier findings on the disciplines’ “contestedness,” our results suggest that the incentives arising from imposing citations as the “currency of our industry” (Coffman et al. Reference Coffman, Niederle and Wilson2017, 3) not only reinforces the risk of marginalizing alternative theoretical approaches within the discipline. It also tends to hamper intellectual exchange of knowledge across its disciplinary borders.

Acknowledgments

The authors want to thank Chenglin Lu for excellent research assistance, Ernest Aigner and Florentin Glötzl for their generous provision of large-scale data on citations in economics, and Vivienne Baldassare for her insightful comments. We also want to thank Matthew L. Wallace, Vincent Larivière and Yves Gingras for their kind provision of large-scale data on interdisciplinary citation behavior in economics and other disciplines. Financial support from the Institute for New Economic Thinking (INET) under grants INO1500038 & INO1500039 (M.A., J.K. & S.S.), from the National Science Foundation (NSF) under grant DMS-1763179 and the Sloan Foundation (both S.S.) is gratefully acknowledged.

Matthias Aistleitner received his undergraduate degree from the Johannes Kepler University Linz (Austria) and is currently a research associate at the Institute for Comprehensive Analysis of the Economy (ICAE) at the same university. His research interests are focused on political economy, economics and philosophy and sociology of economics.

Jakob Kapeller is a professor for socioeconomics at University Duisburg-Essen (Germany) and heads the Institute for Comprehensive Analysis of the Economy (ICAE) at Johannes Kepler University Linz (Austria). His research interests are focused on socio-economic change from a political economy perspective and relate to the history of ideas, the philosophy of the social sciences, distributional issues and heterodox economics.

Stefan Steinerberger received his undergraduate degree from the Johannes Kepler University Linz (Austria) and a PhD in Mathematics from the University of Bonn (Germany). He is currently an Assistant Professor at Yale University and works in Analysis and its Applications

Footnotes

1 All data and code used in this paper is available from the corresponding author’s website (http://www.jakob-kapeller.org/images/data/Aistleitneretal-Replication-CodeData.zip).

2 For example, the prestigious economics departments at the University of Chicago and Harvard University both publish high regarded journals, the Journal of Political Economy (JPE) and the Quarterly Journal of Economics (QJE). Three further high-profile outlets, the American Economic Review (AER), the Journal of Economic Literature (JEL), and the Journal of Economic Perspectives (JEP) are published by the leading professional organization in the field, the American Economic Association (AEA). Within the AEA Leadership, in turn, economists affiliated with one of those departments are dominant: A closer look at the past presidents of the AEA (1886-2016) reveals that a third of them were affiliated with either the University of Chicago or Harvard University (a further quarter at least one of four other prestigious U.S. universities: Princeton University, Yale University, M.I.T., and Stanford University; data retrieved from aeaweb.org). These short examples indicate the dominant position of a specific set of actors, and they also show how prestigious universities, important associations and major journals in economics are institutionally intertwined.

3 Additionally they find, that “‘in-house’ journals favor ‘in-house’ graduates” (Baghestanian & Popov Reference Baghestanian and Popov2017, 17), which obviously leads to a further reinforcement of the stratification mentioned above.

4 The respective universities are: Harvard University, M.I.T., University of Chicago, Stanford University, University of Michigan, University of California Berkeley, Princeton University, Yale University, University of Wisconsin, Columbia University, University of Pennsylvania, and Northwestern University (Hodgson & Rothman Reference Hodgson and Rothman1999, F173)

5 For a more detailed analysis of the AER authorship, see Torgler & Piatti (Reference Torgler and Piatti2013).

6 Before 2016, the JCRs were published by Thomson Reuters.

7 The respective letters stand for specific research fields in physics: (A) atomic, molecular, and optical physics and quantum information, (B) condensed matter physics, materials physics, (C) nuclear physics, (D) particles, fields, gravitation, and cosmology, and (E) statistical, nonlinear, biological, and soft matter physics.

8 Due to some minor data limitations (data for the 1980s does not exist for Sociology of Health and Illness, Political Geography, Annual Review of Political Science, Political Analysis, Nature Physics, and Physical Review E) only 24 of the 30 journals discussed in section 2 enter this analysis.

9 For example, Tsay (Reference Tsay2009) analyzes scientometric data between journals of physics, chemistry and engineering and finds some evidence for differences in citation behavior among scientists in these disciplines, although this comparison is restricted to a period of one year (2002).

10 We want to thank Vivienne Baldassare, Yale Astronomy, for sharing her experience and intuition, which helped us to solidify our interpretation of the result for the astronomy/astrophysics category.

11 We are grateful to Matthew L. Wallace, Vincent Larivière and Yves Gingras for providing the large-scale data necessary to build this figure.

12 We recommend Backhouse (Reference Backhouse2005) or Dobusch & Kapeller (Reference Dobusch and Kapeller2009) for a discussion of the historical roots of this paradigmatic dominance.

13 A more detailed analysis of the data from 1989-2008 furthermore shows that the percentage of citations from the top thirteen heterodox journals exported into mainstream journal literature considered here (2,85% of total references) is driven heavily by statistical outliers: about 80% of these references are caused by only three journals that hold a special position within the economic discourse: On the heterodox side of the sample, the Journal of Economic Behavior and Organization, on the mainstream side of the sample, the Journal of Economic Geography and Economic Geography.

14 We are grateful to Ernest Aigner and Florentin Glötzl for providing the data necessary to perform this analysis.

15 Citing articles from Web of Science Core Collection (accessed October 2018)

References

Abatemarco, Antonio, and Dell’Anno, Roberto. 2012. “The Italian Reform of the Academic Recruitment System: An Appraisal of ANVUR and CUN Benchmarks for Assessing Candidates and Commissioners.Rivista Italiana degli Economisti 3:441480.Google Scholar
Aistleitner, Matthias, Kapeller, Jakob, and Steinerberger, Stefan. 2018. “The Power of Scientometrics and the Development of Economics.Journal of Economic Issues 52(3):816834.CrossRefGoogle Scholar
Akerlof, George A., and Kranton, Rachel E. 2000. “Economics and Identity.The Quarterly Journal of Economics 115(3):715753.CrossRefGoogle Scholar
Arrow, Kenneth J., Bernheim, B. Douglas, Feldstein, Martin S, McFadden, Daniel L, , James MPoterba, , and Solow, Robert M. 2011. “100 Years of the American Economic Review: The Top 20 Articles.The American Economic Review 101(1):18.CrossRefGoogle Scholar
Ashforth, Blake E., and Mael, Fred. 1989. “Social Identity Theory and the Organization.The Academy of Management Review 14(1):2039.CrossRefGoogle Scholar
Attema, Arthur E., Brouwer, Werner B. F, and Van Exel, Job. 2014. “Your Right Arm for a Publication in AER?Economic Inquiry 52(1):495502.CrossRefGoogle Scholar
Backhouse, Roger E. 2005. “The Rise of Free Market Economics: Economists and the Role of the State since 1970.History of Political Economy 37(5):355392.CrossRefGoogle Scholar
Baghestanian, Sascha, and Popov, Sergey V. 2017. “Alma mat(t)er(s): Determinants of Early Career Success in Economics.” Economics Working Papers 17-02. Queens Management School, Queen’s University Belfast.Google Scholar
Bauman, Yoram, and Rose, Elaina. 2011. “Selection or Indoctrination: Why Do Economics Students Donate Less than the Rest?Journal of Economic Behavior & Organization 79(3):318327.CrossRefGoogle Scholar
Bayer, Amanda, and Rouse, Cecilia E. 2016. “Diversity in the Economics Profession: A New Attack on an Old Problem.Journal of Economic Perspectives 30(4):221242.CrossRefGoogle Scholar
Beker, Victor, A. 2010. “On the Economic Crisis and the Crisis of Economics.” Economics Discussion Papers No.2010-18. Kiel Institute for the World Economy. http://www.economics-ejournal.org/economics/discussionpapers/2010-18 (last accessed November 18, 2019)Google Scholar
Bloch, Harry. 2010. “Research Evaluation Down Under: An Outsider’s View from the Inside of the Australian Approach.The American Journal of Economics and Sociology 69(5):15301552.CrossRefGoogle Scholar
Braun, Tibor, Bergstrom, Carl T, Frey, Bruno S, Osterloh, Margit, West, Jevin D, , DavidPendlebury, , and Rohn, Jennifer. 2010. “How to Improve the Use of Metrics.Nature 465(7300):870872.Google Scholar
Card, David, and DellaVigna, Stefano. 2013. “Nine Facts about Top Journals in Economics.Journal of Economic Literature 51(1):144161.CrossRefGoogle Scholar
Coffman, Lucas C., Niederle, Muriel, and Wilson, Alistair J. 2017. “A Proposal to Organize and Promote Replications.The American Economic Review 107(5):4145.CrossRefGoogle Scholar
Colussi, Tommaso. 2018. “Social Ties in Academia: A Friend is a Treasure.The Review of Economics and Statistics 100(1):4550.CrossRefGoogle Scholar
Corsi, Marcella, D’Ippoliti, Carlo, and Lucidi, Federico. 2010. “Pluralism at Risk? Heterodox Economic Approaches and the Evaluation of Economic Research in Italy.The American Journal of Economics and Sociology 69(5):14951529.CrossRefGoogle Scholar
D’Ippoliti, Carlo. 2018. “‘Many-Citedness’: Citations Measure More Than Just Scientific Impact”. INET Working Paper No. 57. https://www.ineteconomics.org/uploads/papers/WP_57-DIppoliti-revised.pdf. (last accessed November 18, 2019).Google Scholar
De Solla Price, Derek J. 1965. “Networks of Scientific Papers.Science 149(3683):510515.CrossRefGoogle Scholar
Diamond, Arthur M. 1989. “The Core Journals of Economics.Current Contents 1:411.Google Scholar
Dobusch, Leonhard, and Kapeller, Jakob. 2009. “Why is Economics not an Evolutionary Science? New Answers to Veblen’s Old Question.Journal of Economic Issues 43(4):867898.CrossRefGoogle Scholar
Dobusch, Leonhard, and Kapeller, Jakob. 2012a. “Heterodox United vs. Mainstream City? Sketching a Framework for Interested Pluralism in Economics.Journal of Economic Issues 46(4):10351057.CrossRefGoogle Scholar
Dobusch, Leonhard, and Kapeller, Jakob. 2012b. “A Guide to Paradigmatic Self-Marginalization – Lessons for Post-Keynesian Economists.Review of Political Economy 24(3):469487.CrossRefGoogle Scholar
Edwards, Marc A., and Roy, Siddhartha. 2017. “Academic Research in the 21st Century: Maintaining Scientific Integrity in a Climate of Perverse Incentives and Hypercompetition.Environmental Engineering Science 34(1):5161.CrossRefGoogle Scholar
Espeland, Wendy N., and Sauder, Michael. 2007. “Rankings and Reactivity: How Public Measures Recreate Social Worlds.American Journal of Sociology 113(1):140.CrossRefGoogle Scholar
Etzioni, Amitai. 2015. “The Moral Effects of Economic Teaching.Sociological Forum 30(1):228233.CrossRefGoogle Scholar
Ferraro, Fabrizio, Pfeffer, Jeffrey, and Sutton, Robert I. 2005. “Economics Language and Assumptions: How Theories Can Become Self-Fulfilling.Academy of Management Review 30(1):824.CrossRefGoogle Scholar
Fourcade, Marion, Ollion, Etienne, and Algan, Yann. 2015. “The Superiority of Economists.Journal of Economic Perspectives 29(1):89114.CrossRefGoogle Scholar
Frank, Robert H., Gilovich, Thomas, and Regan, Dennis T. 1993. “Does Studying Economics Inhibit Cooperation?Journal of Economic Perspectives 7(2):159171.CrossRefGoogle Scholar
Frank, Robert H., Gilovich, Thomas, and Regan, Dennis T. 1996. “Do Economists Make Bad Citizens?Journal of Economic Perspectives 10(1):187192.CrossRefGoogle Scholar
Frey, Bruno S.,Pommerehne, Werner W, and Gygi, Beat. 1993. “Economics Indoctrination or Selection? Some Empirical Results.The Journal of Economic Education 24(3):271281.CrossRefGoogle Scholar
Fusfeld, Daniel R. 1956. “The Program of the American Economic Association Meetings.The American Economic Review 46(4):642644.Google Scholar
Ghoshal, Sumantra. 2005. “Bad Management Theories Are Destroying Good Management Practices.Academy of Management Learning & Education 4(1):7591.CrossRefGoogle Scholar
Gingras, Yves, and Schinckus, Christopher. 2012. “The Institutionalization of Econophysics in the Shadow of Physics.Journal of the History of Economic Thought 34(1):109130.CrossRefGoogle Scholar
Graupe, Silja. 2016. “Der erstarrte Blick. Eine erkenntnistheoretische Kritik der Standardlehrbücher der Volkswirtschaftslehre.In Wirtschaft neu denken. Blinde Flecken der Lehrbuchökonomie, edited by van Treeck, Till, and Urban, Janina, 1829. Berlin: iRights. Media.Google Scholar
Graves, Philip E., Marchand, James R, and Thompson, Randall. 1982. “Economics Departmental Rankings: Research Incentives, Constraints, and Efficiency.The American Economic Review 72(5):11311141.Google Scholar
Glötzl, Florentin, and Aigner, Ernest. 2015. “Pluralism in the Market of Science? A Citation Network Analysis of Economic Research at Universities in Vienna.” Ecological Economic Papers 5. Vienna: WU Vienna University of Economics and Business.Google Scholar
Hall, Peter. A. 1989. The Political Power of Economic Ideas: Keynesianism across Nations. Princeton: Princeton University Press.Google Scholar
Hamermesh, Daniel S. 2018. “Citations in Economics: Measurement, Uses, and Impacts.Journal of Economic Literature 56, no. 1: 115156.CrossRefGoogle Scholar
Han, Shin-Kap. 2003. “Tribal Regimes in Academia: A Comparative Analysis of Market Structure across Disciplines.Social Networks 25(3):251280.CrossRefGoogle Scholar
Haucap, Justus, and Muck, Johannes. 2015. “What Drives the Relevance and Reputation of Economics Journals? An Update from a Survey among Economists.Scientometrics 103(3):849877.CrossRefGoogle Scholar
Heckman, James J., and Moktan, Sidharth. 2018. “Publishing and Promotion in Economics. The Tyranny of the Top Five.” NBER Working Paper No. 25093. http://www.nber.org/papers/w25093 (last accessed November 18, 2019).CrossRefGoogle Scholar
Hodgson, Geoffrey M., and Rothman, Harry. 1999. “The Editors and Authors of Economics Journals: A Case of Institutional Oligopoly?Economic Journal 109(453):F165F186.CrossRefGoogle Scholar
Høyland, Bjørn, Moene, Karl, and Willumsen, Fredrik. 2012. “The tyranny of international index rankings.Journal of Development Economics 97:114.CrossRefGoogle Scholar
Kapeller, Jakob. 2010. “Citation Metrics: Serious Drawbacks, Perverse Incentives and Strategic Options for Heterodox Economics.American Journal of Economics and Sociology 69(5):13761408.CrossRefGoogle Scholar
Kalaitzidakis, Pantelis, Mamuneas, Theofanis, P, and Stengos, Thanasis. 2011. “An Updated Ranking of Academic Journals in Economics.Canadian Journal of Economics 44(4):15251538.CrossRefGoogle Scholar
Kim, E. Han, Morse, Adair, and Zingales, Luigi. 2006. “What Has Mattered to Economics Since 1970?Journal of Economic Perspectives 20(4):189202.CrossRefGoogle Scholar
King, John E. 2003. A History of Post-Keynesian Economics since 1936. Cheltenham: Edward Elgar.Google Scholar
King, John E. 2013. “A Case for Pluralism in Economics”. The Economic and Labour Relations Review 24(1):1731.CrossRefGoogle Scholar
Kuhn, Thomas S. 1962. The Structure of Scientific Revolutions. Chicago: University of Chicago Press.Google Scholar
Laband, David N. 2013. “On the Use and Abuse of Economics Journal Rankings.The Economic Journal 123(570):F223F254.CrossRefGoogle Scholar
Lariviére, Vincent, and Gingras, Yves. 2010. “The Impact Factor’s Matthew Effect: A Natural Experiment in Bibliometrics.Journal of the American Society for Information Science and Technology 61(2):424427.Google Scholar
Larivière, Vincent, Gingras, Yves, and Archambault, Éric. 2009. “The Decline in the Concentration of Citations, 1900–2007.Journal of the Association for Information Science and Technology 60(4):858862.Google Scholar
Lazear, Edward P. 2000. “Economic Imperialism.Quarterly Journal of Economics 115(1):99146.CrossRefGoogle Scholar
Lee, Frederic S. 2007. “The Research Assessment Exercise, the State and the Dominance of Mainstream Economics in British Universities.Cambridge Journal of Economics 31(2):309325.CrossRefGoogle Scholar
Lee, Frederic S, Bruce C. Cronin, McConnell, Scott, and Dean, Erik. 2010. “Research Quality Rankings of Heterodox Economic Journals in a Contested Discipline.The American Journal of Economics and Sociology 69(5):14091452.CrossRefGoogle Scholar
Lee, Frederic S., and Elsner, Wolfram. 2011. Evaluating Economic Research in a Contested Discipline: Ranking, Pluralism, and the Future of Heterodox Economics. New York: John Wiley & Sons.Google Scholar
Leijonhufvud, Axel. 1973. “Life among the Econ.Economic Inquiry 11(3):327337.CrossRefGoogle Scholar
Liebowitz, Stan J., and Palmer, John P. 1984. “Assessing the Relative Impacts of Economics Journals.Journal of Economic Literature 22(1):7788.Google Scholar
Mcdonald, Stuart, and Kam, Jacqueline. 2007. “Ring a Ring o’ Roses: Quality Journals and Gamesmanship in Management Studies.Journal of Management Studies 44(4):640655.CrossRefGoogle Scholar
Medoff, Marshall H. 2006. “Evidence of a Harvard and Chicago Matthew Effect.Journal of Economic Methodology 13(4):485506.CrossRefGoogle Scholar
Merton, Robert K. 1968. “The Matthew Effect in Science.Science 159(3810):5663.CrossRefGoogle Scholar
Mingers, John, and Leydesdorff, Loet. 2015. “A Review of Theory and Practice in Scientometrics.European Journal of Operational Research 246(1):119.CrossRefGoogle Scholar
Mirowski, Philip. 1989. More Heat than Light. Cambridge: Cambridge University Press.CrossRefGoogle Scholar
Moed, Henk F., Colledge, Lisa, Reedijk, Jan, Moya-Anegon, Felix, Guerrero-Bote, Vincente, Plume, Andrew, and Amin, Mayur. 2012. “Citation-based Metrics Are Appropriate Tools in Journal Assessment Provided that They Are Accurate and Used in an Informed Way.Scientometrics 92(2):367376.CrossRefGoogle Scholar
Morgan, Theodore. 1988. “Theory versus Empiricism in Academic Economics: Update and Comparisons.Journal of Economic Perspectives 2(4):159164.CrossRefGoogle Scholar
Necker, Sarah. 2014. “Scientific Misbehavior in Economics.Research Policy 43(10): 17471759.CrossRefGoogle Scholar
Pieters, Rik, and Baumgartner, Hans. 2002. “Who Talks to Whom? Intra- and Interdisciplinary Communication of Economics Journals.Journal of Economic Literature 40(2):483509.CrossRefGoogle Scholar
Rafols, Ismael, Leydesdorff, Loet, O’Hare, Alice, Nightingale, Paul, and Stirling, Andy. 2012. “How Journal Rankings Can Suppress Interdisciplinary Research: A Comparison between Innovation Studies and Business & Management.Research Policy 41(7):12621282.CrossRefGoogle Scholar
Rip, Arie, and Courtial, Jean Pierre. 1984. “Co-Words Maps in Biotechnology: An Example of Cognitive Scientometrics.Scientometrics 6(6):381400.CrossRefGoogle Scholar
Samuelson, Paul A. 1972. “Maximum Principles in Analytical Economics.The American Economic Review 62(3):249262.Google Scholar
Skousen, Mark. 1997. “The Perseverance of Paul Samuelson’s Economics.Journal of Economic Perspectives 11(2):137152.CrossRefGoogle Scholar
Stern, David I. 2013. “Uncertainty Measures for Economics Journal Impact Factors.Journal of Economic Literature 51(1):173189.CrossRefGoogle Scholar
Tajfel, Henri. 1974. “Social Identity and Intergroup Behaviour.Social Science Information 13(2):6593.CrossRefGoogle Scholar
Torgler, Benno, and Piatti, Marco. 2013. A Century of American Economic Review. London: Palgrave.CrossRefGoogle Scholar
Tsay, Ming-Yueh. 2009. “An Analysis and Comparison of Scientometric Data between Journals of Physics, Chemistry and Engineering.Scientometrics 78(2):279293.CrossRefGoogle Scholar
Wallace, Matthew L., Larivière, Vincent, and Gingras, Yves. 2012. “A Small World of Citations? The Influence of Collaboration Networks on Citation Practices.PLoS ONE 7(3):e33339.CrossRefGoogle ScholarPubMed
Wilhite, Allen W., and Fong, Eric A. 2012. “Coercive Citation in Academic Publishing.Science 335(6068):542543.CrossRefGoogle ScholarPubMed
Willmott, Hugh. 2011. “Journal List Fetishism and the Perversion of Scholarship: Reactivity and the ABS List.Organization 18(4):429442.CrossRefGoogle Scholar
Wolfers, Justin. 2010. Interview in Region Focus, Second Quarter 2010, Federal Reserve Bank of Richmond, 13 March 2017 https://www.richmondfed.org/publications/research/econ_focus/2010/q2/full_interview (last accessed November 18, 2019).Google Scholar
Wu, Stephen. 2007. “Recent publishing trends at the AER, JPE and QJE.Applied Economics Letters 14(1):5963.CrossRefGoogle Scholar
Yezer, Anthony M., Goldfarb, Robert S, and Poppen, Paul J. 1996. “Does Studying Economics Discourage Cooperation? Watch What We Do, Not What We Say or How We Play.Journal of Economic Perspectives 10(1):177186.CrossRefGoogle Scholar
Zimmermann, Christian. 2013. “Academic Rankings with RePEc.Econometrics 1:249280.CrossRefGoogle Scholar
Figure 0

Figure 1. The intra-group citation behavior of five core economic journals. (The authors’ own calculation based on data from Clarivate Analytics.).

Figure 1

Figure 2. An interdisciplinary comparison on concentration – the intra-group citation behavior of five core journals in (a) psychology (multidisciplinary), (b) sociology, (c) political science, and (d) physics (multidisciplinary). (The authors’ own calculation based on data from Clarivate Analytics.).

Figure 2

Figure 3. Citation patterns in specialized physics journals – the intra-group citation behavior of five core field journals in physics. (The authors’ own calculation is based on data from Clarivate Analytics.).

Figure 3

Figure 4. Expected effect of a relative increase in journals’ prestige on observed citation patterns.

Figure 4

Figure 5. Change of citation patterns in top journals: 1980s vs. today. (The authors’ own graph is based on data from Clarivate Analytics.).

Figure 5

Figure 6. Citation patterns in the 1980s and today: An aggregated comparison. (The authors’ own graph is based on data from Clarivate Analytics.).

Figure 6

Figure 7. Publication output in the 1980s and today: An aggregated comparison. (The authors’ own graph based on data from Clarivate Analytics.).

Note: Published document types such as editorials, book reviews, bibliographies or biographic items were excluded in this analysis.*Includes sociology, political science and psychology (multidisciplinary).**Includes physics multidisciplinary, physics nuclear, physics atomic molecular chemical, physics condensed matter, and physics particles fields.***The Web of Science Category “Multidisciplinary Sciences” Includes journals that cover a broad spectrum of different specialties, such as PLoS ONE, PNAS, Nature, Science etc… The overlap with other categories (e.g. social sciences, physics) is marginal in both periods (8.1 percent and 3.1 percent of all papers respectively).
Figure 7

Figure 8. Citation patterns in economics and other disciplines: a large scale sample. Authors’ own graph based on data from Wallace et al. (2012).11.

Figure 8

Figure 9. Diversity and the interaction pattern between economic mainstream and heterodox journals. (a) the discourse between a sample of top 13 orthodox and top 13 heterodox journals (1989-2008; taken from Dobusch & Kapeller 2012b). (b) replication with a corresponding control group (1989-2008; the authors’ own calculation based on data from Clarivate Analytics). (c) replication for the post-crisis period (2009-2013; the authors’ own calculation based on data from Clarivate Analytics). (d) Analysis of a large-scale journal sample (1969-2013; taken from Aigner & Glötzl, in this issue).14

You have Access
Open access
7
Cited by

Send article to Kindle

To send this article to your Kindle, first ensure no-reply@cambridge.org is added to your Approved Personal Document E-mail List under your Personal Document Settings on the Manage Your Content and Devices page of your Amazon account. Then enter the ‘name’ part of your Kindle email address below. Find out more about sending to your Kindle. Find out more about sending to your Kindle.

Note you can select to send to either the @free.kindle.com or @kindle.com variations. ‘@free.kindle.com’ emails are free but can only be sent to your device when it is connected to wi-fi. ‘@kindle.com’ emails can be delivered even when you are not connected to wi-fi, but note that service fees apply.

Find out more about the Kindle Personal Document Service.

Citation patterns in economics and beyond
Available formats
×

Send article to Dropbox

To send this article to your Dropbox account, please select one or more formats and confirm that you agree to abide by our usage policies. If this is the first time you use this feature, you will be asked to authorise Cambridge Core to connect with your <service> account. Find out more about sending content to Dropbox.

Citation patterns in economics and beyond
Available formats
×

Send article to Google Drive

To send this article to your Google Drive account, please select one or more formats and confirm that you agree to abide by our usage policies. If this is the first time you use this feature, you will be asked to authorise Cambridge Core to connect with your <service> account. Find out more about sending content to Google Drive.

Citation patterns in economics and beyond
Available formats
×
×

Reply to: Submit a response

Please enter your response.

Your details

Please enter a valid email address.

Conflicting interests

Do you have any conflicting interests? *