The intellectual influence of economic journals: quality versus quantity

Abstract

The evaluation of scientific output has a key role in the allocation of research funds and academic positions. Decisions are often based on quality indicators for academic journals, and over the years, a handful of scoring methods have been proposed for this purpose. Discussing the most prominent methods (de facto standards) we show that they do not distinguish quality from quantity at article level. The systematic bias we find is analytically tractable and implies that the methods are manipulable. We introduce modified methods that correct for this bias, and use them to provide rankings of economic journals. Our methodology is transparent; our results are replicable.

Appendices

Appendices

1.1 Appendix A: The complete ranking of economics journals

Table 2 Modified invariant scores (sc) and ranks (r) of economic journals, 2006–2010

1.2 Appendix B: Data format and source code

We organized the data in three types of files:

• the $m$ file— in this file, for each year $t$, there is a spreadsheet containing the citation matrix $C^t$ where an entry in row $i$, column $j$, is the total number of cites made in year $t$ by articles in journal $j$ to articles in journal $i$ no older than 4 years;

• the $a$ file—in this file, for each year $t$, there is a spreadsheet containing a column with the number of articles $a$ published by each journal in that year;

• the $c$ file—in this file, for each year $t$, there is a spreadsheet containing a column with the total number of citations $c_j$ made by journal $j$ to articles in journals in $J$ no older than 4 years.

To obtain the raw ranking vectors from the above matrices, we used the following code in Wolfram Mathematica 8.0:

The diagonal matrices $A$ and $D_C$ are generated by our code from the data files. Each year we have a different set of journals: for each year the raw score vectors are copied next to the lists of journals and are normalized. The overall ranking is produced by sorting the journals according to their scores.

1.3 Appendix C: Detailed calculations

Writing Equality 7 in detail for the left most and right most terms of the equality, we obtain the following system of equations:

$$\begin{aligned} \left\{ \begin{array}{l} \frac{1}{a_1}[(v_1c_{11}+\dots +v_nc_{1n})-(x_1c_{11}+\dots +x_{j-1}c_{1j-1}+x_{j+1}c_{1j+1}+\dots +x_nc_{1n})] \\ \qquad \quad =\frac{1}{a_1}(v_1c_{11}+\dots +v_nc_{1n})-\delta v_1-\varrho (\Gamma ^{\prime })x_1\\ \vdots \\ \frac{1}{a_j^{\prime }}[(v_1c_{j1}+\dots +v_nc_{jn})-(x_1c_{j1}+\dots +x_{j-1}c_{jj-1}+x_{j+1}c_{jj+1}+\dots +x_nc_{jn})] \\ \qquad \quad = \frac{1}{a_j}(v_1c_{j1}+\dots +v_nc_{jn})-\delta v_j-\varrho (\Gamma ^{\prime })x_j\\ \vdots \\ \frac{1}{a_n}[(v_1c_{n1}+\dots +v_nc_{nn})-(x_1c_{n1}+\dots +x_{j-1}c_{nj-1}+x_{j+1}c_{nj+1}+\dots +x_nc_{nn})] \\ \qquad \quad = \frac{1}{a_n}(v_1c_{n1}+\dots +v_nc_{nn})-\delta v_n-\varrho (\Gamma ^{\prime })x_n \end{array} \right. \end{aligned}$$

After canceling terms and dropping the $j$th row from the system of equations above, we obtain:

$$\begin{aligned} {\left\{ \begin{array}{ll} \frac{1}{a_1}(x_1c_{11}+\dots +x_{j-1}c_{1j-1}+x_{j+1}c_{1j+1}+\dots +x_nc_{1n}) =\delta v_1+\varrho (\Gamma ^{\prime })x_1\\ \vdots \\ \frac{1}{a_{j-1}}(x_1c_{j-11}+\dots +x_{j-1}c_{j-1j-1}+x_{j+1}c_{j-1j+1}+\dots +x_nc_{j-1n}) = \delta v_{j-1}+\varrho (\Gamma ^{\prime })x_{j-1}\\ \frac{1}{a_{j+1}}(x_1c_{j+11}+\dots +x_{j-1}c_{j+1j-1}+x_{j+1}c_{j+1j+1}+\dots +x_nc_{j+1n}) = \delta v_{j+1}+\varrho (\Gamma ^{\prime })x_{j+1}\\ \vdots \\ \frac{1}{a_n}(x_1c_{n1}+\dots +x_{j-1}c_{nj-1}+x_{j+1}c_{nj+1}+\dots +x_nc_{nn}) =\delta v_n+\varrho (\Gamma ^{\prime })x_n \end{array}\right.} \end{aligned}$$

Rewriting the above system of equations using vector and matrix notation yields Eq. 8.