Sequential order as an extraneous factor in editorial decision

Abstract

Does the sequential order in which manuscripts are submitted to an academic journal have any effect on the editorial decision? As an extraneous factor, the order of submission has no relation to the manuscript’s content. However, an editor facing a list of new submissions could be subject to decision fatigue or order bias, which would in turn affect the editorial decision. Empirical analysis of nearly 10,000 (first) submissions to a leading academic journal shows that manuscripts which were submitted earlier on a given day were up to 7% more likely to be desk rejected, without any order effect on the likelihood of a rejection after peer review.

Appendix: robustness of the results

Exogeneity of rank and test for specification errors

A possible explanation for the main results is that there is another variable that simultaneously results in a higher rank and a lower quality of submitted manuscript (in turn leading to higher likelihood of desk rejection). In a different context, Dietrich (2008) suggested that geographic location of the submitting authors could lead to a correlation between their position on a list and other factors affecting the (perceived) quality of their work. For example, if the journal editor is based in Kazakhstan, then submissions from Europe or United States, which on average produce manuscripts of ‘higher quality’ (at least in terms of desk rejection probability), will be received late in the day. In this case, there would be a correlation between higher rank and lower likelihood of desk rejection.

The data from Journal X does not have manuscript-specific information, such as the number of authors, their identities (which could be linked to prior work) or location. As a result, it’s not possible to control for a geographical bias in ranking. However, two steps were taken to address this concern.

First, for a small number of manuscripts that were eventually accepted (after multiple stages of review), it is possible to use the submission and acceptance dates to match the computer-generated first submission IDs to the Digital Object Identifiers of published manuscripts. After matching DOI to (accepted) manuscript ID, it’s possible to examine if the sequential rank on the day of first submission has any correlation with some proxies for quality—the publication’s citation count or the number of downloads. Table 5 shows that sequential rank on the first submission day is not a predictor of manuscript quality.

Table 5 Bibliometric measures of (published) manuscript quality and the sequential order on the day of first submission

Second step is to test for a potential specification error using a link test. If the model is properly specified, then the square of the predicted value should have no explanatory power. This is confirmed using ‘linktest’ command in Stata after the estimations, all squared predicted values are insignificant. Prediction tables could not be generated given the fixed effects specification.

Measurement of rank: accuracy of manuscript ID-based approach

The rank variable is imputed from the manuscript ID-based ordering. This approach is a good proxy for the time of submission at long time scales, for example at the annual frequency the correlation between the manuscript ID-based rank and time of submission-based rank is 0.99 for both Pearson and Spearman correlations. However, as the time scale gets shorter the accuracy of this approach decreases. This is caused by the following feature of the editorial software: whenever an author modifies their submission the original manuscript ID is retained but the date of ’first submission’ is updated to the modification date. For example, if an author submitted a manuscript in August of 2010 and was assigned ID ‘JX-001-10’, but decided to modify the submission in January 2011, then their manuscript ID is retained with the ‘new’ submission date in January 2011.

Since the sequential nature of manuscript IDs is exploited as proxy for the submission order, such submission modifications introduce noise into the measurement of rank at high resolution time scales. For example, on a weekly scale, correlation between time of submission-based and manuscript ID-based ranks is 0.92 for Pearson and 0.90 for Spearman. Without knowing the exact submission time it’s not possible to check how accurate this approach is on a daily scale. However, since the correlation decreased by about 10% when the time scale increased by a factor of 50, assuming that the decline in correlation is proportional, the accuracy on a daily scale is estimated to be well above 0.8 for Spearman correlation. This allows using the manuscript ID-based approach as a reasonable proxy for the actual time of submission.

Subject heterogeneity

Submission to different subject fields vary in their volume, some subjects are more popular and as a result have a larger number of editors to handle the submissions. To check whether the order effect varies depending on the number of subject editors, Eq. 1 was modified by interacting the subject field dummies $I_j$ with manuscript rank:

$$\begin{aligned} {\text{Prob}} (x_{ij,t}=1)= \Lambda \left( \alpha + \beta \times I_j \times R_{ij,t}+ \gamma _{j,t} \right) . \end{aligned}$$

(2)

Fig. 3

Marginal effect of a change in rank. Notes this figure shows the marginal effects of rank measures (lines are given in the following order: rank, first rank, last rank); each marginal effect was calculating by running ‘xtlogit, fe’ with one rank measure at a time; the sample includes only subjects with a large number of submissions; thick and thin lines represents 90 and 95% CI, respectively

Figure 3 shows the marginal effects for rank measures over different subjects. There is some heterogeneity in the rank effects, with strong patterns for subjects with 3–5 editors (subjects 6, 8 and 12). Subjects with 1–2 editors have low submission volumes, which can partly explain the large confidence intervals. Also, that the default setting in the editorial software is to sort incoming manuscripts in the order of submission, but an editor can reverse that order with a single click. Individual editors that do not need to coordinate with colleagues could have specific preference in processing the submissions, which would reduce the accuracy of rank measure (e.g. if the editor checks only once or twice a week).

Measurement of rank: backlog accumulation

Each editor is likely to have their own schedule and time preferences, and hence daily rank might be an inaccurate measure of the order in which the manuscripts are viewed due to accumulation of manuscript backlogs. The editor’s typical schedule could vary over the sample period, since the sample spans several academic terms. To visualise the editorial activity, the decisions on all submissions (including revisions) were aggregated to subject field-day level and relative intensity of activity on any particular day was calculated, see Fig. 4. Panel (a) shows that a large number of editors (to process a larger number of submissions) had activity spread more evenly over the working week, panel (b) for the subject with just 1 editor shows more concentrated activity. This approach relies crucially on the number and types of decisions. Panel (c) shows the activity of a specific editor based on data for published manuscripts. This intensity is based only on decisions to accept, and hence unlikely to fully capture all of the journal-related activities.

Fig. 4

Intensity of daily activity for selected subject fields and individual editors. a Subject 8 (5 Editors). b Subject 15 (1 Editor). c Editor A. Notes the figures show imputed work patterns of editors based on dates of decisions; light blue (light grey for black and white print) indicates days with no activity (during weeks with at least one active day), coloured lines (grey for black and white print) correspond to the intensity specified on the legend bar, and white spaces indicate weeks with no activity; in panels a, b the activity is based on information on all submissions (including revisions); in panel c activity is derived from accepted manuscript only (due to data limitation)

Using this decision activity as a proxy for editor’s activity on processing new submissions, manuscript backlogs were estimated by accumulating first submissions during the editor’s inactive days. The resulting manuscript ranking is highly correlated with the daily ranking ($R_i$): Pearson 0.77, Spearman 0.61. It was expected that this adjusted manuscript ranking would improve estimation for single-editor subjects, however the rank variable is not significant if the sample is restricted to single-editor subjects. This could indicate that the decision data is not an accurate reflection of the timing of first submission review by the editors at the single-editor subject areas, so the rank variable is measured with a lot of noise. It could, of course, also indicate that due to the low submission volume the single editor could have more flexibility in integrating submission review into their existing work schedule reducing any decision fatigue or order effect.¹¹

Large number of submissions

During the sample period the journal had three special issues and a call for papers for another one (deadline was beyond the sample period). A potential concern is that the rank effects could be driven by a number of high-quality submissions which were prepared in advance and submitted before the deadline. In contrast, manuscripts of lower quality could be submitted just before or on the deadline. This would introduce correlation between manuscript quality and imputed rank. The data doesn’t allow distinguishing a special issue submission from a regular submission, moreover special issues could include submissions from multiple subjects. To reduce potential bias caused by these episodes of large submission inflows, two sample restrictions were checked. The first sample restriction excluded all submissions in the months containing a special issue deadline (excluding submissions on the deadline day only could bias the sample by excluding potentially lower-quality submissions). The second sample restriction excluded all submissions on days with more than 5 submissions. The results from these estimations are consistent with the main results.

Table 6 Restricting the sample to 2–5 manuscripts per day

Table 7 Removing submissions during months with a deadline for a special issue

Random effects specification

Using random effects specification moderates the magnitude of the coefficients, but sign and significance are similar to the fixed effects specification (Tables 6, 7, 8).

Table 8 Estimation with random effects specification