Skip to main content

Editorial 25/4: Electronic Markets on reviewing

Dear readers of Electronic Markets,

The last two editorials reported that Electronic Markets managed to improve its standing in two important rankings: the step from C to B in Jourqual3 (Alt et al. 2015a) and the increase of EM’s impact factor from 0.769 to 0.935 (Alt & Zimmermann 2015). Although many influences are relevant for this development, the processes within the journal are certainly one important variable. As known from the literature on process management, processes link activities performed by multiple actors in a specified sequence to attain an output that creates value for the customers of the process (Österle 1995, 62). From the perspective of an academic journal, multiple actors are involved in the submission, review and publishing process. The ultimate goal is to create value for the academic community, which refers to the advancement of the entire field of digital or electronic business (processes) with meaningful and high-quality contributions (Alt and Österle 2014a).

As in a networked world, the value system of an academic journal is complex and dynamic. Isolating a single variable is difficult and very often one outcome is not only determined by many input factors, but also by itself. For example, high-quality contributions are influenced by good submissions and review processes, but also by the journal’s ranking, which in turn has been calculated by the impact of prior – presumably also high-quality – published articles. Thus, a high impact factor and above-average rankings also attract high-quality research in the future. Not all “design variables” of the journal system are amenable to direct action by the journal itself. At Electronic Markets, this is a big and continuous team effort between the editorial team, the senior, associate and guest editors as well as the editorial board and many additional reviewers. The current organization structure has evolved over the last 25 years: as described in a chronological overview (Alt et al. 2015b, 2) the double-blind review process and the international editorial board was introduced in 1999. In 2008, the role of associate editors and in 2012, the role of senior editors was created to secure quality in view of rising submission volumes. Since special issues have been a key characteristic of Electronic Markets almost from the beginning, the guest editor’s role has also had a longer tradition.

However, with a growing number of submissions and participants the challenge of managing the entire system also increased. Everyone, who is familiar with the academic world, has probably experienced the deficits of long response and/or cycle times as an author, the disappointment about brief and/or unenlightened reviews as an author or editor as well as the problem of meeting the expectations of providing substantial feedback in time as a reviewer or editor. Very often, academics act in many roles for multiple journals as well as conferences at the same time. Although reviewing is at the heart of the academic system, it generates a heavy workload. At present, a total of 3’217 journals are listed only in the SSCI index and if each journal includes four issues per year the number of issues accounts for 12’868 issues per year. Assuming that each issue includes a collection of five articles on average, the number of articles raises to 64’340 and if each article receives an average of two reviews, 128’680 reviews alone are produced for accepted articles. Suppose that only 50 % of all articles are rejected some 257’360 reviews are written, which even precludes the usual revisions in subsequent rounds of review. This very conservative calculation for SSCI journals may also serve to estimate the workload, which amounts to an impressive 772’080 h on the basis of three hours per review. Extending the scope to non-SSCI journals and conferences would easily multiply this effort by several dimensions.

Characteristics of reviewing

Again, the assumption of three hours per review for a journal submission is conservative and often exceeded. This is illustrated when taking a deeper look on what is expected from a good review. Much has been published about writing reviews (Zmud 1998; Lee 1995; Australian National University 2014; van Haarten 2015). Overall, these guidelines suggest that a thorough and constructive assessment of the submitted research comprises three parts.

  • First, a brief summary should reflect the reviewer’s perception of the paper’s scope and content. It serves to clarify the arguments developed in the paper as a basis for the subsequent evaluation and critical discussion. This section should not exceed more than a third of the total review and is also the right spot to inform the review readers about the reviewer’s own expertise.

  • The second part of a review covers the critical discussion of the key issues raised in the article, not only mentioning weaknesses, but also strengths. It is good practice to start from the general, overall judgements and then offer a list of specific, numbered point-by-point comments. Wherever possible, criticism should be supported with evidence from the text or from other writings to help the author(s) to further improve their research.

  • Finally, the review should summarize the overall contribution that the paper provides to the field from a scientific as well as from a practical point of view. In particular, the research not only needs to follow an appropriate methodology to answer the research questions, but also needs to present convincing and novel insights. Contrary to the impression of many submissions, empirical work in the information systems field is a means to attain scientific results, but not the research objective itself. Depending on the final decision, the review should point out necessary revisions or the reasons for rejecting the paper.

Obviously, reviews need to take the nature of the paper into account. While the three points mentioned above apply to regular research papers, other formats, such as position papers, will differ since the development of an own point of view is critical for their quality. Usually, it does not require an evaluation concerning the methodological approach or clarity of the hypotheses, since these are not part of a typical position paper. Regardless of the type of paper, reviewers should be aware of the good practices valued in the world of academic reviewing, which relies on providing feedback to peers:

  • As stated by Ives in 1992 the “appointment to certain editorial boards can be an important academic milestone”. The invitation for reviewing is an appreciation of knowledge since the reviewer is regarded an expert in the area of the submitted research contribution. In turn, reviewers should respond to an invitation in time and only agree if the paper actually fits their expertise and interest. If this is not the case and/or if the reviewer sees difficulties in meeting the review deadline (e.g. because they are overcommitted) the editor should be informed as soon as possible either to extend the review period or to give the editor the opportunity to proceed with the review process by inviting another colleague. However, being a member of an editorial board is always linked with a commitment to dedicate a certain capacity for reviewing – at Electronic Markets an average of three papers per year – and declining invitations repeatedly should make board members reflect on whether they actually fulfill their role.

  • Peer reviewing is based on trust and collegial advice. Regardless if the review process follows a single- or double-blind model, the manuscript itself is a confidential document and needs to be treated as such. Since reviewers might also receive reviews when submitting their own work to a journal or conference, they should provide feedback in a fashion that they would expect from reviewers for their own articles. This implies giving the paper the proper recognition and choosing a positive and constructive attitude in the feedback. Thus, reviewers’ suggestions and comments should be constructive, helpful and insightful to provide the author(s) with valuable guidance on how they might address any perceived shortcomings of their paper.

Each journal should be aware that reviewer resources, in particular of those colleagues that are known for elaborate, high-quality feedback, are scarce. Electronic Markets is well aware of this fact and aims to assist all parties involved as much as possible. First of all, reviewers are important to help the responsible editor make a decision on whether a paper is worth publishing, whether it still needs minor or major revisions or is not (yet) mature enough for publication. The second and major role a reviewer needs to fulfill is to advise the author(s) on improving the quality of their manuscript. Being aware that only a limited number of factors are amenable to influence by the journal itself, Electronic Markets has chosen the following corner points for its processes:

Structural organization

First, the structural organization of Electronic Markets recognizes the shared effort of many participants. As described in more detail in the first issue of volume 25 (Alt et al. 2015b, 3) the editorial team consists of the Editors-in-Chief and the Executive Editor, who oversee the journal processes and the relationship to the publisher. A team of senior and associate editors has responsibility for accepted papers and the selection of reviewers, who may be recruited from the editorial board or the community at large. In addition, guest editors organize the review process for submissions to their special issue. Over the years, a variety of measures has emerged, which are helpful in containing cycle times and improving quality:

  • All submissions to Electronic Markets are subject to an early desk reject check by the Editors-in-Chief. This ensures that papers, which are either out of the journal’s scope or immature in nature, are returned early and will not require any editors and reviewer resources.

  • All submissions are assigned a single responsibility and they are double-checked by editors at the same time. In the case of general research submissions, the Executive Editor invites an Associate Editor, who in turn invites a minimum of two reviewers. Special issue submissions are forwarded to the corresponding guest editor, who coordinates within his/her team the responsibility of the paper and the invitation of reviewers. In addition, one Senior Editor double-checks the decisions and reviews for the submissions of a specific special issue as well as for general research. Position papers differ in that they are managed by the Editors-in-Chief and reviewed by Associate Editors.

  • All major activities of the review process and the related communication are managed via the Editorial Manager review system (Editorial Manager 2015a). This is not only the prerequisite of an efficient and structured process for all parties involved, but also creates a centralized view on reviewers, authors, editors and submissions. From the perspective of process management, the transparency allows calculating cycle times and identifying delays as soon as they occur.

  • All reviews are rated by the responsible editors and each year the editorial team suggests five outstanding reviewers as candidates for the annual outstanding reviewer award at the editorial board meeting (EM 2015a). This is an attempt to return recognition to colleagues, who put a high effort into writing constructive in-depth reviews.

Electronic Markets adheres to the “Code of Conduct and Best Practice Guidelines for Journal Editors” formulated by the Committee of Publication Ethics (COPE). Besides fundamental recommendations and best practices for relations with readers, authors, editors, editorial board members as well as journal owners and publishers, three guidelines apply to editorial and peer review processes (COPE 2015, 6):

  • “Editors should strive to ensure that peer review at their journal is fair, unbiased and timely”. Clearly, this is an ongoing challenge for academic reviewing overall. Brief reviews of a mere couple of sentences and possibly even uninspiring feedback, which in addition take months are a nightmare for all journals. Reminding and disinviting may be as important as rewarding and promoting. Electronic Markets not only aims to provide early reminders to all participants, but also identifies committed reviewers and aims to win them as members of the editorial board and later as associate or even senior editors.

  • “Editors should have systems to ensure that material submitted to their journal remains confidential while under review”. In the Electronic Markets Editorial Manager system detailed reviewer guidelines are available (Editorial Manager 2015b), which explain the steps and rules within the review process. Following the settings in the Editorial Manager, authors remain anonymous to the reviewers and all material (reviews, comments, submissions etc.) is only available to the involved individuals.

  • “Editors should take all reasonable steps to ensure the quality of the material they publish, recognizing that journals and sections within journals will have different aims and standards.” First of all, Electronic Markets uses the ithenticate system to check each submission on plagiarism. This may indicate whether a submission declared as being original by the authors, has actually been published in part or as a whole. While this clearly refers to research published at conferences, Electronic Markets either requires the authors to opt-out their work for publication in the conference proceedings or to substantially enhance their paper for the journal submission by approximately one third as rule of thumb.

Process organization

Although the existing processes at Electronic Markets have matured over the last two decades, the need for improvements is a constant undertaking and suggestions are always welcome. The process organization shows, as the second organizational element, how a submission to Electronic Markets currently proceeds through the structural organization. Figure 1 depicts the most important steps using the well-established Business Process Model and Notation (BPMN) language. It further includes the responsible roles and the three sub processes: submission, reviewing and publishing. In the following are some added hints from past experience:

Fig. 1
figure1

Review process at Electronic Markets

  • EM manuscripts need to be submitted using the EM template, which includes page and line numbers. This enables reviewers to refer to a part of the author(s)’ text more easily.

  • In the Editorial Manager, authors should specify the type of research they are submitting. Current types are research papers in the special issue and the general research section, position papers and invited papers as well as case study papers (Alt and Österle 2014b).

  • Although only papers that have passed the desk reject check enter the review process, this does not forestall any decision in the review process. At present, Electronic Markets uses six possible decision terms:

    1. (1)

      Desk reject: The paper in its current form is not regarded suitable for Electronic Markets. Due to the maturity of research, the fit to the scope of Electronic Markets and/or the lack of compelling conclusions, the likelihood of favorable reviews at this stage prevents the paper from being sent to reviewers. A desk reject obviously does not preclude a resubmission of a revised paper.

    2. (2)

      Reject: The decision to reject a manuscript is one of the decisions after a paper has been reviewed. The paper is regarded not suitable for Electronic Markets in terms of content and/or quality. Here, the reviewers and editors do not see reasonable potential for developing the paper in subsequent review rounds.

    3. (3)

      Reject and resubmit: This recommendation signals that the reviewers and editors see merit in the paper, yet have major concerns about its current form. The paper needs to be substantially revised and rewritten. The revised version will be treated as a new submission and whenever possible, be distributed to the same reviewers again.

    4. (4)

      Accept with major revisions: The reviewers have identified significant concerns, still they see enough promise to suggest that the paper should be conditionally accepted. Authors should consider that this decision is not a final accept and that the paper may still be rejected in the successive round(s) of review if the modifications are not convincing. All changes made by the authors in response to the review should be listed in a separate table to facilitate the reviewers in identifying the changes. Typically, addressing flaws in the research methodology and/or argumentation require more substantial changes and only adding single sentences will not be sufficient for a major revision.

    5. (5)

      Accept with minor revisions: This decision means that a paper is conditionally accepted for publication and that the paper has generally convinced the reviewers and editors. Prior to being finally accepted, a limited amount of improvements should be taken care of.

    6. (6)

      Accept without revisions: This is the final verdict of unconditional acceptance and usually occurs at the end of one or more iterations of revisions. Depending on the quality of the paper and its revision(s), several of the above-mentioned review iterations may be needed until the paper is finally accepted (EM 2015b).

  • Decisions two to six are usually taken after a manuscript has been with at least two reviewers and two editors. As soon as reviewers receive the invitation to review a submission, they should provide early feedback, on whether they see difficulties in reviewing the paper content or timewise. The review itself is typically expected after a period of four weeks.

  • In order to guide the reviewer through the most important issues while looking at the paper, some manuscript rating questions need to be answered with a number between one (poor) and five (very good) in the Editorial Manager system:

    1. (1)

      Relevance of paper to electronic markets and collaborative business

    2. (2)

      Strength of theoretical foundations and methodology (e.g. empirical, deductive, experimental, knowledge of state of theory and practice)

    3. (3)

      Contribution of new knowledge to Electronic Markets’ scientific area

    4. (4)

      Value of the paper for business, economy, or society

    5. (5)

      Clarity of paper organization and writing

    6. (6)

      Clarity of tables, figures and exhibits

    7. (7)

      Explication of theoretical concepts and terms by examples or cases

    8. (8)

      Attractiveness for Electronic Markets’ readership

    9. (9)

      Quality of conclusions

    10. (10)

      Your Confidence in the Recommendation

  • Besides the ten questions, two boxes for free text are foreseen in the Editorial Manager review screen. The review itself should be included in the “comments to author(s)” and it is recommended writing the text of the review in a separate file first and copying it into the Editorial Manager upon completion. However, there is the possibility to save partial reviews and continue working on and submitting them later.

  • Finally, the boxes “Recommendation Term” as listed above and the “Overall Reviewer Manuscript Rating”, where a number between 1 and 100 can be provided (81–100 = excellent; 61–80 = good; 41–60 = satisfactory; 21–40 = sufficient; 1–20 = inadequate) should be filled in before the reviewer submits his/her review. The rating itself will be one of the criteria considered in determining the paper of the year.

In summary, reviewing has the potential to also profit from the insights obtained in (business) process management. Processes not only require a proper definition and dedicated responsibilities, but also so-called process customers, which use the output of a (submission, reviewing and/or publishing) process, as well as an integrated electronic support of all major process steps. However, an academic journal differs in important aspects from a for-profit business organization. All participants contribute on a voluntary basis and are part of individual organizations. Thus, establishing or even executing strict processes poses a challenge in such a distributed environment. Editorial offices may suggest and facilitate processes and offer support, which has been the goal of Electronic Markets’ editorial team for many years. Table 1 summarizes some design principles of process management and shows how Electronic Markets aims to address them. We are proud of the effort our editors and board members dedicate to our journal. At the same time, improvements are always possible and the editorial team is always open for any suggestions.

Table 1 Specifics of reviewing at Electronic Markets

Special issue on IOS 2.0

One example of the joint effort of authors, editors and reviewers is the present issue on the topic of “IOS 2.0: New Aspects on Inter-Organizational Integration Through Enterprise 2.0 Technologies”. The Call for Papers for this special issue and the handling of the original submissions was mostly driven by Holger Schrödl, who tragically passed away last year. We want to use this opportunity to give our condolences to his family, friends and colleagues. Holger was a talented researcher and repeatedly contributed to Electronic Markets. We are thankful, that this special issue still makes the publication, which is mainly due to Ali Sunyaev from the University of Cologne, Germany, who took over as corresponding guest editor. Together with Christy M. K. Cheung from Hong Kong Baptist University, Hong Kong, Martin Mocker from Reutlingen University, Germany and MIT, USA, Daniel Schlagwein from UNSW Australia and Klaus Turowski from the University of Magdeburg, Germany, Ali Sunyaev managed to establish an exciting collection of three papers, which address different aspects of inter-organizational integration: The first is titled “How to bridge the boundary? Determinants of inter-organizational social software usage” and authored by Melanie Steinhüser, Alexander Richter and Stefan Smolnik. It is freely available as an open access paper (Steinhüser et al. 2015). The second paper is on “Business value of partner’s IT intensity: value co-creation and appropriation between customers and suppliers” by Pankaj Setia, Vernon Richardson and Rodney J. Smith (Setia et al. 2015). Finally Stephan Schneider and Ali Sunyaev present “CloudLive: A Life Cycle Framework for Cloud Services” (Schneider and Sunyaev 2015). A more detailed description of the individual articles is included in the guest editors’ preface (Cheung et al. 2015).

Furthermore, this issue also includes two general research papers. The first one is Yun Wan’s “The Matthew Effect in Social Commerce: the Case of Online Review Helpfulness”, which focusses on online reviews as one important form of user-generated content, which plays a strategic role in the elimination of information asymmetry (Wan 2015). The second general research paper is titled “The Impact of Customizable Market Interfaces on Trading Performance” and is authored by Florian Teschner, Tobias T. Kranz and Christof Weinhardt. It deals with the inherent complexity that excludes non-sophisticated users as one reason for market failure and is available as an open access paper as well (Teschner et al. 2015).

We hope you enjoy reading this issue. Apart from the Guest Editors’ team, we also want to give special thanks to our former executive editor Christian Hrach, who helped preparing the specification of the journal processes (see Fig. 1). If you consider contributing to Electronic Markets as a reviewer or a guest editor, please feel free to contact us. To qualify as a reviewer for Electronic Markets you should, at least, be an advanced PhD student with profound knowledge in a specific area concerning either our journal’s scope or methodological approaches. Since members of the editorial board are expected to contribute an average of three reviews per year, each individual should consider the workload, which may quickly cumulate with multiple reviewer commitments for different scientific outlets (e.g. other journals, conferences) as indicated in the simple calculation of reviewer effort above.

At the end of this editorial and at the end of another successful year for Electronic Markets we would like to take this opportunity to express our thanks to all who contributed in achieving this. Without a continuous effort of all authors, reviewers and editors this would not have been possible. We wish you happy holidays and all the best for 2016!

Best regards from Leipzig and St. Gallen,

Rainer Alt, Carsta Militzer-Horstmann, Hans-Dieter Zimmermann

References

  1. Alt, R., Militzer-Horstmann, C., & Zimmermann, H.-D. (2015a). Editorial 25/1: 25 years of Electronic Markets. Electronic Markets, 25(1), 1–5. doi:10.1007/s12525-015-0185-y.

    Article  Google Scholar 

  2. Alt, R., Militzer-Horstmann, C., & Zimmermann, H.-D. (2015b). Editorial 25/2: Electronic Markets and privacy. Electronic Markets, 25(2), 87–90. doi:10.1007/s12525-015-0193-y.

    Article  Google Scholar 

  3. Alt, R., & Österle, H. (2014a). Editorial 24/1: Electronic Markets and practice-orientation. Electronic Markets, 24(1), 1–3. doi:10.1007/s12525-014-0152-z.

    Article  Google Scholar 

  4. Alt, R., & Österle, H. (2014b). Editorial 24/2: Electronic Markets and journal sections. Electronic Markets, 24(2), 77–79. doi:10.1007/s12525-014-0159-5.

    Article  Google Scholar 

  5. Alt, R., & Zimmermann, H.-D. (2015). Editorial 25/3: Electronic Markets on ecosystems and tourism. Electronic Markets, 25(3), 169–174. doi:10.1007/s12525-015-0197-7.

    Article  Google Scholar 

  6. Australian National University (2014). Writing a journal article review. https://academicskills.anu.edu.au/node/492. Accessed 9 Oct 2015.

  7. Cheung, C. M. K., Mocker, M., Schlagwein, D., Sunyaev, A. & Turowski, K. (2015). IOS 2.0: New aspects on inter-organizational integration through enterprise 2.0 technologies. Electronic Markets, 25(4), doi:10.1007/s12525-015-0204-z.

  8. COPE (2015). Committee on publishing ethics’ code of conduct and best practice guidelines for journal editors. http://publicationethics.org/files/Code_of_conduct_for_journal_editors_Mar11.pdf.

  9. Editorial Manager (2015a). http://elma.edmgr.com/.

  10. Editorial manager (2015b). Reviewer guidelines. http://www.editorialmanager.com/homepage/DOCS/Reviewer_Tutorial.pdf.

  11. EM (2015a). Electronic Markets’ outstanding reviewers. http://www.electronicmarkets.org/about-em/outstanding-reviewers/. Accessed 9 Oct 2015.

  12. EM (2015b). Electronic Markets’ decisions. http://www.electronicmarkets.org/authors/decisions/. Accessed 9 Oct 2015.

  13. Ives, B. (1992). Editor’s comments. Total quality management of journal reviews. MIS Quarterly, June 1992, xxvii-xxix.

  14. Lee, A. S. (1995). Reviewing a manuscript for publication. Journal of Operations Management, 13(1), 87–92.

    Article  Google Scholar 

  15. Österle, H. (1995). Business in the Information age - heading for new processes, Springer 1995.

  16. Schneider, S. & Sunyaev, A. (2015). Cloud live: a life cycle framework for cloud services. Electronic Markets, 25/4. doi: 10.1007/s12525-015-0205-y.

  17. Setia, P., Richardson, V., & Smith, R. J. (2015). Business value of partner’s IT intensity: value co-creation and appropriation between customers and suppliers. Electronic Markets, 25/4. doi:10.1007/s12525-015-0189-7.

  18. Steinhüser, M., Richter, A. & Smolnik, S. (2015). How to bridge the boundary? Determinants of inter-organizational social software usage. Electronic Markets, 25/4, doi:10.1007/s12525-015-0192-z.

  19. Teschner, F., Kranz, T. T. & Weinhardt, C. (2015). The Impact of customizable Market Interfaces on Trading Performance. Electronic Markets, 25(4), doi:10.1007/s12525-014-0172-8.

  20. van Haarten, J. (2015). A reviewer’s guide to ethics in publishing. https://www.elsevier.com/reviewers-update/story/publishing-ethics/a-reviewers-guide-to-publishing-ethics. Accessed 9 Oct 2015.

  21. Wan, Y. (2015). The Matthew effect in social commerce: the case of online review helpfulness. Electronic Markets, 25(4), doi:10.1007/s12525-015-0186-x.

  22. Zmud, B. (1998). Editor’s comments. MIS Quarterly, 22(2), xiv-xiviii.

Download references

Author information

Affiliations

Authors

Corresponding author

Correspondence to Rainer Alt.

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Alt, R., Militzer-Horstmann, C. & Zimmermann, HD. Editorial 25/4: Electronic Markets on reviewing. Electron Markets 25, 255–261 (2015). https://doi.org/10.1007/s12525-015-0207-9

Download citation