, Volume 110, Issue 1, pp 179–194 | Cite as

Reviewers’ scores do not predict impact: bibliometric analysis of the proceedings of the human–robot interaction conference

  • Christoph Bartneck


The peer review process is an essential component for the progress of science. The ACM/IEEE International Conference on Human–Robot Interaction is the prime publication channel for the field and this study evaluates its peer review process. The results show that the number of accepted papers are unevenly distributed across countries, organizations and authors. The contributions from the US outweigh all others contributions. A Binary Logistic Regression analysis showed that only for 85.5% of the papers the reviewers’ scores accurately predict its acceptance or rejection. Moreover, there was no significant correlation between the reviewers’ scores and the citations the papers later attract. 73% of the authors only ever submitted one paper and the proportion of newcomers at the conferences ranges from 63–77%.


Human–robot interaction HRI Conference Proceedings Acceptance rate 



The author would like to thank Utku Yalcin and Subha Krishna for the data processing.


  1. Bar-Ilan, J. (2008). Which h-index? A comparison of WoS, Scopus and Google Scholar. Scientometrics, 74(2), 257–271. doi: 10.1007/s11192-008-0216-y.CrossRefGoogle Scholar
  2. Bartneck, C. (2010). The all-in publication policy. Fourth International Conference on Digital Society (ICDS 2010), IEEE (pp. 37–40). doi: 10.1109/ICDS.2010.14.
  3. Bartneck, C. (2011). The end of the beginning—a reflection on the first five years of the HRI conference. Scientometrics, 86(2), 487–504. doi: 10.1007/s11192-010-0281-x.CrossRefGoogle Scholar
  4. Bartneck, C., & Jun, H. (2010). The fruits of collaboration in a multidisciplinary field. Scientometrics, 85(1), 41–52. doi: 10.1007/s11192-010-0242-4.CrossRefGoogle Scholar
  5. Bohannon, J. (2013). Who’s afraid of peer review? Science, 342(6154), 60–65. doi: 10.1126/science.342.6154.60.CrossRefGoogle Scholar
  6. Campanario, J. (2009). Rejecting and resisting nobel class discoveries: Accounts by nobel laureates. Scientometrics, 81(2), 549–565. doi: 10.1007/s11192-008-2141-5.CrossRefGoogle Scholar
  7. Campanario, J. M. (1995). Commentary: On influential books and journal articles initially rejected because of negative referees’ evaluations. Science Communication 16(3), 304–325. doi: 10.1177/1075547095016003004.
  8. Eysenbach, G. (2006). Citation advantage of open access articles. PLoS Biology,. doi: 10.1371/journal.pbio.0040157.Google Scholar
  9. Glänzel, W., & Moed, H. F. (2002). Journal impact measures in bibliometric research. Scientometrics, 53(2), 171–193. doi: 10.1023/A:1014848323806.CrossRefGoogle Scholar
  10. Lawrence, S. (2001). Free online availability substantially increases a paper’s impact. Nature, 411(6837), 521–521. doi: 10.1038/35079151.CrossRefGoogle Scholar
  11. Matthews, D. (2016). High rejection rates by journals pointless.
  12. Meho, L. I., & Rogers, Y. (2008). Citation counting, citation ranking, and h-index of human–computer interaction researchers: A comparison of scopus and web of science. Journal of the American Society for Information Science and Technology, 59(11), 1711–1726. doi: 10.1002/asi.20874.CrossRefGoogle Scholar
  13. Meho, L. I., & Yang, K. (2007). Impact of data sources on citation counts and rankings of LIS faculty: Web of science versus scopus and Google Scholar. Journal of the American Society for Information Science and Technology, 58(13), 2105–2125. doi: 10.1002/asi.20677.CrossRefGoogle Scholar
  14. Peters, D. P., & Ceci, S. J. (1982). Peer-review practices of psychological journals: The fate of published articles, submitted again. Behavioral and Brain Sciences, 5(2), 187–195.CrossRefGoogle Scholar
  15. Price, E. (2014). The NIPS experiment.
  16. Safi, M. (2015). Journal accepts bogus paper requesting removal from mailing list.
  17. Rocha da Silva, P. (2015). Selecting for impact: new data debunks old beliefs.

Copyright information

© Akadémiai Kiadó, Budapest, Hungary 2016

Authors and Affiliations

  1. 1.University of CanterburyChristchurchNew Zealand

Personalised recommendations