What Peer-Review Systems Can Learn from Online Rating Sites

  • Edward F. GehringerEmail author
  • Kai Ma
  • Van T. Duong
Conference paper
Part of the Lecture Notes in Educational Technology book series (LNET)


As their core functionality, peer-review systems present ratings of student work. But online ratings are not a new concept. Sites rating products or services have a long history on the Web, and now boast hundreds of millions of users. These sites have developed mechanisms and procedures to improve the accuracy and helpfulness of their reviews. Peer-review systems have much to learn from their experience. Online review sites permit users to flag reviews they consider inappropriate or inaccurate. Peer-review systems could do the same. Online review systems have automated metrics to decide whether reviews should be posted. It would be good for peer-review systems to post only reviews that pass an automatic quality check. Online rating sites give recognition to their best reviewers by means of levels or badges. Recognition is often dependent on upvotes by other users. Online review sites often let readers see helpfulness ratings or other information on reviewers. Peer-review systems could also allow authors to see ratings of the students who reviewed their work.


Peer-review systems Online review sites Reputation systems 



This work has been supported by the U.S. National Science Foundation under grant 1432347.


  1. 1.
    Lin, I. C., Wu, H. J., Li, S. F., & Cheng, C. Y. (2015). A fair reputation system for use in online auctions. Journal of Business Research, 68(4), 878–882.CrossRefGoogle Scholar
  2. 2.
    de Alfaro, L., & Shavlovsky, M. (2014). CrowdGrader: A tool for crowdsourcing the evaluation of homework assignments. In 45th ACM Technical Symposium on Computer Science Education (pp. 415–420).Google Scholar
  3. 3.
    Palanski, M., Babik, D., & Ford, E. (2014). Mobius SLIP: Anonymous, peer-reviewed student writing. In OBTC 2014 at Vanderbilt University.Google Scholar
  4. 4.
    Movshovitz-Attias, D., Movshovitz-Attias, Y., Steenkiste, P., & Faloutsos, C. (2013). Analysis of the reputation system and user contributions on a question answering website: stackoverflow. In Proceedings of the 2013 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining (pp. 886–893).Google Scholar
  5. 5.
    Three insider tips for better TripAdvisor performance. Retrieved June 5, 2015 from
  6. 6.
    Yelp Elite Squad. Retrieved June 5, 2015 from
  7. 7. How ranking works. Retrieved June 5, 2015 from
  8. 8.
    Wright, J., Thornton, C., Leyton-Brown, K., & Mechanical, T. A. (2015). Partially automated high-stakes peer grading. In 46th ACM Technical Symposium on Computer Science Education (SIGCSE ’15) (pp. 96–101). New York, NY, USA: ACM.Google Scholar
  9. 9.
    Gehringer, E. (2009). Expertiza: information management for collaborative learning. In A. A. Juan Perez (Ed.), Monitoring and assessment in online collaborative environments: emergent computational technologies for e-learning support. IGI Global Press.Google Scholar
  10. 10.
    Azzedin, F., & Ridha, A. (2010). Feedback behavior and its role in trust assessment for peer-to-peer systems. Telecommunication Systems, 44(3–4), 253–266.CrossRefGoogle Scholar
  11. 11.
    Meng, X., Ding, Y., & Gong, Y. (2012). @ Trust: A trust model based on feedback-arbitration in structured P2P network. Computer Communications, 35(16), 2044–2053.CrossRefGoogle Scholar
  12. 12.
    Thirunarayan, K., Anantharam, P., Henson, C., & Sheth, A. (2014). Comparative trust management with applications: Bayesian approaches emphasis. Future Generation Computer Systems, 31, 182–199.CrossRefGoogle Scholar
  13. 13.
    About reviewers and rankings. Retrieved June 5, 2015 from
  14. 14.
    Kulkarni, C., Wei, K. P., Le, H., Chia, D., Papadopoulos, K., Cheng, J., Koller, D., & Klemmer, S. R. (2013). Peer and self assessment in massive online classes. ACM Transactions Computer-Human Interaction 20(6), 31–33. doi: 10.1145/2505057
  15. 15.
    Ramachandran, L. (2013). Automated assessment of reviews. Ph.D. dissertation, North Carolina State University.Google Scholar
  16. 16.
    Gehringer, E., Gummadi, A., Kadanjoth, R., & Andres, Y. M. (2010). Motivating effective peer review with extra credit and leaderboards. In American Society for Engineering Education Annual Conference. American Society for Engineering Education.Google Scholar
  17. 17.
    Ghosh, A., & Hummel, P. (2012). Implementing optimal outcomes in social computing: A game-theoretic approach. In Proceedings of the 21st International Conference on the World Wide Web (WWW ’12) (pp. 539–548). New York, NY, USA: ACM. doi: 10.1145/2187836.2187910
  18. 18.
    Easley, D., & Ghosh, A. (2013). Incentives, gamification, and game theory: An economic approach to badge design. In Proceedings of the Fourteenth ACM Conference on Electronic Commerce (EC ’13) (pp. 359–376). New York, NY, USA: ACM. doi: 10.1145/2482540.2482571
  19. 19.
    UrbanSpoon to Zomato: the 5-point rating system. Retrieved June 5, 2015 from
  20. 20.
    Gunersel, A. B., Simpson, N. J., Aufderheide, K. J., & Wang, L. (2008). Effectiveness of calibrated peer reviewTM for improving writing and critical thinking skills in biology undergraduate students. Journal of the Scholarship of Teaching and Learning, 8(2), 25–37.Google Scholar

Copyright information

© Springer Science+Business Media Singapore 2016

Authors and Affiliations

  1. 1.Department of Computer ScienceNorth Carolina State UniversityRaleighUSA

Personalised recommendations