Information Systems Frontiers

, Volume 17, Issue 4, pp 809–825 | Cite as

On the inaccuracy of numerical ratings: dealing with biased opinions in social networks



In this work, we study the potential problems emanating from using numerical ratings in social networks to rank entities regarding their reputation. In particular, we empirically demonstrate how reputation rankings as collected and managed by current systems are likely to be skewed due to subjectivity problems associated with the use of numerical ratings to encapsulate preferences. With the aim of overcoming these problems, we put forward an approach in which users are asked for their opinions about entities in a comparative fashion. In order to select the most appropriate users to be queried, we take advantage of the social structure derived from the interactions among users and entities following a principle of heterogeneity. Finally, we evaluate the proposed approach in the domain of movie ratings by using real datasets collected from different web sites.


Ratings Reputation Pairwise elicitation Opinions Social networks 



Work partially supported by the Spanish Ministry of Science and Innovation through the projects OVAMAH (grant TIN2009-13839-C03-02; co-funded by Plan E) and ”AT” (grant CSD2007-0022; CONSOLIDER-INGENIO 2010) and by the Spanish Ministry of Economy and Competitiveness through the project iHAS (grant TIN2012-36586-C03-02); and by the Region of Madrid through the eMadrid project (grant S2009-TIC1650).


  1. Anderson, M.L., & Magruder, J.R. (2012). Learning from the crowd: Regression discontinuity estimates of the effects of an online review database. Economic Journal, 122 (563), 957–989.CrossRefGoogle Scholar
  2. Balakrishnan, S., & Chopra, S. (2010). Two of a kind or the ratings game? Adaptive pairwise preferences and latent factor models. Proceedings of the 2010 IEEE international conference on data mining. ICDM ’10, (pp. 725–730). Washington, DC: IEEE Computer Society.Google Scholar
  3. Berinsky, A.J. (1999). The two faces of public opinion. American Journal of Political Science, 43 (4), 1209–1230.CrossRefGoogle Scholar
  4. Brams, S.J., & Fishburn, P.C. (2002). Voting procedures In Arrow, K. J., Sen, A. K., Suzumura, K. (Eds.), Handbook of social choice and welfare (Vol. 1, pp. 173–236). New York: Elsevier.Google Scholar
  5. Burke, R. (2000). Knowledge-based Recommender Systems. Encyclopedia of library and information systems Vol. 69.Google Scholar
  6. Chen, Li, & Pu, Pearl (2004). Survey of preference elicitation methods. Tech. rep., Swiss Federal Institure of Technology in Lausanne (EPFL).Google Scholar
  7. Cosley, D., Lam, S.K., Albert, I., Konstan, J.A., Riedl, J. (2003). Is seeing believing?: how recommender system interfaces affect users’ opinions. Proceedings of the sigchi conference on human factors in computing systems. CHI ’03, (pp. 585–592). New York: ACM.Google Scholar
  8. Dalvi, N., Kumar, R., Pang, B. (2013). Para ’normal’ activity: On the distribution of average ratings. Proceedings of 7th international AAAI conference on weblogs and social media, (pp. 110–119).Google Scholar
  9. Dellarocas, C. (2000). Mechanisms for coping with unfair ratings and discriminatory behavior in online reputation reporting systems. ICIS ’00: Proceedings of the 21st international conference on information systems, (pp. 520–525). Atlanta: Association for Information Systems.Google Scholar
  10. Fowler, J.H., & Christakis, N.A. (2010). Cooperative behavior cascades in human social networks. Proceedings of the National Academy of Sciences, 107 (12), 5334–5338.CrossRefGoogle Scholar
  11. Fu, F., Hauert, C., Nowak, M.A., Wang, L. (2008). Reputation-based partner choice promotes cooperation in social networks. Physical Review E, 78.Google Scholar
  12. Fullam, K.K., & Barber, K.S. (2007). Dynamically learning sources of trust information: experience vs. reputation. Proceedings of AAMAS ’07, (pp. 1055–1062). New York: ACM.Google Scholar
  13. Grinstead, C.M., & Snell, J.L. (1997). Introduction to probability, 2nd edn, AMS Bookstore, chap Central Limit Theorem, (pp. 325–360).Google Scholar
  14. Hermoso, R., Centeno, R., Fasli, M. (2013). From blurry numbers to clear preferences: A mechanism to extract reputation in social networks. Expert Systems with Applications, 41 (5), 2269–2285.CrossRefGoogle Scholar
  15. Hill, W., Stead, L., Rosenstein, M., Furnas, G. (1995). Recommending and evaluating choices in a virtual community of use. Proceedings of the sigchi conference on human factors in computing systems. CHI ’95, (pp. 194–201). New York: ACM Press/Addison-Wesley Publishing Co.Google Scholar
  16. Huynh, T.D., Jennings, N.R., Shadbolt, N.R. (2006). An integrated trust and reputation model for open multi-agent systems. Autonomous Agents and Multi-Agent Systems, 13 (2), 119–154.CrossRefGoogle Scholar
  17. Järvelin, K., & Kekäläinen, J. (2002). Cumulated gain-based evaluation of ir techniques. ACM Transactions on Information Systems, 20 (4), 422–446.CrossRefGoogle Scholar
  18. Jøsang, A., & Ismail, R. (2002). The beta reputation system. Proceedings of the 15th bled electronic commerce conference.Google Scholar
  19. Jøsang, A., Ismail, R., Boyd, C. (2007). A survey of trust and reputation systems for online service provision. Decisions Support Systems, 43 (2), 618–644.CrossRefGoogle Scholar
  20. Kendall, M.G. (1938). A New Measure of Rank Correlation. Biometrika, 30 (1/2), 81+. doi:  10.2307/2332226.CrossRefGoogle Scholar
  21. Khosravifar, B., Bentahar, J., Gomrokchi, M., Alam, R. (2012). CRM: An efficient trust and reputation model for agent computing. Knowledge-Based Systems, 30 (0), 1 –16.CrossRefGoogle Scholar
  22. Kramer, M. (2007). Self-selection bias in reputation systems In Etalle, S., & Marsh, S. (Eds.), Trust management. IFIP International Federation for Information Processing (Vol. 238, pp. 255–268). Berlin Heildelberg New York: Springer.Google Scholar
  23. Liu, Y., Li, K., Jin, Y., Zhang, Y., Qu, W. (2011). A novel reputation computation model based on subjective logic for mobile ad hoc networks. Future Generation Computer Systems, 27 (5), 547 –554.CrossRefGoogle Scholar
  24. Luca, M. (2011). Reviews, reputation, and revenue: The case of Harvard Business School Working Papers 12–016, Harvard Business School.Google Scholar
  25. Negahban, S., Oh, S., Shah, D. (2012). Iterative ranking from pair-wise comparisons. Proceedings of the 26th annual conference on neural information processing systems, (pp. 2483–2491).Google Scholar
  26. Pennock, D., Horvitz, E., Lawrence, S., Giles, C.L. (2000). Collaborative filtering by personality diagnosis: A hybrid memory- and model-based approach. Proceedings of the sixteenth conference on uncertainty in artificial intelligence, (pp. 473–480). San Mateo: Morgan Kaufmann.Google Scholar
  27. Pinyol, I., & Sabater-Mir, J. (2013). Computational trust and reputation models for open multi-agent systems: a review. Artificial Intelligence Review, 40 (1), 1–25.CrossRefGoogle Scholar
  28. Ramchurn, S.D., Huynh, D., Jennings, N.R. (2004). Trust in multi-agent systems. Knowledge Engineering Review, 19 (1), 1–25.CrossRefGoogle Scholar
  29. Resnick, P., Kuwabara, K., Zeckhauser, R., Friedman, E. (2000). Reputation systems. Communications of the ACM, 43 (12), 45–48.CrossRefGoogle Scholar
  30. Shearin, S., & Lieberman, H. (2001). Intelligent profiling by example. Proceedings of the 6th international conference on intelligent user interfaces. IUI ’01, (pp. 145–151). New York: ACM.Google Scholar
  31. Sierra, C., & Sabater, J. (2005). Review on computionational trust and reputation models. Artificial Intelligence Review, 24 (1), 33 –60.CrossRefGoogle Scholar
  32. Sparling, E.I., & Sen, S. (2011). Rating: how difficult is it?. Proceedings of the 5th acm conference on recommender systems. RecSys ’11, (pp. 149–156). New York: ACM.Google Scholar
  33. Tanimoto, T. (1958). An elementary mathematical theory of classification and prediction. IBM Internal Report.Google Scholar
  34. Teacy, W.T.L., Patel, J., Jennings, N.R., Luck, M. (2006). Travos: Trust and reputation in the context of inaccurate information sources. Autonomous Agents and Multi-Agent Systems, 12 (2), 183–198.CrossRefGoogle Scholar
  35. Teacy, W.T.L., Luck, M., Rogers, A., Jennings, N.R. (2012). An efficient and versatile approach to trust and reputation using hierarchical bayesian modelling. Artificial Intelligence, 193, 149–185.CrossRefGoogle Scholar
  36. Yu, B., & Singh, M.P. (2002). An evidential model of distributed reputation management. AAMAS ’02: Proceedings of the first international joint conference on autonomous agents and multiagent systems, (pp. 294–301). New York: ACM Press.Google Scholar
  37. Yu, B., Singh, M.P., Sycara, K. (2004). Developing trust in large-scale peer-to-peer systems. Proceedings of first IEEE symposium on multi-agent security and survivability, (pp. 1–10).Google Scholar

Copyright information

© Springer Science+Business Media New York 2014

Authors and Affiliations

  1. 1.Dpto. de Lenguajes y Sistemas Informáticos, UNEDMadridSpain
  2. 2.School of Computer Science and Electronic EngineeringUniversity of EssexColchesterUK

Personalised recommendations