Abstract
In this work, we study the potential problems emanating from using numerical ratings in social networks to rank entities regarding their reputation. In particular, we empirically demonstrate how reputation rankings as collected and managed by current systems are likely to be skewed due to subjectivity problems associated with the use of numerical ratings to encapsulate preferences. With the aim of overcoming these problems, we put forward an approach in which users are asked for their opinions about entities in a comparative fashion. In order to select the most appropriate users to be queried, we take advantage of the social structure derived from the interactions among users and entities following a principle of heterogeneity. Finally, we evaluate the proposed approach in the domain of movie ratings by using real datasets collected from different web sites.
Similar content being viewed by others
Notes
This dataset was used in the 2nd International Workshop on Information Heterogeneity and Fusion in Recommender Systems (See http://ir.ii.uam.es/hetrec2011). It is collected from the MovieLens opinion site, where movies are rated by different users.
As in the MovieLens site, this is an opinion website where movies are rated. However, in this case, movies are also reviewed by professional critics (See http://www.rottentomatoes.com).
References
Anderson, M.L., & Magruder, J.R. (2012). Learning from the crowd: Regression discontinuity estimates of the effects of an online review database. Economic Journal, 122 (563), 957–989.
Balakrishnan, S., & Chopra, S. (2010). Two of a kind or the ratings game? Adaptive pairwise preferences and latent factor models. Proceedings of the 2010 IEEE international conference on data mining. ICDM ’10, (pp. 725–730). Washington, DC: IEEE Computer Society.
Berinsky, A.J. (1999). The two faces of public opinion. American Journal of Political Science, 43 (4), 1209–1230.
Brams, S.J., & Fishburn, P.C. (2002). Voting procedures In Arrow, K. J., Sen, A. K., Suzumura, K. (Eds.), Handbook of social choice and welfare (Vol. 1, pp. 173–236). New York: Elsevier.
Burke, R. (2000). Knowledge-based Recommender Systems. Encyclopedia of library and information systems Vol. 69.
Chen, Li, & Pu, Pearl (2004). Survey of preference elicitation methods. Tech. rep., Swiss Federal Institure of Technology in Lausanne (EPFL).
Cosley, D., Lam, S.K., Albert, I., Konstan, J.A., Riedl, J. (2003). Is seeing believing?: how recommender system interfaces affect users’ opinions. Proceedings of the sigchi conference on human factors in computing systems. CHI ’03, (pp. 585–592). New York: ACM.
Dalvi, N., Kumar, R., Pang, B. (2013). Para ’normal’ activity: On the distribution of average ratings. Proceedings of 7th international AAAI conference on weblogs and social media, (pp. 110–119).
Dellarocas, C. (2000). Mechanisms for coping with unfair ratings and discriminatory behavior in online reputation reporting systems. ICIS ’00: Proceedings of the 21st international conference on information systems, (pp. 520–525). Atlanta: Association for Information Systems.
Fowler, J.H., & Christakis, N.A. (2010). Cooperative behavior cascades in human social networks. Proceedings of the National Academy of Sciences, 107 (12), 5334–5338.
Fu, F., Hauert, C., Nowak, M.A., Wang, L. (2008). Reputation-based partner choice promotes cooperation in social networks. Physical Review E, 78.
Fullam, K.K., & Barber, K.S. (2007). Dynamically learning sources of trust information: experience vs. reputation. Proceedings of AAMAS ’07, (pp. 1055–1062). New York: ACM.
Grinstead, C.M., & Snell, J.L. (1997). Introduction to probability, 2nd edn, AMS Bookstore, chap Central Limit Theorem, (pp. 325–360).
Hermoso, R., Centeno, R., Fasli, M. (2013). From blurry numbers to clear preferences: A mechanism to extract reputation in social networks. Expert Systems with Applications, 41 (5), 2269–2285.
Hill, W., Stead, L., Rosenstein, M., Furnas, G. (1995). Recommending and evaluating choices in a virtual community of use. Proceedings of the sigchi conference on human factors in computing systems. CHI ’95, (pp. 194–201). New York: ACM Press/Addison-Wesley Publishing Co.
Huynh, T.D., Jennings, N.R., Shadbolt, N.R. (2006). An integrated trust and reputation model for open multi-agent systems. Autonomous Agents and Multi-Agent Systems, 13 (2), 119–154.
Järvelin, K., & Kekäläinen, J. (2002). Cumulated gain-based evaluation of ir techniques. ACM Transactions on Information Systems, 20 (4), 422–446.
Jøsang, A., & Ismail, R. (2002). The beta reputation system. Proceedings of the 15th bled electronic commerce conference.
Jøsang, A., Ismail, R., Boyd, C. (2007). A survey of trust and reputation systems for online service provision. Decisions Support Systems, 43 (2), 618–644.
Kendall, M.G. (1938). A New Measure of Rank Correlation. Biometrika, 30 (1/2), 81+. doi: 10.2307/2332226.
Khosravifar, B., Bentahar, J., Gomrokchi, M., Alam, R. (2012). CRM: An efficient trust and reputation model for agent computing. Knowledge-Based Systems, 30 (0), 1 –16.
Kramer, M. (2007). Self-selection bias in reputation systems In Etalle, S., & Marsh, S. (Eds.), Trust management. IFIP International Federation for Information Processing (Vol. 238, pp. 255–268). Berlin Heildelberg New York: Springer.
Liu, Y., Li, K., Jin, Y., Zhang, Y., Qu, W. (2011). A novel reputation computation model based on subjective logic for mobile ad hoc networks. Future Generation Computer Systems, 27 (5), 547 –554.
Luca, M. (2011). Reviews, reputation, and revenue: The case of yelp.com. Harvard Business School Working Papers 12–016, Harvard Business School.
Negahban, S., Oh, S., Shah, D. (2012). Iterative ranking from pair-wise comparisons. Proceedings of the 26th annual conference on neural information processing systems, (pp. 2483–2491).
Pennock, D., Horvitz, E., Lawrence, S., Giles, C.L. (2000). Collaborative filtering by personality diagnosis: A hybrid memory- and model-based approach. Proceedings of the sixteenth conference on uncertainty in artificial intelligence, (pp. 473–480). San Mateo: Morgan Kaufmann.
Pinyol, I., & Sabater-Mir, J. (2013). Computational trust and reputation models for open multi-agent systems: a review. Artificial Intelligence Review, 40 (1), 1–25.
Ramchurn, S.D., Huynh, D., Jennings, N.R. (2004). Trust in multi-agent systems. Knowledge Engineering Review, 19 (1), 1–25.
Resnick, P., Kuwabara, K., Zeckhauser, R., Friedman, E. (2000). Reputation systems. Communications of the ACM, 43 (12), 45–48.
Shearin, S., & Lieberman, H. (2001). Intelligent profiling by example. Proceedings of the 6th international conference on intelligent user interfaces. IUI ’01, (pp. 145–151). New York: ACM.
Sierra, C., & Sabater, J. (2005). Review on computionational trust and reputation models. Artificial Intelligence Review, 24 (1), 33 –60.
Sparling, E.I., & Sen, S. (2011). Rating: how difficult is it?. Proceedings of the 5th acm conference on recommender systems. RecSys ’11, (pp. 149–156). New York: ACM.
Tanimoto, T. (1958). An elementary mathematical theory of classification and prediction. IBM Internal Report.
Teacy, W.T.L., Patel, J., Jennings, N.R., Luck, M. (2006). Travos: Trust and reputation in the context of inaccurate information sources. Autonomous Agents and Multi-Agent Systems, 12 (2), 183–198.
Teacy, W.T.L., Luck, M., Rogers, A., Jennings, N.R. (2012). An efficient and versatile approach to trust and reputation using hierarchical bayesian modelling. Artificial Intelligence, 193, 149–185.
Yu, B., & Singh, M.P. (2002). An evidential model of distributed reputation management. AAMAS ’02: Proceedings of the first international joint conference on autonomous agents and multiagent systems, (pp. 294–301). New York: ACM Press.
Yu, B., Singh, M.P., Sycara, K. (2004). Developing trust in large-scale peer-to-peer systems. Proceedings of first IEEE symposium on multi-agent security and survivability, (pp. 1–10).
Acknowledgments
Work partially supported by the Spanish Ministry of Science and Innovation through the projects OVAMAH (grant TIN2009-13839-C03-02; co-funded by Plan E) and ”AT” (grant CSD2007-0022; CONSOLIDER-INGENIO 2010) and by the Spanish Ministry of Economy and Competitiveness through the project iHAS (grant TIN2012-36586-C03-02); and by the Region of Madrid through the eMadrid project (grant S2009-TIC1650).
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Centeno, R., Hermoso, R. & Fasli, M. On the inaccuracy of numerical ratings: dealing with biased opinions in social networks. Inf Syst Front 17, 809–825 (2015). https://doi.org/10.1007/s10796-014-9526-1
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10796-014-9526-1