Recommender systems: from algorithms to user experience

Abstract

Since their introduction in the early 1990’s, automated recommender systems have revolutionized the marketing and delivery of commerce and content by providing personalized recommendations and predictions over a variety of large and complex product offerings. In this article, we review the key advances in collaborative filtering recommender systems, focusing on the evolution from research concentrated purely on algorithms to research concentrated on the rich set of questions around the user experience with the recommender. We show through examples that the embedding of the algorithm in the user experience dramatically affects the value to the user of the recommender. We argue that evaluating the user experience of a recommender requires a broader set of measures than have been commonly used, and suggest additional measures that have proven effective. Based on our analysis of the state of the field, we identify the most important open research problems, and outline key challenges slowing the advance of the state of the art, and in some cases limiting the relevance of research to real-world applications.

References

  1. Adomavicius G., Kwon Y.: New recommendation techniques for multicriteria rating systems. IEEE Intel. Syst. 22(3), 48–55 (2007)

    Article  Google Scholar 

  2. Adomavicius G., Tuzhlin A.: Towards the next generation of recommender systems: a survey of the state-of-the-art and possible extensions. IEEE Trans. Knowl. Data Eng. 6(17), 734–749 (2005)

    Article  Google Scholar 

  3. Adomavicius G., Sankaranarayanan R., Sen S., Tuzhilin A.: Incorporating contextual information in recommender systems using a multidimensional approach. ACM Trans. Inf. Syst. 23(1), 103–145 (2005)

    Article  Google Scholar 

  4. Ali, K., Van Stam, W.: Tivo: making show recommendations using a distributed collaborative filtering architecture. In: Proceedings of the 10th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 394–401. ACM, New York (2004)

  5. Amatriain, X., Pujol, J.M., Tintarev, N., Oliver, N.: Rate it again: increasing recommendation accuracy by user re-rating. In: Proceedings of the third ACM conference on Recommender systems (RecSys ’09), pp. 173–180. ACM, New York (2009)

  6. Billsus, D., Pazzani, M.J.: Learning collaborative information Filters. In: Proceedings of the Fifteenth International Conference on Machine Learning 54, 47 (1998)

  7. Breese, J.S., Heckerman, D., Kadie, C.: Empirical analysis of predictive algorithms for collaborative filtering. In: Proceedings of the Fourteenth Conference on Uncertainty in Artificial Intelligence, pp. 43–52 (1998)

  8. Burke R.: Hybrid recommender systems: survey and experiments. User Model. User-Adapt. Inter. 12(4), 331–370 (2002)

    MATH  Article  Google Scholar 

  9. Burke R.D., Hammond K.J., Young B.C.: The FindMe approach to assisted browsing. IEEE Expert: Intel. Syst Their Appl. 12(4), 32–40 (1997)

    Google Scholar 

  10. Canny, J.: Collaborative filtering with privacy via factor analysis. In: Proceedings of the 25th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 238–245 (2002)

  11. Chen J., Konstan J.A.: Computer science conference paper selectivity and impact: an analysis. Commun. ACM. 53(6), 79–83 (2009)

    Article  Google Scholar 

  12. Chen, Y., Konstan, J.: Online field experiments: lessons from communityLab. In: Proceedings of the Third International Conference on e-Social Science. Ann Arbor, October 7–9, 2007 (2008)

  13. Chen, L., Pu, P.: Preference-based organization interface: aiding user critiques in recommender systems. In: Proceedings of International Conference on User Modeling (UM2007), pp. 77–86. June 25–29, Corfu (2007a)

  14. Chen, L., Pu, P.: Hybrid critiquing-based recommender systems. In: Proceedings of International Conference on Intelligent User Interfaces (IUI2007), pp. 22–31. Jan 28–31. Hawaii (2007b)

  15. Chen, L., Pu, P.: Critiquing-based recommenders: survey and emerging trends. User Model. User-Adapt. Inter. 22(1–2), 125–150 (2012)

    Google Scholar 

  16. Cosley, D., Frankowski, D., Terveen, L., Riedl, J.: SuggestBot: using intelligent task routing to help people find work in wikipedia. In: Proceedings of the 12th international conference on Intelligent user interfaces (IUI ’07), pp. 32–41. ACM, New York (2007)

  17. Cosley, D., Lam, S.K., Albert, I., Konstan, J.A. and Riedl, J.: Is seeing believing? How recommender system interfaces affect users’ opinions. In: Proceedings of the conference on Human Factors in Computing Systems, pp. 585–592 (2003)

  18. Coyle, M., Smyth, B.: (Web Search) shared: social aspects of a collaborative, community-based search network. In: Proceedings of the 5th international conference on Adaptive Hypermedia and Adaptive Web-based Systems (AH ’08), pp. 103–112 (2008)

  19. Deerwester, S., Dumais, S.T., Furnas, G.W., Landauer, T.K., Harshman, R.: Indexing by latent semantic analysis. J. Am. Soc. Inform. Sci. 41(6) (1990)

  20. Ekstrand, M.D., Kannan, P., Stemper, J.A., Butler, J.T., Konstan, J.A., Riedl, J.T.: Automatically building research reading lists. In: Proceedings of the fourth ACM conference on Recommender systems (RecSys ’10), pp. 159–166. ACM, New York (2010)

  21. Ekstrand, M.D., Riedl, J.T., Konstan, J.A.: Collaborative filtering recommender systems. Found. Trends Hum.-Comput. Interact 4 (2011)

  22. Fleder D., Hosanger K.: Blockbuster culture’s next rise or fall: the impact of recommender systems on sales diversity. Manage. Sci. 55(5), 697–712 (2009)

    Article  Google Scholar 

  23. Frankowski, D., Cosley, D., Sen, S., Terveen, L., Riedl, J.: You are what you say: privacy risks of public mentions. In: Proceedings of the 29th annual international ACM SIGIR conference on Research and development in information retrieval (SIGIR ’06), pp. 565–572. ACM, New York (2006)

  24. Harper, F. Maxwell, Frankowski, D., Drenner, S., Ren, Y., Kiesler, S., Terveen, L., Kraut, R., Riedl, J.: Talk amongst yourselves: inviting users to participate in online conversations. In: Proceedings of the 12th international conference on Intelligent user interfaces (IUI ’07), pp. 62–71. ACM, New York (2007)

  25. Harper, F., Li, X., Chen, Y., Konstan, J.: An economic model of user rating in an online recommender system. In: Proceedings of the 10th International Conference on User Modeling, pp. 216–307. Edinburgh (2005)

  26. Herlocker, J.L., Konstan, J.A., Riedl, J.: Explaining collaborative filtering recommendations. In: Proceedings of the 2000 ACM conference on computer supported cooperative work (CSCW ’00), pp. 241–250 (2000)

  27. Herlocker J.L., Konstan J.A., Terveen L.G., Riedl J.T.: Evaluating collaborative filtering recommender systems. ACM Trans. Inf. Syst. 22(1), 5–53 (2004)

    Article  Google Scholar 

  28. Herlocker, J.L., Konstan, J.A., Borchers, A. and Riedl, J.: An algorithmic framework for performing collaborative filtering. In: Proceedings of the 22nd Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 230–237 (1999)

  29. Hill, W., Stead, L., Rosenstein, M., Furnas, G.: Recommending and evaluating choices in a virtual community of use. In: Proceedings of the 1995 Conference on Human Factors in Computing Systems, pp. 194–201. Denver (1995)

  30. Höök K., Karlgren J., Wærn A., Dahlbäck N., Jansson C.G., Karlgren K., Lemaire B.: A glass box approach to adaptive hypermedia. User Model. User-Adapt. Inter. 6(2–3), 157–184 (1996)

    Article  Google Scholar 

  31. Jannach D., Zanker M., Felfernig A., Friedrich G.: Recommender systems: an introduction. Cambridge University Press, New York (2011)

    Google Scholar 

  32. Jaschke, R., Marinho, L., Hotho, A., Schmidt-Thieme, L., Stumme, G.: Tag Recommendations in Folksonomies. In: Kok, J.N., Koronacki, J., De Mantaras, R.L., Matwin, S., Mladeni, D., Jaschke, R., Skowron, A. (eds.) Proceedings of the 11th European conference on Principles and Practice of Knowledge Discovery in Databases (PKDD 2007), pp. 506–514. Springer-Verlag, Berlin, Heidelberg (2007)

  33. Kapoor, N., Butler, J.T., Fouty, G.C., Stemper, J.A., Konstan, J.A.: A study of citations in users’ online personal collections. In: Proceedings of the ACM 2007 European Conference on Digital Libraries, pp. 404–415. Budapest, Hungary (2007a)

  34. Kapoor, N., Butler, J.T., Fouty, G.C., Stemper, J.A., Konstan, J.A.: Resolvability of references in users’ personal collections. In: Proceedings of the Sixth International Conference on Conceptions of Library and Information Sciences (CoLIS), pp. 43–53. Boras (2007b)

  35. Kapoor, N., Chen, J., Butler, J., Fouty, G.C., Stemper, J.A., Riedl, J., Konstan, J.A.: TechLens: a researcher’s desktop. In: The first ACM conference on recommender systems, extended abstracts, pp. 183–184. Minneapolis (2007c)

  36. Kautz H., Selman B., Shah M.: Referral web: combining social networks and collaborative filtering. Commun. ACM 40(3), 63–65 (1997)

    Article  Google Scholar 

  37. Konstan J., Miller B., Maltz D., Herlocker J., Gordon L., Riedl J.: Applying collaborative filtering to usenet news: the GroupLens system. Commun. ACM 40(3), 77–87 (1997)

    Article  Google Scholar 

  38. Konstan, J.A., Riedl, J., Borchers, A., Herlocker, J.L.: Recommender Systems: A GroupLens perspective. In: Proceedings of the AAAI Workshop on Recommender Systems 1998, pp. 60–64. AAAI Technical Report WS-98-08 (1998)

  39. Knijnenburg, B.P., Willemsen, M.C., Gantner, Z., Soncu, H., Newell, C.: Explaining the user experience of recommender systems. User Model. User-Adapt. Inter. 22 (2012). doi:10.1007/s11257-011-9118-4

  40. Lam, S.K., Riedl, J.: Shilling recommender systems for fun and profit. In: Proceedings of the 13th international conference on World Wide Web (WWW ’04), pp. 393–402. ACM, New York (2004)

  41. Leshed, G., Hancock, J.T., Cosley, D., McLeod, P.L., Gay, G.: Feedback for guiding reflection on teamwork practices. In: Proceedings of the 2007 international ACM conference on Supporting group work (GROUP ’07), pp. 217–220. ACM, New York (2007)

  42. Linton F., Schaefer H.-P.: Recommender systems for learning: building user and expert models through long-term observation of application use. User Model. User-Adapt. Inter. 10(2–3), 181–208 (2000)

    Article  Google Scholar 

  43. Lipczak, M., Hu, Y., Kollet, Y., Milios, E.: Tag sources for recommendation in collaborative tagging systems. In: Proceedings of the ECML-PKDD discovery challenge workshop, pp. 157–172. Bled, Slovenia (2009)

  44. Maes P.: Agents that reduce work and information overload. Commun. ACM 37(7), 30–40 (1994)

    Article  Google Scholar 

  45. Marlin, B.M.,Zemel, R.S.: Collaborative prediction and ranking with non-random missing data. In: Proceedings of the third ACM conference on Recommender systems (RecSys ’09), pp. 5–12. ACM, New York (2009)

  46. Masthoff J., Gatt A.: In pursuit of satisfaction and the prevention of embarrassment: Affective state in group recommender systems. User Model. User Adapt. Inter. 16, 281–319 (2006)

    Article  Google Scholar 

  47. McCarthy, J., Anagnost, T.D.: MusicFX: an arbiter of group preferences for computer supported collaborative workouts. In: Proceedings of the 1998 ACM conference on Computer supported cooperative work (CSCW ’98), pp. 363–372. ACM, New York (1998)

  48. McNee, S.M.: Meeting user information needs in recommender systems. Ph.D. Dissertation, University of Minnesota-Twin Cities (2006)

  49. McNee, S.M., Riedl, J., Konstan, J.A.: Making recommendations better: an analytic model for human-recommender interaction. In: CHI ’06 extended abstracts on Human factors in computing systems (CHI ’06), pp. 1103–1108. ACM, New York (2006a)

  50. McNee, S.M., Kapoor, N., Konstan, J.A.: Don’t look stupid: avoiding pitfalls when recommending research papers. In: Proceedings of the 2006 20th anniversary conference on Computer supported cooperative work (CSCW ’06), pp. 171–180. ACM, New York (2006b)

  51. McNee, S.M., Lam, S.K., Konstan, J.A., Riedl, J.: Interfaces for eliciting new user preferences in recommender systems. In: Proceedings of the 9th International Conference on User Modeling, pp. 178–187 (2003)

  52. McNee, S.M., Albert, I., Cosley, D., Gopalkrishnan, P., Lam, S.K., Rashid, A.M., Konstan, J.A., Riedl, J.: On the recommending of citations for research papers. In: Proceedings of the 2002 ACM conference on Computer supported cooperative work (CSCW ’02), pp. 116–125. ACM, New York (2002)

  53. McSherry, F., Mironov, I.: Differentially private recommender systems: building privacy into the net. In: Proceedings of the 15th ACM SIGKDD international conference on Knowledge discovery and data mining (KDD ’09), pp. 627–636. ACM, New York (2009)

  54. Miller B.N., Konstan J.A., Riedl J.: PocketLens: toward a personal recommender system. ACM Trans. Informat. Syst. 22(3), 437–476 (2004)

    Article  Google Scholar 

  55. Miller, B.N., Albert, I., Lam, S.K., Konstan, J.A., Riedl, J.: MovieLens unplugged: experiences with an occasionally connected recommender system. Poster in IUI ’03: Proceedings of the 8th international conference on intelligent user interfaces, pp. 263–266 (2003)

  56. Miller, B.N., Riedl, J.T., Konstan, J.A.: Experience with GroupLens: making usenet useful again. In: Proceedings of the USENIX 1997 Annual Technical Conference, pp. 219–233 (1997)

  57. Mitchell T.M., Caruana R., Freitag D., McDermott J., Zabowski D.: Experience with a learning personal assistant. Commun. ACM 37(7), 80–91 (1994)

    Article  Google Scholar 

  58. Mobasher, B., Burke, R., Bhaumik, R., Williams, C.: Towards trustworthy recommender systems: an analysis of attack models and algorithm robustness. ACM Transac. Internet Technol. 7(4) Article 23 (2007)

    Google Scholar 

  59. Morita, M., Shinoda, Y.: Information filtering based on user behavior analysis and best match text retrieval. In: Proceedings of the 17th annual international ACM SIGIR conference on Research and development in information retrieval, pp. 272–281. Dublin, Ireland Springer-Verlag, New York (1994)

  60. Mrosek, J., Bussman, S., Albers, H., Posdziech, K., Hengefeld, B., Opperman, N., Robert, S., Spira, G.: Content- and graph-based tag recommendation: two variations. In: Proceedings of the ECML-PKDD Discovery Challenge Workshop, pp. 189–199. Bled, Slovenia, Europe, September 7th (2009)

  61. O’Connor, M., Cosley, D., Konstan, J.A., Riedl, J.: PolyLens: a recommender system for groups of users. In: Prinz, W., Jarke, M., Rogers, Y., Schmidt. K., Wulf, V. (eds.) Proceedings of the Seventh European Conference on Computer-Supported Group Work, pp. 199–218. Bonn, Germany 16–20 September (2001)

  62. Park, M.-H., Hong, J.-H., Cho, S.-B.: Location-based recommendation system using bayesian user’s preference model in mobile devices. In: Indulska, J., Ma, J., Yang, L., Ungerer, L., Cao, J. (Eds.) Proceedings of Ubiquitous Intelligence and Computing, 4611, pp. 1130–1139. Springer, Berlin, Heidelberg (2007)

  63. Pu, P., Chen, L.: Trust building with explanation interfaces. In: Proceedings of the 11th international conference on Intelligent user interfaces (IUI ’06), pp. 93–100. ACM, New York (2006)

  64. Pu, P., Chen, L., Hu, R.: Evaluating recommender systems from the user’s perspective: survey of the state of the art. User Model. User-Adapt. Inter. 22 (2012). doi:10.1007/s11257-011-9115-7

  65. Ramezani, M., Sandvig, J.J., Bhaumik, T., Schimoler, R., Burke, B., Mobasher, B.: Exploring the impact of profile injection attacks in social tagging systems. In: Proceedings of the 2008 WebKDD Workshop, held at ACM KDD’2008 Conference on Knowledge Discovery and Data Mining, pp. 134–143. Las Vegas (2008)

  66. Rashid, A.M., Ling, K., Tassone, R., Resnick, P., Kraut, R., Riedl, J.: Motivating participation by displaying the value of contribution. In: Grinter, R., Rodden, T., Aoki, P., Cutrell, E., Jeffries, R., Olson, G. (eds.) Proceedings of the SIGCHI conference on Human Factors in computing systems (CHI ’06), pp. 955–958. ACM, New York (2006)

  67. Rashid, A.M., Albert, I., Cosley, D., Lam, S.K., McNee, S.M., Konstan, J.A., Riedl, J.: Getting to know you: learning new user preferences in recommender systems. In: Proceedings of the 7th international conference on Intelligent user interfaces (IUI ’02), pp. 127–134. ACM, New York (2002)

  68. Reilly, J., Zhang, J., McGinty, L., Pu, P., Smyth, B.: Evaluating compound critiquing recommenders: a real-user study. In: Proceedings of ACM Conference on Electronic Commerce (EC’07), pp. 114–123, June 11–15. San Diego (2007)

  69. Resnick, P., Sami, R.: 2008. Manipulation-resistant recommender systems through influence limits. SIGecom Exch. 7, 3, Article 10 (November 2008), 4 pages

  70. Resnick, P., Sami, R.: The influence limiter: provably manipulation-resistant recommender systems. In: Proceedings of the 2007 ACM conference on Recommender systems (RecSys ’07), pp. 25–32. ACM, New York (2007)

  71. Resnick, P., Iacovou, N., Suchak, M., Bergstrom, P., Riedl, J.: GroupLens: an open architecture for collaborative filtering of netnews. In: Proceedings of the 1994 ACM conference on Computer supported cooperative work (CSCW ’94), pp. 175–186. ACM, New York (1994)

  72. Ricci, F., Rokach, L., Shapira, B., Kantor, P.B. (eds): Recommender systems handbook. Springer, Berlin (2011)

    Google Scholar 

  73. Sandvig, J.J., Bhaumik, R., Ramezani, M., Burke, R., Mobasher, B.: A framework for the analysis of attacks against social tagging systems. In: Proceedings of the 6th Workshop on Intelligent Techniques for Web Personalization and Recommender Systems, held at AAAI 2008. Chicago, July 2008 (2008)

  74. Sarwar, B., Karypis, G., Konstan, J., Riedl, J.: Item-based collaborative filtering recommendation algorithms. In: Proceedings of the 10th international conference on World Wide Web (WWW ’01), pp. 285–295. ACM, New York (2001)

  75. Sarwar, B., Karypis, G., Konstan, J.A, Riedl, J.: Incremental singular value decomposition algorithms for highly scaleable recommender systems. In: Proceedings of the Fifth International Conference on Computer and Information Technology (2002)

  76. Sarwar, B. M., Karypis, G., Konstan, J.A., Riedl, J.T.: Analysis of recommendation algorithms for e-commerce. In: Proceedings of the 2nd ACM conference on Electronic commerce (EC ’00), pp. 158–167. ACM, New York (2000)

  77. Sen, S., Lam, S.K., Cosley, D., Frankowski, D., Osterhouse, J., Harper, F.M., Riedl, J.: tagging, communities, vocabulary, evolution. In Proceedings of ACM CSCW 2006. Baniff, Alberta, Canada, November. ACM New York, NY, pp. 181–190 (2006)

  78. Shardanand, U., Maes, P.: Social information filtering: algorithms for automating “word of mouth”. In: Katz, I.R., Mack, R., Marks, L., Rosson, M.B., Nielsen, J. (eds.) Proceedings of the SIGCHI conference on Human factors in computing systems (CHI ’95), pp. 210–217. ACM Press/Addison-Wesley Publishing Co., New York (1995)

  79. Sinha, R., Swearingen, K.: Comparing recommendations made by online systems and friends. In: DELOS Workshop: Personalisation and Recommender Systems in Digital Libraries (2001)

  80. Smyth, B.: Adaptive information access and the quest for the personalization-privacy sweetspot. In: Proceedings of the 10th international conference on Intelligent user interfaces (IUI ’05), pp. 2–2. ACM, New York (2005)

  81. Swearingen, K., Sinha, R.: Beyond algorithms: an HCI perspective on recommender systems. In: Proceedings of the SIGIR 2001 Workshop on Recommender Systems (2001)

  82. Tintarev, N., Masthoff, J.: Evaluating the effectiveness of explanations for recommender systems. User Model. User-Adapt. Inter. 22 (2012). doi:10.1007/s11257-011-9117-5

  83. Tintarev, N., Masthoff, J.: The effectiveness of personalized movie explanations: an experiment using commercial meta-data. In: Nejdl, W., Kay, J., Pu, P., Herder, E., (eds.) Adaptive Hypermedia and Adaptive Web-based Systems, 5149, pp. 204–213. 5th International Conference (AH 2008), Hannover, Germany. July 29–August 1st (2008)

  84. Tintarev, N., Masthoff, J.: Effective explanations of recommendations: user-centered design. In: Proceedings of the 2007 ACM conference on Recommender systems (RecSys ’07), pp. 153–156. ACM, New York (2007)

  85. Toch, E. Wang, Y, Cranor, L.F.: Personalization and privacy: a survey of privacy risks and remedies in personalization-based systems. User Model. User-Adapt. Inter. 22(1–2), 203–220 (2012)

  86. Torres, R., McNee, S.M., Abel, M., Konstan, J.A., Riedl, J.: Enhancing digital libraries with TechLens+. In: Proceedings of the 4th ACM/IEEE-CS joint conference on Digital libraries (JCDL ’04), pp. 228–236. ACM, New York (2004)

  87. Van Alstyne M., Brynjolfsson E.: Global village or cyber-balkans? Modeling and measuring the integration of electronic communities. Manage. Sci. 51(6), 851–868 (2005)

    MATH  Article  Google Scholar 

  88. Van Alstyne, M., Brynjolfsson, E.: Electronic communities: Global village or cyberbalkans? In: DeGross, J., Jarvenpaa, S., Srinivasan, A. (eds.) Proceedings of the 17th International Conference on Information Systems (ICIS 1996), pp. 80–98. Cleveland, December (1996)

  89. Vig, J., Sen, S., Riedl, J.: Tagsplanations: explaining recommendations using tags. In: Proceedings of the 13th international conference on Intelligent user interfaces (IUI ’09), pp. 47–56. ACM, New York (2009)

  90. Williams, C., Mobasher, B., Burke, R.: Defending recommender systems: detection of profile injection attacks. J. Serv. Ori. Comput. Appl. 1, 157–170 (2007)

    Google Scholar 

  91. Zha H., Zhang Z.: On matrices with low-rank-plus-shift structure: partial SVD and latent semantic indexing. SIAM J. Matrix Anal. Appl. 21(2), 522–536 (1999)

    MathSciNet  MATH  Article  Google Scholar 

  92. Ziegler, C.N., McNee, S.M., Konstan, J.A., Lausen, G.: Improving recommendation lists through topic diversification. In: Proceedings of the Fourteenth International World Wide Web Conference (WWW 2005), pp. 25–32. ACM, New York (2005)

Download references

Author information

Affiliations

Authors

Corresponding author

Correspondence to Joseph A. Konstan.

Rights and permissions

Reprints and Permissions

About this article

Cite this article

Konstan, J.A., Riedl, J. Recommender systems: from algorithms to user experience. User Model User-Adap Inter 22, 101–123 (2012). https://doi.org/10.1007/s11257-011-9112-x

Download citation

Keywords

  • Recommender systems
  • User experience
  • Collaborative filtering
  • Evaluation
  • Metrics