Usability Guidelines for Product Recommenders Based on Example Critiquing Research

  • Pearl PuEmail author
  • Boi Faltings
  • Li Chen
  • Jiyong Zhang
  • Paolo Viappiani


Over the past decade, our group has developed a suite of decision tools based on example critiquing to help users find their preferred products in e-commerce environments. In this chapter, we survey important usability research work relative to example critiquing and summarize the major results by deriving a set of usability guidelines. Our survey is focused on three key interaction activities between the user and the system: the initial preference elicitation process, the preference revision process, and the presentation of the systems recommendation results. To provide a basis for the derivation of the guidelines, we developed a multi-objective framework of three interacting criteria: accuracy, confidence, and effort (ACE). We use this framework to analyze our past work and provide a specific context for each guideline: when the system should maximize its ability to increase users’ decision accuracy, when to increase user confidence, and when to minimize the interaction effort for the users. Due to the general nature of this multi-criteria model, the set of guidelines that we propose can be used to ease the usability engineering process of other recommender systems, especially those used in e-commerce environments. The ACE framework presented here is also the first in the field to evaluate the performance of preference-based recommenders from a user-centric point of view.

Designers can use these guidelines for the implementation of an effective and successful product recommender.


Recommender System Product Recommender Preference Elicitation Decision Accuracy Usability Guideline 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Adomavicius, G., Tuzhilin, A., Toward the next generation of recommender systems: a survey of the state-of-the-art and possible extensions. IEEE Transactions on Knowledge and Data 542 Pearl Pu, Boi Faltings, Li Chen, Jiyong Zhang and Paolo Viappiani Engineering 17 (6) (2005) 734-749.Google Scholar
  2. 2.
    Agrawal, R., Imielinski, T., Swami, A., Mining association rules between sets of items in large databases. Proceedings of the 1993 ACM SIGMOD International Conference on Management of Data, ACM Press, 1993, 207216.Google Scholar
  3. 3.
    Belkin, N.J., Croft, W.B., Information filtering and information retrieval: two sides of the same coin? Communications of the ACM 35 (12) (1992) 29-38.Google Scholar
  4. 4.
    Bernoulli, D., Exposition of a new theory on the measurement of risk (original 1738). Econometrica 22 (1) (1954) 23-36.CrossRefMathSciNetGoogle Scholar
  5. 5.
    Blythe, J., Visual exploration and incremental utility elicitation. Proceedings of the 18th National Conference on Artificial Intelligence, AAAI press, 2002, 526-532.Google Scholar
  6. 6.
    Bradley, K., Smyth, B., Improving recommendation diversity. Proceedings of the 12th Irish Conference on Artificial Intelligence and Cognitive Science, 2001, 85-94.Google Scholar
  7. 7.
    Burke, R., Hybrid recommender systems: survey and experiments. User Modeling and User- Adapted Interaction 12 (4) (2002) 331-370.Google Scholar
  8. 8.
    Burke, R., Hammond, K., Cooper, E., Knowledge-based navigation of complex information spaces. Proceedings of the 13th National Conference on Artificial Intelligence, AAAI press, 1996, 462-468.Google Scholar
  9. 9.
    Burke, R., Hammond, K., Young, B., The FindMe approach to assisted browsing. IEEE Expert: Intelligent Systems and Their Applications 12 (4) (1997) 32-40.Google Scholar
  10. 10.
    Chen, L., Pu, P., Trust building in recommender agents. Proceedings of the Workshop on Web Personalization, Recommender Systems and Intelligent User Interfaces at the 2nd International Conference on E-Business and Telecommunication Networks, 2005, 135-145.Google Scholar
  11. 11.
    Chen, L., Pu, P., Evaluating critiquing-based recommender agents. Proceedings of TwentyfirstNational Conference on Artificial Intelligence (AAAI-06), 2006, 157-162.Google Scholar
  12. 12.
    Cover, T.M., Hart, P.E., Nearest neighbor pattern classification. IEEE Transactions on Information Theory, IT-13, 1967, 21-27.Google Scholar
  13. 13.
    Faltings, B., Pu, P., Torrens, M., Viappiani, P., Designing example-critiquing interaction. Proceedings of the 9th International Conference on Intelligent User Interfaces (IUI’04), ACM Press, 2004, 22-29.Google Scholar
  14. 14.
    Faltings, B., Torrens, M., Pu, P., Solution generation with qualitative models of preferences. Computational Intelligence 20 (2) (2004), 246-263.Google Scholar
  15. 15.
    Freuder, E.C., Wallace, R.J., Partial constraint satisfaction. Artificial Intelligence 58 (1-3) (1992) 21-70.Google Scholar
  16. 16.
    Goker, M., Thompson, C., The adaptive place advisor: a conversational recommendation system. Proceedings of the 8th German Workshop on Case Based Reasoning, 2000.Google Scholar
  17. 17.
    Goldberg, D., Nichols, D., Oki, B.M., Terry, D., Using collaborative filtering to weave an information tapestry. Communications of the ACM (35) 12, Special issue on information filtering (1992) 61-70.Google Scholar
  18. 18.
    Good, N., Schafer, J.B., Konstan, J.K., Borchers, A., Sarwar, B.M., Herlocker, J.L., Riedl, J., Combining collaborative filtering with personal agents for better recommendations. Proceedings of the 16th National Conference on Artificial Intelligence (AAAI’99), AAAI press, 1999, 439-446.Google Scholar
  19. 19.
    Ha, V.A., Haddawy, P., Problem-focused incremental elicitation of multi-attribute utility models. In Shenoy, P. (ed.), Proceedings of the 13th Conference on Uncertainty in Artificial Intelligence (UAI’97), 1997, 215-222.Google Scholar
  20. 20.
    Haubl, G., Trifts, V., Consumer decision making in online shopping environments: the effects of interactive decision aids. Marketing Science 19 (1) (2000) 4-21.Google Scholar
  21. 21.
    Herstein, I.N., Milnor, J., An axiomatic approach to measurable utility. Econometrica 21 (2)(1953) 291-297.zbMATHCrossRefMathSciNetGoogle Scholar
  22. 22.
    Hill, W., Stead, L., Rosenstein, M., Furnas, G., Recommending and evaluating choices in a virtual community of use. Proceedings of the CHI ’95 Conference on Human Factors in Computing Systems, 1995, 194-201.Google Scholar
  23. 23.
    Keeney, R.L., Value-Focused Thinking: A Path to Creative Decision Making, Harvard University Press (1992).Google Scholar
  24. 24.
    Keeney, R.L., Raiffa, H., Decisions with Multiple Objectives: Preferences and Value Tradeoffs, New York: Wiley (1976).Google Scholar
  25. 25.
    L, J., Kolodner. Case-Based Reasoning. San Mateo, CA: Morgan Kaufmann (1993).Google Scholar
  26. 26.
    Krulwich, B., Lifestyle finder: intelligent user profiling using large-scale demographic data. Artificial Intelligence Magazine 18 (2) (1997) 37-45.Google Scholar
  27. 27.
    Lang, K., Newsweeder: learning to filter news. Proceedings of the 12th International Conference on Machine Learning, 1995, 331-339.Google Scholar
  28. 28.
    Linden, G., Hanks, S., Lesh, N., Interactive assessment of user preference models: the automated travel assistant. Proceedings of the 6th International Conference on User Modeling (UM’97), New York: Springer Wien New York, 1997, 67-78.Google Scholar
  29. 29.
    Marschak, J., Rational Behavior, Uncertain Prospects, and Measurable Utility. Econometrica 18 (2) (1950) 111-141.CrossRefMathSciNetGoogle Scholar
  30. 30.
    McCarthy K., McGinty L., Smyth, B., Reilly, J., A live-user evaluation of incremental dynamic critiquing. Proceedings of the 6th International Conference on Case-Based Reasoning (ICCBR’05), 2005, 339-352.Google Scholar
  31. 31.
    McCarthy K., Reilly, J., L. McGinty, Smyth, B., Thinking positively explanatory feedback for conversational recommender systems. Proceedings of the Workshop on Explanation in CBR at the 7th European Conference on Case-Based Reasoning (ECCB’04), 2004, 115-124.Google Scholar
  32. 32.
    McCarthy K., Reilly, J., L. McGinty, Smyth, B., Experiments in dynamic critiquing. Proceedings of the 10th International Conference on Intelligent User Interfaces (IUI’05), New York: ACM Press, 2005, 175-182.Google Scholar
  33. 33.
    McGinty L., Smyth, B., On the role of diversity in conversational recommender systems. Proceedings of the 5th International Conference on Case-Based Reasoning (ICCBR’03), 2003, 276-290.Google Scholar
  34. 34.
    McNee S.M., Lam, S.K., Konstan, J., Riedl, J., Interfaces for eliciting new user preferences in recommender systems. Proceedings of User Modeling Conference, Springer, 2003, 178–187.Google Scholar
  35. 35.
    McNee S.M., Riedl, J., Konstan, J., Being accurate is not enough: how accuracy metrics have hurt recommender systems. In CHI ’06 Extended Abstracts on Human Factors in Computing Systems (CHI ’06), ACM, New York, NY, 1097-1101.Google Scholar
  36. 36.
    McSherry, D., Diversity-conscious retrieval. In, Craw, S., Preece, A. (eds.), Proceedings of the 6th European Conference on Advances in Case-Based Reasoning, London: Springer-Verlag, 2002, 219-233.Google Scholar
  37. 37.
    McSherry, D., Similarity and compromise. Proceedings of the 5th International Conference on Case-Based Reasoning (ICCBR’03), Springer-Verlag, 2003, 291-305.Google Scholar
  38. 38.
    McSherry, D., Explanation in recommender systems. Workshop Proceedings of the 7th European Conference on Case-Based Reasoning (ECCBR’04), 2004, 125-134.Google Scholar
  39. 39.
    Mongin, P., Expected Utility Theory. Handbook of Economic Methodology, Edward Elgar, 1998, 342-350.Google Scholar
  40. 40.
    Payne, J.W., Bettman, J.R., Johnson, E.J., The Adaptive Decision Maker, Cambridge University Press (1993).Google Scholar
  41. 41.
    Payne, J.W., Bettman, J.R., Schkade, D.A., Measuring constructed preferences: towards a building code. Journal of Risk and Uncertainty 19 (1999) 243-270.Google Scholar
  42. 42.
    Price, B., Messinger, P.R., Optimal recommendation sets: covering uncertainty over user preferences. Proceedings of the 20th National Conference on Artificial Intelligence (AAAI’05), 2005, 541-548.Google Scholar
  43. 43.
    Pu, P., Faltings, B., Enriching buyers’ experiences: the SmartClient approach. Proceedings of the SIGCHI conference on Human factors in computing systems (CHI’00), New York: ACM Press, 2000, 289-296.Google Scholar
  44. 44.
    Pu, P., Faltings, B., Torrens, M., User-involved preference elicitation. Working Notes of the Workshop on Configuration. Eighteenth International Joint Conference on Artificial Intelligence (IJCAI’03), 2003, 56-63.Google Scholar
  45. 45.
    Pu, P., Faltings, B., Decision tradeoff using example-critiquing and constraint programming. Constraints: an International Journal 9 (4) (2004) 289-310. 544 Pearl Pu, Boi Faltings, Li Chen, Jiyong Zhang and Paolo ViappianiCrossRefGoogle Scholar
  46. 46.
    Pu, P., Faltings, B., Torrens, M., Effective interaction principles for online product search environments. Proceedings of the IEEE/WIC/ACM International Joint Conference on Intelligent Agent Technology and Web Intelligence, 2004, 724-727.Google Scholar
  47. 47.
    Pu, P., Kumar, P., Evaluating example-based search tools. Proceedings of the 5th ACM Conference on Electronic Commerce (EC’04), ACM Press, 2004, 208-217.Google Scholar
  48. 48.
    Pu, P., Chen, L., Integrating tradeoff support in product search tools for e-commerce sites. Proceeding of the 6th ACM Conference on Electronic Commerce (EC’05), ACM Press, 2005, 269-278.Google Scholar
  49. 49.
    Pu, P., Chen, L., Trust building with explanation interfaces. Proceedings of the 11th International Conference on Intelligent User Interface (IUI’06), 2006, 93-100.Google Scholar
  50. 50.
    Pu, P., Viappiani, P., Faltings, B., Stimulating decision accuracy using suggestions. SIGCHI conference on Human factors in computing systems (CHI’06), 2006, 121-130.Google Scholar
  51. 51.
    Pu P., Chen L. and Kumar P., Evaluating Product Search and Recommender Systems for ECommerce Environments. Electronic Commerce Research Journal 8(1-2), June 2008, 1-27.Google Scholar
  52. 52.
    Pu P., Chen L., User-Involved Preference Elicitation for Product Search and Recommender Systems. AI Magazine 29(4), Winter 2008, 93-103.Google Scholar
  53. 53.
    Reilly, J., K. McCarthy, L. McGinty, Smyth, B., Dynamic critiquing. Proceedings of the 7th European Conference on Case-Based Reasoning (ECCBR’04), Springer, 2004, 763-777.Google Scholar
  54. 54.
    Resnick, P., Iacovou, N., Suchak, M., Bergstrom, P., Riedl, J., GroupLens: an open architecture for collaborative filtering of Netnews. CSCW ’94: Conference on Computer Supported Cooperative Work, ACM, 1994, 175-186.Google Scholar
  55. 55.
    Rich, E., User modeling via stereotypes. Cognitive Science 3 (1979) 329-354.Google Scholar
  56. 56.
    Rokach, L., Maimon, O., Averbuch, M., Information Retrieval System for Medical Narrative Reports, Lecture Notes in Artificial intelligence 3055, page 217-228 Springer-Verlag, 2004.Google Scholar
  57. 57.
    Schafer, J.B., Konstan, J.A., Riedl, J., Recommender systems in e-commerce. Proceedings of the ACM Conference on Electronic Commerce, ACM, 1999, 158-166.Google Scholar
  58. 58.
    Shardanand, U., Maes, P., Social information filtering: algorithms for automating ”Word of Mouth”. Proceedings of the Conference on Human Factors in Computing Systems (CHI ’95), 1995, 210-217.Google Scholar
  59. 59.
    Schoemaker, P., The Expected Utility Model: Its Variants, Purposes, Evidence and Limitations. Journal of Economic Literature 20 (2) (1982), 529-563.Google Scholar
  60. 60.
    Shimazu, H., ExpertClerk: navigating shoppers’ buying process with the combination of asking and proposing. Proceedings of the 17th International Joint Conference on Artificial Intelligence (IJCAI’01), 2001, 1443-1448Google Scholar
  61. 61.
    Smyth, B., P. McClave. Similarity vs. diversity. Proceedings of the 4th International Conference on Case-Based Reasoning (ICCBR’01), Springer-Verlag, 2001, 347-361.Google Scholar
  62. 62.
    Spiekermann, S., Parachiv, C., Motivating human-agent interaction: transferring insights from behavioral marketing to interface design. Journal of Electronic Commerce Research 2 (3) (2002) 255-285.zbMATHCrossRefGoogle Scholar
  63. 63.
    Stolze, M., Soft navigation in electronic product catalogs. International Journal on Digital Libraries 3 (1) (2000) 60-66.CrossRefGoogle Scholar
  64. 64.
    Torrens, M., Faltings, B., Pu, P., SmartClients: constraint satisfaction as a paradigm forscaleable intelligent information systems. International Journal of Constraints 7 (1) (2002) 49-69.zbMATHCrossRefGoogle Scholar
  65. 65.
    Tversky, A., Contrasting rational and psychological principles in choice. Wise Choices: Decisions, Games, and Negotiations, Boston, MA: Harvard Business School Press (1996) 5-21.Google Scholar
  66. 66.
    Tversky, A., Simonson, I., Context-dependent preferences. Management Science 39 (10) (1993) 1179-1189.Google Scholar
  67. 67.
    Viappiani, P., Faltings, B., V. Schickel-Zuber, Pu, P., Stimulating preference expression using suggestions. Mixed-Initiative Problem-Solving Assistants, AAAI Fall Symposium Series, AAAI press, 2005, 128-133.Google Scholar
  68. 68.
    Viappiani, P., Faltings, B., Pu, P., Evaluating preference-based search tools: a tale of two approaches. Proceedings of the Twenty-first National Conference on Artificial Intelligence (AAAI-06), 2006, 205-210. 16 Usability Guidelines for Product Recommenders 545Google Scholar
  69. 69.
    Viappiani, P., B. Faltings and Pu, P., Preference-based Search using Example-Critiquing with Suggestions. Journal of Artificial Intelligence Research (JAIR), 27, 2006, 465-503.Google Scholar
  70. 70.
    Paolo Viappiani, Pearl Pu, and Boi Faltings. Preference-based Search with Adaptive Recommendations. AI Communications 21>(2-3), 2008, 155-175.zbMATHMathSciNetGoogle Scholar
  71. 71.
    J. von Neumann, Morgenstern, O., The Theory of Games and Economic Behavior, Princeton University Press (1944).Google Scholar
  72. 72.
    Williams, M.D., Tou, F.N., RABBIT: an interface for database access. Proceedings of the ACM ’82 Conference, ACM Press, 1982, 83-87.Google Scholar
  73. 73.
    Zhang, J., Pu, P., Performance evaluation of consumer decision support systems. International Journal of E-Business Research 2 (2006) Idea Group Publishing.Google Scholar
  74. 74.
    Ziegler, C.N., S.M. McNee, Konstan, J.A., Lausen, G., Improving recommendation lists through topic diversification. Proceedings of the 14th International World Wide Web Conference (WWW’05), 2005, 22-32.Google Scholar
  75. 75.
    Zukerman, I., Albrecht, D.W., Predictive statistical models for user modeling. User Modeling and User-Adapted Interaction 11> (1-2) (2001) 5-18.zbMATHCrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media, LLC 2011

Authors and Affiliations

  • Pearl Pu
    • 1
    Email author
  • Boi Faltings
    • 2
  • Li Chen
    • 1
  • Jiyong Zhang
    • 1
  • Paolo Viappiani
    • 3
  1. 1.Human Computer Interaction Group, School of Computer and Communication SciencesSwiss Federal Institute of Technology in Lausanne (EPFL)LausanneSwitzerland
  2. 2.Artificial Intelligence Laboratory, School of Computer and Communication SciencesSwiss Federal Institute of Technology in Lausanne (EPFL)LausanneSwitzerland
  3. 3.Department of Computer ScienceUniversity of TorontoTorontoCanada

Personalised recommendations