Interaction design guidelines on critiquing-based recommender systems

  • Li ChenEmail author
  • Pearl Pu
Original Paper


A critiquing-based recommender system acts like an artificial salesperson. It engages users in a conversational dialog where users can provide feedback in the form of critiques to the sample items that were shown to them. The feedback, in turn, enables the system to refine its understanding of the user’s preferences and prediction of what the user truly wants. The system is then able to recommend products that may better stimulate the user’s interest in the next interaction cycle. In this paper, we report our extensive investigation of comparing various approaches in devising critiquing opportunities designed in these recommender systems. More specifically, we have investigated two major design elements which are necessary for a critiquing-based recommender system: critiquing coverage—one vs. multiple items that are returned during each recommendation cycle to be critiqued; and critiquing aid—system-suggested critiques (i.e., a set of critique suggestions for users to select) vs. user-initiated critiquing facility (i.e., facilitating users to create critiques on their own). Through a series of three user trials, we have measured how real-users reacted to systems with varied setups of the two elements. In particular, it was found that giving users the choice of critiquing one of multiple items (as opposed to just one) has significantly positive impacts on increasing users’ decision accuracy (particularly in the first recommendation cycle) and saving their objective effort (in the later critiquing cycles). As for critiquing aids, the hybrid design with both system-suggested critiques and user-initiated critiquing support exhibits the best performance in inspiring users’ decision confidence and increasing their intention to return, in comparison with the uncombined exclusive approaches. Therefore, the results from our studies shed light on the design guidelines for determining the sweetspot balancing user initiative and system support in the development of an effective and user-centric critiquing-based recommender system.


Critiquing-based recommender systems Decision support Preference revision User control Example critiquing Dynamic critiquing Hybrid critiquing User evaluation Usability Human–computer interaction 


  1. Agrawal, R., Imielinski, T., Swami, A.: Mining association rules between sets of items in large databases. In: 1993 ACM SIGMOD International Conference on Management of Data, Washington, DC, USA, pp. 207–216. (1993)Google Scholar
  2. Ajzen I.: The theory of planned behavior. Organ. Behav. Hum. Decis. Process. 50, 179–211 (1991)CrossRefGoogle Scholar
  3. Ariely D.: Controlling the information flow: effects on consumers. Decision making and preferences. J. Consum. Res. 27, 233–248 (2000)CrossRefGoogle Scholar
  4. Benbasat I., Nault B.R.: An evaluation of empirical research in managerial support systems. Decis. Support Syst. 6(2), 203–226 (1990)CrossRefGoogle Scholar
  5. Bettman J.R., Johnson E.J., Payne J.W.: A componential analysis of cognitive effort in choice. Organ. Behav. Hum. Decis. Process. 45, 111–139 (1990)CrossRefGoogle Scholar
  6. Burke, R.: Knowledge-based recommender systems. Encyclopedia Library Inform. Syst. 69, Supplement 32 (2000)Google Scholar
  7. Burke, R., Hammond, K., Cooper, E.: Knowledge-based navigation of complex information spaces. In: Thirteenth National Conference on Artificial Intelligence, Portland, Oregon, pp. 462–468 (1996)Google Scholar
  8. Burke R., Hammond K., Young B.: The FindMe approach to assisted browsing. IEEE Expert: Intell. Syst. Appl. 12, 32–40 (1997)Google Scholar
  9. Carenini G., Poole D.: Constructed preferences and value-focused thinking: implications for AI research on preference elicitation. In: AAAI-02 Workshop on Preferences in AI and CP:. Symbolic Approaches. Edmonton, Canada (2002)Google Scholar
  10. Chen, L., Pu, P.: Trust building in recommender agents. In: Workshop on Web Personalization, Recommender Systems and Intelligent User Interfaces at the Second International Conference on E-Business and Telecommunication Networks, Reading, UK, pp. 135–145 (2005)Google Scholar
  11. Chen, L., Pu, P.: Evaluating critiquing-based recommender agents. In: Twenty-first National Conference on Artificial Intelligence, Boston, USA, pp. 157–162 (2006)Google Scholar
  12. Einhorn H., Hogarth R.: Confidence in judgment: persistence of the illusion of validity. Psychol. Rev. 85, 395–416 (1978)CrossRefGoogle Scholar
  13. Falk R.F., Miller N.B.: A Primer for Soft Modeling, 1st edn. The University of Akron Press, Akron Ohio (1992)Google Scholar
  14. Faltings B., Torrens M., Pu P.: Solution generation with qualitative models of preferences. Int. J. Comput. Intell. Appl. 20, 246–264 (2004)MathSciNetGoogle Scholar
  15. Grabner-Kräuter S., Kaluscha E.A.: Empirical research in online trust: a review and critical assessment. Int J Hum–Comput Stud 58, 783–812 (2003)CrossRefGoogle Scholar
  16. Grabner-Kräuter, S., Kaluscha, E.A., Fladnitzer, M.: Perspectives of online trust and similar constructs: a conceptual clarification. In: Eighth International Conference on Electronic Commerce, Fredericton, New Brunswick, Canada, pp. 235–243 (2006)Google Scholar
  17. Häubl G., Trifts V.: Consumer decision making in online shopping environments: the effects of interactive decision aids. Mark Sci 19, 4–21 (2000)CrossRefGoogle Scholar
  18. Hopkins, W.: A New View of Statistics. (1997)
  19. Koufaris, M., Hampton-Sosa, W.: Customer trust online: examining the role of the experience with the web-site. CIS Working Paper Series, Zicklin School of Business, Baruch College, New York, NY (2002)Google Scholar
  20. Linden, G., Hanks, S., Lesh, N.: Interactive assessment of user preference models: the automated travel assistant. In: International Conference on User Modeling, Chia Laguna, Sardinia, Italy, pp. 67–78(1997)Google Scholar
  21. McCarthy, K., Reilly, J., McGinty, L., Smyth, B.: On the dynamic generation of compound critiques in conversational recommender systems. In: Third International Conference on Adaptive Hypermedia and Adaptive Web-Based Systems, Eindhoven, Netherlands, pp. 176–184 (2004a)Google Scholar
  22. McCarthy, K., Reilly, J., McGinty, L.,Smyth, B.: Thinking positively \({\frac{1}{\mu}}\) explanatory feedback for conversational recommender systems. In: Workshop on Explanation in CBR at the Seventh European Conference on Case-Based Reasoning, Madrid, Spain, pp. 115–124 (2004b)Google Scholar
  23. McCarthy, K., McGinty, L., Smyth, B., Reilly, J.: A live-user evaluation of incremental dynamic critiquing. In: Sixth International Conference on Case-based Reasoning, Chicago, IL, USA, pp. 339–352 (2005a)Google Scholar
  24. McCarthy, K., McGinty, L., Smyth,B., Reilly J.: On the evaluation of dynamic critiquing: a large-scale user study. In: Twentieth National Conference on Artificial Intelligence and the Seventeenth Innovative Applications of Artificial Intelligence Conference, Pittsburgh, Pennsylvania, USA, pp. 535–540 (2005b)Google Scholar
  25. McCarthy, K., Reilly, J., McGinty, L., Smyth, B.: Experiments in dynamic critiquing. In: Tenth International Conference on Intelligent User Interfaces, San Diego, California, USA, pp. 175–182 (2005c)Google Scholar
  26. McKnight D.H., Chervany N.L.: What trust means in E-commerce customer relationships: conceptual typology. Int J Electron Commer 6(2), 35–59 (2002)Google Scholar
  27. McNee, S.M., Lam, S.K., Konstan, J.A., Riedl, J.: Interfaces for eliciting new user preferences in recommender systems. In: Ninth International Conference on User Modeling, Johnstown, Pennsylvania, USA, pp. 178–188 (2003)Google Scholar
  28. McSherry, D.: Explanation in recommender systems. In: Workshop Proceedings of the Seventh European Conference on Case-Based Reasoning, Madrid, Spain, pp. 125–134 (2004)Google Scholar
  29. Nielsen, J.: Enhancing the explanatory power of usability heuristics. In: SIGCHI Conference on Human factors in Computing Systems, Boston, USA, pp. 152–158 (1994)Google Scholar
  30. Nguyen, Q.N., Ricci, F., Cavada, D.: Critique-based recommendations for mobile users: GUI design and evaluation. In: Third Workshop on “HCI in Mobile Guides” in Conjunction with Sixth International Conference on Human Computer Interaction with Mobile Devices and Services, Glasgow, Scotland (2004)Google Scholar
  31. Novak T.P., Hoffman D.L., Yung Y.-F.: Measuring the customer experience in online environments: a structural modelling approach. Mark Sci 19(1), 22–42 (2000)CrossRefGoogle Scholar
  32. Payne, J.W., Bettman, J.R., Johnson, E.J.: The Adaptive Decision Maker. Cambridge University Press (1993)Google Scholar
  33. Payne J.W., Bettman J.R., Schkade D.A.: Measuring constructed preference: towards a building code. J Risk Uncertainty 19(1–3), 243–270 (1999)zbMATHCrossRefGoogle Scholar
  34. Pu, P., Chen, L.: Integrating tradeoff support in product search tools for E-commerce sites. In: Sixth ACM Conference on Electronic Commerce, Vancouver, BC, Canada, pp. 269–278 (2005)Google Scholar
  35. Pu, P., Faltings, B.: Enriching buyers’ experiences: the SmartClient approach. In: SIGCHI Conference on Human Factors in Computing Systems, Hague, Netherlands, pp. 289–296 (2000)Google Scholar
  36. Pu P., Faltings B.: Decision tradeoff using example critiquing and constraint programming. Special Issue User-Interact Constraint Satisfaction, CONSTRAINT: an Int. J. 9(4), 289–310 (2004)Google Scholar
  37. Pu, P., Kumar, P.: Evaluating example-based search tools. In: Fifth ACM Conference on Electronic Commerce, New York, NY, USA, pp. 208–217 (2004)Google Scholar
  38. Reilly, J., McCarthy, K., McGinty, L., Smyth, B.: Dynamic critiquing. In: Seventh European Conference on Case-based Reasoning, Madrid, Spain, pp. 763–777 (2004)Google Scholar
  39. Reilly J., McCarthy K., McGinty L., Smyth B.: Explaining compound critiques. Artif. Intell. Rev. 24(2), 199–220 (2005)CrossRefGoogle Scholar
  40. Shimazu, H.: ExpertClerk: navigating shoppers’ buying process with the combination of asking and proposing. In: Seventeenth International Joint Conference on Artificial Intelligence, Seattle, Washington, USA, pp. 1443–1450 (2001)Google Scholar
  41. Smyth, B., McGinty L.: An analysis of feedback strategies in conversational recommenders. In: Fourteenth Irish Artificial Intelligence and Cognitive Science Conference, Dublin, Ireland, pp. 211–216 (2003)Google Scholar
  42. Shneiderman, B.: In: Designing the User Interface: Strategies for Effective Human–Computer Interaction, 3rd edn. Addison-Wesley, Reading, MA (1997)Google Scholar
  43. Spiekermann S., Parachiv C.: Motivating human–agent interaction: transferring insights from behavioral marketing to interface design. J Electron Commer Res 2(3), 255–285 (2002)zbMATHCrossRefGoogle Scholar
  44. Torrens M., Faltings B., Pu P.: SmartClients: constraint satisfaction as a paradigm for scaleable intelligent information systems. Int J Constraints 7(1), 49–69 (2002)zbMATHCrossRefGoogle Scholar
  45. Thompson C.A., Goker M.H., Langley P.: A personalized system for conversational recommendations. J. Artif. Intell. Res. 21, 393–428 (2004)Google Scholar
  46. Tversky A., Simonson I.: Context-dependent preferences. Manage. Sci. 39(10), 1179–1189 (1993)zbMATHCrossRefGoogle Scholar
  47. Viappiani P., Faltings B., Pu P.: Preference-based search using example-critiquing with suggestions. J. Artif. Intell. Res. 27, 465–503 (2007)Google Scholar
  48. Williams, M.D., Tou, F.N.: RABBIT: an interface for database access. In: ACM ’82 Conference, pp. 83–87 (1982)Google Scholar

Copyright information

© Springer Science+Business Media B.V. 2008

Authors and Affiliations

  1. 1.Human Computer Interaction Group, School of Computer and Communication SciencesSwiss Federal Institute of Technology in Lausanne (EPFL)LausanneSwitzerland

Personalised recommendations