A Live-User Evaluation of Incremental Dynamic Critiquing

  • Kevin McCarthy
  • Lorraine McGinty
  • Barry Smyth
  • James Reilly
Part of the Lecture Notes in Computer Science book series (LNCS, volume 3620)


Feature critiquing has emerged as an important feedback strategy for conversational recommender systems as it offers a useful balance between user effort and recommendation efficiency. Dynamic critiquing has recently been presented as an extension to conventional (single-feature) critiquing that supports the simultaneous critiquing of multiple features. To date, dynamic critiquing has been evaluated through a variety of artificial user trials to demonstrate its potential advantages, when it comes to improving recommendation efficiency and quality. However these advantages have never been verified through any large-scale user trial. The contribution of this paper is that we present the results of such an evaluation, which confirms the advantages of dynamic critiquing in a realistic online, e-commerce setting. Furthermore we investigate the impact of implicitly maintaining session specific user models to influence the selection of compound critiques. These models are incrementally constructed as the user critiques example recommendations from cycle to cycle. Our live-user evaluation also enabled us to analyse how real users interact with the compound critiques that are produced in this way. The results demonstrate that our incremental critiquing approach has the capability of generating more relevant critique options, and that users frequently recognise the benefits associated with using these as feedback options, leading to significantly shorter recommendation sessions.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Agrawal, R., Mannila, H., Srikant, R., Toivonen, H., Inkeri Verkamo, A.: Fast Discovery of Association Rules in Large Databases. In: Advances in Knowledge Discovery and Data Mining, pp. 307–328 (1996)Google Scholar
  2. 2.
    Burke, R., Hammond, K., Young, B.: Knowledge-based Navigation of Complex Information Spaces. In: Proceedings of the Thirteenth National Conference on Artificial Intelligence, pp. 462–468. AAAI Press/MIT Press, Portland (1996)Google Scholar
  3. 3.
    Burke, R., Hammond, K., Young, B.C.: The FindMe Approach to Assisted Browsing. Journal of IEEE Expert 12(4), 32–40 (1997)CrossRefGoogle Scholar
  4. 4.
    Faltings, B., Pu, P., Torrens, M., Viappiani, P.: Design Example-Critiquing Interaction. In: Proceedings of the International Conference on Intelligent User Interface(IUI 2004), Funchal, Madeira, Portugal, pp. 22–29. ACM Press, New York (2004)CrossRefGoogle Scholar
  5. 5.
    Linden, G., Hanks, S., Lesh, N.: Interactive Assessment of User Preference Models: The Automated Travel Assistant. In: Paris, C., Jameson, A., Tasso, C. (eds.) User Modeling: Proceedings of the Sixth International Conference, pp. 67–78. Springer, Wien (1997)Google Scholar
  6. 6.
    McCarthy, K., Reilly, J., McGinty, L., Smyth, B.: On the Dynamic Generation of Compound Critiques in Conversational Recommender Systems. In: De Bra, P. (ed.) Proceedings of the Third International Conference on Adaptive Hypermedia and Web-Based Systems (AH 2004), pp. 176–184. Springer, Eindhoven (2004)CrossRefGoogle Scholar
  7. 7.
    McCarthy, K., Reilly, J., McGinty, L., Smyth, B.: On the Evaluation of Dynamic Critiquing: A Large-Scale User Study. Submitted to the Twentieth National Conference on Artificial Intelligence (AAAI 2005), Pittsburgh, Pennsylvania, USA (2005)Google Scholar
  8. 8.
    McSherry, D.: Minimizing Dialog Length in Interactive Case-Based Reasoning. In: Nebel, B. (ed.) Proceedings of the Seventeenth International Joint Conference on Artificial Intelligence (IJCAI 2001), pp. 993–998. Morgan Kaufmann, Seattle (2001)Google Scholar
  9. 9.
    McSherry, D.: Similarity and Compromise. In: Ashley, K.D., Bridge, D.G. (eds.) ICCBR 2003. LNCS, vol. 2689, pp. 291–305. Springer, Heidelberg (2003)CrossRefGoogle Scholar
  10. 10.
    Nguyen, Q.N., Ricci, F., Cavada, D.: User Preferences Initialization and Integration in Critique-Based Mobile Recommender Systems. In: Proceedings of Artificial Intelligence in Mobile Systems 2004, in conjunction with UbiComp 2004, pp. 71–78. Iniversitat des Saarlandes Press, Nottingham (2004)Google Scholar
  11. 11.
    Pu, P., Kumar, P.: Evaluating Example-based Search Tools. In: Proceedings of the ACM Conference on Electronic Commerce (EC 2004), pp. 208–217. ACM Press, New York (2004)CrossRefGoogle Scholar
  12. 12.
    Reilly, J., McCarthy, K., McGinty, L., Smyth, B.: Dynamic Critiquing. In: Funk, P., González Calero, P.A. (eds.) ECCBR 2004. LNCS (LNAI), vol. 3155, pp. 763–777. Springer, Heidelberg (2004)CrossRefGoogle Scholar
  13. 13.
    Reilly, J., McCarthy, K., McGinty, L., Smyth, B.: Incremental Critiquing. In: Bramer, M., Coenen, F., Allen, T. (eds.) Research and Development in Intelligent Systems XXI. Proceedings of AI 2004, pp. 101–114. Springer, Cambridge (2004)Google Scholar
  14. 14.
    Sherin, S., Lieberman, H.: Intelligent Profiling by Example. In: Proceedings of the International Conference on Intelligent User Interfaces (IUI 2001), pp. 145–152. ACM Press, Santa Fe (2001)CrossRefGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2005

Authors and Affiliations

  • Kevin McCarthy
    • 1
  • Lorraine McGinty
    • 1
  • Barry Smyth
    • 1
  • James Reilly
    • 1
  1. 1.Adaptive Information Cluster, Smart Media InstituteUniversity College DublinDublinIreland

Personalised recommendations