User Modeling and User-Adapted Interaction

, Volume 22, Issue 4–5, pp 357–397 | Cite as

Designing interfaces for explicit preference elicitation: a user-centered investigation of preference representation and elicitation process

  • Alina PommeranzEmail author
  • Joost Broekens
  • Pascal Wiggers
  • Willem-Paul Brinkman
  • Catholijn M. Jonker
Open Access
Original Paper


Two problems may arise when an intelligent (recommender) system elicits users’ preferences. First, there may be a mismatch between the quantitative preference representations in most preference models and the users’ mental preference models. Giving exact numbers, e.g., such as “I like 30 days of vacation 2.5 times better than 28 days” is difficult for people. Second, the elicitation process can greatly influence the acquired model (e.g., people may prefer different options based on whether a choice is represented as a loss or gain). We explored these issues in three studies. In the first experiment we presented users with different preference elicitation methods and found that cognitively less demanding methods were perceived low in effort and high in liking. However, for methods enabling users to be more expressive, the perceived effort was not an indicator of how much the methods were liked. We thus hypothesized that users are willing to spend more effort if the feedback mechanism enables them to be more expressive. We examined this hypothesis in two follow-up studies. In the second experiment, we explored the trade-off between giving detailed preference feedback and effort. We found that familiarity with and opinion about an item are important factors mediating this trade-off. Additionally, affective feedback was preferred over a finer grained one-dimensional rating scale for giving additional detail. In the third study, we explored the influence of the interface on the elicitation process in a participatory set-up. People considered it helpful to be able to explore the link between their interests, preferences and the desirability of outcomes. We also confirmed that people do not want to spend additional effort in cases where it seemed unnecessary. Based on the findings, we propose four design guidelines to foster interface design of preference elicitation from a user view.


Preference elicitation Constructive preferences Interface design 



We would like to thank the participants of all three studies. This research is supported by the Dutch Technology Foundation stw, the Applied Science Division of nwo and the Technology Program of the Ministry of Economic Affairs. It is part of the Pocket Negotiator project with grant number vici-project 08075.

Open Access

This article is distributed under the terms of the Creative Commons Attribution License which permits any use, distribution, and reproduction in any medium, provided the original author(s) and the source are credited.


  1. Adomavicius G., Tuzhilin A.: Towards the next generation of recommender systems: A survey of the state-of-the-art and possible extensions. IEEE Trans Knowl Data Eng 17(6), 734–749 (2005)CrossRefGoogle Scholar
  2. Aloysius J.A., Davis F.D., Wilson D.D., Taylor A.R., Kottemann J.E.: User acceptance of multi-criteria decision support systems: The impact of preference elicitation techniques. Eur J Oper Res 169, 273–285 (2006)MathSciNetzbMATHCrossRefGoogle Scholar
  3. Ames, M., Naaman, M.: Why we tag: motivations for annotation in mobile and online media. In: Proceedings of the SIGCHI conference on Human factors in computing systems, Gaithersburg (2007)Google Scholar
  4. Ardissono L., Felfernig A., Friedrich G., Goy A., Jannach D., Petrone G., Schäfer R., Zanker M.: A framework for the development of personalized, distributed web-based configuration systems. AI Mag 24, 93–108 (2003)Google Scholar
  5. Barneveld, J., Setten, M.: Designing usable interfaces for tv recommender systems. In: Personalized digital television, human-computer interaction series, vol 6, pp. 259–285. Springer, Amsterdam (2004)Google Scholar
  6. Bellucini E., Zeleznikow J.: Developing negotiation decision support systems that support mediators: a case study of the familywinner system. J Artif Intell Law 13(2), 233–271 (2006)CrossRefGoogle Scholar
  7. Bettman J.R., Luce M.F., Payne J.W.: Constructive consumer choice processes. J Consum Res 25(3), 187–217 (1998)CrossRefGoogle Scholar
  8. Broekens J., Brinkman W.P.: Affectbutton: towards a standard for dynamic affective user feedback. In: Mühl, C., Heylen, D., Nijholt, A. (eds) Proceedings of Affective Computing and Intelligent Interaction (ACII), pp. 1–8. IEEE Computer Society Press, Amsterdam (2009)Google Scholar
  9. Broekens, J., Pommeranz, A., Wiggers, P., Jonker, C.M.: Factors influencing user motivation for giving preference feedback. In: Fifth Multidisciplinary Workshop on Advances in Preference Handling in Conjunction with ECAI 2010, Ulrich Junker, Jérôme Lang and Patrice Perny, pp. 19–24. Lisbon, Portugal, (2010a)Google Scholar
  10. Broekens, J., Pronker, A., Neuteboom, M.: Real time labeling of affect in music using the affectbutton. In: Proceedings of the 3rd international workshop on Affective interaction in natural environments, Ginevra Castellano, Kostas Karpouzis, Jean-Claude Martin, Louis-Philippe Morency, Laurel Riek and Christopher Peters, pp. 21–26. ACM, New York, AFFINE ’10 (2010b)Google Scholar
  11. Burke R.: Knowledge-based recommender systems. In: Kent, A. (ed) Encyclopedia of Library and Information Systems, vol 69, Marcel Dekker, New York (2000)Google Scholar
  12. Burke R.: Hybrid recommender systems: Survey and experiments. User Model User-Adapt Interact 12(4), 331–370 (2002)zbMATHCrossRefGoogle Scholar
  13. Burke, R.D., Hammond, K.J., Young, B.C.: Knowledge-based navigation of complex information spaces. In: 13th National Conference on Artificial Intelligence, pp. 462–468. AAAI Press, Portland, (1996)Google Scholar
  14. Carenini, G., Loyd, J.: Valuecharts: analyzing linear models expressing preferences and evaluations. In: AVI ’04: Proceedings of the working conference on Advanced visual interfaces, pp. 150–157. ACM, New York (2004)Google Scholar
  15. Carenini, G., Poole, D.: Constructed preferences and value-focused thinking: Implications for ai research on preference elicitation. In: AAAI-02 Workshop on Preferences in AI and CP: symbolic approaches, pp. 1–10. AAAI, Edmonton, Canada (2002)Google Scholar
  16. Carenini, G., Smith, J., Poole, D.: Towards more conversational and collaborative recommender systems. In: 8th international conference on intelligent user interfaces, pp. 12–18. ACM, Miami (2003)Google Scholar
  17. Chen, L., Pu, P.: Survey of preference elicitation methods. Tech. rep., Swiss Federal Institute of Technology In Lausanne (EPFL), Lausanne (2004)Google Scholar
  18. Chen L., Pu P.: Interaction design guidelines on critiquing-based recommender systems. User Model User-Adapt Interact J (UMUAI) 19(3), 167–206 (2009)CrossRefGoogle Scholar
  19. Curhan J.R., Neale M.A., Ross L.: Dynamic valuation: preference changes in the context of face-to-face negotiation. J Exp Soc Psychol 40(2), 142–151 (2004)CrossRefGoogle Scholar
  20. Faltings, B., Pu, P., Torrens, M., Viappiani, P.: Designing example-critiquing interaction. In: IUI ’04: Proceedings of the ninth international conference on intelligent user interfaces, pp. 22–29. ACM, New York (2004)Google Scholar
  21. Fano, A., Kurth, S.W.: Personal choice point: helping users visualize what it means to buy a bmw. In: IUI ’03: Proceedings of the eighth international conference on intelligent user interfaces, pp. 46–52. ACM, New York (2003)Google Scholar
  22. Fischer G.W., Carmon Z., Ariely D., Zauberman G.: Goal-based construction of preferences: task goals and the prominence effect. Manag Sci 45(8), 1057–1075 (1999)zbMATHCrossRefGoogle Scholar
  23. Gregorc A.: The Mind Styles Model: Theory, Principles, and Practice. AFG, Columbia (2006)Google Scholar
  24. Herlocker J.L., Konstan J.A., Terveen L.G., Riedl J.T.: Evaluating collaborative filtering recommender systems. ACM Trans Inf Syst 22(1), 5–53 (2004)CrossRefGoogle Scholar
  25. Hunt D., Haynes R., Hanna S., Smith K.: Effects of computer-based clinical decision support systems on physician performance and patient outcomes: a systematic review. J Am Med Assoc 280(15), 1339–1346 (1998)CrossRefGoogle Scholar
  26. Johnson E., Steffel M., Goldstein D.: Making better decisions: from measuring to constructing preferences. Health Psychol 24(8), 17–22 (2005)CrossRefGoogle Scholar
  27. Kay J.: Stereotypes, student models and scrutability. In: Gauthier, G., Frasson, C., VanLehn, K. (eds) Intelligent Tutoring Systems, Lecture Notes in Computer Science, vol 1839, pp. 19–30. Springer, Berlin (2000)Google Scholar
  28. Keeney R.: Value-Focused Thinking: A Path to Creative Decision Making. Harvard University Press, Cambridge (1992)Google Scholar
  29. Keeney R., Raiffa H.: Decisions with Multiple Objectives: Preferences and Value Trade-Offs. Cambridge University Press, Cambridge (1993)Google Scholar
  30. Knijnenburg, B., Willemsen, M., Ganter, Z., Soncu, H., Newell, C.: Explaining the user experience of recommender systems. User Model User-Adapt Interact 22. (2012). doi: 10.1007/s11257-011-9118-4
  31. Kramer T.: The effect of measurement task transparency on preference construction and evaluations of personalized recommendations. J Market Res 44(2), 224–233 (2007)CrossRefGoogle Scholar
  32. Linden, G., Hanks, S., Lesh, N.: Interactive assessment of user preference models: The automated travel assistant. In: User Modeling: Proceedings of the Sixth International Conference, pp. 67–78. Vienna, Austria (1997)Google Scholar
  33. Liu, F.: Changing for the better: preference dynamics and agent diversity. PhD thesis, University of Amsterdam, Amsterdam (2008)Google Scholar
  34. McCarthy, K.J., Reilly, K., McGinty, L., Smyth, B.: Experiments in dynamic critiquing. In: Proceedings of the 10th International Conference on Intelligent User Interfaces, pp. 175–182. ACM, San Diego (2005)Google Scholar
  35. McFadden D.: Rationality for economists?. J Risk Uncert 19(1), 73–105 (1999)zbMATHCrossRefGoogle Scholar
  36. Mehrabian, A.: Basic dimensions for a general psychological theory: Implications for personality, social, environmental, and developmental studies. Cambridge (1980)Google Scholar
  37. Miller, B.N., Albert, I., Lam, S.K., Konstan, J.A., Riedl, J.: Movielens unplugged: experiences with an occasionally connected recommender system. In: Proceedings of the eighth international conference on intelligent user interfaces, pp. 263–266. ACM, New York, IUI ’03 (2003)Google Scholar
  38. Mooney, R., Roy, L.: Content-based book recommending using learning for text categorizations. In: Proceedings of the Fifth ACM Conference on Digital Libraries, pp. 195–204. ACM, San Antonio (2000)Google Scholar
  39. Ono C., Kurokawa M., Motomura Y., Asoh H.: A context-aware movie preference model using a bayesian network for recommendation and promotion. In: Conati, C., McCoy, K., Paliouras, G. (eds) User Modeling 2007, Lecture Notes in Computer Science, vol 4511, pp. 247–257. Springer, Berlin (2007)Google Scholar
  40. Payne J.W., Bettman J.R., Schkade D.A.: Measuring constructed preferences: towards a building code. J Risk Uncert 19(1–3), 243–270 (1999)zbMATHCrossRefGoogle Scholar
  41. Pazzani M.J., Billsus D.: Content-based recommendation systems. In: Brusilovsky, P., Kobsa, A., Nejdl, W. (eds) The adaptive web: methods and strategies of web personalization, pp. 325–341. Springer-Verlag, Berlin (2007)Google Scholar
  42. Peintner B., Viappiani P., Yorke-Smith N.: Preferences in interactive systems: technical challenges and case studies. AI Mag 29(4), 13–24 (2008)Google Scholar
  43. Pommeranz A., Broekens J., Visser W., Brinkman W.P., Wiggers P., Jonker C.: Multi-angle view on preference elicitation for negotiation support systems. In: Brinkman, W.P., Hindriks, K. (eds) Proceedings of First International Working Conference on Human Factors and Computational Models in Negotiation (HuCom08), pp. 19–26. Delft University of Technology, Mediamatica, Delft (2008)Google Scholar
  44. Pommeranz, A., Wiggers, P., Jonker, C.: User-centered design of preference elicitation interfaces for decision support. In: Leitner G, Hitz M, Holzinger A (eds) USAB2010: HCI in Work & Learning, Life & Leisure, Lecture Notes in Computer Science, vol Vol. 6389, pp. 14–33. Springer, Klagenfurt (2010)Google Scholar
  45. Pu P., Chen L.: Trust-inspiring explanation interfaces for recommender systems. Knowledge-Based Systems 20(6), 542–556 (2007)CrossRefGoogle Scholar
  46. Pu P., Chen L.: User-involved preference elicitation for product search and recommender systems. AI Mag 29(4), 93–103 (2008)MathSciNetGoogle Scholar
  47. Pu, P., Faltings, B., Torrens, M.: User-involved preference elicitation. In: Workshop notes of the Workshop on Configuration, the Eighteenth International Joint Conference on Artificial Intelligence (IJCAI’03), pp. 56–63. Menlo Park (2003)Google Scholar
  48. Pu, P., Chen, L., Hu, R.: Evaluating recommender systems from the user’s perspective: Survey of the state of the art. User Modeling and User-Adap Inter 22. (2012). doi: 10.1007/s11257-011-9115-7
  49. Resnick P., Iacovou N., Sushak M., Bergstrom P., Riedl J.: GroupLens: an open architecture for collaborative filtering of netnews. In: Smith, J.B., Smith, F.D., Malone, T.W. (eds) ACM Conference on Computer Supported Collaborative Work Conference, Association of Computing Machinery, pp. 175–186. ACM Press, Chapel Hill (1994)Google Scholar
  50. Shearin S., Lieberman H.: Intelligent profiling by example. In: Sidner, C., Moore, J. (eds) IUI ’01: Proceedings of the Sixth International Conference on Intelligent User Interfaces, pp. 145–151. ACM, New York (2001)CrossRefGoogle Scholar
  51. Shiv B., Fedorikhin A.: Heart and mind in conflict: the interplay of affect and cognition in consumer decision making. J Consum Res 26(3), 278–292 (1999)CrossRefGoogle Scholar
  52. Simon D., Krawczyk D.C., Holyoak K.J.: Construction of preferences by constraint satisfaction. Psychol Sci 15(5), 331–336 (2004)CrossRefGoogle Scholar
  53. Sinha R., Swearingen K.: The role of transparency in recommender systems. In: Wixon, D. (ed) CHI ’02 Extended Abstracts on Human Factors in Computing Systems, pp. 830–831. ACM, New York (2002)CrossRefGoogle Scholar
  54. Smyth B., McGinty L.: The power of suggestion. In: Gottlob, G., Walsh, T. (eds) IJCAI-03, Proceedings of the Eighteenth International Joint Conference on Artificial Intelligence, pp. 127–132. Morgan Kauffman, Acapulco (2003)Google Scholar
  55. Stolze, M., Ströbel, M.: Dealing with learning in ecommerce product navigation and decision support: the teaching salesman problem. In: Piller, F.T., Reichwald, R., Tseng, M. (eds) In: Proceedings of the Second Interdisciplinary World Congress on Mass Customization and Personalization, TUM, Munchen, Working Paper Series of the Department for General and Industrial Management, TUM Business School, Technische Universität München (2003)Google Scholar
  56. Viappiani P., Faltings B., Zuber V.S., Pu P.: Stimulating preference expression using suggestions. In: Aha, D.W., Tecuci, G. (eds) Proceedings of the AAAI Fall Symposium on Mixed-Initiative Problem Solving Assistants (FS-05-07), pp. 128–133. AAAI, Washington, Arlington (2005)Google Scholar
  57. Weber E.U., Johnson E.J.: Constructing Preferences from Memory. Cambridge University Press, Cambridge (2006)Google Scholar

Copyright information

© The Author(s) 2012

Open AccessThis is an open access article distributed under the terms of the Creative Commons Attribution Noncommercial License (, which permits any noncommercial use, distribution, and reproduction in any medium, provided the original author(s) and source are credited.

Authors and Affiliations

  • Alina Pommeranz
    • 1
    Email author
  • Joost Broekens
    • 1
  • Pascal Wiggers
    • 1
  • Willem-Paul Brinkman
    • 1
  • Catholijn M. Jonker
    • 1
  1. 1.Department of Mediamatics, MMI GroupDelft University of TechnologyDelftThe Netherlands

Personalised recommendations