Designing interfaces for explicit preference elicitation: a user-centered investigation of preference representation and elicitation process


Two problems may arise when an intelligent (recommender) system elicits users’ preferences. First, there may be a mismatch between the quantitative preference representations in most preference models and the users’ mental preference models. Giving exact numbers, e.g., such as “I like 30 days of vacation 2.5 times better than 28 days” is difficult for people. Second, the elicitation process can greatly influence the acquired model (e.g., people may prefer different options based on whether a choice is represented as a loss or gain). We explored these issues in three studies. In the first experiment we presented users with different preference elicitation methods and found that cognitively less demanding methods were perceived low in effort and high in liking. However, for methods enabling users to be more expressive, the perceived effort was not an indicator of how much the methods were liked. We thus hypothesized that users are willing to spend more effort if the feedback mechanism enables them to be more expressive. We examined this hypothesis in two follow-up studies. In the second experiment, we explored the trade-off between giving detailed preference feedback and effort. We found that familiarity with and opinion about an item are important factors mediating this trade-off. Additionally, affective feedback was preferred over a finer grained one-dimensional rating scale for giving additional detail. In the third study, we explored the influence of the interface on the elicitation process in a participatory set-up. People considered it helpful to be able to explore the link between their interests, preferences and the desirability of outcomes. We also confirmed that people do not want to spend additional effort in cases where it seemed unnecessary. Based on the findings, we propose four design guidelines to foster interface design of preference elicitation from a user view.


  1. Adomavicius G., Tuzhilin A.: Towards the next generation of recommender systems: A survey of the state-of-the-art and possible extensions. IEEE Trans Knowl Data Eng 17(6), 734–749 (2005)

    Article  Google Scholar 

  2. Aloysius J.A., Davis F.D., Wilson D.D., Taylor A.R., Kottemann J.E.: User acceptance of multi-criteria decision support systems: The impact of preference elicitation techniques. Eur J Oper Res 169, 273–285 (2006)

    MathSciNet  MATH  Article  Google Scholar 

  3. Ames, M., Naaman, M.: Why we tag: motivations for annotation in mobile and online media. In: Proceedings of the SIGCHI conference on Human factors in computing systems, Gaithersburg (2007)

  4. Ardissono L., Felfernig A., Friedrich G., Goy A., Jannach D., Petrone G., Schäfer R., Zanker M.: A framework for the development of personalized, distributed web-based configuration systems. AI Mag 24, 93–108 (2003)

    Google Scholar 

  5. Barneveld, J., Setten, M.: Designing usable interfaces for tv recommender systems. In: Personalized digital television, human-computer interaction series, vol 6, pp. 259–285. Springer, Amsterdam (2004)

  6. Bellucini E., Zeleznikow J.: Developing negotiation decision support systems that support mediators: a case study of the familywinner system. J Artif Intell Law 13(2), 233–271 (2006)

    Article  Google Scholar 

  7. Bettman J.R., Luce M.F., Payne J.W.: Constructive consumer choice processes. J Consum Res 25(3), 187–217 (1998)

    Article  Google Scholar 

  8. Broekens J., Brinkman W.P.: Affectbutton: towards a standard for dynamic affective user feedback. In: Mühl, C., Heylen, D., Nijholt, A. (eds) Proceedings of Affective Computing and Intelligent Interaction (ACII), pp. 1–8. IEEE Computer Society Press, Amsterdam (2009)

    Google Scholar 

  9. Broekens, J., Pommeranz, A., Wiggers, P., Jonker, C.M.: Factors influencing user motivation for giving preference feedback. In: Fifth Multidisciplinary Workshop on Advances in Preference Handling in Conjunction with ECAI 2010, Ulrich Junker, Jérôme Lang and Patrice Perny, pp. 19–24. Lisbon, Portugal, (2010a)

  10. Broekens, J., Pronker, A., Neuteboom, M.: Real time labeling of affect in music using the affectbutton. In: Proceedings of the 3rd international workshop on Affective interaction in natural environments, Ginevra Castellano, Kostas Karpouzis, Jean-Claude Martin, Louis-Philippe Morency, Laurel Riek and Christopher Peters, pp. 21–26. ACM, New York, AFFINE ’10 (2010b)

  11. Burke R.: Knowledge-based recommender systems. In: Kent, A. (ed) Encyclopedia of Library and Information Systems, vol 69, Marcel Dekker, New York (2000)

    Google Scholar 

  12. Burke R.: Hybrid recommender systems: Survey and experiments. User Model User-Adapt Interact 12(4), 331–370 (2002)

    MATH  Article  Google Scholar 

  13. Burke, R.D., Hammond, K.J., Young, B.C.: Knowledge-based navigation of complex information spaces. In: 13th National Conference on Artificial Intelligence, pp. 462–468. AAAI Press, Portland, (1996)

  14. Carenini, G., Loyd, J.: Valuecharts: analyzing linear models expressing preferences and evaluations. In: AVI ’04: Proceedings of the working conference on Advanced visual interfaces, pp. 150–157. ACM, New York (2004)

  15. Carenini, G., Poole, D.: Constructed preferences and value-focused thinking: Implications for ai research on preference elicitation. In: AAAI-02 Workshop on Preferences in AI and CP: symbolic approaches, pp. 1–10. AAAI, Edmonton, Canada (2002)

  16. Carenini, G., Smith, J., Poole, D.: Towards more conversational and collaborative recommender systems. In: 8th international conference on intelligent user interfaces, pp. 12–18. ACM, Miami (2003)

  17. Chen, L., Pu, P.: Survey of preference elicitation methods. Tech. rep., Swiss Federal Institute of Technology In Lausanne (EPFL), Lausanne (2004)

  18. Chen L., Pu P.: Interaction design guidelines on critiquing-based recommender systems. User Model User-Adapt Interact J (UMUAI) 19(3), 167–206 (2009)

    Article  Google Scholar 

  19. Curhan J.R., Neale M.A., Ross L.: Dynamic valuation: preference changes in the context of face-to-face negotiation. J Exp Soc Psychol 40(2), 142–151 (2004)

    Article  Google Scholar 

  20. Faltings, B., Pu, P., Torrens, M., Viappiani, P.: Designing example-critiquing interaction. In: IUI ’04: Proceedings of the ninth international conference on intelligent user interfaces, pp. 22–29. ACM, New York (2004)

  21. Fano, A., Kurth, S.W.: Personal choice point: helping users visualize what it means to buy a bmw. In: IUI ’03: Proceedings of the eighth international conference on intelligent user interfaces, pp. 46–52. ACM, New York (2003)

  22. Fischer G.W., Carmon Z., Ariely D., Zauberman G.: Goal-based construction of preferences: task goals and the prominence effect. Manag Sci 45(8), 1057–1075 (1999)

    MATH  Article  Google Scholar 

  23. Gregorc A.: The Mind Styles Model: Theory, Principles, and Practice. AFG, Columbia (2006)

    Google Scholar 

  24. Herlocker J.L., Konstan J.A., Terveen L.G., Riedl J.T.: Evaluating collaborative filtering recommender systems. ACM Trans Inf Syst 22(1), 5–53 (2004)

    Article  Google Scholar 

  25. Hunt D., Haynes R., Hanna S., Smith K.: Effects of computer-based clinical decision support systems on physician performance and patient outcomes: a systematic review. J Am Med Assoc 280(15), 1339–1346 (1998)

    Article  Google Scholar 

  26. Johnson E., Steffel M., Goldstein D.: Making better decisions: from measuring to constructing preferences. Health Psychol 24(8), 17–22 (2005)

    Article  Google Scholar 

  27. Kay J.: Stereotypes, student models and scrutability. In: Gauthier, G., Frasson, C., VanLehn, K. (eds) Intelligent Tutoring Systems, Lecture Notes in Computer Science, vol 1839, pp. 19–30. Springer, Berlin (2000)

    Google Scholar 

  28. Keeney R.: Value-Focused Thinking: A Path to Creative Decision Making. Harvard University Press, Cambridge (1992)

    Google Scholar 

  29. Keeney R., Raiffa H.: Decisions with Multiple Objectives: Preferences and Value Trade-Offs. Cambridge University Press, Cambridge (1993)

    Google Scholar 

  30. Knijnenburg, B., Willemsen, M., Ganter, Z., Soncu, H., Newell, C.: Explaining the user experience of recommender systems. User Model User-Adapt Interact 22. (2012). doi:10.1007/s11257-011-9118-4

  31. Kramer T.: The effect of measurement task transparency on preference construction and evaluations of personalized recommendations. J Market Res 44(2), 224–233 (2007)

    Article  Google Scholar 

  32. Linden, G., Hanks, S., Lesh, N.: Interactive assessment of user preference models: The automated travel assistant. In: User Modeling: Proceedings of the Sixth International Conference, pp. 67–78. Vienna, Austria (1997)

  33. Liu, F.: Changing for the better: preference dynamics and agent diversity. PhD thesis, University of Amsterdam, Amsterdam (2008)

  34. McCarthy, K.J., Reilly, K., McGinty, L., Smyth, B.: Experiments in dynamic critiquing. In: Proceedings of the 10th International Conference on Intelligent User Interfaces, pp. 175–182. ACM, San Diego (2005)

  35. McFadden D.: Rationality for economists?. J Risk Uncert 19(1), 73–105 (1999)

    MATH  Article  Google Scholar 

  36. Mehrabian, A.: Basic dimensions for a general psychological theory: Implications for personality, social, environmental, and developmental studies. Cambridge (1980)

  37. Miller, B.N., Albert, I., Lam, S.K., Konstan, J.A., Riedl, J.: Movielens unplugged: experiences with an occasionally connected recommender system. In: Proceedings of the eighth international conference on intelligent user interfaces, pp. 263–266. ACM, New York, IUI ’03 (2003)

  38. Mooney, R., Roy, L.: Content-based book recommending using learning for text categorizations. In: Proceedings of the Fifth ACM Conference on Digital Libraries, pp. 195–204. ACM, San Antonio (2000)

  39. Ono C., Kurokawa M., Motomura Y., Asoh H.: A context-aware movie preference model using a bayesian network for recommendation and promotion. In: Conati, C., McCoy, K., Paliouras, G. (eds) User Modeling 2007, Lecture Notes in Computer Science, vol 4511, pp. 247–257. Springer, Berlin (2007)

    Google Scholar 

  40. Payne J.W., Bettman J.R., Schkade D.A.: Measuring constructed preferences: towards a building code. J Risk Uncert 19(1–3), 243–270 (1999)

    MATH  Article  Google Scholar 

  41. Pazzani M.J., Billsus D.: Content-based recommendation systems. In: Brusilovsky, P., Kobsa, A., Nejdl, W. (eds) The adaptive web: methods and strategies of web personalization, pp. 325–341. Springer-Verlag, Berlin (2007)

    Google Scholar 

  42. Peintner B., Viappiani P., Yorke-Smith N.: Preferences in interactive systems: technical challenges and case studies. AI Mag 29(4), 13–24 (2008)

    Google Scholar 

  43. Pommeranz A., Broekens J., Visser W., Brinkman W.P., Wiggers P., Jonker C.: Multi-angle view on preference elicitation for negotiation support systems. In: Brinkman, W.P., Hindriks, K. (eds) Proceedings of First International Working Conference on Human Factors and Computational Models in Negotiation (HuCom08), pp. 19–26. Delft University of Technology, Mediamatica, Delft (2008)

    Google Scholar 

  44. Pommeranz, A., Wiggers, P., Jonker, C.: User-centered design of preference elicitation interfaces for decision support. In: Leitner G, Hitz M, Holzinger A (eds) USAB2010: HCI in Work & Learning, Life & Leisure, Lecture Notes in Computer Science, vol Vol. 6389, pp. 14–33. Springer, Klagenfurt (2010)

  45. Pu P., Chen L.: Trust-inspiring explanation interfaces for recommender systems. Knowledge-Based Systems 20(6), 542–556 (2007)

    Article  Google Scholar 

  46. Pu P., Chen L.: User-involved preference elicitation for product search and recommender systems. AI Mag 29(4), 93–103 (2008)

    MathSciNet  Google Scholar 

  47. Pu, P., Faltings, B., Torrens, M.: User-involved preference elicitation. In: Workshop notes of the Workshop on Configuration, the Eighteenth International Joint Conference on Artificial Intelligence (IJCAI’03), pp. 56–63. Menlo Park (2003)

  48. Pu, P., Chen, L., Hu, R.: Evaluating recommender systems from the user’s perspective: Survey of the state of the art. User Modeling and User-Adap Inter 22. (2012). doi:10.1007/s11257-011-9115-7

  49. Resnick P., Iacovou N., Sushak M., Bergstrom P., Riedl J.: GroupLens: an open architecture for collaborative filtering of netnews. In: Smith, J.B., Smith, F.D., Malone, T.W. (eds) ACM Conference on Computer Supported Collaborative Work Conference, Association of Computing Machinery, pp. 175–186. ACM Press, Chapel Hill (1994)

    Google Scholar 

  50. Shearin S., Lieberman H.: Intelligent profiling by example. In: Sidner, C., Moore, J. (eds) IUI ’01: Proceedings of the Sixth International Conference on Intelligent User Interfaces, pp. 145–151. ACM, New York (2001)

    Google Scholar 

  51. Shiv B., Fedorikhin A.: Heart and mind in conflict: the interplay of affect and cognition in consumer decision making. J Consum Res 26(3), 278–292 (1999)

    Article  Google Scholar 

  52. Simon D., Krawczyk D.C., Holyoak K.J.: Construction of preferences by constraint satisfaction. Psychol Sci 15(5), 331–336 (2004)

    Article  Google Scholar 

  53. Sinha R., Swearingen K.: The role of transparency in recommender systems. In: Wixon, D. (ed) CHI ’02 Extended Abstracts on Human Factors in Computing Systems, pp. 830–831. ACM, New York (2002)

    Google Scholar 

  54. Smyth B., McGinty L.: The power of suggestion. In: Gottlob, G., Walsh, T. (eds) IJCAI-03, Proceedings of the Eighteenth International Joint Conference on Artificial Intelligence, pp. 127–132. Morgan Kauffman, Acapulco (2003)

    Google Scholar 

  55. Stolze, M., Ströbel, M.: Dealing with learning in ecommerce product navigation and decision support: the teaching salesman problem. In: Piller, F.T., Reichwald, R., Tseng, M. (eds) In: Proceedings of the Second Interdisciplinary World Congress on Mass Customization and Personalization, TUM, Munchen, Working Paper Series of the Department for General and Industrial Management, TUM Business School, Technische Universität München (2003)

  56. Viappiani P., Faltings B., Zuber V.S., Pu P.: Stimulating preference expression using suggestions. In: Aha, D.W., Tecuci, G. (eds) Proceedings of the AAAI Fall Symposium on Mixed-Initiative Problem Solving Assistants (FS-05-07), pp. 128–133. AAAI, Washington, Arlington (2005)

    Google Scholar 

  57. Weber E.U., Johnson E.J.: Constructing Preferences from Memory. Cambridge University Press, Cambridge (2006)

    Google Scholar 

Download references


We would like to thank the participants of all three studies. This research is supported by the Dutch Technology Foundation stw, the Applied Science Division of nwo and the Technology Program of the Ministry of Economic Affairs. It is part of the Pocket Negotiator project with grant number vici-project 08075.

Open Access

This article is distributed under the terms of the Creative Commons Attribution License which permits any use, distribution, and reproduction in any medium, provided the original author(s) and the source are credited.

Author information



Corresponding author

Correspondence to Alina Pommeranz.

Rights and permissions

Open Access This is an open access article distributed under the terms of the Creative Commons Attribution Noncommercial License (, which permits any noncommercial use, distribution, and reproduction in any medium, provided the original author(s) and source are credited.

Reprints and Permissions

About this article

Cite this article

Pommeranz, A., Broekens, J., Wiggers, P. et al. Designing interfaces for explicit preference elicitation: a user-centered investigation of preference representation and elicitation process. User Model User-Adap Inter 22, 357–397 (2012).

Download citation


  • Preference elicitation
  • Constructive preferences
  • Interface design