Improving Ranking by Respecting the Multidimensionality and Uncertainty of User Preferences

  • Bettina Berendt
  • Veit Köppen
Part of the Studies in Computational Intelligence book series (SCI, volume 301)


Rankings or ratings are popular methods for structuring large information sets in search engines, e-Commerce, e-Learning, etc. But do they produce the right rankings for their users? In this paper, we give an overview of major evaluation approaches for rankings as well as major challenges facing the use and usability of rankings. We point out the importance of an interdisciplinary perspective for a truly user-centric evaluation of rankings. We then focus on two central problems: the multidimensionality of the criteria that influence both users’ and systems’ rankings, and the randomness inherent in users’ preferences. We propose multicriteria decision analysis and the integration of randomness into rankings as solution approaches to these problems. We close with an outlook on new challenges arising for ranking when systems address not only individuals, but also groups.


Search Engine Analytic Hierarchy Process Recommender System Ranking Function Multi Criterion Decision Analysis 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. [Altman and Tennenholtz 2008]
    Altman, A., Tennenholtz, M.: Axiomatic foundations for ranking systems. Journal of Artificial Intelligence Research 31, 473–495 (2008), (retrieved 2009-06-15)zbMATHMathSciNetGoogle Scholar
  2. [Bamberg and Coenenberg2002]
    Bamberg, G., Coenenberg, A.G.: Betriebswirtschaftliche Entscheidungslehre, 11th edn. Vahlen, Munich (2002)Google Scholar
  3. [Berendt2009]
    Berendt, B.: Ranking – use and usability. To appear in Bulletin of the Belgian Mathematical Society – Simon Stevin (2009)Google Scholar
  4. [Berendt et al.2005]
    Berendt, B., Günther, O., Spiekermann, S.: Privacy in e-commerce: Stated preferences vs. actual behavior. Communications of the ACM 48(4), 101–106 (2005)CrossRefGoogle Scholar
  5. [Berendt and Kralisch2009]
    Berendt, B., Kralisch, A.: A user-centric approach to identifying best deployment strategies for language tools: The impact of content and access language on web user behaviour and attitudes. Journal of Information Retrieval 12(3), 380–399 (2009)CrossRefGoogle Scholar
  6. [Berendt and Trümper2009]
    Berendt, B., Trümper, D.: Semantics-based analysis and navigation of heterogeneous text corpora: The porpoise news and blogs engine. In: Ting, I.-H., Wu, H.-J. (eds.) Web Mining Applications in E-commerce and E-services, pp. 45–64. Springer, Berlin (2009)CrossRefGoogle Scholar
  7. [Chakrabarti2003]
    Chakrabarti, S.: Mining the Web. Morgan Kaufmann, San Francisco (2003)Google Scholar
  8. [Dix et al.1998]
    Dix, A., Finlay, J., Abowd, G., Beale, R.: Human Computer Interaction. Prentice Hall Europe, Englewood Cliffs (1998)Google Scholar
  9. [d’Ocagne1885]
    d’Ocagne, M.: Coordonnées Parallèles et Axiales: Méthode de transformation géométrique et procédé nouveau de calcul graphique déduits de la considération des coordonnées parallèlles. Gauthier-Villars, Paris (1885)Google Scholar
  10. [Eyetools2008]
    Eyetools, Eyetools research and reports: Eyetools, enquiro, and did-it uncover search’s golden triangle (2008), (retrieved 2009-06-15)
  11. [Figueira et al.2005]
    Figueira, J., Greco, S., Erhgott, M. (eds.): Multiple Criteria Decision Analysis: State of the art surveys. Springer Science and Business Media, Boston (2005)zbMATHGoogle Scholar
  12. [Glänzel2008]
    Glänzel, W.: Seven myths in bibliometrics about facts and fiction in quantitative science studies. Collnet Journal of Scientometrics and Information Management 2(1), 9–17 (2008), (retrieved 2009-06-15)Google Scholar
  13. [Hearst2006]
    Hearst, M.A.: Design recommendations for hierarchical faceted search interfaces. In: SIGIR 2006 Faceted Search Workshop (2006), (retrieved 2009-06-15)
  14. [Herlocker et al.2004]
    Herlocker, J.L., Konstan, J.A., Terveen, L.G., Riedl, J.T.: Evaluating collaborative filtering recommender systems. ACM Trans. Inf. Syst. 22(1), 5–53 (2004)CrossRefGoogle Scholar
  15. [Hu et al.2007]
    Hu, J., Zeng, H.-J., Li, H., Niu, C., Chen, Z.: Demographic prediction based on user’s browsing behavior. In: WWW 2007: Proceedings of the 16th international conference on World Wide Web, pp. 151–160. ACM, New York (2007)CrossRefGoogle Scholar
  16. [Inselberg1985]
    Inselberg, A.: The plane with parallel coordinates. Visual Computer 1(4), 69–91 (1985)zbMATHCrossRefGoogle Scholar
  17. [International Organization for Standardization2007]
    International Organization for Standardization, ISO 9241-400:2007. ergonomics of human–system interaction – part 400: Principles and requirements for physical input devices (2007), (retrieved 2009-06-15)
  18. [International Organization for Standardization2008]
    International Organization for Standardization, ISO 9241-151:2008. ergonomics of human-system interaction – part 151: Guidance on world wide web user interfaces (2008), (retrieved 2009-06-15)
  19. [Jameson2003]
    Jameson, A.: Adaptive interfaces and agents. In: Jacko, J.A., Sears, A. (eds.) Human-Computer Interaction Handbook, pp. 305–330. Erlbaum, Mahwah (2003), (retrieved 2009-06-15)Google Scholar
  20. [Jameson and Smyth2007]
    Jameson, A., Smyth, B.: Recommendation to groups. In: Brusilovsky, P., Kobsa, A., Nejdl, W. (eds.) Adaptive Web 2007. LNCS, vol. 4321, pp. 596–627. Springer, Heidelberg (2007)CrossRefGoogle Scholar
  21. [Jansen et al.2007]
    Jansen, B.J., Zhang, M., Zhang, Y.: Brand awareness and the evaluation of search results. In: WWW 2007: Proceedings of the 16th international conference on World Wide Web, pp. 1139–1140. ACM, New York (2007)CrossRefGoogle Scholar
  22. [Kaplan and Norton1996]
    Kaplan, R.S., Norton, D.P.: The Balanced Scorecard. Translating Strategy Into Action. Harvard Business School Press (1996)Google Scholar
  23. [Kaplan and Norton2004]
    Kaplan, R.S., Norton, D.P.: Strategy Maps: Converting Intangible Assets Into Tangible Outcomes. Harvard Business School Press (2004)Google Scholar
  24. [Keeney and Raiffa1976]
    Keeney, R., Raiffa, H.: Decisions with Multiple Objectives; Preferences and Value Tradeoffs. John Wiley & Sons, Chichester (1976)Google Scholar
  25. [Kobsa and Teltzrow2005]
    Kobsa, A., Teltzrow, M.: Impacts of contextualized communication of privacy practices and personalization benefits on purchase behavior and perceived quality of recommendation. In: Beyond Personalization 2005: A Workshop on the Next Stage of Recommender Systems Research (IUI 2005), San Diego, CA, pp. 48–53 (2005)Google Scholar
  26. [Kralisch and Berendt2005]
    Kralisch, A., Berendt, B.: Language-sensitive search behaviour and the role of domain knowledge. The New Review of Hypermedia and Multimedia 11(2), 221–246 (2005)CrossRefGoogle Scholar
  27. [Kralisch and Köppen2005]
    Kralisch, A., Köppen, V.: The impact of language on website use and user satisfaction: Project description. In: Proceedings of the 13th European Conference on Information Systems, Information Systems in a Rapidly Changing Economy, ECIS 2005 (2005)Google Scholar
  28. [Langville and Meyer2006]
    Langville, A.N., Meyer, C.D.: Google’s PageRank and Beyond: The Science of Search Engine Rankings. Princeton University Press, Princeton (2006)zbMATHGoogle Scholar
  29. [Lewandowski and Höchstötter2007]
    Lewandowski, D., Höchstötter, N.: Web searching: A quality measurement perspective. In: Spink, A., Zimmer, M. (eds.) Web Searching: Interdisciplinary Perspectives. Springer, Dordrecht (2007)Google Scholar
  30. [Liu and Mihalcea2007]
    Liu, H., Mihalcea, R.: Of men, women, and computers: Data-driven gender modeling for improved user interfaces. In: Proceedings of the International Conference on Weblogs Social Media (ICWSM), pp. 121–128 (2007)Google Scholar
  31. [Marchant2009]
    Marchant, T.: An axiomatic characterization of the ranking based on the h-index and some other bibliometric rankings of authors. Scientometrics (2009), (retrieved 2009-06-15)
  32. [Mozilla Labs2008]
    Mozilla Labs, Introducing geode (2008), (retrieved 2009-06-15)
  33. [Murakami et al.2008]
    Murakami, T., Mori, K., Orihara, R.: Metrics for evaluating the serendipity of recommendation lists. In: Satoh, K., Inokuchi, A., Nagao, K., Kawamura, T. (eds.) JSAI 2007. LNCS (LNAI), vol. 4914, pp. 40–46. Springer, Heidelberg (2008)CrossRefGoogle Scholar
  34. [Nielsen2006]
    Nielsen, J.: Eyetracking research (2006), (retrieved 2009-06-15)
  35. [Pan et al.2007]
    Pan, B., Hembrooke, H., Joachims, T., Lorigo, L., Gay, G., Granka, L.: In: google we trust: Users’ decisions on rank, position, and relevance. Journal of Computer-Mediated Communication 12(3), 801–823 (2007)CrossRefGoogle Scholar
  36. [Radlinski and Joachims2005]
    Radlinski, F., Joachims, T.: Query chains: learning to rank from implicit feedback. In: Grossman, R., Bayardo, R.J., Bennett, K.P. (eds.) Proceedings of the Eleventh ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 239–248. ACM, New York (2005)CrossRefGoogle Scholar
  37. [Robertson and Zaragoza2007]
    Robertson, S., Zaragoza, H.: On rank-based effectiveness measures and optimization. Inf. Retr. 10(3), 321–339 (2007)CrossRefGoogle Scholar
  38. [Rousseau2008]
    Rousseau, R.: Woeginger’s axiomatisation of the h-index and its relation to the g-index, the h(2)-index and the R2-index. Journal of Informetrics 2(4), 335–340 (2008)CrossRefGoogle Scholar
  39. [Ruthven and Lalmas2003]
    Ruthven, I., Lalmas, M.: A survey on the use of relevance feedback for information access systems. Knowl. Eng. Rev. 18(2), 95–145 (2003)CrossRefGoogle Scholar
  40. [Saaty1980]
    Saaty, T.: The Analytic Hierarchy Process for Decisions in a Complex World. McGraw-Hill, New York (1980)Google Scholar
  41. [Schneeweiß1991]
    Schneeweiß, C.: Planung 1, Systemanalytische und entscheidungstheoretische Grundlagen. Springer, Berlin (1991)Google Scholar
  42. [Sørensen et al.2006]
    Sørensen, P., Lerche, D., Thomsen, M.: Developing decision support based on field data and partial order theory, pp. 259–283. Springer, Berlin (2006)Google Scholar
  43. [Tullis and Albert2008]
    Tullis, T., Albert, W.: Measuring the User Experience: Collecting, Analyzing, and Presenting Usability Metrics. Morgan Kaufmann, San Francisco (2008)Google Scholar
  44. [Tversky and Kahneman1981]
    Tversky, A., Kahneman, D.: The framing of decisions and the psychology of choice. Science 211, 453–458 (1981)CrossRefMathSciNetGoogle Scholar
  45. [Varian2007]
    Varian, H.R.: Intermediate Microeconomics: A Modern Approach, 7th edn. W W Norton & Co. (2007)Google Scholar
  46. [Vaughan2004]
    Vaughan, L.: New measurements for search engine evaluation proposed and tested. Information Processing & Management 40(4), 677–691 (2004)zbMATHCrossRefGoogle Scholar
  47. [Zhang and Hurley2009]
    Zhang, M., Hurley, N.: Statistical modeling of diversity in top-n recommender systems. In: To appear in Proc. of the ACM Web Intelligence Conference WI 2009 (2009)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2010

Authors and Affiliations

  • Bettina Berendt
    • 1
  • Veit Köppen
    • 2
  1. 1.Dept. of Computer ScienceK.U. LeuvenLeuvenBelgium
  2. 2.Dept. of Technical & Business Information SystemsOtto-von-Guericke-Universität MagdeburgMagdeburgGermany

Personalised recommendations