Skip to main content
SpringerLink
Log in

Improving Ranking by Respecting the Multidimensionality and Uncertainty of User Preferences

  • Chapter
Intelligent Information Access

Part of the book series: Studies in Computational Intelligence ((SCI,volume 301))

Abstract

Rankings or ratings are popular methods for structuring large information sets in search engines, e-Commerce, e-Learning, etc. But do they produce the right rankings for their users? In this paper, we give an overview of major evaluation approaches for rankings as well as major challenges facing the use and usability of rankings. We point out the importance of an interdisciplinary perspective for a truly user-centric evaluation of rankings. We then focus on two central problems: the multidimensionality of the criteria that influence both users’ and systems’ rankings, and the randomness inherent in users’ preferences. We propose multicriteria decision analysis and the integration of randomness into rankings as solution approaches to these problems. We close with an outlook on new challenges arising for ranking when systems address not only individuals, but also groups.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 84.99
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 109.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
Hardcover Book
USD 109.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. Altman, A., Tennenholtz, M.: Axiomatic foundations for ranking systems. Journal of Artificial Intelligence Research 31, 473–495 (2008), http://www.jair.org/media/2306/live-2306-3748-jair.pdf (retrieved 2009-06-15)

    MATH  MathSciNet  Google Scholar 

  2. Bamberg, G., Coenenberg, A.G.: Betriebswirtschaftliche Entscheidungslehre, 11th edn. Vahlen, Munich (2002)

    Google Scholar 

  3. Berendt, B.: Ranking – use and usability. To appear in Bulletin of the Belgian Mathematical Society – Simon Stevin (2009)

    Google Scholar 

  4. Berendt, B., Günther, O., Spiekermann, S.: Privacy in e-commerce: Stated preferences vs. actual behavior. Communications of the ACM 48(4), 101–106 (2005)

    Article  Google Scholar 

  5. Berendt, B., Kralisch, A.: A user-centric approach to identifying best deployment strategies for language tools: The impact of content and access language on web user behaviour and attitudes. Journal of Information Retrieval 12(3), 380–399 (2009)

    Article  Google Scholar 

  6. Berendt, B., Trümper, D.: Semantics-based analysis and navigation of heterogeneous text corpora: The porpoise news and blogs engine. In: Ting, I.-H., Wu, H.-J. (eds.) Web Mining Applications in E-commerce and E-services, pp. 45–64. Springer, Berlin (2009)

    Chapter  Google Scholar 

  7. Chakrabarti, S.: Mining the Web. Morgan Kaufmann, San Francisco (2003)

    Google Scholar 

  8. Dix, A., Finlay, J., Abowd, G., Beale, R.: Human Computer Interaction. Prentice Hall Europe, Englewood Cliffs (1998)

    Google Scholar 

  9. d’Ocagne, M.: Coordonnées Parallèles et Axiales: Méthode de transformation géométrique et procédé nouveau de calcul graphique déduits de la considération des coordonnées parallèlles. Gauthier-Villars, Paris (1885)

    Google Scholar 

  10. Eyetools, Eyetools research and reports: Eyetools, enquiro, and did-it uncover search’s golden triangle (2008), http://www.eyetools.com/inpage/research_google_eyetracking_heatmap.htm (retrieved 2009-06-15)

  11. Figueira, J., Greco, S., Erhgott, M. (eds.): Multiple Criteria Decision Analysis: State of the art surveys. Springer Science and Business Media, Boston (2005)

    MATH  Google Scholar 

  12. Glänzel, W.: Seven myths in bibliometrics about facts and fiction in quantitative science studies. Collnet Journal of Scientometrics and Information Management 2(1), 9–17 (2008), http://www.collnet.de/Berlin-2008/GlanzelWIS2008smb.pdf (retrieved 2009-06-15)

    Google Scholar 

  13. Hearst, M.A.: Design recommendations for hierarchical faceted search interfaces. In: SIGIR 2006 Faceted Search Workshop (2006), http://flamenco.berkeley.edu/papers/faceted-workshop06.pdf (retrieved 2009-06-15)

  14. Herlocker, J.L., Konstan, J.A., Terveen, L.G., Riedl, J.T.: Evaluating collaborative filtering recommender systems. ACM Trans. Inf. Syst. 22(1), 5–53 (2004)

    Article  Google Scholar 

  15. Hu, J., Zeng, H.-J., Li, H., Niu, C., Chen, Z.: Demographic prediction based on user’s browsing behavior. In: WWW 2007: Proceedings of the 16th international conference on World Wide Web, pp. 151–160. ACM, New York (2007)

    Chapter  Google Scholar 

  16. Inselberg, A.: The plane with parallel coordinates. Visual Computer 1(4), 69–91 (1985)

    Article  MATH  Google Scholar 

  17. International Organization for Standardization, ISO 9241-400:2007. ergonomics of human–system interaction – part 400: Principles and requirements for physical input devices (2007), http://www.iso.org/iso/iso_catalogue/catalogue_tc/catalogue_detail.htm?csnumber=38896 (retrieved 2009-06-15)

  18. International Organization for Standardization, ISO 9241-151:2008. ergonomics of human-system interaction – part 151: Guidance on world wide web user interfaces (2008), http://www.iso.org/iso/iso_catalogue/catalogue_tc/catalogue_detail.htm?csnumber=37031 (retrieved 2009-06-15)

  19. Jameson, A.: Adaptive interfaces and agents. In: Jacko, J.A., Sears, A. (eds.) Human-Computer Interaction Handbook, pp. 305–330. Erlbaum, Mahwah (2003), http://dfki.de/~jameson/abs/Jameson03Handbook.html (retrieved 2009-06-15)

    Google Scholar 

  20. Jameson, A., Smyth, B.: Recommendation to groups. In: Brusilovsky, P., Kobsa, A., Nejdl, W. (eds.) Adaptive Web 2007. LNCS, vol. 4321, pp. 596–627. Springer, Heidelberg (2007)

    Chapter  Google Scholar 

  21. Jansen, B.J., Zhang, M., Zhang, Y.: Brand awareness and the evaluation of search results. In: WWW 2007: Proceedings of the 16th international conference on World Wide Web, pp. 1139–1140. ACM, New York (2007)

    Chapter  Google Scholar 

  22. Kaplan, R.S., Norton, D.P.: The Balanced Scorecard. Translating Strategy Into Action. Harvard Business School Press (1996)

    Google Scholar 

  23. Kaplan, R.S., Norton, D.P.: Strategy Maps: Converting Intangible Assets Into Tangible Outcomes. Harvard Business School Press (2004)

    Google Scholar 

  24. Keeney, R., Raiffa, H.: Decisions with Multiple Objectives; Preferences and Value Tradeoffs. John Wiley & Sons, Chichester (1976)

    Google Scholar 

  25. Kobsa, A., Teltzrow, M.: Impacts of contextualized communication of privacy practices and personalization benefits on purchase behavior and perceived quality of recommendation. In: Beyond Personalization 2005: A Workshop on the Next Stage of Recommender Systems Research (IUI 2005), San Diego, CA, pp. 48–53 (2005)

    Google Scholar 

  26. Kralisch, A., Berendt, B.: Language-sensitive search behaviour and the role of domain knowledge. The New Review of Hypermedia and Multimedia 11(2), 221–246 (2005)

    Article  Google Scholar 

  27. Kralisch, A., Köppen, V.: The impact of language on website use and user satisfaction: Project description. In: Proceedings of the 13th European Conference on Information Systems, Information Systems in a Rapidly Changing Economy, ECIS 2005 (2005)

    Google Scholar 

  28. Langville, A.N., Meyer, C.D.: Google’s PageRank and Beyond: The Science of Search Engine Rankings. Princeton University Press, Princeton (2006)

    MATH  Google Scholar 

  29. Lewandowski, D., Höchstötter, N.: Web searching: A quality measurement perspective. In: Spink, A., Zimmer, M. (eds.) Web Searching: Interdisciplinary Perspectives. Springer, Dordrecht (2007)

    Google Scholar 

  30. Liu, H., Mihalcea, R.: Of men, women, and computers: Data-driven gender modeling for improved user interfaces. In: Proceedings of the International Conference on Weblogs Social Media (ICWSM), pp. 121–128 (2007)

    Google Scholar 

  31. Marchant, T.: An axiomatic characterization of the ranking based on the h-index and some other bibliometric rankings of authors. Scientometrics (2009), http://www.springerlink.com/content/e71h95u774701j1k (retrieved 2009-06-15)

  32. Mozilla Labs, Introducing geode (2008), http://labs.mozilla.com/2008/10/introducing-geode/ (retrieved 2009-06-15)

  33. Murakami, T., Mori, K., Orihara, R.: Metrics for evaluating the serendipity of recommendation lists. In: Satoh, K., Inokuchi, A., Nagao, K., Kawamura, T. (eds.) JSAI 2007. LNCS (LNAI), vol. 4914, pp. 40–46. Springer, Heidelberg (2008)

    Chapter  Google Scholar 

  34. Nielsen, J.: Eyetracking research (2006), http://www.useit.com/eyetracking (retrieved 2009-06-15)

  35. Pan, B., Hembrooke, H., Joachims, T., Lorigo, L., Gay, G., Granka, L.: In: google we trust: Users’ decisions on rank, position, and relevance. Journal of Computer-Mediated Communication 12(3), 801–823 (2007)

    Article  Google Scholar 

  36. Radlinski, F., Joachims, T.: Query chains: learning to rank from implicit feedback. In: Grossman, R., Bayardo, R.J., Bennett, K.P. (eds.) Proceedings of the Eleventh ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 239–248. ACM, New York (2005)

    Chapter  Google Scholar 

  37. Robertson, S., Zaragoza, H.: On rank-based effectiveness measures and optimization. Inf. Retr. 10(3), 321–339 (2007)

    Article  Google Scholar 

  38. Rousseau, R.: Woeginger’s axiomatisation of the h-index and its relation to the g-index, the h(2)-index and the R2-index. Journal of Informetrics 2(4), 335–340 (2008)

    Article  Google Scholar 

  39. Ruthven, I., Lalmas, M.: A survey on the use of relevance feedback for information access systems. Knowl. Eng. Rev. 18(2), 95–145 (2003)

    Article  Google Scholar 

  40. Saaty, T.: The Analytic Hierarchy Process for Decisions in a Complex World. McGraw-Hill, New York (1980)

    Google Scholar 

  41. Schneeweiß, C.: Planung 1, Systemanalytische und entscheidungstheoretische Grundlagen. Springer, Berlin (1991)

    Google Scholar 

  42. Sørensen, P., Lerche, D., Thomsen, M.: Developing decision support based on field data and partial order theory, pp. 259–283. Springer, Berlin (2006)

    Google Scholar 

  43. Tullis, T., Albert, W.: Measuring the User Experience: Collecting, Analyzing, and Presenting Usability Metrics. Morgan Kaufmann, San Francisco (2008)

    Google Scholar 

  44. Tversky, A., Kahneman, D.: The framing of decisions and the psychology of choice. Science 211, 453–458 (1981)

    Article  MathSciNet  Google Scholar 

  45. Varian, H.R.: Intermediate Microeconomics: A Modern Approach, 7th edn. W W Norton & Co. (2007)

    Google Scholar 

  46. Vaughan, L.: New measurements for search engine evaluation proposed and tested. Information Processing & Management 40(4), 677–691 (2004)

    Article  MATH  Google Scholar 

  47. Zhang, M., Hurley, N.: Statistical modeling of diversity in top-n recommender systems. In: To appear in Proc. of the ACM Web Intelligence Conference WI 2009 (2009)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Dept. of Computer Science, K.U. Leuven, Leuven, Belgium

    Bettina Berendt

  2. Dept. of Technical & Business Information Systems, Otto-von-Guericke-Universität Magdeburg, Magdeburg, Germany

    Veit Köppen

Authors
  1. Bettina Berendt
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Veit Köppen
    View author publications

    You can also search for this author in PubMed Google Scholar

Editor information

Editors and Affiliations

  1. Department of Electrical and Electronic Engineering, University of Cagliari Piazza d’Armi , I09123, Cagliari, Italy

    Giuliano Armano  & Eloisa Vargiu  & 

  2. Department of Informatics, University of Bari “Aldo Moro” , Via E. Orabona, 4, 70126, Bari, Italy

    Marco de Gemmis

  3. Department of Informatics, University of Bari “AldoMoro” , Via E. Orabona, 4, 70126, Bari, Italy

    Giovanni Semeraro

Rights and permissions

Reprints and permissions

Copyright information

© 2010 Springer-Verlag Berlin Heidelberg

About this chapter

Cite this chapter

Berendt, B., Köppen, V. (2010). Improving Ranking by Respecting the Multidimensionality and Uncertainty of User Preferences. In: Armano, G., de Gemmis, M., Semeraro, G., Vargiu, E. (eds) Intelligent Information Access. Studies in Computational Intelligence, vol 301. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-14000-6_3

Download citation

  • DOI: https://doi.org/10.1007/978-3-642-14000-6_3

  • Publisher Name: Springer, Berlin, Heidelberg

  • Print ISBN: 978-3-642-13999-4

  • Online ISBN: 978-3-642-14000-6

  • eBook Packages: EngineeringEngineering (R0)

Publish with us

Policies and ethics

Search

Navigation