Evaluating Relevance Feedback Algorithms for Searching on Small Displays

  • Vishwa Vinay
  • Ingemar J. Cox
  • Natasa Milic-Frayling
  • Ken Wood
Part of the Lecture Notes in Computer Science book series (LNCS, volume 3408)


Searching online information resources using mobile devices is affected by displays on which only a small fraction of the set of ranked documents can be displayed. In this paper, we ask whether the search effort can be reduced, on average, by user feedback indicating a single most relevant document in each display. For small display sizes and limited user actions, we are able to construct a tree representing all possible outcomes. Examination of the tree permits us to compute an upper limit on relevance feedback performance. Three standard feedback algorithms are considered – Rocchio, Robertson/Sparck-Jones and a Bayesian algorithm. Two display strategies are considered, one based on maximizing the immediate information gain and the other on most likely documents. Our results bring out the strengths and weaknesses of the algorithms, and the need for exploratory display strategies with conservative feedback algorithms.


Display Size Relevance Feedback Ideal User Initial Query Bayesian Algorithm 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Buyukkokten, O., Garcia-Molina, H., Paepcke, A., Winograd, T.: Power Browser: Efficient Web Browsing for PDAs. In: The Proceedings of the ACM Conference on Computers and Human Interaction, CHI 2000 (2000)Google Scholar
  2. 2.
    Buyukkokten, O., Garcia-Molina, H., Paepcke, A.: Seeing the Whole in Parts: Text Summarization for Web Browsing on Handheld Devices. In: The Proceedings of the Tenths International World Wide Web Conference, WWW 10 (2001)Google Scholar
  3. 3.
    Campbell, I., van Rijsbergen, C.J.: The Ostensive model of developing information needs. In: Ingwersen, P., Pors, N.O. (eds.) Information Science: Integration in Perspective. Proceedings of CoLIS 2, pp. 251–268 (1996)Google Scholar
  4. 4.
    Chen, Y., Ma, W.-Y., Zhang, H.-J.: Detecting Web Page Structure for Adaptive Viewing on Small Form Factor Devices. In: The Proceedings of the Twelfth World Wide Web Conference, Budapest (May 2003) (to appear)Google Scholar
  5. 5.
    Cox, I.J., Miller, M.L., Minka, T.P., Papathomas, T.V., Yianilos, P.N.: The Bayesian Image Retrieval System, PicHunter: Theory, Implementation and Psychophysical Experiments. IEEE Transactions on Image Processing 9(1), 20–37 (2000)CrossRefGoogle Scholar
  6. 6.
    Evans, D.A., Lefferts, R.G.: CLARIT-TREC experiments. Information Processing and Management 31(3), 494–501 (1995)CrossRefGoogle Scholar
  7. 7.
    Grossman, D.A., Frieder, O., Holmes, D.O., Roberts, D.C.: Integrating Structured Data and Text: A Relational Approach. Journal of the American Society of Information Science 48(2) (February 1997)Google Scholar
  8. 8.
    Harman, D.: Relevance feedback and other query modification techniques. In: Frakes, W., Baeza-Yates, R. (eds.) Information Retrieval. Data Structures and Algorithms, pp. 131–160. Prentice Hall, Englewood Cliffs (1992)Google Scholar
  9. 9.
    Harman, D.: Relevance feedback revisited. In: Proceedings of 15th annual international ACM SIGIR conference on research and development in information retrieval, Copenhagen (1.10, 1992)Google Scholar
  10. 10.
    Jones, M., Marsden, G., Mohd-Nasir, N., Boone, K., Buchanan, G.: Improving Web Interaction on Small Displays. In: The Proceedings of the 8th World Wide Web Conference, Toronto, Canada (May 1999)Google Scholar
  11. 11.
    Jones, M., Marsden, G.: From the Large Screen to the Small Screeñ. Retaining the Designer’s Design for Effective user Interaction. IEEE Colloquium on Issues for Networked Interpersonal Communicators 239(3), 1–4 (1997)Google Scholar
  12. 12.
    Lewis, D., Ringuette, M.: A comparison of two learning algorithms for text categorization. In: Symposium on Document Analysis and Information Retrieval (1994)Google Scholar
  13. 13.
    Magennis, M., van Rijsbergen, C.J.: The potential and actual effectiveness of interactive query expansion. In: Proceedings of 20th annual international ACM SIGIR conference on research and development in information retrieval, Philadelphia (1997)Google Scholar
  14. 14.
    Milic-Frayling, N., Sommerer, R.: SmartView: Enhanced document viewer for mobile devices. Microsoft Technical Report MSR-TR-2002-114 (November 2002)Google Scholar
  15. 15.
    Over, P.: TREC-5 interactive track report. In: Proceedings of the Fifth Text REtrieval Conference, TREC-5 (1996)Google Scholar
  16. 16.
    Harman, D.K. (ed.) Overview of the Fifth Text REtrieval Conference (TREC-5), Gaithersburg, MD: NIST (1997)Google Scholar
  17. 17.
    Rocchio, J.: Relevance feedback information retrieval. In: Salton, G. (ed.) The Smart Retrieval System — Experiments in Automatic Document Processing, pp. 313–323. Prentice-Hall, Englewood Cliffs (1971)Google Scholar
  18. 18.
    Robertson, S.E., Sparck Jones, K.: Relevance weighting of search terms. Journal of the American Society for Information Science 27, 129–146 (1976)CrossRefGoogle Scholar
  19. 19.
    Robertson, S.E., et al.: Okapi at TREC-3. In: Harman, D.K. (ed.) Overview of the Third Text Retrieval Conference (TREC-3), Gaithersburg, MD: NIST (1995) (NIST Special Publication 500-225)Google Scholar
  20. 20.
    Robertson, S., Hull, D.A.: The TREC-9 Filtering Track Final Report. In: Voorhees, E.M., Harman, D.K. (eds.) NIST Special Publication 500-249: The Ninth Text Retrieval Conference (TREC-9), Gaithersburg, MD (2000)Google Scholar
  21. 21.
    Rodden, K., Milic-Frayling, N., Sommerer, R., Blackwell, A.: Effective Web Searching on Mobile Devices. In: The Proceedings of the HCI Conference, Bath (September 2003)Google Scholar
  22. 22.
    Ruvini, J.-D.: Adapting to the User’s Internet Search Strategy. In: IUI 2003, Miami, Florida, January 12-15 (2003)Google Scholar
  23. 23.
    Sellen, A.J., Murphy, R., Shaw, K.L.: How knowledge workers use the web. In: Wixon, D. (ed.) Proceedings of CHI 2002, pp. 227–234. ACM, New York (2002)Google Scholar
  24. 24.
    Sparck Jones, K., Walker, S., Robertson, S.E.: A probabilistic model of information retrieval: development and comparative experiments. Information Processing and Management 36, 779-808, 809-840 (2000)Google Scholar
  25. 25.
    Vinay, V., Cox, I.J., Milic-Frayling, N., Wood, K.: Evaluating Relevance Feedback and Display Strategies for Searching on Small Displays. In: Apostolico, A., Melucci, M. (eds.) SPIRE 2004. LNCS, vol. 3246, pp. 131–133. Springer, Heidelberg (2004)CrossRefGoogle Scholar
  26. 26.
    White, R.W., Jose, J.M., van Rijsbergen, C.J., Ruthven, I.: A Simulated Study of Implicit Feedback Models. In: McDonald, S., Tait, J.I. (eds.) ECIR 2004. LNCS, vol. 2997, pp. 311–326. Springer, Heidelberg (2004)CrossRefGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2005

Authors and Affiliations

  • Vishwa Vinay
    • 1
  • Ingemar J. Cox
    • 1
  • Natasa Milic-Frayling
    • 2
  • Ken Wood
    • 2
  1. 1.Department of Computer ScienceUniversity College LondonUK
  2. 2.Microsoft Research LtdCambridgeUK

Personalised recommendations