A Framework for the Evaluation of Adaptive IR Systems through Implicit Recommendation

  • Catherine Mulwa
  • Seamus Lawless
  • M. Rami Ghorab
  • Eileen O’Donnell
  • Mary Sharp
  • Vincent Wade
Part of the Lecture Notes in Computer Science book series (LNCS, volume 6828)


Personalised Information Retrieval (PIR) has gained considerable attention in recent literature. In PIR different stages of the retrieval process are adapted to the user, such as adapting the user’s query or the results. Personalised recommender frameworks are endowed with intelligent mechanisms to search for products, goods and services that users are interested in. The objective of such tools is to evaluate and filter the huge amount of information available within a specific scope to assist users in their information access processes. This paper presents a web-based adaptive framework for evaluating personalised information retrieval systems. The framework uses implicit recommendation to guide users in deciding which evaluation techniques, metrics and criteria to use. A task-based experiment was conducted to test the functionality and performance of the framework. A Review of evaluation techniques for personalised IR systems was conducted and the results of the analysed survey are presented.


Personalisation Personalised Information Retrieval Systems Implicit Recommendations User-based Evaluation Task-based Evaluation 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Agichtein, E., Brill, E., Dumais, S.: Improving Web Search Ranking by Incorporating User Behavior Information. In: 29th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 2006), ACM, Seattle (2006)Google Scholar
  2. 2.
    Barry, C.L., Schamber, L.: Users’ criteria for relevance evaluation: a cross-situational comparison. Information processing & management 34, 219–236 (1998)CrossRefGoogle Scholar
  3. 3.
    Chirita, P.-A., Firan, C., Nejdl, W.: Personalised Query Expansion for the Web. In: 30th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 2007). ACM, Amsterdam (2007)Google Scholar
  4. 4.
    Cleverdon, C.W., Mills, J., Keen, E.M.: An inquiry in testing of information retrieval systems (vols. 2) (Cranfield, UK: Aslib Cranfield Research Project, College of Aeronautics) (1966)Google Scholar
  5. 5.
    Gao, W., Niu, C., Nie, J.-Y., Zhou, D., Hu, J., Wong, K.-F., Hon, H.-W.: Cross-Lingual Query Suggestion Using Query Logs of Different Languages. In: 30th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 2007). ACM, Amsterdam (2007)Google Scholar
  6. 6.
    Ghorab, M.R., Zhou, D., O’Connor, A., And Wade, V.: Users’ Search History Approach for Personalised Information Retrieval: survey and Classification. Submitted to the Journal of User Modelling and User Adapted Interaction (2011) (Under Review)Google Scholar
  7. 7.
    Lawless, S., Mulwa, C., O’Connor, A.: A Proposal for the Evaluation of Adaptive Personalised Information Retrieval. In: Proceedings of the 2nd International Workshop on Contextual Information Access, Seeking and Retrieval Evaluation, Milton Keynes, UK. 4, March 28 (2010)Google Scholar
  8. 8.
    Koutrika, G., Ioannidis, Y.: Rule-based Query Personalised in Digital Libraries. International Journal on Digital Libraries 4, 60–63 (2004)CrossRefGoogle Scholar
  9. 9.
    Micarelli, A., Sciarrone, F.: Anatomy and Empirical Evaluation of an Adaptive Web-Based Information Filtering System. User Modeling and User-Adapted Interaction 14, 159–200 (2004)CrossRefGoogle Scholar
  10. 10.
    Pretschner, A., Gauch, S.: Ontology Based Personalised Search. In: 11th IEEE International Conference on Tools with Artificial Intelligence (ICTAI 1999). IEEE, Chicago (1999)Google Scholar
  11. 11.
    Pitkow, J., Schutze, H., Cass, T., Cooley, R., Turnbull, D., Edmonds, A., Adar, E., Breuel, T.: Personalised Search. Communications of the ACM 45, 50–55 (2002)CrossRefGoogle Scholar
  12. 12.
    Smyth, B., Balfe, E.: Anonymous Personalised in Collaborative Web Search. Information Retrieval 9, 165–190 (2006)CrossRefGoogle Scholar
  13. 13.
    Speretta, M., Gauch, S.: Misearch. In: IEEE/WIC/ACM International Conference on Web Intelligence (WI 2005), Compiegne University of Technology. IEEE Computer Society, France (2005a)Google Scholar
  14. 14.
    Stamou, S., Ntoulas, A.: Search Personalised Through Query and Page Topical Analysis. User Modeling and User-Adapted Interaction 19, 5–33 (2009)CrossRefGoogle Scholar
  15. 15.
    Stefani, A., Strapparava, C.: Exploiting NLP Techniques to Build User Model for Web Sites: the Use of WordNet in SiteIF Project. In: 2nd Workshop on Adaptive Systems and User Modeling on the World Wide Web, Toronto, Canada (1999)Google Scholar
  16. 16.
    Sugiyama, K., Hatano, K., Yoshikawa, M.: Adaptive Web Search Based on User Profile Constructed without Any Effort from Users. In: 13th International Conference on World Wide Web. ACM, New York (2004)Google Scholar
  17. 17.
    Teevan, J., Dumais, S.T., Horvitz, E.: Personalizing Search via Automated Analysis of Interests and Activities. In: 28th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 2005). ACM, Salvador (2005)Google Scholar
  18. 18.
    Tobar, C.M.: Yet another evaluation framework. In: Second Workshop on Empirical Evaluation of Adaptive Systems is part of the 9th International Conference on User Modeling (2003)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2011

Authors and Affiliations

  • Catherine Mulwa
    • 1
  • Seamus Lawless
    • 1
  • M. Rami Ghorab
    • 1
  • Eileen O’Donnell
    • 1
  • Mary Sharp
    • 1
  • Vincent Wade
    • 1
  1. 1.Knowledge and Data Engineering Research Group, School of Statistics and Computer ScienceTrinity College DublinIreland

Personalised recommendations