A Diary Study-Based Evaluation Framework for Mobile Information Retrieval

  • Ourdia Bouidghaghen
  • Lynda Tamine
  • Mohand Boughanem
Part of the Lecture Notes in Computer Science book series (LNCS, volume 6458)


In this poster, we propose an evaluation framework that investigates the integration of the user context (interests, location and time) into the evaluation process of mobile IR. Our approach is based on a diary study where users are asked to log their queries annotated by their location and time. Users’ interests are explicitly acquired or implicitly learned based on users’ relevance judgments for the retrieved documents answering their queries. We propose two evaluation protocols namely training/test in chronological order and k-fold cross validation. We exploit this framework in order to evaluate the performance of our context-based personalized mobile search approach. Experimental results show the stability performance of our approach according to the proposed evaluation protocols and demonstrate the viability of the diary approach as a means to capture context in evaluation.


Experimental evaluation evaluation framework in mobile context diary study location time user’s interests 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Brown, P.J., Jones, G.J.F.: Context-aware retrieval: Exploring a new environment for information retrieval and information filtering. Personal Ubiquitous Computing 5(4), 253–263 (2001)CrossRefGoogle Scholar
  2. 2.
    Tamine-Lechani, L., Boughanem, M., Daoud, M.: Evaluation of contextual information retrieval effectiveness: Overview of issues and research. KIS 24(1), 1–34 (2009)Google Scholar
  3. 3.
    Bouidghaghen, O., Tamine, L., Daoud, M., Laffaire, C.: Contextual evaluation of mobile search. In: Doan, B.-L., Jose, J., Melucci, M., Tamine, L. (eds.) Workshop on Contextual Information Access, Seeking and Retrieval Evaluation, Milton Keynes, CEUR Workshop Proceedings, vol. 569 (2010)Google Scholar
  4. 4.
    Mizzaro, S., Nazzi, E., Vassena, L.: Retrieval of context-aware applications on mobile devices: How to evaluate? In: Proceedings of IIix, pp. 65–71. ACM, NY (2008)Google Scholar
  5. 5.
    Göker, A., Myrhaug, H.: Evaluation of a mobile information system in context. Information Processing and Management 44(1), 39–65 (2008)CrossRefGoogle Scholar
  6. 6.
    Borlund, P., Ingwersen, P.: Measures of relative relevance and ranked half-life. In: 21st Annual International ACM SIGIR Conference on Research and Development Information Retrieval, pp. 324–331. ACM, NY (1998)Google Scholar
  7. 7.
    Mountain, D., MacFarlane, A.: Geographic information retrieval in a mobile environment: evaluating the needs of mobile individual. JIS 33(5), 515–530 (2007)Google Scholar
  8. 8.
    Rieman, J.: The diary study: a workplace-oriented research tool to guide laboratory efforts. In: INTERACT 1993 and CHI 1993 Conference on Human Factors in Computing Systems, pp. 321–326. ACM, NY (1993)Google Scholar
  9. 9.
    Sohn, T., Li, K.A., Griswold, W.G., Hollan, J.D.: A diary study of mobile information needs. In: CHI, pp. 433–442. ACM, New York (2008)Google Scholar
  10. 10.
    Voorhees, E.M.: The Philosophy of Information Retrieval Evaluation. In: Peters, C., Braschler, M., Gonzalo, J., Kluck, M. (eds.) CLEF 2001. LNCS, vol. 2406, pp. 355–370. Springer, Heidelberg (2002)CrossRefGoogle Scholar
  11. 11.
    Bouidghaghen, O., Tamine-Lechani, L., Boughanem, M.: Dynamically personalizing search results for mobile users. In: Andreasen, T., Bulskov, H. (eds.) FQAS 2009. LNCS, vol. 5822, pp. 99–110. Springer, Heidelberg (2009)CrossRefGoogle Scholar
  12. 12.
    Daoud, M., Tamine, L., Boughanem, M.: Towards a graph based user profile modeling for a session-based personalized search. KIS 21(3), 365–398 (2009)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2010

Authors and Affiliations

  • Ourdia Bouidghaghen
    • 1
  • Lynda Tamine
    • 1
  • Mohand Boughanem
    • 1
  1. 1.IRIT-University Paul SabatierToulouse cedex 09France

Personalised recommendations