A Diary Study-Based Evaluation Framework for Mobile Information Retrieval
In this poster, we propose an evaluation framework that investigates the integration of the user context (interests, location and time) into the evaluation process of mobile IR. Our approach is based on a diary study where users are asked to log their queries annotated by their location and time. Users’ interests are explicitly acquired or implicitly learned based on users’ relevance judgments for the retrieved documents answering their queries. We propose two evaluation protocols namely training/test in chronological order and k-fold cross validation. We exploit this framework in order to evaluate the performance of our context-based personalized mobile search approach. Experimental results show the stability performance of our approach according to the proposed evaluation protocols and demonstrate the viability of the diary approach as a means to capture context in evaluation.
KeywordsExperimental evaluation evaluation framework in mobile context diary study location time user’s interests
Unable to display preview. Download preview PDF.
- 2.Tamine-Lechani, L., Boughanem, M., Daoud, M.: Evaluation of contextual information retrieval effectiveness: Overview of issues and research. KIS 24(1), 1–34 (2009)Google Scholar
- 3.Bouidghaghen, O., Tamine, L., Daoud, M., Laffaire, C.: Contextual evaluation of mobile search. In: Doan, B.-L., Jose, J., Melucci, M., Tamine, L. (eds.) Workshop on Contextual Information Access, Seeking and Retrieval Evaluation, Milton Keynes, CEUR Workshop Proceedings, vol. 569 (2010)Google Scholar
- 4.Mizzaro, S., Nazzi, E., Vassena, L.: Retrieval of context-aware applications on mobile devices: How to evaluate? In: Proceedings of IIix, pp. 65–71. ACM, NY (2008)Google Scholar
- 6.Borlund, P., Ingwersen, P.: Measures of relative relevance and ranked half-life. In: 21st Annual International ACM SIGIR Conference on Research and Development Information Retrieval, pp. 324–331. ACM, NY (1998)Google Scholar
- 7.Mountain, D., MacFarlane, A.: Geographic information retrieval in a mobile environment: evaluating the needs of mobile individual. JIS 33(5), 515–530 (2007)Google Scholar
- 8.Rieman, J.: The diary study: a workplace-oriented research tool to guide laboratory efforts. In: INTERACT 1993 and CHI 1993 Conference on Human Factors in Computing Systems, pp. 321–326. ACM, NY (1993)Google Scholar
- 9.Sohn, T., Li, K.A., Griswold, W.G., Hollan, J.D.: A diary study of mobile information needs. In: CHI, pp. 433–442. ACM, New York (2008)Google Scholar
- 12.Daoud, M., Tamine, L., Boughanem, M.: Towards a graph based user profile modeling for a session-based personalized search. KIS 21(3), 365–398 (2009)Google Scholar