Advertisement

Knowledge and Information Systems

, Volume 24, Issue 1, pp 1–34 | Cite as

Evaluation of contextual information retrieval effectiveness: overview of issues and research

  • Lynda Tamine-Lechani
  • Mohand Boughanem
  • Mariam Daoud
Survey Paper

Abstract

The increasing prominence of information arising from a wide range of sources delivered over electronic media has made traditional information retrieval systems less effective. Indeed, users are overwhelmed by the information delivered by such systems in response to their queries, particularly when the latter are ambiguous. In order to tackle this problem, the state-of-the-art reveals that there is a growing interest towards contextual information retrieval which relies on various sources of evidence issued from the user’s search background and environment like interests, preferences, time and location, in order to improve the retrieval accuracy. Contextual information retrieval systems are based on different definitions of the core concept of user’s context, various user’s context modeling approaches and several techniques of document relevance measurement, but all share the goal of providing the most useful information to the users in accordance with their context. However, the evaluation methodologies conceived in the past several years for traditional information retrieval and widely used in the evaluation campaigns have been challenged by the consideration of user’s context in the information retrieval process. Thus, we recognize that a critical review of existing evaluation methodologies in contextual information retrieval area is needed in order to design and develop standard evaluation frameworks. We present in this paper a comprehensive survey of contextual information retrieval evaluation methodologies and provide insights into how and why they are appropriate to measure the retrieval effectiveness. We also highlight some of the research challenges ahead that would constitute substantive research area for future research.

Keywords

Context Profile Relevance Empirical evaluation 

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. 1.
    Agichtein E, Brill E, Dumais ST (2006) Improving web search ranking by incorporating user behavior information. In: Proceedings of the 29th international SIGIR conference on research and development in information retrieval, pp 19–26Google Scholar
  2. 2.
    Allan J (2002) Challenges in information retrieval and langage modelling. In: Report of a workshop held at the Center for Intelligent Information Retrieval, University of Massachusetts, AmherstGoogle Scholar
  3. 3.
    Allan J (2003) Hard track overview in trec 2003 high accuracy retrieval from documents. In: Proceedings of the 12th text retrieval conference (TREC-12), National Institute of Standards and Technology, NIST special publication, pp 24–37Google Scholar
  4. 4.
    Anand SS, Mobasher B (2007) Introduction to intelligent techniques for web personalization. ACM Trans Internet Technol 7(4): 18CrossRefGoogle Scholar
  5. 5.
    Anderson C, Domingos P, Weld D (2001) Personalizing web sites for mobile users. In: Anderson CR, Domingos P, Weld DS (eds) Personalizing web sites for mobile users. Proceedings of the 10th international WWW conference, pp 565–575Google Scholar
  6. 6.
    Baeza-Yates R, Ribeiro-Neto B (1999) Modern information retrieval. ACM Press, Addison-WesleyGoogle Scholar
  7. 7.
    Beard K, Sharma V (1997) Multidimensional ranking for data in digital spatial libraries. Int J Digit Libr 1(2): 153–160CrossRefGoogle Scholar
  8. 8.
    Belew RK, Hatton J (1996) Rave reviews: acquiring relevance assessments from multiple users. Working notes of the AAAI Spring symposium on Machine learning in information access, Standford, CA, USAGoogle Scholar
  9. 9.
    Bierig R, Göker A (2006) Time, location and interest: an empirical and user-centred study. In: Proceedings of the 1st international conference on information interaction in context (IIiX). ACM, New York, pp 79–87Google Scholar
  10. 10.
    Bilal D (2000) Children’s use of the yahooligans! web search engine: cognitive, physical, and affective behaviors on fact-based search tasks. J Am Soc Inf Sci 51(7): 646–665CrossRefGoogle Scholar
  11. 11.
    Borlund P (2003a) The concept of relevance in IR. J Am Soc Inf Sci Technol 54(10): 913–925CrossRefGoogle Scholar
  12. 12.
    Borlund P (2003b) The IIR evaluation model: a framework for evaluation of interactive information retrieval systems. J Inf Res 8(3): 152Google Scholar
  13. 13.
    Borlund P, Ingwersen P (1998) Measures of relative relevance and ranked half-life: performance indicators in interactive IR. In: Croft WB et al (eds) Proceedings of the 21st ACM SIGIR international conference on research and development, pp 324–331Google Scholar
  14. 14.
    Bouvin NO, Christensen B, Gronbaeck K, Hansen FA (2003) Hycon: a framework for context-aware mobile hypermedia. Hypermedia 9(1): 59–88CrossRefGoogle Scholar
  15. 15.
    Buckley C, Voorhees EM (2004) Retrieval evaluation with incomplete information. In: Proceedings of the 27th annual international ACM SIGIR conference on research and development in information retrieval. ACM, New York, pp 25–32Google Scholar
  16. 16.
    Budzik J, Hammond K (2000) User interactions with every day applications as context for just-in-time information access. In: Proceedings of the 5th international conference on intelligent user interfaces, pp 44–51Google Scholar
  17. 17.
    Bylund M, Espinoza F (2002) Testing and demonstrating context-aware services with quake iii arena. Commun ACM 45(1): 46–48CrossRefGoogle Scholar
  18. 18.
    Bystrom K, Jarvelin K (1995) Task complexity affects information seeking and use. Inf Process Manag 31(2): 191–213CrossRefGoogle Scholar
  19. 19.
    Chandrasekaran P, Joshi A (2002) Mobileiq: a framework for mobile information access. In: Proceedings of the 3rd international conference on mobile data management. IEEE Computer Society, Washington, DC, USA, p 43Google Scholar
  20. 20.
    Cheverst K, Davies N, Mitchell K, Friday A, Efstratiou C (2000) Developing a context-aware electronic tourist guide: some issues and experiences. In: Proceedings of the SIGCHI conference on human factors in computing systems. ACM, New York, pp 17–24Google Scholar
  21. 21.
    Chin DN (2001) Empirical evaluation of user models and user-adapted systems. User Model User-Adapted Interact 11(1-2): 181–194zbMATHCrossRefGoogle Scholar
  22. 22.
    Chittaro, L. (eds) (2003) Human–computer interaction with mobile devices and services. Lecture Notes in Computer Science, vol 2795. Springer, BerlinGoogle Scholar
  23. 23.
    Clarcke C, Craswell N, Soboroff I (2004) Overview of the TREC 2004 terabyte track. In: The text retrieval conference. NISTGoogle Scholar
  24. 24.
    Cleverdon C (1967) The cranfield test on index language devices. In: Aslib, pp 173–194Google Scholar
  25. 25.
    Crestani F, Ruthven I (2007) Introduction to special issue on contextual information retrieval systems. Inf Retr 10: 829–847zbMATHCrossRefGoogle Scholar
  26. 26.
    Daoud M, Tamine-Lechani L, Boughanem M (2008) Learning user interests for a session-based personalized search. In: Proceedings of the 2nd international IIiX symposium (IIiX’08), London, UK, pp 57–64Google Scholar
  27. 27.
    Davies N, Mitchell K, Cheverest K, Blair G (1998) Developing a context sensitive tourist guide. In: First workshop on human computer interaction with mobile devices, GIST Technical Report G98-1Google Scholar
  28. 28.
    Dervin B, Nilan M (1986) Information needs and uses. In: William ME (ed) ARIST, pp 3–33Google Scholar
  29. 29.
    Dey AK (2001) Understanding and using context. Pers Ubiquitous Comput 5(1): 4–7CrossRefGoogle Scholar
  30. 30.
    Diaz A, Garcia A, Gervas P (2008) User-centred versus system-centred evaluation of a personalization system. Inf Process Manag 44(3): 1293–1307CrossRefGoogle Scholar
  31. 31.
    Ding C, Patra J (2007) User modeling for personalized web search with self-organizing map. J Am Soc Inf Sci Technol 58(4): 494–507CrossRefGoogle Scholar
  32. 32.
    Dumais S, Cuttrell E, Cadiz J, Jancke G, Sarin R, Robbins D (2003) Stuff i’ve seen: a system for a personal information retrieval and re-use. In: Proceedings of the 26th ACM SIGIR’, Toronto, pp 72–79Google Scholar
  33. 33.
    Ellis D (1996) The dilemma of measurement in information retrieval research. J Am Soc Inf Sci 47(1): 23–36CrossRefGoogle Scholar
  34. 34.
    Frias-Martinez E, Chen SY, Macredie RD, Liu X (2007) The role of human factors in stereotyping behavior and perception of digital library users: a robust clustering approach. User Model User-Adapted Interact 17(3): 1391–1573CrossRefGoogle Scholar
  35. 35.
    Göker A, Myrhaug H (2008) Evaluation of a mobile information system in context. Inf Process Manag 44(1): 39–65CrossRefGoogle Scholar
  36. 36.
    Göker A, Myrhaug HI (2002) User context and personalisation. In: ECCBR workshop on case based reasoning and personalisation, AberdeenGoogle Scholar
  37. 37.
    Göker A, Watt S, Myrhaug H, Whitehead N, Yakici M, Bierig R, Kanth S, Cumming H (2004) An ambient, personalised, and context-sensitive information system for mobile users. In: Proceedings of the 2nd European union symposium on ambient intelligence (EUSAI), ACM, New York, NY, USA, pp 19–24Google Scholar
  38. 38.
    Gowan JM (2003) A multiple model approach to personalised information access. Master Thesis in computer science, Faculty of science, University College DublinGoogle Scholar
  39. 39.
    Gwizdka J, Chignell M (1999) Towards information retrieval measures for evaluation of web search engines. Unpublished manuscript (1999). Available at http://www.imedia.mie.utoronto.ca/jacekg/pubs.html
  40. 40.
    Harman D (1995) Overview of the 4th text retrieval conference (TREC-4). In: Proceedings of the 4th text retrieval conference (TREC-4). National Institute of Standards and Technology, NIST special publication, pp 1–24Google Scholar
  41. 41.
    Harter S, Hert C (1997) Evaluation of information retrieval systems: approaches, issues, and methods. Ann Rev Inf Sci Technol 32(1): 3–94Google Scholar
  42. 42.
    Hattori S, Tezuka T, Tanaka K (2007) Context-aware query refinement for mobile web search. In: Proceedings of the 2007 international symposium on applications and the internet workshops, IEEE Computer Society, Washington, DC, USA, p 15Google Scholar
  43. 43.
    Haveliwala T (2002) Topic-sensitive page rank. In: International ACM world wide web conference, pp 727–736Google Scholar
  44. 44.
    Held A, Buchholz S, Schill A (2002) (n.d.) Modeling of context information for pervasive computing applications. In: Proceedings of the 6th world multiconference on systemics, cybernetics and informatics (SCI2002)Google Scholar
  45. 45.
    Herlocker JL, Konstan JA, Terveen LG, Riedl JT (2004) Evaluating collaborative filtering recommender systems. ACM Trans Inf Syst 22(1): 5–53CrossRefGoogle Scholar
  46. 46.
    Hersh WR, Elliot DL, Hickam DH, Wolf SL, Molnar A (1995) Towards new measures of information retrieval evaluation. In: Proceedings of the 18th annual international ACM SIGIR conference on research and development in information retrieval, ACM, New York, NY, USA, pp 164–170Google Scholar
  47. 47.
    Hupfer ME, Detlor B (2006) Gender and web information seeking: a self-concept orientation model. J Am Soc Inf Sci Technol 57(8): 1105–1115CrossRefGoogle Scholar
  48. 48.
    Ingwersen P (1994) Polyrepresentation of information needs and semantic entities: elements of a cognitive theory for information retrieval interaction. In: Proceedings of the 17th annual international ACM SIGIR conference on research and development in information retrieval. Springer, New York, pp 101–110Google Scholar
  49. 49.
    Ingwersen P (1996) Cognitive perspectives of information retrieval interaction: Elements of a cognitive theory. J Doc 52(1): 3–50CrossRefGoogle Scholar
  50. 50.
    Ingwersen P, Jarvelin K (2005) The turn: integration of information seeking and information retrieval in context. Springer, BerlinGoogle Scholar
  51. 51.
    Iqbal R, Sturm J, Kulyk O, Wang J, Terken J (2005) User-centred design and evaluation of ubiquitous services. In: Proceedings of the 23rd annual international conference on design of communication (SIGDOC ’05), ACM, New York, NY, USA, pp 138–145Google Scholar
  52. 52.
    Jansen BJ, Booth DL, Spink A (2007a) Determining the user intent of web search engine queries. In: Proceedings of the 16th international conference on World Wide Web, ACM, New York, NY, USA, pp 1149–1150Google Scholar
  53. 53.
    Jansen BJ, Booth DL, Spink A (2007b) Determining the user intent of web search engine queries. In: WWW ’07: proceedings of the 16th international conference on World Wide Web, ACM, New York, NY, USA, pp 1149–1150Google Scholar
  54. 54.
    Jansen BJ, Booth DL, Spink A (2008) Determining the informational, navigational, and transactional intent of web queries. Inf Process Manag 44(3): 1251–1266CrossRefGoogle Scholar
  55. 55.
    Jarvelin K, Kekalainen J (2000) Ir Evaluation methods for highly relevant documents. In: Proceedings of the 23rd annual international ACM SIGIR conference on research and development in information retrieval, Belkin and al, pp 41–48Google Scholar
  56. 56.
    Jarvelin K, Kekalainen J (2002) Cumulative gain-based evaluation of IR techniques. ACM Trans Inf Syst 20(4): 422–446CrossRefGoogle Scholar
  57. 57.
    Joachims T, Granka L, Hembrooke H, Radlinski F, Gay G (2007) Evaluating the accuracy of implicit feedback from clicks and query reformulations in web search. ACM Trans Inf Syst 25(2): 7CrossRefGoogle Scholar
  58. 58.
    John R, Mooney G (2001) Fuzzy user modelling for information retrieval on the world wide web. Knowl Inf Syst 3(1): 81–95zbMATHCrossRefGoogle Scholar
  59. 59.
    Kekalainen J, Jarvelin K (2004) Evaluating information retrieval systems under the challenges of interaction and multidimensional dynamic relevance. In: Ingwersen P, Vakkari P (eds) Proceedings of the 4th CoLIS conference, pp 253–270Google Scholar
  60. 60.
    Kelly D, Fu X (2007) Eliciting better information need descriptions from users of information search systems. Inf Process Manag 43(1): 30–46CrossRefGoogle Scholar
  61. 61.
    Kepler J (2005) Context-awarness in mobile tourism guides-a comprehensive survey. Technical Report, University. Linz, IFS/TKGoogle Scholar
  62. 62.
    Kim K (2008) Effects of emotion control and task on web searching behavior. Inf Process Manag 44(1): 373–385CrossRefGoogle Scholar
  63. 63.
    Kim K, Allen B (2002) Cognitive and task influences on web searching behavior. J Am Soc Inf Sci Technol 53(2): 109–119CrossRefGoogle Scholar
  64. 64.
    Kjeldskov J, Graham C (2003) A review of mobile hci research methods. In: Proceedings mobile HCI, Lecture notes in computer science, pp 317–335Google Scholar
  65. 65.
    Kraft R, Maghoul F, Chang C (2005) Y!q: contextual search at the point of inspiration. In: CIKM ’05: proceedings of the 14th ACM international conference on information and knowledge management, ACM Press, New York, NY, USA, pp 816–823Google Scholar
  66. 66.
    Lang K (1995) NewsWeeder: learning to filter netnews. In: Proceedings of the 12th international conference on machine learning, Morgan Kaufmann, San Mateo, pp 331–339Google Scholar
  67. 67.
    Law E, Klobucar T, Pipan M (2006) User effect in evaluating personalized information retrieval systems. In: EC-TEL, Springer, Berlin, pp 257–271Google Scholar
  68. 68.
    Lee J, Hu X, Downie J (2005) Qa websites: rich research resources for contextualizing information retrieval behaviors. In: Proceedings of the 28th international SIGIR conference on research and development in information retrieval, Workshop on information retrieval in context, pp 33–366Google Scholar
  69. 69.
    Leung CW, Chan SC, Chung F (2006) A collaborative filtering framework based on fuzzy association rules and multiple-level similarity. Knowl Inf Syst 10(3): 357–381CrossRefGoogle Scholar
  70. 70.
    Liu F, Yu C (2004) Personalized web search for improving retrieval effectiveness. IEEE Trans Knowl Data Eng 16(1): 28–40CrossRefGoogle Scholar
  71. 71.
    Marchionini G (1995) Information seeking in electronics environments. Cambridge university press, Cambridge Series on Human-Computer InteractionGoogle Scholar
  72. 72.
    Micarelli A, Sicarrone F (2004) Anatomy and empirical evaluation of an daptive web-based information filtering system. User Model User-Adapted Interact 14(2-3): 159–200CrossRefGoogle Scholar
  73. 73.
    Mitchell T, Chen S, Macredie R (2005) Hypermedia learning and prior knowledge: domain expertise vs. system expertise. J Comput Assist Learn 21(12): 53–64CrossRefGoogle Scholar
  74. 74.
    Mizzaro S. (1998) How many relevances in information retrieval?. Interact Comput 10(3): 303–320CrossRefGoogle Scholar
  75. 75.
    Mobasher B (2007) Data mining for Web personalization. In: Brusilovsky P, Kobsa A, Nejdl W (eds) Lecture notes in computer science. Springer, BerlinGoogle Scholar
  76. 76.
    Mostafa J, Mukhopadhyay S, Palakal M (2003) Simulation studies of different dimensions of users’ interests and their impact on user modeling and information filtering. Inf Retr 6(2): 199–223CrossRefGoogle Scholar
  77. 77.
    Navarro-Prieto R, Scaife M, Rogers Y (2006) Cognitive strategies in web searching. In: 5th conference on human factors & the webGoogle Scholar
  78. 78.
    Petrelli D (2008) On the role of user-centred evaluation in the advancement of interactive information retrieval. Inf Process Manag 44(1): 22–38CrossRefGoogle Scholar
  79. 79.
    Ramesh V, Glass L, Vessey I (2004) Research in computer science: an empirical study. J Syst Softw 70(1-2): 165–176CrossRefGoogle Scholar
  80. 80.
    Rigaux P (2002) Spatial database: with applications to GIS. Morgan Kauffmann, San FranciscoGoogle Scholar
  81. 81.
    Robertson S (2002) Comparing the performance of adaptive filtering and ranked output systems. Inf Retr 5(2-3): 257–268CrossRefGoogle Scholar
  82. 82.
    Robertson SE (1997) The probability ranking principle in IR. Readings in information retrieval, pp 281–286Google Scholar
  83. 83.
    Robertson SE, Hancock-Beaulieu MM (1992) On the evaluation of IR systems. Inf Process Manag 28(4): 457–466CrossRefGoogle Scholar
  84. 84.
    Robertson S, Jones KS (1976) Relevance weighting for search terms. J Am Soc Inf Sci 27(3): 129–146CrossRefGoogle Scholar
  85. 85.
    Rocchio J (1971) Relevance feedback in information retrieval. In: Salton G (eds) The SMART retrieval system—experiments in automatic document processing, Prentice Hall, Englewood CliffsGoogle Scholar
  86. 86.
    Ryan N, Pascoe J, Morse D (1997) Enhanced reality fieldwork: the context-aware archaeological assistant. In: Gaffney V, van Leusen M, Exxon S (eds) Computer Applications in ArcheologyGoogle Scholar
  87. 87.
    Salton G (1971) The SMART information retrieval system. Prentice-Hall, Englewood CliffsGoogle Scholar
  88. 88.
    Schilit B, Adams N, Want R (1994) Context-aware computing applications. In: Proceedings of the workshop on mobile computing systems and applications, IEEE Computer Society, Santa Cruz, CA, pp 85–90Google Scholar
  89. 89.
    Schilit BN, LaMarca A, Borriello G, Griswold WG, McDonald D, Lazowska E, Balachandran A, Hong J, Iverson V (2003) Ubiquitous location-aware computing and the place lab initiative challenge. In: WMASHE’03, The first ACM international workshop on wireless mobile applications and services on WLAN (WMASH 2003), San Diego, CA, September 19, 2003, ACM, New York, NY, USAGoogle Scholar
  90. 90.
    Shamber L (1994) Relevance and information behaviour. In: William ME (ed) ARIST, pp 3–48Google Scholar
  91. 91.
    Shen X, Tan B, Zhai C (2005a) Context-sensitive information retrieval using implicit feedback. In: Proceedings of the 28th annual international ACM SIGIR conference on research and development in information retrieval, ACM, New York, NY, USA, pp 43–50Google Scholar
  92. 92.
    Shen X, Tan B, Zhai C (2005b) Implicit user modeling for personalized search. In: Proceedings of the 14th ACM international conference on Information and knowledge management, ACM, New York, NY, USA, pp 824–831Google Scholar
  93. 93.
    Sieg A, Mobasher B, Burke R (2004) User’s information context: integrating user profiles and concept hierarchies. In: Proceedings of the 2004 meeting of the International Federation of Classification SocietiesGoogle Scholar
  94. 94.
    Sieg A, Mobasher B, Burke R (2007) Web search personalization with ontological user profiles. In: Proceedings of the 16th ACM conference on conference on information and knowledge management, ACM, New York, NY, USA, pp 525–534Google Scholar
  95. 95.
    Smyth B, Balfe E (2006) Anonymous personalization in collaborative web search. Inf Retr 9(2): 165–190CrossRefGoogle Scholar
  96. 96.
    Solomon P (1988) Children’s information retrieval behavior: a case analysis of opac. J Am Soc Inf Sci 44(5): 245CrossRefGoogle Scholar
  97. 97.
    Sonnenwald D, Pejtersen A (1994) Towards a framework to support information needs in design: a concurrent engineering example. Knowledge organization and mangement, pp 161–172Google Scholar
  98. 98.
    Sormunen E (2002) Liberal relevance criteria of TREC-: counting on negligible documents? In: Proceedings of the 25th annual international ACM SIGIR conference on Research and development in information retrieval, ACM, New York, NY, USA, pp 324–330Google Scholar
  99. 99.
    Sparck-Jones K, van Rijsbergen C (1976) Information retrieval test collections. J Doc 32(1): 59–72CrossRefGoogle Scholar
  100. 100.
    Speretta M, Gauch S (2005) Personalized search based on user search histories. In: Proceedings of the IEEE/WIC/ACM international conference on web intelligence, pp 622–628Google Scholar
  101. 101.
    Su LT (1992) Evaluation measures for interactive information retrieval. Inf Process Manag 28(4): 503–516CrossRefGoogle Scholar
  102. 102.
    Tamine L, Boughanem M (2001) Applying heuristics to improve a genetic query optimisation process in information retrieval. In: 23 European colloquium on information retrieval, ECIR’2001, Darmstadt, Germany, 04/04/01-06/04/01, GMD-IPSI, pp 15–23Google Scholar
  103. 103.
    Tamine L, Boughanem M, Zemirli WN (2008) Personalized document ranking: exploiting evidence from multiple user interests for profiling and retrieval. J Digit Inf Manag (in press)Google Scholar
  104. 104.
    Tamine L, Chrisment C, Boughanem M (2003) Multiple query evaluation based on an enchanced genetic algorithm IP&M. Inf Process Manag 39(2):215–231CrossRefGoogle Scholar
  105. 105.
    Tan A, Ong H, Pan H, Ng J, Li Q (2004) Towards personalised web intelligence. Knowl Inf Syst 6(5): 595–616CrossRefGoogle Scholar
  106. 106.
    Tan B, Shen X, Zhai C (2006) Mining long-term search history to improve search accuracy. In: Proceedings of the 12th ACM SIGKDD international conference on Knowledge discovery and data mining, ACM, New York, NY, USA, pp 718–723Google Scholar
  107. 107.
    Tao Y, Mamoulis N, Papadias D (2003) Validity information retrieval for spatio-temporal queries. In: SSTD, 2003Google Scholar
  108. 108.
    Teevan J, Dumais S (2005) Personalizing search via automated analysis of interests and activities. In: Proceedings of the 28th international SIGIR conference on research and development in information retrieval, pp 449–456Google Scholar
  109. 109.
    Tombros A, Ruthven I, Jose JM (2005) How users assess web pages for information seeking. J Am Soc Inf Sci Technol 56(4): 327–344CrossRefGoogle Scholar
  110. 110.
    Trajkova J, Gauch S (2004) Improving ontology-based user profiles. In: Proceedings of RIAO 2004, Vaucluse, FranceGoogle Scholar
  111. 111.
    Turpin H, Hersh W (2001) Why batch and user evaluations do not give the same results. In: ACM conference on research and development in information retrieval, pp 225–231Google Scholar
  112. 112.
    Uzuner O, Katz B, Yuret D (1999) Word sense disambiguation for information retrieval. In: AAAI ’99/IAAI ’99: proceedings of the sixteenth national conference on artificial intelligence and the eleventh innovative applications of artificial intelligence conference innovative applications of artificial intelligence, American Association for Artificial Intelligence, Menlo Park, CA, USA, p 985Google Scholar
  113. 113.
    Vakkari P, Sormunen E (2004) The influence of relevance levels on the effectiveness of interactive information retrieval. J Am Soc Inf Sci Technol 55(11): 963–969CrossRefGoogle Scholar
  114. 114.
    Vieira V, Tedesco P, Salgado AC, Brézillon P (2007) Investigating the specifics of contextual elements management: the cemantika approach. In: CONTEXT, pp 493–506Google Scholar
  115. 115.
    Voorhees E (2000) Variations in relevance judgments and the measurement of retrieval effectiveness. Inf Process Manag 36(1): 697–716CrossRefGoogle Scholar
  116. 116.
    Webb G, Pazzani M, Billsus D (2001) Machine learning for user modeling. User Model User-Adapted Interact 11(1-2): 19–29zbMATHCrossRefGoogle Scholar
  117. 117.
    White R, Ruthven I, Jose J, van Rijsbergen C (2005) Evaluating implicit feedback models using searcher simulations. ACM Trans Inf Syst 23(3): 325–361CrossRefGoogle Scholar
  118. 118.
    Woodruff A, Plaunt C (1994) GIPSY: automated geographic indexing of text documents. J Am Soc Inf Sci 45(9): 645–655CrossRefGoogle Scholar
  119. 119.
    Wu X, Kumar V, Ross Quinlan J, Ghosh J, Yang Q, Motoda H, McLachlan GJ, Ng A, Liu B, Yu PS, Zhou Z-H, Steinbach M, Hand DJ, Steinberg D (2007) Top 10 algorithms in data mining. Knowl Inf Syst 14(1): 1–37CrossRefGoogle Scholar
  120. 120.
    Xie HI (2008) ‘Users’ evaluation of digital libraries (dls): their uses, their criteria, and their assessment. Inf Process Manag 44(3): 1346–1373CrossRefGoogle Scholar
  121. 121.
    Yau S, Huan L, Huang D, Yao Y (2003) Situation-aware personalized information retrieval for mobile internet. In: Proceedings of the 27th annual international computer software and applications conference (COMPSAC)’Google Scholar
  122. 122.
    Yilmaz E, Aslam JA (2008) Estimating average precision when judgments are incomplete. Knowl Inf Syst 16(2): 173–211CrossRefGoogle Scholar
  123. 123.
    Zhou Y, Croft WB (2008) Measuring ranked list robustness for query performance prediction. Knowl Inf Syst 16(2): 155–171CrossRefGoogle Scholar
  124. 124.
    Zobel J (1998) How reliable are the results of large-scale information retrieval experiments?. In: In ACM conference on research and development in information retrieval, pp 307–314Google Scholar

Copyright information

© Springer-Verlag London Limited 2009

Authors and Affiliations

  • Lynda Tamine-Lechani
    • 1
  • Mohand Boughanem
    • 1
  • Mariam Daoud
    • 1
  1. 1.University of Paul Sabatier IRIT-SIGToulouse Cedex 9France

Personalised recommendations