Abstract
We present a user-centred, task-oriented, comparative evaluation of two within-document retrieval tools. ProfileSkim computes a relevance profile for a document with respect to a query, and presents the profile as an interactive bar graph. FindSkim provides similar functionality to the web browser “Find” command. A novel simulated work task was devised, where participants are asked to identify (index) relevant pages of an electronic book, given topics from the existing book index. The original book index provides the ground truth, against which the indexing results of the participants can be compared. We confirmed a major hypothesis, namely ProfileSkim proved significantly more efficient than Find-Skim, as measured by time for task. The study indicates that ProfileSkim was as least as effective as FindSkim in identifying relevant pages, as measured by traditional information retrieval measures, and there is some evidence that ProfileSkim is a precision-enhancing tool. Based on qualitative data from questionnaires, we also provide strong evidence to support our conjecture that the participants would be more satisfied when using ProfileSkim than FindSkim. The experimental study confirmed the potential of relevance profiling for improving within-document retrieval. Relevance profiling should prove highly beneficial for users trying to identify relevant information within long documents.
Article PDF
Similar content being viewed by others
Avoid common mistakes on your manuscript.
References
Beaulieu M, Robertson SE and Rasmussen E (1996) Evaluating interactive systems in TREC. Journal of the American Society for Information Science, 47(1):85–94.
Borlund P and Ingwersen P (1997) The development of a method for the evaluation of interactive information retrieval systems. Journal of Documentation, 53(3):225–250.
procs/ (visited May 15 2003).
Green TRG (1991) Describing information artefacts with cognitive dimensions and structure maps. In: Diaper D and Hammond NV, Eds., Proceedings of the HCI'91 Conference on People and Computers VI. Cambridge University Press, pp. 297-316.
Harman D (1992) Overview of the First Text REtrieval Conference (TREC-1), National Institute of Standards and Technology, Gaithersburg, Maryland, pp. 309–318.
Harper DJ, Coulthard S and Sun Y (2002) A language modelling approach to relevance profiling for document browsing. In: Proceedings of the Joint Conference on Digital Libraries, Oregon, USA, pp. 76-83.
Harper DJ, Koychev I and Sun Y (2003) Query-based document skimming: A user-centred evaluation of relevance profiling. In: Proceedings of 25th European Conference on Information Retrieval. Lecture Notes in Computer Science, Springer-Verlag, Berlin, pp. 377–392.
Hearst MA (1995) TileBars: Visualization of term distribution information in full text information access. In: Proceedings of the ACM SIGCHI Conference on Human Factors in Computing Systems (CHI), Denver, CO, pp. 65-71.
Hersh W and Over P (2001) TREC-9 interactive track report. In: Proceedings of the Ninth Text Retrieval Conference (TREC-9), Gaithersburg, MD: NIST, pp. 42-50.
Hersh W, Pentecost J and Hickam D (1996) A task-oriented approach to information retrieval evaluation. Journal of the American Society for Information Science, 47(1):50–56.
Jansen B, Spink A and Saracevic T (2000) Real life, real users, and real needs: A study and analysis of user queries on the web. Information Processing and Management, 36:207–227.
Jose J, Furner J and Harper DJ (1998) Spatial querying for image retrieval: A user-oriented evaluation. In: Proceedings of the 21st International ACM-SIGIR Conference on Research and Development in Information Retrieval, ACM Press, pp. 232-240.
Kaszkiel M and Zobel J (1997) Passage retrieval revisited. In: Proceedings of the Twentieth International ACM-SIGIR Conference on Research and Development in Information Retrieval, Philadelphia, ACM Press, pp. 178-185.
Pirolli P and Card SK (1999) Information foraging. Psychological Review, 106:643–675.
Ponte J and Croft WB (1998) A language modeling approach to information retrieval. In: Proceedings of the ACM SIGIR Conference on Research and Development in Information Retrieval. ACM Press, pp. 275-281.
Song F and Croft WB (1999) A general language model for information retrieval. In: Proceedings of the ACM SIGIR Conference on Research and Development in Information Retrieval, ACM Press, pp. 279-280.
TREC (2002) Interactive track home page. Available at http://www-nlpir.nist.gov/projects/t11i/ (visited May 15 2003).
Author information
Authors and Affiliations
Rights and permissions
About this article
Cite this article
Harper, D.J., Koychev, I., Sun, Y. et al. Within-Document Retrieval: A User-Centred Evaluation of Relevance Profiling. Information Retrieval 7, 265–290 (2004). https://doi.org/10.1023/B:INRT.0000011207.45988.bb
Issue Date:
DOI: https://doi.org/10.1023/B:INRT.0000011207.45988.bb