Query-Based Document Skimming: A User-Centred Evaluation of Relevance Profiling

  • David J. Harper
  • Ivan Koychev
  • Yixing Sun
Conference paper

DOI: 10.1007/3-540-36618-0_27

Part of the Lecture Notes in Computer Science book series (LNCS, volume 2633)
Cite this paper as:
Harper D.J., Koychev I., Sun Y. (2003) Query-Based Document Skimming: A User-Centred Evaluation of Relevance Profiling. In: Sebastiani F. (eds) Advances in Information Retrieval. ECIR 2003. Lecture Notes in Computer Science, vol 2633. Springer, Berlin, Heidelberg

Abstract

We present a user-centred, task-oriented, comparative evaluation of two query-based document skimming tools. ProfileSkim bases within-document retrieval on computing a relevance profile for a document and query; FindSkim provides similar functionality to the web browser Find-command. A novel simulated work task was devised, where experiment participants are asked to identify (index) relevant pages of an electronic book, given subjects from the existing book index. This subject index provides the ground truth, against which the indexing results can be compared. Our major hypothesis was confirmed, namely ProfileSkim proved significantly more efficient than FindSkim, as measured by time for task. Moreover, indexing task effectiveness, measured by typical IR measures, demonstrated that ProfileSkim was better than FindSkim in identifying relevant pages, although not significantly so. The experiments confirm the potential of relevance profiling to improve query-based document skimming, which should prove highly beneficial for users trying to identify relevant information within long documents.

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

Copyright information

© Springer-Verlag Berlin Heidelberg 2003

Authors and Affiliations

  • David J. Harper
    • 1
  • Ivan Koychev
    • 1
  • Yixing Sun
    • 1
  1. 1.Smart Web Technologies Centre, School of ComputingThe Robert Gordon UniversityAberdeenUK

Personalised recommendations