SIGIR ’94 pp 212-220 | Cite as

Natural Language vs. Boolean Query Evaluation: A Comparison of Retrieval Performance

  • Howard Turtle


The results of experiments comparing the relative performance of natural language and Boolean query formulations are presented. The experiments show that on average a current generation natural language system provides better retrieval performance than expert searchers using a Boolean retrieval system when searching full-text legal materials. Methodological issues are reviewed and the effect of database size on query formulation strategy is discussed.


Natural Language Relevant Document Average Precision Retrieval Model Test Collection 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. [1]
    Karen Sparck Jones. Retrieval system tests 1958–1978. In Karen Sparck Jones, editor, Information Retrieval Experiment, chapter 12, pages 213–255, Butterworth, 1981.Google Scholar
  2. [2]
    Donald R. Swanson. Information retrieval as a trial and error process. Library Quarterly, 47(2):128–148, 1977.CrossRefGoogle Scholar
  3. [3]
    David C. Blair. Language and Representation in Information Retrieval. Elsevier Science Publishers, Amsterdam, 1990.Google Scholar
  4. [4]
    Laura Duncan: New research net drops computerese. Chicago Daily Law Bulletin, 2, September 16 1992.Google Scholar
  5. [5]
    IDP Report. West unveils new natural language interface. Information and Database Publishing Report, 13(35):1,6, September 18 1992.Google Scholar
  6. [6]
    Cary Griffith. WESTLAW’s WIN: not only natural, but new. Information Today, 9–11, October 1992.Google Scholar
  7. [7]
    Cary Griffith. WESTLAW’s winning ways. Law Office Computing, 31–38, February/March 1993.Google Scholar
  8. [8]
    Howard Turtle. Inference Networks for Document Retrieval. PhD thesis, Computer Science Department, University of Massachusetts, Amherst, MA 01003, 1990. Available as COINS Technical Report 90–92.Google Scholar
  9. [9]
    Howard Turtle and W. Bruce Croft. Evaluation of an inference network-based retrieval model. ACM Transactions on Information Systems, 9 (3): 187–222, July 1991.CrossRefGoogle Scholar
  10. [10]
    Edward A. Fox. Extending the Boolean and Vector Space Models of Information Retrieval with P-Norm Queries and Multiple Concept Types. PhD thesis, Cornell University, 1983.Google Scholar
  11. [11]
    Edward A. Fox and Mathew B. Koll. Practical enhanced Boolean retrieval: experiences with the SMART and SIRE systems. Information Processing and Management, 24 (3): 257–267, 1988.CrossRefGoogle Scholar
  12. [12]
    E. M. Keen. Evaluation parameters. Li Gerard Salton, editor, The SMART Retrieval System - Experiments in Automatic Document Processing, chapter 5, pages 74–111, Prentice Hall, 1971.Google Scholar
  13. [13]
    Lynn Evans. An experiment: search strategy variation in SDI profiles. In Karen Sparck Jones, editor, Information Retrieval Experiment, chapter 14, pages 285–315, Butterworth, 1981.Google Scholar
  14. [14]
    D. C. Blair and M. E. Mayon. An evaluation of retrieval effectiveness for a full-text document retrieval system. Communications of the ACM, 28 (3): 290–299, 1985.CrossRefGoogle Scholar
  15. [15]
    Karen Sparck Jones and C. J. van Rijsbergen. Report on the Need for and the Provision of an `Ideal’ Information Retrieval Test Collection. Technical Report British Library Research and Development Report 5266, Computer Laboratory, University of Cambridge, 1975.Google Scholar
  16. [16]
    Karen Sparck Jones and R. G. Bates. Report on a Design Study for the `Ideal’ Information Retrieval Test Collection. Technical Report British Library Research and Development Report 5428, Computer Laboratory, University of Cambridge, 1977.Google Scholar
  17. [17]
    Donna Harman. Overview of the first text retrieval conference (TREC-1). In Donna K. Harman, editor, The First Text Retrieval Conference (TREC-1), pages 1–20, National Institute of Standards and Technology, March 1993. Proceedings available as NIST Special Publication 500–207.Google Scholar

Copyright information

© Springer-Verlag London Limited 1994

Authors and Affiliations

  • Howard Turtle
    • 1
  1. 1.West Publishing CompanyEaganUSA

Personalised recommendations