Evaluating Information Retrieval in the Intellectual Property Domain: The ClefIp Campaign

Abstract

The Clef–Ip track ran for the first time within the Clef 2009 campaign. The purpose of the track was twofold: (a) to encourage and facilitate research in the area of patent retrieval by providing a large clean data set for experimentation; (b) to create a large test collection of patents in the three main European languages for the evaluation of cross-lingual information access. The track focused on the task of prior art search, to which a second task was added in 2010, the patent classification task. The participating teams deployed a variety of Information Retrieval techniques, adapted or custom-made, to tackle with this specific domain and tasks. This chapter reports on activities undertaken to provide a set of topics for the two tasks, to extract the relevance assessments for the provided topics, and on evaluating the effectiveness of the employed retrieval methods.

References

  1. 1.
    Conference on Multilingual and Multimodal Information Access Evaluation (2010). http://clef2010.org/
  2. 2.
    Cross Language Evaluation Forum. http://www.clef-campaign.org
  3. 3.
  4. 4.
    Fujii A, Iwayama M, Kando N (2007) Overview of the patent retrieval task at the NTCIR-6 workshop. In: Kando N, Evans DK (eds) Proceedings of the sixth NTCIR workshop meeting on evaluation of information access technologies: information retrieval, question answering, and cross-lingual information access. National Institute of Informatics, Tokyo, pp 359–365 Google Scholar
  5. 5.
    Graf E, Azzopardi L (2008) A methodology for building a patent test collection for prior art search. In: Proceedings of the second international workshop on evaluating information access (EVIA) Google Scholar
  6. 6.
    Guidelines for Examination in the European Patent Office (2009). http://www.epo.org/patents/law/legal-texts/guidelines.html.
  7. 7.
    Järvelin K, Kekäläinen J (2002) Cumulated gain-based evaluation of IR techniques. ACM Trans Inf Syst 20(4):422–446 CrossRefGoogle Scholar
  8. 8.
    Magdy W, Jones GJF (2010) PRES: A score metric for evaluating recall-oriented information retrieval applications. In: SIGIR Google Scholar
  9. 9.
    NTCIR Project (2010) Evaluation of information access technologies research infrastructure for comparative evaluation of information retrieval and access technologies. http://research.nii.ac.jp/ntcir/index-en.html
  10. 10.
    Peters C, Di Nunzio G, Kurimo M, Mostefa D, Penas A, Roda G (eds) (2010) Multilingual information access evaluation I. Text retrieval experiments. Lecture notes in computer science, vol 6241. Springer, Berlin Google Scholar
  11. 11.
    Piroi F, Tait J (2010) CLEF–IP 2010: Retrieval experiments in the intellectual property domain. Tech Rep IRF-TR-2010-0005, Information Retrieval Facility, Vienna, Austria. URL http://www.ir-facility.org/research/publications-reports/technical-reports/files/irf-tr-2010-00005.pdf
  12. 12.
    Piroi F, Roda G, Zenz V (2009) CLEF-IP 2009 evaluation summary. Tech Rep IRF-TR-2009-00001, Information Retrieval Facility, Vienna, Austria. URL http://www.ir-facility.org/research/technical-reports/files/irf_tr_2009_00001.pdf
  13. 13.
    Roda G, Tait J, Piroi F, Zenz V (2010) CLEF-IP 2009: Retrieval experiments in the intellectual property domain. In: Peters C, Di Nunzio G, Kurimo M, Mostefa D, Penas A, Roda G (eds) Multilingual information access evaluation I. Text retrieval experiments. Lecture notes in computer science, vol 6241. Springer, Berlin, pp 385–409. doi:10.1007/978-3-642-15754-7_47 CrossRefGoogle Scholar
  14. 14.
    Suzan Verberne Eva D’hondt, NOCHK Google Scholar
  15. 15.
    Text Retrieval Conference. http://trec.nist.gov
  16. 16.
    The MAtrixware REsearch Collection (2010). http://ir-facility.net/prototypes/marec/description/overview/

Copyright information

© Springer-Verlag Berlin Heidelberg 2011

Authors and Affiliations

  1. 1.Information Retrieval FacilityViennaAustria
  2. 2.max.recall information systemsViennaAustria

Personalised recommendations