Overview of CLEF-IP 2013 Lab

Information Retrieval in the Patent Domain
  • Florina Piroi
  • Mihai Lupu
  • Allan Hanbury
Part of the Lecture Notes in Computer Science book series (LNCS, volume 8138)


The first Clef-Ip test collection was made available in 2009 to support research in IR methods in the intellectual property domain; only one type of retrieval task (Prior Art Search) was given to the participants. Since then the test collection has been extended with both more content and varied types of tasks, reflecting various specific parts of patent experts’ workflows. In 2013 we organized two tasks – Passage Retrieval Starting from Claims and Structure Recognition – on which we report in this work.


Patent Application Patent Citation Retrieval Task Test Collection Patent Document 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    ***. Citations, (last retrieved: July 2013)
  2. 2.
    ***. International Patent Classification (IPC), (last retrieved: March 2013)
  3. 3.
    Manual of Patent Examining Procedure (MPEP), revision 2012 (2012) (last retrieved: June 2013)Google Scholar
  4. 4.
    Anderka, M., Stein, B.: The ESA Retrieval Model Revisited. In: Allan, J., Aslam, J.A., Sanderson, M., Zhai, C.X., Zobel, J. (eds.) Proceedings of SIGIR, pp. 670–671. ACM (2009)Google Scholar
  5. 5.
    Attar, R., Fraenkel, A.S.: Local Feedback in Full-text Retrieval Systems. J. ACM 24(3), 397–417 (1977)CrossRefzbMATHGoogle Scholar
  6. 6.
    Bunke, H., Shearer, K.: A Graph Distance Metric Based on the Maximal Common Subgraph. Pattern Recognition Letters 19(3-4), 255–259 (1998)CrossRefzbMATHGoogle Scholar
  7. 7.
    Giachanou, A., Salampasis, M., Satratzemi, M., Samaras, N.: Report on the CLEF-IP 2013 Experiments: Multilayer Collection Selection on Topically Organized Patents. In: CLEF (Notebook Papers/LABs/Workshops) (2013)Google Scholar
  8. 8.
    Goto, I., Lu, B., Chow, K.P., Sumita, E., Tsou, B.K.: Overview of the Patent Machine Translation Task at the NTCIR-9 Workshop. In: Proceedings of NTCIR, vol. 9, pp. 559–578 (2011)Google Scholar
  9. 9.
    Graf, E., Azzopardi, L.: A Methodology for Building a Patent Test Collection for Prior art Search. In: Proceedings of the Second International Workshop on Evaluating Information Access (EVIA) (2008)Google Scholar
  10. 10.
    Hanbury, A., Zenz, V., Berger, H.: 1st international workshop on advances in patent information retrieval (AsPIRe 2010). SIGIR Forum 44(1), 19–22 (2010)CrossRefGoogle Scholar
  11. 11.
    Iwayama, M., Fujii, A., Kando, N., Marukawa, Y.: An Empirical Study on Retrieval Models for Different Document Genres: Patents and Newspaper Articles. In: Proc. 26th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR 2003, pp. 251–258. ACM (2003)Google Scholar
  12. 12.
    Kamps, J., Pehcevski, J., Kazai, G., Lalmas, M., Robertson, S.: INEX 2007 Evaluation Measures. In: Fuhr, N., Kamps, J., Lalmas, M., Trotman, A. (eds.) INEX 2007. LNCS, vol. 4862, pp. 24–33. Springer, Heidelberg (2008)CrossRefGoogle Scholar
  13. 13.
    Kando, N., Leong, M.-K.: Workshop on Patent Retrieval (SIGIR 2000 Workshop Report). SIGIR Forum 34(1), 28–30 (2000)CrossRefGoogle Scholar
  14. 14.
    Luo, J., Yang, H.: Query Formulation for Prior Art Search - Georgetown University at CLEF-IP 2013. In: CLEF (Notebook Papers/LABs/Workshops) (2013)Google Scholar
  15. 15.
    Lupu, M., Huang, J., Zhu, J., Tait, J.: TREC-CHEM: Large Scale Chemical Information Retrieval Evaluation at TREC. SIGIR Forum 43(2) (December 2009)Google Scholar
  16. 16.
    Lupu, M., Hanbury, A.: Patent Retrieval. FnTIR. NOW Publishers (2012)Google Scholar
  17. 17.
    Lupu, M., Piroi, F., Hanbury, A.: Evaluating Flowchart Recognition for Patent Retrieval. In: Song, R., Webber, W., Kando, N., Kishida, K. (eds.) The Fifth International Workshop on Evaluating Information Access (EVIA), pp. 37–44 (2013)Google Scholar
  18. 18.
    Lupu, M., Piroi, F., Huang, X., Zhu, J., Tait, J.: Overview of the TREC 2009 Chemical IR Track. In: Voorhees, E.M., Buckland, L.P. (eds.) Proceedings of The Eighteenth Text REtrieval Conference, TREC 2009, Gaithersburg, Maryland, USA, 2009, Gaithersburg, Maryland, USA, November 17-20, Special Publication 500–278. National Institute of Standards and Technology, NIST (2009)Google Scholar
  19. 19.
    Lv, Y., Zhai, C.: When Documents are Very Long, BM25 Fails! In: Ma, W.-Y., Nie, J.-Y., Baeza-Yates, R.A., Chua, T.-S., Croft, W.B. (eds.) Proceedings of SIGIR, pp. 1103–1104. ACM (2011)Google Scholar
  20. 20.
    Magdy, W., Jones, G.J.F.: PRES: A Score Metric for Evaluating Recall-oriented Information Retrieval Applications. In: SIGIR 2010 (2010)Google Scholar
  21. 21.
    Osborn, M., Strzalkowski, T., Marinescu, M.: Evaluating Document Retrieval in Patent Database: A Preliminary Report. In: Proceedings of the Sixth International Conference on Information and Knowledge Management, CIKM 1997, pp. 216–221. ACM, New York (1997)CrossRefGoogle Scholar
  22. 22.
    PCT. Patent Cooperation Treaty (1970), (last retrieved: March 2013)
  23. 23.
    Piroi, F., Lupu, M., Hanbury, A., Zenz, V.: CLEF-IP 2011: Retrieval in the Intellectual Property Domain (September 2011)Google Scholar
  24. 24.
    Piroi, F., Lupu, M., Hanbury, A., Sexton, A.P., Magdy, W., Filippov, I.V.: Clef-IP 2012: Retrieval Experiments in the Intellectual Property Domain. In: CLEF (Online Working Notes/Labs/Workshop) (2012)Google Scholar
  25. 25.
    Piroi, F., Lupu, M., Hanbury, A.: Passage Retrieval Starting from Patent Claims. A Clef-Ip2013 Task Overview. In: CLEF (Online Working Notes/Labs/Workshop) (2013)Google Scholar
  26. 26.
    Potthast, M., Stein, B., Anderka, M.: A Wikipedia-Based Multilingual Retrieval Model. In: Macdonald, C., Ounis, I., Plachouras, V., Ruthven, I., White, R.W. (eds.) ECIR 2008. LNCS, vol. 4956, pp. 522–530. Springer, Heidelberg (2008)CrossRefGoogle Scholar
  27. 27.
    Tait, J., Harris, C., Lupu, M.: The 3rd International Workshop on Patent Information Retrieval (PaIR 2010) (October 2010)Google Scholar
  28. 28.
    Wallis, W.D., Shoubridge, P., Kraetzl, M., Ray, D.: Graph Distances Using Graph Union. Pattern Recognition Letters 22(6/7), 701–704 (2001)CrossRefzbMATHGoogle Scholar
  29. 29.
    WIPO Econimics & Statistics Series. 2013: PCT Yearly Review. WIPO Publication No. 901E/2013 (2013),

Copyright information

© Springer-Verlag Berlin Heidelberg 2013

Authors and Affiliations

  • Florina Piroi
    • 1
  • Mihai Lupu
    • 1
  • Allan Hanbury
    • 1
  1. 1.Institute of Software Technology and Interactive SystemsVienna University of TechnologyViennaAustria

Personalised recommendations