Enhancing Patent Search with Content-Based Image Retrieval

  • Stefanos Vrochidis
  • Anastasia Moumtzidou
  • Ioannis Kompatsiaris
Part of the Lecture Notes in Computer Science book series (LNCS, volume 8830)


Nowadays most of the patent search systems still rely upon text to provide retrieval functionalities. Recently, the intellectual property and information retrieval communities have shown great interest in patent image retrieval, which could augment the current practices of patent search. In this chapter, we present a patent image extraction and retrieval framework, which deals with patent image extraction and multimodal (textual and visual) metadata generation from patent images with a view to provide content-based search and concept-based retrieval functionalities. Patent image extraction builds upon page orientation detection and segmentation, while metadata extraction from images is based on the generation of low level visual and textual features. The content-based retrieval functionality is based on visual low level features, which have been devised to deal with complex black and white drawings. Extraction of concepts builds upon on a supervised machine learning framework realised with Support Vector Machines and a combination of visual and textual features. We evaluate the different retrieval parts of the framework by using a dataset from the footwear and the lithography domain.


patents images retrieval concepts classification hybrid visual 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    List, J.: How Drawings Could Enhance Retrieval in Mechanical and Device Patent Searching. World Patent Information 29, 210–218 (2007)MathSciNetCrossRefGoogle Scholar
  2. 2.
    Adams, S.: Electronic non-text material in patent applications—some questions for patent offices, applicants and searchers. World Patent Information 27(2), 99–103 (2005)CrossRefGoogle Scholar
  3. 3.
    Zeng, Z., Zhao, J., Xu, B.: An Outward-Appearance Patent-Image Retrieval Approach Based on the Contour-Description Matrix. In: Proceedings of the 2007 Japan-China Joint Workshop on Frontier of Computer Science and Technology, pp. 86–99 (2007)Google Scholar
  4. 4.
    Codina, J., Pianta, E., Vrochidis, S., Papadopoulos, S.: Integration of Semantic, Metadata and Image Search Engines with a Text Search Engine for Patent Retrieval. In: Semantic Search 2008 Workshop, Tenerife, Spain (2008)Google Scholar
  5. 5.
    Vrochidis, S., Papadopoulos, S., Moumtzidou, A., Sidiropoulos, P., Pianta, E., Kompatsiaris, I.: Towards Content-based Patent Image Retrieval; A Framework Perspective. World Patent Information Journal 32(2), 94–106 (2010)CrossRefGoogle Scholar
  6. 6.
    Tiwari, A., Bansal, V.: PATSEEK: Content Based Image Retrieval System for Patent Database. In: Proceedings of the International Conference on Electronic Business 2004, Tsinghua University, Beijing, China (2004)Google Scholar
  7. 7.
    Eakins, J.P.: Trademark Image Retrieval. In: Springer-Verlag Principles of Visual Information Retrieval. Berlin (2001)Google Scholar
  8. 8.
    Jain, A.K., Vailaya, A.: Shape-based Retrieval: A case study with trademark image databases. Pattern Recognition 31, 1369–1390 (1998)CrossRefGoogle Scholar
  9. 9.
    Kim, Y.S., Kim, W.Y.: Content-based Trademark Retrieval System Using a Visually Salient Feature. Image and Vision Computing 16, 931–939 (1998)CrossRefGoogle Scholar
  10. 10.
    Wu, J.K., Lam, C.P., Mehtre, B.M., Gao, Y.J., Desai Narasimhalu, A.: Content-based Retrieval for Trademark Registration. Multimedia Tools and Applications 3, 245–267 (1996)CrossRefGoogle Scholar
  11. 11.
    Eakins, J.P., Boardman, J.M., Graham, M.E.: Similarity Retrieval of Trademark Images. IEEE Multimedia 5, 53–63 (1998)CrossRefGoogle Scholar
  12. 12.
    Alwis, S., Austin, J.: Trademark Image Retrieval Using Multiple Features. In: Proceedings of the 1999 International Conference on Challenge of Image Retrieval (IM 1999), Newcastle-upon-Tyne, U.K. (1999)Google Scholar
  13. 13.
    Schietse, J., Eakins, J.P., Veltkamp, R.C.: Practice and Challenges in Trademark Image Retrieval. In: Proceedings of the 6th ACM International Conference on Image and Video Retrieval (CIVR), pp. 518–524 (2007)Google Scholar
  14. 14.
    LTU Technologies,
  15. 15.
  16. 16.
    Huet, B., Kern, N.J., Guarascio, G., Merialdo, B.: Relational Skeletons for Retrieval In Patent Drawings. In: ICIP 2001, vol. 2, pp. 737–740 (2001)Google Scholar
  17. 17.
    Zeng, Z., Zhao, J., Xu, B.: An Outward-Appearance Patent-Image Retrieval Approach Based on the Contour-Description Matrix. In: Proceedings of the 2007 Japan-China Joint Workshop on Frontier of Computer Science and Technology, pp. 86-89 (2007)Google Scholar
  18. 18.
    PATExpert (FP6-028116),
  19. 19.
    Sidiropoulos, P., Vrochidis, S., Kompatsiaris, I.: Content-Based Binary Image Retrieval Using the Adaptive Hierarchical Density Histogram. Pattern Recognition Journal 44(4), 739–750 (2011)CrossRefGoogle Scholar
  20. 20.
    Ypma, G.: Evaluation of Patent Image Retrieval. In: Information Retrieval Facility Symposium 2010 (IRFS 2010), Vienna, Austria (2010)Google Scholar
  21. 21.
    Yan, R., Hsu, W.: Recent Developments in Content-based and Concept-based Image/Video Retrieval. In: Proceedings of the 16th ACM International Conference on Multimedia (MM 2008), New York, USA (2008)Google Scholar
  22. 22.
    Mörzinger, R., Horti, A., Thallinger, G., Bhatti, N., Hanbury, A.: Classifying Patent Images. In: Proceedings of CLEF 2011, Amsterdam (2011) Google Scholar
  23. 23.
    Csurka, G., Renders, J., Jacquet, G.: XRCE’s Participation at Patent Image Classification and Image-based Patent Retrieval Tasks of the Clef-IP 2011. In: Proceedings of CLEF 2011, Amsterdam (2011)Google Scholar
  24. 24.
    Vrochidis, S., Moumtzidou, A., Kompatsiaris, I.: Concept-based Patent Image Retrieval. World Patent Information Journal 34(4), 292–303 (2012)CrossRefGoogle Scholar
  25. 25.
    De Marco, D.: Mechanical Patent Searching: A Moving Target. In: Patent Information Users Group (PIUG), Baltimore, USA (2010)Google Scholar
  26. 26.
    Hoenes, F., Lichter, J.: Layout Extraction of Mixed Mode Documents. Mach. Vision Appl. 7, 237–246 (1994)CrossRefGoogle Scholar
  27. 27.
    Porter, M.F.: An Algorithm for Suffix Stripping. Program 14(3), 130–137 (1980)CrossRefGoogle Scholar
  28. 28.
    The Lemur Toolkit lemur,
  29. 29.
    Boser, B.E., Guyon, I.M., Va, V.N.: A Training Algorithm for Optimal Margin Classifiers. In: Proceedings of the 5th Annual Workshop on Computational Learning Theory (COLT 1992), pp. 144–152. ACM Press, New York (1992)CrossRefGoogle Scholar
  30. 30.
    Chang, C., Lin, C.: LIBSVM: A Library for Support Vector Machines. Software available at:
  31. 31.
    Izquierdo, E., Casas, J., Leonardi, R., Migliorati, P., O’Connor, N., Kompatsiaris, I., Strintzis, M.G.: Advanced Content-Based Semantic Scene Analysis and Information Retrieval: The Schema Project. In: Proceedings Workshop on Image Analysis for Multimedia Interactive Services, London, UK, pp. 519–528 (2003)Google Scholar
  32. 32.
    Vrochidis, S., Moumtzidou, A., Ypma, G., Kompatsiaris, I.: PatMedia: Augmenting Patent Search with Content-based Image Retrieval. In: Proceedings of the 5th IRF Conference, Austria, Vienna (2012)Google Scholar

Copyright information

© Springer International Publishing Switzerland 2014

Authors and Affiliations

  • Stefanos Vrochidis
    • 1
  • Anastasia Moumtzidou
    • 1
  • Ioannis Kompatsiaris
    • 1
  1. 1.Centre for Research & Technology Hellas - Information Technologies InstituteThessalonikiGreece

Personalised recommendations