Integration of Images into the Patent Retrieval Process

  • Wiebke ThodeEmail author
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11438)


Specialized patent retrieval systems mostly use textual information which is difficult enough because of the specialized characteristics of the text. However, in patents there also are drawings which show the invention. Empirical research has shown that patent experts can use these images to determine relevance very quickly. However, these drawings are binary and sometimes abstract and other times very specific; therefore there has not been an effective way to include the visual information into the information retrieval process. In addition, the number and the quality of drawings differs vastly even inside patent classes. This work focuses on the inclusion of images into the patent retrieval process using a combination of visual and textual information. With this multimodal approach it will hopefully be possible to achieve better results than just by using one modality individually. The goal is to develop a prototypical system using an iterative user-centered process.


Patent retrieval Multimodal information retrieval Visual information seeking 


  1. 1.
    Azzopardi, L., Joho, H., Vanderbauwhede, W.: A survey on patent users search behavior, search functionality and system requirements (2010).
  2. 2.
    Balaneshin-kordan, S., Kotov, A.: Deep neural architecture for multi-modal retrieval based on joint embedding space for text and images. In: Proceedings of the Eleventh ACM International Conference on Web Search and Data Mining, WSDM 2018, pp. 28–36. ACM, New York (2018).
  3. 3.
    Bhatti, N., Hanbury, A.: Image search in patents: a review. Int. J. Doc. Anal. Recogn. (IJDAR) 16(4), 309–329 (2013). Scholar
  4. 4.
    Bhatti, N., Hanbury, A., Stottinger, J.: Contextual local primitives for binary patent image retrieval. Multimedia Tools Appl. 77(7), 9111–9151 (2018). Scholar
  5. 5.
    Gordo, A., Almazán, J., Revaud, J., Larlus, D.: End-to-end learning of deepvisual representations for image retrieval. Int. J. Comput. Vis. 124(2), 237–254 (2017). Scholar
  6. 6.
    Hansen, P.: Task-based information seeking and retrieval in the patent domain: processes and relationships. Ph.d. thesis. University of Tampere, Tampere (2011)Google Scholar
  7. 7.
    Jürgens, J.J., Hansen, P., Womser-Hacker, C.: Going beyond CLEF-IP: the ‘reality’ for patent searchers? In: Catarci, T., Forner, P., Hiemstra, D., Peñas, A., Santucci, G. (eds.) CLEF 2012. LNCS, vol. 7488, pp. 30–35. Springer, Heidelberg (2012). Scholar
  8. 8.
    Kravets, A., Lebedev, N., Legenchenko, M.: Patents images retrieval and convolutional neural network training dataset quality improvement. In: Proceedings of the IV International Research Conference Information Technologies in Science, Management, Social Sphere and Medicine (ITSMSSM 2017). Atlantis Press, Paris, France, 12 May 2017–12 August 2017.
  9. 9.
    Lupu, M., Schuster, R., Mörzinger, R., Piroi, F., Schleser, T., Hanbury, A.: Patent images - a glass-encased tool. In: Lindstaedt, S., Granitzer, M. (eds.) Proceedings of the 12th International Conference on Knowledge Management and Knowledge Technologies - i-KNOW 2012, p. 1. ACM Press, New York, New York, USA (2012).
  10. 10.
    Shalaby, W., Zadrozny, W.: Patent retrieval: a literature review.
  11. 11.
    Vrochidis, S., Papadopoulos, S., Moumtzidou, A., Sidiropoulos, P., Pianta, E., Kompatsiaris, I.: Towards content-based patent image retrieval: a framework perspective. World Patent Inf. 32(2), 94–106 (2010). Scholar

Copyright information

© Springer Nature Switzerland AG 2019

Authors and Affiliations

  1. 1.University of HildesheimHildesheimGermany

Personalised recommendations