Skip to main content
SpringerLink
Log in

Optimizing Neural Networks for Patent Classification

  • Conference paper
  • First Online:
Machine Learning and Knowledge Discovery in Databases (ECML PKDD 2019)

Part of the book series: Lecture Notes in Computer Science ((LNAI,volume 11908))

  • 1959 Accesses

Abstract

A great number of patents is filed everyday to the patent offices worldwide. Each of these patents has to be labeled by domain experts with one or many of thousands of categories. This process is not only extremely expensive but also overwhelming for the experts, due to the considerable increase of filed patents over the years and the increasing complexity of the hierarchical categorization structure. Therefore, it is critical to automate the manual classification process using a classification model. In this paper, the automation of the task is carried out based on recent advances in deep learning for NLP and compared to customized approaches. Moreover, an extensive optimization analysis grants insights about hyperparameter importance. Our optimized convolutional neural network achieves a new state-of-the-art performance of \(55.02\%\) accuracy on the public Wipo-Alpha dataset.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 39.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    https://www.epo.org/applying/basics.html (accessed June 20, 2019).

  2. 2.

    https://www.uspto.gov/web/offices/ac/ido/oeip/taf/us_stat.htm (accessed June 20, 2019).

  3. 3.

    https://keras.io/preprocessing/text/ (accessed June 20, 2019).

  4. 4.

    https://www.wipo.int/classifications/ipc/en/ITsupport/Categorization/dataset/index.html (accessed May 4, 2019).

  5. 5.

    http://www.ifs.tuwien.ac.at/~clef-ip/download/2011/index.shtml (accessed May 4, 2019).

  6. 6.

    Specifications of the used machine: OS: CentOS Linux 7.5, RAM: 32 GB Kingston HyperX Fury DDR4, CPU: Intel Core i7-7700, GPU: MSI GeForce GTX 1080 Ti Gaming X 11G.

  7. 7.

    https://github.com/google-research/bert#sentence-and-sentence-pair-classifica-tion-tasks (accessed June 24, 2019).

  8. 8.

    https://github.com/lo2aayy/patent-classification.

References

  1. Benzineb, K., Guyot, J.: Automated patent classification. In: Lupu, M., Mayer, K., Tait, J., Trippe, A. (eds.) Current Challenges in Patent Information Retrieval, pp. 239–261. Springer, Heidelberg (2011). https://doi.org/10.1007/978-3-642-19231-9_12

    Chapter  Google Scholar 

  2. Bojanowski, P., Grave, E., Joulin, A., Mikolov, T.: Enriching word vectors with subword information. Trans. Assoc. Comput. Ling. 5, 135–146 (2017)

    Google Scholar 

  3. Chung, J., Gulcehre, C., Cho, K., Bengio, Y.: Empirical evaluation of gated recurrent neural networks on sequence modeling. In: NIPS 2014 Workshop on Deep Learning (2014)

    Google Scholar 

  4. Caselles-Dupré, H., Lesaint, F., Royo-Letelier, J.: Word2Vec applied to recommendation: hyperparameters matter. In: Proceedings of the 12th ACM Conference on Recommender Systems, pp. 352–356. ACM (2016)

    Google Scholar 

  5. Conneau, A., Schwenk, H., Barrault, L., Lecun, Y.: Very deep convolutional networks for text classification. In: Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics, vol. 1, pp. 1107–1116. ACL (2017)

    Google Scholar 

  6. Devlin, J., Chang, M.W., Lee, K., Toutanova, K.: BERT: pre-training of deep bidirectional transformers for language understanding. In: Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, vol. 1, pp. 4171–4186. ACL (2018)

    Google Scholar 

  7. Falkner, S., Klein, A., Hutter, F.: BOHB: robust and efficient hyperparameter optimization at scale. In: Proceedings of the 35th International Conference on Machine Learning, PMLR, vol. 80, pp. 1437–1446 (2018)

    Google Scholar 

  8. Fall, C.J., Törcsvári, A., Benzineb, K., Karetka, G.: Automated categorization in the international patent classification. In: SIGIR Forum, vol. 1, pp. 10–25. ACM (2003)

    Google Scholar 

  9. Fall, C.J., Benzineb, K.: Literature survey: issues to be considered in the automatic classification of patents. In: World Intellectual Property Organization, vol. 29 (2002)

    Google Scholar 

  10. Fernández-Delgado, M., Cernadas, E., Barro, S., Amorim, D.: Do we need hundreds of classifiers to solve real world classification problems? J. Mach. Learn. Res. 15, 3133–3181 (2014)

    MathSciNet  MATH  Google Scholar 

  11. Gomez, J.C., Moens, M.-F.: A survey of automated hierarchical classification of patents. In: Paltoglou, G., Loizides, F., Hansen, P. (eds.) Professional Search in the Modern World. LNCS, vol. 8830, pp. 215–249. Springer, Cham (2014). https://doi.org/10.1007/978-3-319-12511-4_11

    Chapter  Google Scholar 

  12. Grawe, M.F., Martins, C.A., Bonfante, A.G.: Automated patent classification using word embedding. In: 16th IEEE International Conference on Machine Learning and Applications (ICMLA), pp. 408–411. IEEE (2017)

    Google Scholar 

  13. Howard, J., Ruder, S.: Universal language model fine-tuning for text classification. In: Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, ACL, vol. 1, pp. 328–339 (2018)

    Google Scholar 

  14. Huang, G., Liu, Z., Van Der Maaten, L., Weinberger, K.Q.: Densely connected convolutional networks. In: 30th IEEE Conference on Computer Vision and Pattern Recognition, pp. 2261–2269. IEEE (2017)

    Google Scholar 

  15. Hutter, F., Hoos, H.H., Leyton-Brown, K.: An efficient approach for assessing hyperparameter importance. In: ICML, JMLR Workshop and Conference Proceedings, PMLR, vol. 32, pp. 754–762 (2014)

    Google Scholar 

  16. Joulin, A., Grave, E., Bojanowski, P., Mikolov, T.: Bag of tricks for efficient text classification. In: Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics, vol. 2, pp. 427–431. ACL (2017)

    Google Scholar 

  17. Kalchbrenner, N., Grefenstette, E., Blunsom, P.: A convolutional neural network for modelling sentences. In: Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics, vol. 1, pp. 655–665. ACL (2014)

    Google Scholar 

  18. Kim, Y.: Convolutional neural networks for sentence classification. In: Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing, pp. 1746–1751. ACL (2014)

    Google Scholar 

  19. Kowsari, K., Brown, D.E., Heidarysafa, M., Meimandi, K.J., Gerber, M.S., Barnes, L.E.: HDLTex: hierarchical deep learning for text classification. In: 16th IEEE International Conference on Machine Learning and Applications, pp. 364–371. IEEE (2017)

    Google Scholar 

  20. Leopold, E., Kindermann, J.: Text categorization with support vector machines. How to represent texts in input space? Mach. Learn. 46, 423–444 (2002). https://doi.org/10.1023/A:1012491419635

    Article  MATH  Google Scholar 

  21. Mikolov, T., Chen, K., Corrado, G., Dean, J.: Efficient estimation of word representations in vector space. In: CoPR (2013)

    Google Scholar 

  22. Pennington, J., Socher, R., Manning, C.: GloVe: global vectors for word representation. In: Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing, pp. 1532–1543. ACL (2014)

    Google Scholar 

  23. Peters, M.E., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., Zettlemoyer, L.: Deep contextualized word representations. In: Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 2227–2237. ACL (2018)

    Google Scholar 

  24. Risch, J., Krestel, R.: Domain-specific word embeddings for patent classification. In: Data Technologies and Applications, vol. 53, pp. 108–122. Emerald Publishing Limited (2019)

    Google Scholar 

  25. Steinwart, I., Christmann, A.: Support Vector Machines, 1st edn. Springer, New York (2008). https://doi.org/10.1007/978-0-387-77242-4

    Book  MATH  Google Scholar 

  26. Wu, S., Zhong, S., Liu, Y.: Deep residual learning for image steganalysis. Multimedia Tools Appl. 77(9), 10437–10453 (2017). https://doi.org/10.1007/s11042-017-4440-4

    Article  Google Scholar 

  27. Xiao, Y., Cho, K.: Efficient character-level document classification by combining convolution and recurrent layers. preprint arXiv:1602.00367 (2016)

  28. Yang, Z., Yang, D., Dyer, C., He, X., Smola, A., Hovy, E.: Hierarchical attention networks for document classification. In: Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 1480–1489. ACL (2016)

    Google Scholar 

  29. Zhang, H., Li, D.: Naïve bayes text classifier. In: 2007 IEEE International Conference on Granular Computing (GRC 2007), pp. 708–708. IEEE (2007)

    Google Scholar 

  30. Zhang, X., Zhao, J., LeCun, Y.: Character-level convolutional networks for text classification. In: Proceedings of the 28th International Conference on Neural Information Processing Systems, vol. 1, pp. 649–657. MIT Press (2015)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Averbis GmbH, Freiburg, Germany

    Louay Abdelgawad, Peter Kluegl & Erdan Genc

  2. Machine Learning Institute, Albert-Ludwigs University of Freiburg, Freiburg im Breisgau, Germany

    Stefan Falkner & Frank Hutter

Authors
  1. Louay Abdelgawad
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Peter Kluegl
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Erdan Genc
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Stefan Falkner
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Frank Hutter
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Louay Abdelgawad .

Editor information

Editors and Affiliations

  1. Leuphana University, Lüneburg, Germany

    Ulf Brefeld

  2. IRISA/Inria, Rennes, France

    Elisa Fromont

  3. University of Würzburg, Würzburg, Germany

    Andreas Hotho

  4. Leiden University, Leiden, The Netherlands

    Arno Knobbe

  5. ETH Zurich, Zurich, Switzerland

    Marloes Maathuis

  6. Institut National des Sciences Appliquées, Villeurbanne, France

    Céline Robardet

Rights and permissions

Reprints and permissions

Copyright information

© 2020 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Abdelgawad, L., Kluegl, P., Genc, E., Falkner, S., Hutter, F. (2020). Optimizing Neural Networks for Patent Classification. In: Brefeld, U., Fromont, E., Hotho, A., Knobbe, A., Maathuis, M., Robardet, C. (eds) Machine Learning and Knowledge Discovery in Databases. ECML PKDD 2019. Lecture Notes in Computer Science(), vol 11908. Springer, Cham. https://doi.org/10.1007/978-3-030-46133-1_41

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-46133-1_41

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-46132-4

  • Online ISBN: 978-3-030-46133-1

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Societies and partnerships

Search

Navigation