Deep CNN with Graph Laplacian Regularization for Multi-label Image Annotation

  • Jonathan MojooEmail author
  • Keiichi Kurosawa
  • Takio Kurita
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 10317)


To compensate for incomplete or imprecise tags in training samples, this paper proposes a learning algorithm for the convolutional neural network (CNN) for multi-label image annotation by introducing co-occurrence dependency between tags as a graph Laplacian regularization term. To exploit the co-occurrence dependency, we apply Hayashi’s quantification method-type III to the tags in the training samples and use the distances between the acquired representative vectors to define the weights for graph Laplacian regularization. By introducing this regularization term, the possibility of co-occurrence between tags with high co-occurrence frequency can be increased. To confirm the effectiveness of the proposed algorithm, we have done experiments using Corel5k’s dataset for multi-label image annotation.


Convolutional neural network Multi-label classification Laplacian regularization Hayashi’s quantification method Co-occurrence between tags 



This work was partly supported by JSPS KAKENHI Grant Number 16K00239.


  1. 1.
  2. 2.
  3. 3.
    Ames, M., Naaman, M.: Why we tag: motivations for annotation in mobile and online media. In: Proceedings of the Conference on Human Factors in Computing Systems, pp. 971–980. ACM (2007)Google Scholar
  4. 4.
    Guillaumin, M., Mensink, T., Verbeek, J., Schmid, C.: TagProp: discriminative metric learning in nearest neighbor models for image auto-annotation. In: 2009 International Conference Computer Vision, pp. 309–316. IEEE (2009)Google Scholar
  5. 5.
    Barnard, K., Duygulu, P., Forsyth, D., Freitas, N.D., Blei, D.M., Jordan, M.I.: Matching words and pictures. J. Mach. Learn. Res. 3(Feb), 1107–1135 (2003)zbMATHGoogle Scholar
  6. 6.
    Grangier, D., Bengio, S.: A discriminative kernel-based approach to rank images from text queries. IEEE Trans. Pattern Anal. Mach. Intell. 30(8), 1371–1384 (2008)CrossRefGoogle Scholar
  7. 7.
    Hertz, T., Bar-Hillel, A., Weinshall, D.: Learning distance functions for image retrieval. In: 2004 Computer Society Conference on Computer Vision and Pattern Recognition, vol. 2, p. II. IEEE (2004)Google Scholar
  8. 8.
    Makadia, A., Pavlovic, V., Kumar, S.: A new baseline for image annotation. In: Forsyth, D., Torr, P., Zisserman, A. (eds.) ECCV 2008. LNCS, vol. 5304, pp. 316–329. Springer, Heidelberg (2008). doi: 10.1007/978-3-540-88690-7_24 CrossRefGoogle Scholar
  9. 9.
    LeCun, Y., Boser, B., Denker, J.S., Henderson, D., Howard, R.E., Hubbard, W., Jackel, L.D.: Backpropagation applied to handwritten zip code recognition. Neural Comput. 1(4), 541–551 (1989)CrossRefGoogle Scholar
  10. 10.
    Krizhevsky, A., Sutskever, I., Hinton, G.E.: Imagenet classification with deep convolutional neural networks. In: Advances in Neural Information Processing Systems, pp. 1097–1105 (2012)Google Scholar
  11. 11.
    Gong, Y., Jia, Y., Leung, T., Toshev, A., Ioffe, S.: Deep convolutional ranking for multilabel image annotation arXiv:1312.4894 (2013)
  12. 12.
    Wei, Y., Xia, W., Huang, J., Ni, B., Dong, J., Zhao, Y., Yan, S.: CNN: single-label to multi-label arXiv:1406.5726 (2014)
  13. 13.
    Xue, X., Zhang, W., Zhang, J., Wu, B., Fan, J., Lu, Y.: Correlative multi-label multi-instance image annotation. In: 2011 IEEE International Conference Computer Vision, pp. 651–658. IEEE (2011)Google Scholar
  14. 14.
    Guo, Y., Gu, S.: Multi-label classification using conditional dependency networks. In: IJCAI Proceedings-International Joint Conference on Artificial Intelligence, vol. 22, p. 1300 (2011)Google Scholar
  15. 15.
    Wang, J., Yang, Y., Mao, J., Huang, Z., Huang, C., Xu, W.: CNN-RNN: a unified framework for multi-label image classification. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 2285–2294 (2016)Google Scholar
  16. 16.
    Hayashi, C.: Multidimensional quantification. I. Proc. Jpn. Acad. 30(2), 61–65 (1954)MathSciNetCrossRefzbMATHGoogle Scholar
  17. 17.
    Benzecri, J.P.: Lanalyse des donnees, tome II. Lanalyse des correspondances (1973)Google Scholar
  18. 18.
    Nishisato, S.: Analysis of Categorical Data: Dual Scaling and Its Applications. JSTOR (1980)Google Scholar
  19. 19.
    Belkin, M., Niyogi, P.: Laplacian eigenmaps and spectral techniques for embedding and clustering. In: NIPS, vol. 14, pp. 585–591 (2001)Google Scholar
  20. 20.
    Watanabe, K., Kurita, T.: Locality preserving multi-nominal logistic regression. In: ICPR, pp. 1–4. IEEE (2008)Google Scholar

Copyright information

© Springer International Publishing AG 2017

Authors and Affiliations

  • Jonathan Mojoo
    • 1
    Email author
  • Keiichi Kurosawa
    • 2
  • Takio Kurita
    • 1
  1. 1.Department of Information EngineeringHiroshima UniversityHiroshimaJapan
  2. 2.Faculty of EngineeringHiroshima UniversityHiroshimaJapan

Personalised recommendations