Taxonomy-Regularized Semantic Deep Convolutional Neural Networks

  • Wonjoon Goo
  • Juyong Kim
  • Gunhee Kim
  • Sung Ju HwangEmail author
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 9906)


We propose a novel convolutional network architecture that abstracts and differentiates the categories based on a given class hierarchy. We exploit grouped and discriminative information provided by the taxonomy, by focusing on the general and specific components that comprise each category, through the min- and difference-pooling operations. Without using any additional parameters or substantial increase in time complexity, our model is able to learn the features that are discriminative for classifying often confused sub-classes belonging to the same superclass, and thus improve the overall classification performance. We validate our method on CIFAR-100, Places-205, and ImageNet Animal datasets, on which our model obtains significant improvements over the base convolutional networks.


Deep learning Object categorization Taxonomy Ontology 



This work was supported by Samsung Research Funding Center of Samsung Electronics under Project Number SRFC-IT1502-03.

Supplementary material

419974_1_En_6_MOESM1_ESM.pdf (1.4 mb)
Supplementary material 1 (pdf 1389 KB)


  1. 1.
    Bengio, S., Weston, J., Grangier, D.: Label embedding trees for large multi-class task. In: NIPS (2010)Google Scholar
  2. 2.
    Deng, J., Ding, N., Jia, Y., Frome, A., Murphy, K., Bengio, S., Li, Y., Neven, H., Adam, H.: Large-scale object classification using label relation graphs. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) ECCV 2014. LNCS, vol. 8689, pp. 48–64. Springer, Heidelberg (2014). doi: 10.1007/978-3-319-10590-1_4 Google Scholar
  3. 3.
    Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., Fei-Fei, L.: ImageNet: a large-scale hierarchical image database. In: CVPR (2009)Google Scholar
  4. 4.
    Frome, A., Corrado, G., Shlens, J., Bengio, S., Dean, J., Ranzato, M., Mikolov, T.: Devise: a deep visual-semantic embedding model. In: NIPS (2013)Google Scholar
  5. 5.
    Gao, T., Koller, D.: Discriminative learning of relaxed hierarchy for large-scale visual recognition. In: ICCV (2011)Google Scholar
  6. 6.
    Hwang, S.J.: Discriminative object categorization with external semantic knowledge. Ph.D. dissertation, The University of Texas at Austin (2013)Google Scholar
  7. 7.
    Hwang, S.J., Grauman, K., Sha, F.: Learning a tree of metrics with disjoint visual features. In: NIPS (2011)Google Scholar
  8. 8.
    Hwang, S.J., Grauman, K., Sha, F.: Semantic kernel forests from multiple taxonomies. In: NIPS (2012)Google Scholar
  9. 9.
    Hwang, S.J., Sigal, L.: A unified semantic embedding: relating taxonomies and attributes. In: NIPS (2014)Google Scholar
  10. 10.
    Jia, Y.: Caffe: an open source convolutional architecture for fast feature embedding. arXiv:1408.5093 (2013)
  11. 11.
    Krizhevsky, A.: Learning multiple layers of features from tiny images. Master’s thesis, University of Toronto (2009)Google Scholar
  12. 12.
    Krizhevsky, A., Sutskever, I., Hinton, G.E.: ImageNet classification with deep convolutional neural networks. In: NIPS (2012)Google Scholar
  13. 13.
    Lecun, Y., Bottou, L., Bengio, Y., Haffner, P.: Gradient-based learning applied to document recognition. Proc. IEEE 86(11), 2278–2324 (1998)CrossRefGoogle Scholar
  14. 14.
    Lin, M., Chen, Q., Yan, S.: Network in network. In: ICLR (2014)Google Scholar
  15. 15.
    Marszałek, M., Schmid, C.: Constructing category hierarchies for visual recognition. In: Forsyth, D., Torr, P., Zisserman, A. (eds.) ECCV 2008. LNCS, vol. 5305, pp. 479–491. Springer, Heidelberg (2008). doi: 10.1007/978-3-540-88693-8_35 CrossRefGoogle Scholar
  16. 16.
    Miller, G.A., Beckwith, R., Fellbaum, C.D., Gross, D., Miller, K.: WordNet: an online lexical database. Int. J. Lexicograph 3(4), 235–244 (1990)CrossRefGoogle Scholar
  17. 17.
    Salakhutdinov, R.R., Tenenbaum, J.B., Torralba, A.: Learning to learn with compound HD models. In: NIPS (2011)Google Scholar
  18. 18.
    Simonyan, K., Zisserman, A.: Imagenet classification with deep convolutional neural networks. In: ICLR (2015)Google Scholar
  19. 19.
    Srivastava, N., Salakhutdinov, R.R.: Discriminative transfer learning with tree-based priors. In: NIPS (2013)Google Scholar
  20. 20.
    Stollenga, M.F., Masci, J., Gomez, F., Schmidhuber, J.: Deep networks with internal selective attention through feedback connections. In: NIPS (2014)Google Scholar
  21. 21.
    Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., Rabinovich, A.: Going deeper with convolutions. In: CVPR (2015)Google Scholar
  22. 22.
    Xiao, T., Zhang, J., Yang, K., Peng, Y., Zhang, Z.: Error-driven incremental learning in deep convolutional neural network for large-scale image classification. In: ACM MM (2014)Google Scholar
  23. 23.
    Yan, Z., Zhang, H., Jagadeesh, V., DeCoste, D., Di, W., Yu, Y.: HD-CNN: hierarchical deep convolutional neural network for image classification. In: ICCV (2015)Google Scholar
  24. 24.
    Zhao, B., Fei-Fei, L., Xing, E.P.: Large-scale category structure aware image categorization. In: NIPS (2011)Google Scholar
  25. 25.
    Zhou, B., Lapedriza, A., Xiao, J., Torralba, A., Oliva, A.: Learning deep features for scene recognition using places database. In: NIPS (2014)Google Scholar
  26. 26.
    Zhou, D., Xiao, L., Wu, M.: Hierarchical classification via orthogonal transfer. In: ICML (2011)Google Scholar
  27. 27.
    Zhou, Y., Jin, R., Hoi, S.C.H.: Exclusive lasso for multi-task feature selection. JMLR 9, 988–995 (2010)Google Scholar

Copyright information

© Springer International Publishing AG 2016

Authors and Affiliations

  • Wonjoon Goo
    • 1
  • Juyong Kim
    • 1
  • Gunhee Kim
    • 1
  • Sung Ju Hwang
    • 2
    Email author
  1. 1.Computer Science and EngineeringSeoul National UniversitySeoulKorea
  2. 2.School of Electrical and Computer EngineeringUNISTUlsanSouth Korea

Personalised recommendations