Advertisement

ReDro: Efficiently Learning Large-Sized SPD Visual Representation

Conference paper
  • 484 Downloads
Part of the Lecture Notes in Computer Science book series (LNCS, volume 12360)

Abstract

Symmetric positive definite (SPD) matrix has recently been used as an effective visual representation. When learning this representation in deep networks, eigen-decomposition of covariance matrix is usually needed for a key step called matrix normalisation. This could result in significant computational cost, especially when facing the increasing number of channels in recent advanced deep networks.

This work proposes a novel scheme called Relation Dropout (ReDro). It is inspired by the fact that eigen-decomposition of a block diagonal matrix can be efficiently obtained by decomposing each of its diagonal square matrices, which are of smaller sizes. Instead of using a full covariance matrix as in the literature, we generate a block diagonal one by randomly grouping the channels and only considering the covariance within the same group. We insert ReDro as an additional layer before the step of matrix normalisation and make its random grouping transparent to all subsequent layers. Additionally, we can view the ReDro scheme as a dropout-like regularisation, which drops the channel relationship across groups. As experimentally demonstrated, for the SPD methods typically involving the matrix normalisation step, ReDro can effectively help them reduce computational cost in learning large-sized SPD visual representation and also help to improve image recognition performance.

Keywords

Block diagonal matrix Covariance Eigen-decomposition SPD representation Fine-grained image recognition 

Notes

Acknowledgement

This work was supported by the CSIRO Data61 Scholarship; the University of Wollongong Australia IPTA scholarship; the Australian Research Council (grant number DP200101289); and the Multi-modal Australian ScienceS Imaging and Visualisation Environment (MASSIVE).

Supplementary material

504470_1_En_1_MOESM1_ESM.pdf (357 kb)
Supplementary material 1 (pdf 356 KB)

References

  1. 1.
    Brooks, D., Schwander, O., Barbaresco, F., Schneider, J.Y., Cord, M.: Riemannian batch normalization for SPD neural networks. In: Advances in Neural Information Processing Systems, pp. 15489–15500 (2019)Google Scholar
  2. 2.
    Chakraborty, R., Bouza, J., Manton, J., Vemuri, B.C.: A deep neural network for manifold-valued data with applications to neuroimaging. In: Chung, A.C.S., Gee, J.C., Yushkevich, P.A., Bao, S. (eds.) IPMI 2019. LNCS, vol. 11492, pp. 112–124. Springer, Cham (2019).  https://doi.org/10.1007/978-3-030-20351-1_9CrossRefGoogle Scholar
  3. 3.
    Cimpoi, M., Maji, S., Vedaldi, A.: Deep filter banks for texture recognition and segmentation. In: Proceedings of the International Conference on Computer Vision and Pattern Recognition, pp. 3828–3836. IEEE (2015)Google Scholar
  4. 4.
    Cui, Y., Zhou, F., Wang, J., Liu, X., Lin, Y., Belongie, S.: Kernel pooling for convolutional neural networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2921–2930 (2017)Google Scholar
  5. 5.
    Demmel, J., Dumitriu, I., Holtz, O.: Fast linear algebra is stable. Numer. Math. 108(1), 59–91 (2007)MathSciNetCrossRefGoogle Scholar
  6. 6.
    Engin, M., Wang, L., Zhou, L., Liu, X.: DeepKSPD: learning kernel-matrix-based SPD representation for fine-grained image recognition. In: Ferrari, V., Hebert, M., Sminchisescu, C., Weiss, Y. (eds.) ECCV 2018. LNCS, vol. 11206, pp. 629–645. Springer, Cham (2018).  https://doi.org/10.1007/978-3-030-01216-8_38CrossRefGoogle Scholar
  7. 7.
    Gao, Y., Beijbom, O., Zhang, N., Darrell, T.: Compact bilinear pooling. In: Proceedings of the Conference on Computer Vision and Pattern Recognition, pp. 317–326. IEEE (2016)Google Scholar
  8. 8.
    Ghiasi, G., Lin, T.Y., Le, Q.V.: Dropblock: a regularization method for convolutional networks. In: Proceedings of the Advances in Neural Information Processing Systems, pp. 10727–10737 (2018)Google Scholar
  9. 9.
    Hou, S., Wang, Z.: Weighted channel dropout for regularization of deep convolutional neural network. Proc. AAAI Conf. Artif. Intell. 33, 8425–8432 (2019)Google Scholar
  10. 10.
    Huang, Z., Van Gool, L.: A Riemannian network for SPD matrix learning. In: Thirty-First AAAI Conference on Artificial Intelligence (2017)Google Scholar
  11. 11.
    Ionescu, C., Vantzos, O., Sminchisescu, C.: Matrix backpropagation for deep networks with structured layers. In: Proceedings of the International Conference on Computer Vision, pp. 2965–2973. IEEE (2015)Google Scholar
  12. 12.
    Jayasumana, S., Hartley, R., Salzmann, M., Li, H., Harandi, M.: Kernel methods on Riemannian manifolds with Gaussian RBF kernels. IEEE Trans. Pattern Anal. Mach. Intell. 37(12), 2464–2477 (2015) CrossRefGoogle Scholar
  13. 13.
    Kong, S., Fowlkes, C.: Low-rank bilinear pooling for fine-grained classification. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 365–374 (2017)Google Scholar
  14. 14.
    Koniusz, P., Wang, L., Cherian, A.: Tensor representations for action recognition. IEEE Trans. Pattern Anal. Mach. Intell. (2020)Google Scholar
  15. 15.
    Koniusz, P., Zhang, H.: Power normalizations in fine-grained image, few-shot image and graph classification. IEEE Trans. Pattern Anal. Mach. Intell. (2020)Google Scholar
  16. 16.
    Koniusz, P., Zhang, H., Porikli, F.: A deeper look at power normalizations. In: 2018 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2018, Salt Lake City, UT, USA, 18–22 June 2018, pp. 5774–5783. IEEE Computer Society (2018).  https://doi.org/10.1109/CVPR.2018.00605
  17. 17.
    Krause, J., Stark, M., Deng, J., Fei-Fei, L.: 3D object representations for fine-grained categorization. In: Proceedings of the International Conference on Computer Vision Workshops, pp. 554–561. IEEE (2013)Google Scholar
  18. 18.
    Li, P., Xie, J., Wang, Q., Gao, Z.: Towards faster training of global covariance pooling networks by iterative matrix square root normalization. In: Proceedings of the Conference on Computer Vision and Pattern Recognition, pp. 947–955. IEEE (2018)Google Scholar
  19. 19.
    Li, P., Xie, J., Wang, Q., Zuo, W.: Is second-order information helpful for large-scale visual recognition? In: Proceedings of the International Conference on Computer Vision, pp. 2070–2078. IEEE (2017)Google Scholar
  20. 20.
    Lin, T.Y., Maji, S.: Improved Bilinear Pooling with CNNs. arXiv preprint arXiv:1707.06772 (2017)
  21. 21.
    Lin, T.Y., RoyChowdhury, A., Maji, S.: Bilinear CNN models for fine-grained visual recognition. In: Proceedings of the International Conference on Computer Vision, pp. 1449–1457. IEEE (2015)Google Scholar
  22. 22.
    Maji, S., Rahtu, E., Kannala, J., Blaschko, M., Vedaldi, A.: Fine-grained visual classification of aircraft. arXiv preprint arXiv:1306.5151 (2013)
  23. 23.
    Ng, A.Y., Jordan, M.I., Weiss, Y.: On spectral clustering: analysis and an algorithm. In: Proceedings of the Advances in Neural Information Processing Systems, pp. 849–856 (2002)Google Scholar
  24. 24.
    Nguyen, X.S., Brun, L., Lézoray, O., Bougleux, S.: A neural network based on SPD manifold learning for skeleton-based hand gesture recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 12036–12045 (2019)Google Scholar
  25. 25.
    Porikli, F., Tuzel, O., Meer, P.: Covariance tracking using model update based on lie algebra. In: Proceedings of the Conference on Computer Vision and Pattern Recognition, vol. 1, pp. 728–735. IEEE (2006)Google Scholar
  26. 26.
    Quattoni, A., Torralba, A.: Recognizing indoor scenes. In: Proceedings of the Conference on Computer Vision and Pattern Recognition, pp. 413–420. IEEE (2009)Google Scholar
  27. 27.
    Srivastava, N., Hinton, G., Krizhevsky, A., Sutskever, I., Salakhutdinov, R.: Dropout: a simple way to prevent neural networks from overfitting. J. Mach. Learn. Res. 15(1), 1929–1958 (2014)MathSciNetzbMATHGoogle Scholar
  28. 28.
    Tompson, J., Goroshin, R., Jain, A., LeCun, Y., Bregler, C.: Efficient object localization using convolutional networks. In: Proceedings of the Conference on Computer Vision and Pattern Recognition, pp. 648–656. IEEE (2015)Google Scholar
  29. 29.
    Tuzel, O., Porikli, F., Meer, P.: Region covariance: a fast descriptor for detection and classification. In: Leonardis, A., Bischof, H., Pinz, A. (eds.) ECCV 2006. LNCS, vol. 3952, pp. 589–600. Springer, Heidelberg (2006).  https://doi.org/10.1007/11744047_45CrossRefGoogle Scholar
  30. 30.
    Tuzel, O., Porikli, F., Meer, P.: Human detection via classification on riemannian manifolds. In: Proceedings of the Conference on Computer Vision and Pattern Recognition, pp. 1–8. IEEE (2007)Google Scholar
  31. 31.
    Vedaldi, A., Lenc, K.: MatConvNet: convolutional neural networks for Matlab. In: Proceedings of the 23rd ACM International Conference on Multimedia, pp. 689–692 (2015)Google Scholar
  32. 32.
    Wang, Q., Xie, J., Zuo, W., Zhang, L., Li, P.: Deep CNNs meet global covariance pooling: better representation and generalization. IEEE Trans. Pattern Anal. Mach. Intell. (2020)Google Scholar
  33. 33.
    Wang, R., Guo, H., Davis, L.S., Dai, Q.: Covariance discriminative learning: A natural and efficient approach to image set classification. In: Proceedings of the Conference on Computer Vision and Pattern Recognition, pp. 2496–2503. IEEE (2012)Google Scholar
  34. 34.
    Welinder, P., et al.: Caltech-UCSD birds 200. Technical report CNS-TR-2010-001, California Institute of Technology (2010)Google Scholar
  35. 35.
    Yu, K., Salzmann, M.: Second-order convolutional neural networks. arXiv preprint arXiv:1703.06817 (2017)
  36. 36.
    Zhang, H., Zhang, J., Koniusz, P.: Few-shot learning via saliency-guided hallucination of samples. In: Proceedings of the Conference on Computer Vision and Pattern Recognition, pp. 2770–2779. IEEE (2019)Google Scholar
  37. 37.
    Zhang, H., Zhang, L., Qui, X., Li, H., Torr, P.H.S., Koniusz, P.: Few-shot action recognition with permutation-invariant attention. In: Proceedings of the European Conference on Computer Vision (ECCV) (2020)Google Scholar
  38. 38.
    Zheng, H., Fu, J., Zha, Z.J., Luo, J.: Learning deep bilinear transformation for fine-grained image representation. In: Proceedings of the Advances in Neural Information Processing Systems, pp. 4279–4288 (2019)Google Scholar
  39. 39.
    Zhu, X., Xu, C., Hui, L., Lu, C., Tao, D.: Approximated bilinear modules for temporal modeling. In: Proceedings of the International Conference on Computer Vision, pp. 3494–3503. IEEE (2019)Google Scholar

Copyright information

© Springer Nature Switzerland AG 2020

Authors and Affiliations

  1. 1.VILA, School of Computing and Information TechnologyUniversity of WollongongWollongongAustralia
  2. 2.CSIRO Data61EppingAustralia
  3. 3.School of Electrical and Information EngineeringUniversity of SydneySydneyAustralia

Personalised recommendations